From 4c9beb05b7fcf6a95e1c9df1420f3321be9e0b34 Mon Sep 17 00:00:00 2001 From: Rahul Steiger Date: Wed, 10 Jun 2026 07:46:09 -0700 Subject: [PATCH 01/23] squashed with main Signed-off-by: Rahul Steiger --- docs/source/models/visual-generation.md | 25 +- examples/visual_gen/models/wan_t2v.py | 40 +- tensorrt_llm/_torch/modules/linear.py | 15 +- tensorrt_llm/_torch/visual_gen/config.py | 17 +- tensorrt_llm/_torch/visual_gen/executor.py | 53 ++ .../models/cosmos3/pipeline_cosmos3.py | 120 ++- .../models/cosmos3/transformer_cosmos3.py | 47 +- .../visual_gen/models/wan/pipeline_wan.py | 112 ++- .../visual_gen/models/wan/pipeline_wan_i2v.py | 2 +- tensorrt_llm/_torch/visual_gen/offloading.py | 697 ++++++++++++++++++ tensorrt_llm/_torch/visual_gen/pipeline.py | 71 +- .../_torch/visual_gen/pipeline_loader.py | 85 ++- tensorrt_llm/visual_gen/__init__.py | 2 + tensorrt_llm/visual_gen/args.py | 56 ++ .../test_lists/test-db/l0_b200.yml | 3 + .../visual_gen/test_cosmos3_pipeline.py | 2 +- .../visual_gen/test_cosmos3_t2v_offload.py | 216 ++++++ .../visual_gen/test_cosmos3_transformer.py | 7 +- .../_torch/visual_gen/test_offloading.py | 589 +++++++++++++++ .../visual_gen/test_wan22_t2v_offload.py | 257 +++++++ .../visual_gen/test_wan22_t2v_pipeline.py | 36 +- 21 files changed, 2350 insertions(+), 102 deletions(-) create mode 100644 tensorrt_llm/_torch/visual_gen/offloading.py create mode 100644 tests/unittest/_torch/visual_gen/test_cosmos3_t2v_offload.py create mode 100644 tests/unittest/_torch/visual_gen/test_offloading.py create mode 100644 tests/unittest/_torch/visual_gen/test_wan22_t2v_offload.py diff --git a/docs/source/models/visual-generation.md b/docs/source/models/visual-generation.md index d1e376b9a13e..37143a792d10 100644 --- a/docs/source/models/visual-generation.md +++ b/docs/source/models/visual-generation.md @@ -18,6 +18,7 @@ TensorRT-LLM **VisualGen** provides a unified inference stack for diffusion mode - Sparse attention support: see [VisualGen Sparse Attention](../visual-gen/features/sparse-attention.md). - Multi-GPU parallelism (CFG parallel, Ulysses sequence parallel, Tensor parallelism). - **Step caching** — two runtime caching backends (**TeaCache** and **Cache-DiT**) that skip transformer computation on steps where the step-to-step change is small. +- CPU offloading for Wan text-to-video pipelines to reduce peak GPU memory usage. - `trtllm-serve` integration with OpenAI-compatible API endpoints for image and video generation. ## Supported Models @@ -44,16 +45,16 @@ Models are auto-detected from the checkpoint directory. Diffusers-format models ### Feature Matrix -| Model | FP8 blockwise | NVFP4 | TeaCache | Cache-DiT | CFG Parallelism | Ulysses Parallelism | Parallel VAE | CUDA Graph | torch.compile | trtllm-serve | Attention2D | Ring Attention | Tensor Parallelism | VSA | -|---|---|---|---|---|---|---|---|---|---|---|--|--|--|--| -| **FLUX.1** | Yes | Yes | Yes | Yes | No [^1] | Yes | No | Yes | Yes | Yes | Yes | Yes | Yes | No | -| **FLUX.2** | Yes | Yes | Yes | Yes | No [^1] | Yes | No | Yes | Yes | Yes | Yes | Yes | Yes | No | -| **Wan 2.1** | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | -| **Wan 2.1 VSA** [^2] | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | No | Yes | Yes | -| **Wan 2.2** | Yes | Yes | Yes [^3] | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | -| **LTX-2** | Yes | Yes | Yes [^4] | Yes | Yes | Yes | No | No | Yes | Yes | Yes | Yes | No | No | -| **Qwen-Image** [^5] | Yes | Yes | No | No | No | Yes | No | Yes | Yes | Yes | Yes | Yes | No | No | -| **Cosmos3** | Yes | Yes | No | No | Yes | Yes | Yes | Yes | Yes | Yes | No | No | Yes | No | +| Model | FP8 blockwise | NVFP4 | TeaCache | Cache-DiT | CPU Offloading | CFG Parallelism | Ulysses Parallelism | Parallel VAE | CUDA Graph | torch.compile | trtllm-serve | Attention2D | Ring Attention | Tensor Parallelism | VSA | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| **FLUX.1** | Yes | Yes | Yes | Yes | No | No [^1] | Yes | No | Yes | Yes | Yes | Yes | Yes | Yes | No | +| **FLUX.2** | Yes | Yes | Yes | Yes | No | No [^1] | Yes | No | Yes | Yes | Yes | Yes | Yes | Yes | No | +| **Wan 2.1** | Yes | Yes | Yes | Yes | T2V only | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | +| **Wan 2.1 VSA** [^2] | Yes | Yes | Yes | Yes | T2V only | Yes | Yes | Yes | Yes | Yes | Yes | No | No | Yes | Yes | +| **Wan 2.2** | Yes | Yes | Yes [^3] | Yes | T2V only | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | +| **LTX-2** | Yes | Yes | Yes [^4] | Yes | No | Yes | Yes | No | No | Yes | Yes | Yes | Yes | No | No | +| **Qwen-Image** [^5] | Yes | Yes | No | No | No | No | Yes | No | Yes | Yes | Yes | Yes | Yes | No | No | +| **Cosmos3** | Yes | Yes | No | No | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | No | Yes | No | [^1]: FLUX models use embedded guidance and do not have a separate negative prompt path, so CFG parallelism is not applicable. @@ -267,6 +268,9 @@ attention_config: vsa_sparsity: 0.90 ``` +### CPU Offloading (Wan T2V only) + +CPU offloading stages move selected Wan T2V pipeline components between CPU and GPU to reduce peak GPU memory usage; enable it with `cpu_offload_config.enable: true` or `--enable_offloading` in the example script. ### Multi-GPU Parallelism @@ -280,6 +284,7 @@ Configured under `VisualGenArgs.parallel_config`. Modes can be combined: - **Attention2D** (`attn2d_size: [N, M]`): Shards the sequence axis across an `N × M` device mesh (CP degree = `N · M`; total SP degree = `N · M · ulysses_size`). - **Ring Attention** (`ring_size: N`): Shards the sequence axis across a 1D ring of `N` ranks, streaming K/V blocks (CP degree = `N`; total SP degree = `N · ulysses_size`; mutually exclusive with Attention2D). - **Tensor Parallelism** (`tp_size: N`): Splits attention heads and transformer MLPs across GPUs for faster compute and reduced memory usage. + ## Developer Guide ### Architecture Overview diff --git a/examples/visual_gen/models/wan_t2v.py b/examples/visual_gen/models/wan_t2v.py index d2530d918ee9..c3b7629773aa 100644 --- a/examples/visual_gen/models/wan_t2v.py +++ b/examples/visual_gen/models/wan_t2v.py @@ -18,6 +18,8 @@ Usage: python wan_t2v.py python wan_t2v.py --visual_gen_args ../configs/wan2.2-t2v-fp4-1gpu.yaml + python wan_t2v.py --enable_offloading \ + --offload_stages text_encoder,denoising_transformer,denoising_transformer_2,vae """ import argparse @@ -25,6 +27,15 @@ from tensorrt_llm import VisualGen, VisualGenArgs +def _parse_offload_stages(offload_stages: str | None) -> list[str] | None: + if offload_stages is None: + return None + stages = [stage.strip() for stage in offload_stages.split(",") if stage.strip()] + if not stages: + raise ValueError("--offload_stages must contain at least one stage name") + return stages + + def main(): parser = argparse.ArgumentParser(description="Wan Text-to-Video example") parser.add_argument( @@ -40,6 +51,21 @@ def main(): default=None, help="Path to YAML config (same as trtllm-serve --visual_gen_args)", ) + parser.add_argument( + "--enable_offloading", + action="store_true", + help="Enable Wan T2V text-encoder/transformer-block CPU offloading.", + ) + parser.add_argument( + "--offload_stages", + type=str, + default=None, + help=( + "Comma-separated offload stage names, e.g. " + "text_encoder,denoising_transformer,denoising_transformer_2,vae. " + "Implies --enable_offloading." + ), + ) parser.add_argument( "--output_path", type=str, @@ -49,8 +75,18 @@ def main(): args = parser.parse_args() # Engine config from shared YAML (optional); model-specific defaults apply otherwise. - extra_args = VisualGenArgs.from_yaml(args.visual_gen_args) if args.visual_gen_args else None - visual_gen = VisualGen(model=args.model, args=extra_args) + if args.visual_gen_args: + visual_gen_args = VisualGenArgs.from_yaml(args.visual_gen_args) + else: + visual_gen_args = VisualGenArgs() + + offload_stages = _parse_offload_stages(args.offload_stages) + if args.enable_offloading or offload_stages is not None: + visual_gen_args.cpu_offload_config.enable = True + if offload_stages is not None: + visual_gen_args.cpu_offload_config.stages = offload_stages + + visual_gen = VisualGen(model=args.model, args=visual_gen_args) # --- Model-specific: T2V request construction --- # Start from per-model defaults (steps, guidance, seed, etc.) and override resolution/frames. diff --git a/tensorrt_llm/_torch/modules/linear.py b/tensorrt_llm/_torch/modules/linear.py index e8da57a36c0e..e9459639e4c1 100644 --- a/tensorrt_llm/_torch/modules/linear.py +++ b/tensorrt_llm/_torch/modules/linear.py @@ -814,11 +814,14 @@ def rescale_fused_weights(self, module: Linear): """ weight_mode = module.weights_loading_config.weight_mode shard_key_to_index = weight_mode.shard_key_to_index + compute_device = module.weight.device + if compute_device.type == "cpu" and torch.cuda.is_available(): + compute_device = torch.device("cuda") # Handle input_scale if hasattr(module, "has_static_input_scale"): # Compute max and replace input_scale with a new parameter - max_input_scale = module.tmp_input_scales.max() + max_input_scale = module.tmp_input_scales.to(compute_device).max() module.input_scale.data.copy_(max_input_scale) # Also update inv_input_scale for static quantization if hasattr( @@ -832,13 +835,13 @@ def rescale_fused_weights(self, module: Linear): module.inv_input_scale = None # Compute max weight_scale - max_weight_scale = module.tmp_weight_scales.max() + max_weight_scale = module.tmp_weight_scales.to(compute_device).max() module.weight_scale.data.copy_(max_weight_scale) # Rescale each weight shard: (weight * original_scale) / max_scale for shard_key in weight_mode.shard_keys: idx = shard_key_to_index[shard_key] - original_scale = module.tmp_weight_scales[idx] + original_scale = module.tmp_weight_scales[idx].to(compute_device) # Get shard position from mapping shard_offset, shard_size = module.fused_weight_shard_indices_mapping[ @@ -847,12 +850,14 @@ def rescale_fused_weights(self, module: Linear): # Rescale: FP8 -> BF16 -> multiply by original_scale -> divide by max_scale -> FP8 weight_shard = module.weight.data[shard_offset:shard_offset + shard_size] - rescaled_weight = weight_shard.to(module.dtype).mul_(original_scale) + rescaled_weight = weight_shard.to(compute_device).to( + module.dtype).mul_(original_scale) rescaled_weight = rescaled_weight.div_( max_weight_scale.to(rescaled_weight.device)).to( torch.float8_e4m3fn) module.weight.data[shard_offset:shard_offset + - shard_size] = rescaled_weight + shard_size] = rescaled_weight.to( + module.weight.device) delattr(module, "tmp_input_scales") delattr(module, "tmp_weight_scales") diff --git a/tensorrt_llm/_torch/visual_gen/config.py b/tensorrt_llm/_torch/visual_gen/config.py index c84cc1c4b6b0..e4f4f0f327b0 100644 --- a/tensorrt_llm/_torch/visual_gen/config.py +++ b/tensorrt_llm/_torch/visual_gen/config.py @@ -35,6 +35,7 @@ CacheConfig, CacheDiTConfig, CompilationConfig, + CpuOffloadConfig, CudaGraphConfig, ParallelConfig, TeaCacheConfig, @@ -108,6 +109,7 @@ class DiffusionModelConfig(_VisualGenConfigBase): # Unified parallelism mapping copied from the owning pipeline config. visual_gen_mapping: Optional[Any] = None # VisualGenMapping (lazy import) + device: str = "cuda" dynamic_weight_quant: bool = False @@ -118,6 +120,7 @@ class DiffusionModelConfig(_VisualGenConfigBase): compilation: CompilationConfig = PydanticField(default_factory=CompilationConfig) torch_compile: TorchCompileConfig = PydanticField(default_factory=TorchCompileConfig) cuda_graph: CudaGraphConfig = PydanticField(default_factory=CudaGraphConfig) + cpu_offload_config: CpuOffloadConfig = PydanticField(default_factory=CpuOffloadConfig) attention: AttentionConfig = PydanticField(default_factory=AttentionConfig) attention_metadata_state: Optional[Dict[str, Any]] = None parallel: ParallelConfig = PydanticField(default_factory=ParallelConfig) @@ -173,6 +176,7 @@ class DiffusionPipelineConfig(_VisualGenConfigBase): # Unified parallelism mapping (populated by setup_visual_gen_mapping) visual_gen_mapping: Optional[Any] = None # VisualGenMapping (lazy import) + device: str = "cuda" dynamic_weight_quant: bool = False @@ -183,6 +187,7 @@ class DiffusionPipelineConfig(_VisualGenConfigBase): compilation: CompilationConfig = PydanticField(default_factory=CompilationConfig) torch_compile: TorchCompileConfig = PydanticField(default_factory=TorchCompileConfig) cuda_graph: CudaGraphConfig = PydanticField(default_factory=CudaGraphConfig) + cpu_offload_config: CpuOffloadConfig = PydanticField(default_factory=CpuOffloadConfig) attention: AttentionConfig = PydanticField(default_factory=AttentionConfig) attention_metadata_state: Optional[Dict[str, Any]] = None parallel: ParallelConfig = PydanticField(default_factory=ParallelConfig) @@ -240,12 +245,14 @@ def _make_model_config( extra_attrs=_model_config_value(self.extra_attrs), # Topology mappings carry distributed process-group handles. visual_gen_mapping=_model_config_value(self.visual_gen_mapping, deep_copy=False), + device=_model_config_value(self.device), dynamic_weight_quant=_model_config_value(self.dynamic_weight_quant), quant_config=_model_config_value(self.quant_config), quant_config_dict=_model_config_value(self.quant_config_dict), compilation=_model_config_value(self.compilation), torch_compile=_model_config_value(self.torch_compile), cuda_graph=_model_config_value(self.cuda_graph), + cpu_offload_config=_model_config_value(self.cpu_offload_config), attention=_model_config_value(self.attention), attention_metadata_state=_model_config_value(self.attention_metadata_state), parallel=_model_config_value(self.parallel), @@ -460,8 +467,9 @@ def from_pretrained( Args: checkpoint_dir: Path to checkpoint args: VisualGenArgs containing user config - - (compilation, torch_compile, cuda_graph, pipeline, attention, parallel, teacache, - cache_backend, cache_dit) + - (compilation_config, torch_compile_config, cuda_graph_config, + cpu_offload_config, pipeline_config, attention_config, parallel_config, + cache_config) **kwargs: Additional config options (e.g., mapping) """ kwargs.pop("trust_remote_code", None) @@ -470,6 +478,7 @@ def from_pretrained( compilation_cfg = args.compilation_config if args else CompilationConfig() torch_compile_cfg = args.torch_compile_config if args else TorchCompileConfig() cuda_graph_cfg = args.cuda_graph_config if args else CudaGraphConfig() + offload_cfg = args.cpu_offload_config if args else CpuOffloadConfig() attention_cfg = args.attention_config if args else AttentionConfig() parallel_cfg = args.parallel_config if args else ParallelConfig() cache_cfg = args.cache_config if args else None @@ -612,6 +621,8 @@ def from_pretrained( create_attention_metadata_state() if attention_cfg.backend == "TRTLLM" else None ) + device = kwargs.pop("device", "cuda") + pipeline_config = cls( quant_config=quant_config, quant_config_dict=quant_config_dict, @@ -621,10 +632,12 @@ def from_pretrained( compilation=compilation_cfg, torch_compile=torch_compile_cfg, cuda_graph=cuda_graph_cfg, + cpu_offload_config=offload_cfg, attention=attention_cfg, attention_metadata_state=attention_metadata_state, parallel=parallel_cfg, cache=cache_cfg, + device=device, enable_layerwise_nvtx_marker=enable_layerwise_nvtx_marker, skip_create_weights_in_init=True, extra_attrs=extra_attrs, diff --git a/tensorrt_llm/_torch/visual_gen/executor.py b/tensorrt_llm/_torch/visual_gen/executor.py index 0a1463c2d5c5..c41d99e69594 100644 --- a/tensorrt_llm/_torch/visual_gen/executor.py +++ b/tensorrt_llm/_torch/visual_gen/executor.py @@ -227,6 +227,21 @@ def _detect_external_launch() -> Optional[Tuple[int, int, int, str, int]]: return None +def _cuda_memory_logging_enabled() -> bool: + """Whether per-request CUDA peak-memory logging is enabled. + + This is a development-only knob exposed as an environment variable + (rather than a public ``VisualGenArgs`` field) to keep the engine + config surface clean, mirroring the nsys trace knob + ``TLLM_PROFILE_VISUAL_GEN_START_STOP``. + """ + return os.environ.get("TLLM_VISUAL_GEN_LOG_CUDA_MEMORY", "0").strip().lower() in ( + "1", + "true", + "yes", + ) + + @dataclass class DiffusionRequest: """Request for diffusion inference. @@ -415,6 +430,9 @@ def _merge_defaults(self, req: DiffusionRequest): def process_request(self, req: DiffusionRequest): """Process a single request.""" + log_cuda_memory = _cuda_memory_logging_enabled() + if log_cuda_memory: + self._reset_cuda_peak_memory_stats() try: self._merge_defaults(req) cache_key = self.pipeline.warmup_cache_key( @@ -433,7 +451,10 @@ def process_request(self, req: DiffusionRequest): # that the per-phase CUDA-event timings on PipelineOutput do not. generation_start = time.perf_counter() output = self.pipeline.infer(req) + if log_cuda_memory: + self._log_cuda_peak_memory(req.request_id) generation = time.perf_counter() - generation_start # seconds + if self.rank == 0: output.to_handle() self.response_queue.put( @@ -444,6 +465,8 @@ def process_request(self, req: DiffusionRequest): ) ) except Exception as e: + if log_cuda_memory: + self._log_cuda_peak_memory(req.request_id) logger.error(f"Worker {self.device_id}: Error: {e}") logger.error(traceback.format_exc()) if self.rank == 0: @@ -451,6 +474,36 @@ def process_request(self, req: DiffusionRequest): DiffusionResponse(request_id=req.request_id, error_msg=str(e)) ) + def _reset_cuda_peak_memory_stats(self) -> None: + """Reset CUDA peak memory stats for this worker if CUDA is available.""" + if not torch.cuda.is_available(): + return + + try: + torch.cuda.reset_peak_memory_stats(self.device_id) + except RuntimeError as e: + logger.warning( + f"Worker {self.device_id} rank {self.rank}: " + f"Unable to reset CUDA peak memory stats: {e}" + ) + + def _log_cuda_peak_memory(self, request_id: int) -> None: + """Log peak CUDA memory observed for one request.""" + if not torch.cuda.is_available(): + return + + try: + peak_allocated = torch.cuda.max_memory_allocated(self.device_id) + logger.info( + f"Worker {self.device_id} rank {self.rank}: " + f"Request {request_id} peak CUDA memory: {peak_allocated / 2**30:.2f} GiB" + ) + except RuntimeError as e: + logger.warning( + f"Worker {self.device_id} rank {self.rank}: " + f"Unable to log CUDA peak memory for request {request_id}: {e}" + ) + def run_diffusion_worker( rank: int, diff --git a/tensorrt_llm/_torch/visual_gen/models/cosmos3/pipeline_cosmos3.py b/tensorrt_llm/_torch/visual_gen/models/cosmos3/pipeline_cosmos3.py index 2dac241f2115..0cb5e2046ad9 100644 --- a/tensorrt_llm/_torch/visual_gen/models/cosmos3/pipeline_cosmos3.py +++ b/tensorrt_llm/_torch/visual_gen/models/cosmos3/pipeline_cosmos3.py @@ -21,6 +21,7 @@ import PIL.Image import torch +import torch.nn as nn from diffusers import AutoencoderKLWan, UniPCMultistepScheduler from diffusers.utils.torch_utils import randn_tensor from diffusers.video_processor import VideoProcessor @@ -52,6 +53,22 @@ TRTLLM_DISABLE_COSMOS3_GUARDRAILS = os.environ.get("TRTLLM_DISABLE_COSMOS3_GUARDRAILS", "0") == "1" +# Public offload component names for the two transformer towers. The "reasoner" +# (understanding) pathway is the causal language model that processes text; the +# "generator" (generation) pathway is the stack of cross-attention layers that +# produces video tokens. Only the heavy decoder-layer ModuleLists are offloaded; +# the small shared embeddings/projections/norms stay resident on GPU. +COSMOS3_REASONER_OFFLOAD_COMPONENT = "reasoner" +COSMOS3_GENERATOR_OFFLOAD_COMPONENT = "generator" +# Opt-in guardrail offload components (rank 0 only): the Qwen3Guard text checker and +# the RetinaFace video face-blur model. Kept out of the CPU defaults below. +COSMOS3_TEXT_GUARDRAIL_OFFLOAD_COMPONENT = "text_guardrail" +COSMOS3_VIDEO_GUARDRAIL_OFFLOAD_COMPONENT = "video_guardrail" +_COSMOS3_DEFAULT_OFFLOAD_STAGES = ( + (COSMOS3_REASONER_OFFLOAD_COMPONENT,), + (COSMOS3_GENERATOR_OFFLOAD_COMPONENT,), +) + @register_pipeline( "Cosmos3OmniMoTPipeline", @@ -87,6 +104,79 @@ def _init_transformer(self) -> None: model_config = self.pipeline_config.model_configs["transformer"] self.transformer = Cosmos3VFMTransformer(model_config) + # ========================================================================= + # Offloading + # ========================================================================= + + def default_offload_stages(self) -> tuple[tuple[str, ...], ...]: + """Offload the reasoner and generator towers as separate stages. + + Only invoked when ``cpu_offload_config.enable`` is true (the base class + short-circuits before calling this). Defaults are CPU-staged today; + non-CPU offload devices skip the defaults and require explicit + ``cpu_offload_config.stages`` from the user. + """ + cpu_offload_config = self.pipeline_config.cpu_offload_config + if cpu_offload_config.device != "cpu": + logger.warning( + "Cosmos3 default offload stages are CPU-only; " + f"cpu_offload_config.device='{cpu_offload_config.device}' has no " + "default stages. Set cpu_offload_config.stages explicitly to " + "stage components on a non-CPU device." + ) + return () + return _COSMOS3_DEFAULT_OFFLOAD_STAGES + + def offload_pipeline_components(self) -> dict[str, nn.Module]: + """Expose the two transformer towers, VAE, and guardrails as offload components. + + Cosmos3 packs both pathways into a single ``transformer`` module, so the + default ``BasePipeline.offload_pipeline_components`` (which looks for a + ``transformer.blocks`` ModuleList) does not apply. We expose the heavy + decoder-layer ModuleLists of each tower individually so they can be + brought on/off the GPU independently. The opt-in guardrail components wrap + the underlying safety nn.Modules (loaded on rank 0 only). + """ + components: dict[str, nn.Module] = {} + + transformer = getattr(self, "transformer", None) + if transformer is not None: + language_model = getattr(transformer, "language_model", None) + reasoner_layers = ( + getattr(language_model, "layers", None) if language_model is not None else None + ) + if reasoner_layers is not None: + components[COSMOS3_REASONER_OFFLOAD_COMPONENT] = reasoner_layers + + generator_layers = getattr(transformer, "gen_layers", None) + if generator_layers is not None: + components[COSMOS3_GENERATOR_OFFLOAD_COMPONENT] = generator_layers + + vae = getattr(self, PipelineComponent.VAE.value, None) + if vae is not None: + components[PipelineComponent.VAE.value] = vae + + # Guardrails (rank 0 only). CosmosSafetyChecker is an nn.Module but its + # GuardrailRunner children are plain objects, so expose the real safety + # nn.Modules (Qwen3Guard, RetinaFaceFilter) wrapped in a ModuleList. + safety_checker = getattr(self, "safety_checker", None) + if safety_checker is not None: + for component_name, runner in ( + (COSMOS3_TEXT_GUARDRAIL_OFFLOAD_COMPONENT, safety_checker.text_guardrail), + (COSMOS3_VIDEO_GUARDRAIL_OFFLOAD_COMPONENT, safety_checker.video_guardrail), + ): + modules = [m for m in runner.models if isinstance(m, nn.Module)] + if modules: + components[component_name] = nn.ModuleList(modules) + + return components + + def extra_offload_component_names(self) -> set[str]: + # Guardrails load on rank 0 only; treat their names as valid on all ranks + # so explicit multi-GPU stages don't fail during validation. Other ranks + # drop them later via the offloader's stage filtering. + return {COSMOS3_TEXT_GUARDRAIL_OFFLOAD_COMPONENT, COSMOS3_VIDEO_GUARDRAIL_OFFLOAD_COMPONENT} + def load_weights(self, weights: dict) -> None: if self.transformer is not None and hasattr(self.transformer, "load_weights"): transformer_weights = weights.get("transformer", weights) @@ -123,11 +213,16 @@ def load_standard_components( if PipelineComponent.VAE not in skip_components: logger.info("Loading VAE...") + vae_device = ( + torch.device("cpu") + if PipelineComponent.VAE.value in self.offloader.requested_components() + else device + ) self.vae = AutoencoderKLWan.from_pretrained( checkpoint_dir, subfolder=PipelineComponent.VAE, torch_dtype=torch.bfloat16, # load VAE in BF16 for memory saving - ).to(device) + ).to(vae_device) self.vae_scale_factor_temporal = getattr( self.vae.config, "scale_factor_temporal", self.vae_scale_factor_temporal @@ -486,7 +581,8 @@ def _encode_conditioning_video( video = image_tensor.unsqueeze(2).expand(-1, -1, num_frames, -1, -1).contiguous() video = video.to(device=self.device, dtype=self.vae.dtype) - latent = self.vae.encode(video).latent_dist.mode() + with self.offloader.context_if_requested(PipelineComponent.VAE.value): + latent = self.vae.encode(video).latent_dist.mode() # Normalize (inverse of _decode_latents denormalization) if hasattr(self.vae.config, "latents_mean") and hasattr(self.vae.config, "latents_std"): @@ -589,7 +685,8 @@ def _decode_latents(self, latents): scaling_factor = self.vae.config.get("scaling_factor", 1.0) latents = latents / scaling_factor - video = self.vae.decode(latents, return_dict=False)[0] + with self.offloader.context_if_requested(PipelineComponent.VAE.value): + video = self.vae.decode(latents, return_dict=False)[0] video = postprocess_video_tensor(video) return video @@ -682,12 +779,13 @@ def forward( prompts_to_check = list(prompt) if negative_prompt is not None: prompts_to_check.append(negative_prompt) - for p in prompts_to_check: - is_safe = self.safety_checker.check_text_safety(p) - if not is_safe: - logger.warning("Text guardrail blocked prompt") - text_blocked.fill_(1) - break + with self.offloader.context_if_requested(COSMOS3_TEXT_GUARDRAIL_OFFLOAD_COMPONENT): + for p in prompts_to_check: + is_safe = self.safety_checker.check_text_safety(p) + if not is_safe: + logger.warning("Text guardrail blocked prompt") + text_blocked.fill_(1) + break if torch.distributed.is_available() and torch.distributed.is_initialized(): torch.distributed.broadcast(text_blocked, src=0) @@ -826,6 +924,7 @@ def forward_fn( fps=frame_rate, noisy_frame_mask=velocity_mask, audio_latents=current_audio, + offload_context=self.offloader.context_if_requested, ) video_noise_pred = result.video @@ -894,7 +993,8 @@ def forward_fn( logger.info(f"Total pipeline time: {time.time() - pipeline_start:.2f}s") if use_guardrails and self.safety_checker is not None: - video = check_video_safety(video, self.safety_checker) + with self.offloader.context_if_requested(COSMOS3_VIDEO_GUARDRAIL_OFFLOAD_COMPONENT): + video = check_video_safety(video, self.safety_checker) timer.mark_end() diff --git a/tensorrt_llm/_torch/visual_gen/models/cosmos3/transformer_cosmos3.py b/tensorrt_llm/_torch/visual_gen/models/cosmos3/transformer_cosmos3.py index b3df78a06ec1..fca93a96ea7b 100644 --- a/tensorrt_llm/_torch/visual_gen/models/cosmos3/transformer_cosmos3.py +++ b/tensorrt_llm/_torch/visual_gen/models/cosmos3/transformer_cosmos3.py @@ -14,8 +14,9 @@ # limitations under the License. import math +from contextlib import nullcontext from dataclasses import dataclass -from typing import Optional, Tuple +from typing import Callable, ContextManager, Optional, Tuple import torch import torch.nn as nn @@ -35,6 +36,11 @@ from tensorrt_llm.models.modeling_utils import QuantConfig +def _noop_offload_context(_tower_name: str) -> ContextManager: + """Default offload context: no staging, used when callers don't offload.""" + return nullcontext() + + class Qwen3VLTextRMSNorm(nn.Module): def __init__( self, hidden_size: int, eps: float = 1e-6, dtype: torch.dtype = torch.bfloat16 @@ -976,6 +982,7 @@ def forward( text_ids: Optional[torch.Tensor] = None, text_mask: Optional[torch.Tensor] = None, video_shape: Optional[Tuple[int, int, int]] = None, + offload_context: Callable[[str], ContextManager] = _noop_offload_context, fps: float | None = None, noisy_frame_mask: torch.Tensor | None = None, audio_latents: Optional[torch.Tensor] = None, @@ -992,6 +999,12 @@ def forward( text_ids: [B, S_text] tokenized text input text_mask: [B, S_text] attention mask for text (1=real, 0=pad) video_shape: (T, H, W) in latent space + offload_context: Callable supplied by the pipeline that maps a tower + name ("reasoner" for the understanding pathway, "generator" for + the generation pathway) to a context manager staging that tower's + weights onto the GPU for the duration of its execution. The + pipeline returns a no-op context for towers that are not being + offloaded, so the transformer itself stays offload-agnostic. fps: video frame rate; when provided, temporal mRoPE positions are scaled to reflect real time (FPS modulation). noisy_frame_mask: Optional [B, 1, T, 1, 1] mask where 1=noisy (add @@ -1048,12 +1061,13 @@ def forward( hidden_states.device, hidden_states.dtype, ) - cached_kv_full = self.language_model( - text_ids, - text_mask, - freqs_und, - timestep=timestep, - ) + with offload_context("reasoner"): + cached_kv_full = self.language_model( + text_ids, + text_mask, + freqs_und, + timestep=timestep, + ) self.cached_freqs_gen = freqs_gen if self.sharder.is_active: @@ -1108,11 +1122,12 @@ def forward( sin = self.sharder.shard(sin, dim=1, pad_to_multiple=True) freqs_gen = (cos, sin) - for i, layer in enumerate(self.gen_layers): - k_und, v_und = self.cached_kv[i] - if not self.sharder.is_active: - k_und = k_und[:, :max_real_len] - v_und = v_und[:, :max_real_len] + with offload_context("generator"): + for i, layer in enumerate(self.gen_layers): + k_und, v_und = self.cached_kv[i] + if not self.sharder.is_active: + k_und = k_und[:, :max_real_len] + v_und = v_und[:, :max_real_len] hidden_gen = layer( hidden_gen, k_und, @@ -1121,14 +1136,6 @@ def forward( timestep=timestep, real_text_lens=real_text_lens, ) - else: - hidden_gen = layer( - hidden_gen, - k_und, - v_und, - freqs_gen, - timestep=timestep, - ) hidden_gen = self.sharder.gather(hidden_gen, dim=1, unpad_to=S_gen) diff --git a/tensorrt_llm/_torch/visual_gen/models/wan/pipeline_wan.py b/tensorrt_llm/_torch/visual_gen/models/wan/pipeline_wan.py index 41a284e040c3..f961c3a4446a 100644 --- a/tensorrt_llm/_torch/visual_gen/models/wan/pipeline_wan.py +++ b/tensorrt_llm/_torch/visual_gen/models/wan/pipeline_wan.py @@ -83,6 +83,16 @@ "fused fingers, motionless image, cluttered background, three legs, many people in the background, walking backward" ) +_WAN_SINGLE_TRANSFORMER_OFFLOAD_STAGES = ( + ("text_encoder",), + ("denoising_transformer",), +) +_WAN_TWO_TRANSFORMER_OFFLOAD_STAGES = ( + ("text_encoder",), + ("denoising_transformer",), + ("denoising_transformer_2",), +) + @register_pipeline( "WanPipeline", @@ -153,10 +163,10 @@ def dtype(self): @property def device(self): - return self.transformer.device + return super().device @property - def transformer_components(self) -> list: + def transformer_components(self) -> list[str]: """Return list of transformer components this pipeline needs.""" if self.transformer_2 is not None: return ["transformer", "transformer_2"] @@ -204,6 +214,28 @@ def _init_transformer(self) -> None: model_config=self.pipeline_config.model_configs["transformer_2"] ) + def default_offload_stages(self) -> tuple[tuple[str, ...], ...]: + """Return WAN offload stages for the loaded transformer topology. + + Only invoked when ``cpu_offload_config.enable`` is true (the base + class short-circuits before calling this). Defaults are CPU-staged + today; non-CPU offload devices skip the defaults and require + explicit ``cpu_offload_config.stages`` from the user. + """ + cpu_offload_config = self.pipeline_config.cpu_offload_config + if cpu_offload_config.device != "cpu": + logger.warning( + "Wan default offload stages are CPU-only; " + f"cpu_offload_config.device='{cpu_offload_config.device}' has no " + "default stages. Set cpu_offload_config.stages explicitly to " + "stage components on a non-CPU device." + ) + return () + + if self.transformer_2 is not None: + return _WAN_TWO_TRANSFORMER_OFFLOAD_STAGES + return _WAN_SINGLE_TRANSFORMER_OFFLOAD_STAGES + def load_standard_components( self, checkpoint_dir: str, @@ -240,17 +272,27 @@ def load_standard_components( if PipelineComponent.TEXT_ENCODER not in skip_components: logger.info("Loading text encoder...") + text_encoder_device = ( + torch.device("cpu") + if PipelineComponent.TEXT_ENCODER.value in self.offloader.requested_components() + else device + ) self.text_encoder = UMT5EncoderModel.from_pretrained( checkpoint_dir, subfolder=PipelineComponent.TEXT_ENCODER, torch_dtype=self.pipeline_config.torch_dtype, - ).to(device) + ).to(text_encoder_device) if PipelineComponent.VAE not in skip_components: logger.info("Loading VAE...") + vae_device = ( + torch.device("cpu") + if PipelineComponent.VAE.value in self.offloader.requested_components() + else device + ) self.vae = load_wan_vae( checkpoint_dir, - device, + vae_device, self.pipeline_config.visual_gen_mapping, dtype=self.pipeline_config.torch_dtype, ) @@ -560,6 +602,7 @@ def forward_fn( current_t = timestep if timestep.dim() == 0 else timestep[0] # Select model based on timestep (if two-stage denoising is enabled) + offload_stage = "denoising_transformer" if boundary_timestep is not None and self.transformer_2 is not None: if current_t >= boundary_timestep: current_model = self.transformer @@ -567,6 +610,7 @@ def forward_fn( else: current_model = self.transformer_2 model_name = "transformer_2 (low-noise)" + offload_stage = "denoising_transformer_2" # Log when switching models if last_model_used[0] != model_name: @@ -590,29 +634,30 @@ def forward_fn( # T2V: current_t for all frames timestep = current_t.reshape(1, 1).expand(latents.shape[0], nf * nh * nw) - if _vsa_active and _vsa_builder is not None: - # latents: [B, C, T_latent, H_latent, W_latent] - raw_latent_shape = (latents.shape[2], latents.shape[3], latents.shape[4]) - vsa_metadata = _vsa_builder.build( - current_timestep=_vsa_step_counter[0], - raw_latent_shape=raw_latent_shape, - patch_size=_vsa_patch_size, - vsa_sparsity=_vsa_sparsity, - device=latents.device, - ) - _vsa_step_counter[0] += 1 - with set_vsa_forward_context(vsa_metadata): - return current_model( - hidden_states=latents, - timestep=timestep / self.scheduler.config.num_train_timesteps, - encoder_hidden_states=encoder_hidden_states, + with self.offloader.context_if_requested(offload_stage): + if _vsa_active and _vsa_builder is not None: + # latents: [B, C, T_latent, H_latent, W_latent] + raw_latent_shape = (latents.shape[2], latents.shape[3], latents.shape[4]) + vsa_metadata = _vsa_builder.build( + current_timestep=_vsa_step_counter[0], + raw_latent_shape=raw_latent_shape, + patch_size=_vsa_patch_size, + vsa_sparsity=_vsa_sparsity, + device=latents.device, ) - - return current_model( - hidden_states=latents, - timestep=timestep / self.scheduler.config.num_train_timesteps, - encoder_hidden_states=encoder_hidden_states, - ) + _vsa_step_counter[0] += 1 + with set_vsa_forward_context(vsa_metadata): + return current_model( + hidden_states=latents, + timestep=timestep / self.scheduler.config.num_train_timesteps, + encoder_hidden_states=encoder_hidden_states, + ) + + return current_model( + hidden_states=latents, + timestep=timestep / self.scheduler.config.num_train_timesteps, + encoder_hidden_states=encoder_hidden_states, + ) # Pin reference image to latent after each scheduler step (Wan 2.2 5B I2V only) def _pin_i2v_first_frame(x): @@ -668,7 +713,10 @@ def get_embeds(texts): input_ids = text_inputs.input_ids.to(self.device) attention_mask = text_inputs.attention_mask.to(self.device) - embeds = self.text_encoder(input_ids, attention_mask=attention_mask).last_hidden_state + with self.offloader.context_if_requested(PipelineComponent.TEXT_ENCODER.value): + embeds = self.text_encoder( + input_ids, attention_mask=attention_mask + ).last_hidden_state embeds = embeds.to(self.dtype) # Zero-out padded tokens based on mask @@ -759,9 +807,10 @@ def _prepare_latents_wan22_5B_i2v( ) # Encode video condition through VAE - latent_condition = retrieve_latents(self.vae.encode(image), sample_mode="argmax").to( - self.dtype - ) + with self.offloader.context_if_requested(PipelineComponent.VAE.value): + latent_condition = retrieve_latents(self.vae.encode(image), sample_mode="argmax").to( + self.dtype + ) if batch_size > 1: latent_condition = latent_condition.repeat(batch_size, 1, 1, 1, 1) @@ -817,7 +866,8 @@ def _decode_latents(self, latents: torch.Tensor) -> torch.Tensor: latents = latents / scaling_factor # VAE decode: returns (B, C, T, H, W) - video = self.vae.decode(latents, return_dict=False)[0] + with self.offloader.context_if_requested(PipelineComponent.VAE.value): + video = self.vae.decode(latents, return_dict=False)[0] # Post-process video tensor: (B, C, T, H, W) -> (B, T, H, W, C) video = postprocess_video_tensor(video) diff --git a/tensorrt_llm/_torch/visual_gen/models/wan/pipeline_wan_i2v.py b/tensorrt_llm/_torch/visual_gen/models/wan/pipeline_wan_i2v.py index f472c4c1fe45..b79c45933f78 100644 --- a/tensorrt_llm/_torch/visual_gen/models/wan/pipeline_wan_i2v.py +++ b/tensorrt_llm/_torch/visual_gen/models/wan/pipeline_wan_i2v.py @@ -142,7 +142,7 @@ def device(self): return self.transformer.device @property - def transformer_components(self) -> list: + def transformer_components(self) -> list[str]: if self.transformer_2 is not None: return ["transformer", "transformer_2"] return ["transformer"] diff --git a/tensorrt_llm/_torch/visual_gen/offloading.py b/tensorrt_llm/_torch/visual_gen/offloading.py new file mode 100644 index 000000000000..34b18659ff17 --- /dev/null +++ b/tensorrt_llm/_torch/visual_gen/offloading.py @@ -0,0 +1,697 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Component-offloading utilities for visual generation pipelines. + +Terminology used throughout this module: + +- **component**: a named, public sub-model of the pipeline (e.g. ``text_encoder``, + ``denoising_transformer``, ``vae``). +- **stage**: one step of the offload schedule — a single component, or a group + of components that are co-resident on the GPU and run together before being + evicted back to CPU. + +The offload path keeps model loading and quantization unchanged: weights are +loaded into the modules first, then the components of each stage are copied into +packed CPU storage. At runtime one stage at a time is brought onto a reusable GPU +arena and the original module parameters/buffers are rebound to views of that +storage. +""" + +import time +import weakref +from contextlib import AbstractContextManager, nullcontext +from dataclasses import dataclass +from typing import TYPE_CHECKING, Iterator, Mapping, Sequence + +import torch +import torch.nn as nn + +from tensorrt_llm.logger import logger + +if TYPE_CHECKING: + from tensorrt_llm._torch.visual_gen.pipeline import BasePipeline + + +def _align_offset(offset: int, alignment: int = 256) -> int: + return ((offset + alignment - 1) // alignment) * alignment + + +def transformer_component_offload_name(component_name: str, transformer_prefix: str) -> str: + """Map an internal transformer attribute to its public offload component name.""" + if component_name == transformer_prefix: + return "denoising_transformer" + if component_name.startswith(f"{transformer_prefix}_"): + return f"denoising_transformer_{component_name[len(transformer_prefix) + 1 :]}" + return component_name + + +def _format_bytes(num_bytes: int) -> str: + return f"{num_bytes / (1024**3):.2f} GiB" + + +# FlashInfer and other custom kernels can require tensor data pointers to be at +# least 16-byte aligned even for smaller dtypes such as BF16. +_PACKED_TENSOR_ALIGNMENT = 16 + + +OffloadPipelineStage = tuple[str, ...] + + +@dataclass +class _FlatTensorSpec: + owner: nn.Module + name: str + qualified_name: str + is_parameter: bool + shape: tuple[int, ...] + stride: tuple[int, ...] + dtype: torch.dtype + requires_grad: bool + persistent: bool + offset: int + nbytes: int + + +@dataclass +class _StageLayout: + """Packed storage layout and rebound views for one offload stage.""" + + name: str + nbytes: int + specs: list[_FlatTensorSpec] + cpu_storage: torch.Tensor | None = None + cpu_views: tuple[nn.Parameter | torch.Tensor, ...] = () + gpu_views: tuple[nn.Parameter | torch.Tensor, ...] = () + + +class ModuleOffloadManager: + """Pack component stages into CPU storage and bring one stage onto the GPU. + + The manager owns packed byte buffers: + - each layout owns ``cpu_storage`` for one offloaded stage. + - ``gpu_arena`` is reused for whichever stage is currently active. + + Initializing the manager packs and rebinds one stage at a time. This + requires enough host memory to allocate the current stage's packed CPU + storage before that stage's original tensors are released. + """ + + def __init__( + self, + stages: Mapping[str, nn.Module], + device: torch.device | str, + pin_memory: bool = True, + ) -> None: + if not stages: + raise ValueError("At least one offload stage must be provided") + + self.stages = dict(stages) + self.device = torch.device(device) + self.pin_memory = pin_memory + self.gpu_arena: torch.Tensor | None = None + self.layouts: dict[str, _StageLayout] = {} + self.active_stage_name: str | None = None + + for name, module in self.stages.items(): + if not name: + raise ValueError("Offload stage names must be non-empty") + if not isinstance(module, nn.Module): + raise TypeError(f"Offload stage '{name}' must contain an nn.Module") + + @staticmethod + def _owner_and_name(root: nn.Module, qualified_name: str) -> tuple[nn.Module, str]: + if "." not in qualified_name: + return root, qualified_name + module_path, name = qualified_name.rsplit(".", 1) + return root.get_submodule(module_path), name + + @staticmethod + def _tensor_nbytes(tensor: torch.Tensor) -> int: + return tensor.numel() * tensor.element_size() + + @staticmethod + def _storage_key(tensor: torch.Tensor) -> tuple[int, int] | None: + if tensor.numel() == 0: + return None + storage_offset_bytes = tensor.storage_offset() * tensor.element_size() + return tensor.untyped_storage().data_ptr(), storage_offset_bytes + + def _get_alias_spec( + self, + seen_tensors: dict[tuple[int, int], _FlatTensorSpec], + tensor: torch.Tensor, + display_name: str, + ) -> _FlatTensorSpec | None: + key = self._storage_key(tensor) + if key is None: + return None + canonical = seen_tensors.get(key) + if canonical is None: + return None + if self._tensor_nbytes(tensor) != canonical.nbytes or tensor.dtype != canonical.dtype: + raise ValueError( + "Shared parameters or buffers with different sizes or dtypes are " + f"not supported by ModuleOffloadManager: '{display_name}' aliases " + f"'{canonical.qualified_name}'" + ) + return canonical + + def _build_spec( + self, + stage_name: str, + stage_module: nn.Module, + qualified_name: str, + tensor: torch.Tensor, + is_parameter: bool, + offset: int, + ) -> _FlatTensorSpec: + display_name = f"{stage_name}.{qualified_name}" + if not tensor.is_contiguous(): + raise ValueError( + f"Cannot offload non-contiguous tensor '{display_name}' " + f"with stride {tuple(tensor.stride())}" + ) + + owner, name = self._owner_and_name(stage_module, qualified_name) + return _FlatTensorSpec( + owner=owner, + name=name, + qualified_name=display_name, + is_parameter=is_parameter, + shape=tuple(tensor.shape), + stride=tuple(tensor.stride()), + dtype=tensor.dtype, + requires_grad=tensor.requires_grad if is_parameter else False, + persistent=True if is_parameter else name not in owner._non_persistent_buffers_set, + offset=offset, + nbytes=self._tensor_nbytes(tensor), + ) + + def _iter_stage_tensors( + self, stage_module: nn.Module + ) -> Iterator[tuple[str, torch.Tensor, bool]]: + for qualified_name, param in stage_module.named_parameters( + recurse=True, + remove_duplicate=False, + ): + yield qualified_name, param.detach(), True + for qualified_name, buffer in stage_module.named_buffers( + recurse=True, + remove_duplicate=False, + ): + yield qualified_name, buffer.detach(), False + + def _append_layout_spec( + self, + stage_name: str, + stage_module: nn.Module, + qualified_name: str, + tensor: torch.Tensor, + is_parameter: bool, + offset: int, + seen_tensors: dict[tuple[int, int], _FlatTensorSpec], + specs: list[_FlatTensorSpec], + ) -> int: + """Append a tensor spec and return the next stage-local byte offset. + + This handles three layout concerns in one place: alias reuse, packed + tensor alignment, and spec construction. The offset is local to this + stage and is shared by the CPU storage and reusable GPU arena views. + """ + display_name = f"{stage_name}.{qualified_name}" + alias = self._get_alias_spec(seen_tensors, tensor, display_name) + if alias is None: + offset = _align_offset(offset, _PACKED_TENSOR_ALIGNMENT) + + spec = self._build_spec( + stage_name=stage_name, + stage_module=stage_module, + qualified_name=qualified_name, + tensor=tensor, + is_parameter=is_parameter, + offset=alias.offset if alias is not None else offset, + ) + specs.append(spec) + + if alias is not None: + return offset + + key = self._storage_key(tensor) + if key is not None: + seen_tensors[key] = spec + return offset + spec.nbytes + + def _collect_stage_layout(self, stage_name: str, stage_module: nn.Module) -> _StageLayout: + """Build the packed storage layout for one named component stage.""" + offset = 0 + specs: list[_FlatTensorSpec] = [] + seen_tensors: dict[tuple[int, int], _FlatTensorSpec] = {} + + for qualified_name, tensor, is_parameter in self._iter_stage_tensors(stage_module): + offset = self._append_layout_spec( + stage_name=stage_name, + stage_module=stage_module, + qualified_name=qualified_name, + tensor=tensor, + is_parameter=is_parameter, + offset=offset, + seen_tensors=seen_tensors, + specs=specs, + ) + + if not specs: + raise ValueError(f"Offload stage '{stage_name}' has no parameters or buffers") + + return _StageLayout( + name=stage_name, + nbytes=_align_offset(offset), + specs=specs, + ) + + def _copy_stage_to_cpu_storage(self, layout: _StageLayout) -> None: + if layout.cpu_storage is None: + raise RuntimeError( + f"CPU storage for offload stage '{layout.name}' has not been allocated" + ) + for spec in layout.specs: + if spec.nbytes == 0: + continue + try: + tensor = getattr(spec.owner, spec.name).detach() + tensor_bytes = tensor.reshape(-1).view(torch.uint8).cpu() + layout.cpu_storage.narrow(0, spec.offset, spec.nbytes).copy_(tensor_bytes) + except RuntimeError as e: + raise RuntimeError( + f"Failed to copy offload tensor '{spec.qualified_name}' " + f"({_format_bytes(spec.nbytes)}, shape={spec.shape}, dtype={spec.dtype}) " + f"to packed CPU storage at offset {spec.offset}." + ) from e + + def _stage_size_summary(self) -> str: + return ", ".join( + f"{name}={_format_bytes(layout.nbytes)}" for name, layout in self.layouts.items() + ) + + def _cuda_allocation_hint(self) -> str: + if self.device.type != "cuda": + return "" + return ( + " If this is due to CUDA memory fragmentation, try setting " + "PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True before starting the process." + ) + + def _allocate_cpu_storage(self, num_bytes: int, stage_name: str | None = None) -> torch.Tensor: + try: + return torch.empty( + num_bytes, + dtype=torch.uint8, + device="cpu", + pin_memory=self.pin_memory, + ) + except RuntimeError as e: + stage_context = f", stage='{stage_name}'" if stage_name is not None else "" + raise RuntimeError( + "Failed to allocate packed CPU storage for visual generation offload " + f"({_format_bytes(num_bytes)}, {num_bytes} bytes{stage_context}, " + f"pin_memory={self.pin_memory}, stages=[{self._stage_size_summary()}])." + ) from e + + def _allocate_gpu_arena(self, num_bytes: int) -> torch.Tensor: + try: + return torch.empty(num_bytes, dtype=torch.uint8, device=self.device) + except RuntimeError as e: + raise RuntimeError( + "Failed to allocate GPU arena for visual generation offload " + f"({_format_bytes(num_bytes)}, {num_bytes} bytes, " + f"device={self.device}, stages=[{self._stage_size_summary()}])." + f"{self._cuda_allocation_hint()}" + ) from e + + def _make_views( + self, + layout: _StageLayout, + storage: torch.Tensor, + ) -> tuple[nn.Parameter | torch.Tensor, ...]: + views: list[nn.Parameter | torch.Tensor] = [] + for spec in layout.specs: + view = storage.narrow(0, spec.offset, spec.nbytes).view(spec.dtype) + view = view.as_strided(spec.shape, spec.stride) + if spec.is_parameter: + views.append(nn.Parameter(view, requires_grad=spec.requires_grad)) + else: + views.append(view) + return tuple(views) + + def _bind_views( + self, + layout: _StageLayout, + views: tuple[nn.Parameter | torch.Tensor, ...], + ) -> None: + for spec, view in zip(layout.specs, views, strict=True): + if spec.is_parameter: + if not isinstance(view, nn.Parameter): + raise TypeError( + f"Expected offload view '{spec.name}' to be an nn.Parameter, " + f"got {type(view).__name__}" + ) + spec.owner.register_parameter(spec.name, view) + else: + if not isinstance(view, torch.Tensor): + raise TypeError( + f"Expected offload view '{spec.name}' to be a torch.Tensor, " + f"got {type(view).__name__}" + ) + spec.owner.register_buffer(spec.name, view, persistent=spec.persistent) + + def initialize(self) -> None: + """Allocate packed storage, copy current tensors, and bind CPU views.""" + if self.layouts: + raise RuntimeError("ModuleOffloadManager has already been initialized") + + start_time = time.time() + for name, module in self.stages.items(): + layout = self._collect_stage_layout(name, module) + self.layouts[name] = layout + + total_cpu_bytes = sum(layout.nbytes for layout in self.layouts.values()) + max_gpu_bytes = max(layout.nbytes for layout in self.layouts.values()) + logger.info( + "Module offload storage layout: " + f"cpu_total={_format_bytes(total_cpu_bytes)}, " + f"gpu_arena={_format_bytes(max_gpu_bytes)}, " + f"stages=[{self._stage_size_summary()}], device={self.device}" + ) + + # Pack and rebind one stage at a time. This keeps setup simple and fast: + # offloading requires enough host memory to allocate one stage's packed + # CPU storage before that stage's original tensors are released. + for layout in self.layouts.values(): + logger.info( + "Module offload packing stage into CPU storage: " + f"{layout.name} ({_format_bytes(layout.nbytes)})" + ) + layout.cpu_storage = self._allocate_cpu_storage(layout.nbytes, stage_name=layout.name) + self._copy_stage_to_cpu_storage(layout) + layout.cpu_views = self._make_views(layout, layout.cpu_storage) + self._rebind_to_cpu(layout.name) + + self.gpu_arena = self._allocate_gpu_arena(max_gpu_bytes) + for layout in self.layouts.values(): + layout.gpu_views = self._make_views(layout, self.gpu_arena) + + logger.info(f"Module offload setup completed in {time.time() - start_time:.2f}s") + + def _get_layout(self, name: str) -> _StageLayout: + try: + return self.layouts[name] + except KeyError as e: + raise KeyError( + f"Unknown offload stage '{name}'. Available stages: {sorted(self.layouts)}" + ) from e + + def stage(self, name: str) -> None: + """Bring one offload stage onto the GPU arena and rebind its tensors.""" + layout = self._get_layout(name) + if self.active_stage_name == name: + return + if layout.cpu_storage is None or self.gpu_arena is None: + raise RuntimeError("ModuleOffloadManager must be initialized before staging") + + if self.active_stage_name is not None: + self._rebind_to_cpu(self.active_stage_name) + self.active_stage_name = None + + src = layout.cpu_storage.narrow(0, 0, layout.nbytes) + dst = self.gpu_arena.narrow(0, 0, layout.nbytes) + try: + dst.copy_(src, non_blocking=layout.cpu_storage.is_pinned()) + if self.device.type == "cuda": + torch.cuda.synchronize(self.device) + except RuntimeError as e: + raise RuntimeError( + f"Failed to bring offload stage '{name}' ({_format_bytes(layout.nbytes)}) " + f"onto {self.device}" + ) from e + self._rebind_to_gpu(name) + self.active_stage_name = name + + def _rebind_to_cpu(self, name: str) -> None: + layout = self._get_layout(name) + if not layout.cpu_views: + raise RuntimeError("ModuleOffloadManager must be initialized before staging") + self._bind_views(layout, layout.cpu_views) + + def _rebind_to_gpu(self, name: str) -> None: + layout = self._get_layout(name) + if not layout.gpu_views: + raise RuntimeError("ModuleOffloadManager must be initialized before staging") + self._bind_views(layout, layout.gpu_views) + + +class OffloadPipeline: + """Bring offload stages onto the GPU explicitly from model call-site contexts. + + This class intentionally does not use forward hooks. Pipeline code must wrap + the relevant call site with ``with self.offloader.context("stage")`` so staging + happens before the model invocation and outside any later CUDA graph capture. + """ + + def __init__( + self, + stages: Sequence[Sequence[str] | str], + components: Mapping[str, nn.Module], + device: torch.device | str, + pin_memory: bool = True, + ) -> None: + if not stages: + raise ValueError("At least one offload pipeline stage must be provided") + + self.stages = tuple( + (stage,) if isinstance(stage, str) else tuple(stage) for stage in stages + ) + self.components = dict(components) + self.device = torch.device(device) + self.pin_memory = pin_memory + self.manager = ModuleOffloadManager( + stages=self._build_stage_modules(), + device=self.device, + pin_memory=self.pin_memory, + ) + + def _build_stage_modules(self) -> dict[str, nn.Module]: + stage_modules: dict[str, nn.Module] = {} + for stage in self.stages: + stage_name = self._stage_name(stage) + if not stage: + raise ValueError("Offload pipeline stages must have at least one component") + if stage_name in stage_modules: + raise ValueError(f"Duplicate offload pipeline stage: {stage_name}") + + modules: list[nn.Module] = [] + for component_name in stage: + try: + component = self.components[component_name] + except KeyError as e: + raise KeyError( + f"Unknown offload pipeline component '{component_name}' for stage " + f"'{stage_name}'. Available components: {sorted(self.components)}" + ) from e + modules.append(component) + + stage_module = modules[0] if len(modules) == 1 else nn.ModuleList(modules) + stage_modules[stage_name] = stage_module + + return stage_modules + + def initialize(self) -> None: + """Allocate and populate backing storage for all configured stages.""" + self.manager.initialize() + + @staticmethod + def _stage_name(stage: Sequence[str] | str) -> str: + return stage if isinstance(stage, str) else "+".join(stage) + + def context(self, stage_name: str): + """Bring stage ``stage_name`` onto the GPU and return a no-op context manager.""" + self.manager.stage(stage_name) + # The active stage intentionally stays resident after the call site. + # The next stage() call rebinds it back to CPU before bringing another + # stage on, and cleanup() handles the final rebind when the pipeline exits. + return nullcontext() + + def cleanup(self) -> None: + """Return the active stage to CPU-backed views.""" + if self.manager.active_stage_name is not None: + self.manager._rebind_to_cpu(self.manager.active_stage_name) + self.manager.active_stage_name = None + + +class PipelineOffloader: + """Resolve, build, and drive component offloading for a visual-gen pipeline. + + Keeps ``BasePipeline`` free of offload bookkeeping: the pipeline only supplies + model-specific hooks (``offload_pipeline_components``, ``default_offload_stages``, + ``extra_offload_component_names``), while this coordinator resolves configured + stages, validates/filters them, builds the backing :class:`OffloadPipeline`, + and brings stages on/off the GPU at explicit call sites. + """ + + def __init__(self, pipeline: "BasePipeline") -> None: + # Hold the owning pipeline weakly: the pipeline owns this offloader + # (``pipeline.offloader``), so a strong back-reference would form a + # reference cycle and keep the whole pipeline (including its GPU + # weights) alive until a ``gc.collect()`` runs, defeating prompt + # release on ``del pipeline``. + self._pipeline_ref = weakref.ref(pipeline) + self._offload_pipeline: OffloadPipeline | None = None + self._stage_name_by_component: dict[str, str] = {} + + @property + def _pipeline(self) -> "BasePipeline": + pipeline = self._pipeline_ref() + if pipeline is None: + raise ReferenceError("PipelineOffloader's owning pipeline has already been released") + return pipeline + + @property + def offload_pipeline(self) -> OffloadPipeline | None: + """The backing :class:`OffloadPipeline`, or ``None`` if not initialized.""" + return self._offload_pipeline + + @staticmethod + def _normalize_stages( + stages: list[str | list[str]], + ) -> tuple[OffloadPipelineStage, ...]: + """Normalize user-configured offload stages to tuple form.""" + normalized_stages: list[OffloadPipelineStage] = [] + for stage in stages: + components = (stage,) if isinstance(stage, str) else tuple(stage) + if not components: + raise ValueError("Offload stages must contain at least one component") + normalized_stages.append(components) + return tuple(normalized_stages) + + def stages(self) -> tuple[OffloadPipelineStage, ...]: + """Return offload stages resolved from the pipeline configuration.""" + cpu_offload_config = getattr(self._pipeline.pipeline_config, "cpu_offload_config", None) + if cpu_offload_config is None or not cpu_offload_config.enable: + return () + configured_stages = getattr(cpu_offload_config, "stages", None) + if configured_stages is not None: + return self._normalize_stages(configured_stages) + return self._pipeline.default_offload_stages() + + def requested_components(self) -> set[str]: + """Return all component names referenced by the configured stages.""" + return {component for stage in self.stages() for component in stage} + + def validate_configured_stages( + self, + stages: tuple[OffloadPipelineStage, ...], + available_components: Mapping[str, nn.Module], + ) -> None: + """Validate user-configured stages against this pipeline's component names.""" + known = dict(available_components) + for name in self._pipeline.extra_offload_component_names(): + known.setdefault(name, None) + unknown_components = sorted( + {component for stage in stages for component in stage if component not in known} + ) + if unknown_components: + available_names = ", ".join(sorted(known)) or "" + unknown_names = ", ".join(unknown_components) + raise ValueError( + f"Unknown cpu_offload_config.stages entries for this model: {unknown_names}. " + f"Available components: {available_names}." + ) + + @staticmethod + def filter_available_stages( + stages: tuple[OffloadPipelineStage, ...], + available_components: Mapping[str, nn.Module], + ) -> tuple[OffloadPipelineStage, ...]: + """Drop stage components that are unavailable for the loaded pipeline.""" + return tuple( + tuple(component for component in stage if component in available_components) + for stage in stages + if any(component in available_components for component in stage) + ) + + def initialize(self) -> None: + """Create and initialize the offload pipeline after weights are loaded.""" + configured_stages = self.stages() + if not configured_stages or self._offload_pipeline is not None: + return + + pipeline = self._pipeline + available_components = pipeline.offload_pipeline_components() + cpu_offload_config = getattr(pipeline.pipeline_config, "cpu_offload_config", None) + has_user_stages = ( + cpu_offload_config is not None + and getattr(cpu_offload_config, "stages", None) is not None + ) + if has_user_stages: + self.validate_configured_stages(configured_stages, available_components) + + stages = self.filter_available_stages(configured_stages, available_components) + if not stages: + return + + if pipeline._cuda_graph_runners: + raise NotImplementedError( + "CUDA graphs are not supported with visual generation offloading yet. " + "Disable either cuda_graph_config.enable or cpu_offload_config.enable." + ) + + pin_memory = bool(getattr(cpu_offload_config, "pin_memory", True)) + stage_summary = " -> ".join("+".join(stage) for stage in stages) + logger.info(f"{pipeline.__class__.__name__} offload pipeline enabled: {stage_summary}") + offload_pipeline = OffloadPipeline( + stages=stages, + components=available_components, + device=torch.device(pipeline.device), + pin_memory=pin_memory, + ) + offload_pipeline.initialize() + self._offload_pipeline = offload_pipeline + self._stage_name_by_component = { + component: "+".join(stage) for stage in stages for component in stage + } + + def cleanup(self) -> None: + """Tear down the offload pipeline and reset state.""" + if self._offload_pipeline is not None: + self._offload_pipeline.cleanup() + self._offload_pipeline = None + self._stage_name_by_component = {} + + def context(self, component_name: str, enable: bool = True) -> AbstractContextManager: + """Bring the stage containing ``component_name`` onto the GPU for one call site.""" + if not enable: + return nullcontext() + if self.stages() and self._offload_pipeline is None: + raise RuntimeError( + "Visual generation offloading is configured but the offload pipeline " + "has not been initialized. Call initialize_offload_pipeline() after " + "loading weights." + ) + if self._offload_pipeline is None: + return nullcontext() + stage_name = self._stage_name_by_component.get(component_name, component_name) + return self._offload_pipeline.context(stage_name) + + def context_if_requested(self, component_name: str) -> AbstractContextManager: + """Return an offload context for ``component_name``, active only if configured.""" + return self.context(component_name, enable=component_name in self.requested_components()) diff --git a/tensorrt_llm/_torch/visual_gen/pipeline.py b/tensorrt_llm/_torch/visual_gen/pipeline.py index 13cbe77bdf12..aaf2558535bc 100644 --- a/tensorrt_llm/_torch/visual_gen/pipeline.py +++ b/tensorrt_llm/_torch/visual_gen/pipeline.py @@ -18,6 +18,7 @@ from .checkpoints import WeightLoader from .cuda_graph_runner import CUDAGraphRunner, CUDAGraphRunnerConfig, SharedGraphPool from .modules.vae.parallel_vae_interface import ParallelVAEFactory +from .offloading import OffloadPipelineStage, PipelineOffloader, transformer_component_offload_name class ExtraParamSchema(StrictBaseModel): @@ -122,7 +123,9 @@ def __init__(self, pipeline_config: "DiffusionPipelineConfig"): self.pipeline_config = pipeline_config self.config = pipeline_config.primary_pretrained_config self.mapping: Mapping = getattr(pipeline_config, "mapping", None) or Mapping() + self._device = torch.device(getattr(pipeline_config, "device", "cuda")) self._cuda_graph_runners: Dict[str, CUDAGraphRunner] = {} + self.offloader = PipelineOffloader(self) self._parallel_vae_enabled: bool = False self._warmed_up_shapes: Set[tuple] = set() @@ -183,6 +186,12 @@ def _setup_cuda_graphs(self): if not self.pipeline_config.cuda_graph.enable: return + if self.offloader.stages(): + raise NotImplementedError( + "CUDA graphs are not supported with visual generation offloading yet. " + "Disable either cuda_graph_config.enable or cpu_offload_config.enable." + ) + if len(self.transformer_components) > 1: logger.info( "CUDA graph runner: multiple transformer components, using shared graph pool" @@ -220,12 +229,12 @@ def dtype(self): @property def device(self): - return self.transformer.device + return self._device @property - def transformer_components(self) -> list: - """Return list of transformer components this pipeline needs.""" - return [PipelineComponent.TRANSFORMER] if self.transformer is not None else [] + def transformer_components(self) -> list[str]: + """Return names of transformer components this pipeline needs.""" + return [PipelineComponent.TRANSFORMER.value] if self.transformer is not None else [] def warmup_cache_key(self, height: int, width: int, num_frames: int) -> tuple: """Return the cache key for a given warmup shape. @@ -402,6 +411,59 @@ def post_load_weights(self) -> None: if self.transformer is not None and hasattr(self.transformer, "post_load_weights"): self.transformer.post_load_weights() + def default_offload_stages(self) -> tuple[OffloadPipelineStage, ...]: + """Return model-specific offload stages requested by shorthand config. + + Each stage is a tuple of component names from ``offload_pipeline_components``. + Components in the same stage are brought onto the GPU together. + """ + return () + + def extra_offload_component_names(self) -> set[str]: + """Return component names that are valid in configured stages but may be absent. + + These are accepted during stage validation even when missing from + ``offload_pipeline_components`` (e.g. rank-0-only components that other ranks + drop later). Defaults to an empty set; models override as needed. + """ + return set() + + def offload_pipeline_components(self) -> dict[str, nn.Module]: + """Expose standard component subtrees for offloading. + + This default assumes the common diffusers-style layout: a ``text_encoder``, + a ``vae``, and one or more transformers each exposing their decoder layers + as ``.blocks``. Models that deviate from this naming/structure (e.g. Cosmos3, + whose towers live inside a single ``transformer`` module) must override this + to expose their own offloadable components. + + NOTE: this attribute-name based detection is convenient but brittle; if we + extend offloading to more model families it may be worth replacing with an + explicit per-model declaration instead of relying on these conventions. + """ + components: dict[str, nn.Module] = {} + text_encoder = getattr(self, PipelineComponent.TEXT_ENCODER.value, None) + if text_encoder is not None: + components[PipelineComponent.TEXT_ENCODER.value] = text_encoder + + vae = getattr(self, PipelineComponent.VAE.value, None) + if vae is not None: + components[PipelineComponent.VAE.value] = vae + + for component_name in self.transformer_components: + transformer = getattr(self, component_name, None) + blocks = getattr(transformer, "blocks", None) if transformer is not None else None + if blocks is not None: + public_name = transformer_component_offload_name( + component_name, PipelineComponent.TRANSFORMER.value + ) + components[public_name] = blocks + return components + + def initialize_offload_pipeline(self) -> None: + """Create and initialize the offload pipeline after weights are loaded.""" + self.offloader.initialize() + def _apply_teacache_coefficients(self, coefficients: Optional[Dict] = None) -> None: """Resolve TeaCache polynomial coefficients into pipeline_config.cache (TeaCacheConfig). @@ -1231,6 +1293,7 @@ def cleanup(self): """Call before dist.destroy_process_group().""" self._cuda_profiler_stop() + self.offloader.cleanup() for name, runner in self._cuda_graph_runners.items(): logger.info(f"Releasing CUDA graphs for {name}") runner.clear() diff --git a/tensorrt_llm/_torch/visual_gen/pipeline_loader.py b/tensorrt_llm/_torch/visual_gen/pipeline_loader.py index 3551371fa173..2b2740270bae 100644 --- a/tensorrt_llm/_torch/visual_gen/pipeline_loader.py +++ b/tensorrt_llm/_torch/visual_gen/pipeline_loader.py @@ -254,9 +254,10 @@ def load( with MetaInitMode(): pipeline = AutoPipeline.from_config(config, checkpoint_dir) - # Convert meta tensors to CUDA tensors - self._materialize_meta_tensors(pipeline) - pipeline.to(self.device) + # Convert meta tensors to their runtime devices. Offloaded submodules + # stay on CPU until they are explicitly staged. + cpu_offload_modules = self._get_load_time_cpu_offload_modules(pipeline) + self._materialize_meta_tensors(pipeline, cpu_offload_modules) # ===================================================================== # STEP 3: Load Transformer Weights @@ -296,6 +297,7 @@ def load( if hasattr(pipeline, "post_load_weights"): pipeline.post_load_weights() + pipeline.initialize_offload_pipeline() if config.torch_compile.enable: torch._dynamo.config.cache_size_limit = 128 @@ -331,20 +333,83 @@ def load( ) return pipeline - def _materialize_meta_tensors(self, module: torch.nn.Module) -> None: + def _get_load_time_cpu_offload_modules(self, pipeline: "BasePipeline") -> list[torch.nn.Module]: + """Return offloaded modules that should materialize on CPU at load time. + + The offload manager is initialized only after weights and quantization + hooks run, but MetaInit materialization happens before loading. This + helper mirrors the pipeline's configured stages so offloaded towers never + need to materialize on GPU first. + """ + available_components = pipeline.offload_pipeline_components() + modules: list[torch.nn.Module] = [] + seen: set[int] = set() + + for stage in pipeline.offloader.filter_available_stages( + pipeline.offloader.stages(), available_components + ): + for component in stage: + offload_module = available_components[component] + module_id = id(offload_module) + if module_id not in seen: + modules.append(offload_module) + seen.add(module_id) + + return modules + + def _materialize_meta_tensors( + self, + module: torch.nn.Module, + cpu_offload_modules: Optional[list[torch.nn.Module]] = None, + ) -> None: """ - Convert meta tensors to CUDA tensors. + Convert meta tensors to tensors on their runtime devices. Meta tensors are placeholders that don't allocate GPU memory. After model structure is defined, we materialize them to real tensors. + Submodules listed in ``cpu_offload_modules`` are materialized on CPU and + skipped by the final move to ``self.device`` so checkpoint loading never + needs the full offloaded tower resident on GPU. """ - memo = {} + cpu_offload_modules = cpu_offload_modules or [] + meta_device = torch.device("meta") + + cpu_memo = {} + + def init_cpu_tensor(t: torch.Tensor) -> torch.Tensor: + if t.device != meta_device: + return t.to("cpu") + if t not in cpu_memo: + cpu_memo[t] = torch.empty_like(t, device="cpu") + return cpu_memo[t] + + for offload_module in cpu_offload_modules: + offload_module._apply(init_cpu_tensor) + + cpu_tensor_ids = self._collect_module_tensor_ids(cpu_offload_modules) + target_memo = {} def init_meta_tensor(t: torch.Tensor) -> torch.Tensor: - if t.device != torch.device("meta"): + if id(t) in cpu_tensor_ids: return t - if t not in memo: - memo[t] = torch.empty_like(t, device=self.device) - return memo[t] + if t.device != meta_device: + return t.to(self.device) + if t not in target_memo: + target_memo[t] = torch.empty_like(t, device=self.device) + return target_memo[t] module._apply(init_meta_tensor) + + @staticmethod + def _collect_module_tensor_ids(modules: list[torch.nn.Module]) -> set[int]: + """Collect parameter and buffer object IDs owned by ``modules``.""" + tensor_ids: set[int] = set() + for module in modules: + for child in module.modules(): + for param in child._parameters.values(): + if param is not None: + tensor_ids.add(id(param)) + for buffer in child._buffers.values(): + if buffer is not None: + tensor_ids.add(id(buffer)) + return tensor_ids diff --git a/tensorrt_llm/visual_gen/__init__.py b/tensorrt_llm/visual_gen/__init__.py index f15294afec4c..a1026b244631 100644 --- a/tensorrt_llm/visual_gen/__init__.py +++ b/tensorrt_llm/visual_gen/__init__.py @@ -29,6 +29,7 @@ CacheConfig, CacheDiTConfig, CompilationConfig, + CpuOffloadConfig, CudaGraphConfig, ParallelConfig, QuantAttentionConfig, @@ -55,6 +56,7 @@ # Cross-cutting sub-configs "CompilationConfig", "CudaGraphConfig", + "CpuOffloadConfig", "TorchCompileConfig", "ParallelConfig", "AttentionConfig", diff --git a/tensorrt_llm/visual_gen/args.py b/tensorrt_llm/visual_gen/args.py index 06c2cb4f603d..2e290f91ed20 100644 --- a/tensorrt_llm/visual_gen/args.py +++ b/tensorrt_llm/visual_gen/args.py @@ -471,6 +471,58 @@ class CudaGraphConfig(StrictBaseModel): enable: bool = Field(False, status="prototype") +class CpuOffloadConfig(StrictBaseModel): + """Configuration for offloading visual-generation model components to CPU. + + A diffusion pipeline is a sequence of large *components* (text encoder, + denoising transformer, VAE, optional guardrails, ...) that run one after + another, so only one is needed on the GPU at any moment. Offloading + exploits this: weights are loaded and quantized normally, then held in + packed host (CPU) storage and brought onto the GPU one *stage* at a time, + rebinding each module's parameters/buffers to a reusable GPU arena while + that stage runs and evicting it back to CPU afterwards. This lowers peak + GPU memory to roughly the largest single stage rather than the whole + pipeline, letting larger models fit on limited VRAM, at the cost of + host<->device copies between stages (mitigated by ``pin_memory``). + + Terminology (used consistently across these fields): + + - **component**: a named, public sub-model of the pipeline, e.g. + ``text_encoder``, ``denoising_transformer``, ``vae``. + - **stage**: one step of the offload schedule — a single component, or a + group of components that are co-resident on the GPU and run together + before being evicted. The ordered list of stages is set by ``stages``. + """ + + enable: bool = Field( + False, + status="prototype", + description="Enable offloading of model components to CPU between pipeline stages.", + ) + device: Literal["cpu", "cuda"] = Field( + "cpu", + status="prototype", + description="Device where components are held while not the active stage on the GPU.", + ) + pin_memory: bool = Field( + True, + status="prototype", + description="Allocate pinned CPU storage for faster host-to-device copies.", + ) + stages: Optional[List[Union[str, List[str]]]] = Field( + default=None, + status="prototype", + description=( + "Optional ordered list of stages, named with model-specific public component " + "names. Each entry is a single component, or a list of components that are " + "co-resident on the GPU and run together as one stage, e.g. " + "['text_encoder', 'denoising_transformer', 'vae'] for Wan T2V, or " + "['reasoner', 'generator', 'text_guardrail', 'video_guardrail', 'vae'] for Cosmos3. " + "If omitted, the model chooses default stages." + ), + ) + + class CompilationConfig(StrictBaseModel): """Configuration for torch.compile / CUDA graph warmup shapes. @@ -567,6 +619,10 @@ class VisualGenArgs(StrictBaseModel): default_factory=CudaGraphConfig, status="prototype", ) + cpu_offload_config: CpuOffloadConfig = Field( + default_factory=CpuOffloadConfig, + status="prototype", + ) attention_config: AttentionConfig = Field( default_factory=AttentionConfig, status="prototype", diff --git a/tests/integration/test_lists/test-db/l0_b200.yml b/tests/integration/test_lists/test-db/l0_b200.yml index f43b08b9fb72..4de0e77ed034 100644 --- a/tests/integration/test_lists/test-db/l0_b200.yml +++ b/tests/integration/test_lists/test-db/l0_b200.yml @@ -245,6 +245,9 @@ l0_b200: - unittest/_torch/visual_gen/test_wan21_t2v_teacache_user_coefficients.py - unittest/_torch/visual_gen/test_wan22_i2v_teacache.py - unittest/_torch/visual_gen/test_wan22_t2v_teacache.py + - unittest/_torch/visual_gen/test_offloading.py + - unittest/_torch/visual_gen/test_wan22_t2v_offload.py + - unittest/_torch/visual_gen/test_cosmos3_t2v_offload.py - unittest/_torch/visual_gen/test_wan_transformer.py - unittest/_torch/visual_gen/test_cosmos3_transformer.py - unittest/_torch/visual_gen/test_cosmos3_pipeline.py diff --git a/tests/unittest/_torch/visual_gen/test_cosmos3_pipeline.py b/tests/unittest/_torch/visual_gen/test_cosmos3_pipeline.py index c0329b71a060..eaa3f56a442c 100644 --- a/tests/unittest/_torch/visual_gen/test_cosmos3_pipeline.py +++ b/tests/unittest/_torch/visual_gen/test_cosmos3_pipeline.py @@ -497,7 +497,7 @@ def test_fp8_load_and_t2v(self): checkpoint = _require_checkpoint() pipeline = _load_pipeline(checkpoint, quant_config=COSMOS3_FP8_QUANT_CONFIG) try: - assert pipeline.transformer.model_config.quant_config.quant_algo is not None + assert pipeline.pipeline_config.quant_config.quant_algo is not None result = _run_forward(pipeline, image=None, num_frames=NUM_FRAMES) _assert_valid_video(result.video, num_frames=NUM_FRAMES) assert result.frame_rate == FRAME_RATE diff --git a/tests/unittest/_torch/visual_gen/test_cosmos3_t2v_offload.py b/tests/unittest/_torch/visual_gen/test_cosmos3_t2v_offload.py new file mode 100644 index 000000000000..c43fd275d723 --- /dev/null +++ b/tests/unittest/_torch/visual_gen/test_cosmos3_t2v_offload.py @@ -0,0 +1,216 @@ +# SPDX-FileCopyrightText: Copyright (c) 2022-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Pipeline-level integration tests for Cosmos3 T2V CPU offloading. + +Loads the full model and calls pipeline.forward() with cpu_offload_config.enable=True; +offloaded output is checked against the non-offload TRT-LLM baseline. CUDA graphs +with offloading must raise. These skip unless the checkpoint is available. + +Fast, GPU-free offload unit tests (including the Cosmos3-specific offload wiring) +live in test_offloading.py. + +Run: + pytest tests/unittest/_torch/visual_gen/test_cosmos3_t2v_offload.py -v -s + +Override checkpoint path: + DIFFUSION_MODEL_PATH_COSMOS3=/path/to/cosmos3 \\ + pytest tests/unittest/_torch/visual_gen/test_cosmos3_t2v_offload.py -v -s +""" + +import os + +os.environ["TLLM_DISABLE_MPI"] = "1" +# Cosmos3 guardrails require the cosmos_guardrail package (NVIDIA Open Model +# License), which is not installed in CI. Disable guardrails by default so the +# test runs without it: the guardrail offload stages below stay valid for +# validation and are filtered out at runtime when the safety checker is absent. +# Set this to "0" locally (with cosmos_guardrail installed) to also exercise +# guardrail offloading. +os.environ.setdefault("TRTLLM_DISABLE_COSMOS3_GUARDRAILS", "1") + +import gc +from pathlib import Path + +import pytest +import torch +import torch.nn.functional as F + +from tensorrt_llm._torch.visual_gen.models.cosmos3.pipeline_cosmos3 import ( + COSMOS3_GENERATOR_OFFLOAD_COMPONENT, + COSMOS3_REASONER_OFFLOAD_COMPONENT, + COSMOS3_TEXT_GUARDRAIL_OFFLOAD_COMPONENT, + COSMOS3_VIDEO_GUARDRAIL_OFFLOAD_COMPONENT, +) + + +@pytest.fixture(autouse=True, scope="module") +def _cleanup_mpi_env(): + yield + os.environ.pop("TLLM_DISABLE_MPI", None) + os.environ.pop("TRTLLM_DISABLE_COSMOS3_GUARDRAILS", None) + + +@pytest.fixture(autouse=True) +def _cleanup_gpu(): + gc.collect() + torch.cuda.empty_cache() + yield + gc.collect() + torch.cuda.empty_cache() + + +def _llm_models_root() -> Path: + if "LLM_MODELS_ROOT" in os.environ: + root = Path(os.environ["LLM_MODELS_ROOT"]) + else: + root = Path("/home/scratch.trt_llm_data_ci/llm-models/") + if not root.exists(): + root = Path("/scratch.trt_llm_data/llm-models/") + assert root.exists(), ( + "Set LLM_MODELS_ROOT or ensure /home/scratch.trt_llm_data_ci/llm-models/ is accessible." + ) + return root + + +def _checkpoint(env_var: str, default_name: str) -> str: + return os.environ.get(env_var) or str(_llm_models_root() / default_name) + + +COSMOS3_PATH = _checkpoint("DIFFUSION_MODEL_PATH_COSMOS3", "Cosmos3-Nano") + +# Mirrors a validated manual offload run; reduced steps keep the test fast. +INFER_PROMPT = "A cute cat playing piano." +# None -> Cosmos3 uses its safe default negative prompt and skips the guardrail +# blocklist check on it (an empty string is flagged as "Input is empty"). +INFER_NEGATIVE_PROMPT = None +INFER_HEIGHT = 720 +INFER_WIDTH = 1080 +INFER_NUM_FRAMES = 33 +INFER_NUM_STEPS = 4 +INFER_GUIDANCE_SCALE = 6.0 +INFER_SEED = 42 +# Exercise both transformer towers, the VAE, and both guardrails. +INFER_OFFLOAD_STAGES = [ + COSMOS3_REASONER_OFFLOAD_COMPONENT, + COSMOS3_GENERATOR_OFFLOAD_COMPONENT, + "vae", + COSMOS3_TEXT_GUARDRAIL_OFFLOAD_COMPONENT, + COSMOS3_VIDEO_GUARDRAIL_OFFLOAD_COMPONENT, +] +COS_SIM_THRESHOLD = 0.99 + + +def _make_pipeline(checkpoint_path: str, *, enable_offload: bool = False): + from tensorrt_llm._torch.visual_gen.pipeline_loader import PipelineLoader + from tensorrt_llm.visual_gen.args import CpuOffloadConfig as ArgsOffloadConfig + from tensorrt_llm.visual_gen.args import TorchCompileConfig, VisualGenArgs + + if not os.path.exists(checkpoint_path): + pytest.skip(f"Checkpoint not found: {checkpoint_path}") + args = VisualGenArgs( + model=checkpoint_path, + torch_compile_config=TorchCompileConfig(enable=False), + cpu_offload_config=ArgsOffloadConfig( + enable=enable_offload, + stages=INFER_OFFLOAD_STAGES if enable_offload else None, + ), + ) + pipeline = PipelineLoader(args).load(skip_warmup=True) + assert pipeline.transformer is not None, "Expected a loaded Cosmos3 transformer" + return pipeline + + +def _capture_video(pipe) -> torch.Tensor: + with torch.no_grad(): + result = pipe.forward( + prompt=INFER_PROMPT, + negative_prompt=INFER_NEGATIVE_PROMPT, + height=INFER_HEIGHT, + width=INFER_WIDTH, + num_frames=INFER_NUM_FRAMES, + num_inference_steps=INFER_NUM_STEPS, + guidance_scale=INFER_GUIDANCE_SCALE, + seed=INFER_SEED, + ) + assert result.video is not None, "Pipeline returned no video (guardrail blocked?)" + return result.video.float() / 255.0 + + +@pytest.fixture +def cosmos3_offload_pipeline(): + pipeline = _make_pipeline(COSMOS3_PATH, enable_offload=True) + yield pipeline + del pipeline + gc.collect() + torch.cuda.empty_cache() + + +@pytest.mark.integration +class TestCosmos3Offload: + """Cosmos3 T2V with CPU offloading of both towers, VAE, and guardrails.""" + + def test_cosmos3_offload_forward(self, cosmos3_offload_pipeline): + pipeline = cosmos3_offload_pipeline + assert pipeline.offloader.stages(), ( + "cpu_offload_config.enable=True but offloader.stages() is empty" + ) + assert pipeline.offloader.offload_pipeline is not None, ( + "offload pipeline was not initialized after load" + ) + _capture_video(pipeline) + print(f"\n ===== Offload — Cosmos3 {INFER_HEIGHT}x{INFER_WIDTH} =====") + print(f" stages: {pipeline.offloader.stages()}") + + def test_cosmos3_offload_matches_baseline(self): + offload_pipe = _make_pipeline(COSMOS3_PATH, enable_offload=True) + offload_video = _capture_video(offload_pipe) + del offload_pipe + gc.collect() + torch.cuda.empty_cache() + + baseline_pipe = _make_pipeline(COSMOS3_PATH, enable_offload=False) + baseline_video = _capture_video(baseline_pipe) + del baseline_pipe + gc.collect() + torch.cuda.empty_cache() + + assert offload_video.shape == baseline_video.shape, ( + f"shape mismatch offload {offload_video.shape} vs baseline {baseline_video.shape}" + ) + a_flat = offload_video.float().cpu().reshape(-1) + b_flat = baseline_video.float().cpu().reshape(-1) + cos_sim = ( + F.cosine_similarity(a_flat.unsqueeze(0), b_flat.unsqueeze(0)).clamp(-1.0, 1.0).item() + ) + print( + f"\n ===== Offload accuracy — Cosmos3 =====\n" + f" cosine_similarity={cos_sim:.6f} (threshold={COS_SIM_THRESHOLD})" + ) + assert cos_sim >= COS_SIM_THRESHOLD, ( + f"offload vs baseline cosine similarity {cos_sim:.6f} < {COS_SIM_THRESHOLD}" + ) + + +@pytest.mark.integration +class TestCosmos3OffloadCudaGraphRaisesError: + """CUDA graphs plus offloading must raise NotImplementedError on pipeline load.""" + + def test_cosmos3_raises_if_cuda_graph_and_offload_enabled(self): + from tensorrt_llm._torch.visual_gen.pipeline_loader import PipelineLoader + from tensorrt_llm.visual_gen.args import CpuOffloadConfig as ArgsOffloadConfig + from tensorrt_llm.visual_gen.args import CudaGraphConfig, TorchCompileConfig, VisualGenArgs + + if not os.path.exists(COSMOS3_PATH): + pytest.skip(f"Checkpoint not found: {COSMOS3_PATH}") + args = VisualGenArgs( + model=COSMOS3_PATH, + torch_compile_config=TorchCompileConfig(enable=False), + cuda_graph_config=CudaGraphConfig(enable=True), + cpu_offload_config=ArgsOffloadConfig(enable=True, stages=INFER_OFFLOAD_STAGES), + ) + with pytest.raises( + NotImplementedError, + match="CUDA graphs are not supported with visual generation offloading", + ): + PipelineLoader(args).load(skip_warmup=True) diff --git a/tests/unittest/_torch/visual_gen/test_cosmos3_transformer.py b/tests/unittest/_torch/visual_gen/test_cosmos3_transformer.py index 76e6c1a01a8c..428e744fba2c 100644 --- a/tests/unittest/_torch/visual_gen/test_cosmos3_transformer.py +++ b/tests/unittest/_torch/visual_gen/test_cosmos3_transformer.py @@ -28,7 +28,10 @@ import torch from tensorrt_llm._torch.modules.linear import Linear -from tensorrt_llm._torch.visual_gen.config import DiffusionModelConfig, DiffusionPipelineConfig +from tensorrt_llm._torch.visual_gen.config import ( + DiffusionModelConfig, + DiffusionPipelineConfig, +) from tensorrt_llm._torch.visual_gen.models.cosmos3.transformer_cosmos3 import Cosmos3VFMTransformer from tensorrt_llm._torch.visual_gen.pipeline_loader import PipelineComponent, PipelineLoader from tensorrt_llm.visual_gen.args import TorchCompileConfig, VisualGenArgs @@ -447,7 +450,7 @@ def test_load_fp8_quantization(self, quant_algo: str): ) pipeline = PipelineLoader(args).load(skip_warmup=True, skip_components=_SKIP_AUX) try: - assert pipeline.transformer.model_config.quant_config.quant_algo is not None + assert pipeline.pipeline_config.quant_config.quant_algo is not None transformer = pipeline.transformer c = transformer.latent_channel_size hs, ts, text_ids, text_mask, video_shape = _cosmos3_inputs( diff --git a/tests/unittest/_torch/visual_gen/test_offloading.py b/tests/unittest/_torch/visual_gen/test_offloading.py new file mode 100644 index 000000000000..19b57a04a23b --- /dev/null +++ b/tests/unittest/_torch/visual_gen/test_offloading.py @@ -0,0 +1,589 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Tests for visual generation module parameter offloading.""" + +from types import SimpleNamespace +from typing import Optional +from unittest.mock import patch + +import pytest +import torch +import torch.nn as nn + +from tensorrt_llm._torch.visual_gen.config import CpuOffloadConfig, VisualGenArgs +from tensorrt_llm._torch.visual_gen.models.cosmos3.pipeline_cosmos3 import ( + COSMOS3_GENERATOR_OFFLOAD_COMPONENT, + COSMOS3_REASONER_OFFLOAD_COMPONENT, + COSMOS3_TEXT_GUARDRAIL_OFFLOAD_COMPONENT, + COSMOS3_VIDEO_GUARDRAIL_OFFLOAD_COMPONENT, + Cosmos3OmniMoTPipeline, +) +from tensorrt_llm._torch.visual_gen.offloading import ModuleOffloadManager, OffloadPipeline +from tensorrt_llm._torch.visual_gen.pipeline import BasePipeline + + +class _ToyModule(nn.Module): + def __init__(self, weight_value: float, bias_value: float) -> None: + super().__init__() + self.weight = nn.Parameter(torch.full((2, 2), weight_value)) + self.register_buffer("bias", torch.full((2,), bias_value)) + + +class _AlignmentToyModule(nn.Module): + def __init__(self) -> None: + super().__init__() + self.prefix = nn.Parameter(torch.ones((1,), dtype=torch.bfloat16)) + self.alignment_sensitive = nn.Parameter(torch.ones((3,), dtype=torch.bfloat16)) + + +class _ToyTransformer(nn.Module): + def __init__(self) -> None: + super().__init__() + self.blocks = _ToyModule(weight_value=3.0, bias_value=30.0) + + @property + def device(self): + return next(self.parameters()).device + + +class _OffloadCudaGraphPipeline(BasePipeline): + def _init_transformer(self) -> None: + self.transformer = _ToyModule(weight_value=1.0, bias_value=10.0) + + def default_offload_stages(self) -> tuple[tuple[str, ...], ...]: + return (("denoising_transformer",),) + + +class _CustomOffloadPipeline(BasePipeline): + def _init_transformer(self) -> None: + self.transformer = _ToyTransformer() + self.text_encoder = _ToyModule(weight_value=1.0, bias_value=10.0) + self.vae = _ToyModule(weight_value=2.0, bias_value=20.0) + + +def _make_config( + *, + offload_enable: bool = True, + offload_stages: list[str | list[str]] | None = None, + device: str = "cuda", +): + return SimpleNamespace( + primary_pretrained_config=SimpleNamespace(), + device=device, + cuda_graph=SimpleNamespace(enable=False), + torch_compile=SimpleNamespace(enable=False), + cpu_offload_config=CpuOffloadConfig( + enable=offload_enable, + stages=offload_stages, + ), + ) + + +def test_pipeline_device_uses_config_not_transformer_parameter_device(): + pipeline = _CustomOffloadPipeline(_make_config(device="cuda:3")) + + assert pipeline.transformer.device == torch.device("cpu") + assert pipeline.device == torch.device("cuda:3") + + +def _make_manager() -> tuple[ModuleOffloadManager, _ToyModule, _ToyModule]: + stage_a = _ToyModule(weight_value=1.0, bias_value=10.0) + stage_b = _ToyModule(weight_value=2.0, bias_value=20.0) + manager = ModuleOffloadManager( + stages={ + "stage_a": stage_a, + "stage_b": stage_b, + }, + device="cpu", + pin_memory=False, + ) + manager.initialize() + return manager, stage_a, stage_b + + +def _storage_ptr(tensor: torch.Tensor) -> int: + return tensor.untyped_storage().data_ptr() + + +def test_cuda_graphs_with_offload_raise_not_implemented(): + config = _make_config() + config.cuda_graph.enable = True + + with pytest.raises( + NotImplementedError, + match="CUDA graphs are not supported with visual generation offloading", + ): + _OffloadCudaGraphPipeline(config) + + +def test_offload_context_requires_initialized_pipeline_when_configured(): + pipeline = _OffloadCudaGraphPipeline(_make_config()) + + with pipeline.offloader.context("transformer", enable=False): + pass + + with pytest.raises( + RuntimeError, + match="offload pipeline has not been initialized", + ): + with pipeline.offloader.context("transformer"): + pass + + +def test_configured_offload_stages_override_model_defaults_and_expose_vae(): + pipeline = _CustomOffloadPipeline( + _make_config( + offload_stages=[ + "text_encoder", + ["denoising_transformer", "vae"], + "denoising_transformer_2", + ] + ) + ) + + assert pipeline.offloader.stages() == ( + ("text_encoder",), + ("denoising_transformer", "vae"), + ("denoising_transformer_2",), + ) + + components = pipeline.offload_pipeline_components() + assert components["text_encoder"] is pipeline.text_encoder + assert components["denoising_transformer"] is pipeline.transformer.blocks + assert components["vae"] is pipeline.vae + + assert pipeline.offloader.filter_available_stages(pipeline.offloader.stages(), components) == ( + ("text_encoder",), + ("denoising_transformer", "vae"), + ) + + +def test_offload_context_resolves_component_to_stageed_stage(): + pipeline = _CustomOffloadPipeline( + _make_config(offload_stages=[["denoising_transformer", "vae"]]) + ) + pipeline.initialize_offload_pipeline() + + with pipeline.offloader.context("vae"): + assert pipeline.offloader.offload_pipeline is not None + assert ( + pipeline.offloader.offload_pipeline.manager.active_stage_name + == "denoising_transformer+vae" + ) + + pipeline.cleanup() + + +def test_initialize_rejects_configured_stage_components_that_are_unavailable_for_model(): + pipeline = _CustomOffloadPipeline( + _make_config(offload_stages=[["denoising_transformer", "denoising_transformer_2"]]) + ) + + with pytest.raises( + ValueError, + match=r"Unknown cpu_offload_config\.stages entries.*denoising_transformer_2.*denoising_transformer", + ): + pipeline.initialize_offload_pipeline() + + +def test_visual_gen_args_loads_yaml_offload_stages(tmp_path): + config_path = tmp_path / "visual_gen.yaml" + config_path.write_text( + """ +cpu_offload_config: + enable: true + stages: + - text_encoder + - [denoising_transformer, vae] +""", + encoding="utf-8", + ) + + args = VisualGenArgs.from_yaml(config_path) + + assert args.cpu_offload_config.enable is True + assert args.cpu_offload_config.stages == [ + "text_encoder", + ["denoising_transformer", "vae"], + ] + + +def test_visual_gen_args_allows_model_specific_offload_stage_names(): + args = VisualGenArgs( + cpu_offload_config={ + "enable": True, + "stages": ["future_model_stage"], + } + ) + + assert args.cpu_offload_config.stages == ["future_model_stage"] + + +def test_pipeline_rejects_unknown_offload_stage_names_for_this_model(): + pipeline = _CustomOffloadPipeline(_make_config(offload_stages=["transformer.blocks"])) + + with pytest.raises( + ValueError, + match=r"Unknown cpu_offload_config\.stages entries.*transformer\.blocks.*denoising_transformer", + ): + pipeline.initialize_offload_pipeline() + + +def test_initialize_reports_cpu_storage_allocation_context(): + stage = _ToyModule(weight_value=1.0, bias_value=10.0) + manager = ModuleOffloadManager( + stages={"stage": stage}, + device="cpu", + pin_memory=False, + ) + gpu_arena = torch.empty(1024, dtype=torch.uint8, device="cpu") + + def fail_cpu_storage(*args, **kwargs): + raise RuntimeError("cpu allocation failed") + + with ( + patch.object(manager, "_allocate_gpu_arena", return_value=gpu_arena), + patch("torch.empty", side_effect=fail_cpu_storage), + ): + with pytest.raises( + RuntimeError, + match="Failed to allocate packed CPU storage for visual generation offload", + ) as exc_info: + manager.initialize() + + message = str(exc_info.value) + assert "Failed to allocate packed CPU storage for visual generation offload" in message + assert "pin_memory=False" in message + assert "stages=[stage=" in message + + +def test_initialize_reports_cuda_arena_allocation_hint(): + stage = _ToyModule(weight_value=1.0, bias_value=10.0) + manager = ModuleOffloadManager( + stages={"stage": stage}, + device="cuda", + pin_memory=False, + ) + original_empty = torch.empty + + def fail_cuda_arena(*args, **kwargs): + device = torch.device(kwargs.get("device", "cpu")) + if device.type == "cuda": + raise RuntimeError("cuda allocation failed") + return original_empty(*args, **kwargs) + + with patch("torch.empty", side_effect=fail_cuda_arena): + with pytest.raises( + RuntimeError, + match="Failed to allocate GPU arena for visual generation offload", + ) as exc_info: + manager.initialize() + + message = str(exc_info.value) + assert "Failed to allocate GPU arena for visual generation offload" in message + assert "device=cuda" in message + assert "stages=[stage=" in message + assert "PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True" in message + + +def test_initialize_packs_stage_before_rebinding_to_cpu(): + stage_a = _ToyModule(weight_value=1.0, bias_value=10.0) + stage_b = _ToyModule(weight_value=2.0, bias_value=20.0) + manager = ModuleOffloadManager( + stages={ + "stage_a": stage_a, + "stage_b": stage_b, + }, + device="cpu", + pin_memory=False, + ) + events = [] + original_rebind_to_cpu = manager._rebind_to_cpu + original_allocate_cpu_storage = manager._allocate_cpu_storage + + def record_rebind_to_cpu(name): + events.append(("rebind_cpu", name)) + return original_rebind_to_cpu(name) + + def record_allocate_cpu_storage(num_bytes, stage_name=None): + events.append(("allocate_cpu", stage_name)) + return original_allocate_cpu_storage(num_bytes, stage_name=stage_name) + + with ( + patch.object(manager, "_rebind_to_cpu", side_effect=record_rebind_to_cpu), + patch.object( + manager, + "_allocate_cpu_storage", + side_effect=record_allocate_cpu_storage, + ), + ): + manager.initialize() + + assert events.index(("allocate_cpu", "stage_a")) < events.index(("rebind_cpu", "stage_a")) + assert events.index(("allocate_cpu", "stage_b")) < events.index(("rebind_cpu", "stage_b")) + + +def test_packed_tensor_views_are_sufficiently_aligned(): + stage = _AlignmentToyModule() + manager = ModuleOffloadManager( + stages={"stage": stage}, + device="cpu", + pin_memory=False, + ) + manager.initialize() + + layout = manager.layouts["stage"] + for spec in layout.specs: + assert spec.offset % 16 == 0 + + manager.stage("stage") + assert stage.prefix.data_ptr() % 16 == 0 + assert stage.alignment_sensitive.data_ptr() % 16 == 0 + + +def test_offload_pipeline_context_stages_requested_stage(): + stage_a = _ToyModule(weight_value=1.0, bias_value=10.0) + stage_b = _ToyModule(weight_value=2.0, bias_value=20.0) + pipeline = OffloadPipeline( + stages=(("stage_a",), ("stage_b",)), + components={ + "stage_a": stage_a, + "stage_b": stage_b, + }, + device="cpu", + pin_memory=False, + ) + pipeline.initialize() + + assert pipeline.manager.active_stage_name is None + with pipeline.context("stage_a"): + assert pipeline.manager.active_stage_name == "stage_a" + assert pipeline.manager.active_stage_name == "stage_a" + + with pipeline.context("stage_b"): + assert pipeline.manager.active_stage_name == "stage_b" + + +def test_inactive_stage_stays_cpu_backed_after_staging_another_stage(): + manager, stage_a, stage_b = _make_manager() + assert manager.gpu_arena is not None + stage_a_cpu_storage = manager.layouts["stage_a"].cpu_storage + stage_b_cpu_storage = manager.layouts["stage_b"].cpu_storage + assert stage_a_cpu_storage is not None + assert stage_b_cpu_storage is not None + stage_a_cpu_storage_ptr = _storage_ptr(stage_a_cpu_storage) + stage_b_cpu_storage_ptr = _storage_ptr(stage_b_cpu_storage) + gpu_arena_ptr = _storage_ptr(manager.gpu_arena) + + manager.stage("stage_a") + + assert manager.active_stage_name == "stage_a" + assert _storage_ptr(stage_a.weight) == gpu_arena_ptr + assert _storage_ptr(stage_a.bias) == gpu_arena_ptr + assert _storage_ptr(stage_b.weight) == stage_b_cpu_storage_ptr + assert _storage_ptr(stage_b.bias) == stage_b_cpu_storage_ptr + torch.testing.assert_close(stage_b.weight, torch.full((2, 2), 2.0)) + torch.testing.assert_close(stage_b.bias, torch.full((2,), 20.0)) + + manager.stage("stage_b") + + assert manager.active_stage_name == "stage_b" + assert _storage_ptr(stage_a.weight) == stage_a_cpu_storage_ptr + assert _storage_ptr(stage_a.bias) == stage_a_cpu_storage_ptr + assert _storage_ptr(stage_b.weight) == gpu_arena_ptr + assert _storage_ptr(stage_b.bias) == gpu_arena_ptr + torch.testing.assert_close(stage_a.weight, torch.full((2, 2), 1.0)) + torch.testing.assert_close(stage_a.bias, torch.full((2,), 10.0)) + + +def test_state_dict_reads_correct_inactive_stage_data(): + manager, stage_a, stage_b = _make_manager() + + manager.stage("stage_a") + stage_b_state = stage_b.state_dict() + torch.testing.assert_close(stage_b_state["weight"], torch.full((2, 2), 2.0)) + torch.testing.assert_close(stage_b_state["bias"], torch.full((2,), 20.0)) + + manager.stage("stage_b") + stage_a_state = stage_a.state_dict() + torch.testing.assert_close(stage_a_state["weight"], torch.full((2, 2), 1.0)) + torch.testing.assert_close(stage_a_state["bias"], torch.full((2,), 10.0)) + + +def test_rebinding_reuses_cached_view_objects(): + manager, stage_a, _ = _make_manager() + cpu_weight = stage_a.weight + cpu_bias = stage_a.bias + + manager.stage("stage_a") + gpu_weight = stage_a.weight + gpu_bias = stage_a.bias + + assert gpu_weight is not cpu_weight + assert gpu_bias is not cpu_bias + + manager.stage("stage_b") + assert stage_a.weight is cpu_weight + assert stage_a.bias is cpu_bias + + manager.stage("stage_a") + assert stage_a.weight is gpu_weight + assert stage_a.bias is gpu_bias + + +# ============================================================================ +# Cosmos3-specific offload wiring (no GPU / no checkpoint) +# ============================================================================ + + +class _FakeRunner: + """Stand-in for cosmos_guardrail.GuardrailRunner (a plain object).""" + + def __init__(self, models: list) -> None: + self._models = models + + @property + def models(self) -> list: + return self._models + + +class _FakeSafetyChecker: + def __init__(self, text_models: list, video_models: list) -> None: + self.text_guardrail = _FakeRunner(text_models) + self.video_guardrail = _FakeRunner(video_models) + + +class _FakeCosmos3Transformer(nn.Module): + def __init__(self) -> None: + super().__init__() + self.language_model = nn.Module() + self.language_model.layers = nn.ModuleList([nn.Linear(2, 2)]) + self.gen_layers = nn.ModuleList([nn.Linear(2, 2)]) + + +class _Cosmos3OffloadUnitPipeline(Cosmos3OmniMoTPipeline): + """Cosmos3 pipeline with a fake transformer to avoid loading the model.""" + + def _init_transformer(self) -> None: + self.transformer = _FakeCosmos3Transformer() + + +def _make_cosmos3_unit_config( + *, + offload_enable: bool = True, + offload_stages: Optional[list] = None, + device: str = "cpu", +): + return SimpleNamespace( + primary_pretrained_config=SimpleNamespace(), + device=device, + cuda_graph=SimpleNamespace(enable=False), + torch_compile=SimpleNamespace(enable=False), + cpu_offload_config=CpuOffloadConfig(enable=offload_enable, stages=offload_stages), + ) + + +def _make_cosmos3_unit_pipeline(**config_kwargs) -> _Cosmos3OffloadUnitPipeline: + return _Cosmos3OffloadUnitPipeline(_make_cosmos3_unit_config(**config_kwargs)) + + +def test_cosmos3_offload_components_expose_towers_vae_and_guardrails(): + pipeline = _make_cosmos3_unit_pipeline() + pipeline.vae = nn.Linear(2, 2) + text_model = nn.Linear(2, 2) + video_model = nn.Linear(2, 2) + pipeline.safety_checker = _FakeSafetyChecker( + text_models=[text_model], video_models=[video_model] + ) + + components = pipeline.offload_pipeline_components() + + assert ( + components[COSMOS3_REASONER_OFFLOAD_COMPONENT] is pipeline.transformer.language_model.layers + ) + assert components[COSMOS3_GENERATOR_OFFLOAD_COMPONENT] is pipeline.transformer.gen_layers + assert components["vae"] is pipeline.vae + + text_component = components[COSMOS3_TEXT_GUARDRAIL_OFFLOAD_COMPONENT] + video_component = components[COSMOS3_VIDEO_GUARDRAIL_OFFLOAD_COMPONENT] + assert isinstance(text_component, nn.ModuleList) and list(text_component) == [text_model] + assert isinstance(video_component, nn.ModuleList) and list(video_component) == [video_model] + + +def test_cosmos3_guardrail_components_absent_without_safety_checker(): + pipeline = _make_cosmos3_unit_pipeline() + # safety_checker is loaded on rank 0 only; emulate ranks where it is missing. + components = pipeline.offload_pipeline_components() + + assert COSMOS3_REASONER_OFFLOAD_COMPONENT in components + assert COSMOS3_GENERATOR_OFFLOAD_COMPONENT in components + assert COSMOS3_TEXT_GUARDRAIL_OFFLOAD_COMPONENT not in components + assert COSMOS3_VIDEO_GUARDRAIL_OFFLOAD_COMPONENT not in components + + +def test_cosmos3_guardrail_components_wrap_only_nn_modules(): + pipeline = _make_cosmos3_unit_pipeline() + nn_model = nn.Linear(2, 2) + # Blocklist is a plain object (not nn.Module) and must be skipped. + pipeline.safety_checker = _FakeSafetyChecker( + text_models=[object(), nn_model], video_models=[object()] + ) + + components = pipeline.offload_pipeline_components() + + assert list(components[COSMOS3_TEXT_GUARDRAIL_OFFLOAD_COMPONENT]) == [nn_model] + # video guardrail has no nn.Module members -> no component registered. + assert COSMOS3_VIDEO_GUARDRAIL_OFFLOAD_COMPONENT not in components + + +def test_cosmos3_default_offload_stages_exclude_guardrails(): + pipeline = _make_cosmos3_unit_pipeline() + + assert pipeline.default_offload_stages() == ( + (COSMOS3_REASONER_OFFLOAD_COMPONENT,), + (COSMOS3_GENERATOR_OFFLOAD_COMPONENT,), + ) + + +def test_cosmos3_validate_accepts_guardrail_names_even_when_not_loaded(): + # Emulates a non-rank-0 process: guardrail components are configured but absent + # from offload_pipeline_components. Validation must not raise on the guardrails. + pipeline = _make_cosmos3_unit_pipeline( + offload_stages=[ + COSMOS3_REASONER_OFFLOAD_COMPONENT, + COSMOS3_TEXT_GUARDRAIL_OFFLOAD_COMPONENT, + COSMOS3_VIDEO_GUARDRAIL_OFFLOAD_COMPONENT, + ] + ) + available_components = { + COSMOS3_REASONER_OFFLOAD_COMPONENT: pipeline.transformer.language_model.layers + } + + # Guardrail names are tolerated even though they are not in available_components. + pipeline.offloader.validate_configured_stages(pipeline.offloader.stages(), available_components) + + +def test_cosmos3_validate_still_rejects_unknown_component_names(): + pipeline = _make_cosmos3_unit_pipeline(offload_stages=["not_a_real_component"]) + available_components = { + COSMOS3_REASONER_OFFLOAD_COMPONENT: pipeline.transformer.language_model.layers + } + + with pytest.raises( + ValueError, match=r"Unknown cpu_offload_config\.stages entries.*not_a_real_component" + ): + pipeline.offloader.validate_configured_stages( + pipeline.offloader.stages(), available_components + ) diff --git a/tests/unittest/_torch/visual_gen/test_wan22_t2v_offload.py b/tests/unittest/_torch/visual_gen/test_wan22_t2v_offload.py new file mode 100644 index 000000000000..f99119715d45 --- /dev/null +++ b/tests/unittest/_torch/visual_gen/test_wan22_t2v_offload.py @@ -0,0 +1,257 @@ +# SPDX-FileCopyrightText: Copyright (c) 2022-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Tests for Wan 2.2 T2V CPU offloading using the full pipeline. + +Wan 2.2 two-stage T2V is tested with cpu_offload_config.enable=True: + - Wan2.2-T2V-A14B-Diffusers 480x832 + +Offloaded output is checked against the non-offload TRT-LLM baseline. HuggingFace parity +with offloading is in test_wan22_t2v_pipeline.py::test_cosine_similarity_with_offload. +FP8 dynamic quantization is checked against the TRT-LLM baseline only. +CUDA graphs with offloading are not supported (test_wan22_raises_if_cuda_graph_and_offload_enabled). +CUDA-graph correctness is in test_wan22_t2v_pipeline.py. + +Each test loads all pipeline components and calls pipeline.forward(). + +Run all: + pytest tests/unittest/_torch/visual_gen/test_wan22_t2v_offload.py -v -s + +Override checkpoint path: + DIFFUSION_MODEL_PATH_WAN22_T2V=/path/to/wan22 \\ + pytest tests/unittest/_torch/visual_gen/test_wan22_t2v_offload.py -v -s +""" + +import os + +os.environ["TLLM_DISABLE_MPI"] = "1" + +import gc +from pathlib import Path +from typing import Optional + +import pytest +import torch +import torch.nn.functional as F + +from tensorrt_llm._torch.visual_gen.pipeline_loader import PipelineLoader +from tensorrt_llm.visual_gen.args import ( + CpuOffloadConfig, + CudaGraphConfig, + TorchCompileConfig, + VisualGenArgs, +) + + +@pytest.fixture(autouse=True, scope="module") +def _cleanup_mpi_env(): + yield + os.environ.pop("TLLM_DISABLE_MPI", None) + + +@pytest.fixture(autouse=True) +def _cleanup_gpu(): + gc.collect() + torch.cuda.empty_cache() + yield + gc.collect() + torch.cuda.empty_cache() + + +# ============================================================================ +# Path helpers +# ============================================================================ + + +def _llm_models_root() -> Path: + if "LLM_MODELS_ROOT" in os.environ: + root = Path(os.environ["LLM_MODELS_ROOT"]) + else: + root = Path("/home/scratch.trt_llm_data_ci/llm-models/") + if not root.exists(): + root = Path("/scratch.trt_llm_data/llm-models/") + assert root.exists(), ( + "Set LLM_MODELS_ROOT or ensure /home/scratch.trt_llm_data_ci/llm-models/ is accessible." + ) + return root + + +def _checkpoint(env_var: str, default_name: str) -> str: + return os.environ.get(env_var) or str(_llm_models_root() / default_name) + + +WAN22_A14B_PATH = _checkpoint("DIFFUSION_MODEL_PATH_WAN22_T2V", "Wan2.2-T2V-A14B-Diffusers") + +INFER_PROMPT = "A cat sitting on a sunny windowsill watching birds outside." +INFER_NEGATIVE_PROMPT = "" +INFER_NUM_FRAMES = 9 +INFER_NUM_STEPS = 4 +INFER_SEED = 42 +INFER_GUIDANCE_SCALE = 4.0 +COS_SIM_THRESHOLD = 0.99 + + +# ============================================================================ +# Pipeline fixture factory +# ============================================================================ + + +def _make_pipeline( + checkpoint_path: str, + *, + enable_offload: bool = False, + quant_config: Optional[dict] = None, +): + if not os.path.exists(checkpoint_path): + pytest.skip(f"Checkpoint not found: {checkpoint_path}") + args = VisualGenArgs( + model=checkpoint_path, + torch_compile_config=TorchCompileConfig(enable=False), + cpu_offload_config=CpuOffloadConfig(enable=enable_offload), + quant_config=quant_config, + ) + pipeline = PipelineLoader(args).load(skip_warmup=True) + assert pipeline.transformer_2 is not None, ( + "Expected two-stage Wan 2.2 pipeline (transformer_2 should not be None)" + ) + return pipeline + + +def _require_fp8_quant_ops() -> None: + try: + _ = torch.ops.tensorrt_llm.quantize_e4m3_per_tensor + _ = torch.ops.tensorrt_llm.quantize_e4m3_activation + except (AttributeError, RuntimeError) as e: + pytest.skip(f"FP8 quantization ops not available: {e}") + + +@pytest.fixture +def wan22_offload_pipeline(): + pipeline = _make_pipeline(WAN22_A14B_PATH, enable_offload=True) + yield pipeline + del pipeline + torch.cuda.empty_cache() + + +# ============================================================================ +# Shared assertion helpers +# ============================================================================ + + +def _assert_offload_forward(pipeline, *, model: str = "") -> None: + assert pipeline.offloader.stages(), ( + f"{model}: cpu_offload_config.enable=True but offloader.stages() is empty" + ) + assert pipeline.offloader.offload_pipeline is not None, ( + f"{model}: offload pipeline was not initialized after load" + ) + + with torch.no_grad(): + pipeline.forward( + prompt=INFER_PROMPT, + negative_prompt=INFER_NEGATIVE_PROMPT, + height=480, + width=832, + num_frames=INFER_NUM_FRAMES, + num_inference_steps=INFER_NUM_STEPS, + guidance_scale=INFER_GUIDANCE_SCALE, + seed=INFER_SEED, + ) + + print(f"\n ===== Offload — Wan 2.2 {model} 480x832 =====") + print(f" stages: {pipeline.offloader.stages()}") + print(" =================================================") + + +def _assert_offload_matches_baseline( + checkpoint_path: str, + *, + model: str, + quant_config: Optional[dict] = None, +) -> None: + def _capture_video(pipe) -> torch.Tensor: + with torch.no_grad(): + result = pipe.forward( + prompt=INFER_PROMPT, + negative_prompt=INFER_NEGATIVE_PROMPT, + height=480, + width=832, + num_frames=INFER_NUM_FRAMES, + num_inference_steps=INFER_NUM_STEPS, + guidance_scale=INFER_GUIDANCE_SCALE, + seed=INFER_SEED, + ) + return result.video.float() / 255.0 + + offload_pipe = _make_pipeline(checkpoint_path, enable_offload=True, quant_config=quant_config) + offload_video = _capture_video(offload_pipe) + del offload_pipe + gc.collect() + torch.cuda.empty_cache() + + baseline_pipe = _make_pipeline(checkpoint_path, enable_offload=False, quant_config=quant_config) + baseline_video = _capture_video(baseline_pipe) + del baseline_pipe + gc.collect() + torch.cuda.empty_cache() + + assert offload_video.shape == baseline_video.shape, ( + f"{model}: shape mismatch offload {offload_video.shape} vs baseline {baseline_video.shape}" + ) + a_flat = offload_video.float().cpu().reshape(-1) + b_flat = baseline_video.float().cpu().reshape(-1) + cos_sim = F.cosine_similarity(a_flat.unsqueeze(0), b_flat.unsqueeze(0)).clamp(-1.0, 1.0).item() + print( + f"\n ===== Offload accuracy — {model} =====\n" + f" cosine_similarity={cos_sim:.6f} (threshold={COS_SIM_THRESHOLD})\n" + f" ========================================================" + ) + assert cos_sim >= COS_SIM_THRESHOLD, ( + f"{model}: offload vs baseline cosine similarity {cos_sim:.6f} < {COS_SIM_THRESHOLD}" + ) + + +# ============================================================================ +# Tests +# ============================================================================ + + +@pytest.mark.integration +@pytest.mark.wan_t2v +class TestWan22_A14B_Offload: + """Wan2.2-T2V-A14B 480x832 two-stage with CPU offloading.""" + + def test_wan22_offload(self, wan22_offload_pipeline): + _assert_offload_forward(wan22_offload_pipeline, model="T2V-A14B") + + def test_wan22_offload_matches_baseline(self): + _assert_offload_matches_baseline(WAN22_A14B_PATH, model="Wan2.2-T2V-A14B") + + def test_wan22_fp8_offload_matches_baseline(self): + _require_fp8_quant_ops() + _assert_offload_matches_baseline( + WAN22_A14B_PATH, + model="Wan2.2-T2V-A14B FP8", + quant_config={"quant_algo": "FP8", "dynamic": True}, + ) + + +@pytest.mark.integration +@pytest.mark.wan_t2v +class TestWanT2VOffloadCudaGraphRaisesError: + """CUDA graphs plus offloading must raise NotImplementedError on pipeline load.""" + + def test_wan22_raises_if_cuda_graph_and_offload_enabled(self): + if not os.path.exists(WAN22_A14B_PATH): + pytest.skip(f"Checkpoint not found: {WAN22_A14B_PATH}") + args = VisualGenArgs( + model=WAN22_A14B_PATH, + torch_compile_config=TorchCompileConfig(enable=False), + cuda_graph_config=CudaGraphConfig(enable=True), + cpu_offload_config=CpuOffloadConfig(enable=True), + ) + with pytest.raises( + NotImplementedError, + match="CUDA graphs are not supported with visual generation offloading", + ): + PipelineLoader(args).load(skip_warmup=True) diff --git a/tests/unittest/_torch/visual_gen/test_wan22_t2v_pipeline.py b/tests/unittest/_torch/visual_gen/test_wan22_t2v_pipeline.py index c05b506efce4..042b0123c75c 100644 --- a/tests/unittest/_torch/visual_gen/test_wan22_t2v_pipeline.py +++ b/tests/unittest/_torch/visual_gen/test_wan22_t2v_pipeline.py @@ -13,6 +13,9 @@ Model tested: - Wan2.2-T2V-A14B-Diffusers (480x832, 33 frames) +Also covers offloading vs HuggingFace. Offload-specific baseline and CUDA graph +incompatibility tests are in test_wan22_t2v_offload.py. + Run: pytest tests/unittest/_torch/visual_gen/test_wan22_t2v_pipeline.py -v -s @@ -39,6 +42,7 @@ from tensorrt_llm.visual_gen.args import ( AttentionConfig, CacheDiTConfig, + CpuOffloadConfig, TorchCompileConfig, VisualGenArgs, ) @@ -90,15 +94,20 @@ def _checkpoint(env_var: str, default_name: str) -> str: # ============================================================================ -def _load_trtllm_pipeline(checkpoint_path: str): +def _load_trtllm_pipeline( + checkpoint_path: str, + *, + enable_offload: bool = False, +): """Load TRTLLM WanPipeline (two-stage) without torch.compile or warmup.""" if not os.path.exists(checkpoint_path): pytest.skip(f"Checkpoint not found: {checkpoint_path}") - args = VisualGenArgs( + kwargs = dict( model=checkpoint_path, torch_compile_config=TorchCompileConfig(enable=False), + cpu_offload_config=CpuOffloadConfig(enable=enable_offload), ) - return PipelineLoader(args).load(skip_warmup=True) + return PipelineLoader(VisualGenArgs(**kwargs)).load(skip_warmup=True) def _load_hf_pipeline(checkpoint_path: str): @@ -183,10 +192,18 @@ def _assert_pipeline_matches_hf( num_frames: int, guidance_scale: float, model_label: str, + *, + enable_offload: bool = False, ) -> None: """Run TRTLLM and HF pipelines sequentially, compare decoded video output.""" # --- TRTLLM --- - trtllm_pipe = _load_trtllm_pipeline(checkpoint_path) + trtllm_pipe = _load_trtllm_pipeline( + checkpoint_path, + enable_offload=enable_offload, + ) + if enable_offload: + assert trtllm_pipe.offloader.stages(), f"{model_label}: offload stages must be configured" + assert trtllm_pipe.offloader.offload_pipeline is not None # Confirm two-stage denoising is active (Wan 2.2 specific sanity check) assert trtllm_pipe.transformer_2 is not None, ( @@ -271,6 +288,17 @@ def test_cosine_similarity(self): model_label="Wan2.2-T2V-A14B", ) + def test_cosine_similarity_with_offload(self): + _assert_pipeline_matches_hf( + checkpoint_path=WAN22_A14B_PATH, + height=480, + width=832, + num_frames=9, + guidance_scale=4.0, + model_label="Wan2.2-T2V-A14B (offload)", + enable_offload=True, + ) + # ============================================================================ # Two-stage feature fixtures (loaded once per module) From a11d7c76bf9025e18c74725957cbb34b9bbb42a9 Mon Sep 17 00:00:00 2001 From: Rahul Steiger Date: Thu, 11 Jun 2026 02:13:08 -0700 Subject: [PATCH 02/23] updated tests to run post merge Signed-off-by: Rahul Steiger --- tests/integration/test_lists/test-db/l0_b200.yml | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/tests/integration/test_lists/test-db/l0_b200.yml b/tests/integration/test_lists/test-db/l0_b200.yml index 4de0e77ed034..6b6d4f982642 100644 --- a/tests/integration/test_lists/test-db/l0_b200.yml +++ b/tests/integration/test_lists/test-db/l0_b200.yml @@ -246,8 +246,12 @@ l0_b200: - unittest/_torch/visual_gen/test_wan22_i2v_teacache.py - unittest/_torch/visual_gen/test_wan22_t2v_teacache.py - unittest/_torch/visual_gen/test_offloading.py - - unittest/_torch/visual_gen/test_wan22_t2v_offload.py - - unittest/_torch/visual_gen/test_cosmos3_t2v_offload.py + # Heavyweight offload accuracy tests (double pipeline load) run in post_merge; + # keep only lightweight smoke + error-path coverage in pre_merge. + - unittest/_torch/visual_gen/test_wan22_t2v_offload.py::TestWan22_A14B_Offload::test_wan22_offload + - unittest/_torch/visual_gen/test_wan22_t2v_offload.py::TestWanT2VOffloadCudaGraphRaisesError::test_wan22_raises_if_cuda_graph_and_offload_enabled + - unittest/_torch/visual_gen/test_cosmos3_t2v_offload.py::TestCosmos3Offload::test_cosmos3_offload_forward + - unittest/_torch/visual_gen/test_cosmos3_t2v_offload.py::TestCosmos3OffloadCudaGraphRaisesError::test_cosmos3_raises_if_cuda_graph_and_offload_enabled - unittest/_torch/visual_gen/test_wan_transformer.py - unittest/_torch/visual_gen/test_cosmos3_transformer.py - unittest/_torch/visual_gen/test_cosmos3_pipeline.py @@ -359,6 +363,12 @@ l0_b200: - examples/visual_gen/test_visual_gen.py::test_qwenimage_lpips_against_golden TIMEOUT (10) - examples/visual_gen/test_visual_gen.py::test_cosmos3_nano_t2i_lpips_against_golden TIMEOUT (10) - examples/visual_gen/test_visual_gen.py::test_cosmos3_nano_t2v_lpips_against_golden TIMEOUT (15) + # Heavyweight CPU-offload accuracy tests (each loads the full pipeline twice: + # offload + baseline). Kept in post_merge to bound the B200 pre_merge wall-time. + - unittest/_torch/visual_gen/test_wan22_t2v_offload.py::TestWan22_A14B_Offload::test_wan22_offload_matches_baseline + - unittest/_torch/visual_gen/test_wan22_t2v_offload.py::TestWan22_A14B_Offload::test_wan22_fp8_offload_matches_baseline + - unittest/_torch/visual_gen/test_cosmos3_t2v_offload.py::TestCosmos3Offload::test_cosmos3_offload_matches_baseline + - examples/visual_gen/test_visual_gen.py::test_wan_t2v_example_with_offloading - visual_gen/test_visual_gen_benchmark.py::test_offline_benchmark - visual_gen/test_visual_gen_benchmark.py::test_online_benchmark[openai-videos] # ---- moved to post-merge (MoE CI optimization) ---- From f2632d58d8899388bbb84d98f2bd064e2a121edd Mon Sep 17 00:00:00 2001 From: Rahul Steiger Date: Wed, 17 Jun 2026 08:53:28 -0700 Subject: [PATCH 03/23] fix precommit Signed-off-by: Rahul Steiger --- tests/unittest/_torch/visual_gen/test_cosmos3_transformer.py | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/tests/unittest/_torch/visual_gen/test_cosmos3_transformer.py b/tests/unittest/_torch/visual_gen/test_cosmos3_transformer.py index 428e744fba2c..dddd6a55c282 100644 --- a/tests/unittest/_torch/visual_gen/test_cosmos3_transformer.py +++ b/tests/unittest/_torch/visual_gen/test_cosmos3_transformer.py @@ -28,10 +28,7 @@ import torch from tensorrt_llm._torch.modules.linear import Linear -from tensorrt_llm._torch.visual_gen.config import ( - DiffusionModelConfig, - DiffusionPipelineConfig, -) +from tensorrt_llm._torch.visual_gen.config import DiffusionModelConfig, DiffusionPipelineConfig from tensorrt_llm._torch.visual_gen.models.cosmos3.transformer_cosmos3 import Cosmos3VFMTransformer from tensorrt_llm._torch.visual_gen.pipeline_loader import PipelineComponent, PipelineLoader from tensorrt_llm.visual_gen.args import TorchCompileConfig, VisualGenArgs From cc212bb8afbb8e766023cea35b8b797554489cc5 Mon Sep 17 00:00:00 2001 From: Rahul Steiger Date: Thu, 25 Jun 2026 01:29:49 -0700 Subject: [PATCH 04/23] addressed api review Signed-off-by: Rahul Steiger --- examples/visual_gen/models/wan_t2v.py | 32 ------------------- .../models/cosmos3/pipeline_cosmos3.py | 14 ++------ .../visual_gen/models/wan/pipeline_wan.py | 30 ++++++----------- tensorrt_llm/_torch/visual_gen/offloading.py | 13 ++++---- tensorrt_llm/visual_gen/args.py | 9 ++---- .../_torch/visual_gen/test_offloading.py | 30 ++++++++--------- 6 files changed, 35 insertions(+), 93 deletions(-) diff --git a/examples/visual_gen/models/wan_t2v.py b/examples/visual_gen/models/wan_t2v.py index c3b7629773aa..260c8d3a53f6 100644 --- a/examples/visual_gen/models/wan_t2v.py +++ b/examples/visual_gen/models/wan_t2v.py @@ -18,8 +18,6 @@ Usage: python wan_t2v.py python wan_t2v.py --visual_gen_args ../configs/wan2.2-t2v-fp4-1gpu.yaml - python wan_t2v.py --enable_offloading \ - --offload_stages text_encoder,denoising_transformer,denoising_transformer_2,vae """ import argparse @@ -27,15 +25,6 @@ from tensorrt_llm import VisualGen, VisualGenArgs -def _parse_offload_stages(offload_stages: str | None) -> list[str] | None: - if offload_stages is None: - return None - stages = [stage.strip() for stage in offload_stages.split(",") if stage.strip()] - if not stages: - raise ValueError("--offload_stages must contain at least one stage name") - return stages - - def main(): parser = argparse.ArgumentParser(description="Wan Text-to-Video example") parser.add_argument( @@ -51,21 +40,6 @@ def main(): default=None, help="Path to YAML config (same as trtllm-serve --visual_gen_args)", ) - parser.add_argument( - "--enable_offloading", - action="store_true", - help="Enable Wan T2V text-encoder/transformer-block CPU offloading.", - ) - parser.add_argument( - "--offload_stages", - type=str, - default=None, - help=( - "Comma-separated offload stage names, e.g. " - "text_encoder,denoising_transformer,denoising_transformer_2,vae. " - "Implies --enable_offloading." - ), - ) parser.add_argument( "--output_path", type=str, @@ -80,12 +54,6 @@ def main(): else: visual_gen_args = VisualGenArgs() - offload_stages = _parse_offload_stages(args.offload_stages) - if args.enable_offloading or offload_stages is not None: - visual_gen_args.cpu_offload_config.enable = True - if offload_stages is not None: - visual_gen_args.cpu_offload_config.stages = offload_stages - visual_gen = VisualGen(model=args.model, args=visual_gen_args) # --- Model-specific: T2V request construction --- diff --git a/tensorrt_llm/_torch/visual_gen/models/cosmos3/pipeline_cosmos3.py b/tensorrt_llm/_torch/visual_gen/models/cosmos3/pipeline_cosmos3.py index 0cb5e2046ad9..25367d5e0bf1 100644 --- a/tensorrt_llm/_torch/visual_gen/models/cosmos3/pipeline_cosmos3.py +++ b/tensorrt_llm/_torch/visual_gen/models/cosmos3/pipeline_cosmos3.py @@ -112,19 +112,9 @@ def default_offload_stages(self) -> tuple[tuple[str, ...], ...]: """Offload the reasoner and generator towers as separate stages. Only invoked when ``cpu_offload_config.enable`` is true (the base class - short-circuits before calling this). Defaults are CPU-staged today; - non-CPU offload devices skip the defaults and require explicit - ``cpu_offload_config.stages`` from the user. + short-circuits before calling this). Stages are held in CPU storage + while inactive and moved onto the pipeline GPU when active. """ - cpu_offload_config = self.pipeline_config.cpu_offload_config - if cpu_offload_config.device != "cpu": - logger.warning( - "Cosmos3 default offload stages are CPU-only; " - f"cpu_offload_config.device='{cpu_offload_config.device}' has no " - "default stages. Set cpu_offload_config.stages explicitly to " - "stage components on a non-CPU device." - ) - return () return _COSMOS3_DEFAULT_OFFLOAD_STAGES def offload_pipeline_components(self) -> dict[str, nn.Module]: diff --git a/tensorrt_llm/_torch/visual_gen/models/wan/pipeline_wan.py b/tensorrt_llm/_torch/visual_gen/models/wan/pipeline_wan.py index f961c3a4446a..95b0eeeef7b5 100644 --- a/tensorrt_llm/_torch/visual_gen/models/wan/pipeline_wan.py +++ b/tensorrt_llm/_torch/visual_gen/models/wan/pipeline_wan.py @@ -85,12 +85,12 @@ _WAN_SINGLE_TRANSFORMER_OFFLOAD_STAGES = ( ("text_encoder",), - ("denoising_transformer",), + ("transformer",), ) _WAN_TWO_TRANSFORMER_OFFLOAD_STAGES = ( ("text_encoder",), - ("denoising_transformer",), - ("denoising_transformer_2",), + ("transformer",), + ("transformer_2",), ) @@ -218,20 +218,9 @@ def default_offload_stages(self) -> tuple[tuple[str, ...], ...]: """Return WAN offload stages for the loaded transformer topology. Only invoked when ``cpu_offload_config.enable`` is true (the base - class short-circuits before calling this). Defaults are CPU-staged - today; non-CPU offload devices skip the defaults and require - explicit ``cpu_offload_config.stages`` from the user. + class short-circuits before calling this). Stages are held in CPU + storage while inactive and moved onto the pipeline GPU when active. """ - cpu_offload_config = self.pipeline_config.cpu_offload_config - if cpu_offload_config.device != "cpu": - logger.warning( - "Wan default offload stages are CPU-only; " - f"cpu_offload_config.device='{cpu_offload_config.device}' has no " - "default stages. Set cpu_offload_config.stages explicitly to " - "stage components on a non-CPU device." - ) - return () - if self.transformer_2 is not None: return _WAN_TWO_TRANSFORMER_OFFLOAD_STAGES return _WAN_SINGLE_TRANSFORMER_OFFLOAD_STAGES @@ -601,8 +590,9 @@ def forward_fn( # Extract scalar timestep current_t = timestep if timestep.dim() == 0 else timestep[0] - # Select model based on timestep (if two-stage denoising is enabled) - offload_stage = "denoising_transformer" + # Select the transformer for this timestep and keep its public + # component name paired for optional offload staging. + transformer_component_name = "transformer" if boundary_timestep is not None and self.transformer_2 is not None: if current_t >= boundary_timestep: current_model = self.transformer @@ -610,7 +600,7 @@ def forward_fn( else: current_model = self.transformer_2 model_name = "transformer_2 (low-noise)" - offload_stage = "denoising_transformer_2" + transformer_component_name = "transformer_2" # Log when switching models if last_model_used[0] != model_name: @@ -634,7 +624,7 @@ def forward_fn( # T2V: current_t for all frames timestep = current_t.reshape(1, 1).expand(latents.shape[0], nf * nh * nw) - with self.offloader.context_if_requested(offload_stage): + with self.offloader.context_if_requested(transformer_component_name): if _vsa_active and _vsa_builder is not None: # latents: [B, C, T_latent, H_latent, W_latent] raw_latent_shape = (latents.shape[2], latents.shape[3], latents.shape[4]) diff --git a/tensorrt_llm/_torch/visual_gen/offloading.py b/tensorrt_llm/_torch/visual_gen/offloading.py index 34b18659ff17..c7eff0ca7714 100644 --- a/tensorrt_llm/_torch/visual_gen/offloading.py +++ b/tensorrt_llm/_torch/visual_gen/offloading.py @@ -17,7 +17,7 @@ Terminology used throughout this module: - **component**: a named, public sub-model of the pipeline (e.g. ``text_encoder``, - ``denoising_transformer``, ``vae``). + ``transformer``, ``vae``). - **stage**: one step of the offload schedule — a single component, or a group of components that are co-resident on the GPU and run together before being evicted back to CPU. @@ -49,11 +49,12 @@ def _align_offset(offset: int, alignment: int = 256) -> int: def transformer_component_offload_name(component_name: str, transformer_prefix: str) -> str: - """Map an internal transformer attribute to its public offload component name.""" - if component_name == transformer_prefix: - return "denoising_transformer" - if component_name.startswith(f"{transformer_prefix}_"): - return f"denoising_transformer_{component_name[len(transformer_prefix) + 1 :]}" + """Map an internal transformer attribute to its public offload component name. + + Transformer component names follow the diffusers ``model_index.json`` keys + such as ``transformer`` and ``transformer_2``. + """ + del transformer_prefix return component_name diff --git a/tensorrt_llm/visual_gen/args.py b/tensorrt_llm/visual_gen/args.py index 2e290f91ed20..16325e9a6476 100644 --- a/tensorrt_llm/visual_gen/args.py +++ b/tensorrt_llm/visual_gen/args.py @@ -488,7 +488,7 @@ class CpuOffloadConfig(StrictBaseModel): Terminology (used consistently across these fields): - **component**: a named, public sub-model of the pipeline, e.g. - ``text_encoder``, ``denoising_transformer``, ``vae``. + ``text_encoder``, ``transformer``, ``vae``. - **stage**: one step of the offload schedule — a single component, or a group of components that are co-resident on the GPU and run together before being evicted. The ordered list of stages is set by ``stages``. @@ -499,11 +499,6 @@ class CpuOffloadConfig(StrictBaseModel): status="prototype", description="Enable offloading of model components to CPU between pipeline stages.", ) - device: Literal["cpu", "cuda"] = Field( - "cpu", - status="prototype", - description="Device where components are held while not the active stage on the GPU.", - ) pin_memory: bool = Field( True, status="prototype", @@ -516,7 +511,7 @@ class CpuOffloadConfig(StrictBaseModel): "Optional ordered list of stages, named with model-specific public component " "names. Each entry is a single component, or a list of components that are " "co-resident on the GPU and run together as one stage, e.g. " - "['text_encoder', 'denoising_transformer', 'vae'] for Wan T2V, or " + "['text_encoder', 'transformer', 'vae'] for Wan T2V, or " "['reasoner', 'generator', 'text_guardrail', 'video_guardrail', 'vae'] for Cosmos3. " "If omitted, the model chooses default stages." ), diff --git a/tests/unittest/_torch/visual_gen/test_offloading.py b/tests/unittest/_torch/visual_gen/test_offloading.py index 19b57a04a23b..ebe57a06a781 100644 --- a/tests/unittest/_torch/visual_gen/test_offloading.py +++ b/tests/unittest/_torch/visual_gen/test_offloading.py @@ -63,7 +63,7 @@ def _init_transformer(self) -> None: self.transformer = _ToyModule(weight_value=1.0, bias_value=10.0) def default_offload_stages(self) -> tuple[tuple[str, ...], ...]: - return (("denoising_transformer",),) + return (("transformer",),) class _CustomOffloadPipeline(BasePipeline): @@ -147,40 +147,38 @@ def test_configured_offload_stages_override_model_defaults_and_expose_vae(): _make_config( offload_stages=[ "text_encoder", - ["denoising_transformer", "vae"], - "denoising_transformer_2", + ["transformer", "vae"], + "transformer_2", ] ) ) assert pipeline.offloader.stages() == ( ("text_encoder",), - ("denoising_transformer", "vae"), - ("denoising_transformer_2",), + ("transformer", "vae"), + ("transformer_2",), ) components = pipeline.offload_pipeline_components() assert components["text_encoder"] is pipeline.text_encoder - assert components["denoising_transformer"] is pipeline.transformer.blocks + assert components["transformer"] is pipeline.transformer.blocks assert components["vae"] is pipeline.vae assert pipeline.offloader.filter_available_stages(pipeline.offloader.stages(), components) == ( ("text_encoder",), - ("denoising_transformer", "vae"), + ("transformer", "vae"), ) def test_offload_context_resolves_component_to_stageed_stage(): - pipeline = _CustomOffloadPipeline( - _make_config(offload_stages=[["denoising_transformer", "vae"]]) - ) + pipeline = _CustomOffloadPipeline(_make_config(offload_stages=[["transformer", "vae"]])) pipeline.initialize_offload_pipeline() with pipeline.offloader.context("vae"): assert pipeline.offloader.offload_pipeline is not None assert ( pipeline.offloader.offload_pipeline.manager.active_stage_name - == "denoising_transformer+vae" + == "transformer+vae" ) pipeline.cleanup() @@ -188,12 +186,12 @@ def test_offload_context_resolves_component_to_stageed_stage(): def test_initialize_rejects_configured_stage_components_that_are_unavailable_for_model(): pipeline = _CustomOffloadPipeline( - _make_config(offload_stages=[["denoising_transformer", "denoising_transformer_2"]]) + _make_config(offload_stages=[["transformer", "transformer_2"]]) ) with pytest.raises( ValueError, - match=r"Unknown cpu_offload_config\.stages entries.*denoising_transformer_2.*denoising_transformer", + match=r"Unknown cpu_offload_config\.stages entries.*transformer_2.*transformer", ): pipeline.initialize_offload_pipeline() @@ -206,7 +204,7 @@ def test_visual_gen_args_loads_yaml_offload_stages(tmp_path): enable: true stages: - text_encoder - - [denoising_transformer, vae] + - [transformer, vae] """, encoding="utf-8", ) @@ -216,7 +214,7 @@ def test_visual_gen_args_loads_yaml_offload_stages(tmp_path): assert args.cpu_offload_config.enable is True assert args.cpu_offload_config.stages == [ "text_encoder", - ["denoising_transformer", "vae"], + ["transformer", "vae"], ] @@ -236,7 +234,7 @@ def test_pipeline_rejects_unknown_offload_stage_names_for_this_model(): with pytest.raises( ValueError, - match=r"Unknown cpu_offload_config\.stages entries.*transformer\.blocks.*denoising_transformer", + match=r"Unknown cpu_offload_config\.stages entries.*transformer\.blocks.*transformer", ): pipeline.initialize_offload_pipeline() From c377b203ad595b51d2700affaaa50578dd6518b3 Mon Sep 17 00:00:00 2001 From: Rahul Steiger Date: Thu, 25 Jun 2026 02:27:10 -0700 Subject: [PATCH 05/23] fixed pre-commit Signed-off-by: Rahul Steiger --- tests/unittest/_torch/visual_gen/test_offloading.py | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/tests/unittest/_torch/visual_gen/test_offloading.py b/tests/unittest/_torch/visual_gen/test_offloading.py index ebe57a06a781..f9af79a67db5 100644 --- a/tests/unittest/_torch/visual_gen/test_offloading.py +++ b/tests/unittest/_torch/visual_gen/test_offloading.py @@ -176,10 +176,7 @@ def test_offload_context_resolves_component_to_stageed_stage(): with pipeline.offloader.context("vae"): assert pipeline.offloader.offload_pipeline is not None - assert ( - pipeline.offloader.offload_pipeline.manager.active_stage_name - == "transformer+vae" - ) + assert pipeline.offloader.offload_pipeline.manager.active_stage_name == "transformer+vae" pipeline.cleanup() From 11ebadbd52b7aa998e936058dbc10b487c9b5bdf Mon Sep 17 00:00:00 2001 From: Rahul Steiger Date: Thu, 25 Jun 2026 02:32:20 -0700 Subject: [PATCH 06/23] removed stale test Signed-off-by: Rahul Steiger --- tests/integration/test_lists/test-db/l0_b200.yml | 1 - 1 file changed, 1 deletion(-) diff --git a/tests/integration/test_lists/test-db/l0_b200.yml b/tests/integration/test_lists/test-db/l0_b200.yml index 6b6d4f982642..409f57207f74 100644 --- a/tests/integration/test_lists/test-db/l0_b200.yml +++ b/tests/integration/test_lists/test-db/l0_b200.yml @@ -368,7 +368,6 @@ l0_b200: - unittest/_torch/visual_gen/test_wan22_t2v_offload.py::TestWan22_A14B_Offload::test_wan22_offload_matches_baseline - unittest/_torch/visual_gen/test_wan22_t2v_offload.py::TestWan22_A14B_Offload::test_wan22_fp8_offload_matches_baseline - unittest/_torch/visual_gen/test_cosmos3_t2v_offload.py::TestCosmos3Offload::test_cosmos3_offload_matches_baseline - - examples/visual_gen/test_visual_gen.py::test_wan_t2v_example_with_offloading - visual_gen/test_visual_gen_benchmark.py::test_offline_benchmark - visual_gen/test_visual_gen_benchmark.py::test_online_benchmark[openai-videos] # ---- moved to post-merge (MoE CI optimization) ---- From 06ebc452bdb2ea43206621fe64cfa0206906c3d6 Mon Sep 17 00:00:00 2001 From: Rahul Steiger Date: Fri, 26 Jun 2026 10:59:55 +0200 Subject: [PATCH 07/23] Update docs/source/models/visual-generation.md Co-authored-by: Zhenhua Wang <4936589+zhenhuaw-me@users.noreply.github.com> Signed-off-by: Rahul Steiger --- docs/source/models/visual-generation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/models/visual-generation.md b/docs/source/models/visual-generation.md index 37143a792d10..0edc43b92cf0 100644 --- a/docs/source/models/visual-generation.md +++ b/docs/source/models/visual-generation.md @@ -270,7 +270,7 @@ attention_config: ### CPU Offloading (Wan T2V only) -CPU offloading stages move selected Wan T2V pipeline components between CPU and GPU to reduce peak GPU memory usage; enable it with `cpu_offload_config.enable: true` or `--enable_offloading` in the example script. +CPU offloading stages move selected Wan T2V pipeline components between CPU and GPU to reduce peak GPU memory usage; enable it with `cpu_offload_config.enable: true` ### Multi-GPU Parallelism From e963008b301ee782d913e88f69ced87453a4775a Mon Sep 17 00:00:00 2001 From: Rahul Steiger Date: Fri, 26 Jun 2026 02:03:52 -0700 Subject: [PATCH 08/23] reverted change Signed-off-by: Rahul Steiger --- examples/visual_gen/models/wan_t2v.py | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/examples/visual_gen/models/wan_t2v.py b/examples/visual_gen/models/wan_t2v.py index 260c8d3a53f6..d2530d918ee9 100644 --- a/examples/visual_gen/models/wan_t2v.py +++ b/examples/visual_gen/models/wan_t2v.py @@ -49,12 +49,8 @@ def main(): args = parser.parse_args() # Engine config from shared YAML (optional); model-specific defaults apply otherwise. - if args.visual_gen_args: - visual_gen_args = VisualGenArgs.from_yaml(args.visual_gen_args) - else: - visual_gen_args = VisualGenArgs() - - visual_gen = VisualGen(model=args.model, args=visual_gen_args) + extra_args = VisualGenArgs.from_yaml(args.visual_gen_args) if args.visual_gen_args else None + visual_gen = VisualGen(model=args.model, args=extra_args) # --- Model-specific: T2V request construction --- # Start from per-model defaults (steps, guidance, seed, etc.) and override resolution/frames. From 0eecb07018b94d85f1ea125bbb4020cd5485930c Mon Sep 17 00:00:00 2001 From: Rahul Steiger Date: Fri, 26 Jun 2026 02:06:37 -0700 Subject: [PATCH 09/23] added co-resident stage example Signed-off-by: Rahul Steiger --- tensorrt_llm/visual_gen/args.py | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/tensorrt_llm/visual_gen/args.py b/tensorrt_llm/visual_gen/args.py index 16325e9a6476..558835e53410 100644 --- a/tensorrt_llm/visual_gen/args.py +++ b/tensorrt_llm/visual_gen/args.py @@ -491,7 +491,9 @@ class CpuOffloadConfig(StrictBaseModel): ``text_encoder``, ``transformer``, ``vae``. - **stage**: one step of the offload schedule — a single component, or a group of components that are co-resident on the GPU and run together - before being evicted. The ordered list of stages is set by ``stages``. + before being evicted. For example, a stage ``["text_encoder", "transformer"]`` + keeps both components on GPU until that stage completes. The ordered list + of stages is set by ``stages``. """ enable: bool = Field( @@ -510,7 +512,9 @@ class CpuOffloadConfig(StrictBaseModel): description=( "Optional ordered list of stages, named with model-specific public component " "names. Each entry is a single component, or a list of components that are " - "co-resident on the GPU and run together as one stage, e.g. " + "co-resident on the GPU and run together as one stage. For example, " + "[['text_encoder', 'transformer'], 'vae'] keeps text_encoder and " + "transformer co-resident before moving to vae. Example components: " "['text_encoder', 'transformer', 'vae'] for Wan T2V, or " "['reasoner', 'generator', 'text_guardrail', 'video_guardrail', 'vae'] for Cosmos3. " "If omitted, the model chooses default stages." From 2bf1e32d02ef5dde357f50aee7140f445016be30 Mon Sep 17 00:00:00 2001 From: Rahul Steiger Date: Wed, 8 Jul 2026 05:39:23 -0700 Subject: [PATCH 10/23] updated feature matrix Signed-off-by: Rahul Steiger --- docs/source/models/visual-generation.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/source/models/visual-generation.md b/docs/source/models/visual-generation.md index 0edc43b92cf0..4bea82185491 100644 --- a/docs/source/models/visual-generation.md +++ b/docs/source/models/visual-generation.md @@ -49,12 +49,12 @@ Models are auto-detected from the checkpoint directory. Diffusers-format models |---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| | **FLUX.1** | Yes | Yes | Yes | Yes | No | No [^1] | Yes | No | Yes | Yes | Yes | Yes | Yes | Yes | No | | **FLUX.2** | Yes | Yes | Yes | Yes | No | No [^1] | Yes | No | Yes | Yes | Yes | Yes | Yes | Yes | No | -| **Wan 2.1** | Yes | Yes | Yes | Yes | T2V only | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | -| **Wan 2.1 VSA** [^2] | Yes | Yes | Yes | Yes | T2V only | Yes | Yes | Yes | Yes | Yes | Yes | No | No | Yes | Yes | -| **Wan 2.2** | Yes | Yes | Yes [^3] | Yes | T2V only | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | +| **Wan 2.1** | Yes | Yes | Yes | Yes | Yes [^6] | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | +| **Wan 2.1 VSA** [^2] | Yes | Yes | Yes | Yes | Yes [^6] | Yes | Yes | Yes | Yes | Yes | Yes | No | No | Yes | Yes | +| **Wan 2.2** | Yes | Yes | Yes [^3] | Yes | Yes [^6] | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | | **LTX-2** | Yes | Yes | Yes [^4] | Yes | No | Yes | Yes | No | No | Yes | Yes | Yes | Yes | No | No | | **Qwen-Image** [^5] | Yes | Yes | No | No | No | No | Yes | No | Yes | Yes | Yes | Yes | Yes | No | No | -| **Cosmos3** | Yes | Yes | No | No | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | No | Yes | No | +| **Cosmos3** | Yes | Yes | No | No | Yes [^6] | Yes | Yes | Yes | Yes | Yes | Yes | No | No | Yes | No | [^1]: FLUX models use embedded guidance and do not have a separate negative prompt path, so CFG parallelism is not applicable. @@ -66,6 +66,8 @@ Models are auto-detected from the checkpoint directory. Diffusers-format models [^5]: Qwen-Image ships a native BF16 implementation with per-module numerical parity against `diffusers.QwenImagePipeline` (cosine similarity >= 0.999 on the full 20B transformer) and supports `trtllm-serve` / `/v1/images/generations`. VisualGen supports FP8 blockwise and NVFP4 dynamic quantization from BF16 checkpoints, as well as direct loading of statically quantized FP8 and NVFP4 ModelOpt checkpoints. +[^6]: CPU offloading is currently validated for Wan and Cosmos3 text-to-video pipelines. + ## Quick Start Here is a simple example to generate a video with Wan 2.1: From 6c5aaa79e9a80e8fd96f2ea3f77e24d68a4a1530 Mon Sep 17 00:00:00 2001 From: Rahul Steiger Date: Thu, 9 Jul 2026 03:49:06 -0700 Subject: [PATCH 11/23] remove stale name Signed-off-by: Rahul Steiger --- docs/source/models/visual-generation.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source/models/visual-generation.md b/docs/source/models/visual-generation.md index 4bea82185491..27047a9292e6 100644 --- a/docs/source/models/visual-generation.md +++ b/docs/source/models/visual-generation.md @@ -270,9 +270,9 @@ attention_config: vsa_sparsity: 0.90 ``` -### CPU Offloading (Wan T2V only) +### CPU Offloading (Wan and Cosmos3 T2V) -CPU offloading stages move selected Wan T2V pipeline components between CPU and GPU to reduce peak GPU memory usage; enable it with `cpu_offload_config.enable: true` +CPU offloading stages move selected Wan and Cosmos3 T2V pipeline components between CPU and GPU to reduce peak GPU memory usage; enable it with `cpu_offload_config.enable: true`. ### Multi-GPU Parallelism From 434b47c669c2221b76d7183b2a7d16c0e5e5b209 Mon Sep 17 00:00:00 2001 From: Rahul Steiger Date: Fri, 10 Jul 2026 11:11:36 +0200 Subject: [PATCH 12/23] Update docs/source/models/visual-generation.md Co-authored-by: Zhenhua Wang <4936589+zhenhuaw-me@users.noreply.github.com> Signed-off-by: Rahul Steiger --- docs/source/models/visual-generation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/models/visual-generation.md b/docs/source/models/visual-generation.md index 27047a9292e6..9409280b33fd 100644 --- a/docs/source/models/visual-generation.md +++ b/docs/source/models/visual-generation.md @@ -18,7 +18,7 @@ TensorRT-LLM **VisualGen** provides a unified inference stack for diffusion mode - Sparse attention support: see [VisualGen Sparse Attention](../visual-gen/features/sparse-attention.md). - Multi-GPU parallelism (CFG parallel, Ulysses sequence parallel, Tensor parallelism). - **Step caching** — two runtime caching backends (**TeaCache** and **Cache-DiT**) that skip transformer computation on steps where the step-to-step change is small. -- CPU offloading for Wan text-to-video pipelines to reduce peak GPU memory usage. +- CPU offloading to reduce peak GPU memory usage. - `trtllm-serve` integration with OpenAI-compatible API endpoints for image and video generation. ## Supported Models From 2dc9d391be031f4ac0ae060661846eaa3f0d610b Mon Sep 17 00:00:00 2001 From: Rahul Steiger Date: Fri, 10 Jul 2026 11:12:00 +0200 Subject: [PATCH 13/23] Update tests/integration/test_lists/test-db/l0_b200.yml Co-authored-by: Zhenhua Wang <4936589+zhenhuaw-me@users.noreply.github.com> Signed-off-by: Rahul Steiger --- tests/integration/test_lists/test-db/l0_b200.yml | 6 ------ 1 file changed, 6 deletions(-) diff --git a/tests/integration/test_lists/test-db/l0_b200.yml b/tests/integration/test_lists/test-db/l0_b200.yml index 409f57207f74..07d4ca7f6bd5 100644 --- a/tests/integration/test_lists/test-db/l0_b200.yml +++ b/tests/integration/test_lists/test-db/l0_b200.yml @@ -246,12 +246,6 @@ l0_b200: - unittest/_torch/visual_gen/test_wan22_i2v_teacache.py - unittest/_torch/visual_gen/test_wan22_t2v_teacache.py - unittest/_torch/visual_gen/test_offloading.py - # Heavyweight offload accuracy tests (double pipeline load) run in post_merge; - # keep only lightweight smoke + error-path coverage in pre_merge. - - unittest/_torch/visual_gen/test_wan22_t2v_offload.py::TestWan22_A14B_Offload::test_wan22_offload - - unittest/_torch/visual_gen/test_wan22_t2v_offload.py::TestWanT2VOffloadCudaGraphRaisesError::test_wan22_raises_if_cuda_graph_and_offload_enabled - - unittest/_torch/visual_gen/test_cosmos3_t2v_offload.py::TestCosmos3Offload::test_cosmos3_offload_forward - - unittest/_torch/visual_gen/test_cosmos3_t2v_offload.py::TestCosmos3OffloadCudaGraphRaisesError::test_cosmos3_raises_if_cuda_graph_and_offload_enabled - unittest/_torch/visual_gen/test_wan_transformer.py - unittest/_torch/visual_gen/test_cosmos3_transformer.py - unittest/_torch/visual_gen/test_cosmos3_pipeline.py From 1d7602545661f1333b833245fbe5659c096ef4bb Mon Sep 17 00:00:00 2001 From: Rahul Steiger Date: Tue, 14 Jul 2026 10:43:50 +0200 Subject: [PATCH 14/23] Update docs/source/models/visual-generation.md Co-authored-by: Zhenhua Wang <4936589+zhenhuaw-me@users.noreply.github.com> Signed-off-by: Rahul Steiger --- docs/source/models/visual-generation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/models/visual-generation.md b/docs/source/models/visual-generation.md index 9409280b33fd..94007d1fd365 100644 --- a/docs/source/models/visual-generation.md +++ b/docs/source/models/visual-generation.md @@ -270,7 +270,7 @@ attention_config: vsa_sparsity: 0.90 ``` -### CPU Offloading (Wan and Cosmos3 T2V) +### CPU Offloading CPU offloading stages move selected Wan and Cosmos3 T2V pipeline components between CPU and GPU to reduce peak GPU memory usage; enable it with `cpu_offload_config.enable: true`. From 864c1a3b16c1dd37fd69afd8994914e615e8ebef Mon Sep 17 00:00:00 2001 From: Rahul Steiger Date: Tue, 14 Jul 2026 10:44:09 +0200 Subject: [PATCH 15/23] Update docs/source/models/visual-generation.md Co-authored-by: Zhenhua Wang <4936589+zhenhuaw-me@users.noreply.github.com> Signed-off-by: Rahul Steiger --- docs/source/models/visual-generation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/models/visual-generation.md b/docs/source/models/visual-generation.md index 94007d1fd365..bdb63afda643 100644 --- a/docs/source/models/visual-generation.md +++ b/docs/source/models/visual-generation.md @@ -66,7 +66,7 @@ Models are auto-detected from the checkpoint directory. Diffusers-format models [^5]: Qwen-Image ships a native BF16 implementation with per-module numerical parity against `diffusers.QwenImagePipeline` (cosine similarity >= 0.999 on the full 20B transformer) and supports `trtllm-serve` / `/v1/images/generations`. VisualGen supports FP8 blockwise and NVFP4 dynamic quantization from BF16 checkpoints, as well as direct loading of statically quantized FP8 and NVFP4 ModelOpt checkpoints. -[^6]: CPU offloading is currently validated for Wan and Cosmos3 text-to-video pipelines. +[^6]: CPU offloading of Cosmos3 is currently validated the text-to-video pipelines. ## Quick Start From 45b47a81c935c38947daf566165b872f2895678c Mon Sep 17 00:00:00 2001 From: Rahul Steiger Date: Wed, 15 Jul 2026 01:23:30 -0700 Subject: [PATCH 16/23] fixed real_text_lens issue Signed-off-by: Rahul Steiger --- .../models/cosmos3/transformer_cosmos3.py | 24 ++++++++++++------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/tensorrt_llm/_torch/visual_gen/models/cosmos3/transformer_cosmos3.py b/tensorrt_llm/_torch/visual_gen/models/cosmos3/transformer_cosmos3.py index fca93a96ea7b..547a097b4ad0 100644 --- a/tensorrt_llm/_torch/visual_gen/models/cosmos3/transformer_cosmos3.py +++ b/tensorrt_llm/_torch/visual_gen/models/cosmos3/transformer_cosmos3.py @@ -1128,14 +1128,22 @@ def forward( if not self.sharder.is_active: k_und = k_und[:, :max_real_len] v_und = v_und[:, :max_real_len] - hidden_gen = layer( - hidden_gen, - k_und, - v_und, - freqs_gen, - timestep=timestep, - real_text_lens=real_text_lens, - ) + hidden_gen = layer( + hidden_gen, + k_und, + v_und, + freqs_gen, + timestep=timestep, + real_text_lens=real_text_lens, + ) + else: + hidden_gen = layer( + hidden_gen, + k_und, + v_und, + freqs_gen, + timestep=timestep, + ) hidden_gen = self.sharder.gather(hidden_gen, dim=1, unpad_to=S_gen) From 6078df4a54f86b496b198dba20a566a12c2ed2ef Mon Sep 17 00:00:00 2001 From: Rahul Steiger Date: Sun, 2 Aug 2026 12:33:04 -0700 Subject: [PATCH 17/23] fix ltx test Signed-off-by: Rahul Steiger --- tests/unittest/_torch/visual_gen/test_ltx2_pipeline.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/tests/unittest/_torch/visual_gen/test_ltx2_pipeline.py b/tests/unittest/_torch/visual_gen/test_ltx2_pipeline.py index 282fc81cf8c0..8604d2fc9105 100644 --- a/tests/unittest/_torch/visual_gen/test_ltx2_pipeline.py +++ b/tests/unittest/_torch/visual_gen/test_ltx2_pipeline.py @@ -1281,6 +1281,9 @@ def forward(self, x): return self.lin(x) pipeline = object.__new__(ltx2_two_stages.LTX2TwoStagesPipeline) + # __init__ is bypassed here; BasePipeline.device now reads self._device + # (set in __init__), so initialize it explicitly for the warmup path. + pipeline._device = torch.device("cuda") pipeline.pipeline_config = DiffusionPipelineConfig( cuda_graph=CudaGraphConfig(enable=True), torch_compile=TorchCompileConfig(enable=False), From 456aba9b22821431f5652cd9e0592d15f471c4a8 Mon Sep 17 00:00:00 2001 From: Chang Liu <9713593+chang-l@users.noreply.github.com> Date: Mon, 3 Aug 2026 17:05:36 -0700 Subject: [PATCH 18/23] Fix Cosmos3 bare pipeline test devices Signed-off-by: Chang Liu <9713593+chang-l@users.noreply.github.com> --- tests/unittest/_torch/visual_gen/test_cosmos3_pipeline.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/tests/unittest/_torch/visual_gen/test_cosmos3_pipeline.py b/tests/unittest/_torch/visual_gen/test_cosmos3_pipeline.py index ec835fa13e22..64be12be6990 100644 --- a/tests/unittest/_torch/visual_gen/test_cosmos3_pipeline.py +++ b/tests/unittest/_torch/visual_gen/test_cosmos3_pipeline.py @@ -265,7 +265,9 @@ def _make_test_image() -> PIL.Image.Image: @pytest.fixture def cosmos3_format_pipeline(): """Minimal pipeline for prompt formatting helpers (no checkpoint).""" - return Cosmos3OmniMoTPipeline.__new__(Cosmos3OmniMoTPipeline) + pipeline = Cosmos3OmniMoTPipeline.__new__(Cosmos3OmniMoTPipeline) + pipeline._device = torch.device("cpu") + return pipeline def _format_prompt_with_metadata( @@ -664,6 +666,7 @@ def test_v2v_keep_last_smoke(self, cosmos3_pipeline): def test_v2v_flow_shift_override_request_path(self): pipeline = Cosmos3OmniMoTPipeline.__new__(Cosmos3OmniMoTPipeline) + pipeline._device = torch.device("cpu") pipeline.transformer = SimpleNamespace(device=torch.device("cpu")) pipeline.audio_gen = False calls = [] @@ -730,6 +733,7 @@ def test_v2v_rebuilds_the_audio_scheduler_too(self): the checkpoint's flow shift / Karras sigmas and the streams step on different schedules.""" pipeline = Cosmos3OmniMoTPipeline.__new__(Cosmos3OmniMoTPipeline) + pipeline._device = torch.device("cpu") pipeline.transformer = SimpleNamespace(device=torch.device("cpu")) pipeline.audio_gen = True rebuilt = [] From 10138379e570495052a35f107cea0ffaaa82cca9 Mon Sep 17 00:00:00 2001 From: Rahul Steiger Date: Tue, 11 Aug 2026 06:01:25 -0700 Subject: [PATCH 19/23] Merge upstream/main into visual_gen_offloading Signed-off-by: Rahul Steiger --- .github/workflows/blossom-ci.yml | 1 + .pre-commit-config.yaml | 12 +- ATTRIBUTIONS-Python.md | 2 +- README.md | 2 +- cpp/cmake/modules/cuda_configuration.cmake | 6 +- .../batch_manager/peftCacheManager.h | 12 +- cpp/include/tensorrt_llm/common/cudaUtils.h | 8 +- cpp/include/tensorrt_llm/runtime/loraCache.h | 14 +- cpp/tensorrt_llm/CMakeLists.txt | 2 + .../kv_cache_manager_v2/AGENTS.md | 302 + .../kv_cache_manager_v2/CMakeLists.txt | 1 + .../kv_cache_manager_v2/blockRadixTree.cpp | 245 +- .../kv_cache_manager_v2/blockRadixTree.h | 110 +- .../kv_cache_manager_v2/common.h | 166 +- .../kv_cache_manager_v2/config.h | 6 + .../kv_cache_manager_v2/eventManager.cpp | 12 +- .../kv_cache_manager_v2/kvCache.cpp | 48 +- .../kv_cache_manager_v2/kvCache.h | 58 +- .../kv_cache_manager_v2/kvCacheManager.cpp | 40 +- .../kv_cache_manager_v2/kvCacheManager.h | 34 +- .../batch_manager/kv_cache_manager_v2/page.h | 1 - .../kv_cache_manager_v2/tokenIdExt.cpp | 235 + .../kv_cache_manager_v2/tokenIdExt.h | 243 + .../kv_cache_manager_v2/utils/math.h | 9 + .../batch_manager/peftCacheManager.cpp | 46 +- cpp/tensorrt_llm/common/attentionOp.cpp | 4 +- .../nixl_utils/agentBindings.cpp | 10 +- cpp/tensorrt_llm/kernels/CMakeLists.txt | 16 +- .../kernels/IndexerKCacheGather.h | 2 +- .../kernels/IndexerKCacheScatter.h | 2 +- .../moeAlltoAllKernels.cu | 62 +- .../communicationKernels/moeAlltoAllKernels.h | 24 + .../kernels/cuda_graph_grouped_gemm.cu | 219 +- .../cutlass_kernels/include/moe_kernels.h | 1 + .../cutlass_kernels/moe_gemm/moe_kernels.cu | 72 +- cpp/tensorrt_llm/kernels/fmhaDispatcher.cpp | 4 +- .../kernels/fusedQKNormRopeKernel.cu | 178 +- .../kernels/fusedQKNormRopeKernel.h | 10 + cpp/tensorrt_llm/kernels/groupGemm.cu | 293 +- cpp/tensorrt_llm/kernels/groupGemm.h | 10 +- .../kernels/indexerKCacheGather.cu | 2 +- .../kernels/indexerKCacheScatter.cu | 2 +- .../kernels/kdaDecode/CMakeLists.txt | 21 + .../kernels/kdaDecode/kdaDecode.cu | 1611 ++ .../kernels/kdaDecode/kdaDecode.h | 89 + .../kernels/kimiK3AttnRes/CMakeLists.txt | 31 + .../kernels/kimiK3AttnRes/attnResFwd.cu | 1557 ++ .../kernels/kimiK3AttnRes/attnResFwd.h | 59 + cpp/tensorrt_llm/kernels/lora/lora.cpp | 23 +- cpp/tensorrt_llm/kernels/marlin/marlin.cuh | 9 + .../kernels/marlin/marlin_nvfp4.h | 5 + .../kernels/marlin/marlin_nvfp4_gemm.cu | 6 +- .../kernels/marlin/marlin_nvfp4_template.h | 4 +- .../kernels/marlin/marlin_repack.cu | 6 +- .../kernels/sparseAttentionKernels.h | 4 +- cpp/tensorrt_llm/kernels/splitkGroupGemm.cu | 16 +- cpp/tensorrt_llm/kernels/splitkGroupGemm.h | 2 +- .../batchedGemm/KernelRunner.cpp | 48 +- .../batchedGemm/KernelRunner.h | 26 +- .../BatchedGemmInterface.h | 57 +- .../trtllmGen_bmm_export/BatchedGemmOptions.h | 77 +- .../GemmGatedActOptions.h | 23 +- .../trtllmGen_bmm_export/GemmOptions.h | 399 +- .../trtllmGen_bmm_export/KernelMetaInfo.h | 13114 +++++++++++----- .../trtllmGen_bmm_export/KernelParams.h | 38 +- .../trtllmGen_bmm_export/KernelParamsDecl.h | 34 +- .../trtllmGen_bmm_export/KernelTraits.h | 129 +- .../trtllmGen_bmm_export/config.json | 1780 ++- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2N_bM_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...Ov_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...Ov_bM_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...asFp32M_bN_rgTma_clmp_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...asFp32M_bN_rgTma_clmp_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst | 4 +- ...x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...dS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...dS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...dS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...dS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...dS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...dS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...W8_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...W8_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...dS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...dS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...dS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...dS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...dS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...dS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ..._schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst | 4 +- ...x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...dS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ..._schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst | 4 +- ...x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...dS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...dS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...dS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ..._schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst | 4 +- ...S_relu2_bN_rgTma_clmp_sm100f.cubin.tar.zst | 4 +- ..._schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst | 4 +- ...S_relu2_bN_rgTma_clmp_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...Ov_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...2M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...stsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...stsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...lmp_geGlu_lsfaW2_dynB_sm100f.cubin.tar.zst | 4 +- ...lmp_geGlu_lsfaW4_dynB_sm100f.cubin.tar.zst | 4 +- ...mp_swiGlu_lsfaW2_dynB_sm100f.cubin.tar.zst | 4 +- ...mp_swiGlu_lsfaW4_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_lsfaW2_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_lsfaW4_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_lsfaW2_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_lsfaW4_dynB_sm100f.cubin.tar.zst | 4 +- ...rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...stsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...stsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...eGlu_lbW4_lsfbW4_dynB_sm100f.cubin.tar.zst | 4 +- ...iGlu_lbW4_lsfbW4_dynB_sm100f.cubin.tar.zst | 4 +- ...clmp_lbW4_lsfbW4_dynB_sm100f.cubin.tar.zst | 4 +- ...clmp_lbW4_lsfbW4_dynB_sm100f.cubin.tar.zst | 4 +- ...rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...stsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...stsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...stsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...stsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...stsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...stsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...stsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...stsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...eGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst | 4 +- ...iGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst | 4 +- ..._clmp_geGlu_lbW4_dynB_sm100f.cubin.tar.zst | 4 +- ...clmp_swiGlu_lbW4_dynB_sm100f.cubin.tar.zst | 4 +- ...clmp_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst | 4 +- ..._rgTma_clmp_lbW4_dynB_sm100f.cubin.tar.zst | 4 +- ...clmp_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst | 4 +- ..._rgTma_clmp_lbW4_dynB_sm100f.cubin.tar.zst | 4 +- ...eGlu_laW8_lsfaW4_dynB_sm100f.cubin.tar.zst | 4 +- ...iGlu_laW8_lsfaW4_dynB_sm100f.cubin.tar.zst | 4 +- ...clmp_laW8_lsfaW4_dynB_sm100f.cubin.tar.zst | 4 +- ...clmp_laW8_lsfaW4_dynB_sm100f.cubin.tar.zst | 4 +- ...rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...stsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...stsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...stsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...stsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...stsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...stsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...stsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...stsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...stsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...stsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...stsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...stsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...stsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...stsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...stsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...stsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...stsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...stsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...stsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...stsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...asFp32M_bN_rgTma_clmp_sm100f.cubin.tar.zst | 4 +- ...rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...asFp32M_bN_rgTma_clmp_sm100f.cubin.tar.zst | 4 +- ...rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst | 4 +- ...clmp_swiGlu_lbW8_dynB_sm100f.cubin.tar.zst | 4 +- ..._rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst | 4 +- ..._rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst | 4 +- ...clmp_swiGlu_lbW8_dynB_sm100f.cubin.tar.zst | 4 +- ..._rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst | 4 +- ..._rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...clmp_swiGlu_lbW8_dynB_sm100f.cubin.tar.zst | 4 +- ..._rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst | 4 +- ..._rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst | 4 +- ...clmp_swiGlu_lbW8_dynB_sm100f.cubin.tar.zst | 4 +- ..._rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst | 4 +- ..._rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst | 4 +- ...clmp_swiGlu_lbW8_dynB_sm100f.cubin.tar.zst | 4 +- ..._rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst | 4 +- ..._rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst | 4 +- ...clmp_swiGlu_lbW8_dynB_sm100f.cubin.tar.zst | 4 +- ..._rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst | 4 +- ..._rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...clmp_swiGlu_lbW8_dynB_sm100f.cubin.tar.zst | 4 +- ..._rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst | 4 +- ..._rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst | 4 +- ...clmp_swiGlu_lbW8_dynB_sm100f.cubin.tar.zst | 4 +- ..._rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst | 4 +- ..._rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ...tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst | 4 +- ..._rgTma_clmp_lbW4_dynB_sm100f.cubin.tar.zst | 4 +- ..._rgTma_clmp_lbW4_dynB_sm100f.cubin.tar.zst | 4 +- ..._rgTma_clmp_lbW4_dynB_sm100f.cubin.tar.zst | 4 +- ..._rgTma_clmp_lbW4_dynB_sm100f.cubin.tar.zst | 4 +- ..._rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst | 4 +- ..._rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst | 4 +- ..._rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst | 4 +- ..._rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst | 4 +- ..._rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst | 4 +- ..._rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst | 4 +- ..._rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst | 4 +- ..._rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst | 4 +- ..._rgTma_clmp_lbW4_dynB_sm100f.cubin.tar.zst | 4 +- ..._rgTma_clmp_lbW4_dynB_sm100f.cubin.tar.zst | 4 +- ..._rgTma_clmp_lbW4_dynB_sm100f.cubin.tar.zst | 4 +- ..._rgTma_clmp_lbW4_dynB_sm100f.cubin.tar.zst | 4 +- ..._schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst | 4 +- ..._rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst | 4 +- ..._rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst | 4 +- ..._schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst | 4 +- ..._rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst | 4 +- ..._rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst | 4 +- ..._schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst | 4 +- ...S_relu2_bN_rgTma_clmp_sm100f.cubin.tar.zst | 4 +- ..._schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst | 4 +- ...S_relu2_bN_rgTma_clmp_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...asFp32M_bN_rgTma_clmp_sm100f.cubin.tar.zst | 4 +- ...asFp32M_bN_rgTma_clmp_sm100f.cubin.tar.zst | 4 +- ..._schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst | 4 +- ..._schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst | 4 +- ..._schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst | 4 +- ...S_relu2_bN_rgTma_clmp_sm100f.cubin.tar.zst | 4 +- ..._schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst | 4 +- ...S_relu2_bN_rgTma_clmp_sm100f.cubin.tar.zst | 4 +- ...uGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst | 3 + ...iGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst | 4 +- ...iGlu_lbW4_lsfbW4_dynB_sm100f.cubin.tar.zst | 4 +- ...uGlu_lbW4_lsfbW4_dynB_sm100f.cubin.tar.zst | 3 + ...uGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst | 3 + ...iGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst | 4 +- ...uGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst | 3 + ...iGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst | 4 +- ...iGlu_lbW4_lsfbW4_dynB_sm100f.cubin.tar.zst | 4 +- ...uGlu_lbW4_lsfbW4_dynB_sm100f.cubin.tar.zst | 3 + ...uGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst | 3 + ...iGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst | 4 +- ...Tma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst | 3 + ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...Tma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst | 3 + ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...Tma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst | 3 + ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...Tma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst | 3 + ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...Tma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst | 3 + ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...Tma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst | 3 + ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...Tma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst | 3 + ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...Tma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst | 3 + ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...iGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst | 4 +- ...uGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst | 3 + ...iGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst | 4 +- ...uGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst | 3 + ...Tma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst | 3 + ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...Tma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst | 3 + ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...Tma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst | 3 + ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...Tma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst | 3 + ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...Tma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst | 3 + ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...Tma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst | 3 + ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...Tma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst | 3 + ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...Tma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst | 3 + ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...uGlu_lbW1_lsfbW1_dynB_sm100f.cubin.tar.zst | 3 + ...iGlu_lbW1_lsfbW1_dynB_sm100f.cubin.tar.zst | 4 +- ...uGlu_lbW1_lsfbW1_dynB_sm100f.cubin.tar.zst | 3 + ...iGlu_lbW1_lsfbW1_dynB_sm100f.cubin.tar.zst | 4 +- ...uGlu_lbW1_lsfbW1_dynB_sm100f.cubin.tar.zst | 3 + ...iGlu_lbW1_lsfbW1_dynB_sm100f.cubin.tar.zst | 4 +- ...uGlu_lbW1_lsfbW1_dynB_sm100f.cubin.tar.zst | 3 + ...iGlu_lbW1_lsfbW1_dynB_sm100f.cubin.tar.zst | 4 +- ...Tma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst | 3 + ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...Tma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst | 3 + ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...Tma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst | 3 + ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...Tma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst | 3 + ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...Tma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst | 3 + ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...Tma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst | 3 + ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...Tma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst | 3 + ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...Tma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst | 3 + ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...Tma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst | 3 + ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...Tma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst | 3 + ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...Tma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst | 3 + ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...Tma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst | 3 + ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...Tma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst | 3 + ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...Tma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst | 3 + ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...Tma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst | 3 + ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...Tma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst | 3 + ...gTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst | 4 +- ...iGlu_laW8_lsfaW4_dynB_sm100f.cubin.tar.zst | 4 +- ...iGlu_laW8_lsfaW4_dynB_sm100f.cubin.tar.zst | 4 +- .../blockScaleMoe/DevKernel.cu | 29 +- .../blockScaleMoe/routing/RoutingCustom.cu | 422 +- .../routing/RoutingCustomPolicy.cuh | 22 + .../blockScaleMoe/routing/RoutingKernel.h | 6 + .../trtllmGenKernels/blockScaleMoe/runner.cu | 18 +- .../nanobind/batch_manager/bindings.cpp | 164 + .../nanobind/batch_manager/kvCacheManager.cpp | 3 + .../batch_manager/kvCacheManagerV2.cpp | 424 +- cpp/tensorrt_llm/nanobind/thop/bindings.cpp | 8 +- cpp/tensorrt_llm/runtime/loraCache.cpp | 32 +- cpp/tensorrt_llm/runtime/loraUtils.cpp | 10 +- cpp/tensorrt_llm/runtime/loraUtils.h | 6 +- cpp/tensorrt_llm/thop/CMakeLists.txt | 2 + cpp/tensorrt_llm/thop/attentionOp.cpp | 45 +- cpp/tensorrt_llm/thop/attentionOp.h | 7 +- cpp/tensorrt_llm/thop/attnResOp.cpp | 142 + cpp/tensorrt_llm/thop/fusedQKNormRopeOp.cpp | 117 +- cpp/tensorrt_llm/thop/kdaDecodeOp.cpp | 252 + cpp/tensorrt_llm/thop/loraOp.cpp | 19 +- cpp/tensorrt_llm/thop/moeAlltoAllOp.cpp | 15 + cpp/tensorrt_llm/thop/moeOp.cpp | 56 +- .../unit_tests/batch_manager/CMakeLists.txt | 5 + .../kvCacheManagerV2DigestPoolTest.cpp | 251 + .../kvCacheManagerV2StatsTest.cpp | 4 +- .../batch_manager/peftCacheManagerTest.cpp | 69 +- docker/Dockerfile.multi | 7 + docker/README.md | 8 +- ...blog17_Sparse_Attention_in_TensorRT-LLM.md | 4 +- ...ion_Across_NVL72_Rack_with_TensorRT-LLM.md | 44 +- ..._Workload_Optimizations_in_TensorRT-LLM.md | 10 +- docs/source/commands/trtllm-bench.rst | 142 +- ...loyment-guide-for-deepseek-r1-on-trtllm.md | 6 +- .../deployment-guide-for-kimi-k3-on-trtllm.md | 359 + docs/source/deployment-guide/index.rst | 1 + docs/source/developer-guide/overview.md | 29 + .../sparse-attention-development-guide.md | 32 +- docs/source/developer-guide/telemetry.md | 2 +- docs/source/features/kvcache.md | 36 +- docs/source/features/sampling.md | 53 +- .../legacy/performance/perf-benchmarking.md | 4 +- docs/source/models/encoder-decoder.md | 152 +- docs/source/models/supported-models.md | 22 + docs/source/models/visual-generation.md | 15 +- docs/source/torch/adding_custom_kernels.md | 4 +- examples/constraints.txt | 2 +- .../slurm/benchmark/run_benchmark.sh | 16 + .../slurm/benchmark/start_server.sh | 24 +- .../slurm/benchmark/start_worker.sh | 35 +- .../configs/kda_payload_kimi_k3.yaml | 83 + .../run_cache_transceiver_test.py | 40 +- examples/kimi_k3/README.md | 211 + examples/kimi_k3/disagg/README.md | 165 + .../disagg/benchmark_kimi_k3_dep16.yaml | 136 + examples/kimi_k3/disagg/ctx_config.yaml | 67 + .../kimi_k3/disagg/disagg_proxy_config.yaml | 23 + examples/kimi_k3/disagg/gen_config_no_sa.yaml | 50 + examples/kimi_k3/eval_extra_llm_options.yaml | 19 + .../kimi_k3/eval_extra_llm_options_reuse.yaml | 30 + examples/kimi_k3/perf_sweep/acc_sweep.sbatch | 177 + examples/kimi_k3/perf_sweep/perf_sweep.sbatch | 264 + .../kimi_k3/perf_sweep/submit_acc_sweep.sh | 63 + .../kimi_k3/perf_sweep/submit_perf_sweep.sh | 71 + examples/kimi_k3/quick_start_kimi_k3.py | 89 + examples/kimi_k3/quick_start_kimi_k3.sbatch | 116 + examples/kimi_k3/run_gsm8k_kimi_k3.sbatch | 158 + .../run_serving_benchmark_kimi_k3.sbatch | 233 + examples/layer_wise_benchmarks/README.md | 20 +- examples/layer_wise_benchmarks/run.py | 4 +- examples/llm-api/llm_sparse_attention.py | 4 +- examples/llm-api/quickstart_advanced.py | 4 +- examples/longbench/eval_longbench_v1.py | 4 +- examples/models/core/exaone/README.md | 2 +- examples/wide_ep/README.md | 4 +- examples/wide_ep/ep_load_balancer/README.md | 7 +- examples/wide_ep/slurm_scripts/config.yaml | 2 +- .../slurm_scripts/kimi-k2-thinking.yaml | 2 +- jenkins/Build.groovy | 39 +- jenkins/BuildDockerImage.groovy | 85 +- jenkins/L0_MergeRequest.groovy | 102 +- jenkins/L0_Test.groovy | 347 +- jenkins/TensorRT_LLM_PLC.groovy | 117 +- jenkins/controlCCache.groovy | 5 +- jenkins/current_image_tags.properties | 12 +- jenkins/runPerfSanityTriage.groovy | 5 +- jenkins/scripts/cbts/README.md | 4 +- jenkins/scripts/cbts/coverage_utils/README.md | 1 + .../cbts/coverage_utils/sitecustomize.py | 14 +- jenkins/scripts/cbts/rules/README.md | 29 +- jenkins/scripts/cbts/rules/visual_gen_rule.py | 34 +- jenkins/scripts/perf/local/run_disagg.sh | 3 +- jenkins/scripts/perf/submit.py | 65 +- .../pulse_in_pipeline_scanning/main.py | 126 +- .../submit_preapproved_candidates.py | 40 + .../submit_report.py | 226 +- .../utils/common.py | 1 + .../pulse_in_pipeline_scanning/utils/es.py | 221 +- .../utils/report.py | 4 +- .../utils/triage.py | 162 + legacy-files.txt | 5 +- pyproject.toml | 5 +- requirements-dev.txt | 2 + requirements.txt | 2 +- ruff-legacy.toml | 5 +- scripts/build_wheel.py | 33 +- scripts/rename_docker_images.py | 10 +- security_scanning/docs/poetry.lock | 30 +- security_scanning/examples/apps/poetry.lock | 10 +- .../examples/auto_deploy/poetry.lock | 144 +- .../models/contrib/hyperclovax/poetry.lock | 16 +- security_scanning/examples/ngram/poetry.lock | 216 +- .../examples/quantization/poetry.lock | 240 +- .../examples/ray_orchestrator/poetry.lock | 120 +- .../examples/ray_orchestrator/pyproject.toml | 2 +- security_scanning/examples/serve/poetry.lock | 6360 -------- .../examples/serve/pyproject.toml | 1 - .../examples/trtllm-eval/poetry.lock | 256 +- security_scanning/metadata.json | 4 +- security_scanning/poetry.lock | 271 +- security_scanning/pyproject.toml | 10 +- security_scanning/triton_backend/poetry.lock | 226 +- setup.py | 14 +- tensorrt_llm/__init__.py | 211 +- tensorrt_llm/_bootstrap.py | 176 + tensorrt_llm/_common.py | 83 +- tensorrt_llm/_mnnvl_utils.py | 80 +- .../_torch/attention_backend/fmha/__init__.py | 2 + .../attention_backend/fmha/cute_dsl_mla.py | 544 + .../_torch/attention_backend/fmha/fallback.py | 31 +- .../fmha/flashinfer_trtllm_gen.py | 13 +- .../attention_backend/fmha/msa_sparse_gqa.py | 6 +- .../_torch/attention_backend/fmha/registry.py | 2 + .../_torch/attention_backend/interface.py | 36 +- .../attention_backend/sparse/__init__.py | 11 +- .../sparse/deepseek_v4/__init__.py | 16 +- .../sparse/deepseek_v4/backend.py | 271 + .../sparse/deepseek_v4/cache_manager.py | 65 +- .../sparse/deepseek_v4/compressor.py | 7 +- .../sparse/deepseek_v4/indexer.py | 393 + .../sparse/deepseek_v4/kernels.py | 287 + .../{deepseek_v4.py => metadata.py} | 876 +- .../sparse/deepseek_v4/module.py | 859 + .../sparse/deepseek_v4/params.py | 100 + .../_torch/attention_backend/sparse/dsa.py | 3538 ----- .../attention_backend/sparse/dsa/__init__.py | 53 + .../attention_backend/sparse/dsa/backend.py | 191 + .../sparse/dsa/cache_manager.py | 622 + .../sparse/dsa/custom_ops.py | 148 + .../attention_backend/sparse/dsa/indexer.py | 2092 +++ .../attention_backend/sparse/dsa/kernels.py | 294 + .../attention_backend/sparse/dsa/metadata.py | 1217 ++ .../attention_backend/sparse/dsa/module.py | 621 + .../attention_backend/sparse/dsa/params.py | 68 + .../_torch/attention_backend/sparse/hooks.py | 245 + .../sparse/minimax_m3/msa_backend.py | 13 +- .../_torch/attention_backend/sparse/params.py | 57 +- .../sparse/{utils.py => registry.py} | 37 +- .../_torch/attention_backend/sparse/rocket.py | 1186 -- .../sparse/rocket/__init__.py | 19 + .../sparse/rocket/backend.py | 601 + .../sparse/rocket/cache_manager.py | 293 + .../sparse/{kernel.py => rocket/kernels.py} | 2751 ++-- .../sparse/rocket/metadata.py | 337 + .../attention_backend/sparse/rocket/module.py | 33 + .../attention_backend/sparse/rocket/params.py | 47 + .../sparse/skip_softmax/__init__.py | 28 + .../params.py} | 63 +- .../_torch/attention_backend/trtllm.py | 194 +- .../_torch/attention_backend/utils.py | 106 + .../_torch/attention_backend/vanilla.py | 5 +- .../custom_ops/attention/trtllm_attention.py | 2 +- .../auto_deploy/custom_ops/mla/trtllm_mla.py | 8 +- tensorrt_llm/_torch/auto_deploy/llm_args.py | 7 + .../models/custom/modeling_eagle.py | 2 +- .../_torch/auto_deploy/shim/ad_executor.py | 4 +- tensorrt_llm/_torch/compilation/backend.py | 6 - .../_torch/compilation/piecewise_optimizer.py | 1 - tensorrt_llm/_torch/compilation/utils.py | 33 + tensorrt_llm/_torch/configs/__init__.py | 8 + tensorrt_llm/_torch/configs/kimi_linear.py | 175 + tensorrt_llm/_torch/custom_ops/__init__.py | 18 + .../_torch/custom_ops/cpp_custom_ops.py | 54 + .../_torch/custom_ops/cute_dsl_custom_ops.py | 779 +- .../custom_ops/cute_dsl_kimi_k3_custom_ops.py | 1352 ++ .../cute_dsl_kimi_k3_kda_mtp_ops.py | 755 + .../_torch/custom_ops/torch_custom_ops.py | 2 +- .../custom_ops/trtllm_gen_custom_ops.py | 8 +- .../blackwell/attention/__init__.py | 2 + .../blackwell/attention/mla/__init__.py | 10 + .../attention/mla/mla_decode_fp16.py | 4504 ++++++ .../blackwell/attention/mla/mla_decode_fp8.py | 4461 ++++++ .../blackwell/attention/mla/mla_helpers.py | 310 + .../blackwell/kimi_k3_kda/__init__.py | 15 + .../blackwell/kimi_k3_kda/akk_inverse.py | 1075 ++ .../blackwell/kimi_k3_kda/fused_k123.py | 2305 +++ .../blackwell/kimi_k3_kda/fused_k1234.py | 2278 +++ .../blackwell/kimi_k3_kda/k4_persistent.py | 1496 ++ .../blackwell/kimi_k3_kda/kda_mtp_decode.py | 779 + .../_torch/disaggregation/native/auxiliary.py | 75 + .../disaggregation/native/bounce/config.py | 56 +- .../disaggregation/native/bounce/core.py | 13 +- .../disaggregation/native/bounce/impl.py | 111 +- .../disaggregation/native/mixers/ssm/peer.py | 129 + .../_torch/disaggregation/native/peer.py | 27 +- .../_torch/disaggregation/native/transfer.py | 105 +- .../_torch/disaggregation/transceiver.py | 38 +- tensorrt_llm/_torch/model_config.py | 46 +- tensorrt_llm/_torch/models/__init__.py | 152 +- tensorrt_llm/_torch/models/_arch_index.py | 223 + .../models/checkpoints/hf/weight_loader.py | 61 + tensorrt_llm/_torch/models/modeling_auto.py | 14 +- .../_torch/models/modeling_deepseekv3.py | 12 +- .../_torch/models/modeling_deepseekv4.py | 19 +- .../_torch/models/modeling_exaone4_5.py | 27 + .../_torch/models/modeling_gemma4mm.py | 120 +- .../_torch/models/modeling_gpt_oss.py | 24 + .../_torch/models/modeling_kimi_k25.py | 8 + .../_torch/models/modeling_kimi_linear.py | 2703 ++++ tensorrt_llm/_torch/models/modeling_laguna.py | 10 +- tensorrt_llm/_torch/models/modeling_llama.py | 6 +- .../_torch/models/modeling_minimaxm2.py | 9 +- .../_torch/models/modeling_minimaxm3.py | 105 +- .../_torch/models/modeling_mistral.py | 3 +- .../_torch/models/modeling_nemotron_h.py | 20 +- .../_torch/models/modeling_qwen3_5.py | 47 +- .../_torch/models/modeling_qwen3_next.py | 20 +- .../_torch/models/modeling_speculative.py | 226 + tensorrt_llm/_torch/models/modeling_utils.py | 164 +- .../modules/ATTENTION_DEVELOPER_GUIDE.md | 76 +- tensorrt_llm/_torch/modules/attention.py | 67 +- .../_torch/modules/fla/cached_replay.py | 137 +- .../modules/fused_moe/MOE_DEVELOPER_GUIDE.md | 7 +- .../communication/communication_factory.py | 16 +- .../communication/deep_ep_low_latency.py | 17 + .../nvlink_two_sided_flashinfer.py | 58 +- .../modules/fused_moe/configurable_moe.py | 49 +- .../_torch/modules/fused_moe/create_moe.py | 107 +- .../fused_moe/fused_moe_cute_dsl_b12x.py | 43 +- .../modules/fused_moe/fused_moe_cutlass.py | 2 + .../modules/fused_moe/fused_moe_marlin.py | 8 +- .../modules/fused_moe/fused_moe_trtllm_gen.py | 146 +- .../_torch/modules/fused_moe/impl_base.py | 75 + .../_torch/modules/fused_moe/impl_contract.py | 305 + .../_torch/modules/fused_moe/impl_identity.py | 149 + .../_torch/modules/fused_moe/interface.py | 5 + .../modules/fused_moe/moe_op_backend.py | 33 + .../_torch/modules/fused_moe/moe_scheduler.py | 19 +- .../_torch/modules/fused_moe/quantization.py | 200 +- .../fused_ops/fused_qk_norm_rope_gate.py | 145 +- tensorrt_llm/_torch/modules/gated_mlp.py | 22 +- .../modules/kimi_k3_attn_res/__init__.py | 30 + .../kimi_k3_attn_res/_attn_res_kernels.py | 84 + .../kimi_k3_attn_res/kimi_k3_attn_res.py | 403 + .../_torch/modules/kimi_k3_mla/__init__.py | 25 + .../kimi_k3_mla/kimi_k3_mla_attention.py | 242 + .../_torch/modules/kimi_k3_moe/__init__.py | 18 + .../_torch/modules/kimi_k3_moe/_mlp.py | 302 + .../modules/kimi_k3_moe/kimi_k3_moe_gate.py | 281 + .../_torch/modules/kimi_kda/__init__.py | 17 + .../_torch/modules/kimi_kda/_kda_decode.py | 221 + .../_torch/modules/kimi_kda/_kda_kernels.py | 390 + .../_torch/modules/kimi_kda/kimi_kda_mixer.py | 723 + tensorrt_llm/_torch/modules/linear.py | 473 +- .../_torch/modules/mamba/gdn_mixer.py | 4 +- .../_torch/modules/mamba/layernorm_gated.py | 38 +- .../_torch/modules/mamba/mamba2_metadata.py | 27 +- tensorrt_llm/_torch/modules/mla.py | 2237 +-- tensorrt_llm/_torch/modules/mlp.py | 43 +- .../_torch/modules/qk_norm_attention.py | 11 +- .../peft/lora/cuda_graph_lora_manager.py | 16 + tensorrt_llm/_torch/peft/lora/layer.py | 106 +- tensorrt_llm/_torch/peft/lora/validation.py | 2 +- tensorrt_llm/_torch/pyexecutor/_util.py | 227 +- .../_torch/pyexecutor/config_utils.py | 101 +- .../_torch/pyexecutor/cuda_graph_runner.py | 963 +- .../_torch/pyexecutor/hang_detector.py | 245 + .../_torch/pyexecutor/kv_cache_manager_v2.py | 59 +- tensorrt_llm/_torch/pyexecutor/llm_request.py | 29 + .../_torch/pyexecutor/mamba_cache_manager.py | 577 +- .../_torch/pyexecutor/model_engine.py | 1308 +- .../_torch/pyexecutor/model_loader.py | 72 +- .../_torch/pyexecutor/perf_metrics_manager.py | 2 +- tensorrt_llm/_torch/pyexecutor/py_executor.py | 687 +- .../_torch/pyexecutor/py_executor_creator.py | 9 +- .../_torch/pyexecutor/resource_manager.py | 167 +- .../pyexecutor/sampler/ops/flashinfer.py | 25 + .../_torch/pyexecutor/sampler/penalties.py | 8 +- .../_torch/pyexecutor/sampler/sampler.py | 207 +- .../_torch/pyexecutor/sampler/top_p_decay.py | 2 +- tensorrt_llm/_torch/speculative/__init__.py | 10 +- .../_torch/speculative/accept_stats.py | 390 + tensorrt_llm/_torch/speculative/dflash.py | 155 +- .../_torch/speculative/draft_target.py | 13 - tensorrt_llm/_torch/speculative/drafter.py | 7 +- tensorrt_llm/_torch/speculative/eagle3.py | 48 +- tensorrt_llm/_torch/speculative/interface.py | 175 +- tensorrt_llm/_torch/speculative/mtp.py | 50 +- tensorrt_llm/_torch/speculative/sa_worker.py | 20 - .../_torch/speculative/spec_sampler_base.py | 153 +- .../_torch/speculative/suffix_automaton.py | 45 +- tensorrt_llm/_torch/speculative/utils.py | 56 +- tensorrt_llm/_torch/utils.py | 11 +- .../_torch/visual_gen/models/__init__.py | 3 +- .../_torch/visual_gen/models/wan/__init__.py | 2 + .../_torch/visual_gen/models/wan/defaults.py | 12 + .../_torch/visual_gen/models/wan/dup_up3d.py | 229 + .../visual_gen/models/wan/parallel_vae.py | 53 +- .../visual_gen/models/wan/pipeline_fastwan.py | 197 + .../_torch/visual_gen/models/wan/wan_vae.py | 45 +- .../_torch/visual_gen/modules/vae/conv.py | 230 +- tensorrt_llm/_utils.py | 55 +- tensorrt_llm/commands/bench.py | 3 +- tensorrt_llm/commands/serve.py | 34 +- tensorrt_llm/commands/utils.py | 18 +- tensorrt_llm/evaluate/lm_eval.py | 55 +- tensorrt_llm/executor/base_worker.py | 51 +- tensorrt_llm/executor/proxy.py | 7 +- tensorrt_llm/executor/request.py | 2 - tensorrt_llm/executor/result.py | 34 + tensorrt_llm/executor/worker.py | 3 +- tensorrt_llm/inputs/registry.py | 86 +- tensorrt_llm/inputs/utils.py | 27 +- tensorrt_llm/llmapi/llm.py | 18 + tensorrt_llm/llmapi/llm_args.py | 129 +- tensorrt_llm/llmapi/llm_utils.py | 127 +- tensorrt_llm/llmapi/reasoning_parser.py | 229 + tensorrt_llm/llmapi/rlhf_utils.py | 12 +- tensorrt_llm/llmapi/serialization.py | 224 + tensorrt_llm/lora_manager.py | 202 +- tensorrt_llm/mapping.py | 11 + tensorrt_llm/models/quant_config_utils.py | 54 +- .../runtime/kv_cache_manager_v2/AGENTS.md | 2 +- .../runtime/kv_cache_manager_v2/__init__.py | 13 +- .../runtime/kv_cache_manager_v2/__init__.pyi | 10 +- .../kv_cache_manager_v2/_block_radix_tree.py | 187 +- .../runtime/kv_cache_manager_v2/_cache_key.py | 134 - .../runtime/kv_cache_manager_v2/_config.py | 10 + .../kv_cache_manager_v2/_core/_kv_cache.py | 34 +- .../_core/_kv_cache_manager.py | 70 +- .../kv_cache_manager_v2/_event_manager.py | 12 +- .../kv_cache_manager_v2/_exceptions.py | 3 + .../kv_cache_manager_v2/_introspection.py | 111 + .../runtime/kv_cache_manager_v2/_page.py | 3 - tensorrt_llm/sampling_params.py | 57 +- tensorrt_llm/serialization.py | 268 +- tensorrt_llm/serve/openai_protocol.py | 14 +- tensorrt_llm/serve/openai_server.py | 86 +- tensorrt_llm/serve/openai_video_routes.py | 9 +- tensorrt_llm/serve/responses_utils.py | 31 +- .../serve/tool_parser/kimi_k3_tool_parser.py | 201 + .../serve/tool_parser/tool_parser_factory.py | 3 + tensorrt_llm/serve/visual_gen_utils.py | 10 +- .../tokenizer/deepseek_v32/tokenizer.py | 6 + .../tokenizer/deepseek_v4/tokenizer.py | 6 + .../tools/layer_wise_benchmarks/runner.py | 1 - .../usage/llm_args_golden_manifest.json | 91 +- tensorrt_llm/version.py | 2 +- tensorrt_llm/visual_gen/__init__.py | 102 +- tests/integration/defs/.test_durations | 3118 ++-- .../defs/accuracy/references/gsm8k.yaml | 19 +- .../defs/accuracy/references/mmlu.yaml | 3 + .../accuracy/test_disaggregated_serving.py | 4 +- .../defs/accuracy/test_llm_api_pytorch.py | 141 +- .../test_llm_api_pytorch_multimodal.py | 1 + tests/integration/defs/conftest.py | 12 +- ...deepseek_v3_lite_attention_dp_overlap.yaml | 14 +- .../disagg_config_mamba_bs1_concurrency2.yaml | 36 + ...lap_transceiver_runtime_python_bounce.yaml | 2 +- .../defs/disaggregated/test_disaggregated.py | 95 +- .../defs/examples/visual_gen/conftest.py | 29 +- ...smos3_nano_fp8_blockwise_lpips_golden.json | 27 + .../cosmos3_nano_nvfp4_lpips_golden.json | 27 + .../fastwan_lpips_golden_video.json | 16 + .../flux1_cuda_graph_lpips_golden.json | 25 + .../flux1_fp8_blockwise_lpips_golden.json | 25 + .../flux1_nvfp4_lpips_golden.json | 25 + .../flux2_cuda_graph_lpips_golden.json | 25 + .../flux2_fp8_blockwise_lpips_golden.json | 25 + .../flux2_nvfp4_lpips_golden.json | 25 + .../ltx2_cuda_graph_lpips_golden_video.json | 28 + ...ltx2_fp8_blockwise_lpips_golden_video.json | 28 + .../ltx2_nvfp4_lpips_golden_video.json | 28 + .../qwenimage_cuda_graph_lpips_golden.json | 26 + .../qwenimage_fp8_blockwise_lpips_golden.json | 26 + .../qwenimage_nvfp4_lpips_golden.json | 26 + .../visual_gen_lpips_golden_media.zip | 4 +- .../wan21_cuda_graph_lpips_golden_video.json | 28 + ...an21_fp8_blockwise_lpips_golden_video.json | 28 + .../wan21_nvfp4_lpips_golden_video.json | 28 + .../wan22_cuda_graph_lpips_golden_video.json | 28 + ...an22_fp8_blockwise_lpips_golden_video.json | 28 + .../wan22_nvfp4_lpips_golden_video.json | 28 + .../examples/visual_gen/test_visual_gen.py | 2452 --- .../visual_gen/test_visual_gen_cosmos3.py | 638 + .../visual_gen/test_visual_gen_flux.py | 368 + .../visual_gen/test_visual_gen_ltx2.py | 622 + .../visual_gen/test_visual_gen_multi_gpu.py | 2 +- .../visual_gen/test_visual_gen_qwen_image.py | 509 + .../visual_gen/test_visual_gen_wan.py | 721 + .../visual_gen/visual_gen_test_utils.py | 1000 ++ .../integration/defs/kimi_k3_disagg_parity.py | 792 + tests/integration/defs/kimi_k3_sa_harness.py | 558 + ...t_final_single_token_context_cuda_graph.py | 4 +- .../kv_cache/test_prefix_aware_scheduling.py | 5 + .../defs/llmapi/test_llm_api_pytorch_bart.py | 156 + .../defs/llmapi/test_llm_api_pytorch_t5.py | 96 +- tests/integration/defs/perf/_model_paths.py | 13 - .../defs/perf/pytorch_model_config.py | 156 +- tests/integration/defs/perf/test_perf.py | 5 - .../integration/defs/perf/test_perf_sanity.py | 166 +- tests/integration/defs/test_e2e.py | 16 +- .../integration/defs/test_kimi_k3_specdec.py | 95 + .../test_lists/qa/llm_function_core.txt | 15 +- .../test_lists/qa/llm_function_rtx6k.txt | 8 +- .../test_lists/qa/llm_perf_core.yml | 168 +- .../test_lists/qa/llm_perf_disagg.yml | 14 - .../test_lists/qa/llm_perf_multinode.txt | 13 - .../integration/test_lists/test-db/l0_a10.yml | 23 +- .../test_lists/test-db/l0_b200.yml | 92 +- .../integration/test_lists/test-db/l0_cpu.yml | 23 +- .../test_lists/test-db/l0_dgx_b200.yml | 7 +- .../test_lists/test-db/l0_dgx_h100.yml | 5 +- .../test_lists/test-db/l0_gb10.yml | 2 + .../test-db/l0_gb200_multi_gpus.yml | 7 - .../test-db/l0_gb300_multi_gpus.yml | 16 + .../test_lists/test-db/l0_gh200.yml | 4 +- .../test_lists/test-db/l0_h100.yml | 22 +- .../test_lists/test-db/l0_l40s.yml | 6 +- .../test_lists/test-db/l0_rtx_pro_6000.yml | 4 +- tests/integration/test_lists/waives.txt | 143 +- tests/microbenchmarks/bench_moe/worker.py | 2 +- .../cache_transceiver_precheck/README.md | 8 +- .../precheck_config.py | 17 +- .../run_precheck.py | 26 +- ...x1_dep4_gen1_tep8_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ...x1_dep4_gen1_dep8_eplb0_mtp1_ccb-NIXL.yaml | 4 +- ...x1_dep4_gen1_dep8_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ...x1_dep4_gen1_dep8_eplb0_mtp1_ccb-NIXL.yaml | 4 +- ...x1_dep4_gen1_tep8_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ...x1_dep4_gen1_dep8_eplb0_mtp1_ccb-NIXL.yaml | 4 +- ...x1_pp8_gen1_dep16_eplb0_mtp1_ccb-NIXL.yaml | 6 +- ...tx1_pp8_gen1_tep8_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ...1_dep4_gen1_dep32_eplb0_mtp0_ccb-NIXL.yaml | 3 +- ...1_dep4_gen1_dep32_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ...x1_dep4_gen1_dep8_eplb0_mtp0_ccb-NIXL.yaml | 3 +- ...x1_dep4_gen1_tep8_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ...2_dep4_gen1_dep16_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ...dep4_gen1_dep16_eplb288_mtp3_ccb-NIXL.yaml | 4 +- ...x1_dep4_gen1_dep4_eplb0_mtp1_ccb-NIXL.yaml | 4 +- ...1_dep4_gen1_dep32_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ...x1_dep4_gen1_tep8_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ...1_dep4_gen1_dep16_eplb0_mtp1_ccb-NIXL.yaml | 4 +- ...dep4_gen1_dep32_eplb256_mtp3_ccb-NIXL.yaml | 4 +- ...x1_dep4_gen1_tep8_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ...x1_dep4_gen1_dep4_eplb0_mtp1_ccb-NIXL.yaml | 4 +- ...x1_dep4_gen1_tep8_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ...dep4_gen1_dep32_eplb288_mtp1_ccb-NIXL.yaml | 4 +- ...1_dep4_gen1_dep32_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ...x1_dep8_gen1_dep8_eplb0_mtp0_ccb-NIXL.yaml | 3 +- ...dep4_gen1_dep32_eplb256_mtp3_ccb-NIXL.yaml | 4 +- ...x1_dep4_gen1_tep8_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ...dep4_gen1_dep32_eplb256_mtp0_ccb-NIXL.yaml | 4 +- ...ctx1_tp1_gen1_tp1_eplb0_mtp0_ccb-NIXL.yaml | 2 +- ...x1_dep4_gen1_tep4_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ..._dep4_gen1_dep8_eplb256_mtp1_ccb-NIXL.yaml | 4 +- ...1_dep4_gen1_dep32_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ..._dep4_gen1_dep8_eplb256_mtp1_ccb-NIXL.yaml | 4 +- ...x1_dep4_gen1_tep8_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ...1_dep4_gen1_dep32_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ...tx1_tp1_gen1_dep2_eplb0_mtp0_ccb-NIXL.yaml | 4 +- ...tx1_tp1_gen1_dep2_eplb0_mtp0_ccb-NIXL.yaml | 4 +- ...ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-NIXL.yaml | 4 +- ...ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-NIXL.yaml | 2 +- ...ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-NIXL.yaml | 4 +- ...ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-NIXL.yaml | 4 +- ...tx1_tp1_gen1_dep2_eplb0_mtp0_ccb-NIXL.yaml | 4 +- ...1_dep4_gen1_dep32_eplb0_mtp0_ccb-NIXL.yaml | 4 +- ...x1_dep4_gen1_dep8_eplb0_mtp0_ccb-NIXL.yaml | 4 +- ...x1_dep4_gen1_tep4_eplb0_mtp0_ccb-NIXL.yaml | 4 +- ...dep4_gen1_dep32_eplb416_mtp3_ccb-NIXL.yaml | 4 +- ...1_dep4_gen1_dep16_eplb0_mtp0_ccb-NIXL.yaml | 4 +- ...x1_dep4_gen1_tep8_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ...ctx1_tp1_gen1_tp1_eplb0_mtp0_ccb-NIXL.yaml | 2 +- ...tx1_tp1_gen1_dep8_eplb0_mtp0_ccb-NIXL.yaml | 4 +- ...tx1_tp1_gen1_tep4_eplb0_mtp0_ccb-NIXL.yaml | 4 +- ...tx1_tp1_gen1_tep4_eplb0_mtp0_ccb-NIXL.yaml | 4 +- ...tx1_pp4_gen1_tep8_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ...tx1_pp4_gen1_dep8_eplb0_mtp1_ccb-NIXL.yaml | 4 +- ...x1_pp4_gen1_dep16_eplb0_mtp1_ccb-NIXL.yaml | 4 +- ...1_dep4_gen1_dep32_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ...x1_dep4_gen1_tep8_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ...x1_dep4_gen1_dep4_eplb0_mtp1_ccb-NIXL.yaml | 4 +- ...1_dep4_gen1_dep32_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ...x1_dep4_gen1_tep8_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ...1_dep4_gen1_dep16_eplb0_mtp1_ccb-NIXL.yaml | 4 +- ...dep4_gen1_dep32_eplb256_mtp3_ccb-NIXL.yaml | 4 +- ...x1_dep4_gen1_tep8_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ...x1_dep4_gen1_dep4_eplb0_mtp1_ccb-NIXL.yaml | 4 +- ...x1_dep4_gen1_tep8_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ...dep4_gen1_dep32_eplb288_mtp1_ccb-NIXL.yaml | 4 +- ...1_dep4_gen1_dep32_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ...x1_dep8_gen1_dep8_eplb0_mtp0_ccb-NIXL.yaml | 3 +- ...dep4_gen1_dep32_eplb256_mtp3_ccb-NIXL.yaml | 4 +- ...x1_dep4_gen1_tep8_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ...dep4_gen1_dep32_eplb256_mtp0_ccb-NIXL.yaml | 4 +- ...dep4_gen1_dep32_eplb384_mtp3_ccb-NIXL.yaml | 2 +- ..._dep4_gen1_dep8_eplb384_mtp1_ccb-NIXL.yaml | 2 +- ...dep4_gen1_dep16_eplb384_mtp3_ccb-NIXL.yaml | 2 +- ...x1_dep4_gen4_tep8_eplb0_mtp3_ccb-NIXL.yaml | 2 +- ...x1_dep2_gen1_tep4_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ..._dep2_gen1_dep8_eplb256_mtp1_ccb-NIXL.yaml | 4 +- ...1_dep2_gen1_dep32_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ..._dep2_gen1_dep8_eplb256_mtp1_ccb-NIXL.yaml | 4 +- ...x1_dep2_gen1_tep8_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ...1_dep2_gen1_dep32_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ...1_dep4_gen1_dep32_eplb0_mtp0_ccb-NIXL.yaml | 4 +- ...x1_dep4_gen1_dep8_eplb0_mtp0_ccb-NIXL.yaml | 4 +- ...x1_dep4_gen1_tep4_eplb0_mtp0_ccb-NIXL.yaml | 4 +- ...dep4_gen1_dep32_eplb416_mtp3_ccb-NIXL.yaml | 4 +- ...1_dep4_gen1_dep16_eplb0_mtp0_ccb-NIXL.yaml | 4 +- ...x1_dep4_gen1_tep8_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ..._ctx1_tp2_gen1_tp2_eplb0_mtp0_ccb-UCX.yaml | 2 +- ..._tp2_gen1_tep4_eplb0_mtp0_ccb-DEFAULT.yaml | 2 +- ...1_tp1_gen1_tp2_eplb0_mtp0_ccb-DEFAULT.yaml | 2 +- ...x1_pp8_gen1_dep16_eplb0_mtp1_ccb-NIXL.yaml | 4 +- ...tx1_pp8_gen1_dep16_eplb0_mtp1_ccb-UCX.yaml | 104 - ...tx1_pp8_gen1_tep8_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ...x1_pp8_gen1_dep32_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ...pp4_gen8_pp4_bs2_eplb0_mtp0_con2-NIXL.yaml | 3 +- ...1_dep4_gen1_dep32_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ...x1_dep4_gen1_tep8_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ...x1_dep4_gen1_dep4_eplb0_mtp1_ccb-NIXL.yaml | 4 +- ...1_dep4_gen1_dep32_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ...x1_dep4_gen1_tep8_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ...1_dep4_gen1_dep16_eplb0_mtp1_ccb-NIXL.yaml | 2 +- ...x1_dep4_gen1_dep16_eplb0_mtp1_ccb-UCX.yaml | 105 - ...n3_tep8_bs32_eplb0_mtp0_con1_ccb-NIXL.yaml | 3 +- ...x1_dep4_gen1_tep8_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ...dep4_gen1_dep32_eplb288_mtp1_ccb-NIXL.yaml | 4 +- ..._dep4_gen1_dep32_eplb288_mtp1_ccb-UCX.yaml | 122 - ...1_dep4_gen1_dep32_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ...x1_dep8_gen1_dep8_eplb0_mtp0_ccb-NIXL.yaml | 3 +- ...dep4_gen1_dep32_eplb256_mtp3_ccb-NIXL.yaml | 4 +- ...x1_dep4_gen1_tep8_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ...dep4_gen1_dep32_eplb256_mtp0_ccb-NIXL.yaml | 4 +- ..._dep4_gen1_dep32_eplb256_mtp0_ccb-UCX.yaml | 112 - ..._dep4_gen1_dep8_eplb256_mtp1_ccb-NIXL.yaml | 4 +- ...1_dep4_gen1_dep8_eplb256_mtp1_ccb-UCX.yaml | 108 - ...x1_dep4_gen1_tep8_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ...1_dep4_gen1_dep32_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ..._ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-UCX.yaml | 104 - ...ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-NIXL.yaml | 4 +- ...ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-NIXL.yaml | 4 +- ...tx1_tp1_gen1_dep2_eplb0_mtp0_ccb-NIXL.yaml | 4 +- ...1_dep4_gen1_dep32_eplb0_mtp0_ccb-NIXL.yaml | 4 +- ...x1_dep4_gen1_dep8_eplb0_mtp0_ccb-NIXL.yaml | 4 +- ...x1_dep4_gen1_tep4_eplb0_mtp0_ccb-NIXL.yaml | 4 +- ...dep4_gen1_dep32_eplb416_mtp3_ccb-NIXL.yaml | 4 +- ...1_dep4_gen1_dep16_eplb0_mtp0_ccb-NIXL.yaml | 4 +- ...x1_dep4_gen1_dep16_eplb0_mtp0_ccb-UCX.yaml | 103 - ...x1_dep4_gen1_tep8_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ...tx1_tp1_gen1_dep8_eplb0_mtp0_ccb-NIXL.yaml | 4 +- ...ctx1_tp1_gen1_dep8_eplb0_mtp0_ccb-UCX.yaml | 100 - ...tx1_tp1_gen1_tep4_eplb0_mtp0_ccb-NIXL.yaml | 4 +- ...tx1_tp1_gen1_tep4_eplb0_mtp0_ccb-NIXL.yaml | 4 +- ...x1_tp1_gen1_tp4_eplb0_eagle3_ccb-NIXL.yaml | 4 +- ...2_bs128_eplb288_mtp3_con1024_ccb-NIXL.yaml | 3 +- ...2_bs128_eplb288_mtp3_con1024_ccb-NIXL.yaml | 3 +- ...en1_dep32_bs128_eplb288_mtp3_ccb-NIXL.yaml | 3 +- ...tx1_pp4_gen1_tep8_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ...tx1_pp4_gen1_dep8_eplb0_mtp1_ccb-NIXL.yaml | 4 +- ...ctx1_pp4_gen1_dep8_eplb0_mtp1_ccb-UCX.yaml | 107 - ...x1_pp4_gen1_dep16_eplb0_mtp1_ccb-NIXL.yaml | 4 +- ...1_dep4_gen1_dep32_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ...x1_dep4_gen1_tep8_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ...1_dep4_gen1_dep16_eplb0_mtp1_ccb-NIXL.yaml | 4 +- ...x1_dep4_gen1_dep16_eplb0_mtp1_ccb-UCX.yaml | 105 - ...x1_dep4_gen1_tep8_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ...dep4_gen1_dep32_eplb288_mtp1_ccb-NIXL.yaml | 4 +- ..._dep4_gen1_dep32_eplb288_mtp1_ccb-UCX.yaml | 122 - ...1_dep4_gen1_dep32_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ...x1_dep8_gen1_dep8_eplb0_mtp0_ccb-NIXL.yaml | 3 +- ...dep4_gen1_dep32_eplb256_mtp3_ccb-NIXL.yaml | 4 +- ...x1_dep4_gen1_tep8_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ...dep4_gen1_dep32_eplb256_mtp0_ccb-NIXL.yaml | 2 +- ..._dep4_gen1_dep32_eplb256_mtp0_ccb-UCX.yaml | 112 - ..._dep4_gen1_dep8_eplb384_mtp3_ccb-NIXL.yaml | 2 +- ...x1_dep4_gen5_tep4_eplb0_mtp3_ccb-NIXL.yaml | 2 +- ...dep4_gen1_dep32_eplb384_mtp3_ccb-NIXL.yaml | 2 +- ...x1_dep2_gen1_tep4_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ..._dep2_gen1_dep8_eplb256_mtp1_ccb-NIXL.yaml | 4 +- ...1_dep2_gen1_dep32_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ..._dep2_gen1_dep8_eplb256_mtp1_ccb-NIXL.yaml | 4 +- ...1_dep2_gen1_dep8_eplb256_mtp1_ccb-UCX.yaml | 108 - ...x1_dep2_gen1_tep8_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ...1_dep2_gen1_dep32_eplb0_mtp3_ccb-NIXL.yaml | 2 +- ...dep4_gen1_dep32_eplb416_mtp3_ccb-NIXL.yaml | 4 +- ...1_dep4_gen1_dep16_eplb0_mtp0_ccb-NIXL.yaml | 2 +- ...x1_dep4_gen1_dep16_eplb0_mtp0_ccb-UCX.yaml | 103 - ...x1_dep4_gen1_tep8_eplb0_mtp3_ccb-NIXL.yaml | 4 +- ...2_bs128_eplb288_mtp3_con1024_ccb-NIXL.yaml | 3 +- ...2_bs128_eplb288_mtp3_con1024_ccb-NIXL.yaml | 3 +- .../deepseek_v4/test_compressor_module.py | 10 +- .../test_deepseek_v4_cache_manager.py | 2 +- .../test_deepseek_v4_fp4_indexer.py | 6 +- .../test_deepseek_v4_indices_transform.py | 6 +- .../deepseek_v4/test_deepseek_v4_o_proj.py | 9 +- .../test_deepseek_v4_sparse_mla.py | 18 +- .../attention/sparse/dsa/test_dsa_indexer.py | 313 +- .../sparse/dsa/test_dsa_sparse_mla.py | 13 +- .../test_kernels.py} | 5 +- .../sparse/dsa/test_req_idx_per_token.py | 103 + .../sparse/dsa/test_short_seq_mha.py | 47 +- .../test_kernels.py} | 5 +- .../attention/sparse/test_cpp_custom_ops.py | 1 + .../sparse/test_minimax_m3_msa_backend.py | 2 +- .../attention/sparse/test_sparse_attention.py | 124 +- .../sparse/test_sparse_mla_forward.py | 98 +- .../attention/sparse/test_triton_topk.py | 228 - .../_torch/attention/test_attention_mla.py | 3 +- .../attention/test_attention_op_sync.py | 8 +- .../_torch/attention/test_fmha_page_index.py | 73 +- .../singlegpu/models/test_gpt_oss_modeling.py | 1 + .../compilation/test_remove_copy_pass.py | 58 +- .../custom_ops/test_deepseek_v4_q_norm.py | 32 +- .../cute_dsl/test_kimi_k3_kda_ptx_patch.py | 174 + .../_torch/executor/test_benchmark_disagg.py | 7 +- .../executor/test_hang_detector_kill.py | 916 +- .../executor/test_kv_cache_estimation.py | 2 +- .../executor/test_kv_cache_manager_v2.py | 25 +- .../test_kv_cache_v2_extra_buffers.py | 30 +- .../executor/test_mamba_cache_manager.py | 734 +- .../_torch/executor/test_model_loader_mx.py | 4 +- .../_torch/executor/test_py_executor.py | 410 +- .../executor/test_pytorch_model_engine.py | 328 +- .../test_pytorch_model_engine_warmup.py | 46 + .../_torch/executor/test_resource_manager.py | 52 + .../_torch/executor/test_seq_slot_sizing.py | 45 +- .../test_fp8_lora_grouped_gemm_regressions.py | 298 + .../_torch/lora/test_moe_lora_validator.py | 3 +- .../_torch/memory/test_gms_backend.py | 2 + .../misc/test_moe_a2a_warmup_timeout.py | 78 + .../_torch/modeling/test_gemma4_multimodal.py | 211 +- .../test_kda_mtp_decode_cute_parity.py | 540 + .../test_kimi_k3_fp8_weight_read_gates.py | 82 + .../test_kimi_kda_fused_verify_parity.py | 237 + .../modeling/test_kimi_kda_verify_parity.py | 105 + .../_torch/modeling/test_modeling_afmoe.py | 5 + .../modeling/test_modeling_deepseekv4.py | 52 +- .../modeling/test_modeling_exaone4_5.py | 31 + .../_torch/modeling/test_modeling_gpt_oss.py | 60 +- .../modeling/test_modeling_llava_next.py | 1 + .../test_modeling_nemotron_h_moe_quant.py | 145 + .../test_modeling_nemotron_nano_v2_vl.py | 12 + .../_torch/modeling/test_modeling_parakeet.py | 1 + .../modeling/test_modeling_qwen3_5_vl_moe.py | 44 +- .../_torch/modeling/test_modeling_qwen3vl.py | 1 + .../test_modeling_qwen3vl_preprocess.py | 4 + .../_torch/modeling/test_modeling_step3p7.py | 4 + .../test_nemotron_nano_preprocessing.py | 2 + .../checkpoints/hf/test_weight_loader.py | 2 + .../models/checkpoints/laguna/__init__.py | 2 + .../laguna/test_laguna_weight_mapper.py | 2 + .../models/checkpoints/mistral/__init__.py | 2 + .../checkpoints/mistral/test_config_loader.py | 4 + .../mistral/test_mistral_weight_mapper.py | 2 + .../mx/test_mx_checkpoint_loader.py | 3 + .../_torch/models/test_nemotron_h_puzzle.py | 2 + .../models/test_qwen3_next_moe_quant.py | 3 +- .../fused_moe/test_configurable_moe.py | 122 + .../fused_ops/test_fused_qk_norm_rope_gate.py | 76 +- .../kimi_k3_attn_res/test_attn_res_op.py | 86 + .../kimi_kda/test_kda_cache_soundness.py | 382 + .../modules/kimi_kda/test_kda_decode_op.py | 401 + .../modules/kimi_kda/test_kda_prefill_op.py | 323 + .../kimi_kda/test_kda_prefill_state_parity.py | 397 + .../mamba/test_gdn_kernel_optimizations.py | 38 +- .../mamba/test_gdn_replay_recurrent.py | 62 +- .../moe/kimi_k3_ref_moe/_moe_kernels.py | 411 + .../modules/moe/kimi_k3_ref_moe/_mxfp4.py | 163 + .../moe/kimi_k3_ref_moe/kimi_k3_moe_block.py | 688 + .../_torch/modules/moe/quantize_utils.py | 58 +- .../moe/test_cute_dsl_b12x_moe_backend.py | 162 +- .../modules/moe/test_cutlass_moe_op_smoke.py | 51 + .../_torch/modules/moe/test_kimi_k3_mlp.py | 117 + .../modules/moe/test_kimi_k3_moe_gate.py | 127 + .../modules/moe/test_kimi_k3_situ_and_mul.py | 143 + .../modules/moe/test_kimi_k3_situ_moe.py | 689 + .../_torch/modules/moe/test_moe_backend.py | 39 +- .../_torch/modules/moe/test_moe_comm.py | 6 + .../unittest/_torch/modules/test_mla_helix.py | 2 +- .../_torch/modules/test_w4a16_nvfp4_linear.py | 593 + .../tests_lora_modules/test_qwen3_sanity.py | 6 +- .../multimodal/test_external_embedding.py | 2 + .../multimodal/test_multimodal_mixin.py | 13 +- .../multimodal/test_multimodal_runtime.py | 2 + .../multimodal/test_qwen3vl_disagg_prompt.py | 3 + .../multimodal/test_share_multiparams.py | 2 + .../test_llm_update_weights_multi_gpu.py | 23 +- .../_torch/sampler/test_torch_sampler.py | 222 +- .../hw_agnostic/test_dflash_accept_stats.py | 271 + .../test_kimi_k3_dflash_scaffold.py | 344 + .../test_kimi_k3_dspark_semantics.py | 494 + .../_torch/speculative/hw_agnostic/test_sa.py | 277 + .../_torch/speculative/test_eagle3.py | 128 +- tests/unittest/_torch/test_connector.py | 2 + .../_torch/test_custom_config_registration.py | 3 + tests/unittest/_torch/test_hf_quant_config.py | 4 + tests/unittest/_torch/test_mnnvl_utils.py | 244 + tests/unittest/_torch/test_model_config.py | 2 + .../parallel/test_dense_gemm_act_fusion.py | 9 +- .../test_finegrained_mixed_dtype_gemm.py | 1 + .../_torch/thop/parallel/test_fp4_linear.py | 30 +- .../_torch/thop/parallel/test_kda_decode.py | 457 + .../parallel_hw_agnostic/test_custom_ops.py | 17 + .../test_fused_qk_norm_rope.py | 117 +- tests/unittest/_torch/thop/serial/test_moe.py | 207 +- .../thop/serial/test_nvfp4_marlin_gemm.py | 7 +- .../multi_gpu/test_parallel_conv.py | 287 +- .../sparse_attention/test_skip_softmax.py | 2 +- .../visual_gen/test_fastwan_dmd_math.py | 201 + .../visual_gen/test_fastwan_pipeline.py | 136 + .../visual_gen/test_trtllm_serve_endpoints.py | 11 +- .../visual_gen/test_visual_gen_params.py | 28 + .../visual_gen/test_wan22_t2v_pipeline.py | 6 +- .../_torch/visual_gen/test_wan_dup_up3d.py | 140 + .../_torch/visual_gen/test_wan_vae_halo.py | 68 + .../api_stability/references/llm.yaml | 12 + .../references/trtllm_serve_cli.yaml | 13 +- .../custom_ops/test_ad_dist_strategies.py | 2 + .../library/test_tp_sharding.py | 2 + .../singlegpu/compile/test_captured_graph.py | 22 + .../compile/test_piecewise_runner.py | 6 + .../singlegpu/compile/test_piecewise_utils.py | 4 + .../attention/test_torch_attention_op.py | 3 + .../custom_ops/mla/test_torch_mla_op.py | 1 + .../custom_ops/mla/test_triton_mla_op.py | 1 + .../custom_ops/moe/test_mxfp4_moe_layout.py | 3 + .../custom_ops/quantization/test_quant.py | 18 +- .../custom_ops/rope/test_rope_op_variants.py | 1 + .../singlegpu/custom_ops/test_gptq_op.py | 2 + .../custom_ops/test_resource_handlers.py | 3 +- .../custom_ops/test_update_kv_cache.py | 3 + .../singlegpu/mlir/test_decompose.py | 3 + .../singlegpu/mlir/test_dialect.py | 3 + .../mlir/test_elementwise_fusion_e2e.py | 7 + .../singlegpu/mlir/test_fx_mlir_roundtrip.py | 2 + .../singlegpu/mlir/test_string_codegen.py | 1 + .../singlegpu/mlir/test_subgraph_discovery.py | 3 + .../singlegpu/models/test_cohere_modeling.py | 7 + .../singlegpu/models/test_decilm_modeling.py | 3 + .../models/test_deepseek_v2_modeling.py | 1 + .../singlegpu/models/test_eagle.py | 1 + .../singlegpu/models/test_exaone_modeling.py | 4 + .../singlegpu/models/test_gemma2_modeling.py | 5 + .../singlegpu/models/test_gemma3n_modeling.py | 10 + .../singlegpu/models/test_gemma4_modeling.py | 8 + .../singlegpu/models/test_gemma_modeling.py | 5 + .../models/test_glm4_moe_lite_modeling.py | 3 + .../models/test_glm4_moe_modeling.py | 4 + .../models/test_glm_moe_dsa_modeling.py | 4 + .../singlegpu/models/test_granite_modeling.py | 4 + .../auto_deploy/singlegpu/models/test_hf.py | 6 + .../models/test_hunyuan_moe_modeling.py | 5 + .../models/test_internlm3_modeling.py | 3 + .../singlegpu/models/test_kimi_k2_modeling.py | 5 + .../singlegpu/models/test_llama3_modeling.py | 3 + .../singlegpu/models/test_llama4_modeling.py | 5 + .../models/test_minimax_m2_modeling.py | 4 +- .../models/test_mistral3_modeling.py | 5 + .../singlegpu/models/test_mistral_modeling.py | 3 + .../singlegpu/models/test_olmo3_modeling.py | 5 + .../singlegpu/models/test_openelm_modeling.py | 5 + .../singlegpu/models/test_qwen3_5_moe.py | 2 + .../models/test_qwen3_moe_modeling.py | 4 + .../models/test_qwen3_next_modeling.py | 3 + .../models/test_seed_oss_modeling.py | 3 + .../models/test_skywork_r1v2_modeling.py | 3 + .../singlegpu/models/test_smollm3_modeling.py | 4 + .../models/test_starcoder2_modeling.py | 3 + .../singlegpu/models/test_step3p7_modeling.py | 1 + .../singlegpu/shim/test_create_ad_executor.py | 128 +- .../singlegpu/shim/test_llm_config.py | 17 +- .../smoke/test_ad_speculative_decoding.py | 2 +- .../singlegpu/test_graph_canonicalize.py | 2 + .../singlegpu/test_hf_export_info.py | 3 + .../test_mistral_small_4_tokenizer_bridge.py | 3 + .../singlegpu/test_pattern_matcher.py | 2 + .../library/test_bmm_moe_hooks.py | 2 + .../library/test_fuse_silu_mul.py | 4 + .../library/test_fuse_swiglu.py | 3 + .../library/test_gemm_fusion.py | 1 + .../library/test_kvcache_vswa_metadata.py | 2 + .../library/test_mrope_delta_cache.py | 3 + .../library/test_quant_fusion.py | 12 + .../library/test_quantization.py | 1 + .../library/test_shared_kv_attention.py | 12 + .../singlegpu/transformations/test_export.py | 2 + .../transformations/test_pipeline_cache.py | 3 + .../singlegpu/utils/test_config.py | 2 + .../utils/test_create_derived_custom_op.py | 4 + .../singlegpu/utils/test_dist_config.py | 3 + .../singlegpu/utils/test_example_configs.py | 3 + .../utils/test_multi_stream_utils.py | 1 + .../utils/test_node_utils_sharding.py | 3 + .../utils/test_quantization_utils.py | 3 + .../standalone/test_standalone_import.py | 3 + tests/unittest/bindings/test_bindings_ut.py | 2 + .../bindings/test_executor_bindings.py | 47 + .../bindings/test_transfer_agent_bindings.py | 58 + .../unittest/disaggregated/region/test_aux.py | 61 +- .../disaggregated/region/test_block.py | 3 + .../disaggregated/region/test_page.py | 3 + .../disaggregated/region/test_region.py | 2 + tests/unittest/disaggregated/test_agent.py | 1 + .../test_agent_multi_backends.py | 3 + .../test_aux_buffer_registration.py | 195 + tests/unittest/disaggregated/test_bounce.py | 234 +- .../disaggregated/test_cache_reuse_adapter.py | 94 + .../disaggregated/test_cluster_storage.py | 2 + .../test_deepseek_v4_kv_transfer.py | 2 +- .../test_disagg_cluster_manager_worker.py | 2 + .../test_disagg_openai_client.py | 2 + .../disaggregated/test_disagg_utils.py | 2 + .../test_disaggregated_params.py | 2 + .../unittest/disaggregated/test_extractor.py | 3 + .../disaggregated/test_kda_mamba_transfer.py | 589 + .../disaggregated/test_kv_transfer.py | 16 +- .../unittest/disaggregated/test_messenger.py | 3 + .../test_openai_disagg_server.py | 2 + .../test_openai_disagg_service.py | 2 + tests/unittest/disaggregated/test_peer.py | 120 +- .../disaggregated/test_perf_logger.py | 4 + .../unittest/disaggregated/test_rank_info.py | 2 + .../disaggregated/test_remoteDictionary.py | 4 + .../unittest/disaggregated/test_request_id.py | 2 + tests/unittest/disaggregated/test_router.py | 2 + tests/unittest/executor/test_base_worker.py | 10 +- .../test_event_loop_error_broadcast.py | 75 +- .../executor/test_spec_dec_perf_metrics.py | 105 + .../executor/test_spec_dec_stats_pairing.py | 212 + .../inputs/test_chat_template_dispatch.py | 99 + .../test_kv_cache_event_manager.py | 446 +- .../test_kv_cache_manager_v2.py | 189 +- .../test_kv_cache_salting.py | 12 +- .../test_kv_cache_stats_behavior.py | 12 +- .../unittest/llmapi/apps/test_tool_parsers.py | 287 + tests/unittest/llmapi/test_config_database.py | 4 +- .../llmapi/test_disagg_telemetry_launcher.py | 2 + tests/unittest/llmapi/test_llm.py | 2 +- tests/unittest/llmapi/test_llm_args.py | 367 +- tests/unittest/llmapi/test_llm_utils.py | 48 + .../unittest/llmapi/test_reasoning_parser.py | 262 + tests/unittest/llmapi/test_responses_utils.py | 89 + tests/unittest/llmapi/test_sampling_params.py | 136 + tests/unittest/llmapi/test_serialization.py | 2 +- tests/unittest/metrics/test_collector.py | 2 + tests/unittest/models/test_modeling_utils.py | 4 + .../models/test_quant_config_utils.py | 29 + tests/unittest/others/test_bench.py | 3 + .../test_cache_transceiver_precheck_config.py | 5 +- .../test_cache_transceiver_precheck_run.py | 28 +- tests/unittest/others/test_convert_utils.py | 3 + tests/unittest/others/test_exception.py | 4 + tests/unittest/others/test_export.py | 3 + tests/unittest/others/test_lazy_model_zoo.py | 386 + tests/unittest/others/test_lm_eval.py | 94 +- tests/unittest/others/test_lora_manager.py | 415 +- .../unittest/others/test_lora_module_count.py | 4 + tests/unittest/others/test_mapping.py | 13 +- .../others/test_multimodal_registry.py | 4 + .../unittest/others/test_pretrained_config.py | 17 +- tests/unittest/others/test_time_breakdown.py | 4 + .../others/test_triton_kernels_vendoring.py | 4 + tests/unittest/quantization/test_mode.py | 4 + .../scripts/test_check_model_registry.py | 3 + tests/unittest/scripts/test_fetch_cache.py | 3 + tests/unittest/scripts/test_legacy_utils.py | 3 + .../scripts/test_perf_sanity_helpers.py | 82 + tests/unittest/scripts/test_perf_submit.py | 53 +- .../tools/test_config_database_sync.py | 3 + tests/unittest/tools/test_host_profiler.py | 16 +- .../tools/test_layer_wise_benchmarks.py | 5 + tests/unittest/tools/test_prepare_dataset.py | 2 + .../tools/test_test_to_stage_mapping.py | 2 + .../utils/test_confidential_compute.py | 172 + tests/unittest/utils/test_logger.py | 2 + .../utils/test_prebuilt_whl_cpp_extensions.py | 4 + 1594 files changed, 101325 insertions(+), 34028 deletions(-) create mode 100644 cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/AGENTS.md create mode 100644 cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/tokenIdExt.cpp create mode 100644 cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/tokenIdExt.h create mode 100644 cpp/tensorrt_llm/kernels/kdaDecode/CMakeLists.txt create mode 100644 cpp/tensorrt_llm/kernels/kdaDecode/kdaDecode.cu create mode 100644 cpp/tensorrt_llm/kernels/kdaDecode/kdaDecode.h create mode 100644 cpp/tensorrt_llm/kernels/kimiK3AttnRes/CMakeLists.txt create mode 100644 cpp/tensorrt_llm/kernels/kimiK3AttnRes/attnResFwd.cu create mode 100644 cpp/tensorrt_llm/kernels/kimiK3AttnRes/attnResFwd.h create mode 100644 cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst create mode 100644 cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7x7x7x7x1x7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_siTuGlu_lbW4_lsfbW4_dynB_sm100f.cubin.tar.zst create mode 100644 cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128u2_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst create mode 100644 cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst create mode 100644 cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4x4x4x4x1x4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_siTuGlu_lbW4_lsfbW4_dynB_sm100f.cubin.tar.zst create mode 100644 cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256u2_s4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst create mode 100644 cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst create mode 100644 cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst create mode 100644 cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst create mode 100644 cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst create mode 100644 cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst create mode 100644 cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst create mode 100644 cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst create mode 100644 cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst create mode 100644 cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x256x128_s6x6x6x6x1x6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_fCp_tmOv_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst create mode 100644 cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x256x256_s3x3x3x3x1x3_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_fCp_tmOv_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst create mode 100644 cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst create mode 100644 cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst create mode 100644 cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst create mode 100644 cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst create mode 100644 cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst create mode 100644 cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst create mode 100644 cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst create mode 100644 cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst create mode 100644 cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x128_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW1_lsfbW1_dynB_sm100f.cubin.tar.zst create mode 100644 cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x128u2_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW1_lsfbW1_dynB_sm100f.cubin.tar.zst create mode 100644 cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x256_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW1_lsfbW1_dynB_sm100f.cubin.tar.zst create mode 100644 cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x256u2_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW1_lsfbW1_dynB_sm100f.cubin.tar.zst create mode 100644 cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst create mode 100644 cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst create mode 100644 cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst create mode 100644 cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst create mode 100644 cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst create mode 100644 cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst create mode 100644 cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst create mode 100644 cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst create mode 100644 cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst create mode 100644 cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst create mode 100644 cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst create mode 100644 cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst create mode 100644 cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst create mode 100644 cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst create mode 100644 cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst create mode 100644 cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst create mode 100644 cpp/tensorrt_llm/thop/attnResOp.cpp create mode 100644 cpp/tensorrt_llm/thop/kdaDecodeOp.cpp create mode 100644 cpp/tests/unit_tests/batch_manager/kvCacheManagerV2DigestPoolTest.cpp create mode 100644 docs/source/deployment-guide/deployment-guide-for-kimi-k3-on-trtllm.md create mode 100644 examples/disaggregated/slurm/cache_transceiver_test/configs/kda_payload_kimi_k3.yaml create mode 100644 examples/kimi_k3/README.md create mode 100644 examples/kimi_k3/disagg/README.md create mode 100644 examples/kimi_k3/disagg/benchmark_kimi_k3_dep16.yaml create mode 100644 examples/kimi_k3/disagg/ctx_config.yaml create mode 100644 examples/kimi_k3/disagg/disagg_proxy_config.yaml create mode 100644 examples/kimi_k3/disagg/gen_config_no_sa.yaml create mode 100644 examples/kimi_k3/eval_extra_llm_options.yaml create mode 100644 examples/kimi_k3/eval_extra_llm_options_reuse.yaml create mode 100644 examples/kimi_k3/perf_sweep/acc_sweep.sbatch create mode 100644 examples/kimi_k3/perf_sweep/perf_sweep.sbatch create mode 100755 examples/kimi_k3/perf_sweep/submit_acc_sweep.sh create mode 100644 examples/kimi_k3/perf_sweep/submit_perf_sweep.sh create mode 100644 examples/kimi_k3/quick_start_kimi_k3.py create mode 100644 examples/kimi_k3/quick_start_kimi_k3.sbatch create mode 100644 examples/kimi_k3/run_gsm8k_kimi_k3.sbatch create mode 100644 examples/kimi_k3/run_serving_benchmark_kimi_k3.sbatch create mode 100644 jenkins/scripts/pulse_in_pipeline_scanning/submit_preapproved_candidates.py create mode 100644 jenkins/scripts/pulse_in_pipeline_scanning/utils/triage.py delete mode 100644 security_scanning/examples/serve/poetry.lock create mode 100644 tensorrt_llm/_bootstrap.py create mode 100644 tensorrt_llm/_torch/attention_backend/fmha/cute_dsl_mla.py create mode 100644 tensorrt_llm/_torch/attention_backend/sparse/deepseek_v4/backend.py create mode 100644 tensorrt_llm/_torch/attention_backend/sparse/deepseek_v4/indexer.py create mode 100644 tensorrt_llm/_torch/attention_backend/sparse/deepseek_v4/kernels.py rename tensorrt_llm/_torch/attention_backend/sparse/deepseek_v4/{deepseek_v4.py => metadata.py} (50%) create mode 100644 tensorrt_llm/_torch/attention_backend/sparse/deepseek_v4/module.py create mode 100644 tensorrt_llm/_torch/attention_backend/sparse/deepseek_v4/params.py delete mode 100644 tensorrt_llm/_torch/attention_backend/sparse/dsa.py create mode 100644 tensorrt_llm/_torch/attention_backend/sparse/dsa/__init__.py create mode 100644 tensorrt_llm/_torch/attention_backend/sparse/dsa/backend.py create mode 100644 tensorrt_llm/_torch/attention_backend/sparse/dsa/cache_manager.py create mode 100644 tensorrt_llm/_torch/attention_backend/sparse/dsa/custom_ops.py create mode 100644 tensorrt_llm/_torch/attention_backend/sparse/dsa/indexer.py create mode 100644 tensorrt_llm/_torch/attention_backend/sparse/dsa/kernels.py create mode 100644 tensorrt_llm/_torch/attention_backend/sparse/dsa/metadata.py create mode 100644 tensorrt_llm/_torch/attention_backend/sparse/dsa/module.py create mode 100644 tensorrt_llm/_torch/attention_backend/sparse/dsa/params.py create mode 100644 tensorrt_llm/_torch/attention_backend/sparse/hooks.py rename tensorrt_llm/_torch/attention_backend/sparse/{utils.py => registry.py} (79%) delete mode 100644 tensorrt_llm/_torch/attention_backend/sparse/rocket.py create mode 100644 tensorrt_llm/_torch/attention_backend/sparse/rocket/__init__.py create mode 100644 tensorrt_llm/_torch/attention_backend/sparse/rocket/backend.py create mode 100644 tensorrt_llm/_torch/attention_backend/sparse/rocket/cache_manager.py rename tensorrt_llm/_torch/attention_backend/sparse/{kernel.py => rocket/kernels.py} (51%) create mode 100644 tensorrt_llm/_torch/attention_backend/sparse/rocket/metadata.py create mode 100644 tensorrt_llm/_torch/attention_backend/sparse/rocket/module.py create mode 100644 tensorrt_llm/_torch/attention_backend/sparse/rocket/params.py create mode 100644 tensorrt_llm/_torch/attention_backend/sparse/skip_softmax/__init__.py rename tensorrt_llm/_torch/attention_backend/sparse/{skip_softmax.py => skip_softmax/params.py} (88%) create mode 100644 tensorrt_llm/_torch/configs/kimi_linear.py create mode 100644 tensorrt_llm/_torch/custom_ops/cute_dsl_kimi_k3_custom_ops.py create mode 100644 tensorrt_llm/_torch/custom_ops/cute_dsl_kimi_k3_kda_mtp_ops.py create mode 100644 tensorrt_llm/_torch/cute_dsl_kernels/blackwell/attention/__init__.py create mode 100644 tensorrt_llm/_torch/cute_dsl_kernels/blackwell/attention/mla/__init__.py create mode 100644 tensorrt_llm/_torch/cute_dsl_kernels/blackwell/attention/mla/mla_decode_fp16.py create mode 100644 tensorrt_llm/_torch/cute_dsl_kernels/blackwell/attention/mla/mla_decode_fp8.py create mode 100644 tensorrt_llm/_torch/cute_dsl_kernels/blackwell/attention/mla/mla_helpers.py create mode 100644 tensorrt_llm/_torch/cute_dsl_kernels/blackwell/kimi_k3_kda/__init__.py create mode 100644 tensorrt_llm/_torch/cute_dsl_kernels/blackwell/kimi_k3_kda/akk_inverse.py create mode 100644 tensorrt_llm/_torch/cute_dsl_kernels/blackwell/kimi_k3_kda/fused_k123.py create mode 100644 tensorrt_llm/_torch/cute_dsl_kernels/blackwell/kimi_k3_kda/fused_k1234.py create mode 100644 tensorrt_llm/_torch/cute_dsl_kernels/blackwell/kimi_k3_kda/k4_persistent.py create mode 100644 tensorrt_llm/_torch/cute_dsl_kernels/blackwell/kimi_k3_kda/kda_mtp_decode.py create mode 100644 tensorrt_llm/_torch/models/_arch_index.py create mode 100644 tensorrt_llm/_torch/models/modeling_kimi_linear.py create mode 100644 tensorrt_llm/_torch/modules/fused_moe/impl_base.py create mode 100644 tensorrt_llm/_torch/modules/fused_moe/impl_contract.py create mode 100644 tensorrt_llm/_torch/modules/fused_moe/impl_identity.py create mode 100644 tensorrt_llm/_torch/modules/kimi_k3_attn_res/__init__.py create mode 100644 tensorrt_llm/_torch/modules/kimi_k3_attn_res/_attn_res_kernels.py create mode 100644 tensorrt_llm/_torch/modules/kimi_k3_attn_res/kimi_k3_attn_res.py create mode 100644 tensorrt_llm/_torch/modules/kimi_k3_mla/__init__.py create mode 100644 tensorrt_llm/_torch/modules/kimi_k3_mla/kimi_k3_mla_attention.py create mode 100644 tensorrt_llm/_torch/modules/kimi_k3_moe/__init__.py create mode 100644 tensorrt_llm/_torch/modules/kimi_k3_moe/_mlp.py create mode 100644 tensorrt_llm/_torch/modules/kimi_k3_moe/kimi_k3_moe_gate.py create mode 100644 tensorrt_llm/_torch/modules/kimi_kda/__init__.py create mode 100644 tensorrt_llm/_torch/modules/kimi_kda/_kda_decode.py create mode 100644 tensorrt_llm/_torch/modules/kimi_kda/_kda_kernels.py create mode 100644 tensorrt_llm/_torch/modules/kimi_kda/kimi_kda_mixer.py create mode 100644 tensorrt_llm/_torch/speculative/accept_stats.py create mode 100644 tensorrt_llm/_torch/visual_gen/models/wan/dup_up3d.py create mode 100644 tensorrt_llm/_torch/visual_gen/models/wan/pipeline_fastwan.py create mode 100644 tensorrt_llm/llmapi/serialization.py delete mode 100644 tensorrt_llm/runtime/kv_cache_manager_v2/_cache_key.py create mode 100644 tensorrt_llm/serve/tool_parser/kimi_k3_tool_parser.py create mode 100644 tests/integration/defs/disaggregated/test_configs/disagg_config_mamba_bs1_concurrency2.yaml create mode 100644 tests/integration/defs/examples/visual_gen/golden/visual_gen_lpips/cosmos3_nano_fp8_blockwise_lpips_golden.json create mode 100644 tests/integration/defs/examples/visual_gen/golden/visual_gen_lpips/cosmos3_nano_nvfp4_lpips_golden.json create mode 100644 tests/integration/defs/examples/visual_gen/golden/visual_gen_lpips/fastwan_lpips_golden_video.json create mode 100644 tests/integration/defs/examples/visual_gen/golden/visual_gen_lpips/flux1_cuda_graph_lpips_golden.json create mode 100644 tests/integration/defs/examples/visual_gen/golden/visual_gen_lpips/flux1_fp8_blockwise_lpips_golden.json create mode 100644 tests/integration/defs/examples/visual_gen/golden/visual_gen_lpips/flux1_nvfp4_lpips_golden.json create mode 100644 tests/integration/defs/examples/visual_gen/golden/visual_gen_lpips/flux2_cuda_graph_lpips_golden.json create mode 100644 tests/integration/defs/examples/visual_gen/golden/visual_gen_lpips/flux2_fp8_blockwise_lpips_golden.json create mode 100644 tests/integration/defs/examples/visual_gen/golden/visual_gen_lpips/flux2_nvfp4_lpips_golden.json create mode 100644 tests/integration/defs/examples/visual_gen/golden/visual_gen_lpips/ltx2_cuda_graph_lpips_golden_video.json create mode 100644 tests/integration/defs/examples/visual_gen/golden/visual_gen_lpips/ltx2_fp8_blockwise_lpips_golden_video.json create mode 100644 tests/integration/defs/examples/visual_gen/golden/visual_gen_lpips/ltx2_nvfp4_lpips_golden_video.json create mode 100644 tests/integration/defs/examples/visual_gen/golden/visual_gen_lpips/qwenimage_cuda_graph_lpips_golden.json create mode 100644 tests/integration/defs/examples/visual_gen/golden/visual_gen_lpips/qwenimage_fp8_blockwise_lpips_golden.json create mode 100644 tests/integration/defs/examples/visual_gen/golden/visual_gen_lpips/qwenimage_nvfp4_lpips_golden.json create mode 100644 tests/integration/defs/examples/visual_gen/golden/visual_gen_lpips/wan21_cuda_graph_lpips_golden_video.json create mode 100644 tests/integration/defs/examples/visual_gen/golden/visual_gen_lpips/wan21_fp8_blockwise_lpips_golden_video.json create mode 100644 tests/integration/defs/examples/visual_gen/golden/visual_gen_lpips/wan21_nvfp4_lpips_golden_video.json create mode 100644 tests/integration/defs/examples/visual_gen/golden/visual_gen_lpips/wan22_cuda_graph_lpips_golden_video.json create mode 100644 tests/integration/defs/examples/visual_gen/golden/visual_gen_lpips/wan22_fp8_blockwise_lpips_golden_video.json create mode 100644 tests/integration/defs/examples/visual_gen/golden/visual_gen_lpips/wan22_nvfp4_lpips_golden_video.json delete mode 100644 tests/integration/defs/examples/visual_gen/test_visual_gen.py create mode 100644 tests/integration/defs/examples/visual_gen/test_visual_gen_cosmos3.py create mode 100644 tests/integration/defs/examples/visual_gen/test_visual_gen_flux.py create mode 100644 tests/integration/defs/examples/visual_gen/test_visual_gen_ltx2.py create mode 100644 tests/integration/defs/examples/visual_gen/test_visual_gen_qwen_image.py create mode 100644 tests/integration/defs/examples/visual_gen/test_visual_gen_wan.py create mode 100644 tests/integration/defs/examples/visual_gen/visual_gen_test_utils.py create mode 100644 tests/integration/defs/kimi_k3_disagg_parity.py create mode 100644 tests/integration/defs/kimi_k3_sa_harness.py create mode 100644 tests/integration/defs/test_kimi_k3_specdec.py delete mode 100644 tests/scripts/perf/disaggregated/gb200_deepseek-r1-fp4_128k8k_con128_ctx1_pp8_gen1_dep16_eplb0_mtp1_ccb-UCX.yaml delete mode 100644 tests/scripts/perf/disaggregated/gb200_deepseek-r1-fp4_8k1k_con4096_ctx1_dep4_gen1_dep16_eplb0_mtp1_ccb-UCX.yaml delete mode 100644 tests/scripts/perf/disaggregated/gb200_deepseek-v32-fp4_32k4k_con2048_ctx1_dep4_gen1_dep32_eplb288_mtp1_ccb-UCX.yaml delete mode 100644 tests/scripts/perf/disaggregated/gb200_deepseek-v32-fp4_8k1k_con4096_ctx1_dep4_gen1_dep32_eplb256_mtp0_ccb-UCX.yaml delete mode 100644 tests/scripts/perf/disaggregated/gb200_glm-5-fp4_8k1k_con1024_ctx1_dep4_gen1_dep8_eplb256_mtp1_ccb-UCX.yaml delete mode 100644 tests/scripts/perf/disaggregated/gb200_gpt-oss-120b-fp4_8k1k_con1024_ctx1_tp1_gen1_tp4_eplb0_mtp0_ccb-UCX.yaml delete mode 100644 tests/scripts/perf/disaggregated/gb200_kimi-k25-thinking-fp4_8k1k_con4096_ctx1_dep4_gen1_dep16_eplb0_mtp0_ccb-UCX.yaml delete mode 100644 tests/scripts/perf/disaggregated/gb200_qwen3-235b-fp4_8k1k_con1024_ctx1_tp1_gen1_dep8_eplb0_mtp0_ccb-UCX.yaml delete mode 100644 tests/scripts/perf/disaggregated/gb300_deepseek-r1-fp4_128k8k_con256_ctx1_pp4_gen1_dep8_eplb0_mtp1_ccb-UCX.yaml delete mode 100644 tests/scripts/perf/disaggregated/gb300_deepseek-r1-fp4_8k1k_con4096_ctx1_dep4_gen1_dep16_eplb0_mtp1_ccb-UCX.yaml delete mode 100644 tests/scripts/perf/disaggregated/gb300_deepseek-v32-fp4_32k4k_con2048_ctx1_dep4_gen1_dep32_eplb288_mtp1_ccb-UCX.yaml delete mode 100644 tests/scripts/perf/disaggregated/gb300_deepseek-v32-fp4_8k1k_con4096_ctx1_dep4_gen1_dep32_eplb256_mtp0_ccb-UCX.yaml delete mode 100644 tests/scripts/perf/disaggregated/gb300_glm-5-fp4_8k1k_con1024_ctx1_dep2_gen1_dep8_eplb256_mtp1_ccb-UCX.yaml delete mode 100644 tests/scripts/perf/disaggregated/gb300_kimi-k25-thinking-fp4_8k1k_con4096_ctx1_dep4_gen1_dep16_eplb0_mtp0_ccb-UCX.yaml rename tests/unittest/_torch/attention/sparse/{test_triton_gather_k_cache.py => dsa/test_kernels.py} (95%) create mode 100644 tests/unittest/_torch/attention/sparse/dsa/test_req_idx_per_token.py rename tests/unittest/_torch/attention/sparse/{kernel/test_triton.py => rocketkv/test_kernels.py} (98%) delete mode 100644 tests/unittest/_torch/attention/sparse/test_triton_topk.py create mode 100644 tests/unittest/_torch/cute_dsl/test_kimi_k3_kda_ptx_patch.py create mode 100644 tests/unittest/_torch/lora/test_fp8_lora_grouped_gemm_regressions.py create mode 100644 tests/unittest/_torch/misc/test_moe_a2a_warmup_timeout.py create mode 100644 tests/unittest/_torch/modeling/test_kda_mtp_decode_cute_parity.py create mode 100644 tests/unittest/_torch/modeling/test_kimi_k3_fp8_weight_read_gates.py create mode 100644 tests/unittest/_torch/modeling/test_kimi_kda_fused_verify_parity.py create mode 100644 tests/unittest/_torch/modeling/test_kimi_kda_verify_parity.py create mode 100644 tests/unittest/_torch/modeling/test_modeling_nemotron_h_moe_quant.py create mode 100644 tests/unittest/_torch/models/checkpoints/laguna/__init__.py create mode 100644 tests/unittest/_torch/models/checkpoints/mistral/__init__.py create mode 100644 tests/unittest/_torch/modules/fused_moe/test_configurable_moe.py create mode 100644 tests/unittest/_torch/modules/kimi_k3_attn_res/test_attn_res_op.py create mode 100644 tests/unittest/_torch/modules/kimi_kda/test_kda_cache_soundness.py create mode 100644 tests/unittest/_torch/modules/kimi_kda/test_kda_decode_op.py create mode 100644 tests/unittest/_torch/modules/kimi_kda/test_kda_prefill_op.py create mode 100644 tests/unittest/_torch/modules/kimi_kda/test_kda_prefill_state_parity.py create mode 100644 tests/unittest/_torch/modules/moe/kimi_k3_ref_moe/_moe_kernels.py create mode 100644 tests/unittest/_torch/modules/moe/kimi_k3_ref_moe/_mxfp4.py create mode 100644 tests/unittest/_torch/modules/moe/kimi_k3_ref_moe/kimi_k3_moe_block.py create mode 100644 tests/unittest/_torch/modules/moe/test_kimi_k3_mlp.py create mode 100644 tests/unittest/_torch/modules/moe/test_kimi_k3_moe_gate.py create mode 100644 tests/unittest/_torch/modules/moe/test_kimi_k3_situ_and_mul.py create mode 100644 tests/unittest/_torch/modules/moe/test_kimi_k3_situ_moe.py create mode 100644 tests/unittest/_torch/modules/test_w4a16_nvfp4_linear.py create mode 100644 tests/unittest/_torch/speculative/hw_agnostic/test_dflash_accept_stats.py create mode 100644 tests/unittest/_torch/speculative/hw_agnostic/test_kimi_k3_dflash_scaffold.py create mode 100644 tests/unittest/_torch/speculative/hw_agnostic/test_kimi_k3_dspark_semantics.py create mode 100644 tests/unittest/_torch/test_mnnvl_utils.py create mode 100644 tests/unittest/_torch/thop/parallel/test_kda_decode.py create mode 100644 tests/unittest/_torch/visual_gen/test_fastwan_dmd_math.py create mode 100644 tests/unittest/_torch/visual_gen/test_fastwan_pipeline.py create mode 100644 tests/unittest/_torch/visual_gen/test_wan_dup_up3d.py create mode 100644 tests/unittest/_torch/visual_gen/test_wan_vae_halo.py create mode 100644 tests/unittest/disaggregated/test_aux_buffer_registration.py create mode 100644 tests/unittest/disaggregated/test_kda_mamba_transfer.py create mode 100644 tests/unittest/executor/test_spec_dec_perf_metrics.py create mode 100644 tests/unittest/executor/test_spec_dec_stats_pairing.py create mode 100644 tests/unittest/llmapi/test_responses_utils.py create mode 100644 tests/unittest/others/test_lazy_model_zoo.py create mode 100644 tests/unittest/scripts/test_perf_sanity_helpers.py create mode 100644 tests/unittest/utils/test_confidential_compute.py diff --git a/.github/workflows/blossom-ci.yml b/.github/workflows/blossom-ci.yml index 91ca8ba0c6b0..2695c79656d6 100644 --- a/.github/workflows/blossom-ci.yml +++ b/.github/workflows/blossom-ci.yml @@ -144,6 +144,7 @@ jobs: "guqiqi", "h-guo18", "HandongLi-01", + "hanjingtian", "haow-nv", "hchings", "hello-11", diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 59d45940179f..1876bedff89e 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -140,6 +140,9 @@ common-files: &common_files | tensorrt_llm/_torch/custom_ops/userbuffers_custom_ops.py | tensorrt_llm/_torch/cute_dsl_kernels/__init__.py | tensorrt_llm/_torch/cute_dsl_kernels/blackwell/__init__.py | + tensorrt_llm/_torch/cute_dsl_kernels/blackwell/attention/mla/mla_decode_fp16.py | + tensorrt_llm/_torch/cute_dsl_kernels/blackwell/attention/mla/mla_decode_fp8.py | + tensorrt_llm/_torch/cute_dsl_kernels/blackwell/attention/mla/mla_helpers.py | tensorrt_llm/_torch/cute_dsl_kernels/blackwell/custom_pipeline.py | tensorrt_llm/_torch/cute_dsl_kernels/blackwell/dense_blockscaled_gemm_persistent.py | tensorrt_llm/_torch/cute_dsl_kernels/blackwell/utils.py | @@ -402,6 +405,7 @@ common-files: &common_files | tensorrt_llm/llmapi/mm_encoder.py | tensorrt_llm/llmapi/mpi_session.py | tensorrt_llm/llmapi/reasoning_parser.py | + tensorrt_llm/llmapi/serialization.py | tensorrt_llm/llmapi/tokenizer.py | tensorrt_llm/llmapi/tracer.py | tensorrt_llm/llmapi/tracing.py | @@ -452,7 +456,6 @@ common-files: &common_files | tensorrt_llm/scaffolding/task_collection.py | tensorrt_llm/scaffolding/worker.py | tensorrt_llm/scheduling_params.py | - tensorrt_llm/serialization.py | tensorrt_llm/serve/__init__.py | tensorrt_llm/serve/chat_utils.py | tensorrt_llm/serve/cluster_storage.py | @@ -911,6 +914,9 @@ legacy-files: &legacy_files | tensorrt_llm/_torch/custom_ops/userbuffers_custom_ops.py | tensorrt_llm/_torch/cute_dsl_kernels/__init__.py | tensorrt_llm/_torch/cute_dsl_kernels/blackwell/__init__.py | + tensorrt_llm/_torch/cute_dsl_kernels/blackwell/attention/mla/mla_decode_fp16.py | + tensorrt_llm/_torch/cute_dsl_kernels/blackwell/attention/mla/mla_decode_fp8.py | + tensorrt_llm/_torch/cute_dsl_kernels/blackwell/attention/mla/mla_helpers.py | tensorrt_llm/_torch/cute_dsl_kernels/blackwell/custom_pipeline.py | tensorrt_llm/_torch/cute_dsl_kernels/blackwell/dense_blockscaled_gemm_persistent.py | tensorrt_llm/_torch/cute_dsl_kernels/blackwell/utils.py | @@ -1173,6 +1179,7 @@ legacy-files: &legacy_files | tensorrt_llm/llmapi/mm_encoder.py | tensorrt_llm/llmapi/mpi_session.py | tensorrt_llm/llmapi/reasoning_parser.py | + tensorrt_llm/llmapi/serialization.py | tensorrt_llm/llmapi/tokenizer.py | tensorrt_llm/llmapi/tracer.py | tensorrt_llm/llmapi/tracing.py | @@ -1223,7 +1230,6 @@ legacy-files: &legacy_files | tensorrt_llm/scaffolding/task_collection.py | tensorrt_llm/scaffolding/worker.py | tensorrt_llm/scheduling_params.py | - tensorrt_llm/serialization.py | tensorrt_llm/serve/__init__.py | tensorrt_llm/serve/chat_utils.py | tensorrt_llm/serve/cluster_storage.py | @@ -1561,7 +1567,7 @@ static-analysis-files: &static_analysis_files | )$ # Global exclude: vendored code + trtllm-gen FMHA artifacts (cubin pointers, export headers, cuda_ptx) -exclude: '(^cpp/tensorrt_llm/common/sha256/|^triton_kernels/|trtllmGenKernels/fmha/cubin/kernelMetaInfo\.h$|cubin\.cpp$|cubin\.h$|trtllmGenKernels/fmha/trtllmGen_fmha_export/|trtllmGenKernels/fmha/cuda_ptx/)' +exclude: '(^cpp/tensorrt_llm/common/sha256/|^triton_kernels/|trtllmGenKernels/fmha/cubin/kernelMetaInfo\.h$|cubin\.cpp$|cubin\.h$|trtllmGenKernels/fmha/trtllmGen_fmha_export/|trtllmGenKernels/fmha/cuda_ptx/|trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/KernelMetaInfo\.h$)' default_install_hook_types: [pre-commit, commit-msg] repos: diff --git a/ATTRIBUTIONS-Python.md b/ATTRIBUTIONS-Python.md index 7e74b846a85f..4d2111f72cdd 100644 --- a/ATTRIBUTIONS-Python.md +++ b/ATTRIBUTIONS-Python.md @@ -5261,7 +5261,7 @@ For more information, please refer to - `Tracker`: https://github.com/tox-dev/py-filelock/issues -## flashinfer-python (0.6.15) +## flashinfer-python (0.6.16) ### Licenses License: `Apache-2.0` diff --git a/README.md b/README.md index 6c5f86b73102..0fe1bf535445 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ TensorRT LLM [![python](https://img.shields.io/badge/python-3.10-green)](https://www.python.org/downloads/release/python-31012/) [![cuda](https://img.shields.io/badge/cuda-13.2.1-green)](https://developer.nvidia.com/cuda-downloads) [![torch](https://img.shields.io/badge/torch-2.11.0-green)](https://pytorch.org) -[![version](https://img.shields.io/badge/release-1.3.0rc24-green)](https://github.com/NVIDIA/TensorRT-LLM/blob/main/tensorrt_llm/version.py) +[![version](https://img.shields.io/badge/release-1.3.0rc25-green)](https://github.com/NVIDIA/TensorRT-LLM/blob/main/tensorrt_llm/version.py) [![license](https://img.shields.io/badge/license-Apache%202-blue)](https://github.com/NVIDIA/TensorRT-LLM/blob/main/LICENSE) [Architecture](https://nvidia.github.io/TensorRT-LLM/developer-guide/overview.html)   |   [Performance](https://nvidia.github.io/TensorRT-LLM/developer-guide/perf-overview.html)   |   [Examples](https://nvidia.github.io/TensorRT-LLM/quick-start-guide.html)   |   [Documentation](https://nvidia.github.io/TensorRT-LLM/)   |   [Roadmap](https://github.com/NVIDIA/TensorRT-LLM/issues?q=is%3Aissue%20state%3Aopen%20label%3Aroadmap) diff --git a/cpp/cmake/modules/cuda_configuration.cmake b/cpp/cmake/modules/cuda_configuration.cmake index 98be47b7f5ff..740bcf08c3e1 100644 --- a/cpp/cmake/modules/cuda_configuration.cmake +++ b/cpp/cmake/modules/cuda_configuration.cmake @@ -61,7 +61,7 @@ Functions ``all`` or unset Resolves to architectures TensorRT-LLM is optimized for and the - compiler supports (80, 86, 89, 90, 100, 103, 120 depending on CUDA version). + compiler supports (80, 86, 89, 90, 100, 103, 107, 120 depending on CUDA version). ``all-major`` Unsupported. Results in a fatal error. @@ -341,6 +341,9 @@ function(setup_cuda_architectures) if(CMAKE_CUDA_COMPILER_VERSION VERSION_GREATER_EQUAL "12.9") list(APPEND CMAKE_CUDA_ARCHITECTURES_RAW 103) endif() + if(CMAKE_CUDA_COMPILER_VERSION VERSION_GREATER_EQUAL "13.4") + list(APPEND CMAKE_CUDA_ARCHITECTURES_RAW 107) + endif() endif() # CMAKE_CUDA_ARCHITECTURES_ORIG contains all architectures enabled, without @@ -431,6 +434,7 @@ function(setup_cuda_architectures) 90 100 103 + 107 120) foreach(CUDA_ARCH IN LISTS ARCHITECTURES_WITH_KERNELS) if(NOT ${CUDA_ARCH} IN_LIST CMAKE_CUDA_ARCHITECTURES_ORIG) diff --git a/cpp/include/tensorrt_llm/batch_manager/peftCacheManager.h b/cpp/include/tensorrt_llm/batch_manager/peftCacheManager.h index ed928e96d811..0438687c7ff9 100644 --- a/cpp/include/tensorrt_llm/batch_manager/peftCacheManager.h +++ b/cpp/include/tensorrt_llm/batch_manager/peftCacheManager.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved. + * Copyright (c) 2024-2026, NVIDIA CORPORATION. All rights reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -29,6 +29,8 @@ #include #include +#include +#include #include #include #include @@ -125,6 +127,11 @@ class PeftCacheManager : public BasePeftCacheManager [[nodiscard]] bool isTaskCachedDevice(uint64_t const taskId) const; + [[nodiscard]] tensorrt_llm::DataType getDataType() const; + + //! Configure the homogeneous LoRA weight dtype before inserting an adapter. + void configureDataType(tensorrt_llm::DataType dataType); + void resetDeviceCache() override; void markRequestDone(LlmRequest const& llmReq, bool pause = false) override; @@ -175,6 +182,9 @@ class PeftCacheManager : public BasePeftCacheManager runtime::ModelConfig mModelConfig; runtime::WorldConfig mWorldConfig; + mutable std::mutex mDataTypeMutex; + std::optional mDataType; + int mDevice{-1}; }; diff --git a/cpp/include/tensorrt_llm/common/cudaUtils.h b/cpp/include/tensorrt_llm/common/cudaUtils.h index a4ca3082b319..cca1e89e7270 100644 --- a/cpp/include/tensorrt_llm/common/cudaUtils.h +++ b/cpp/include/tensorrt_llm/common/cudaUtils.h @@ -306,10 +306,12 @@ inline int getSMVersion(bool queryRealSmArch = false) return sm; } -inline bool isSM100Family() +inline bool isSM100Family(std::optional sm = std::nullopt) { - int const sm = getSMVersion(); - return sm == 100 || sm == 103; // To be continued... + // Not value_or(): its argument is evaluated eagerly, so an explicit sm + // would still trigger a device query (which throws with no device present). + int smVersion = sm.has_value() ? *sm : getSMVersion(); + return smVersion >= 100 && smVersion < 110; } inline int getDevice() diff --git a/cpp/include/tensorrt_llm/runtime/loraCache.h b/cpp/include/tensorrt_llm/runtime/loraCache.h index eb4ef57494ea..d6f8ca964539 100644 --- a/cpp/include/tensorrt_llm/runtime/loraCache.h +++ b/cpp/include/tensorrt_llm/runtime/loraCache.h @@ -1,5 +1,5 @@ -/*loraCac - * Copyright (c) 2022-2023, NVIDIA CORPORATION. All rights reserved. +/* + * Copyright (c) 2022-2026, NVIDIA CORPORATION. All rights reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -186,6 +186,16 @@ class LoraCache LoraCache(LoraCachePageManagerConfig const& pageManagerConfig, ModelConfig const& modelConfig, WorldConfig const& worldConfig, BufferManager const& bufferManager); + /** + * \brief Reinitialize an empty cache to store the given data type. + * + * The page geometry remains unchanged. This is intended for selecting a + * homogeneous LoRA cache dtype before the first task is inserted. + */ + void setDataType(tensorrt_llm::DataType dataType); + + [[nodiscard]] tensorrt_llm::DataType getDataType() const; + /** * \brief put a task in the cache, and claim pages for it, and optionally load task weights. * diff --git a/cpp/tensorrt_llm/CMakeLists.txt b/cpp/tensorrt_llm/CMakeLists.txt index 5f5e37836a05..d9a7853bc9d5 100644 --- a/cpp/tensorrt_llm/CMakeLists.txt +++ b/cpp/tensorrt_llm/CMakeLists.txt @@ -186,6 +186,8 @@ set(TRTLLM_LINK_LIBS trtllm_gen_batched_gemm selective_scan_src mamba2_mtp_ssm_cache_src + kda_decode_src + kimi_k3_attn_res_src ws_layernorm_src fusedGatedRMSNormQuant_src fpA_intB_gemm_src diff --git a/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/AGENTS.md b/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/AGENTS.md new file mode 100644 index 000000000000..001d3296a829 --- /dev/null +++ b/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/AGENTS.md @@ -0,0 +1,302 @@ + + +# KVCacheManagerV2 C++ Guide + +This directory contains the C++ implementation of KVCacheManagerV2: allocation, +prefix reuse, eviction, GPU/host/disk movement, page locking, and KV-cache event +generation. Python accesses it through the nanobind bindings in +`cpp/tensorrt_llm/nanobind/batch_manager/kvCacheManagerV2.cpp`. + +This guide is self-contained. Do not make changes here depend on the Python +KVCacheManagerV2 implementation or its migration documents; those files are +temporary and will be removed after the migration. + +The current C++ headers and tests are the source of truth. Historical Python +behavior remains useful for compatibility testing, but migration-era proposed +layouts or ownership models must not override the implementation. + +## Layout + +- `common.h`, `config.h`, `tokenIdExt.h`, and `exceptions.h`: shared types, + configuration, token encoding, and error types. +- `blockRadixTree.*`: the shared prefix-reuse tree and SHA-256 block keys. +- `page.*`, `kvCache.*`, and `kvCacheManager.*`: page lifecycle, per-request + cache state, and the top-level manager. +- `storage/`, `storageManager.*`, `evictionController.*`, and `copyEngine.*`: + pools, eviction ownership, migration, and data movement. +- `lifeCycleRegistry.*`: layer-group/lifecycle mapping, including attention and + SSM behavior. +- `eventManager.*` and `eventSink.h`: KV-cache event derivation and delivery. +- `utils/`: typed indices, ownership helpers, CUDA events, host memory, and + math utilities. + +## Design principles + +- Prefer composition when it expresses ownership or containment. Use + inheritance only for a real IS-A relationship or required virtual dispatch, + such as `CommittedPage : Page` and `EventManager : EventSink`. +- Preserve strongly typed indices. `LayerGroupId` is a public lifecycle + semantic, `PoolGroupIndex` is a storage-layout index, `BlockOrdinal` is a + sequence position, and `SlotId`/page indices address storage. Do not convert + or compare them implicitly. +- Keep hot-path checks under `TLLM_CHECK_DEBUG*` or `gDebug` when they are too + expensive for release builds. Do not remove debug invariants simply because + production does not execute them. +- Keep the core independent of Python. C++ storage, copying, CUDA, hashing, and + lifecycle operations call C++ APIs directly; Python adaptation belongs in + nanobind. + +## Architecture and request flow + +`KvCacheManager` owns the global services: a `LifeCycleRegistry`, +`BlockRadixTree`, and `StorageManager`. It creates a `KvCache` per request. +The request cache starts `SUSPENDED`, becomes `ACTIVE` through `resume()`, and +must eventually be `close()`d. Its normal flow is: + +1. Match the input `TokenSpan` against `BlockRadixTree` within a `ReuseScope`. +2. Allocate request-local pages for unmatched blocks through `StorageManager`. +3. Lock or migrate required committed pages to GPU before model execution. +4. Commit completed blocks to the tree, making their immutable pages available + for later requests; `stopCommitting()` finalizes this process. +5. Suspend or close the request, returning pages to holding/eviction ownership. + +`LifeCycleRegistry` maps model layers to lifecycle groups. Attention lifecycles +may have sliding-window and sink-token rules; SSM lifecycles represent a +recurrent-state checkpoint. A pool-group index is a storage-layout index and is +not interchangeable with a layer ID or lifecycle ID. + +`StorageManager` coordinates GPU, host, and disk cache levels. It allocates +slots, schedules pages for eviction, migrates pages between levels, and resizes +pools. `CopyEngine` performs the actual batched transfers; C++ code calls it +directly and must not round-trip through Python bindings. + +The dependency direction is broadly: + +```text +types/config/exceptions + -> lifecycle + memory/CUDA utilities + -> storage pools + eviction + copy engine + radix tree + -> pages + storage manager + -> per-request KvCache + -> KvCacheManager + -> nanobind API +``` + +## State machines + +### Per-request cache + +- `SUSPENDED`: no active CUDA-stream use; committed pages can be held or + evicted. +- `ACTIVE`: pages required by the request are locked to GPU and use the cache's + CUDA stream. +- `CLOSED`: resources are released; further use is invalid. + +`commit()` finalizes full blocks. Its `isEnd=true` form is a terminal-memory +contract: later writes to the request's KV memory are invalid, because final +live pages may be moved into the radix tree rather than copied. + +`stopCommitting()` is a distinct transition and must not call `commit()`: +doing so would append the same tokens twice. It also releases stale held SWA +pages and performs final commit-state bookkeeping. + +### Page status + +- `LOCKED`: required on GPU; neither eviction nor dropping is permitted. +- `HELD`: eviction is allowed, but dropping is not. +- `DROPPABLE`: both eviction and dropping are allowed. + +The `PageHolder`, `UniqPageLock`, and `SharedPageLock` types implement these +transitions. CUDA ready/finish events are part of their correctness contract: +they establish write completion, migration ordering, and safe reuse across +streams. A stream change for an active cache intentionally synchronizes the +new stream with the old one. + +## Ownership and lifetime + +The high-level ownership shape is: + +```text +KvCacheManager +|- LifeCycleRegistry (value) +|- StorageManager (shared) +|- BlockRadixTree (shared) +`- living KvCache registry (non-owning pointers) + +KvCache +|- KvCacheManager (shared; cache keeps manager alive) +`- per-beam/per-block page holders and locks + +BlockRadixTree +`- roots -> child Blocks (strong ownership through next maps) + `- lifecycle page entries (raw observer links) + +Eviction controller +`- prioritized LRU lists (strong ownership of droppable Pages) +``` + +- Follow the existing `SharedPtr`/`WeakPtr` conventions in `utils/sharedPtr.h`. + Treat every ownership edge as intentional; do not replace weak edges with + strong ones merely to simplify access. +- `KvCache` keeps its `KvCacheManager` alive. The manager's registry of living + caches must not create the reverse strong-reference cycle. +- A committed page is referenced by the radix tree without making the tree its + permanent owner. Eviction queues may be the only strong owner of a droppable + page, so never store a raw pointer past the operation that obtained it. +- The eviction queue stores strong page ownership and its `NodeRef` is valid + only for the eviction policy that created it. Exclude a page from eviction + before moving it to another cache level; only then schedule it in the new + level's policy. +- Destructors can trigger tree detachment, page unlinking, or eviction updates. + Keep teardown order explicit, make cleanup idempotent where needed, and audit + re-entrancy before changing a destructor or `close`/`shutdown` path. +- `CommittedPage::numTokensInBlock` can be smaller than its block's token span. + For attention it describes a reusable prefix; for SSM it is an exact state + checkpoint. Do not assume every page covers its whole block. +- `Block::prev`, `Block::storage`, `CommittedPage::block`, and page manager + pointers are observer/back-reference links with lifetime invariants, not + ownership. Explicit unlinking and teardown order keep them valid. +- Orphan blocks may retain pages while a live `KvCache` still references them; + `Block::~Block()` reclaims those pages when the last block owner releases them. + Every `KvCache` must be closed before manager shutdown so this deferred cleanup + runs while `StorageManager` is still alive. + +## Correctness invariants + +- Block keys are SHA-256 digests over the reuse scope and the token sequence. + They are a security boundary for cross-request reuse. Do not replace, + truncate, or use a non-cryptographic hash unless prefix matching also gains a + token-content equality check. +- `knownNoDigest=true` is an external guarantee (for example `text_only`), not + a hint derived by scanning tokens. Passing it incorrectly corrupts key hashes. +- The radix tree owns child blocks and a child never outlives its parent. Preserve + parent/child attachment order when replacing or removing blocks. +- Root removal is deferred through `proposeToEraseEmptyRoot()` and drained only + at tree safe points. Do not erase roots directly from a destructor chain. +- Pages are immutable after commit. Locking and CUDA events establish when their + data is safe to read or migrate; preserve those synchronization boundaries. +- Event payloads describe complete blocks. A lifecycle with partial page coverage + must not emit an event for the full block. +- Hash bytes and token encoding must stay compatible across all callers. Normal + token IDs use the little-endian `TokenIdExt` representation; digest tokens and + `ReuseScope` values participate in the same chained block-key protocol. +- Partial attention coverage uses a `>=` prefix check. SSM coverage names an + exact recurrent-state checkpoint and must be truncated to that boundary. +- Multi-beam block arrays, sliding-window stale ranges, sink tokens, partial + block reuse, SSM snapshots, intra-batch rebasing, and rollback after OOM are + coupled inside `KvCache`; changes there require broad invariant testing. + +## Storage, eviction, and memory + +- Every cache level has storage and a per-level eviction controller; each pool + group has a priority-sorted set of LRU queues. Lower priority is evicted first, + then least-recently-used within that priority. +- `NodeRef` is a stable `std::list` iterator, but only within the exact + `LRUEvictionPolicy` that issued it. Iterators from different lists are not + comparable or interchangeable. +- Eviction failure must preserve queue consistency. If a multi-pool eviction + cannot satisfy all requested slots, restore pages already removed before + propagating `OutOfPagesError`. +- Host and disk memory code directly uses `mmap`, `munmap`, `mremap`, + `madvise`, `posix_fallocate`, and CUDA host registration. Preserve cleanup on + partial failure, CUDA unregister/register ordering across resize, and the + distinction between host and disk OOM errors. +- CUDA virtual memory and copy operations use the driver/runtime APIs directly. + Keep allocation granularity, stream ordering, and source/destination lifetime + valid through asynchronous copies. + +## Interfaces, bindings, and build + +- `TokenSpan` is non-owning. The caller retains its backing storage for the full + call; the manager reads tokens but never stores the span itself. This enables + the int32 zero-copy matching path. +- Keep C++ implementation sources co-located here and add every compiled source + to this directory's `CMakeLists.txt`. The parent target consumes its source + list; do not add a separate shared library for this subsystem. +- SHA-256 support is vendored under `cpp/tensorrt_llm/common/sha256` + and configured by this directory's `CMakeLists.txt`. Preserve the + architecture-specific SHA extension flags when changing the hash integration. + Do not add OpenSSL/libcrypto merely for block hashing; avoiding that dependency + is intentional for wheel portability. +- Nanobind bindings belong in + `cpp/tensorrt_llm/nanobind/batch_manager/kvCacheManagerV2.cpp`. Keep the + public Python surface compatible with the runtime package; use the + introspection API only for white-box tests and diagnostics. + +## Nanobind and concurrency + +- C++ public APIs are called under the executor's single-threaded KV-cache + access model. Do not add mutexes or relax that model without auditing every + manager, cache, page, and callback path. +- Binding code may release the GIL only while it touches no Python objects. Keep + Python conversions and callbacks under the GIL, and keep non-owning token + buffers alive for the entire C++ call. +- Reacquire the GIL before invoking a Python callback, wrapping a C++ result, + creating an `nb::object`, or changing Python reference counts. If the + single-threaded access precondition is ever relaxed, concurrency protection + must be designed for the whole manager rather than added piecemeal. +- Exceptions crossing the binding boundary need an explicit nanobind mapping. + Preserve Python exception type and attributes when adding or changing a C++ + exception. + +## High-risk changes + +Use extra review and tests for changes involving: + +- destructor, `shutdown()`, `close()`, tree detachment, or page unlink order; +- eviction ownership, `NodeRef`, migration, pool resizing, or OOM rollback; +- block hashes, token encoding, salts/LoRA reuse scopes, or `knownNoDigest`; +- partial attention coverage, SSM checkpoints, SWA windows/sinks, or final + snapshots; +- CUDA stream/event lifetime or host-memory registration; +- `KvCache` commit state, beam forks, reuse rebasing, or page-index buffers; +- nanobind GIL release, callbacks, non-owning buffers, or exception translation. + +## Development and tests + +- Focused C++ unit tests are in `cpp/tests/unit_tests/batch_manager/`, notably + `radixBlockTreeTest.cpp`, `kvCacheManagerTest.cpp`, + `kvCacheManagerV2DigestPoolTest.cpp`, `kvCacheManagerV2HostMemTest.cpp`, + `kvCacheManagerV2StatsTest.cpp`, and `kvCacheManagerV2TypedIndexTest.cpp`. +- Python behavior and backend-parity tests are in + `tests/unittest/kv_cache_manager_v2_tests/`. During development, prefer the + fast path below: set `PYTHONPATH` to `tensorrt_llm/runtime/` and execute the + test file directly with `python`. Do not use `pytest` for this fast path; the + file's test runner avoids importing the full `tensorrt_llm` package. + + ```bash + REPO_ROOT="$(git rev-parse --show-toplevel)" + PYTHONPATH="$REPO_ROOT/tensorrt_llm/runtime/" \ + python "$REPO_ROOT/tests/unittest/kv_cache_manager_v2_tests/test_kv_cache_manager_v2.py" -v + ``` + +- Run one test class or method by passing its unittest name: + + ```bash + REPO_ROOT="$(git rev-parse --show-toplevel)" + PYTHONPATH="$REPO_ROOT/tensorrt_llm/runtime/" \ + python "$REPO_ROOT/tests/unittest/kv_cache_manager_v2_tests/test_kv_cache_manager_v2.py" \ + TestNoBatching.test_basic -v + ``` + +- Before final validation, also exercise the production import path: + + ```bash + REPO_ROOT="$(git rev-parse --show-toplevel)" + PYTHONPATH="$REPO_ROOT/" \ + python "$REPO_ROOT/tests/unittest/kv_cache_manager_v2_tests/test_kv_cache_manager_v2.py" -v + ``` + +- Run `test_kv_cache_event_manager.py` after event changes, + `test_kv_cache_salting.py` after hashing/reuse-scope changes, and the stats + tests after allocation or event-accounting changes. Run both available + backends for changes shared by the C++ and Python surfaces. +- Run focused tests when possible, then the affected KVCacheManagerV2 Python + suite on both available backends. Use `TLLM_DEBUG_MODE=1` when diagnosing an + invariant failure. +- Follow the repository C++ standards in `CODING_GUIDELINES.md`: Allman braces, + east-const, explicit ownership, and clang-format. Do not make unrelated + formatting changes in this high-churn subsystem. diff --git a/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/CMakeLists.txt b/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/CMakeLists.txt index 163ad82ffd3f..9a7d3af13671 100644 --- a/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/CMakeLists.txt +++ b/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/CMakeLists.txt @@ -49,6 +49,7 @@ set(KV_CACHE_MANAGER_V2_SRCS kv_cache_manager_v2/storage/core.cpp kv_cache_manager_v2/evictionController.cpp kv_cache_manager_v2/copyEngine.cpp + kv_cache_manager_v2/tokenIdExt.cpp kv_cache_manager_v2/blockRadixTree.cpp kv_cache_manager_v2/eventManager.cpp kv_cache_manager_v2/page.cpp diff --git a/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/blockRadixTree.cpp b/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/blockRadixTree.cpp index 5d0d4bc314c0..a6db17976f30 100644 --- a/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/blockRadixTree.cpp +++ b/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/blockRadixTree.cpp @@ -25,11 +25,19 @@ #include "tensorrt_llm/common/assert.h" #include +#include #include #include #include #include +// Token hashing reinterprets TokenIdExt bytes as a raw little-endian stream (both +// the per-element and the bulk paths), so a normal token's 4 bytes equal its +// integer id. Guard the assumption at compile time (std::endian is C++20; C++17 here). +#if defined(__BYTE_ORDER__) && (__BYTE_ORDER__ != __ORDER_LITTLE_ENDIAN__) +#error "kv_cache_manager_v2 block hashing requires a little-endian target" +#endif + namespace tensorrt_llm::batch_manager::kv_cache_manager_v2 { @@ -82,17 +90,6 @@ void ensureSha256Detected() } } // namespace -static void hashInt64(CSHA256& h, int64_t v) -{ - unsigned char buf[8]; - auto const unsignedValue = static_cast(v); - for (int i = 0; i < 8; ++i) - { - buf[i] = static_cast((unsignedValue >> (8 * i)) & 0xFFU); - } - h.Write(buf, sizeof(buf)); -} - Hasher::Hasher() { ensureSha256Detected(); @@ -114,13 +111,15 @@ Hasher& Hasher::update(ReuseScope const& scope) Hasher& Hasher::update(TokenId token) { - hashInt64(mState, static_cast(token)); + static_assert(sizeof(TokenId) == sizeof(TokenIdExt)); + assert(TokenIdExt(token).tokenId() == token); + mState.Write(reinterpret_cast(&token), sizeof(token)); return *this; } -Hasher& Hasher::update(BlockKey const& key) +Hasher& Hasher::update(Digest const& digest) { - mState.Write(reinterpret_cast(key.data()), key.size()); + mState.Write(reinterpret_cast(digest.data()), digest.size()); return *this; } @@ -132,26 +131,50 @@ Hasher& Hasher::update(std::vector const& bytes) Hasher& Hasher::update(TokenIdExt const& tokenExt) { - std::visit( - [this](auto const& v) - { - using T = std::decay_t; - if constexpr (std::is_same_v) - hashInt64(mState, static_cast(v)); - else - mState.Write(reinterpret_cast(v.data()), v.size()); - }, - tokenExt); - return *this; + if (tokenExt.isDigest()) + { + return update(tokenExt.digest()); + } + return update(tokenExt.tokenId()); } -Hasher& Hasher::update(TokenIdExt const* tokens, size_t count) +Hasher& Hasher::update(TokenIdExt const* tokens, size_t count, bool knownNoDigest) { - // Python uses array("Q", data).tobytes() to reduce per-token interpreter - // overhead. In C++ the compiler inlines each update() call, so the loop - // is already optimal; batching would only add a heap allocation. - for (size_t i = 0; i < count; ++i) - update(tokens[i]); + TokenIdExt const* const end = tokens + count; + + // Bulk-write a contiguous run of normal tokens [begin, stop) as one raw + // little-endian uint32 block — the whole point of the 4-byte layout. A single + // Write of k tokens is byte-identical to k per-token writes. + auto const writeNormalRun = [this](TokenIdExt const* begin, TokenIdExt const* stop) + { + if (begin != stop) + { + mState.Write( + reinterpret_cast(begin), static_cast(stop - begin) * sizeof(TokenIdExt)); + } + }; + + if (knownNoDigest) + { + TLLM_CHECK_DEBUG(std::none_of(tokens, end, [](TokenIdExt const& t) { return t.isDigest(); })); + writeNormalRun(tokens, end); + return *this; + } + + // Unknown/digest-bearing: bulk-write each maximal run of normal tokens and + // hash each (rare, multi-modal) digest on its own. All-normal collapses to a + // single Write. + for (TokenIdExt const* pos = tokens; pos != end;) + { + TokenIdExt const* const digestIt = std::find_if(pos, end, [](TokenIdExt const& t) { return t.isDigest(); }); + writeNormalRun(pos, digestIt); + if (digestIt == end) + { + break; + } + update(digestIt->digest()); // 32 digest bytes + pos = digestIt + 1; + } return *this; } @@ -166,10 +189,10 @@ BlockKey Hasher::digest() const } // --------------------------------------------------------------------------- -// genMultiModalTokens +// genMultimodalCacheKeyTokens // --------------------------------------------------------------------------- -std::vector genMultiModalTokens( +std::vector genMultimodalCacheKeyTokens( int idOffset, std::vector const& multiModalDataDigest, int numTokens, int tokenOffset) { TLLM_CHECK_DEBUG(numTokens > 0); @@ -181,68 +204,18 @@ std::vector genMultiModalTokens( { if (tokenOffset + i == 0) { - Digest d; - std::memcpy(d.data(), multiModalDataDigest.data(), kDIGEST_LEN); - result.emplace_back(DigestToken(d)); + Digest digest; + std::memcpy(digest.data(), multiModalDataDigest.data(), kDIGEST_LEN); + result.emplace_back(digest); } else { - result.emplace_back(TokenId(idOffset + tokenOffset + i)); + result.emplace_back(TokenId{idOffset + tokenOffset + i}); } } return result; } -// --------------------------------------------------------------------------- -// makeBlockchainKeyGenerator — lazy key generator. -// Returns a callable that yields one BlockKey per call (nullopt when done). -// First call yields root entry (empty token block). Mirrors Python's generator. -// --------------------------------------------------------------------------- - -static auto makeBlockchainKeyGenerator( - int tokensPerBlock, ReuseScope reuseScope, TokenIdExt const* tokens, size_t numTokens) -{ - // digest carries the running hash from the previous block. - BlockKey digest = Hasher(reuseScope).digest(); - // ordinal = -1: next call yields root (reuseScope digest). - // ordinal >= 0: next call yields key for tokens[ordinal*tpb .. (ordinal+1)*tpb). - int ordinal = -1; - - return [=]() mutable -> std::optional - { - if (ordinal == -1) - { - ordinal++; - return digest; // root key - } - - size_t beg = static_cast(ordinal) * static_cast(tokensPerBlock); - if (beg >= numTokens) - return std::nullopt; - - size_t end = std::min(beg + static_cast(tokensPerBlock), numTokens); - - Hasher h; - h.update(digest); - h.update(tokens + beg, end - beg); - digest = h.digest(); - - ordinal++; - return digest; - }; -} - -// Eager wrapper for callers that need all keys at once. -std::vector sequenceToBlockchainKeys( - int tokensPerBlock, ReuseScope const& reuseScope, std::vector const& tokens) -{ - std::vector result; - auto gen = makeBlockchainKeyGenerator(tokensPerBlock, reuseScope, tokens.data(), tokens.size()); - while (auto key = gen()) - result.push_back(*key); - return result; -} - // --------------------------------------------------------------------------- // RootBlock // --------------------------------------------------------------------------- @@ -309,35 +282,55 @@ SharedPtr NodeBase::detachNext(BlockKey const& blockKey) namespace { -static bool isPrefix(std::vector const& prefix, std::vector const& full) +// Takes raw (ptr, size) so it works uniformly over any TokenIdExt buffer +// (the query vector and the std::vector that backs Block::tokens). +bool isPrefix(TokenIdExt const* prefix, size_t prefixLen, TokenIdExt const* full, size_t fullLen) { - if (prefix.size() > full.size()) + if (prefixLen > fullLen) + { return false; - for (size_t i = 0; i < prefix.size(); ++i) + } + for (size_t i = 0; i < prefixLen; ++i) { if (prefix[i] != full[i]) + { return false; + } } return true; } } // anonymous namespace -BlockKey Block::makeKey(BlockKey const& prevKey, TokenIdExt const* tokens, size_t count) +BlockKey Block::makeKey(BlockKey const& prevKey, TokenIdExt const* tokens, size_t count, bool knownNoDigest) { Hasher h; h.update(prevKey); - h.update(tokens, count); + h.update(tokens, count, knownNoDigest); return h.digest(); } -Block::Block(BlockKey k, std::vector toks, NodeBase* prevNode, LifeCycleId numLifeCycles) +Block::Block(BlockKey k, std::vector toks, NodeBase* prevNode) : NodeBase(k, prevNode->eventSink) , tokens(std::move(toks)) , prev(prevNode) - , storage(numLifeCycles, nullptr) + , storage(prevNode->numLifeCycles(), nullptr) // tree-wide count, derived from prev , mOrdinal(prevNode->ordinal() + 1) { + // key is a caller-supplied second source of truth (precomputed for the + // pre-construction dedup lookup). Verify it matches what we'd derive from + // prev + tokens so the two can never silently drift. Debug-only and fully + // compiled out in release: this re-hashes (exactly the recomputation the param + // exists to avoid). knownNoDigest=false lets makeKey's update() scan for digests + // itself — correct regardless of content, so no separate scan is needed here. + TLLM_CHECK_DEBUG(k == Block::makeKey(prevNode->key, tokens.data(), tokens.size(), /*knownNoDigest=*/false)); +} + +// Delegates to the tree, mirroring Python's RootBlock.num_life_cycles. Defined +// out-of-line so BlockRadixTree is complete at the point of use. +LifeCycleId RootBlock::numLifeCycles() const noexcept +{ + return tree->numLifeCycles(); } int Block::tokensPerBlock() const noexcept @@ -508,8 +501,7 @@ std::vector> Block::clearStaleBlocksAfterPageUnlink( // addOrGetExistingBlock // --------------------------------------------------------------------------- -SharedPtr addOrGetExistingBlock( - NodeBase* prev, LifeCycleId numLifeCycles, std::vector tokens, bool* isNew) +SharedPtr addOrGetExistingBlock(NodeBase* prev, std::vector tokens, bool knownNoDigest, bool* isNew) { TLLM_CHECK_DEBUG_WITH_INFO(prev, "prev must not be null"); @@ -521,7 +513,7 @@ SharedPtr addOrGetExistingBlock( auto& prevNext = prev->next; int const tpb = prev->tokensPerBlock(); - BlockKey newKey = Block::makeKey(prev->key, tokens.data(), tokens.size()); + BlockKey newKey = Block::makeKey(prev->key, tokens.data(), tokens.size(), knownNoDigest); // Exact match: return existing block (not new — mirrors Python's UselessBlockError path). auto it = prevNext.find(newKey); @@ -538,7 +530,8 @@ SharedPtr addOrGetExistingBlock( { for (auto const& [k, sibling] : prevNext) { - if (sibling->tokens.size() >= tokens.size() && isPrefix(tokens, sibling->tokens)) + if (sibling->tokens.size() >= tokens.size() + && isPrefix(tokens.data(), tokens.size(), sibling->tokens.data(), sibling->tokens.size())) throw UselessBlockError(sibling); } } @@ -552,7 +545,8 @@ SharedPtr addOrGetExistingBlock( std::vector toRemove; for (auto const& [k, sibling] : prevNext) { - if (sibling->tokens.size() < tokens.size() && isPrefix(sibling->tokens, tokens)) + if (sibling->tokens.size() < tokens.size() + && isPrefix(sibling->tokens.data(), sibling->tokens.size(), tokens.data(), tokens.size())) { TLLM_CHECK_DEBUG(!sibling->isFull() && sibling->key == k && sibling->next.empty()); toRemove.push_back(k); @@ -562,8 +556,9 @@ SharedPtr addOrGetExistingBlock( // would already have replaced the shorter one. TLLM_CHECK_DEBUG(toRemove.size() <= 1); - // Create the new block. ordinal and tokensPerBlock are derived from prev inside the Block ctor. - auto block = makeShared(newKey, std::move(tokens), prev, numLifeCycles); + // Create the new block. ordinal, tokensPerBlock, and numLifeCycles are all + // derived from prev. Block stores the tokens as a plain vector (moved in). + auto block = makeShared(newKey, std::move(tokens), prev); // Keep the parent attached while covered children are replaced. Adding the replacement // first prevents detachNext() from pruning an emptied RootBlock out of the tree. @@ -593,10 +588,6 @@ SharedPtr removeSubtree(Block& root) // Post-order traversal using prev/next links — O(1) extra space. // Descend to leaves first, remove on the way back up. - // Each block's pages are reclaimed eagerly via releasePages() while the - // StorageManager is still alive, rather than deferring to ~Block(): an external - // reference can keep a Block alive past StorageManager teardown, after which - // page->manager would be dangling. Mirrors Python's remove_subtree(). while (true) { // Descend: if the current block has children, go to the first child. @@ -606,10 +597,7 @@ SharedPtr removeSubtree(Block& root) } else { - current->releasePages(); - // Remove this block from its parent's next map. - // Null prev to detach — the block may outlive the tree if held - // externally (e.g., by nanobind/Python shared_ptr). + // Remove this block from its parent's next map and null prev to detach it. NodeBase* parent = current->prev; BlockKey const currentKey = current->key; auto detached = parent->detachNext(currentKey); @@ -644,7 +632,7 @@ BlockRadixTree::BlockRadixTree( BlockRadixTree::~BlockRadixTree() { - // Clear all roots (which will drop all blocks). + // Clear all roots (which will drop all blocks without external owners). mRoots.clear(); } @@ -728,51 +716,50 @@ int numMatchedTokens(std::vector const& matched, in } // anonymous namespace std::vector BlockRadixTree::matchTokenPath( - ReuseScope const& reuseScope, std::vector const& tokens, bool enablePartialMatch) const + ReuseScope const& reuseScope, TokenSpan tokens, bool knownNoDigest, bool enablePartialMatch) const { drainPendingRootErases(); std::vector results; // Lazily compute one key per iteration — no wasted hashing on early miss. - auto gen = makeBlockchainKeyGenerator(mTokensPerBlock, reuseScope, tokens.data(), tokens.size()); + auto gen = sequenceToBlockchainKeys(mTokensPerBlock, reuseScope, tokens.begin(), tokens.size(), knownNoDigest); - // First key is the root key. - auto rootKey = gen(); - if (!rootKey) + // First step is the root key (empty token range). + auto rootStep = gen(); + if (!rootStep) return results; - auto rootIt = mRoots.find(*rootKey); + auto rootIt = mRoots.find(rootStep->key); if (rootIt == mRoots.end()) return results; RootBlock const& root = *rootIt->second; std::unordered_map> const* currentNext = &root.next; - // ordinal tracks which block we're on (0-based, after root). - BlockOrdinal ordinal{0}; + // Token range of the first unmatched block, captured on miss for the partial pass. + HalfOpenRange missedRange; bool missed = false; - while (auto key = gen()) + // Each step carries the block's key and its token range — no need to re-derive + // block boundaries here. + while (auto step = gen()) { - auto blockIt = currentNext->find(*key); + auto blockIt = currentNext->find(step->key); if (blockIt == currentNext->end()) { + missedRange = step->tokens; missed = true; break; } - size_t beg = toSizeT(ordinal) * static_cast(mTokensPerBlock); - int numTokens = static_cast(std::min(static_cast(mTokensPerBlock), tokens.size() - beg)); Block* block = blockIt->second.get(); - results.push_back({block, numTokens}); + results.push_back({block, static_cast(step->tokens.length())}); currentNext = &block->next; - ordinal++; } // Partial match in children of current node. if (missed && enablePartialMatch) { - size_t beg = toSizeT(ordinal) * static_cast(mTokensPerBlock); - size_t missedCount = std::min(static_cast(mTokensPerBlock), tokens.size() - beg); - auto [best, bestMatch] = findBestPartialMatchInNextNodes(*currentNext, tokens.data() + beg, missedCount); + auto [best, bestMatch] + = findBestPartialMatchInNextNodes(*currentNext, tokens.begin() + missedRange.beg, missedRange.length()); if (best) results.push_back({best, bestMatch}); } @@ -874,9 +861,9 @@ std::vector BlockRadixTree::pruneMatch(std::vector< } BlockRadixTree::ReuseMatch BlockRadixTree::match( - ReuseScope const& reuseScope, std::vector const& tokens, bool enablePartialMatch) const + ReuseScope const& reuseScope, TokenSpan tokens, bool knownNoDigest, bool enablePartialMatch) const { - auto const matched = pruneMatch(matchTokenPath(reuseScope, tokens, enablePartialMatch)); + auto const matched = pruneMatch(matchTokenPath(reuseScope, tokens, knownNoDigest, enablePartialMatch)); ReuseMatch result{}; result.numTokens = numMatchedTokens(matched, mTokensPerBlock); result.numLookupTokens = static_cast(tokens.size()); diff --git a/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/blockRadixTree.h b/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/blockRadixTree.h index c4a1debcb093..aff76497dc40 100644 --- a/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/blockRadixTree.h +++ b/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/blockRadixTree.h @@ -20,10 +20,13 @@ #include "kv_cache_manager_v2/common.h" #include "kv_cache_manager_v2/eventSink.h" #include "kv_cache_manager_v2/lifeCycleRegistry.h" +#include "kv_cache_manager_v2/tokenIdExt.h" +#include "kv_cache_manager_v2/utils/math.h" // HalfOpenRange #include "kv_cache_manager_v2/utils/sharedPtr.h" #include "sha256.h" +#include #include #include #include @@ -88,11 +91,14 @@ class Hasher explicit Hasher(ReuseScope const& seed); Hasher& update(TokenId token); - Hasher& update(BlockKey const& key); + Hasher& update(Digest const& digest); // 32 raw bytes (BlockKey is a Digest alias) Hasher& update(ReuseScope const& scope); Hasher& update(std::vector const& bytes); Hasher& update(TokenIdExt const& tokenExt); - Hasher& update(TokenIdExt const* tokens, size_t count); + // knownNoDigest: caller guarantees the range holds no digest, enabling the bulk + // fast path. Only pass true from external knowledge (request/model text_only) — never + // from scanning the tokens, since false already makes update() scan internally. + Hasher& update(TokenIdExt const* tokens, size_t count, bool knownNoDigest = false); BlockKey digest() const; @@ -100,16 +106,61 @@ class Hasher CSHA256 mState; }; +// One step of the blockchain-key generator: a block's key plus the half-open token +// index range [beg, end) it covers ([0, 0) for the root). Carrying the range lets +// callers slice the tokens without re-deriving block boundaries, mirroring Python's +// (token_block, key) pairs. +struct BlockchainKeyStep +{ + BlockKey key; + HalfOpenRange tokens; +}; + // --------------------------------------------------------------------------- -// Utility: convert a token sequence → list of BlockKeys. -// First key is the root (reuseScope digest), then one per token block. -// Mirrors Python's sequence_to_blockchain_keys(). +// sequenceToBlockchainKeys — lazy per-block key generator. +// Returns a callable yielding one BlockchainKeyStep per call (nullopt when done): +// the first call yields the root (reuseScope digest, empty [0,0) range), then one +// step per tokensPerBlock chunk chained on the previous digest. Mirrors Python's +// sequence_to_blockchain_keys(). Lazy so a caller that stops early (e.g. on the +// first mismatch) skips the remaining hashing. Inline so both the tree +// (matchTokenPath) and the nanobind layer drive the same generator. +// knownNoDigest: from external text_only knowledge, never a scan (see Hasher::update). // --------------------------------------------------------------------------- -std::vector sequenceToBlockchainKeys( - int tokensPerBlock, ReuseScope const& reuseScope, std::vector const& tokens); +inline auto sequenceToBlockchainKeys( + int tokensPerBlock, ReuseScope reuseScope, TokenIdExt const* tokens, size_t numTokens, bool knownNoDigest = false) +{ + // digest carries the running hash from the previous block. + BlockKey digest = Hasher(reuseScope).digest(); + // ordinal = -1: next call yields root (reuseScope digest). + // ordinal >= 0: next call yields key for tokens[ordinal*tpb .. (ordinal+1)*tpb). + int ordinal = -1; + + return [=]() mutable -> std::optional + { + if (ordinal == -1) + { + ordinal++; + return BlockchainKeyStep{digest, {}}; // root key, empty [0,0) token range + } -// Generate multi-modal token IDs (mirrors gen_multi_modal_tokens in Python). -std::vector genMultiModalTokens( + size_t beg = static_cast(ordinal) * static_cast(tokensPerBlock); + if (beg >= numTokens) + return std::nullopt; + + size_t end = std::min(beg + static_cast(tokensPerBlock), numTokens); + + Hasher h; + h.update(digest); + h.update(tokens + beg, end - beg, knownNoDigest); + digest = h.digest(); + + ordinal++; + return BlockchainKeyStep{digest, {beg, end}}; + }; +} + +// Generate multi-modal token IDs (mirrors gen_multimodal_cache_key_tokens in Python). +std::vector genMultimodalCacheKeyTokens( int idOffset, std::vector const& multiModalDataDigest, int numTokens, int tokenOffset = 0); // --------------------------------------------------------------------------- @@ -139,6 +190,10 @@ struct NodeBase /// RootBlock: delegates to tree. Block: len(prev->tokens) or prev->tokensPerBlock(). virtual int tokensPerBlock() const noexcept = 0; + /// Tree-wide life-cycle count. RootBlock: delegates to tree. Block: storage.size(). + /// Mirrors Python's num_life_cycles property. + virtual LifeCycleId numLifeCycles() const noexcept = 0; + protected: NodeBase(BlockKey k, EventSink* sink) : key(k) @@ -172,6 +227,7 @@ struct RootBlock : NodeBase } int tokensPerBlock() const noexcept override; + LifeCycleId numLifeCycles() const noexcept override; // delegates to tree }; // --------------------------------------------------------------------------- @@ -181,6 +237,8 @@ struct RootBlock : NodeBase // --------------------------------------------------------------------------- struct Block : NodeBase, EnableSharedFromThis { + // A block's tokens are written once and never re-hashed because its key is + // computed before construction, so store them in a plain vector. std::vector tokens; // Previous node in the chain (RootBlock or Block). Null after detaching from the tree. @@ -189,10 +247,14 @@ struct Block : NodeBase, EnableSharedFromThis TypedVec storage; - Block(BlockKey key, std::vector tokens, NodeBase* prev, LifeCycleId numLifeCycles); + // key is precomputed by the caller (for the pre-construction dedup lookup); + // numLifeCycles is derived from prev. tokens is moved in as a plain vector. + Block(BlockKey key, std::vector tokens, NodeBase* prev); ~Block() override; - static BlockKey makeKey(BlockKey const& prevKey, TokenIdExt const* tokens, size_t count); + // knownNoDigest: from external text_only knowledge, never a scan (see Hasher::update). + static BlockKey makeKey( + BlockKey const& prevKey, TokenIdExt const* tokens, size_t count, bool knownNoDigest = false); Type type() const noexcept override { @@ -206,7 +268,7 @@ struct Block : NodeBase, EnableSharedFromThis int tokensPerBlock() const noexcept override; - LifeCycleId numLifeCycles() const noexcept + LifeCycleId numLifeCycles() const noexcept override { return storage.size(); } @@ -266,11 +328,10 @@ struct Block : NodeBase, EnableSharedFromThis // Reclaim every page held by this block: null each page's back-pointer and, for // DROPPABLE pages still scheduled for eviction, remove them from the eviction - // controller (releasing their storage slots). Idempotent. Must run during tree - // teardown (removeSubtree) rather than being deferred to ~Block(), so page - // reclamation does not depend on this Block's destruction timing — an external - // reference can keep a Block alive past StorageManager teardown, after which - // page->manager would be dangling. Mirrors Python's Block._release_pages(). + // controller (releasing their storage slots). Idempotent. Cleanup is normally + // deferred to ~Block(): an orphan block may remain referenced by a live KvCache, + // and every KvCache must close before StorageManager teardown. Mirrors Python's + // Block._release_pages(). void releasePages(); private: @@ -309,10 +370,13 @@ class BlockRadixTree int numLookupTokens; }; - ReuseMatch match( - ReuseScope const& reuseScope, std::vector const& tokens, bool enablePartialMatch = false) const; + // knownNoDigest: from external text_only knowledge, never a scan (see Hasher::update). + // Takes a non-owning TokenSpan so a zero-copy int32 token buffer can be matched without + // allocating/copying (the hot path). Callers holding a std::vector pass toSpan(vec). + ReuseMatch match(ReuseScope const& reuseScope, TokenSpan tokens, bool knownNoDigest = false, + bool enablePartialMatch = false) const; - // Clear all cached pages. ~Block() handles excludeFromEviction for DROPPABLE pages. + // Detach all cached blocks. ~Block() releases pages when the last owner drops a block. void clear(); int tokensPerBlock() const noexcept @@ -346,8 +410,9 @@ class BlockRadixTree } private: + // knownNoDigest: from external text_only knowledge, never a scan (see Hasher::update). std::vector matchTokenPath( - ReuseScope const& reuseScope, std::vector const& tokens, bool enablePartialMatch) const; + ReuseScope const& reuseScope, TokenSpan tokens, bool knownNoDigest, bool enablePartialMatch) const; std::vector pruneMatch(std::vector matched) const; // Erase any pending empty root blocks from mRoots. @@ -371,8 +436,9 @@ class BlockRadixTree // prefix of an existing sibling — mirrors Python's UselessBlockError. // If isNew is non-null, *isNew is set to true if a new block was created, false // if an existing block was returned. +// knownNoDigest: from external text_only knowledge, never a scan (see Hasher::update). SharedPtr addOrGetExistingBlock( - NodeBase* prev, LifeCycleId numLifeCycles, std::vector tokens, bool* isNew = nullptr); + NodeBase* prev, std::vector tokens, bool knownNoDigest, bool* isNew = nullptr); // Post-order traversal: remove a subtree rooted at `root` from its parent's // next map. ~Block() handles page cleanup. Mirrors Python's remove_subtree(). diff --git a/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/common.h b/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/common.h index 0f53938b95e2..459997a042e3 100644 --- a/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/common.h +++ b/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/common.h @@ -17,17 +17,21 @@ #pragma once +#include "kv_cache_manager_v2/tokenIdExt.h" // TokenId, Digest, TokenIdExt #include "kv_cache_manager_v2/utils/typedIndex.h" #include "tensorrt_llm/batch_manager/common.h" +#include "tensorrt_llm/common/assert.h" #include #include #include -#include +#include #include #include #include +#include #include +#include namespace tensorrt_llm::batch_manager::kv_cache_manager_v2 { @@ -76,96 +80,12 @@ enum class PageIndexMode : int using CacheLevel = StrongIndex; inline constexpr CacheLevel kGpuLevel{0}; -// Vocabulary token identifier (normal tokens only). -using TokenId = int64_t; - // Opaque request identifier shared with the rest of the batch manager. using RequestIdType = tensorrt_llm::batch_manager::RequestIdType; // Opaque LoRA task identifier shared with the rest of the batch manager. using LoraTaskIdType = tensorrt_llm::runtime::LoraTaskIdType; -// 32-byte aligned to enable SIMD. -inline constexpr int kDIGEST_LEN = 32; - -struct alignas(kDIGEST_LEN) Digest : std::array -{ - // Custom operator== needed to emit SIMD code - bool operator==(Digest const& o) const noexcept - { - return std::memcmp(this, &o, kDIGEST_LEN) == 0; - } - - bool operator!=(Digest const& o) const noexcept - { - return !(*this == o); - } -}; - -// Heap-allocated digest token for multi-modal tokens. -// Copyable (deep-copies the digest) with value-based equality. -// Digest tokens are rare, so unique_ptr keeps sizeof(TokenIdExt) small. -class DigestToken -{ -public: - explicit DigestToken(Digest const& d) - : mData(std::make_unique(d)) - { - } - - explicit DigestToken(std::unique_ptr d) - : mData(std::move(d)) - { - } - - DigestToken(DigestToken const& o) - : mData(std::make_unique(*o.mData)) - { - } - - DigestToken(DigestToken&&) noexcept = default; - - DigestToken& operator=(DigestToken const& o) - { - if (this != &o) - mData = std::make_unique(*o.mData); - return *this; - } - - DigestToken& operator=(DigestToken&&) noexcept = default; - - bool operator==(DigestToken const& o) const - { - return *mData == *o.mData; - } - - bool operator!=(DigestToken const& o) const - { - return !(*this == o); - } - - std::byte const* data() const noexcept - { - return mData->data(); - } - - size_t size() const noexcept - { - return mData->size(); - } - - Digest const& digest() const noexcept - { - return *mData; - } - -private: - std::unique_ptr mData; -}; - -// Extended token id: normal TokenId or a heap-allocated digest for multi-modal tokens. -using TokenIdExt = std::variant; - // Ordinal index of a KV cache block (sequence of tokens). using BlockOrdinal = StrongIndex; inline constexpr BlockOrdinal kBadBlockOrdinal{-1}; @@ -203,6 +123,69 @@ inline constexpr Priority kPriorityDefault = 35; // Optional sliding window size (nullopt = no sliding window). using SlidingWindowSize = std::optional; +// --------------------------------------------------------------------------- +//! Non-owning view into a contiguous buffer (a C++17 stand-in for std::span). +//! +//! The referenced buffer must outlive the view. This remains an aggregate, so +//! `Span{}` creates an empty view and `Span{ptr, len}` is plain brace-init. +//! Supports operator[] for uniform access with std::vector. +// --------------------------------------------------------------------------- +template +struct Span +{ + T* ptr = nullptr; + int len = 0; + + T& operator[](int idx) + { + return ptr[idx]; + } + + T const& operator[](int idx) const + { + return ptr[idx]; + } + + int size() const noexcept + { + return len; + } + + T* data() const noexcept + { + return ptr; + } + + T* begin() const noexcept + { + return ptr; + } + + T* end() const noexcept + { + return ptr + len; + } +}; + +//! Create a non-owning const Span over a std::vector. +//! +//! The source vector must outlive the returned view and must not reallocate while +//! the view is in use. +template +inline Span toSpan(std::vector const& vec) noexcept +{ + TLLM_CHECK_DEBUG(vec.size() <= static_cast(std::numeric_limits::max())); + return Span{vec.data(), static_cast(vec.size())}; +} + +//! Non-owning view of a token sequence; the source buffer must outlive the view. +//! +//! Used on the hot ingest path: a digest-free int32 token buffer can be +//! reinterpret_cast to TokenIdExt const* and matched/hashed with no per-token +//! copy. TokenIdExt is 4 bytes and bit-identical to a normal int32 token (see +//! tokenIdExt.h). +using TokenSpan = Span; + // --------------------------------------------------------------------------- // Address types // --------------------------------------------------------------------------- @@ -228,16 +211,3 @@ using Address = std::variant; using DataRole = std::string; } // namespace tensorrt_llm::batch_manager::kv_cache_manager_v2 - -// std::hash specialization for Digest/BlockKey so unordered_map works without a custom hasher. -template <> -struct std::hash -{ - size_t operator()(tensorrt_llm::batch_manager::kv_cache_manager_v2::Digest const& k) const noexcept - { - // First 8 bytes of a SHA-256 digest are already well-distributed. - uint64_t v; - std::memcpy(&v, k.data(), sizeof(v)); - return static_cast(v); - } -}; diff --git a/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/config.h b/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/config.h index 6852ed51f5a1..0471a7f0f3ea 100644 --- a/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/config.h +++ b/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/config.h @@ -293,6 +293,12 @@ struct KVCacheManagerConfig // Collect V2 KV cache allocation, reuse, and transfer statistics. bool enableStats = true; + // Deployment-level guarantee that no request carries multi-modal content, so token + // sequences never contain digests. Lets block-key hashing take the digest-free fast + // path without scanning. A per-KvCache text_only override may only tighten this + // (a text-only deployment forbids a request claiming otherwise). Default false. + bool textOnly = false; + bool enableSwaScratchReuse() const noexcept { return swaScratchReuse.has_value(); diff --git a/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/eventManager.cpp b/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/eventManager.cpp index 301d941d1cfc..e8fce2216564 100644 --- a/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/eventManager.cpp +++ b/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/eventManager.cpp @@ -564,17 +564,16 @@ std::optional EventManager::storedBlockFromBlock( tokens.reserve(block.tokens.size()); for (auto const& token : block.tokens) { - if (auto const* tokenId = std::get_if(&token)) + if (!token.isDigest()) { UniqueToken uniqueToken; - uniqueToken.tokenId = EventTokenId{std::in_place_index<0>, *tokenId}; + uniqueToken.tokenId = EventTokenId{std::in_place_index<0>, token.tokenId()}; tokens.push_back(std::move(uniqueToken)); } else { UniqueToken uniqueToken; - uniqueToken.tokenId - = EventTokenId{std::in_place_index<1>, digestToHex(std::get(token).digest())}; + uniqueToken.tokenId = EventTokenId{std::in_place_index<1>, digestToHex(token.digest())}; tokens.push_back(std::move(uniqueToken)); } } @@ -645,13 +644,12 @@ uint64_t EventManager::v1HashFromBlock(Block const& block) { for (auto const& token : currentBlock.tokens) { - auto const* tokenId = std::get_if(&token); - if (tokenId == nullptr) + if (token.isDigest()) { parentIsV1Compatible = false; break; } - textTokens.push_back(*tokenId); + textTokens.push_back(token.tokenId()); } } if (parentIsV1Compatible) diff --git a/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCache.cpp b/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCache.cpp index 6beb2bd4f9eb..c6a1c51c5388 100644 --- a/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCache.cpp +++ b/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCache.cpp @@ -58,7 +58,8 @@ void copySlotData(StorageManager& storageMgr, CacheLevel dstLevel, CacheLevel sr // --------------------------------------------------------------------------- KvCache::KvCache(KvCacheManager& manager, ReuseScope reuseScope, std::optional reuseMatch, - std::optional mId, PriorityCb priorityCb, std::optional expectedPromptLength) + std::optional mId, PriorityCb priorityCb, std::optional expectedPromptLength, + std::optional textOnly) : id(mId) , mManager(manager.shared_from_this()) , mReuseScope(std::move(reuseScope)) @@ -92,6 +93,10 @@ KvCache::KvCache(KvCacheManager& manager, ReuseScope reuseScope, std::optional{false}), + "text_only=false is not allowed when the manager is configured text_only=true"); + mTextOnly = textOnly.value_or(manager.textOnly()); + if (reuseMatch.has_value()) { _setupForReuse(*reuseMatch); @@ -873,7 +878,6 @@ void KvCache::_snapshotPartialBlockToTree(BlockOrdinal ordinal, bool commitSsm) int const numTokens = static_cast(tokens.size()); TLLM_CHECK_DEBUG(0 < numTokens && numTokens < mTokensPerBlock); - LifeCycleId numLc = mManager->storage().numLifeCycles(); NodeBase* prevNode = nullptr; RootBlock& root = mManager->radixTree().addOrGetExisting(mReuseScope); if (ordinal == BlockOrdinal{0}) @@ -889,7 +893,7 @@ void KvCache::_snapshotPartialBlockToTree(BlockOrdinal ordinal, bool commitSsm) SharedPtr treeBlock; try { - treeBlock = addOrGetExistingBlock(prevNode, numLc, tokens, &isNew); + treeBlock = addOrGetExistingBlock(prevNode, tokens, textOnly(), &isNew); } catch (UselessBlockError const& e) { @@ -1535,7 +1539,7 @@ void KvCache::_commitBlock(int ord, bool isLast, bool commitSsm, bool moveSsm) SharedPtr newBlock; try { - newBlock = addOrGetExistingBlock(prevNode, numLc, tokenBlock, &blockIsNew); + newBlock = addOrGetExistingBlock(prevNode, tokenBlock, textOnly(), &blockIsNew); } catch (UselessBlockError const& e) { @@ -1713,12 +1717,12 @@ void KvCache::_commitBlock(int ord, bool isLast, bool commitSsm, bool moveSsm) // commit // --------------------------------------------------------------------------- -void KvCache::commit(std::vector const& tokens, bool isEnd) +void KvCache::commit(TokenSpan tokens, bool isEnd) { TLLM_CHECK_DEBUG(mStatus == Status::ACTIVE); if (mBeamWidth != BeamIndex{1}) throw LogicError("Not implemented yet for beam search"); - if (tokens.empty()) + if (tokens.size() == 0) { if (isEnd) stopCommitting(); @@ -1727,6 +1731,10 @@ void KvCache::commit(std::vector const& tokens, bool isEnd) if (mCommitState == CommitState::USER_STOP) throw LogicError("Cannot commit tokens after stop_committing()"); + TLLM_CHECK_DEBUG_WITH_INFO( + !mTextOnly || std::none_of(tokens.begin(), tokens.end(), [](TokenIdExt const& t) { return t.isDigest(); }), + "Cannot commit digest tokens to a text-only KV cache"); + bool const commitMinSnapshot = mManager->commitMinSnapshot(); auto ssmLcId = mManager->lifeCycles().ssmLifeCycleId(); if (commitMinSnapshot) @@ -1902,7 +1910,7 @@ std::unique_ptr KvCache::planCommittedBlockDrop() if (numCommittedTokens() == 0) return nullptr; - auto const match = mManager->matchReuse(mReuseScope, mCommittedTokens); + auto const match = mManager->matchReuse(mReuseScope, toSpan(mCommittedTokens), textOnly()); if (match.numTokens != numCommittedTokens() || match.blocks.empty()) return nullptr; @@ -2519,6 +2527,32 @@ void KvCache::setEnableSwaScratchReuse(bool enable) mEnableSwaScratchReuse = false; } +bool KvCache::textOnly() const noexcept +{ + return mTextOnly; +} + +void KvCache::setTextOnly(bool textOnly) +{ + // A text-only deployment is a hard guarantee: a request may not opt out. + if (!textOnly && mManager->textOnly()) + { + throw std::invalid_argument( + "Cannot set text_only=false for a request when the KV cache manager is configured text_only=true"); + } + // Claiming text-only is a fast-path claim; verify the committed tokens are digest-free. + if (textOnly) + { + bool const hasDigest = std::any_of( + mCommittedTokens.begin(), mCommittedTokens.end(), [](TokenIdExt const& t) { return t.isDigest(); }); + if (hasDigest) + { + throw std::invalid_argument("Cannot set text_only=true: this sequence has already committed digest tokens"); + } + } + mTextOnly = textOnly; +} + KvCache::DeltaScratchSlots KvCache::_takeExcessScratchSlots(int capacity, int historyLength) { LifeCycleId numLc = mManager->storage().numLifeCycles(); diff --git a/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCache.h b/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCache.h index 2fca81c8d17e..afb6e3cce152 100644 --- a/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCache.h +++ b/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCache.h @@ -113,47 +113,6 @@ struct SeqBlock } }; -// --------------------------------------------------------------------------- -// Span — non-owning view into a contiguous buffer. -// Supports operator[] for uniform access with std::vector. -// --------------------------------------------------------------------------- -template -struct Span -{ - T* ptr; - int32_t len; - - T& operator[](int idx) - { - return ptr[idx]; - } - - T operator[](int idx) const - { - return ptr[idx]; - } - - int size() const noexcept - { - return len; - } - - T* data() const noexcept - { - return ptr; - } - - T* begin() const noexcept - { - return ptr; - } - - T* end() const noexcept - { - return ptr + len; - } -}; - // --------------------------------------------------------------------------- // PlannedDropHandle — tracks committed pages planned for dropping without // owning them. Mirrors Python's PlannedDropHandle in _core/_kv_cache.py. @@ -211,7 +170,8 @@ class KvCache : public std::enable_shared_from_this using PriorityCb = std::function; KvCache(KvCacheManager& manager, ReuseScope reuseScope, std::optional reuseMatch, - std::optional id, PriorityCb priorityCb, std::optional expectedPromptLength = std::nullopt); + std::optional id, PriorityCb priorityCb, std::optional expectedPromptLength = std::nullopt, + std::optional textOnly = std::nullopt); ~KvCache(); @@ -257,7 +217,7 @@ class KvCache : public std::enable_shared_from_this // This is a terminal-memory contract: callers must not perform later writes // to this KvCache's memory. The final live pages may be moved into the radix // tree instead of copied (SSM state and the last partial block). - void commit(std::vector const& tokens, bool isEnd = false); + void commit(TokenSpan tokens, bool isEnd = false); // Stop committing (called by close() automatically). void stopCommitting(); @@ -431,6 +391,16 @@ class KvCache : public std::enable_shared_from_this // Enable or disable SWA scratch reuse. Throws if the transition is invalid. void setEnableSwaScratchReuse(bool enable); + // Resolved text-only status: the per-sequence override, else the manager config + // default. When true, this sequence's tokens are known to be digest-free, letting + // block-key hashing skip the digest scan. + bool textOnly() const noexcept; + + // Set the per-sequence text-only override. Throws if the transition is invalid: + // a text-only deployment (config.textOnly) forbids setting false, and setting true + // requires the already-committed tokens to be digest-free (verified by a scan). + void setTextOnly(bool textOnly); + // Whether the given page index mode is supported (SHARED requires no scratch slots). bool supportsIndexMode(PageIndexMode mode) const; @@ -609,6 +579,8 @@ class KvCache : public std::enable_shared_from_this TypedVec mBlocks; std::vector mCommittedTokens; + // Resolved per-sequence text-only state after applying the manager default. + bool mTextOnly = false; int mNumCommittedBlocks; std::optional mFinishEvent; int mTokensPerBlock; diff --git a/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCacheManager.cpp b/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCacheManager.cpp index ae136b0306f7..ba61fcfbb04a 100644 --- a/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCacheManager.cpp +++ b/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCacheManager.cpp @@ -23,6 +23,7 @@ #include "kv_cache_manager_v2/utils/math.h" #include "tensorrt_llm/common/assert.h" +#include "tensorrt_llm/common/logger.h" #include #include #include @@ -127,11 +128,25 @@ KvCacheManager::KvCacheManager(KVCacheManagerConfig const& config, std::shared_p KvCacheManager::~KvCacheManager() { - shutdown(); + try + { + shutdown(); + } + catch (std::exception const& e) + { + TLLM_LOG_ERROR("%s", e.what()); + } +} + +void KvCacheManager::_checkNoLivingKvCaches(char const* api) const +{ + TLLM_CHECK_WITH_INFO(mLivingKvCaches.empty(), + "%s with %zu KV cache(s) still open; close them (or drain the engine) first", api, mLivingKvCaches.size()); } void KvCacheManager::shutdown() { + _checkNoLivingKvCaches("shutdown()"); clearReusableBlocks(); TLLM_CHECK_DEBUG(mStorage); @@ -150,20 +165,21 @@ void KvCacheManager::shutdown() void KvCacheManager::clearReusableBlocks() { + _checkNoLivingKvCaches("clear_reusable_blocks()"); TLLM_CHECK_DEBUG(mRadixTree); mRadixTree->clear(); } -std::shared_ptr KvCacheManager::createKvCache(ReuseScope reuseScope, - std::vector const& inputTokens, std::optional id, KvCache::PriorityCb priorityCb, - std::optional expectedPromptLength) +std::shared_ptr KvCacheManager::createKvCache(ReuseScope reuseScope, TokenSpan inputTokens, + std::optional id, KvCache::PriorityCb priorityCb, std::optional expectedPromptLength, + std::optional textOnly) { if (!priorityCb) { priorityCb = [](BlockOrdinal, LifeCycleId) { return kPriorityDefault; }; } - if (!expectedPromptLength.has_value() && !inputTokens.empty()) + if (!expectedPromptLength.has_value() && inputTokens.size() != 0) { expectedPromptLength = static_cast(inputTokens.size()); } @@ -172,24 +188,24 @@ std::shared_ptr KvCacheManager::createKvCache(ReuseScope reuseScope, // hand it to the KvCache, rather than having the cache re-walk the radix tree. // Mirrors Python KVCacheManager.allocate() passing a ReuseMatch into _KVCache. std::optional reuseMatch; - if (!inputTokens.empty()) + if (inputTokens.size() != 0) { - reuseMatch = matchReuse(reuseScope, inputTokens); + reuseMatch = matchReuse(reuseScope, inputTokens, textOnly.value_or(this->textOnly())); } return std::make_shared(*this, std::move(reuseScope), std::move(reuseMatch), std::move(id), - std::move(priorityCb), expectedPromptLength); + std::move(priorityCb), expectedPromptLength, textOnly); } BlockRadixTree::ReuseMatch KvCacheManager::matchReuse( - ReuseScope const& reuseScope, std::vector const& inputTokens) const + ReuseScope const& reuseScope, TokenSpan inputTokens, bool knownNoDigest) const { - return mRadixTree->match(reuseScope, inputTokens, enablePartialMatch()); + return mRadixTree->match(reuseScope, inputTokens, knownNoDigest, enablePartialMatch()); } -int KvCacheManager::probeReuse(ReuseScope reuseScope, std::vector const& inputTokens) const +int KvCacheManager::probeReuse(ReuseScope reuseScope, TokenSpan inputTokens, bool knownNoDigest) const { - return matchReuse(reuseScope, inputTokens).numTokens; + return matchReuse(reuseScope, inputTokens, knownNoDigest).numTokens; } // ---- Memory pool queries -------------------------------------------------- diff --git a/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCacheManager.h b/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCacheManager.h index a9737b5b282c..0f5990862524 100644 --- a/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCacheManager.h +++ b/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/kvCacheManager.h @@ -133,15 +133,25 @@ class KvCacheManager : public std::enable_shared_from_this // ---- KvCache creation ------------------------------------------------- // Create a new KvCache. Returned cache is SUSPENDED; call activate() with a stream. - // input_tokens: optional sequence to match against existing cached blocks. - // priorityCb: optional priority override per block. - std::shared_ptr createKvCache(ReuseScope reuseScope = {}, std::vector const& inputTokens = {}, + // input_tokens: optional sequence to match against existing cached blocks. + // priorityCb: optional priority override per block. + // expectedPromptLength: token count marking the prefill->generation boundary; once + // historyLength reaches it, later capacity growth is recorded as + // generation-phase allocation stats (defaults to inputTokens.size()). + // Stats-only: no effect on allocation, reuse, or correctness. + // textOnly: per-sequence override of the text-only (digest-free) guarantee; + // nullopt inherits the manager config default. + // inputTokens is a non-owning view; the caller must keep the underlying buffer alive for the + // duration of the call (matching reads it but never stores it). + std::shared_ptr createKvCache(ReuseScope reuseScope = {}, TokenSpan inputTokens = {}, std::optional id = std::nullopt, KvCache::PriorityCb priorityCb = {}, - std::optional expectedPromptLength = std::nullopt); + std::optional expectedPromptLength = std::nullopt, std::optional textOnly = std::nullopt); + // knownNoDigest: from external text_only knowledge, never a scan (see Hasher::update). + // Defaults false (safe: the scanning path is taken). BlockRadixTree::ReuseMatch matchReuse( - ReuseScope const& reuseScope, std::vector const& inputTokens) const; - int probeReuse(ReuseScope reuseScope = {}, std::vector const& inputTokens = {}) const; + ReuseScope const& reuseScope, TokenSpan inputTokens, bool knownNoDigest = false) const; + int probeReuse(ReuseScope reuseScope = {}, TokenSpan inputTokens = {}, bool knownNoDigest = false) const; // ---- Memory pool queries ----------------------------------------------- @@ -175,6 +185,12 @@ class KvCacheManager : public std::enable_shared_from_this return mConfig.commitMinSnapshot; } + // Deployment-level text-only guarantee (see KVCacheManagerConfig::textOnly). + bool textOnly() const noexcept + { + return mConfig.textOnly; + } + bool isSwaScratchReuseEnabled() const noexcept { return mConfig.enableSwaScratchReuse(); @@ -298,6 +314,10 @@ class KvCacheManager : public std::enable_shared_from_this friend class KvCacheIntrospection; private: + // Throw unless every KvCache has been closed. `api` names the caller so the message + // points at the mistake rather than at whatever breaks later. + void _checkNoLivingKvCaches(char const* api) const; + void _adjustLevel(CacheLevel level, size_t quota); bool _needAdjustment(CacheLevel level) const; TypedVec const& _getTargetRatioList(CacheLevel level) const; @@ -314,8 +334,8 @@ class KvCacheManager : public std::enable_shared_from_this KVCacheManagerConfig mConfig; LifeCycleRegistry mLifeCycles; std::shared_ptr mEventSink; - std::shared_ptr mRadixTree; std::shared_ptr mStorage; + std::shared_ptr mRadixTree; // Weak references to all living KvCaches. std::set mLivingKvCaches; diff --git a/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/page.h b/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/page.h index fc3a02252471..a446196177cd 100644 --- a/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/page.h +++ b/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/page.h @@ -130,7 +130,6 @@ class UncommittedPage : public Page KvCache* kvCache; BlockOrdinal ordinal; BeamIndex beamIndex; - std::vector tokens; UncommittedPage(KvCache& kvc, BlockOrdinal ord, LifeCycleId lc, CacheLevel level, BeamIndex bi = kDefaultBeamIndex); diff --git a/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/tokenIdExt.cpp b/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/tokenIdExt.cpp new file mode 100644 index 000000000000..b6ab10dab190 --- /dev/null +++ b/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/tokenIdExt.cpp @@ -0,0 +1,235 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "kv_cache_manager_v2/tokenIdExt.h" + +#include "kv_cache_manager_v2/utils/math.h" // DynamicBitset +#include "tensorrt_llm/common/assert.h" + +#include +#include +#include +#include +#include + +namespace tensorrt_llm::batch_manager::kv_cache_manager_v2 +{ + +// --------------------------------------------------------------------------- +// DigestPool — process-global, address-stable store of 32-byte multi-modal +// Digests, referenced by a 31-bit slot index packed into a TokenIdExt. It is a +// pure implementation detail of TokenIdExt, so it lives here (anonymous +// namespace) rather than in the header. +// +// Digests are rare (multi-modal only), so all access is guarded by a single +// mutex; the all-normal-token hashing fast path never touches the pool. Storage +// is a std::deque (element references stay valid across push_back — unlike a +// reallocating std::vector). Occupancy is a DynamicBitset; alloc() takes the +// LOWEST free slot (front-packing via a rolling minFreeHint) so live digests +// cluster at the front and free() can pop trailing free slots off the deque +// (never remapping a live index → outstanding borrows stay valid). The index is +// unobservable (equality/hashing use the 32 bytes, never the index), so a single +// static singleton is safe. +// --------------------------------------------------------------------------- +namespace +{ + +class DigestPool +{ +public: + static DigestPool& instance() + { + static DigestPool pool; + return pool; + } + + ~DigestPool() + { + // Runs at process exit (single-threaded → no lock). Every digest + // TokenIdExt should have freed its slot by now; a nonzero count means one + // outlived the pool — a lifetime bug. Use fprintf, not the logger, whose + // lifetime at static destruction is not guaranteed. + if (size_t const inUse = mInUse.numSetBits(); inUse != 0) + { + (void) std::fprintf(stderr, "[ERROR] DigestPool destroyed with %zu digest slot(s) still in use\n", inUse); + } + } + + DigestPool(DigestPool const&) = delete; + DigestPool& operator=(DigestPool const&) = delete; + DigestPool(DigestPool&&) = delete; + DigestPool& operator=(DigestPool&&) = delete; + + // Store a copy of `digest` in the lowest free slot; return its index. + uint32_t alloc(Digest const& digest) + { + std::lock_guard const lock(mMutex); + return allocLocked(digest); + } + + // Duplicate the digest at slot `idx` into a fresh slot; return the new index. + uint32_t duplicate(uint32_t idx) + { + std::lock_guard const lock(mMutex); + return allocLocked(mStore[idx]); // Safe for deque + } + + // The digest at slot `idx`. The reference stays valid after the lock is + // released and across later shrinks (which only pop free tail slots). Uses + // at() so a bad index (e.g. the sentinel of a moved-from handle) throws + // rather than reading out of bounds; digests are rare so the check is cheap. + [[nodiscard]] Digest const& get(uint32_t idx) const + { + std::lock_guard const lock(mMutex); + return mStore.at(idx); + } + + // Clear slot `idx` and reclaim trailing free slots. + void free(uint32_t idx) + { + if (idx == TokenIdExt::kValueMask) + { + return; // the default / moved-from sentinel index — nothing to free + } + std::lock_guard const lock(mMutex); + mInUse.clear(idx); + if (idx < mMinFreeHint) + { + mMinFreeHint = idx; // a lower slot is now free + } + shrinkTailLocked(); + } + + [[nodiscard]] size_t liveCount() const + { + std::lock_guard const lock(mMutex); + return mInUse.numSetBits(); + } + +private: + DigestPool() = default; + + // Slot count. Also checks the occupancy bitset stays sized to the store. + // Precondition: caller holds mMutex and mStore/mInUse are in sync (i.e. not + // called between growing/shrinking one and resizing the other). + [[nodiscard]] size_t capacity() const + { + TLLM_CHECK_DEBUG(mInUse.size() == mStore.size()); + return mStore.size(); + } + + // Precondition: caller holds mMutex. Store `digest` in the lowest free slot + // (front-packing), growing the deque only when no free slot exists. + uint32_t allocLocked(Digest const& digest) + { + size_t const cap = capacity(); + size_t idx = mMinFreeHint; + while (idx < cap && mInUse.get(idx)) + { + ++idx; + } + if (idx == cap) + { + // No free slot below the high-water mark — grow by one. Indices stay + // strictly below kValueMask, which is reserved as the bad-handle sentinel. + TLLM_CHECK_WITH_INFO(cap < TokenIdExt::kValueMask, "DigestPool exhausted the 31-bit index space"); + mStore.push_back(digest); + mInUse.resize(mStore.size()); // re-sync the bitset; new bit is clear + } + else + { + mStore[idx] = digest; + } + mInUse.set(idx); + mMinFreeHint = idx + 1; // everything below is now occupied + return static_cast(idx); + } + + // Precondition: caller holds mMutex. Pop free tail slots off the deque when + // the trailing-free run reaches kSlackHigh, leaving ~kSlackLow of buffer. + void shrinkTailLocked() + { + size_t const cap = capacity(); + // Highest in-use index + 1 (scan down over the trailing free run only). + size_t liveEnd = cap; + while (liveEnd > 0 && !mInUse.get(liveEnd - 1)) + { + --liveEnd; + } + size_t const trailingFree = cap - liveEnd; + if (trailingFree < kSlackHigh) + { + return; + } + size_t const newCapacity = liveEnd + kSlackLow; // leave a small buffer + // Only free tail slots are dropped, so no live index is remapped. + mStore.resize(newCapacity); + mInUse.resize(newCapacity); + if (mMinFreeHint > liveEnd) + { + mMinFreeHint = liveEnd; // lowest free slot is now at the reclaimed tail + } + } + + // Absolute slack (in slots), NOT a ratio: a deque grows/shrinks one fixed + // chunk at a time, so we only damp boundary-block churn. + static constexpr size_t kSlackHigh = 256; + static constexpr size_t kSlackLow = 64; + + mutable std::mutex mMutex; + std::deque mStore; // slot storage; mStore.size() == the slot count (== bitset capacity) + DynamicBitset mInUse{0}; // bit i set == slot i occupied + size_t mMinFreeHint{0}; // lower bound on the lowest free slot index +}; + +} // namespace + +// --------------------------------------------------------------------------- +// TokenIdExt — RAII members that touch the pool (construct/copy=alloc, dtor=free). +// --------------------------------------------------------------------------- + +TokenIdExt::TokenIdExt(Digest const& digestValue) + : mBits(DigestPool::instance().alloc(digestValue) | kTagMask) +{ +} + +void TokenIdExt::freeSlot(uint32_t index) noexcept +{ + DigestPool::instance().free(index); +} + +uint32_t TokenIdExt::duplicateSlot(uint32_t index) +{ + return DigestPool::instance().duplicate(index); +} + +Digest const& TokenIdExt::digest() const +{ + return DigestPool::instance().get(digestIndex()); +} + +namespace detail +{ + +size_t digestPoolLiveCount() +{ + return DigestPool::instance().liveCount(); +} + +} // namespace detail + +} // namespace tensorrt_llm::batch_manager::kv_cache_manager_v2 diff --git a/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/tokenIdExt.h b/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/tokenIdExt.h new file mode 100644 index 000000000000..c0248361dd9e --- /dev/null +++ b/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/tokenIdExt.h @@ -0,0 +1,243 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/assert.h" + +#include +#include +#include +#include +#include +#include + +namespace tensorrt_llm::batch_manager::kv_cache_manager_v2 +{ + +// Vocabulary token identifier (normal tokens only). 32-bit, matching the +// runtime-wide tensorrt_llm::runtime::TokenIdType — real token ids are < vocab +// size (~10^5-10^6), well within int32. +using TokenId = std::int32_t; + +// 32-byte aligned to enable SIMD. +inline constexpr int kDIGEST_LEN = 32; + +struct alignas(kDIGEST_LEN) Digest : std::array +{ + // Custom operator== needed to emit SIMD code + bool operator==(Digest const& o) const noexcept + { + return std::memcmp(this, &o, kDIGEST_LEN) == 0; + } + + bool operator!=(Digest const& o) const noexcept + { + return !(*this == o); + } +}; + +// --------------------------------------------------------------------------- +// TokenIdExt — 4-byte self-describing token handle (RAII value type). +// +// One uint32_t; the high bit tags the low 31 bits: +// - tag 0: normal token id (stored verbatim). An all-normal array is a +// contiguous little-endian int32 array, hashed in one CSHA256::Write(N*4). +// - tag 1: multi-modal digest; low bits index a slot in an internal pool that +// holds the 32-byte Digest. +// +// A digest handle owns its pool slot: construct from a Digest to allocate, copy +// to clone into a fresh slot, and destroy to free. Normal handles own nothing. +// Pool-touching members are defined out-of-line in tokenIdExt.cpp. +// --------------------------------------------------------------------------- +class TokenIdExt +{ +public: + static constexpr uint32_t kTagMask = 0x80000000U; + static constexpr uint32_t kValueMask = 0x7FFFFFFFU; + // Sentinel for default / moved-from handles: a *digest* (tag 1) whose index is + // the reserved value kValueMask — never a real slot. Tagging it as a digest + // leaves the whole tag-0 space to real token ids, and its destructor is a + // no-op because DigestPool::free ignores that index. Copying or dereferencing + // a moved-from handle is a bug and fails loudly (duplicate/get hit a bad slot). + static constexpr uint32_t kBadToken = kTagMask | kValueMask; // 0xFFFFFFFF + // Maximum normal token id — the full 31-bit range. (Digest slot indices stay + // strictly below kValueMask, which is reserved for the sentinel above.) + static constexpr uint32_t kMaxValue = kValueMask; + + TokenIdExt() noexcept = default; // kBadToken (see mBits initializer) + + // Normal token id (tag 0). Precondition: 0 <= id <= kMaxValue (so the value + // fits the 31-bit field and leaves the digest tag bit clear). + explicit TokenIdExt(TokenId id) + : mBits(static_cast(id)) + { + TLLM_CHECK_DEBUG(id >= 0 && id <= static_cast(kMaxValue)); + } + + // Multi-modal digest (tag 1): copies `digest` into a fresh pool slot. + explicit TokenIdExt(Digest const& digest); + + ~TokenIdExt() + { + if (isDigest()) + { + freeSlot(digestIndex()); + } + } + + // Clones a digest slot; a normal token just copies its bits. + TokenIdExt(TokenIdExt const& other) + : mBits(other.isDigest() ? (duplicateSlot(other.digestIndex()) | kTagMask) : other.mBits) + { + } + + // Clones a digest slot; a normal token just copies its bits. + TokenIdExt& operator=(TokenIdExt const& other) + { + if (this != &other) + { + // Allocate the clone before releasing our slot (self-safe, exception-safe). + uint32_t const newBits = other.isDigest() ? (duplicateSlot(other.digestIndex()) | kTagMask) : other.mBits; + if (isDigest()) + { + freeSlot(digestIndex()); + } + mBits = newBits; + } + return *this; + } + + TokenIdExt(TokenIdExt&& other) noexcept + : mBits(other.mBits) + { + other.mBits = kBadToken; // steal the slot; leave source empty + } + + TokenIdExt& operator=(TokenIdExt&& other) noexcept + { + if (this != &other) + { + if (isDigest()) + { + freeSlot(digestIndex()); + } + mBits = other.mBits; + other.mBits = kBadToken; + } + return *this; + } + + [[nodiscard]] bool isDigest() const noexcept + { + return (mBits & kTagMask) != 0; + } + + // Valid iff !isDigest(). A normal handle's tag bit is clear by construction + // (the ctor range-checks id <= kMaxValue), so the raw bits are already the id. + [[nodiscard]] TokenId tokenId() const noexcept + { + TLLM_CHECK_DEBUG(!isDigest()); + return static_cast(mBits); + } + + // The pooled 32-byte digest. Precondition: isDigest(). + [[nodiscard]] Digest const& digest() const; + + // Raw 4-byte payload — consumed by the bulk-hash fast path. + [[nodiscard]] uint32_t raw() const noexcept + { + return mBits; + } + + // Value equality: normal/tag-mismatch compare raw bits; digest-vs-digest + // compares the pooled 32 bytes (equal content in different slots is equal). + bool operator==(TokenIdExt const& other) const + { + if (mBits == other.mBits) + { + return true; + } + // Distinct bits are unequal unless both are digests whose 32-byte contents + // match (equal digests may live in different pool slots — no dedup). + if (isDigest() && other.isDigest()) + { + return digest() == other.digest(); + } + return false; + } + + bool operator!=(TokenIdExt const& other) const + { + return !(*this == other); + } + +private: + [[nodiscard]] uint32_t digestIndex() const noexcept + { + TLLM_CHECK_DEBUG(isDigest()); + return mBits & kValueMask; + } + + static void freeSlot(uint32_t index) noexcept; + [[nodiscard]] static uint32_t duplicateSlot(uint32_t index); + + uint32_t mBits{kBadToken}; +}; + +// Binary compatibility with int32_t is a deliberate design contract, not a coincidence: a +// non-negative token id has the same 4-byte little-endian image as the TokenIdExt holding it. +// The nanobind zero-copy ingest path relies on it to view a contiguous int32 token buffer as +// TokenIdExt without copying, and Hasher::update relies on it to bulk-hash an all-normal run in +// one Write. +static_assert(sizeof(TokenIdExt) == 4, "TokenIdExt must be exactly 4 bytes for bulk hashing"); +static_assert(std::is_standard_layout_v, "TokenIdExt must be standard-layout for byte-stream hashing"); +static_assert(alignof(TokenIdExt) == alignof(int32_t), "TokenIdExt must align like int32_t for zero-copy token views"); + +// The digest pool that backs digest-tagged TokenIdExt is an implementation +// detail hidden entirely in tokenIdExt.cpp (anonymous namespace). Only this +// introspection hook is exposed, for tests. +namespace detail +{ + +// Live digest-pool slot count. Compare as a delta vs a captured baseline because +// the pool is a process-global singleton. +[[nodiscard]] size_t digestPoolLiveCount(); + +} // namespace detail + +// Token sequences are held directly as std::vector. TokenIdExt is an +// RAII value type (copy = clone the digest slot, destroy = free it), so a plain +// vector copies/destroys correctly with no wrapper. The "is this sequence +// digest-free" summary that once lived here is now an explicit knownNoDigest / +// text_only flag threaded from the request/model level (see blockRadixTree Hasher +// and KvCache::textOnly). + +} // namespace tensorrt_llm::batch_manager::kv_cache_manager_v2 + +// std::hash specialization for Digest/BlockKey so unordered_map works without a custom hasher. +template <> +struct std::hash +{ + size_t operator()(tensorrt_llm::batch_manager::kv_cache_manager_v2::Digest const& k) const noexcept + { + // First 8 bytes of a SHA-256 digest are already well-distributed. + uint64_t v; + std::memcpy(&v, k.data(), sizeof(v)); + return static_cast(v); + } +}; diff --git a/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/math.h b/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/math.h index bdd1fd503708..c946a6810607 100644 --- a/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/math.h +++ b/cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/utils/math.h @@ -231,9 +231,16 @@ class DynamicBitset explicit DynamicBitset(size_t capacity) : mWords(divUp(capacity, size_t{64}), uint64_t{0}) , mNumSetBits(0) + , mSize(capacity) { } + // Logical bit capacity (as last set by the constructor / resize()). + [[nodiscard]] size_t size() const noexcept + { + return mSize; + } + void set(size_t index) { if (!get(index)) @@ -289,6 +296,7 @@ class DynamicBitset // Grow (zero-filled) or shrink storage to the new word count. mWords.resize(newWords, uint64_t{0}); + mSize = newCapacity; } // Returns true if any bit in [start, end) is set. @@ -329,6 +337,7 @@ class DynamicBitset private: std::vector mWords; size_t mNumSetBits; + size_t mSize; // logical bit capacity (mWords holds divUp(mSize, 64) words) }; // --------------------------------------------------------------------------- diff --git a/cpp/tensorrt_llm/batch_manager/peftCacheManager.cpp b/cpp/tensorrt_llm/batch_manager/peftCacheManager.cpp index 89cc475b82ee..3e51ddc4c518 100644 --- a/cpp/tensorrt_llm/batch_manager/peftCacheManager.cpp +++ b/cpp/tensorrt_llm/batch_manager/peftCacheManager.cpp @@ -1,5 +1,5 @@ /* - * SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. * SPDX-License-Identifier: Apache-2.0 * * Licensed under the Apache License, Version 2.0 (the "License"); @@ -224,6 +224,7 @@ void PeftCacheManager::prefetchLoraWeights(std::string const& modelDir, runtime: TensorPtr weights = runtime::utils::loadNpy(bufferManager, weightsFn, runtime::MemoryType::kCPU); TensorPtr config = runtime::utils::loadNpy(bufferManager, configFn, runtime::MemoryType::kCPU); TLLM_LOG_DEBUG("prefetch lora task %s", tasks[taskId].c_str()); + configureDataType(weights->getDataType()); mHostLoraCache->put(std::stoi(tasks[taskId]), weights, config, true); } TLLM_LOG_DEBUG("%s stop", __PRETTY_FUNCTION__); @@ -263,7 +264,15 @@ void PeftCacheManager::addRequestPeft(std::shared_ptr llmRequest, bo auto optLoraConfig = llmRequest->getLoraConfig(); if (optTaskId || optLoraWeights || optLoraConfig) { - runtime::lora::loraValidateRequestTensors(optTaskId, optLoraWeights, optLoraConfig, mModelConfig, mWorldConfig); + auto const requestDataType = optLoraWeights + ? std::optional{optLoraWeights.value()->getDataType()} + : std::nullopt; + runtime::lora::loraValidateRequestTensors( + optTaskId, optLoraWeights, optLoraConfig, mModelConfig, mWorldConfig, requestDataType); + if (optLoraWeights) + { + configureDataType(optLoraWeights.value()->getDataType()); + } } else { @@ -374,6 +383,39 @@ void PeftCacheManager::addRequestPeft(std::shared_ptr llmRequest, bo TLLM_LOG_DEBUG("%s stop", __PRETTY_FUNCTION__); } +void PeftCacheManager::configureDataType(tensorrt_llm::DataType dataType) +{ + auto const modelDataType = mModelConfig.getDataType(); +#ifdef ENABLE_FP8 + TLLM_CHECK_WITH_INFO(dataType == modelDataType || dataType == tensorrt_llm::DataType::kFP8, + "Unsupported LoRA weights dtype %s for PEFT cache; expected model dtype %s or FP8", + runtime::IBuffer::getDataTypeName(dataType), runtime::IBuffer::getDataTypeName(modelDataType)); +#else + TLLM_CHECK_WITH_INFO(dataType == modelDataType, + "Unsupported LoRA weights dtype %s for PEFT cache; expected model dtype %s", + runtime::IBuffer::getDataTypeName(dataType), runtime::IBuffer::getDataTypeName(modelDataType)); +#endif + + std::lock_guard lock(mDataTypeMutex); + if (mDataType) + { + TLLM_CHECK_WITH_INFO(mDataType.value() == dataType, + "PEFT cache supports one homogeneous LoRA dtype; cache is %s but received %s", + runtime::IBuffer::getDataTypeName(mDataType.value()), runtime::IBuffer::getDataTypeName(dataType)); + return; + } + + mHostLoraCache->setDataType(dataType); + mDeviceLoraCache->setDataType(dataType); + mDataType = dataType; +} + +tensorrt_llm::DataType PeftCacheManager::getDataType() const +{ + std::lock_guard lock(mDataTypeMutex); + return mDataType.value_or(mModelConfig.getDataType()); +} + std::tuple>, BasePeftCacheManager::TaskIdToReqIds> PeftCacheManager::getTaskMaps(RequestVector const& contextRequests, RequestVector const& generationRequests) { diff --git a/cpp/tensorrt_llm/common/attentionOp.cpp b/cpp/tensorrt_llm/common/attentionOp.cpp index bcf8f671939e..7493542a1fc1 100644 --- a/cpp/tensorrt_llm/common/attentionOp.cpp +++ b/cpp/tensorrt_llm/common/attentionOp.cpp @@ -1224,11 +1224,11 @@ int AttentionOp::mlaGeneration( // Set the following parameters if sparseAttention is used. if (useSparseMLA()) { - bool const useDynamicSparseMLA = mRuntimeSparseAttentionParams.sparse_mla_topk_lens != nullptr; + bool const useDynamicSparseMLA = mRuntimeSparseAttentionParams.sparse_attn_kv_lens != nullptr; tllmRunnerParams.mSparseAttention = useDynamicSparseMLA ? SparseType::DynamicTokenSparse : SparseType::StaticTokenSparse; tllmRunnerParams.mSparseTopK = mRuntimeSparseAttentionParams.num_sparse_topk; - tllmRunnerParams.ptrSparseMlaTopKLens = mRuntimeSparseAttentionParams.sparse_mla_topk_lens; + tllmRunnerParams.ptrSparseMlaTopKLens = mRuntimeSparseAttentionParams.sparse_attn_kv_lens; tllmRunnerParams.kvPageIdxPtr = reinterpret_cast( mRuntimeSparseAttentionParams.sparse_attn_indices); if (useDynamicSparseMLA) diff --git a/cpp/tensorrt_llm/executor/cache_transmission/nixl_utils/agentBindings.cpp b/cpp/tensorrt_llm/executor/cache_transmission/nixl_utils/agentBindings.cpp index 040979e7197d..6fd2d523ee71 100644 --- a/cpp/tensorrt_llm/executor/cache_transmission/nixl_utils/agentBindings.cpp +++ b/cpp/tensorrt_llm/executor/cache_transmission/nixl_utils/agentBindings.cpp @@ -83,9 +83,9 @@ NB_MODULE(tensorrt_llm_transfer_agent_binding, m) // Classmethod: batch construction from numpy arrays .def_static( "from_arrays", - [](kvc::MemoryType type, nb::ndarray, nb::c_contig, nb::device::cpu> addrs, - nb::ndarray, nb::c_contig, nb::device::cpu> sizes, - nb::ndarray, nb::c_contig, nb::device::cpu> deviceIds) + [](kvc::MemoryType type, nb::ndarray, nb::c_contig, nb::device::cpu> addrs, + nb::ndarray, nb::c_contig, nb::device::cpu> sizes, + nb::ndarray, nb::c_contig, nb::device::cpu> deviceIds) { size_t n = addrs.shape(0); auto const* a = addrs.data(); @@ -105,8 +105,8 @@ NB_MODULE(tensorrt_llm_transfer_agent_binding, m) // Classmethod: batch construction with uniform device_id (avoids np.full allocation) .def_static( "from_arrays_uniform_device", - [](kvc::MemoryType type, nb::ndarray, nb::c_contig, nb::device::cpu> addrs, - nb::ndarray, nb::c_contig, nb::device::cpu> sizes, uint32_t deviceId) + [](kvc::MemoryType type, nb::ndarray, nb::c_contig, nb::device::cpu> addrs, + nb::ndarray, nb::c_contig, nb::device::cpu> sizes, uint32_t deviceId) { size_t n = addrs.shape(0); auto const* a = addrs.data(); diff --git a/cpp/tensorrt_llm/kernels/CMakeLists.txt b/cpp/tensorrt_llm/kernels/CMakeLists.txt index 6cb73708b01c..c6c6fe4a292a 100644 --- a/cpp/tensorrt_llm/kernels/CMakeLists.txt +++ b/cpp/tensorrt_llm/kernels/CMakeLists.txt @@ -1,5 +1,5 @@ # -# SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & +# SPDX-FileCopyrightText: Copyright (c) 1993-2026 NVIDIA CORPORATION & # AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 # # Licensed under the Apache License, Version 2.0 (the "License"); you may not @@ -30,6 +30,8 @@ add_subdirectory(dsv3MinLatencyKernels) add_subdirectory(causalConv1d) add_subdirectory(fusedGatedRMSNormQuant) add_subdirectory(mamba2MTPSSMCache) +add_subdirectory(kdaDecode) +add_subdirectory(kimiK3AttnRes) add_subdirectory(mhcKernels) add_subdirectory(compressorKernels) @@ -57,11 +59,15 @@ list(FILTER SRC_CU EXCLUDE REGEX "userbuffers/.*") list(FILTER SRC_CU EXCLUDE REGEX "fusedLayernormKernels/.*") list(FILTER SRC_CU EXCLUDE REGEX "fusedGatedRMSNormQuant/.*") list(FILTER SRC_CU EXCLUDE REGEX "mamba2MTPSSMCache/.*") +list(FILTER SRC_CPP EXCLUDE REGEX "kdaDecode/.*") +list(FILTER SRC_CU EXCLUDE REGEX "kdaDecode/.*") +list(FILTER SRC_CPP EXCLUDE REGEX "kimiK3AttnRes/.*") +list(FILTER SRC_CU EXCLUDE REGEX "kimiK3AttnRes/.*") list(FILTER SRC_CPP EXCLUDE REGEX "mhcKernels/.*") list(FILTER SRC_CU EXCLUDE REGEX "mhcKernels/.*") list(FILTER SRC_CPP EXCLUDE REGEX "compressorKernels/.*") list(FILTER SRC_CU EXCLUDE REGEX "compressorKernels/.*") -# Marlin is built as its own Hopper-only OBJECT library below. +# Marlin is built as its own architecture-scoped OBJECT library below. list(FILTER SRC_CPP EXCLUDE REGEX "marlin/.*") list(FILTER SRC_CU EXCLUDE REGEX "marlin/.*") @@ -82,8 +88,8 @@ if(FAST_BUILD) STATUS "FAST_BUILD enabled for kernels: using -O1 for CUDA compilation") endif() -# Marlin NVFP4: Ada (SM89, e.g. L40S) and Hopper (SM90) OBJECT library. Pinned -# to sm_89/sm_90 so the global CMAKE_CUDA_ARCHITECTURES doesn't propagate. +# Marlin NVFP4: build Ada/Hopper kernels and the dense W4A16/repack kernels used +# on SM12x. Unsupported device passes emit empty kernel stubs. file(GLOB_RECURSE MARLIN_SRC "marlin/*.cu" "marlin/*.cpp") if(MARLIN_SRC) add_library(marlin_src OBJECT ${MARLIN_SRC}) @@ -95,7 +101,7 @@ if(MARLIN_SRC) $ ) target_link_libraries(marlin_src PRIVATE trtllm_gen_fmha_interface) - set_cuda_architectures(marlin_src 89 90) + set_cuda_architectures(marlin_src 89 90 120f) endif() add_library( diff --git a/cpp/tensorrt_llm/kernels/IndexerKCacheGather.h b/cpp/tensorrt_llm/kernels/IndexerKCacheGather.h index 60422dea5910..c0a461e3c72d 100644 --- a/cpp/tensorrt_llm/kernels/IndexerKCacheGather.h +++ b/cpp/tensorrt_llm/kernels/IndexerKCacheGather.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2022-2025, NVIDIA CORPORATION. All rights reserved. + * Copyright (c) 2022-2026, NVIDIA CORPORATION. All rights reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/cpp/tensorrt_llm/kernels/IndexerKCacheScatter.h b/cpp/tensorrt_llm/kernels/IndexerKCacheScatter.h index 9e316c0b4ebe..a62da181cbb5 100644 --- a/cpp/tensorrt_llm/kernels/IndexerKCacheScatter.h +++ b/cpp/tensorrt_llm/kernels/IndexerKCacheScatter.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2022-2025, NVIDIA CORPORATION. All rights reserved. + * Copyright (c) 2022-2026, NVIDIA CORPORATION. All rights reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/cpp/tensorrt_llm/kernels/communicationKernels/moeAlltoAllKernels.cu b/cpp/tensorrt_llm/kernels/communicationKernels/moeAlltoAllKernels.cu index 74e40dbb2b81..8559653b6a5b 100644 --- a/cpp/tensorrt_llm/kernels/communicationKernels/moeAlltoAllKernels.cu +++ b/cpp/tensorrt_llm/kernels/communicationKernels/moeAlltoAllKernels.cu @@ -21,8 +21,10 @@ #include "tensorrt_llm/common/vec_dtypes.cuh" #include "tensorrt_llm/kernels/communicationKernels/moeAlltoAllKernels.h" #include "tensorrt_llm/kernels/quantization.cuh" +#include #include #include +#include #include TRTLLM_NAMESPACE_BEGIN @@ -32,6 +34,54 @@ namespace kernels::moe_comm using tensorrt_llm::common::launchWithPdlWhenEnabled; +// Resolve the completion-flag wait budget; see the header. Seconds are converted at +// an assumed 2 GHz SM clock, so they are nominal rather than wall-clock. +int64_t moeA2AGetTimeoutCycles(bool is_warmup) +{ + static constexpr int64_t kAssumedClockHz = 2000ll * 1000ll * 1000ll; + static constexpr int64_t kDefaultTimeoutSec = 300; + // Warmup contains one-time per-rank costs (JIT compilation, autotuning, module + // loading) that can run for minutes and are not synchronized against this + // collective, so it needs a larger budget than steady state. + static constexpr int64_t kDefaultWarmupTimeoutSec = 1800; + + // Reject trailing garbage, out-of-range values and anything that would overflow + // the cycle multiplication. + auto const readEnv = [](char const* name, int64_t fallback) -> int64_t + { + static constexpr int64_t kMaxSec = 24 * 60 * 60; // 1 day; * 2e9 stays well inside int64 + char const* v = std::getenv(name); + if (v == nullptr || *v == '\0') + { + return fallback; + } + errno = 0; + char* end = nullptr; + int64_t parsed = std::strtoll(v, &end, 10); + bool const trailingGarbage = (end == v) || (*end != '\0'); + if (trailingGarbage || errno == ERANGE || parsed <= 0 || parsed > kMaxSec) + { + TLLM_LOG_WARNING("Ignoring invalid %s=\"%s\" (expected 1..%ld seconds); using %ld s", name, v, + static_cast(kMaxSec), static_cast(fallback)); + return fallback; + } + return parsed; + }; + + static int64_t const sSteadySec = readEnv("TRTLLM_MOE_A2A_TIMEOUT_SEC", kDefaultTimeoutSec); + static int64_t const sWarmupSec = readEnv("TRTLLM_MOE_A2A_WARMUP_TIMEOUT_SEC", kDefaultWarmupTimeoutSec); + static bool const sLogged = []() + { + TLLM_LOG_INFO( + "MoE all-to-all completion-flag budget: steady=%ld s, warmup=%ld s (nominal, at an " + "assumed 2 GHz clock64 rate)", + static_cast(sSteadySec), static_cast(sWarmupSec)); + return true; + }(); + (void) sLogged; + return (is_warmup ? sWarmupSec : sSteadySec) * kAssumedClockHz; +} + #define ENABLE_DEBUG_PRINT 0 #define DISABLE_SYNC_FOR_PROFILING 0 @@ -161,10 +211,10 @@ using tensorrt_llm::common::launchWithPdlWhenEnabled; } #if DISABLE_TIMEOUT -#define check_timeout(s) false +#define check_timeout(s, budget) false #else -// 300 * 2000 MHz - should be high enough on any GPU but will prevent a hang -#define check_timeout(s) ((clock64() - (s)) > (300ll * 2000ll * 1000ll * 1000ll)) +// `budget` is in clock64() cycles, resolved on the host by moeA2AGetTimeoutCycles(). +#define check_timeout(s, budget) ((clock64() - (s)) > (budget)) #endif // ============================================================================ @@ -618,7 +668,7 @@ __global__ void moeA2ADispatchKernel(int32_t const* token_selected_experts, // [ rank_id, peer_rank, flag_value, expected_value, flag_ptr); #endif flag_set = flag_value == expected_value; - } while (!flag_set && !check_timeout(s)); + } while (!flag_set && !check_timeout(s, ptrs.timeout_cycles)); if (__builtin_expect(!flag_set, 0)) { @@ -659,6 +709,7 @@ void moe_a2a_dispatch_launch(MoeA2ADispatchParams const& params) // Prepare kernel pointers struct DispatchKernelPointers kernel_ptrs = {}; + kernel_ptrs.timeout_cycles = params.timeout_cycles; // Fill source data pointers and payload sizes for (int i = 0; i < params.num_payloads; i++) @@ -1258,7 +1309,7 @@ __global__ void moeA2ACombineKernel( rank_id, peer_rank, flag_value, expected_value, flag_ptr); #endif flag_set = flag_value == expected_value; - } while (!flag_set && !check_timeout(s)); + } while (!flag_set && !check_timeout(s, ptrs.timeout_cycles)); if (__builtin_expect(!flag_set, 0)) { @@ -1365,6 +1416,7 @@ void moe_a2a_combine_launch(MoeA2ACombineParams const& params) // Prepare kernel pointers struct for combine CombineKernelPointers kernel_ptrs = {}; // Zero-initialize + kernel_ptrs.timeout_cycles = params.timeout_cycles; // Set output data pointer in src_data_ptrs[0] kernel_ptrs.src_data_ptrs[0] = params.output_data; diff --git a/cpp/tensorrt_llm/kernels/communicationKernels/moeAlltoAllKernels.h b/cpp/tensorrt_llm/kernels/communicationKernels/moeAlltoAllKernels.h index 5184878ffc51..0b25e94a9e26 100644 --- a/cpp/tensorrt_llm/kernels/communicationKernels/moeAlltoAllKernels.h +++ b/cpp/tensorrt_llm/kernels/communicationKernels/moeAlltoAllKernels.h @@ -33,6 +33,9 @@ static constexpr int kRankMaskWords = 2; // uint64 words to hold the active-rank // (kRankMaskWords * 64 must be >= kMaxRanks) static_assert(kRankMaskWords * 64 >= kMaxRanks, "active_rank_mask too small for kMaxRanks"); +// Default completion-flag wait budget: 300 s at an assumed 2 GHz clock64 rate. +static constexpr int64_t kDefaultTimeoutCycles = 300ll * 2000ll * 1000ll * 1000ll; + // Describes a single payload type to be communicated struct PayloadDescriptor { @@ -73,6 +76,9 @@ struct DispatchKernelPointers // Word 0 covers ranks 0..63; word 1 covers ranks 64..127. The masked kernel // rejects inactive route targets and skips their peer counters, stats, and flags. uint64_t active_rank_mask[kRankMaskWords]; + + // Completion-flag wait budget in clock64() cycles; see moeA2AGetTimeoutCycles(). + int64_t timeout_cycles{kDefaultTimeoutCycles}; }; // Combine kernel pointers - non-const output in src_data_ptrs[0], const recv buffers @@ -94,6 +100,9 @@ struct CombineKernelPointers // Active-rank bitmask: see DispatchKernelPointers::active_rank_mask. Combine skips // completion flag writes/waits to/from inactive peers. uint64_t active_rank_mask[kRankMaskWords]; + + // Completion-flag wait budget in clock64() cycles; see moeA2AGetTimeoutCycles(). + int64_t timeout_cycles{kDefaultTimeoutCycles}; }; // Dispatch phase parameters @@ -147,10 +156,22 @@ struct MoeA2ADispatchParams // CUDA graph replay until generation-scoped invalidation and recapture are available. uint64_t active_rank_mask[kRankMaskWords] = {~uint64_t{0}, ~uint64_t{0}}; + // Completion-flag wait budget in clock64() cycles; see moeA2AGetTimeoutCycles(). + int64_t timeout_cycles{kDefaultTimeoutCycles}; + // CUDA stream cudaStream_t stream; }; +// Resolve the completion-flag wait budget, in clock64() cycles. +// +// No collective separates a rank's first-touch JIT/autotune work from its dispatch +// launch, so this device-side budget is in effect a deadline on the slowest peer's +// host-side progress. Warmup therefore uses a larger budget than steady state. +// Overridable via TRTLLM_MOE_A2A_TIMEOUT_SEC / TRTLLM_MOE_A2A_WARMUP_TIMEOUT_SEC. +// See nvbugs/6482566. +int64_t moeA2AGetTimeoutCycles(bool is_warmup); + // Dispatch kernels void moe_a2a_dispatch_launch(MoeA2ADispatchParams const& params); // Prepare for dispatch: zero send_counters, local_token_counter and increment flag_val @@ -202,6 +223,9 @@ struct MoeA2ACombineParams // CUDA graph replay until generation-scoped invalidation and recapture are available. uint64_t active_rank_mask[kRankMaskWords] = {~uint64_t{0}, ~uint64_t{0}}; + // Completion-flag wait budget in clock64() cycles; see moeA2AGetTimeoutCycles(). + int64_t timeout_cycles{kDefaultTimeoutCycles}; + // CUDA stream cudaStream_t stream; }; diff --git a/cpp/tensorrt_llm/kernels/cuda_graph_grouped_gemm.cu b/cpp/tensorrt_llm/kernels/cuda_graph_grouped_gemm.cu index 8af43b2b4914..7fd7979ee4bd 100644 --- a/cpp/tensorrt_llm/kernels/cuda_graph_grouped_gemm.cu +++ b/cpp/tensorrt_llm/kernels/cuda_graph_grouped_gemm.cu @@ -1,5 +1,5 @@ /* - * Copyright (c) 2024, NVIDIA CORPORATION. All rights reserved. + * Copyright (c) 2024-2026, NVIDIA CORPORATION. All rights reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -15,11 +15,14 @@ */ #include "cuda_graph_grouped_gemm.h" +#include "groupGemm.h" #include "tensorrt_llm/common/assert.h" #include "tensorrt_llm/common/cudaUtils.h" #include +#include + #include "cutlass/cutlass.h" #include "cutlass/gemm/device/gemm_grouped.h" #include "cutlass/gemm/kernel/default_gemm_grouped.h" @@ -27,10 +30,57 @@ #include "tensorrt_llm/cutlass_extensions/include/cutlass_extensions/gemm/device/splitk_gemm_grouped.h" #include "tensorrt_llm/cutlass_extensions/include/cutlass_extensions/gemm/kernel/default_splitk_gemm_grouped.h" +#ifdef ENABLE_FP8 +#include "cute/tensor.hpp" +#include "cutlass/epilogue/collective/collective_builder.hpp" +#include "cutlass/gemm/collective/collective_builder.hpp" +#include "cutlass/gemm/device/gemm_universal_adapter.h" +#include "cutlass/gemm/dispatch_policy.hpp" +#include "cutlass/gemm/group_array_problem_shape.hpp" +#include "cutlass/gemm/kernel/gemm_universal.hpp" +#include "cutlass/util/packed_stride.hpp" +#endif // ENABLE_FP8 + TRTLLM_NAMESPACE_BEGIN namespace kernels { +#ifdef ENABLE_FP8 +#if defined(CUTLASS_ARCH_MMA_MODIFIABLE_TMA_SM90_SUPPORTED) +namespace +{ + +void checkFp8CudaGraphAlignment( + cutlass::gemm::GemmCoord const* hostMaxProblemSizesPtr, int problemCount, int minKN, char const* kernelName) +{ + static int const smVersion = tensorrt_llm::common::getSMVersion(); + // CUTLASS also exposes this kernel level on SM120/SM121; enable those after validation. + TLLM_CHECK_WITH_INFO( + smVersion == 90, "%s requires Hopper (SM90), but the current device is SM%d", kernelName, smVersion); + + TLLM_CHECK_WITH_INFO(minKN >= kFp8TmaAlignment && minKN % kFp8TmaAlignment == 0, + "%s requires active LoRA ranks to be multiples of %d elements for 128-bit TMA alignment. " + "The minimum K/N dimension is %d.", + kernelName, kFp8TmaAlignment, minKN); + + for (int problemIdx = 0; problemIdx < problemCount; ++problemIdx) + { + auto const& problem = hostMaxProblemSizesPtr[problemIdx]; + if (problem.m() == 0 || problem.n() == 0 || problem.k() == 0) + { + continue; + } + + TLLM_CHECK_WITH_INFO(problem.n() % kFp8TmaAlignment == 0 && problem.k() % kFp8TmaAlignment == 0, + "%s requires GEMM N and K dimensions to be multiples of %d elements for 128-bit TMA alignment. " + "Max problem %d has M=%d, N=%d, K=%d.", + kernelName, kFp8TmaAlignment, problemIdx, problem.m(), problem.n(), problem.k()); + } +} + +} // namespace +#endif // CUTLASS_ARCH_MMA_MODIFIABLE_TMA_SM90_SUPPORTED +#endif // ENABLE_FP8 /** * Template for CUDA Graph compatible grouped GEMM that directly uses GPU tensors @@ -140,10 +190,157 @@ void cudaGraphGroupedGemmType(cutlass::gemm::GemmCoord* problemSizesPtr, int pro } } +#ifdef ENABLE_FP8 +#if defined(CUTLASS_ARCH_MMA_MODIFIABLE_TMA_SM90_SUPPORTED) + +// ==================================================================== +// FP8 CUDA-graph-compatible grouped GEMM using CUTLASS 3.x. +// +// The CUDA graph variant receives live problem sizes, pointers, and leading +// dimensions already on the GPU and reuses their compatible storage directly. +// ==================================================================== + +void fp8CudaGraphGroupedGemm(cutlass::gemm::GemmCoord* problemSizesPtr, int problemCount, void** ptrAGpu, + void** ptrBGpu, void** ptrCGpu, void** ptrDGpu, int64_t* ldaGpu, int64_t* ldbGpu, int64_t* ldcGpu, int64_t* lddGpu, + cudaStream_t stream) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + using namespace cute; + + using ElementA = cutlass::float_e4m3_t; + using ElementB = cutlass::float_e4m3_t; + using ElementC = cutlass::float_e4m3_t; + using ElementD = cutlass::float_e4m3_t; + using ElementAccumulator = float; + + using LayoutA = cutlass::layout::RowMajor; + using LayoutB = cutlass::layout::ColumnMajor; + using LayoutC = cutlass::layout::RowMajor; + using LayoutD = cutlass::layout::RowMajor; + + static constexpr int kAlignmentA = 128 / cutlass::sizeof_bits::value; + static constexpr int kAlignmentB = 128 / cutlass::sizeof_bits::value; + static constexpr int kAlignmentC = 128 / cutlass::sizeof_bits::value; + static constexpr int kAlignmentD = 128 / cutlass::sizeof_bits::value; + + using ArchTag = cutlass::arch::Sm90; + using OperatorClass = cutlass::arch::OpClassTensorOp; + + using TileShape = Shape<_128, _128, _128>; + using ClusterShape = Shape<_1, _2, _1>; + + using KernelSchedule = cutlass::gemm::KernelPtrArrayTmaWarpSpecializedCooperativeFP8FastAccum; + using EpilogueSchedule = cutlass::epilogue::PtrArrayTmaWarpSpecializedCooperative; + + using ProblemShape = cutlass::gemm::GroupProblemShape>; + + using CollectiveEpilogue = + typename cutlass::epilogue::collective::CollectiveBuilder::CollectiveOp; + + using CollectiveMainloop = typename cutlass::gemm::collective::CollectiveBuilder( + sizeof(typename CollectiveEpilogue::SharedStorage))>, + KernelSchedule>::CollectiveOp; + + using GemmKernel = cutlass::gemm::kernel::GemmUniversal; + using Gemm = cutlass::gemm::device::GemmUniversalAdapter; + + using StrideA = typename GemmKernel::InternalStrideA; + using StrideB = typename GemmKernel::InternalStrideB; + using StrideC = typename GemmKernel::InternalStrideC; + using StrideD = typename GemmKernel::InternalStrideD; + + using UnderlyingProblemShape = typename ProblemShape::UnderlyingProblemShape; + using PackedStride = cute::Stride; + static_assert(std::is_same_v>); + static_assert(sizeof(UnderlyingProblemShape) == sizeof(cutlass::gemm::GemmCoord)); + static_assert(alignof(UnderlyingProblemShape) == alignof(cutlass::gemm::GemmCoord)); + static_assert(std::is_same_v); + static_assert(std::is_same_v); + static_assert(std::is_same_v); + static_assert(std::is_same_v); + static_assert(sizeof(PackedStride) == sizeof(int64_t)); + static_assert(alignof(PackedStride) == alignof(int64_t)); + + // The fused LoRA parameter kernel already stores problem shapes as contiguous M/N/K int32 values and each + // packed stride's single dynamic component as an int64 leading dimension. + auto* problemShapes = reinterpret_cast(problemSizesPtr); + auto* ptrA = const_cast(reinterpret_cast(ptrAGpu)); + auto* ptrB = const_cast(reinterpret_cast(ptrBGpu)); + auto* ptrC = const_cast(reinterpret_cast(ptrCGpu)); + auto* ptrD = reinterpret_cast(ptrDGpu); + auto* strideA = reinterpret_cast(ldaGpu); + auto* strideB = reinterpret_cast(ldbGpu); + auto* strideC = reinterpret_cast(ldcGpu); + auto* strideD = reinterpret_cast(lddGpu); + + cutlass::KernelHardwareInfo hwInfo; + hwInfo.device_id = 0; + cudaGetDevice(&hwInfo.device_id); + hwInfo.sm_count = cutlass::KernelHardwareInfo::query_device_multiprocessor_count(hwInfo.device_id); + + typename Gemm::Arguments arguments{cutlass::gemm::GemmUniversalMode::kGrouped, + {problemCount, problemShapes, nullptr}, {ptrA, strideA, ptrB, strideB}, + {{1.0f, 0.0f}, ptrC, strideC, ptrD, strideD}, hwInfo}; + + Gemm gemm; + + size_t const requiredWorkspace = Gemm::get_workspace_size(arguments); + at::Tensor workspace; + void* gemmWorkspace = nullptr; + if (requiredWorkspace > 0) + { + auto const tensorOpts = at::TensorOptions().dtype(at::kByte).device(at::kCUDA); + workspace = at::empty({static_cast(requiredWorkspace)}, tensorOpts); + gemmWorkspace = workspace.data_ptr(); + } + + cutlass::Status status = gemm.can_implement(arguments); + TLLM_CHECK_WITH_INFO(status == cutlass::Status::kSuccess, "FP8 CUDA graph grouped GEMM can_implement failed: %s", + cutlass::cutlassGetStatusString(status)); + + status = gemm.initialize(arguments, gemmWorkspace, stream); + TLLM_CHECK_WITH_INFO(status == cutlass::Status::kSuccess, "FP8 CUDA graph grouped GEMM initialize failed: %s", + cutlass::cutlassGetStatusString(status)); + + status = gemm.run(stream); + sync_check_cuda_error(stream); + TLLM_CHECK_WITH_INFO(status == cutlass::Status::kSuccess, "FP8 CUDA graph grouped GEMM run failed: %s", + cutlass::cutlassGetStatusString(status)); + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +#endif // CUTLASS_ARCH_MMA_MODIFIABLE_TMA_SM90_SUPPORTED +#endif // ENABLE_FP8 + void cudaGraphGroupedGemm(cutlass::gemm::GemmCoord* problemSizesPtr, int problemCount, void** ptrAGpu, void** ptrBGpu, void** ptrCGpu, void** ptrDGpu, int64_t* ldaGpu, int64_t* ldbGpu, int64_t* ldcGpu, int64_t* lddGpu, bool isLoraIn, tensorrt_llm::DataType dataType, int minKN, cutlass::gemm::GemmCoord* hostMaxProblemSizesPtr, cudaStream_t stream) { +#ifdef ENABLE_FP8 +#if defined(CUTLASS_ARCH_MMA_MODIFIABLE_TMA_SM90_SUPPORTED) + if (dataType == tensorrt_llm::DataType::kFP8) + { + checkFp8CudaGraphAlignment(hostMaxProblemSizesPtr, problemCount, minKN, "FP8 CUDA graph grouped GEMM"); + fp8CudaGraphGroupedGemm( + problemSizesPtr, problemCount, ptrAGpu, ptrBGpu, ptrCGpu, ptrDGpu, ldaGpu, ldbGpu, ldcGpu, lddGpu, stream); + return; + } +#else + if (dataType == tensorrt_llm::DataType::kFP8) + { + TLLM_CHECK_WITH_INFO(false, + "FP8 CUDA graph grouped GEMM requires CUTLASS modifiable TMA support (CUDA 12.3+ and Hopper SM90 " + "kernels)."); + } +#endif // CUTLASS_ARCH_MMA_MODIFIABLE_TMA_SM90_SUPPORTED +#endif // ENABLE_FP8 + if (isLoraIn) { if (minKN >= 8) @@ -312,6 +509,26 @@ void cudaGraphSplitKGroupedGemm(cutlass::gemm::GemmCoord* problemSizesPtr, int p bool isLoraIn, tensorrt_llm::DataType dataType, int splitKSlices, int minKN, cutlass::gemm::GemmCoord* hostMaxProblemSizesPtr, int64_t* splitKOffsetsGpu, cudaStream_t stream) { +#ifdef ENABLE_FP8 +#if defined(CUTLASS_ARCH_MMA_MODIFIABLE_TMA_SM90_SUPPORTED) + if (dataType == tensorrt_llm::DataType::kFP8) + { + // Reuse the non-split-K fp8 path; CUTLASS 3.x cooperative schedule handles large-K efficiently. + checkFp8CudaGraphAlignment(hostMaxProblemSizesPtr, problemCount, minKN, "FP8 CUDA graph split-K grouped GEMM"); + fp8CudaGraphGroupedGemm( + problemSizesPtr, problemCount, ptrAGpu, ptrBGpu, ptrCGpu, ptrDGpu, ldaGpu, ldbGpu, ldcGpu, lddGpu, stream); + return; + } +#else + if (dataType == tensorrt_llm::DataType::kFP8) + { + TLLM_CHECK_WITH_INFO(false, + "FP8 CUDA graph split-K grouped GEMM requires CUTLASS modifiable TMA support (CUDA 12.3+ and Hopper SM90 " + "kernels)."); + } +#endif // CUTLASS_ARCH_MMA_MODIFIABLE_TMA_SM90_SUPPORTED +#endif // ENABLE_FP8 + if (isLoraIn) { if (minKN >= 8) diff --git a/cpp/tensorrt_llm/kernels/cutlass_kernels/include/moe_kernels.h b/cpp/tensorrt_llm/kernels/cutlass_kernels/include/moe_kernels.h index ab7ed876257d..4b307c86ddac 100644 --- a/cpp/tensorrt_llm/kernels/cutlass_kernels/include/moe_kernels.h +++ b/cpp/tensorrt_llm/kernels/cutlass_kernels/include/moe_kernels.h @@ -993,6 +993,7 @@ class CutlassMoeFCRunner : public CutlassMoeFCRunnerInterface float* permuted_token_final_scales_{}; int64_t* expert_first_token_offset_{}; + int64_t* gemm_expert_first_token_offset_{}; void* glu_inter_result_{}; void* fc2_result_{}; diff --git a/cpp/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_kernels.cu b/cpp/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_kernels.cu index 8bed9c16b58e..b259c613f30e 100644 --- a/cpp/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_kernels.cu +++ b/cpp/tensorrt_llm/kernels/cutlass_kernels/moe_gemm/moe_kernels.cu @@ -1686,6 +1686,43 @@ void expandInputRowsKernelLauncher(InputActivationsType const* unpermuted_input, num_experts_per_node, reinterpret_cast(prequant_scales)); } +template +__global__ void padEmptyFp8BlockScaleMoeInputKernel(T* permuted_input, int64_t const* expert_first_token_offset, + int64_t* gemm_expert_first_token_offset, int num_experts_per_node, int64_t hidden_size) +{ + for (int index = threadIdx.x; index <= num_experts_per_node; index += blockDim.x) + { + gemm_expert_first_token_offset[index] = expert_first_token_offset[index]; + } + + if (expert_first_token_offset[num_experts_per_node] != 0) + { + return; + } + + for (int64_t index = threadIdx.x; index < hidden_size; index += blockDim.x) + { + permuted_input[index] = T(0); + } + __syncthreads(); + + // Assign the zero row to the final local expert. The all-gather/reduce-scatter + // finalizer ignores it because that expert was not selected by any real token. + if (threadIdx.x == 0) + { + gemm_expert_first_token_offset[num_experts_per_node] = 1; + } +} + +template +void padEmptyFp8BlockScaleMoeInput(T* permuted_input, int64_t const* expert_first_token_offset, + int64_t* gemm_expert_first_token_offset, int num_experts_per_node, int64_t hidden_size, cudaStream_t stream) +{ + constexpr int threads = 256; + padEmptyFp8BlockScaleMoeInputKernel<<<1, threads, 0, stream>>>( + permuted_input, expert_first_token_offset, gemm_expert_first_token_offset, num_experts_per_node, hidden_size); +} + #define INSTANTIATE_EXPAND_INPUT_ROWS(InputActivationsType, ExpandedActivationsType) \ template void expandInputRowsKernelLauncher( \ InputActivationsType const* unpermuted_input, ExpandedActivationsType* permuted_output, \ @@ -2985,6 +3022,8 @@ CutlassMoeFCRunner:: size_t const permuted_data_size = permuted_elems * dtype_size; size_t const expert_first_token_offset_size = (num_experts_per_node + 1) * sizeof(int64_t); + size_t const gemm_expert_first_token_offset_size + = use_deepseek_fp8_block_scale ? expert_first_token_offset_size : 0; size_t const permuted_token_final_scales_size = mayHaveFinalizeFused() ? num_moe_inputs * sizeof(float) : 0; size_t const glu_inter_size = glu_inter_elems * gemm_output_dtype; // May be an intermediate type for quantization size_t const fc1_result_size = interbuf_elems * dtype_size; // Activation quantizes so back to dtype_size @@ -3084,6 +3123,7 @@ CutlassMoeFCRunner:: ADD(blocked_expert_counts_cumsum); ADD(blocked_row_to_unpermuted_row); ADD(expert_first_token_offset); + ADD(gemm_expert_first_token_offset); ADD(permuted_token_final_scales); ADD(overlapped_gemm1_gemm2_inputs); ADD(overlapped_gemm1_gemm2_outputs); @@ -3145,6 +3185,7 @@ void CutlassMoeFCRunner 1 && !enable_alltoall) + { + TLLM_CHECK(gemm_expert_first_token_offset_ != nullptr); + padEmptyFp8BlockScaleMoeInput(gemm1_input_expand, expert_first_token_offset_, + gemm_expert_first_token_offset_, num_experts_per_node, hidden_size, stream); + gemm_expert_first_token_offset = gemm_expert_first_token_offset_; + } auto const* gemm1_input = gemm1_input_expand; sync_check_cuda_error(stream); @@ -4349,12 +4405,12 @@ void CutlassMoeFCRunner(smoothed_act_) : fc1_result_; Self::gemm1(moe_gemm_runner_, blockscale_gemm_runner, gemm1_input, gemm1_output, glu_inter_result_, - expert_first_token_offset_, gemm1_tma_ws_input, fc1_expert_weights, fc1_expert_biases, num_valid_tokens_ptr, - fc1_int_scales, fc1_fp8_dequant, use_wfp4afp8 ? fc2_wfp4afp8_quant_scale : fc2_fp8_quant, - fc1_fp4_act_scale_, fc2_fp4_act_scale_, quant_params, num_rows, expanded_num_rows, - expected_tokens_per_expert, hidden_size, inter_size, num_experts_per_node, fc1_activation_type, - alpha_scale_ptr_array_fc1_, !use_lora, stream, *gemm1_config_, false, nullptr, nullptr, - fc2_prequant_scale_ptr); + gemm_expert_first_token_offset, gemm1_tma_ws_input, fc1_expert_weights, fc1_expert_biases, + num_valid_tokens_ptr, fc1_int_scales, fc1_fp8_dequant, + use_wfp4afp8 ? fc2_wfp4afp8_quant_scale : fc2_fp8_quant, fc1_fp4_act_scale_, fc2_fp4_act_scale_, + quant_params, num_rows, expanded_num_rows, expected_tokens_per_expert, hidden_size, inter_size, + num_experts_per_node, fc1_activation_type, alpha_scale_ptr_array_fc1_, !use_lora, stream, *gemm1_config_, + false, nullptr, nullptr, fc2_prequant_scale_ptr); sync_check_cuda_error(stream); if (use_lora) @@ -4371,11 +4427,11 @@ void CutlassMoeFCRunner 1. if (mFixedParams.useTllmGenSparseAttention) { - bool const useDynamicSparseMLA = runnerParams.sparse_params.sparse_mla_topk_lens != nullptr; + bool const useDynamicSparseMLA = runnerParams.sparse_params.sparse_attn_kv_lens != nullptr; tllmRunnerParams.mSparseAttention = useDynamicSparseMLA ? SparseType::DynamicTokenSparse : SparseType::StaticTokenSparse; tllmRunnerParams.mSparseTopK = runnerParams.sparse_params.num_sparse_topk; - tllmRunnerParams.ptrSparseMlaTopKLens = runnerParams.sparse_params.sparse_mla_topk_lens; + tllmRunnerParams.ptrSparseMlaTopKLens = runnerParams.sparse_params.sparse_attn_kv_lens; tllmRunnerParams.mKernelType = FmhaKernelType::Generation; tllmRunnerParams.mUseGenKernelForPrefill = true; tllmRunnerParams.mMaskType = TrtllmGenAttentionMaskType::Causal; diff --git a/cpp/tensorrt_llm/kernels/fusedQKNormRopeKernel.cu b/cpp/tensorrt_llm/kernels/fusedQKNormRopeKernel.cu index 6e26dfce65bf..2eaf4ba15fa6 100644 --- a/cpp/tensorrt_llm/kernels/fusedQKNormRopeKernel.cu +++ b/cpp/tensorrt_llm/kernels/fusedQKNormRopeKernel.cu @@ -24,6 +24,7 @@ #include #include #include +#include TRTLLM_NAMESPACE_BEGIN @@ -56,15 +57,50 @@ __device__ __forceinline__ float selectMRopePosId(int const* position_ids, int t return static_cast(position_ids[sec * num_tokens + tokenIdx]); } -// Perform per-head QK Norm and RoPE in a single kernel. +// Store a per-thread run of `numElemsPerThread` float elements, converting to +// OutT. The FP8 path saturates to +/-448, whereas torch's .to(float8_e4m3fn) +// produces NaN for out-of-range values. +template +__device__ __forceinline__ void storeHeadElements( + OutT* out, int offsetThread, float const (&elements)[numElemsPerThread]) +{ + using vec_T = typename tensorrt_llm::common::packed_as::type; + if constexpr (std::is_same_v) + { + vec_T vec; +#pragma unroll + for (int i = 0; i < vecSize; i++) + { + __nv_bfloat162 vals = __float22bfloat162_rn(make_float2(elements[2 * i], elements[2 * i + 1])); + reinterpret_cast<__nv_bfloat162&>(*(reinterpret_cast(&vec) + i)) = vals; + } + *reinterpret_cast(&out[offsetThread]) = vec; + } + else // __nv_fp8_e4m3 + { + static_assert(numElemsPerThread % 2 == 0, "FP8 store expects an even element count per thread"); +#pragma unroll + for (int i = 0; i < numElemsPerThread; i += 2) + { + __nv_fp8x2_e4m3 packed(make_float2(elements[i], elements[i + 1])); + reinterpret_cast<__nv_fp8x2_storage_t*>(&out[offsetThread])[i / 2] = packed.__x; + } + } +} + +// Perform per-head QK Norm and RoPE in a single kernel, reading a BF16 input and +// writing to a (possibly different-dtype) output buffer. // head_dim: the dimension of each head // interleave: interleave=!is_neox. -template +// OutT: output element type (__nv_bfloat16 or __nv_fp8_e4m3). +template __global__ void fusedQKNormRopeKernel( - __nv_bfloat16* qkv, // Combined QKV tensor [num_tokens, (num_heads_q+num_heads_k+num_heads_v)*head_dim] + __nv_bfloat16 const* qkv_in, // Combined QKV input [num_tokens, (num_heads_q+num_heads_k+num_heads_v)*head_dim] + OutT* qkv_out, // Output buffer, same layout as qkv_in int const num_heads_q, // Number of query heads int const num_heads_k, // Number of key heads int const num_heads_v, // Number of value heads + bool const process_v, // Whether to copy-cast V heads into qkv_out int const rotary_dim, // Dimension for RoPE float const eps, // Epsilon for RMS normalization __nv_bfloat16 const* q_weight, // RMSNorm weights for query @@ -95,17 +131,35 @@ __global__ void fusedQKNormRopeKernel( // Total number of attention heads (Q and K) int const total_qk_heads = num_heads_q + num_heads_k; + int const total_proc_heads = total_qk_heads + (process_v ? num_heads_v : 0); - // Determine which token and head type (Q or K) this warp processes - int const tokenIdx = globalWarpIdx / total_qk_heads; - int const localHeadIdx = globalWarpIdx % total_qk_heads; + // Determine which token and head this warp processes + int const tokenIdx = globalWarpIdx / total_proc_heads; + int const localHeadIdx = globalWarpIdx % total_proc_heads; // Skip if this warp is assigned beyond the number of tokens if (tokenIdx >= num_tokens) return; bool const isQ = localHeadIdx < num_heads_q; - int const headIdx = isQ ? localHeadIdx : localHeadIdx - num_heads_q; + bool const isV = localHeadIdx >= total_qk_heads; + int headIdx; // index within the head's own Q/K/V segment + int segStart; // element offset of the segment start within a token row + if (isQ) + { + headIdx = localHeadIdx; + segStart = 0; + } + else if (!isV) + { + headIdx = localHeadIdx - num_heads_q; + segStart = num_heads_q * head_dim; + } + else + { + headIdx = localHeadIdx - total_qk_heads; + segStart = total_qk_heads * head_dim; + } int const num_heads = num_heads_q + num_heads_k + num_heads_v; @@ -119,17 +173,7 @@ __global__ void fusedQKNormRopeKernel( constexpr int vecSize = elemSizeBytes / 4; // Use packed_as to perform loading/saving. using vec_T = typename tensorrt_llm::common::packed_as::type; - int offsetWarp; // Offset for the warp - if (isQ) - { - // Q segment: token offset + head offset within Q segment - offsetWarp = tokenIdx * num_heads * head_dim + headIdx * head_dim; - } - else - { - // K segment: token offset + entire Q segment + head offset within K segment - offsetWarp = tokenIdx * num_heads * head_dim + num_heads_q * head_dim + headIdx * head_dim; - } + int const offsetWarp = tokenIdx * num_heads * head_dim + segStart + headIdx * head_dim; int offsetThread = offsetWarp + laneId * numElemsPerThread; // Sum of squares for RMSNorm @@ -137,7 +181,8 @@ __global__ void fusedQKNormRopeKernel( // Load. { - vec_T vec = *reinterpret_cast(&qkv[offsetThread]); + vec_T vec = *reinterpret_cast(&qkv_in[offsetThread]); +#pragma unroll for (int i = 0; i < vecSize; i++) { float2 vals = __bfloat1622float2(*reinterpret_cast<__nv_bfloat162*>(reinterpret_cast(&vec) + i)); @@ -149,6 +194,13 @@ __global__ void fusedQKNormRopeKernel( } } + // V heads are copy-cast only: no norm, no RoPE. + if (isV) + { + storeHeadElements(qkv_out, offsetThread, elements); + return; + } + if (is_qk_norm) { // Reduce sum across warp using the utility function @@ -296,16 +348,7 @@ __global__ void fusedQKNormRopeKernel( } // Store. - { - vec_T vec; - for (int i = 0; i < vecSize; i++) - { - __nv_bfloat162 vals = __float22bfloat162_rn(make_float2(elements[2 * i], elements[2 * i + 1])); - reinterpret_cast<__nv_bfloat162&>(*(reinterpret_cast(&vec) + i)) = vals; - } - vec_T* outputPtr = reinterpret_cast(&qkv[offsetThread]); - *outputPtr = vec; - } + storeHeadElements(qkv_out, offsetThread, elements); } // Borrowed from @@ -322,18 +365,25 @@ __global__ void fusedQKNormRopeKernel( __VA_ARGS__ \ } -void launchFusedQKNormRope(void* qkv, int const num_tokens, int const num_heads_q, int const num_heads_k, - int const num_heads_v, int const head_dim, int const rotary_dim, float const eps, void const* q_weight, - void const* k_weight, float const base, bool const interleave, int const* position_ids, float factor, float low, - float high, float attention_factor, cudaStream_t stream, bool is_qk_norm, bool use_gemma, bool use_mrope, - int mrope_section1, int mrope_section2) +template +static void launchFusedQKNormRopeImpl(__nv_bfloat16 const* qkv_in, OutT* qkv_out, bool const process_v, + int const num_tokens, int const num_heads_q, int const num_heads_k, int const num_heads_v, int const head_dim, + int const rotary_dim, float const eps, __nv_bfloat16 const* q_weight, __nv_bfloat16 const* k_weight, + float const base, bool const interleave, int const* position_ids, float factor, float low, float high, + float attention_factor, cudaStream_t stream, bool is_qk_norm, bool use_gemma, bool use_mrope, int mrope_section1, + int mrope_section2) { if (factor == 1.0f) { TLLM_CHECK(attention_factor == 1.0f); } - TLLM_CHECK_WITH_INFO(rotary_dim % 2 == 0, "rotary_dim must be even"); + TLLM_CHECK_WITH_INFO(rotary_dim > 0 && rotary_dim <= head_dim && rotary_dim % 2 == 0, + "rotary_dim must be positive, even and no greater than head_dim (got rotary_dim=%d, head_dim=%d)", rotary_dim, + head_dim); + // Skipping V leaves the output's V slots untouched, which is only meaningful in place. + TLLM_CHECK_WITH_INFO(process_v || static_cast(qkv_in) == static_cast(qkv_out), + "process_v=false requires qkv_in and qkv_out to alias"); if (!interleave) { // To allow warp-level pairing for partial rope @@ -344,8 +394,8 @@ void launchFusedQKNormRope(void* qkv, int const num_tokens, int const num_heads_ constexpr int blockSize = 256; int const warpsPerBlock = blockSize / 32; - int const totalQKHeads = num_heads_q + num_heads_k; - int const totalWarps = num_tokens * totalQKHeads; + int const totalProcHeads = num_heads_q + num_heads_k + (process_v ? num_heads_v : 0); + int const totalWarps = num_tokens * totalProcHeads; int const gridSize = common::divUp(totalWarps, warpsPerBlock); dim3 gridDim(gridSize); @@ -357,34 +407,58 @@ void launchFusedQKNormRope(void* qkv, int const num_tokens, int const num_heads_ { case 64: DISPATCH_INTERLEAVE(interleave, INTERLEAVE, { - fusedQKNormRopeKernel<64, INTERLEAVE> - <<>>(reinterpret_cast<__nv_bfloat16*>(qkv), num_heads_q, num_heads_k, - num_heads_v, rotary_dim, eps, reinterpret_cast<__nv_bfloat16 const*>(q_weight), - reinterpret_cast<__nv_bfloat16 const*>(k_weight), base, position_ids, num_tokens, factor, low, high, - attention_factor, is_qk_norm, use_gemma, use_mrope, mrope_section1, mrope_section2); + fusedQKNormRopeKernel<64, INTERLEAVE, OutT><<>>(qkv_in, qkv_out, num_heads_q, + num_heads_k, num_heads_v, process_v, rotary_dim, eps, q_weight, k_weight, base, position_ids, + num_tokens, factor, low, high, attention_factor, is_qk_norm, use_gemma, use_mrope, mrope_section1, + mrope_section2); }); break; case 128: DISPATCH_INTERLEAVE(interleave, INTERLEAVE, { - fusedQKNormRopeKernel<128, INTERLEAVE> - <<>>(reinterpret_cast<__nv_bfloat16*>(qkv), num_heads_q, num_heads_k, - num_heads_v, rotary_dim, eps, reinterpret_cast<__nv_bfloat16 const*>(q_weight), - reinterpret_cast<__nv_bfloat16 const*>(k_weight), base, position_ids, num_tokens, factor, low, high, - attention_factor, is_qk_norm, use_gemma, use_mrope, mrope_section1, mrope_section2); + fusedQKNormRopeKernel<128, INTERLEAVE, OutT><<>>(qkv_in, qkv_out, num_heads_q, + num_heads_k, num_heads_v, process_v, rotary_dim, eps, q_weight, k_weight, base, position_ids, + num_tokens, factor, low, high, attention_factor, is_qk_norm, use_gemma, use_mrope, mrope_section1, + mrope_section2); }); break; case 256: DISPATCH_INTERLEAVE(interleave, INTERLEAVE, { - fusedQKNormRopeKernel<256, INTERLEAVE> - <<>>(reinterpret_cast<__nv_bfloat16*>(qkv), num_heads_q, num_heads_k, - num_heads_v, rotary_dim, eps, reinterpret_cast<__nv_bfloat16 const*>(q_weight), - reinterpret_cast<__nv_bfloat16 const*>(k_weight), base, position_ids, num_tokens, factor, low, high, - attention_factor, is_qk_norm, use_gemma, use_mrope, mrope_section1, mrope_section2); + fusedQKNormRopeKernel<256, INTERLEAVE, OutT><<>>(qkv_in, qkv_out, num_heads_q, + num_heads_k, num_heads_v, process_v, rotary_dim, eps, q_weight, k_weight, base, position_ids, + num_tokens, factor, low, high, attention_factor, is_qk_norm, use_gemma, use_mrope, mrope_section1, + mrope_section2); }); break; default: TLLM_THROW("Unsupported head dimension for fusedQKNormRope: %d", head_dim); } } + +void launchFusedQKNormRope(void* qkv, int const num_tokens, int const num_heads_q, int const num_heads_k, + int const num_heads_v, int const head_dim, int const rotary_dim, float const eps, void const* q_weight, + void const* k_weight, float const base, bool const interleave, int const* position_ids, float factor, float low, + float high, float attention_factor, cudaStream_t stream, bool is_qk_norm, bool use_gemma, bool use_mrope, + int mrope_section1, int mrope_section2) +{ + launchFusedQKNormRopeImpl<__nv_bfloat16>(static_cast<__nv_bfloat16 const*>(qkv), static_cast<__nv_bfloat16*>(qkv), + /*process_v=*/false, num_tokens, num_heads_q, num_heads_k, num_heads_v, head_dim, rotary_dim, eps, + static_cast<__nv_bfloat16 const*>(q_weight), static_cast<__nv_bfloat16 const*>(k_weight), base, interleave, + position_ids, factor, low, high, attention_factor, stream, is_qk_norm, use_gemma, use_mrope, mrope_section1, + mrope_section2); +} + +void launchFusedQKNormRopeToFp8(void const* qkv_in, void* qkv_out, int const num_tokens, int const num_heads_q, + int const num_heads_k, int const num_heads_v, int const head_dim, int const rotary_dim, float const eps, + void const* q_weight, void const* k_weight, float const base, bool const interleave, int const* position_ids, + float factor, float low, float high, float attention_factor, cudaStream_t stream, bool is_qk_norm, bool use_gemma, + bool use_mrope, int mrope_section1, int mrope_section2) +{ + // Out-of-place, so V has to be copy-cast rather than left untouched. + launchFusedQKNormRopeImpl<__nv_fp8_e4m3>(static_cast<__nv_bfloat16 const*>(qkv_in), + static_cast<__nv_fp8_e4m3*>(qkv_out), /*process_v=*/true, num_tokens, num_heads_q, num_heads_k, num_heads_v, + head_dim, rotary_dim, eps, static_cast<__nv_bfloat16 const*>(q_weight), + static_cast<__nv_bfloat16 const*>(k_weight), base, interleave, position_ids, factor, low, high, + attention_factor, stream, is_qk_norm, use_gemma, use_mrope, mrope_section1, mrope_section2); +} } // namespace kernels TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/fusedQKNormRopeKernel.h b/cpp/tensorrt_llm/kernels/fusedQKNormRopeKernel.h index 4e2421cb57a2..fd2401f592f1 100644 --- a/cpp/tensorrt_llm/kernels/fusedQKNormRopeKernel.h +++ b/cpp/tensorrt_llm/kernels/fusedQKNormRopeKernel.h @@ -51,6 +51,16 @@ void launchFusedQKNormRope( int mrope_section1, // mrope_section[1] (height) int mrope_section2); // mrope_section[2] (width) +// Out-of-place FP8 variant of launchFusedQKNormRope, folding the FP8 +// activation-quant into the norm+RoPE epilogue. Q and K get RMSNorm + RoPE; V is +// copy-cast only. +void launchFusedQKNormRopeToFp8(void const* qkv_in, // BF16 input [num_tokens, total_heads*head_dim] + void* qkv_out, // FP8 E4M3 output buffer, same layout as input + int const num_tokens, int const num_heads_q, int const num_heads_k, int const num_heads_v, int const head_dim, + int const rotary_dim, float const eps, void const* q_weight, void const* k_weight, float const base, + bool const interleave, int const* position_ids, float factor, float low, float high, float attention_factor, + cudaStream_t stream, bool is_qk_norm, bool use_gemma, bool use_mrope, int mrope_section1, int mrope_section2); + } // namespace kernels TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/groupGemm.cu b/cpp/tensorrt_llm/kernels/groupGemm.cu index b41021ffe6f7..87e5b94c6645 100644 --- a/cpp/tensorrt_llm/kernels/groupGemm.cu +++ b/cpp/tensorrt_llm/kernels/groupGemm.cu @@ -1,5 +1,5 @@ /* - * SPDX-FileCopyrightText: Copyright (c) 1993-2022 NVIDIA CORPORATION & + * SPDX-FileCopyrightText: Copyright (c) 1993-2026 NVIDIA CORPORATION & * AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 * * Licensed under the Apache License, Version 2.0 (the "License"); @@ -22,6 +22,17 @@ #include "cutlass/gemm/kernel/default_gemm_grouped.h" #include "cutlass/gemm/kernel/gemm_grouped.h" +#ifdef ENABLE_FP8 +#include "cute/tensor.hpp" +#include "cutlass/epilogue/collective/collective_builder.hpp" +#include "cutlass/gemm/collective/collective_builder.hpp" +#include "cutlass/gemm/device/gemm_universal_adapter.h" +#include "cutlass/gemm/dispatch_policy.hpp" +#include "cutlass/gemm/group_array_problem_shape.hpp" +#include "cutlass/gemm/kernel/gemm_universal.hpp" +#include "cutlass/util/packed_stride.hpp" +#endif // ENABLE_FP8 + #include "groupGemm.h" #include "tensorrt_llm/common/assert.h" #include "tensorrt_llm/common/config.h" @@ -34,6 +45,36 @@ TRTLLM_NAMESPACE_BEGIN namespace kernels { +#ifdef ENABLE_FP8 +#if defined(CUTLASS_ARCH_MMA_MODIFIABLE_TMA_SM90_SUPPORTED) +namespace +{ + +void checkFp8GroupedGemmAlignment(std::vector const& problemSizes, char const* kernelName) +{ + int const smVersion = tensorrt_llm::common::getSMVersion(); + // CUTLASS also exposes this kernel level on SM120/SM121; enable those after validation. + TLLM_CHECK_WITH_INFO( + smVersion == 90, "%s requires Hopper (SM90), but the current device is SM%d", kernelName, smVersion); + + for (size_t problemIdx = 0; problemIdx < problemSizes.size(); ++problemIdx) + { + auto const& problem = problemSizes[problemIdx]; + if (problem.m() == 0 || problem.n() == 0 || problem.k() == 0) + { + continue; + } + + TLLM_CHECK_WITH_INFO(problem.n() % kFp8TmaAlignment == 0 && problem.k() % kFp8TmaAlignment == 0, + "%s requires GEMM N and K dimensions to be multiples of %d elements for 128-bit TMA alignment. " + "Problem %zu has M=%d, N=%d, K=%d.", + kernelName, kFp8TmaAlignment, problemIdx, problem.m(), problem.n(), problem.k()); + } +} + +} // namespace +#endif // CUTLASS_ARCH_MMA_MODIFIABLE_TMA_SM90_SUPPORTED +#endif // ENABLE_FP8 int64_t inline getGemmCoordSize(int64_t problemCount) { @@ -59,6 +100,22 @@ int64_t getGroupedGemmParamsWorkSpaceSize(int64_t problemCount) return gemm_coord_size + ptr_size + ldd_size; } +int64_t getFp8GroupedGemmParamsWorkSpaceSize(int64_t problemCount) +{ + // The FP8 grouped GEMM (CUTLASS 3.x) stores on device: + // - problem shapes: problemCount * 12 bytes (Shape) + // - 4 pointer arrays: 4 * problemCount * 8 bytes + // - 4 stride arrays: 4 * problemCount * 16 bytes (conservative upper bound for cute strides) + // All 16-byte aligned. + auto align16 = [](int64_t bytes) -> int64_t { return (bytes + 15) / 16 * 16; }; + + int64_t const szProblemShapes = align16(problemCount * 12); + int64_t const szPtrs = 4 * align16(problemCount * 8); + int64_t const szStrides = 4 * align16(problemCount * 16); + + return szProblemShapes + szPtrs + szStrides; +} + template void groupedGemm_(std::vector problem_sizes, std::vector const& ptrA, @@ -201,12 +258,246 @@ void groupedGemmType_(std::vector problem_sizes, std:: #endif } +#ifdef ENABLE_FP8 + +// ==================================================================== +// FP8 grouped GEMM using CUTLASS 3.x collective API (Hopper SM90). +// +// The legacy CUTLASS 2.x DefaultGemmGrouped does NOT support fp8 element +// types. This implementation uses the CUTLASS 3.x CollectiveBuilder and +// GemmUniversal kernel with PtrArray schedules, following the pattern of +// cutlass/examples/57_hopper_grouped_gemm. +// +// Layout convention: +// A: RowMajor (M x K) +// B: ColumnMajor (K x N, stored column-major) +// D: RowMajor (M x N, output; C is aliased to D for beta=0) +// +// The 3.x grouped API uses pointer-array arguments: arrays of per-group +// pointers and strides on the GPU. We build these on the host and copy +// them to device memory via the provided workspace. +// ==================================================================== + +#if defined(CUTLASS_ARCH_MMA_MODIFIABLE_TMA_SM90_SUPPORTED) + +void fp8GroupedGemm(std::vector const& problemSizes, std::vector const& ptrA, + std::vector const& ptrB, std::vector const& ptrC, std::vector const& ptrD, + void* gemmParamsWorkSpace, int64_t gemmParamsWorkSpaceSize, void* gemmWorkSpace, int64_t gemmWorkspaceSize, + cudaStream_t stream) +{ + TLLM_LOG_TRACE("%s start", __PRETTY_FUNCTION__); + + using namespace cute; + + using ElementA = cutlass::float_e4m3_t; + using ElementB = cutlass::float_e4m3_t; + using ElementC = cutlass::float_e4m3_t; + using ElementD = cutlass::float_e4m3_t; + using ElementAccumulator = float; + + using LayoutA = cutlass::layout::RowMajor; + using LayoutB = cutlass::layout::ColumnMajor; + using LayoutC = cutlass::layout::RowMajor; + using LayoutD = cutlass::layout::RowMajor; + + // 128-bit TMA alignment: 128 bits / 8 bits per fp8 element = 16 elements + static constexpr int kAlignmentA = 128 / cutlass::sizeof_bits::value; + static constexpr int kAlignmentB = 128 / cutlass::sizeof_bits::value; + static constexpr int kAlignmentC = 128 / cutlass::sizeof_bits::value; + static constexpr int kAlignmentD = 128 / cutlass::sizeof_bits::value; + + using ArchTag = cutlass::arch::Sm90; + using OperatorClass = cutlass::arch::OpClassTensorOp; + + // Tile and cluster shapes chosen for fp8 on Hopper. + using TileShape = Shape<_128, _128, _128>; + using ClusterShape = Shape<_1, _2, _1>; + + // Kernel and epilogue schedule for fp8 grouped GEMM with PtrArray TMA. + using KernelSchedule = cutlass::gemm::KernelPtrArrayTmaWarpSpecializedCooperativeFP8FastAccum; + using EpilogueSchedule = cutlass::epilogue::PtrArrayTmaWarpSpecializedCooperative; + + using ProblemShape = cutlass::gemm::GroupProblemShape>; + + using CollectiveEpilogue = + typename cutlass::epilogue::collective::CollectiveBuilder::CollectiveOp; + + using CollectiveMainloop = typename cutlass::gemm::collective::CollectiveBuilder( + sizeof(typename CollectiveEpilogue::SharedStorage))>, + KernelSchedule>::CollectiveOp; + + using GemmKernel = cutlass::gemm::kernel::GemmUniversal; + using Gemm = cutlass::gemm::device::GemmUniversalAdapter; + + using StrideA = typename GemmKernel::InternalStrideA; + using StrideB = typename GemmKernel::InternalStrideB; + using StrideC = typename GemmKernel::InternalStrideC; + using StrideD = typename GemmKernel::InternalStrideD; + + int const problemCount = static_cast(problemSizes.size()); + + // Build host-side arrays for problem shapes, pointers, and strides. + std::vector problemShapesHost(problemCount); + std::vector ptrAHost(problemCount); + std::vector ptrBHost(problemCount); + std::vector ptrCHost(problemCount); + std::vector ptrDHost(problemCount); + std::vector strideAHost(problemCount); + std::vector strideBHost(problemCount); + std::vector strideCHost(problemCount); + std::vector strideDHost(problemCount); + + for (int i = 0; i < problemCount; ++i) + { + auto const& coord = problemSizes[i]; + int const m = coord.m(); + int const n = coord.n(); + int const k = coord.k(); + + problemShapesHost[i] = cute::make_shape(m, n, k); + ptrAHost[i] = static_cast(ptrA[i]); + ptrBHost[i] = static_cast(ptrB[i]); + ptrCHost[i] = static_cast(ptrC[i]); + ptrDHost[i] = static_cast(ptrD[i]); + + strideAHost[i] = cutlass::make_cute_packed_stride(StrideA{}, cute::make_shape(m, k, 1)); + strideBHost[i] = cutlass::make_cute_packed_stride(StrideB{}, cute::make_shape(n, k, 1)); + strideCHost[i] = cutlass::make_cute_packed_stride(StrideC{}, cute::make_shape(m, n, 1)); + strideDHost[i] = cutlass::make_cute_packed_stride(StrideD{}, cute::make_shape(m, n, 1)); + } + + // Compute workspace layout sizes (all 16-byte aligned). + auto const align16 = [](int64_t bytes) -> int64_t { return (bytes + 15) / 16 * 16; }; + + int64_t const szProblemShapes = align16(problemCount * sizeof(typename ProblemShape::UnderlyingProblemShape)); + int64_t const szPtrA = align16(problemCount * sizeof(ElementA const*)); + int64_t const szPtrB = align16(problemCount * sizeof(ElementB const*)); + int64_t const szPtrC = align16(problemCount * sizeof(ElementC const*)); + int64_t const szPtrD = align16(problemCount * sizeof(ElementD*)); + int64_t const szStrideA = align16(problemCount * sizeof(StrideA)); + int64_t const szStrideB = align16(problemCount * sizeof(StrideB)); + int64_t const szStrideC = align16(problemCount * sizeof(StrideC)); + int64_t const szStrideD = align16(problemCount * sizeof(StrideD)); + + int64_t const totalParamsBytes + = szProblemShapes + szPtrA + szPtrB + szPtrC + szPtrD + szStrideA + szStrideB + szStrideC + szStrideD; + TLLM_CHECK_WITH_INFO(totalParamsBytes <= gemmParamsWorkSpaceSize, + "FP8 grouped GEMM params workspace too small: need %ld, have %ld", totalParamsBytes, gemmParamsWorkSpaceSize); + + // Stage everything into a single host buffer, then memcpy to device. + std::vector hostBuf(totalParamsBytes); + char* cursor = hostBuf.data(); + + auto copyToHostBuf = [&cursor](void const* src, int64_t bytes) + { + std::memcpy(cursor, src, bytes); + char* base = cursor; + cursor += (bytes + 15) / 16 * 16; + return base; + }; + + copyToHostBuf(problemShapesHost.data(), problemCount * sizeof(typename ProblemShape::UnderlyingProblemShape)); + copyToHostBuf(ptrAHost.data(), problemCount * sizeof(ElementA const*)); + copyToHostBuf(ptrBHost.data(), problemCount * sizeof(ElementB const*)); + copyToHostBuf(ptrCHost.data(), problemCount * sizeof(ElementC const*)); + copyToHostBuf(ptrDHost.data(), problemCount * sizeof(ElementD*)); + copyToHostBuf(strideAHost.data(), problemCount * sizeof(StrideA)); + copyToHostBuf(strideBHost.data(), problemCount * sizeof(StrideB)); + copyToHostBuf(strideCHost.data(), problemCount * sizeof(StrideC)); + copyToHostBuf(strideDHost.data(), problemCount * sizeof(StrideD)); + + tensorrt_llm::common::cudaAutoCpy(reinterpret_cast(gemmParamsWorkSpace), + reinterpret_cast(hostBuf.data()), totalParamsBytes, stream); + + // Compute device pointers from the workspace base. + char* devBase = static_cast(gemmParamsWorkSpace); + int64_t devOff = 0; + + auto devPtr = [&devBase, &devOff](int64_t sz) -> void* + { + void* p = devBase + devOff; + devOff += (sz + 15) / 16 * 16; + return p; + }; + + auto* devProblemShapes = static_cast(devPtr(szProblemShapes)); + auto* devPtrA = static_cast(devPtr(szPtrA)); + auto* devPtrB = static_cast(devPtr(szPtrB)); + auto* devPtrC = static_cast(devPtr(szPtrC)); + auto* devPtrD = static_cast(devPtr(szPtrD)); + auto* devStrideA = static_cast(devPtr(szStrideA)); + auto* devStrideB = static_cast(devPtr(szStrideB)); + auto* devStrideC = static_cast(devPtr(szStrideC)); + auto* devStrideD = static_cast(devPtr(szStrideD)); + + // Hardware info for kernel launch. + cutlass::KernelHardwareInfo hwInfo; + hwInfo.device_id = 0; + cudaGetDevice(&hwInfo.device_id); + hwInfo.sm_count = cutlass::KernelHardwareInfo::query_device_multiprocessor_count(hwInfo.device_id); + + // Build CUTLASS 3.x arguments. alpha=1, beta=0 for LoRA GEMM. + typename Gemm::Arguments arguments{cutlass::gemm::GemmUniversalMode::kGrouped, + {problemCount, devProblemShapes, nullptr}, {devPtrA, devStrideA, devPtrB, devStrideB}, + {{1.0f, 0.0f}, devPtrC, devStrideC, devPtrD, devStrideD}, hwInfo}; + + Gemm gemm; + + size_t const workspaceSize = Gemm::get_workspace_size(arguments); + TLLM_CHECK_WITH_INFO(static_cast(workspaceSize) <= gemmWorkspaceSize, + "FP8 grouped GEMM workspace too small: need %lu, have %ld", workspaceSize, gemmWorkspaceSize); + + cutlass::Status status = gemm.can_implement(arguments); + TLLM_CHECK_WITH_INFO(status == cutlass::Status::kSuccess, "FP8 grouped GEMM can_implement failed: %s", + cutlass::cutlassGetStatusString(status)); + + status = gemm.initialize(arguments, gemmWorkSpace, stream); + TLLM_CHECK_WITH_INFO(status == cutlass::Status::kSuccess, "FP8 grouped GEMM initialize failed: %s", + cutlass::cutlassGetStatusString(status)); + + status = gemm.run(stream); + TLLM_CHECK_WITH_INFO(status == cutlass::Status::kSuccess, "FP8 grouped GEMM run failed: %s", + cutlass::cutlassGetStatusString(status)); + + sync_check_cuda_error(stream); + + TLLM_LOG_TRACE("%s stop", __PRETTY_FUNCTION__); +} + +#endif // CUTLASS_ARCH_MMA_MODIFIABLE_TMA_SM90_SUPPORTED + +#endif // ENABLE_FP8 + void groupedGemm(std::vector problem_sizes, std::vector const& ptrA, std::vector const& ptrB, std::vector const& ptrC, std::vector const& ptrD, void* gemmParamsWorkSpace, int64_t gemmParamsWorkSpaceSize, void* gemmWorkSpace, int64_t gemmWorkspaceSize, bool isLoraIn, tensorrt_llm::DataType dataType, int minKN, cudaStream_t stream) { TLLM_LOG_TRACE("%s start, isLoraIn: %d, minKN = %d", __PRETTY_FUNCTION__, static_cast(isLoraIn), minKN); + +#ifdef ENABLE_FP8 +#if defined(CUTLASS_ARCH_MMA_MODIFIABLE_TMA_SM90_SUPPORTED) + if (dataType == tensorrt_llm::DataType::kFP8) + { + checkFp8GroupedGemmAlignment(problem_sizes, "FP8 grouped GEMM"); + + fp8GroupedGemm(problem_sizes, ptrA, ptrB, ptrC, ptrD, gemmParamsWorkSpace, gemmParamsWorkSpaceSize, + gemmWorkSpace, gemmWorkspaceSize, stream); + return; + } +#else + if (dataType == tensorrt_llm::DataType::kFP8) + { + TLLM_CHECK_WITH_INFO( + false, "FP8 grouped GEMM requires CUTLASS modifiable TMA support (CUDA 12.3+ and Hopper SM90 kernels)."); + } +#endif // CUTLASS_ARCH_MMA_MODIFIABLE_TMA_SM90_SUPPORTED +#endif // ENABLE_FP8 + if (isLoraIn) { // K >> N, like K = 1024, N = 8 diff --git a/cpp/tensorrt_llm/kernels/groupGemm.h b/cpp/tensorrt_llm/kernels/groupGemm.h index c526e08e986c..48b38eb82d65 100644 --- a/cpp/tensorrt_llm/kernels/groupGemm.h +++ b/cpp/tensorrt_llm/kernels/groupGemm.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2019-2023, NVIDIA CORPORATION. All rights reserved. + * Copyright (c) 2019-2026, NVIDIA CORPORATION. All rights reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -24,8 +24,16 @@ TRTLLM_NAMESPACE_BEGIN namespace kernels { +inline constexpr int kFp8TmaAlignment = 16; + int64_t getGroupedGemmParamsWorkSpaceSize(int64_t problem_count); +//! @brief Returns the parameter workspace size for the FP8 grouped GEMM path. +//! +//! The FP8 path (CUTLASS 3.x) stores problem shapes, pointer arrays, and +//! cute stride arrays on the device. This function returns the required bytes. +int64_t getFp8GroupedGemmParamsWorkSpaceSize(int64_t problemCount); + void groupedGemm(std::vector problem_sizes, std::vector const& ptrA, std::vector const& ptrB, std::vector const& ptrC, std::vector const& ptrD, void* gemmParamsWorkspace, int64_t gemmParamsWorkSpaceSize, void* gemmWorkSpace, int64_t gemmWorkspaceSize, diff --git a/cpp/tensorrt_llm/kernels/indexerKCacheGather.cu b/cpp/tensorrt_llm/kernels/indexerKCacheGather.cu index cbcb31f07111..cd33862afb4c 100644 --- a/cpp/tensorrt_llm/kernels/indexerKCacheGather.cu +++ b/cpp/tensorrt_llm/kernels/indexerKCacheGather.cu @@ -63,7 +63,7 @@ __device__ __forceinline__ int64_t flatIndexToMemoryOffset( * @param out_scale Output scale data [num_tokens, scale_size] contiguous * @param k_token_start Start offset into slot_mapping arrays * @param num_tokens Number of tokens to gather - * @param head_dim Head dimension (must be 128) + * @param head_dim Payload byte width (128 for FP8, 64 for packed FP4) * @param scale_size Scale size in bytes (must be 4) * @param cache_stride_0 Stride for k_cache dimension 0 (in bytes) * @param cache_stride_1 Stride for k_cache dimension 1 (in bytes) diff --git a/cpp/tensorrt_llm/kernels/indexerKCacheScatter.cu b/cpp/tensorrt_llm/kernels/indexerKCacheScatter.cu index f0a41a5009fc..f0c7c27a235c 100644 --- a/cpp/tensorrt_llm/kernels/indexerKCacheScatter.cu +++ b/cpp/tensorrt_llm/kernels/indexerKCacheScatter.cu @@ -61,7 +61,7 @@ __device__ __forceinline__ int64_t flatIndexToMemoryOffset( * @param slot_mapping_fp8 Flat element index for FP8 data start position [num_tokens] * @param slot_mapping_scale Flat element index for scale data start position [num_tokens] * @param num_tokens Number of tokens - * @param head_dim Head dimension (must be 128) + * @param head_dim Payload byte width (128 for FP8, 64 for packed FP4) * @param scale_size Scale size in bytes (must be 4) * @param cache_stride_0 Stride for k_cache dimension 0 (in bytes) * @param cache_stride_1 Stride for k_cache dimension 1 (in bytes) diff --git a/cpp/tensorrt_llm/kernels/kdaDecode/CMakeLists.txt b/cpp/tensorrt_llm/kernels/kdaDecode/CMakeLists.txt new file mode 100644 index 000000000000..81560913c019 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/kdaDecode/CMakeLists.txt @@ -0,0 +1,21 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. +# All rights reserved. SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of +# the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations under +# the License. + +add_library(kda_decode_src OBJECT kdaDecode.cu) + +set_property(TARGET kda_decode_src PROPERTY POSITION_INDEPENDENT_CODE ON) +set_property(TARGET kda_decode_src PROPERTY CUDA_RESOLVE_DEVICE_SYMBOLS ON) +target_compile_options(kda_decode_src + PRIVATE $<$:--use_fast_math>) diff --git a/cpp/tensorrt_llm/kernels/kdaDecode/kdaDecode.cu b/cpp/tensorrt_llm/kernels/kdaDecode/kdaDecode.cu new file mode 100644 index 000000000000..1249779454ad --- /dev/null +++ b/cpp/tensorrt_llm/kernels/kdaDecode/kdaDecode.cu @@ -0,0 +1,1611 @@ +/* + * Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/kdaDecode/kdaDecode.h" + +#include "tensorrt_llm/common/cudaUtils.h" + +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels::kdaDecode +{ + +namespace +{ + +constexpr int kDimK = 128; +constexpr int kDimV = 128; +constexpr int kKernelWidth = 4; +constexpr int kConvStateWidth = kKernelWidth - 1; +constexpr int kThreads = 256; +constexpr int kWarps = kThreads / 32; +constexpr int kChunkV = 32; +constexpr int kNumChunks = kDimV / kChunkV; +constexpr int kRowsPerWarp = kChunkV / kWarps; + +template +__device__ __forceinline__ float bf16_load(__nv_bfloat16 const* ptr, Index idx) +{ + return __bfloat162float(ptr[idx]); +} + +__device__ __forceinline__ __nv_bfloat16 bf16_store(float value) +{ + return __float2bfloat16(value); +} + +template +__device__ __forceinline__ void store_state_float4(float* ptr, float4 value) +{ + if constexpr (kUseCacheGlobalStore) + { + __stcg(reinterpret_cast(ptr), value); + } + else + { + *reinterpret_cast(ptr) = value; + } +} + +__device__ __forceinline__ float sigmoid_fast(float x) +{ + return 1.0f / (1.0f + __expf(-x)); +} + +__device__ __forceinline__ float silu_fast(float x) +{ + return x * sigmoid_fast(x); +} + +__device__ __forceinline__ float softplus_fast(float x) +{ + return x > 20.0f ? x : log1pf(__expf(x)); +} + +__device__ __forceinline__ float warp_reduce_sum(float value) +{ +#pragma unroll + for (int offset = 16; offset > 0; offset >>= 1) + { + value += __shfl_xor_sync(0xffffffffu, value, offset); + } + return value; +} + +// cp.async requires sm_80+; pre-SM80 falls back to a synchronous copy, so the +// commit/wait helpers become no-ops there. +__device__ __forceinline__ void cp_async_cg_16b(float* smem_ptr, float const* gmem_ptr) +{ +#if defined(__CUDA_ARCH__) && __CUDA_ARCH__ >= 800 + uint32_t smem_addr = static_cast(__cvta_generic_to_shared(smem_ptr)); + asm volatile("cp.async.cg.shared.global [%0], [%1], 16;\n" : : "r"(smem_addr), "l"(gmem_ptr)); +#else + *reinterpret_cast(smem_ptr) = *reinterpret_cast(gmem_ptr); +#endif +} + +__device__ __forceinline__ void cp_async_commit() +{ +#if defined(__CUDA_ARCH__) && __CUDA_ARCH__ >= 800 + asm volatile("cp.async.commit_group;\n" ::); +#endif +} + +__device__ __forceinline__ void cp_async_wait_all() +{ +#if defined(__CUDA_ARCH__) && __CUDA_ARCH__ >= 800 + asm volatile("cp.async.wait_all;\n" ::); +#endif +} + +__device__ __forceinline__ void cp_async_wait_group_0() +{ +#if defined(__CUDA_ARCH__) && __CUDA_ARCH__ >= 800 + asm volatile("cp.async.wait_group 0;\n" ::); +#endif +} + +__device__ __forceinline__ void cp_async_wait_group_1() +{ +#if defined(__CUDA_ARCH__) && __CUDA_ARCH__ >= 800 + asm volatile("cp.async.wait_group 1;\n" ::); +#endif +} + +__device__ __forceinline__ void cp_async_wait_group_2() +{ +#if defined(__CUDA_ARCH__) && __CUDA_ARCH__ >= 800 + asm volatile("cp.async.wait_group 2;\n" ::); +#endif +} + +template +__device__ __forceinline__ void cp_async_state_chunk_stage( + float* s_state, float const* state, int slot, int i_hv, int64_t state_slot_stride, int chunk, int stage) +{ + constexpr int kFloat4PerChunk = kStageChunkV * kDimK / 4; + int const tid = threadIdx.x; + int const v_base = chunk * kStageChunkV; + for (int linear4 = tid; linear4 < kFloat4PerChunk; linear4 += kThreads) + { + int const elem = linear4 * 4; + int const row = elem / kDimK; + int const k = elem - row * kDimK; + float* dst = s_state + (stage * kStageChunkV + row) * kDimK + k; + int64_t const state_offset = static_cast(slot) * state_slot_stride + + (static_cast(i_hv) * kDimV + v_base + row) * kDimK + k; + float const* src = state + state_offset; + cp_async_cg_16b(dst, src); + } + cp_async_commit(); +} + +template +__device__ __forceinline__ void cp_async_state_chunk_for( + float* s_state, float const* state, int slot, int i_hv, int64_t state_slot_stride, int chunk) +{ + constexpr int kFloat4PerChunk = kChunkV * kDimK / 4; + int const tid = threadIdx.x; + int const stage = chunk & 1; + int const v_base = chunk * kChunkV; + for (int linear4 = tid; linear4 < kFloat4PerChunk; linear4 += kCopyThreads) + { + int const elem = linear4 * 4; + int const row = elem / kDimK; + int const k = elem - row * kDimK; + float* dst = s_state + (stage * kChunkV + row) * kDimK + k; + int64_t const state_offset = static_cast(slot) * state_slot_stride + + (static_cast(i_hv) * kDimV + v_base + row) * kDimK + k; + float const* src = state + state_offset; + cp_async_cg_16b(dst, src); + } + cp_async_commit(); +} + +__device__ __forceinline__ void cp_async_state_chunk( + float* s_state, float const* state, int slot, int i_hv, int64_t state_slot_stride, int chunk) +{ + cp_async_state_chunk_for(s_state, state, slot, i_hv, state_slot_stride, chunk); +} + +__device__ __forceinline__ float block_reduce_sum(float value, float* scratch) +{ + int const tid = threadIdx.x; + int const lane = tid & 31; + int const warp = tid >> 5; + + float warp_total = warp_reduce_sum(value); + if (lane == 0) + { + scratch[warp] = warp_total; + } + __syncthreads(); + + float block_total = 0.0f; + if (warp == 0) + { + block_total = lane < kWarps ? scratch[lane] : 0.0f; + block_total = warp_reduce_sum(block_total); + if (lane == 0) + { + scratch[0] = block_total; + } + } + __syncthreads(); + return scratch[0]; +} + +struct Sum2 +{ + float x; + float y; +}; + +__device__ __forceinline__ Sum2 warp_reduce_sum_pair(float x, float y) +{ +#pragma unroll + for (int offset = 16; offset > 0; offset >>= 1) + { + x += __shfl_xor_sync(0xffffffffu, x, offset); + y += __shfl_xor_sync(0xffffffffu, y, offset); + } + return {x, y}; +} + +struct Sum4 +{ + float a; + float b; + float c; + float d; +}; + +__device__ __forceinline__ Sum4 warp_reduce_sum4(float a, float b, float c, float d) +{ +#pragma unroll + for (int offset = 16; offset > 0; offset >>= 1) + { + a += __shfl_xor_sync(0xffffffffu, a, offset); + b += __shfl_xor_sync(0xffffffffu, b, offset); + c += __shfl_xor_sync(0xffffffffu, c, offset); + d += __shfl_xor_sync(0xffffffffu, d, offset); + } + return {a, b, c, d}; +} + +template +__device__ __forceinline__ Sum2 block_reduce_sum2_for(float x, float y, float* scratch) +{ + int const lane = threadIdx.x & 31; + int const warp = threadIdx.x >> 5; + + float const warp_x = warp_reduce_sum(x); + float const warp_y = warp_reduce_sum(y); + if (lane == 0) + { + scratch[warp] = warp_x; + scratch[kReduceWarps + warp] = warp_y; + } + __syncthreads(); + + float block_x = 0.0f; + float block_y = 0.0f; + if (warp == 0) + { + block_x = lane < kReduceWarps ? scratch[lane] : 0.0f; + block_y = lane < kReduceWarps ? scratch[kReduceWarps + lane] : 0.0f; + block_x = warp_reduce_sum(block_x); + block_y = warp_reduce_sum(block_y); + if (lane == 0) + { + scratch[0] = block_x; + scratch[1] = block_y; + } + } + __syncthreads(); + return {scratch[0], scratch[1]}; +} + +__device__ __forceinline__ Sum2 block_reduce_sum2(float x, float y, float* scratch) +{ + return block_reduce_sum2_for(x, y, scratch); +} + +template +__device__ __forceinline__ float block_reduce_sum_active_for(float value, float* scratch) +{ + int const lane = threadIdx.x & 31; + int const warp = threadIdx.x >> 5; + + float warp_total = 0.0f; + if (warp < kReduceWarps) + { + warp_total = warp_reduce_sum(value); + } + if (lane == 0 && warp < kReduceWarps) + { + scratch[warp] = warp_total; + } + __syncthreads(); + + float block_total = 0.0f; + if (warp == 0) + { + block_total = lane < kReduceWarps ? scratch[lane] : 0.0f; + block_total = warp_reduce_sum(block_total); + if (lane == 0) + { + scratch[0] = block_total; + } + } + __syncthreads(); + return scratch[0]; +} + +template +__device__ __forceinline__ Sum2 block_reduce_sum2_active_for(float x, float y, float* scratch) +{ + int const lane = threadIdx.x & 31; + int const warp = threadIdx.x >> 5; + + float warp_x = 0.0f; + float warp_y = 0.0f; + if (warp < kReduceWarps) + { + warp_x = warp_reduce_sum(x); + warp_y = warp_reduce_sum(y); + } + if (lane == 0 && warp < kReduceWarps) + { + scratch[warp] = warp_x; + scratch[kReduceWarps + warp] = warp_y; + } + __syncthreads(); + + float block_x = 0.0f; + float block_y = 0.0f; + if (warp == 0) + { + block_x = lane < kReduceWarps ? scratch[lane] : 0.0f; + block_y = lane < kReduceWarps ? scratch[kReduceWarps + lane] : 0.0f; + block_x = warp_reduce_sum(block_x); + block_y = warp_reduce_sum(block_y); + if (lane == 0) + { + scratch[0] = block_x; + scratch[1] = block_y; + } + } + __syncthreads(); + return {scratch[0], scratch[1]}; +} + +template +__global__ __launch_bounds__(kThreads, 2) void kda_decode_fusion_compact_heads_kernel( + __nv_bfloat16 const* __restrict__ x_q, __nv_bfloat16 const* __restrict__ x_k, __nv_bfloat16 const* __restrict__ x_v, + __nv_bfloat16 const* __restrict__ w_q_t, __nv_bfloat16 const* __restrict__ w_k_t, + __nv_bfloat16 const* __restrict__ w_v_t, __nv_bfloat16 const* __restrict__ bias_q, + __nv_bfloat16 const* __restrict__ bias_k, __nv_bfloat16 const* __restrict__ bias_v, + __nv_bfloat16* __restrict__ cs_q, __nv_bfloat16* __restrict__ cs_k, __nv_bfloat16* __restrict__ cs_v, + float const* __restrict__ a_log, __nv_bfloat16 const* __restrict__ g, float const* __restrict__ dt_bias, + __nv_bfloat16 const* __restrict__ beta, __nv_bfloat16 const* __restrict__ onorm_g, + float const* __restrict__ onorm_weight, int const* __restrict__ ssm_state_indices, + int const* __restrict__ cu_seqlens, float* __restrict__ state, int64_t state_slot_stride, + __nv_bfloat16* __restrict__ out, int B, int H, int HV, float lower_bound, float scale, float onorm_eps) +{ + int const tid = threadIdx.x; + int const lane = tid & 31; + int const warp = tid >> 5; + int i_n; + int i_hv; + int i_h; + int bos; + int slot; + if constexpr (kUseStaticDecodeLayout) + { + if constexpr (kUseHeadGrid) + { + i_n = blockIdx.x; + i_hv = blockIdx.y; + } + else + { + int const nhv = blockIdx.x; + i_n = nhv / kFixedValueHeads; + i_hv = nhv - i_n * kFixedValueHeads; + } + i_h = i_hv; + bos = i_n; + slot = i_n; + } + else + { + int const nhv = blockIdx.x; + i_n = nhv / HV; + i_hv = nhv - i_n * HV; + int const hv_per_h = HV / H; + i_h = i_hv / hv_per_h; + + bos = cu_seqlens == nullptr ? i_n : cu_seqlens[i_n]; + int const eos = cu_seqlens == nullptr ? i_n + 1 : cu_seqlens[i_n + 1]; + if (eos <= bos) + { + return; + } + slot = ssm_state_indices[i_n]; + } + + int const hk_off = i_h * kDimK; + int const hv_off = i_hv * kDimV; + int const h_count = kUseStaticDecodeLayout ? kFixedHeads : H; + int const hv_count = kUseStaticDecodeLayout ? kFixedValueHeads : HV; + int const hkv_dim = h_count * kDimK; + int const hvv_dim = hv_count * kDimV; + int const conv_slot = kUpdateConvState ? slot : i_n; + int64_t const qk_input_head = static_cast(bos) * h_count + i_h; + int64_t const value_input_head = static_cast(bos) * hv_count + i_hv; + int64_t const output_head = static_cast(i_n) * hv_count + i_hv; + + constexpr int kStageChunkV = 32; + constexpr int kStageCount = 3; + constexpr int kStageNumChunks = kDimV / kStageChunkV; + extern __shared__ float s_state[]; + __shared__ float s_q[kDimK]; + __shared__ float s_k[kDimK]; + __shared__ float s_decay[kDimK]; + __shared__ float s_v[kDimV]; + __shared__ float s_o[kDimV]; + __shared__ float s_reduce[kThreads]; + __shared__ float s_beta; + float pre_onorm_gate = 0.0f; + float pre_onorm_weight = 0.0f; + + cp_async_state_chunk_stage(s_state, state, slot, i_hv, state_slot_stride, 0, 0); + cp_async_state_chunk_stage(s_state, state, slot, i_hv, state_slot_stride, 1, 1); + + if constexpr (kUpdateConvState) + { + if (tid < kDimK) + { + int const k = tid; + int const hk = hk_off + k; + int64_t const cs_base = static_cast(slot) * hkv_dim * kConvStateWidth + hk; + int64_t const xq_idx = qk_input_head * kDimK + k; + float const exp_a = __shfl_sync(0xffffffffu, lane == 0 ? __expf(a_log[i_h]) : 0.0f, 0); + + float q_acc = bf16_load(bias_q, hk); + float k_acc = bf16_load(bias_k, hk); + __nv_bfloat16 q_shift0 = __float2bfloat16(0.0f); + __nv_bfloat16 q_shift1 = __float2bfloat16(0.0f); + __nv_bfloat16 k_shift0 = __float2bfloat16(0.0f); + __nv_bfloat16 k_shift1 = __float2bfloat16(0.0f); +#pragma unroll + for (int w = 0; w < kConvStateWidth; ++w) + { + const __nv_bfloat16 q_state = cs_q[cs_base + w * hkv_dim]; + const __nv_bfloat16 k_state = cs_k[cs_base + w * hkv_dim]; + q_acc += __bfloat162float(q_state) * bf16_load(w_q_t, w * hkv_dim + hk); + k_acc += __bfloat162float(k_state) * bf16_load(w_k_t, w * hkv_dim + hk); + if (w == 1) + { + q_shift0 = q_state; + k_shift0 = k_state; + } + else if (w == 2) + { + q_shift1 = q_state; + k_shift1 = k_state; + } + } + const __nv_bfloat16 q_new = x_q[xq_idx]; + const __nv_bfloat16 k_new = x_k[xq_idx]; + q_acc += __bfloat162float(q_new) * bf16_load(w_q_t, (kKernelWidth - 1) * hkv_dim + hk); + k_acc += __bfloat162float(k_new) * bf16_load(w_k_t, (kKernelWidth - 1) * hkv_dim + hk); + + cs_q[cs_base + 0] = q_shift0; + cs_q[cs_base + hkv_dim] = q_shift1; + cs_q[cs_base + 2 * hkv_dim] = q_new; + cs_k[cs_base + 0] = k_shift0; + cs_k[cs_base + hkv_dim] = k_shift1; + cs_k[cs_base + 2 * hkv_dim] = k_new; + + s_q[k] = silu_fast(q_acc); + s_k[k] = silu_fast(k_acc); + + float const g_raw = bf16_load(g, value_input_head * kDimK + k) + dt_bias[hk]; + if constexpr (kUseLowerBound) + { + s_decay[k] = __expf(lower_bound * sigmoid_fast(exp_a * g_raw)); + } + else + { + s_decay[k] = __expf(-exp_a * softplus_fast(g_raw)); + } + } + } + else + { + if (tid < kDimK) + { + int const k = tid; + int const hk = hk_off + k; + float const exp_a = __shfl_sync(0xffffffffu, lane == 0 ? __expf(a_log[i_h]) : 0.0f, 0); + + float q_acc = bf16_load(bias_q, hk); + float k_acc = bf16_load(bias_k, hk); +#pragma unroll + for (int w = 0; w < kConvStateWidth; ++w) + { + int64_t const cs_idx = (static_cast(conv_slot) * hkv_dim + hk) * kConvStateWidth + w; + q_acc += bf16_load(cs_q, cs_idx) * bf16_load(w_q_t, w * hkv_dim + hk); + k_acc += bf16_load(cs_k, cs_idx) * bf16_load(w_k_t, w * hkv_dim + hk); + } + q_acc += bf16_load(x_q, qk_input_head * kDimK + k) * bf16_load(w_q_t, (kKernelWidth - 1) * hkv_dim + hk); + k_acc += bf16_load(x_k, qk_input_head * kDimK + k) * bf16_load(w_k_t, (kKernelWidth - 1) * hkv_dim + hk); + + s_q[k] = silu_fast(q_acc); + s_k[k] = silu_fast(k_acc); + + float const g_raw = bf16_load(g, value_input_head * kDimK + k) + dt_bias[hk]; + if constexpr (kUseLowerBound) + { + s_decay[k] = __expf(lower_bound * sigmoid_fast(exp_a * g_raw)); + } + else + { + s_decay[k] = __expf(-exp_a * softplus_fast(g_raw)); + } + } + } + + if constexpr (kUpdateConvState) + { + if (tid < kDimV) + { + int const v = tid; + int const hvv = hv_off + v; + int64_t const cs_base = static_cast(slot) * hvv_dim * kConvStateWidth + hvv; + int64_t const xv_idx = value_input_head * kDimV + v; + + float v_acc = bf16_load(bias_v, hvv); + __nv_bfloat16 v_shift0 = __float2bfloat16(0.0f); + __nv_bfloat16 v_shift1 = __float2bfloat16(0.0f); +#pragma unroll + for (int w = 0; w < kConvStateWidth; ++w) + { + const __nv_bfloat16 v_state = cs_v[cs_base + w * hvv_dim]; + v_acc += __bfloat162float(v_state) * bf16_load(w_v_t, w * hvv_dim + hvv); + if (w == 1) + { + v_shift0 = v_state; + } + else if (w == 2) + { + v_shift1 = v_state; + } + } + const __nv_bfloat16 v_new = x_v[xv_idx]; + v_acc += __bfloat162float(v_new) * bf16_load(w_v_t, (kKernelWidth - 1) * hvv_dim + hvv); + cs_v[cs_base + 0] = v_shift0; + cs_v[cs_base + hvv_dim] = v_shift1; + cs_v[cs_base + 2 * hvv_dim] = v_new; + s_v[v] = silu_fast(v_acc); + + if constexpr (kApplyOnorm && kPreloadOnormParams) + { + int64_t const out_idx = output_head * kDimV + v; + pre_onorm_gate = sigmoid_fast(bf16_load(onorm_g, out_idx)); + pre_onorm_weight = onorm_weight[v]; + } + } + } + else + { + if (tid < kDimV) + { + int const v = tid; + int const hvv = hv_off + v; + + float v_acc = bf16_load(bias_v, hvv); +#pragma unroll + for (int w = 0; w < kConvStateWidth; ++w) + { + int64_t const cs_idx = (static_cast(conv_slot) * hvv_dim + hvv) * kConvStateWidth + w; + v_acc += bf16_load(cs_v, cs_idx) * bf16_load(w_v_t, w * hvv_dim + hvv); + } + v_acc + += bf16_load(x_v, value_input_head * kDimV + v) * bf16_load(w_v_t, (kKernelWidth - 1) * hvv_dim + hvv); + s_v[v] = silu_fast(v_acc); + + if constexpr (kApplyOnorm && kPreloadOnormParams) + { + int64_t const out_idx = output_head * kDimV + v; + pre_onorm_gate = sigmoid_fast(bf16_load(onorm_g, out_idx)); + pre_onorm_weight = onorm_weight[v]; + } + } + } + + if (tid == 0) + { + float const beta_raw = bf16_load(beta, value_input_head); + if constexpr (kApplyBetaSigmoid) + { + s_beta = sigmoid_fast(beta_raw); + } + else + { + s_beta = beta_raw; + } + } + __syncthreads(); + + if constexpr (kIssueThirdStatePrefetchEarly && kStageNumChunks > 2) + { + cp_async_state_chunk_stage(s_state, state, slot, i_hv, state_slot_stride, 2, 2); + } + + float const q_sq = tid < kDimK ? s_q[tid] * s_q[tid] : 0.0f; + float const k_sq = tid < kDimK ? s_k[tid] * s_k[tid] : 0.0f; + Sum2 qk_sum; + if constexpr (kUseActiveQkReduction) + { + qk_sum = block_reduce_sum2_active_for(q_sq, k_sq, s_reduce); + } + else + { + qk_sum = block_reduce_sum2(q_sq, k_sq, s_reduce); + } + if (tid < kDimK) + { + s_q[tid] *= rsqrtf(qk_sum.x + 1.0e-6f) * scale; + s_k[tid] *= rsqrtf(qk_sum.y + 1.0e-6f); + } + __syncthreads(); + + int const k_base = lane * 4; + const float4 q4 = *reinterpret_cast(s_q + k_base); + const float4 k4 = *reinterpret_cast(s_k + k_base); + const float4 decay4 = *reinterpret_cast(s_decay + k_base); + float r_q[4] = {q4.x, q4.y, q4.z, q4.w}; + float r_k[4] = {k4.x, k4.y, k4.z, k4.w}; + float r_decay[4] = {decay4.x, decay4.y, decay4.z, decay4.w}; + float o_sumsq = 0.0f; + +#pragma unroll + for (int chunk = 0; chunk < kStageNumChunks; ++chunk) + { + if constexpr (kIssueThirdStatePrefetchEarly) + { + if (chunk == 0) + { + cp_async_wait_group_2(); + } + else if (chunk + 1 < kStageNumChunks) + { + cp_async_wait_group_1(); + } + else + { + cp_async_wait_group_0(); + } + } + else + { + if (chunk + 1 < kStageNumChunks) + { + cp_async_wait_group_1(); + } + else + { + cp_async_wait_group_0(); + } + } + __syncwarp(); + + int const prefetch = chunk + 2; + if constexpr (kIssueThirdStatePrefetchEarly) + { + if (prefetch < kStageNumChunks && prefetch >= 3) + { + cp_async_state_chunk_stage( + s_state, state, slot, i_hv, state_slot_stride, prefetch, prefetch % kStageCount); + } + } + else + { + if (prefetch < kStageNumChunks) + { + cp_async_state_chunk_stage( + s_state, state, slot, i_hv, state_slot_stride, prefetch, prefetch % kStageCount); + } + } + + int const v_row_a = warp; + int const v_row_b = warp + kWarps; + int const v_row_c = warp + 2 * kWarps; + int const v_row_d = warp + 3 * kWarps; + int const v0 = chunk * kChunkV + v_row_a; + int const v1 = chunk * kChunkV + v_row_b; + int const v2 = chunk * kChunkV + v_row_c; + int const v3 = chunk * kChunkV + v_row_d; + float h_a_vals[4]; + float h_b_vals[4]; + float h_c_vals[4]; + float h_d_vals[4]; + float dot_hk_a = 0.0f; + float dot_hk_b = 0.0f; + float dot_hk_c = 0.0f; + float dot_hk_d = 0.0f; + + float const* state_stage = s_state + (chunk % kStageCount) * kStageChunkV * kDimK; + const float4 raw_h_a = *reinterpret_cast(state_stage + v_row_a * kDimK + k_base); + const float4 raw_h_b = *reinterpret_cast(state_stage + v_row_b * kDimK + k_base); + const float4 raw_h_c = *reinterpret_cast(state_stage + v_row_c * kDimK + k_base); + const float4 raw_h_d = *reinterpret_cast(state_stage + v_row_d * kDimK + k_base); + h_a_vals[0] = raw_h_a.x * r_decay[0]; + h_a_vals[1] = raw_h_a.y * r_decay[1]; + h_a_vals[2] = raw_h_a.z * r_decay[2]; + h_a_vals[3] = raw_h_a.w * r_decay[3]; + h_b_vals[0] = raw_h_b.x * r_decay[0]; + h_b_vals[1] = raw_h_b.y * r_decay[1]; + h_b_vals[2] = raw_h_b.z * r_decay[2]; + h_b_vals[3] = raw_h_b.w * r_decay[3]; + h_c_vals[0] = raw_h_c.x * r_decay[0]; + h_c_vals[1] = raw_h_c.y * r_decay[1]; + h_c_vals[2] = raw_h_c.z * r_decay[2]; + h_c_vals[3] = raw_h_c.w * r_decay[3]; + h_d_vals[0] = raw_h_d.x * r_decay[0]; + h_d_vals[1] = raw_h_d.y * r_decay[1]; + h_d_vals[2] = raw_h_d.z * r_decay[2]; + h_d_vals[3] = raw_h_d.w * r_decay[3]; + dot_hk_a = h_a_vals[0] * r_k[0] + h_a_vals[1] * r_k[1] + h_a_vals[2] * r_k[2] + h_a_vals[3] * r_k[3]; + dot_hk_b = h_b_vals[0] * r_k[0] + h_b_vals[1] * r_k[1] + h_b_vals[2] * r_k[2] + h_b_vals[3] * r_k[3]; + dot_hk_c = h_c_vals[0] * r_k[0] + h_c_vals[1] * r_k[1] + h_c_vals[2] * r_k[2] + h_c_vals[3] * r_k[3]; + dot_hk_d = h_d_vals[0] * r_k[0] + h_d_vals[1] * r_k[1] + h_d_vals[2] * r_k[2] + h_d_vals[3] * r_k[3]; + + const Sum4 dot_hk = warp_reduce_sum4(dot_hk_a, dot_hk_b, dot_hk_c, dot_hk_d); + float const v_new0 = (s_v[v0] - dot_hk.a) * s_beta; + float const v_new1 = (s_v[v1] - dot_hk.b) * s_beta; + float const v_new2 = (s_v[v2] - dot_hk.c) * s_beta; + float const v_new3 = (s_v[v3] - dot_hk.d) * s_beta; + + float dot_hq_a = 0.0f; + float dot_hq_b = 0.0f; + float dot_hq_c = 0.0f; + float dot_hq_d = 0.0f; + int64_t const state_head_offset + = static_cast(slot) * state_slot_stride + static_cast(i_hv) * kDimV * kDimK; + int64_t const state_idx_a = state_head_offset + static_cast(v0) * kDimK + k_base; + int64_t const state_idx_b = state_head_offset + static_cast(v1) * kDimK + k_base; + int64_t const state_idx_c = state_head_offset + static_cast(v2) * kDimK + k_base; + int64_t const state_idx_d = state_head_offset + static_cast(v3) * kDimK + k_base; + float const h_a_0 = h_a_vals[0] + r_k[0] * v_new0; + float const h_a_1 = h_a_vals[1] + r_k[1] * v_new0; + float const h_a_2 = h_a_vals[2] + r_k[2] * v_new0; + float const h_a_3 = h_a_vals[3] + r_k[3] * v_new0; + float const h_b_0 = h_b_vals[0] + r_k[0] * v_new1; + float const h_b_1 = h_b_vals[1] + r_k[1] * v_new1; + float const h_b_2 = h_b_vals[2] + r_k[2] * v_new1; + float const h_b_3 = h_b_vals[3] + r_k[3] * v_new1; + float const h_c_0 = h_c_vals[0] + r_k[0] * v_new2; + float const h_c_1 = h_c_vals[1] + r_k[1] * v_new2; + float const h_c_2 = h_c_vals[2] + r_k[2] * v_new2; + float const h_c_3 = h_c_vals[3] + r_k[3] * v_new2; + float const h_d_0 = h_d_vals[0] + r_k[0] * v_new3; + float const h_d_1 = h_d_vals[1] + r_k[1] * v_new3; + float const h_d_2 = h_d_vals[2] + r_k[2] * v_new3; + float const h_d_3 = h_d_vals[3] + r_k[3] * v_new3; + if constexpr (kComputeOutputBeforeStore) + { + dot_hq_a = h_a_0 * r_q[0] + h_a_1 * r_q[1] + h_a_2 * r_q[2] + h_a_3 * r_q[3]; + dot_hq_b = h_b_0 * r_q[0] + h_b_1 * r_q[1] + h_b_2 * r_q[2] + h_b_3 * r_q[3]; + dot_hq_c = h_c_0 * r_q[0] + h_c_1 * r_q[1] + h_c_2 * r_q[2] + h_c_3 * r_q[3]; + dot_hq_d = h_d_0 * r_q[0] + h_d_1 * r_q[1] + h_d_2 * r_q[2] + h_d_3 * r_q[3]; + store_state_float4(state + state_idx_a, make_float4(h_a_0, h_a_1, h_a_2, h_a_3)); + store_state_float4(state + state_idx_b, make_float4(h_b_0, h_b_1, h_b_2, h_b_3)); + store_state_float4(state + state_idx_c, make_float4(h_c_0, h_c_1, h_c_2, h_c_3)); + store_state_float4(state + state_idx_d, make_float4(h_d_0, h_d_1, h_d_2, h_d_3)); + } + else + { + store_state_float4(state + state_idx_a, make_float4(h_a_0, h_a_1, h_a_2, h_a_3)); + store_state_float4(state + state_idx_b, make_float4(h_b_0, h_b_1, h_b_2, h_b_3)); + store_state_float4(state + state_idx_c, make_float4(h_c_0, h_c_1, h_c_2, h_c_3)); + store_state_float4(state + state_idx_d, make_float4(h_d_0, h_d_1, h_d_2, h_d_3)); + dot_hq_a = h_a_0 * r_q[0] + h_a_1 * r_q[1] + h_a_2 * r_q[2] + h_a_3 * r_q[3]; + dot_hq_b = h_b_0 * r_q[0] + h_b_1 * r_q[1] + h_b_2 * r_q[2] + h_b_3 * r_q[3]; + dot_hq_c = h_c_0 * r_q[0] + h_c_1 * r_q[1] + h_c_2 * r_q[2] + h_c_3 * r_q[3]; + dot_hq_d = h_d_0 * r_q[0] + h_d_1 * r_q[1] + h_d_2 * r_q[2] + h_d_3 * r_q[3]; + } + + const Sum4 dot_hq = warp_reduce_sum4(dot_hq_a, dot_hq_b, dot_hq_c, dot_hq_d); + if (lane == 0) + { + s_o[v0] = dot_hq.a; + s_o[v1] = dot_hq.b; + s_o[v2] = dot_hq.c; + s_o[v3] = dot_hq.d; + if constexpr (kApplyOnorm && kAccumulateOnormSumsq) + { + o_sumsq += dot_hq.a * dot_hq.a + dot_hq.b * dot_hq.b + dot_hq.c * dot_hq.c + dot_hq.d * dot_hq.d; + } + } + } + __syncthreads(); + + if constexpr (kApplyOnorm) + { + if constexpr (kAccumulateOnormSumsq) + { + if (lane == 0) + { + s_reduce[warp] = o_sumsq; + } + __syncthreads(); + + float total_sumsq = 0.0f; + if (warp == 0) + { + total_sumsq = lane < kWarps ? s_reduce[lane] : 0.0f; + total_sumsq = warp_reduce_sum(total_sumsq); + if (lane == 0) + { + s_reduce[0] = total_sumsq; + } + } + __syncthreads(); + + if (tid < kDimV) + { + int64_t const out_idx = output_head * kDimV + tid; + float const raw_o = s_o[tid]; + float const rstd = rsqrtf(s_reduce[0] / static_cast(kDimV) + onorm_eps); + float gate; + float weight; + if constexpr (kPreloadOnormParams) + { + gate = pre_onorm_gate; + weight = pre_onorm_weight; + } + else + { + gate = sigmoid_fast(bf16_load(onorm_g, out_idx)); + weight = onorm_weight[tid]; + } + float const y = raw_o * rstd * weight * gate; + out[out_idx] = bf16_store(y); + } + } + else + { + float const raw_o = tid < kDimV ? s_o[tid] : 0.0f; + float const o_sq = raw_o * raw_o; + float const sumsq = block_reduce_sum(o_sq, s_reduce); + + if (tid < kDimV) + { + int64_t const out_idx = output_head * kDimV + tid; + float const rstd = rsqrtf(sumsq / static_cast(kDimV) + onorm_eps); + float gate; + float weight; + if constexpr (kPreloadOnormParams) + { + gate = pre_onorm_gate; + weight = pre_onorm_weight; + } + else + { + gate = sigmoid_fast(bf16_load(onorm_g, out_idx)); + weight = onorm_weight[tid]; + } + float const y = raw_o * rstd * weight * gate; + out[out_idx] = bf16_store(y); + } + } + } + else + { + if (tid < kDimV) + { + int64_t const out_idx = output_head * kDimV + tid; + out[out_idx] = bf16_store(s_o[tid]); + } + } +} + +template +__global__ __launch_bounds__(kThreads, 2) void kda_decode_fusion_many_heads_kernel( + __nv_bfloat16 const* __restrict__ x_q, __nv_bfloat16 const* __restrict__ x_k, __nv_bfloat16 const* __restrict__ x_v, + __nv_bfloat16 const* __restrict__ w_q_t, __nv_bfloat16 const* __restrict__ w_k_t, + __nv_bfloat16 const* __restrict__ w_v_t, __nv_bfloat16 const* __restrict__ bias_q, + __nv_bfloat16 const* __restrict__ bias_k, __nv_bfloat16 const* __restrict__ bias_v, + __nv_bfloat16* __restrict__ cs_q, __nv_bfloat16* __restrict__ cs_k, __nv_bfloat16* __restrict__ cs_v, + float const* __restrict__ a_log, __nv_bfloat16 const* __restrict__ g, float const* __restrict__ dt_bias, + __nv_bfloat16 const* __restrict__ beta, __nv_bfloat16 const* __restrict__ onorm_g, + float const* __restrict__ onorm_weight, int const* __restrict__ ssm_state_indices, + int const* __restrict__ cu_seqlens, float* __restrict__ state, int64_t state_slot_stride, + __nv_bfloat16* __restrict__ out, int B, int H, int HV, float lower_bound, float scale, float onorm_eps) +{ + int const tid = threadIdx.x; + int const lane = tid & 31; + int const warp = tid >> 5; + int i_n; + int i_hv; + int i_h; + int bos; + int slot; + if constexpr (kUseStaticDecodeLayout) + { + if constexpr (kUseHeadGrid) + { + i_n = blockIdx.x; + i_hv = blockIdx.y; + } + else + { + int const nhv = blockIdx.x; + i_n = nhv / kFixedValueHeads; + i_hv = nhv - i_n * kFixedValueHeads; + } + i_h = i_hv; + bos = i_n; + slot = i_n; + } + else + { + int const nhv = blockIdx.x; + i_n = nhv / HV; + i_hv = nhv - i_n * HV; + int const hv_per_h = HV / H; + i_h = i_hv / hv_per_h; + + bos = cu_seqlens == nullptr ? i_n : cu_seqlens[i_n]; + int const eos = cu_seqlens == nullptr ? i_n + 1 : cu_seqlens[i_n + 1]; + if (eos <= bos) + { + return; + } + slot = ssm_state_indices[i_n]; + } + + int const hk_off = i_h * kDimK; + int const hv_off = i_hv * kDimV; + int const h_count = kUseStaticDecodeLayout ? kFixedHeads : H; + int const hv_count = kUseStaticDecodeLayout ? kFixedValueHeads : HV; + int const hkv_dim = h_count * kDimK; + int const hvv_dim = hv_count * kDimV; + int const conv_slot = kUpdateConvState ? slot : i_n; + int64_t const qk_input_head = static_cast(bos) * h_count + i_h; + int64_t const value_input_head = static_cast(bos) * hv_count + i_hv; + int64_t const output_head = static_cast(i_n) * hv_count + i_hv; + + __shared__ float s_state[2][kChunkV][kDimK]; + __shared__ float s_q[kDimK]; + __shared__ float s_k[kDimK]; + __shared__ float s_decay[kDimK]; + __shared__ float s_v[kDimV]; + __shared__ float s_o[kDimV]; + __shared__ float s_reduce[kThreads]; + __shared__ float s_beta; + float pre_onorm_gate = 0.0f; + float pre_onorm_weight = 0.0f; + + cp_async_state_chunk(&s_state[0][0][0], state, slot, i_hv, state_slot_stride, 0); + + if constexpr (kUpdateConvState) + { + if (tid < kDimK) + { + int const k = tid; + int const hk = hk_off + k; + int64_t const cs_base = static_cast(slot) * hkv_dim * kConvStateWidth + hk; + int64_t const xq_idx = qk_input_head * kDimK + k; + float const exp_a = __shfl_sync(0xffffffffu, lane == 0 ? __expf(a_log[i_h]) : 0.0f, 0); + + float q_acc = bf16_load(bias_q, hk); + float k_acc = bf16_load(bias_k, hk); + __nv_bfloat16 q_shift0 = __float2bfloat16(0.0f); + __nv_bfloat16 q_shift1 = __float2bfloat16(0.0f); + __nv_bfloat16 k_shift0 = __float2bfloat16(0.0f); + __nv_bfloat16 k_shift1 = __float2bfloat16(0.0f); +#pragma unroll + for (int w = 0; w < kConvStateWidth; ++w) + { + const __nv_bfloat16 q_state = cs_q[cs_base + w * hkv_dim]; + const __nv_bfloat16 k_state = cs_k[cs_base + w * hkv_dim]; + q_acc += __bfloat162float(q_state) * bf16_load(w_q_t, w * hkv_dim + hk); + k_acc += __bfloat162float(k_state) * bf16_load(w_k_t, w * hkv_dim + hk); + if (w == 1) + { + q_shift0 = q_state; + k_shift0 = k_state; + } + else if (w == 2) + { + q_shift1 = q_state; + k_shift1 = k_state; + } + } + const __nv_bfloat16 q_new = x_q[xq_idx]; + const __nv_bfloat16 k_new = x_k[xq_idx]; + q_acc += __bfloat162float(q_new) * bf16_load(w_q_t, (kKernelWidth - 1) * hkv_dim + hk); + k_acc += __bfloat162float(k_new) * bf16_load(w_k_t, (kKernelWidth - 1) * hkv_dim + hk); + + cs_q[cs_base + 0] = q_shift0; + cs_q[cs_base + hkv_dim] = q_shift1; + cs_q[cs_base + 2 * hkv_dim] = q_new; + cs_k[cs_base + 0] = k_shift0; + cs_k[cs_base + hkv_dim] = k_shift1; + cs_k[cs_base + 2 * hkv_dim] = k_new; + + s_q[k] = silu_fast(q_acc); + s_k[k] = silu_fast(k_acc); + + float const g_raw = bf16_load(g, value_input_head * kDimK + k) + dt_bias[hk]; + if constexpr (kUseLowerBound) + { + s_decay[k] = __expf(lower_bound * sigmoid_fast(exp_a * g_raw)); + } + else + { + s_decay[k] = __expf(-exp_a * softplus_fast(g_raw)); + } + } + } + else + { + if (tid < kDimK) + { + int const k = tid; + int const hk = hk_off + k; + float const exp_a = __shfl_sync(0xffffffffu, lane == 0 ? __expf(a_log[i_h]) : 0.0f, 0); + + float q_acc = bf16_load(bias_q, hk); + float k_acc = bf16_load(bias_k, hk); +#pragma unroll + for (int w = 0; w < kConvStateWidth; ++w) + { + int64_t const cs_idx = (static_cast(conv_slot) * hkv_dim + hk) * kConvStateWidth + w; + q_acc += bf16_load(cs_q, cs_idx) * bf16_load(w_q_t, w * hkv_dim + hk); + k_acc += bf16_load(cs_k, cs_idx) * bf16_load(w_k_t, w * hkv_dim + hk); + } + q_acc += bf16_load(x_q, qk_input_head * kDimK + k) * bf16_load(w_q_t, (kKernelWidth - 1) * hkv_dim + hk); + k_acc += bf16_load(x_k, qk_input_head * kDimK + k) * bf16_load(w_k_t, (kKernelWidth - 1) * hkv_dim + hk); + + s_q[k] = silu_fast(q_acc); + s_k[k] = silu_fast(k_acc); + + float const g_raw = bf16_load(g, value_input_head * kDimK + k) + dt_bias[hk]; + if constexpr (kUseLowerBound) + { + s_decay[k] = __expf(lower_bound * sigmoid_fast(exp_a * g_raw)); + } + else + { + s_decay[k] = __expf(-exp_a * softplus_fast(g_raw)); + } + } + } + + if constexpr (kUpdateConvState) + { + if (tid < kDimV) + { + int const v = tid; + int const hvv = hv_off + v; + int64_t const cs_base = static_cast(slot) * hvv_dim * kConvStateWidth + hvv; + int64_t const xv_idx = value_input_head * kDimV + v; + + float v_acc = bf16_load(bias_v, hvv); + __nv_bfloat16 v_shift0 = __float2bfloat16(0.0f); + __nv_bfloat16 v_shift1 = __float2bfloat16(0.0f); +#pragma unroll + for (int w = 0; w < kConvStateWidth; ++w) + { + const __nv_bfloat16 v_state = cs_v[cs_base + w * hvv_dim]; + v_acc += __bfloat162float(v_state) * bf16_load(w_v_t, w * hvv_dim + hvv); + if (w == 1) + { + v_shift0 = v_state; + } + else if (w == 2) + { + v_shift1 = v_state; + } + } + const __nv_bfloat16 v_new = x_v[xv_idx]; + v_acc += __bfloat162float(v_new) * bf16_load(w_v_t, (kKernelWidth - 1) * hvv_dim + hvv); + cs_v[cs_base + 0] = v_shift0; + cs_v[cs_base + hvv_dim] = v_shift1; + cs_v[cs_base + 2 * hvv_dim] = v_new; + s_v[v] = silu_fast(v_acc); + + if constexpr (kApplyOnorm && kPreloadOnormParams) + { + int64_t const out_idx = output_head * kDimV + v; + pre_onorm_gate = sigmoid_fast(bf16_load(onorm_g, out_idx)); + pre_onorm_weight = onorm_weight[v]; + } + } + } + else + { + if (tid < kDimV) + { + int const v = tid; + int const hvv = hv_off + v; + + float v_acc = bf16_load(bias_v, hvv); +#pragma unroll + for (int w = 0; w < kConvStateWidth; ++w) + { + int64_t const cs_idx = (static_cast(conv_slot) * hvv_dim + hvv) * kConvStateWidth + w; + v_acc += bf16_load(cs_v, cs_idx) * bf16_load(w_v_t, w * hvv_dim + hvv); + } + v_acc + += bf16_load(x_v, value_input_head * kDimV + v) * bf16_load(w_v_t, (kKernelWidth - 1) * hvv_dim + hvv); + s_v[v] = silu_fast(v_acc); + + if constexpr (kApplyOnorm && kPreloadOnormParams) + { + int64_t const out_idx = output_head * kDimV + v; + pre_onorm_gate = sigmoid_fast(bf16_load(onorm_g, out_idx)); + pre_onorm_weight = onorm_weight[v]; + } + } + } + + if (tid == 0) + { + float const beta_raw = bf16_load(beta, value_input_head); + if constexpr (kApplyBetaSigmoid) + { + s_beta = sigmoid_fast(beta_raw); + } + else + { + s_beta = beta_raw; + } + } + __syncthreads(); + + if constexpr (kPrefetchNextStateChunk && kNumChunks > 1) + { + cp_async_state_chunk(&s_state[0][0][0], state, slot, i_hv, state_slot_stride, 1); + } + + float const q_sq = tid < kDimK ? s_q[tid] * s_q[tid] : 0.0f; + float const k_sq = tid < kDimK ? s_k[tid] * s_k[tid] : 0.0f; + Sum2 qk_sum; + if constexpr (kUseActiveQkReduction) + { + qk_sum = block_reduce_sum2_active_for(q_sq, k_sq, s_reduce); + } + else + { + qk_sum = block_reduce_sum2(q_sq, k_sq, s_reduce); + } + if (tid < kDimK) + { + s_q[tid] *= rsqrtf(qk_sum.x + 1.0e-6f) * scale; + s_k[tid] *= rsqrtf(qk_sum.y + 1.0e-6f); + } + __syncthreads(); + + int const k_base = lane * 4; + const float4 q4 = *reinterpret_cast(s_q + k_base); + const float4 k4 = *reinterpret_cast(s_k + k_base); + const float4 decay4 = *reinterpret_cast(s_decay + k_base); + float r_q[4] = {q4.x, q4.y, q4.z, q4.w}; + float r_k[4] = {k4.x, k4.y, k4.z, k4.w}; + float r_decay[4] = {decay4.x, decay4.y, decay4.z, decay4.w}; + float o_sumsq = 0.0f; + +#pragma unroll + for (int chunk = 0; chunk < kNumChunks; ++chunk) + { + if constexpr (kPrefetchNextStateChunk && kNumChunks > 1) + { + if (chunk + 1 < kNumChunks) + { + cp_async_wait_group_1(); + } + else + { + cp_async_wait_all(); + } + } + else + { + cp_async_wait_all(); + } + if constexpr (!kSkipWarpSync) + { + __syncwarp(); + } + + if constexpr (!kPrefetchNextStateChunk) + { + if (chunk + 1 < kNumChunks) + { + cp_async_state_chunk(&s_state[0][0][0], state, slot, i_hv, state_slot_stride, chunk + 1); + } + } + +#pragma unroll + for (int row = 0; row < kRowsPerWarp; row += 2) + { + int const v_row_a = warp + row * kWarps; + int const v_row_b = warp + (row + 1) * kWarps; + int const v0 = chunk * kChunkV + v_row_a; + int const v1 = chunk * kChunkV + v_row_b; + float h_a_vals[4]; + float h_b_vals[4]; + float dot_hk_a = 0.0f; + float dot_hk_b = 0.0f; + + const float4 raw_h_a = *reinterpret_cast(&s_state[chunk & 1][v_row_a][k_base]); + const float4 raw_h_b = *reinterpret_cast(&s_state[chunk & 1][v_row_b][k_base]); + h_a_vals[0] = raw_h_a.x * r_decay[0]; + h_a_vals[1] = raw_h_a.y * r_decay[1]; + h_a_vals[2] = raw_h_a.z * r_decay[2]; + h_a_vals[3] = raw_h_a.w * r_decay[3]; + h_b_vals[0] = raw_h_b.x * r_decay[0]; + h_b_vals[1] = raw_h_b.y * r_decay[1]; + h_b_vals[2] = raw_h_b.z * r_decay[2]; + h_b_vals[3] = raw_h_b.w * r_decay[3]; + dot_hk_a = h_a_vals[0] * r_k[0] + h_a_vals[1] * r_k[1] + h_a_vals[2] * r_k[2] + h_a_vals[3] * r_k[3]; + dot_hk_b = h_b_vals[0] * r_k[0] + h_b_vals[1] * r_k[1] + h_b_vals[2] * r_k[2] + h_b_vals[3] * r_k[3]; + + const Sum2 dot_hk = warp_reduce_sum_pair(dot_hk_a, dot_hk_b); + float const v_new0 = (s_v[v0] - dot_hk.x) * s_beta; + float const v_new1 = (s_v[v1] - dot_hk.y) * s_beta; + + float dot_hq_a = 0.0f; + float dot_hq_b = 0.0f; + int64_t const state_head_offset + = static_cast(slot) * state_slot_stride + static_cast(i_hv) * kDimV * kDimK; + int64_t const state_idx_a = state_head_offset + static_cast(v0) * kDimK + k_base; + int64_t const state_idx_b = state_head_offset + static_cast(v1) * kDimK + k_base; + float const h_a_0 = h_a_vals[0] + r_k[0] * v_new0; + float const h_a_1 = h_a_vals[1] + r_k[1] * v_new0; + float const h_a_2 = h_a_vals[2] + r_k[2] * v_new0; + float const h_a_3 = h_a_vals[3] + r_k[3] * v_new0; + float const h_b_0 = h_b_vals[0] + r_k[0] * v_new1; + float const h_b_1 = h_b_vals[1] + r_k[1] * v_new1; + float const h_b_2 = h_b_vals[2] + r_k[2] * v_new1; + float const h_b_3 = h_b_vals[3] + r_k[3] * v_new1; + if constexpr (kComputeOutputBeforeStore) + { + dot_hq_a = h_a_0 * r_q[0] + h_a_1 * r_q[1] + h_a_2 * r_q[2] + h_a_3 * r_q[3]; + dot_hq_b = h_b_0 * r_q[0] + h_b_1 * r_q[1] + h_b_2 * r_q[2] + h_b_3 * r_q[3]; + store_state_float4(state + state_idx_a, make_float4(h_a_0, h_a_1, h_a_2, h_a_3)); + store_state_float4(state + state_idx_b, make_float4(h_b_0, h_b_1, h_b_2, h_b_3)); + } + else + { + store_state_float4(state + state_idx_a, make_float4(h_a_0, h_a_1, h_a_2, h_a_3)); + store_state_float4(state + state_idx_b, make_float4(h_b_0, h_b_1, h_b_2, h_b_3)); + dot_hq_a = h_a_0 * r_q[0] + h_a_1 * r_q[1] + h_a_2 * r_q[2] + h_a_3 * r_q[3]; + dot_hq_b = h_b_0 * r_q[0] + h_b_1 * r_q[1] + h_b_2 * r_q[2] + h_b_3 * r_q[3]; + } + + const Sum2 dot_hq = warp_reduce_sum_pair(dot_hq_a, dot_hq_b); + if (lane == 0) + { + s_o[v0] = dot_hq.x; + s_o[v1] = dot_hq.y; + if constexpr (kApplyOnorm && kAccumulateOnormSumsq) + { + o_sumsq += dot_hq.x * dot_hq.x + dot_hq.y * dot_hq.y; + } + } + } + + if constexpr (kPrefetchNextStateChunk) + { + if (chunk + 2 < kNumChunks) + { + cp_async_state_chunk(&s_state[0][0][0], state, slot, i_hv, state_slot_stride, chunk + 2); + } + } + } + __syncthreads(); + + if constexpr (kApplyOnorm) + { + if constexpr (kAccumulateOnormSumsq) + { + if (lane == 0) + { + s_reduce[warp] = o_sumsq; + } + __syncthreads(); + + float total_sumsq = 0.0f; + if (warp == 0) + { + total_sumsq = lane < kWarps ? s_reduce[lane] : 0.0f; + total_sumsq = warp_reduce_sum(total_sumsq); + if (lane == 0) + { + s_reduce[0] = total_sumsq; + } + } + __syncthreads(); + + if (tid < kDimV) + { + int64_t const out_idx = output_head * kDimV + tid; + float const raw_o = s_o[tid]; + float const rstd = rsqrtf(s_reduce[0] / static_cast(kDimV) + onorm_eps); + float gate; + float weight; + if constexpr (kPreloadOnormParams) + { + gate = pre_onorm_gate; + weight = pre_onorm_weight; + } + else + { + gate = sigmoid_fast(bf16_load(onorm_g, out_idx)); + weight = onorm_weight[tid]; + } + float const y = raw_o * rstd * weight * gate; + out[out_idx] = bf16_store(y); + } + } + else + { + float const raw_o = tid < kDimV ? s_o[tid] : 0.0f; + float const o_sq = raw_o * raw_o; + float sumsq; + if constexpr (kUseActiveOnormReduction || kUseActiveQkReduction) + { + sumsq = block_reduce_sum_active_for(o_sq, s_reduce); + } + else + { + sumsq = block_reduce_sum(o_sq, s_reduce); + } + + if (tid < kDimV) + { + int64_t const out_idx = output_head * kDimV + tid; + float const rstd = rsqrtf(sumsq / static_cast(kDimV) + onorm_eps); + float gate; + float weight; + if constexpr (kPreloadOnormParams) + { + gate = pre_onorm_gate; + weight = pre_onorm_weight; + } + else + { + gate = sigmoid_fast(bf16_load(onorm_g, out_idx)); + weight = onorm_weight[tid]; + } + float const y = raw_o * rstd * weight * gate; + out[out_idx] = bf16_store(y); + } + } + } + else + { + if (tid < kDimV) + { + int64_t const out_idx = output_head * kDimV + tid; + out[out_idx] = bf16_store(s_o[tid]); + } + } +} + +struct KdaDecodeLaunchParams +{ + void const* x_q; + void const* x_k; + void const* x_v; + void const* w_q_t; + void const* w_k_t; + void const* w_v_t; + void const* bias_q; + void const* bias_k; + void const* bias_v; + void* cs_q; + void* cs_k; + void* cs_v; + float const* a_log; + void const* g; + float const* dt_bias; + void const* beta; + void const* onorm_g; + float const* onorm_weight; + int const* ssm_state_indices; + int const* cu_seqlens; + float* state; + int64_t state_slot_stride; + void* out; + int B; + int H; + int HV; + bool apply_onorm; + bool update_conv_cache; + bool use_lower_bound; + bool apply_beta_sigmoid; + float lower_bound; + float scale; + float onorm_eps; + cudaStream_t stream; +}; + +template +void launch_kda_decode_compact_heads_raw(KdaDecodeLaunchParams const& p) +{ + constexpr int kStageDynamicSmemBytes = 3 * 32 * kDimK * static_cast(sizeof(float)); + TLLM_CUDA_CHECK(cudaFuncSetAttribute( + kda_decode_fusion_compact_heads_kernel, + cudaFuncAttributeMaxDynamicSharedMemorySize, kStageDynamicSmemBytes)); + kda_decode_fusion_compact_heads_kernel + <<>>( + reinterpret_cast<__nv_bfloat16 const*>(p.x_q), reinterpret_cast<__nv_bfloat16 const*>(p.x_k), + reinterpret_cast<__nv_bfloat16 const*>(p.x_v), reinterpret_cast<__nv_bfloat16 const*>(p.w_q_t), + reinterpret_cast<__nv_bfloat16 const*>(p.w_k_t), reinterpret_cast<__nv_bfloat16 const*>(p.w_v_t), + reinterpret_cast<__nv_bfloat16 const*>(p.bias_q), reinterpret_cast<__nv_bfloat16 const*>(p.bias_k), + reinterpret_cast<__nv_bfloat16 const*>(p.bias_v), reinterpret_cast<__nv_bfloat16*>(p.cs_q), + reinterpret_cast<__nv_bfloat16*>(p.cs_k), reinterpret_cast<__nv_bfloat16*>(p.cs_v), p.a_log, + reinterpret_cast<__nv_bfloat16 const*>(p.g), p.dt_bias, reinterpret_cast<__nv_bfloat16 const*>(p.beta), + reinterpret_cast<__nv_bfloat16 const*>(p.onorm_g), p.onorm_weight, p.ssm_state_indices, p.cu_seqlens, + p.state, p.state_slot_stride, reinterpret_cast<__nv_bfloat16*>(p.out), p.B, p.H, p.HV, p.lower_bound, + p.scale, p.onorm_eps); +} + +template +void launch_kda_decode_many_heads_raw(KdaDecodeLaunchParams const& p) +{ + constexpr bool kUseHeadGrid = kUseStaticDecodeLayout; + dim3 const grid = kUseStaticDecodeLayout ? dim3(p.B, p.HV) : dim3(p.B * p.HV); + kda_decode_fusion_many_heads_kernel + <<>>(reinterpret_cast<__nv_bfloat16 const*>(p.x_q), + reinterpret_cast<__nv_bfloat16 const*>(p.x_k), reinterpret_cast<__nv_bfloat16 const*>(p.x_v), + reinterpret_cast<__nv_bfloat16 const*>(p.w_q_t), reinterpret_cast<__nv_bfloat16 const*>(p.w_k_t), + reinterpret_cast<__nv_bfloat16 const*>(p.w_v_t), reinterpret_cast<__nv_bfloat16 const*>(p.bias_q), + reinterpret_cast<__nv_bfloat16 const*>(p.bias_k), reinterpret_cast<__nv_bfloat16 const*>(p.bias_v), + reinterpret_cast<__nv_bfloat16*>(p.cs_q), reinterpret_cast<__nv_bfloat16*>(p.cs_k), + reinterpret_cast<__nv_bfloat16*>(p.cs_v), p.a_log, reinterpret_cast<__nv_bfloat16 const*>(p.g), p.dt_bias, + reinterpret_cast<__nv_bfloat16 const*>(p.beta), reinterpret_cast<__nv_bfloat16 const*>(p.onorm_g), + p.onorm_weight, p.ssm_state_indices, p.cu_seqlens, p.state, p.state_slot_stride, + reinterpret_cast<__nv_bfloat16*>(p.out), p.B, p.H, p.HV, p.lower_bound, p.scale, p.onorm_eps); +} + +template +void launch_kda_decode_raw(KdaDecodeLaunchParams const& p) +{ + if constexpr (kCompact) + { + launch_kda_decode_compact_heads_raw(p); + } + else + { + launch_kda_decode_many_heads_raw(p); + } +} + +template +void launch_kda_decode_selected_backend(KdaDecodeLaunchParams const& p) +{ + if (p.update_conv_cache) + { + launch_kda_decode_raw(p); + } + else + { + launch_kda_decode_raw(p); + } +} + +template +void dispatch_kda_decode_beta(KdaDecodeLaunchParams const& p) +{ + if (p.apply_beta_sigmoid) + { + launch_kda_decode_selected_backend( + p); + } + else + { + launch_kda_decode_selected_backend(p); + } +} + +template +void dispatch_kda_decode_decay(KdaDecodeLaunchParams const& p) +{ + if (p.use_lower_bound) + { + dispatch_kda_decode_beta(p); + } + else + { + dispatch_kda_decode_beta(p); + } +} + +template +void dispatch_kda_decode_features(KdaDecodeLaunchParams const& p) +{ + if (p.apply_onorm) + { + dispatch_kda_decode_decay(p); + } + else + { + dispatch_kda_decode_decay(p); + } +} + +template +void dispatch_kda_decode_layout(KdaDecodeLaunchParams const& p) +{ + if (p.ssm_state_indices == nullptr) + { + dispatch_kda_decode_features(p); + } + else + { + dispatch_kda_decode_features(p); + } +} + +template +void dispatch_kda_decode_heads(KdaDecodeLaunchParams const& p) +{ + switch (p.H) + { + case 1: dispatch_kda_decode_layout(p); break; + case 2: dispatch_kda_decode_layout(p); break; + case 3: dispatch_kda_decode_layout(p); break; + case 4: dispatch_kda_decode_layout(p); break; + case 6: dispatch_kda_decode_layout(p); break; + case 8: dispatch_kda_decode_layout(p); break; + case 12: dispatch_kda_decode_layout(p); break; + case 16: dispatch_kda_decode_layout(p); break; + case 24: dispatch_kda_decode_layout(p); break; + case 32: dispatch_kda_decode_layout(p); break; + case 48: dispatch_kda_decode_layout(p); break; + case 96: dispatch_kda_decode_layout(p); break; + default: + if constexpr (kCompact) + { + TLLM_CHECK_WITH_INFO(false, "KDA compact-heads decode does not support numHeads=%d", p.H); + } + else + { + TLLM_CHECK_WITH_INFO(false, "KDA decode does not support numHeads=%d", p.H); + } + } +} + +} // namespace + +void invokeKdaDecode(KdaDecodeParams const& params, cudaStream_t stream) +{ + TLLM_CHECK_WITH_INFO(params.numHeads == params.numValueHeads, "KDA decode requires numHeads == numValueHeads"); + bool const useCompactHeads = shouldUseCompactHeads(params.batchSize, params.numHeads, params.numValueHeads); + KdaDecodeLaunchParams const launchParams{params.xQ, params.xK, params.xV, params.wQT, params.wKT, params.wVT, + params.biasQ, params.biasK, params.biasV, params.convStateQ, params.convStateK, params.convStateV, params.logA, + params.gate, params.dtBias, params.beta, params.outputNormGate, params.outputNormWeight, params.ssmStateIndices, + params.cuSeqlens, params.state, params.stateSlotStride, params.output, params.batchSize, params.numHeads, + params.numValueHeads, params.applyOutputNorm, params.updateConvCache, params.useLowerBound, + params.applyBetaSigmoid, params.lowerBound, params.scale, params.outputNormEps, stream}; + if (useCompactHeads) + { + dispatch_kda_decode_heads(launchParams); + } + else + { + dispatch_kda_decode_heads(launchParams); + } + TLLM_CUDA_CHECK(cudaGetLastError()); +} + +} // namespace kernels::kdaDecode + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/kdaDecode/kdaDecode.h b/cpp/tensorrt_llm/kernels/kdaDecode/kdaDecode.h new file mode 100644 index 000000000000..4d030ce8e4f5 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/kdaDecode/kdaDecode.h @@ -0,0 +1,89 @@ +/* + * Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels::kdaDecode +{ + +constexpr int kCompactHeadsWorkThreshold = 144; + +constexpr bool isSupportedHeadCount(int numHeads) +{ + return numHeads == 1 || numHeads == 2 || numHeads == 3 || numHeads == 4 || numHeads == 6 || numHeads == 8 + || numHeads == 12 || numHeads == 16 || numHeads == 24 || numHeads == 32 || numHeads == 48 || numHeads == 96; +} + +//! Select the compact-head kernel within the measured KDA decode work threshold. +//! Division keeps the B*H threshold overflow-safe. +constexpr bool shouldUseCompactHeads(int batchSize, int numHeads, int numValueHeads) +{ + return batchSize > 0 && numHeads == numValueHeads && isSupportedHeadCount(numHeads) + && batchSize <= kCompactHeadsWorkThreshold / numHeads; +} + +//! Parameters for the fused, single-token KDA decode kernel. +struct KdaDecodeParams +{ + void const* xQ; + void const* xK; + void const* xV; + void const* wQT; + void const* wKT; + void const* wVT; + void const* biasQ; + void const* biasK; + void const* biasV; + void* convStateQ; + void* convStateK; + void* convStateV; + float const* logA; // Named differently from a_log to satisfy codespell. + void const* gate; + float const* dtBias; + void const* beta; + void const* outputNormGate; + float const* outputNormWeight; + int const* ssmStateIndices; + //! Must be arange(batchSize + 1): the kernel only advances each state by one token. + int const* cuSeqlens; + float* state; + int64_t stateSlotStride; + void* output; + int batchSize; + int numHeads; + int numValueHeads; + bool applyOutputNorm; + bool updateConvCache; + bool useLowerBound; + bool applyBetaSigmoid; + float lowerBound; + float scale; + float outputNormEps; +}; + +//! Launches the tuned KDA decode kernel on the supplied CUDA stream. +void invokeKdaDecode(KdaDecodeParams const& params, cudaStream_t stream); + +} // namespace kernels::kdaDecode + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/kimiK3AttnRes/CMakeLists.txt b/cpp/tensorrt_llm/kernels/kimiK3AttnRes/CMakeLists.txt new file mode 100644 index 000000000000..d282ac9ee3f8 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/kimiK3AttnRes/CMakeLists.txt @@ -0,0 +1,31 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. +# All rights reserved. SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may obtain a copy of +# the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations under +# the License. + +add_library(kimi_k3_attn_res_src OBJECT attnResFwd.cu) + +# The kernel is warp-specialized for the SM100 (datacenter Blackwell) family and +# uses tcgen05/TMEM PTX instructions that ptxas rejects for other architectures +# (e.g. sm_120f), so pin the target to the sm_100 family instead of inheriting +# the global architecture list. On builds without any sm_100-family architecture +# the target still compiles (the tcgen05 code paths are guarded by +# __CUDA_ARCH__), and the Torch-op bridge rejects unsupported devices at +# runtime. +set_cuda_architectures(kimi_k3_attn_res_src 100f) + +set_property(TARGET kimi_k3_attn_res_src PROPERTY POSITION_INDEPENDENT_CODE ON) +set_property(TARGET kimi_k3_attn_res_src PROPERTY CUDA_RESOLVE_DEVICE_SYMBOLS + ON) +target_compile_options(kimi_k3_attn_res_src + PRIVATE $<$:--use_fast_math>) diff --git a/cpp/tensorrt_llm/kernels/kimiK3AttnRes/attnResFwd.cu b/cpp/tensorrt_llm/kernels/kimiK3AttnRes/attnResFwd.cu new file mode 100644 index 000000000000..f1e284118d6f --- /dev/null +++ b/cpp/tensorrt_llm/kernels/kimiK3AttnRes/attnResFwd.cu @@ -0,0 +1,1557 @@ +/* + * Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// Fused Kimi K3 attention-residual forward for Blackwell (sm_100 family). +// +// Warp-specialized online softmax + residual + RMSNorm: +// - 1 producer warp issues cp.async.bulk row loads into shared memory. +// - 8 consumer warps compute reductions and output. +// - Q=res_weight*rms_weight remains in registers across persistent tokens. +// - V rows are converted once and cached as FP32 in TMEM between passes. +// +// Contract: B=1, N<=12, H in [4096,8192] and divisible by 1024; checked at +// the Torch-op bridge. The kernel uses separate layer/block residual inputs +// and does not require a concatenated V tensor. +// +// Source-integrated from the NVIDIA+Moonshot jointly developed +// Attention_residual kernel at e7f934124acc915575f9f7561f9d1e373ab43089. + +#include "tensorrt_llm/common/cudaUtils.h" +#include "tensorrt_llm/kernels/kimiK3AttnRes/attnResFwd.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace +{ + +using bf16_t = __nv_bfloat16; + +static constexpr int ATTN_RES_BLOCK = 256; +static constexpr int ATTN_RES_WARPS = ATTN_RES_BLOCK / 32; + +__inline__ __device__ float warp_reduce_sum(float val) +{ +#pragma unroll + for (int offset = 16; offset > 0; offset >>= 1) + val += __shfl_xor_sync(0xffffffff, val, offset); + return val; +} + +__inline__ __device__ float block_reduce_sum(float val, float* ws) +{ + int lane = threadIdx.x & 31; + int wid = threadIdx.x >> 5; + val = warp_reduce_sum(val); + if (lane == 0) + ws[wid] = val; + __syncthreads(); + val = (threadIdx.x < ATTN_RES_WARPS) ? ws[threadIdx.x] : 0.f; + if (wid == 0) + val = warp_reduce_sum(val); + return val; +} + +__device__ __forceinline__ bf16_t const* v_addr( + bf16_t const* block_res, bf16_t const* layer_res, int n, int N, int t, int b, int T, int B, int H) +{ + if (n < N - 1) + return block_res + (((long long) n * T + t) * B + b) * H; + return layer_res + ((long long) t * B + b) * H; +} + +namespace sm100 +{ + +CUTE_DEVICE +void tcgen05_after_thread_sync() +{ + asm volatile("tcgen05.fence::after_thread_sync;"); +} + +CUTE_DEVICE +void umma_arrive_noelect(uint64_t& bar_ptr) +{ + uint64_t bar_addr = cute::cast_smem_ptr_to_uint(&bar_ptr); + asm volatile("tcgen05.commit.cta_group::1.mbarrier::arrive::one.shared::cluster.b64 [%0];" : : "l"(bar_addr)); +} + +CUTE_DEVICE +float2 float2_sub(float2 const& a, float2 const& b) +{ + float2 c; + asm volatile("sub.f32x2 %0, %1, %2;\n" + : "=l"(reinterpret_cast(c)) + : "l"(reinterpret_cast(a)), "l"(reinterpret_cast(b))); + return c; +} + +CUTE_DEVICE +float2 float2_mul(float2 const& a, float2 const& b) +{ + float2 c; + asm volatile("mul.f32x2 %0, %1, %2;\n" + : "=l"(reinterpret_cast(c)) + : "l"(reinterpret_cast(a)), "l"(reinterpret_cast(b))); + return c; +} + +CUTE_DEVICE +float2 float2_fma(float2 const& a, float2 const& b, float2 const& c) +{ + float2 d; + asm volatile("fma.rn.f32x2 %0, %1, %2, %3;\n" + : "=l"(reinterpret_cast(d)) + : "l"(reinterpret_cast(a)), "l"(reinterpret_cast(b)), + "l"(reinterpret_cast(c))); + return d; +} + +CUTE_DEVICE +float2 float2_add(float2 const& a, float2 const& b) +{ + float2 c; + asm volatile("add.rn.f32x2 %0, %1, %2;\n" + : "=l"(reinterpret_cast(c)) + : "l"(reinterpret_cast(a)), "l"(reinterpret_cast(b))); + return c; +} + +template +CUTE_DEVICE void tmem_ld_32dp32bNx(uint32_t const& src_addr, T* dst_ptr_) +{ + uint32_t* dst_ptr = reinterpret_cast(dst_ptr_); + if constexpr (N == 8) + { + asm volatile( + "tcgen05.ld.sync.aligned.32x32b.x8.b32" + "{%0, %1, %2, %3, %4, %5, %6, %7}," + "[%8];\n" + : "=r"(dst_ptr[0]), "=r"(dst_ptr[1]), "=r"(dst_ptr[2]), "=r"(dst_ptr[3]), "=r"(dst_ptr[4]), + "=r"(dst_ptr[5]), "=r"(dst_ptr[6]), "=r"(dst_ptr[7]) + : "r"(src_addr)); + } + else + { + static_assert(N == 4, "attn_res TMEM helpers support x4 and x8"); + asm volatile( + "tcgen05.ld.sync.aligned.32x32b.x4.b32" + "{%0, %1, %2, %3}, [%4];\n" + : "=r"(dst_ptr[0]), "=r"(dst_ptr[1]), "=r"(dst_ptr[2]), "=r"(dst_ptr[3]) + : "r"(src_addr)); + } +} + +template +CUTE_DEVICE void tmem_st_32dp32bNx(uint32_t const& dst_addr, T* src_ptr_) +{ + uint32_t* src_ptr = reinterpret_cast(src_ptr_); + if constexpr (N == 8) + { + asm volatile( + "tcgen05.st.sync.aligned.32x32b.x8.b32" + "[%8], {%0, %1, %2, %3, %4, %5, %6, %7};\n" + : + : "r"(src_ptr[0]), "r"(src_ptr[1]), "r"(src_ptr[2]), "r"(src_ptr[3]), "r"(src_ptr[4]), "r"(src_ptr[5]), + "r"(src_ptr[6]), "r"(src_ptr[7]), "r"(dst_addr)); + } + else + { + static_assert(N == 4, "attn_res TMEM helpers support x4 and x8"); + asm volatile( + "tcgen05.st.sync.aligned.32x32b.x4.b32" + "[%4], {%0, %1, %2, %3};\n" + : + : "r"(src_ptr[0]), "r"(src_ptr[1]), "r"(src_ptr[2]), "r"(src_ptr[3]), "r"(dst_addr)); + } +} + +namespace fwd_prod_v2 +{ + +using namespace cute; + +constexpr int K_TILE = 1024; +constexpr int N_MAX = 12; +constexpr int N_CHUNK_DEFAULT = 4; +constexpr int CHUNK_DEPTH = 2; +constexpr int BLK = 288; // 1 producer warp + 8 consumer warps +constexpr int CONSUMER_THREADS = BLK - 32; // 256 +constexpr int CONSUMER_WARPS = CONSUMER_THREADS / 32; +constexpr int CONSUMER_GROUPS = 2; // two 128-thread consumer groups +constexpr int CONSUMER_THREADS_PER_GROUP = CONSUMER_THREADS / CONSUMER_GROUPS; +constexpr int TMEM_Q_COLS_PER_GROUP = 32; +constexpr int TMEM_Q_COLS_TOTAL = 2 * TMEM_Q_COLS_PER_GROUP; + +template +struct FwdSmemPlan +{ + alignas(16) uint64_t bar_ready[CHUNK_DEPTH]; + alignas(16) uint64_t bar_consumed[CHUNK_DEPTH]; + alignas(16) float2 ws_stats[CONSUMER_WARPS][NC]; + alignas(16) float logits_all[N_MAX]; + uint32_t tmem_base; +}; + +__device__ __forceinline__ void cp_async_bulk(void* smem_dst, void const* gmem_src, int bytes, uint64_t& mbar) +{ + uint32_t s = cute::cast_smem_ptr_to_uint(smem_dst); + uint32_t m = cute::cast_smem_ptr_to_uint(&mbar); + asm volatile("cp.async.bulk.shared::cta.global.mbarrier::complete_tx::bytes [%0], [%1], %2, [%3];\n" ::"r"(s), + "l"(gmem_src), "r"(bytes), "r"(m) + : "memory"); +} + +template +__global__ void __launch_bounds__(BLK, 1) attn_res_fwd_online_v2_kernel(bf16_t const* __restrict__ block_res, + bf16_t const* __restrict__ layer_res, bf16_t const* __restrict__ res_w, bf16_t const* __restrict__ rms_w, + bf16_t* __restrict__ output, float* __restrict__ rsigma_out, float* __restrict__ probs_out, + float* __restrict__ logits_out, int N, int T, int B, float rms_eps) +{ +#if defined(__CUDA_ARCH__) && __CUDA_ARCH__ >= 1000 + constexpr float LOG2_E = 1.4426950408889634f; + constexpr int N_CHUNK = NC; + constexpr int NUM_BUFS = CHUNK_DEPTH * NC; + constexpr int NHT = H / K_TILE; + constexpr int SLICES_PER_GROUP = (NHT + CONSUMER_GROUPS - 1) / CONSUMER_GROUPS; + constexpr int VEC = 8; + constexpr int ACC_PER_THREAD = H == 7168 ? 28 : SLICES_PER_GROUP * VEC; + constexpr int TMEM_V_COLS_PER_GROUP = SLICES_PER_GROUP * N_CHUNK * VEC; + constexpr int TMEM_COLS_TOTAL = CONSUMER_GROUPS * TMEM_V_COLS_PER_GROUP; + constexpr int TMEM_COLS_ALLOC = 256; + static_assert(TMEM_COLS_TOTAL <= TMEM_COLS_ALLOC); + static_assert(H >= 4096 && H <= 8192); + static_assert(H % K_TILE == 0); + + int const tid = threadIdx.x; + int const wid = tid >> 5; + int const lane = tid & 31; + int const TB = FULL_N12 ? 1024 : T; + int const num_ctas = gridDim.x; + int const num_chunks = (N + N_CHUNK - 1) / N_CHUNK; + + int const comp_wid = wid - 1; + int const comp_tid = tid - 32; + int const group = (comp_wid >= 4) ? 1 : 0; + int const ct_in_group = (comp_tid >= 0) ? (comp_tid & (CONSUMER_THREADS_PER_GROUP - 1)) : -1; + int const k_local = ct_in_group * VEC; + + extern __shared__ char smem_raw[]; + bf16_t* v_bufs = reinterpret_cast(smem_raw); // [NUM_BUFS][H] + constexpr size_t V_BYTES = (size_t) NUM_BUFS * H * sizeof(bf16_t); + FwdSmemPlan& plan = *reinterpret_cast*>(smem_raw + V_BYTES); + + auto slot_of = [](long long gci, int n) { return (int) (gci % CHUNK_DEPTH) * N_CHUNK + n; }; + auto phase_of = [](long long gci) { return (int) ((gci / CHUNK_DEPTH) & 1); }; + auto buf_ptr = [&](int slot) -> bf16_t* { return v_bufs + slot * H; }; + + if (wid == 0 && elect_one_sync()) + { +#pragma unroll + for (int i = 0; i < CHUNK_DEPTH; i++) + { + cute::initialize_barrier(plan.bar_ready[i], 1); + cute::initialize_barrier(plan.bar_consumed[i], CONSUMER_WARPS); + } + cutlass::arch::fence_barrier_init(); + } + if (wid == 1) + { + cute::TMEM::Allocator1Sm alloc; + alloc.allocate(TMEM_COLS_ALLOC, &plan.tmem_base); + if constexpr (RELEASE_TMEM) + { + alloc.release_allocation_lock(); + } + } + __syncthreads(); + + const uint32_t my_v_tmem = (comp_tid >= 0) ? (plan.tmem_base + group * TMEM_V_COLS_PER_GROUP) : 0; + + float q_cache[ACC_PER_THREAD]; + if (comp_tid >= 0) + { +#pragma unroll + for (int si = 0; si < SLICES_PER_GROUP; si++) + { + if constexpr (H == 7168) + { + if (si == SLICES_PER_GROUP - 1) + { + int h_base = 6 * K_TILE + group * (K_TILE / 2) + ct_in_group * 4; +#pragma unroll + for (int j = 0; j < 4; j++) + { + int h = h_base + j; + q_cache[si * VEC + j] = __bfloat162float(rms_w[h]) * __bfloat162float(res_w[h]); + } + continue; + } + } + int dt = si * CONSUMER_GROUPS + group; + if (dt >= NHT) + continue; + int h_base = dt * K_TILE + k_local; +#pragma unroll + for (int j = 0; j < VEC; j++) + { + int h = h_base + j; + q_cache[si * VEC + j] = __bfloat162float(rms_w[h]) * __bfloat162float(res_w[h]); + } + } + } + + if (wid == 0) + { + if (elect_one_sync()) + { + long long gci = 0; + for (int tb = blockIdx.x; tb < TB; tb += num_ctas) + { + for (int ci = 0; ci < num_chunks; ci++, gci++) + { + int ns = ci * N_CHUNK; + int an = FULL_N12 ? N_CHUNK : min(N_CHUNK, N - ns); + int chunk_slot = (int) (gci % CHUNK_DEPTH); + int pc = phase_of(gci); + cute::wait_barrier(plan.bar_consumed[chunk_slot], pc ^ 1); + cute::set_barrier_transaction_bytes(plan.bar_ready[chunk_slot], an * H * (int) sizeof(bf16_t)); +#pragma unroll + for (int n = 0; n < N_CHUNK; n++) + { + if constexpr (!FULL_N12) + { + if (n >= an) + continue; + } + int slot = slot_of(gci, n); + int const ng = ns + n; + bf16_t const* src = (ng < (FULL_N12 ? 11 : N - 1)) ? block_res + ((long long) ng * T + tb) * H + : layer_res + (long long) tb * H; + cp_async_bulk(buf_ptr(slot), src, H * sizeof(bf16_t), plan.bar_ready[chunk_slot]); + } + } + } + } + } + else + { + float acc32[ACC_PER_THREAD] = {}; + float eps_cache; + asm volatile("mov.b32 %0, %1;" : "=f"(eps_cache) : "f"(rms_eps)); + + long long gci = 0; + for (int tb = blockIdx.x; tb < TB; tb += num_ctas) + { + float m_running = -FLT_MAX; + float s_running = 0.f; +#pragma unroll + for (int i = 0; i < ACC_PER_THREAD; i++) + { + acc32[i] = 0.f; + } + + for (int ci = 0; ci < num_chunks; ci++, gci++) + { + int ns = ci * N_CHUNK; + int an = FULL_N12 ? N_CHUNK : min(N_CHUNK, N - ns); + int chunk_slot = (int) (gci % CHUNK_DEPTH); + int pr = phase_of(gci); + float2 sq_local[N_CHUNK] = {}; + float2 dot_local[N_CHUNK] = {}; + cute::wait_barrier(plan.bar_ready[chunk_slot], pr); + + auto pass_A_body = [&](auto AN_TOK) + { + constexpr int AN = decltype(AN_TOK)::value; +#pragma unroll + for (int si = 0; si < SLICES_PER_GROUP; si++) + { + if constexpr (H == 7168) + { + if (si == SLICES_PER_GROUP - 1) + { + int h_base = 6 * K_TILE + group * (K_TILE / 2) + ct_in_group * 4; + const float* qv = &q_cache[si * VEC]; +#pragma unroll + for (int n = 0; n < AN; n++) + { + int slot = slot_of(gci, n); + int2 vp = *reinterpret_cast(buf_ptr(slot) + h_base); + __nv_bfloat162* v2 = reinterpret_cast<__nv_bfloat162*>(&vp); + float2 f[2] = {__bfloat1622float2(v2[0]), __bfloat1622float2(v2[1])}; + if constexpr (FULL_N12) + { + if (n == AN - 1 && lane == 0) + { + cute::arrive_barrier(plan.bar_consumed[chunk_slot]); + } + } + tmem_st_32dp32bNx<4>( + my_v_tmem + (si * N_CHUNK + n) * VEC, reinterpret_cast(f)); + sq_local[n] = float2_fma(f[0], f[0], sq_local[n]); + sq_local[n] = float2_fma(f[1], f[1], sq_local[n]); + dot_local[n] = float2_fma(f[0], make_float2(qv[0], qv[1]), dot_local[n]); + dot_local[n] = float2_fma(f[1], make_float2(qv[2], qv[3]), dot_local[n]); + } + continue; + } + } + int dt = si * CONSUMER_GROUPS + group; + if (dt >= NHT) + continue; + const float* qv = &q_cache[si * VEC]; + +#pragma unroll + for (int n = 0; n < AN; n++) + { + int slot = slot_of(gci, n); + int4 vp = *reinterpret_cast(buf_ptr(slot) + dt * K_TILE + k_local); + __nv_bfloat162* v2 = reinterpret_cast<__nv_bfloat162*>(&vp); + float2 f[4] = {__bfloat1622float2(v2[0]), __bfloat1622float2(v2[1]), + __bfloat1622float2(v2[2]), __bfloat1622float2(v2[3])}; + tmem_st_32dp32bNx(my_v_tmem + (si * N_CHUNK + n) * VEC, reinterpret_cast(f)); + sq_local[n] = float2_fma(f[0], f[0], sq_local[n]); + sq_local[n] = float2_fma(f[1], f[1], sq_local[n]); + sq_local[n] = float2_fma(f[2], f[2], sq_local[n]); + sq_local[n] = float2_fma(f[3], f[3], sq_local[n]); + dot_local[n] = float2_fma(f[0], make_float2(qv[0], qv[1]), dot_local[n]); + dot_local[n] = float2_fma(f[1], make_float2(qv[2], qv[3]), dot_local[n]); + dot_local[n] = float2_fma(f[2], make_float2(qv[4], qv[5]), dot_local[n]); + dot_local[n] = float2_fma(f[3], make_float2(qv[6], qv[7]), dot_local[n]); + } + } + if constexpr (!FULL_N12) + { + cutlass::arch::fence_view_async_tmem_store(); + } + }; + if constexpr (FULL_N12) + { + pass_A_body(std::integral_constant{}); + } + else if constexpr (NC == 4) + { + switch (an) + { + case 4: pass_A_body(std::integral_constant{}); break; + case 3: pass_A_body(std::integral_constant{}); break; + case 2: pass_A_body(std::integral_constant{}); break; + case 1: pass_A_body(std::integral_constant{}); break; + default: __builtin_unreachable(); + } + } + else if constexpr (NC == 3) + { + switch (an) + { + case 3: pass_A_body(std::integral_constant{}); break; + case 2: pass_A_body(std::integral_constant{}); break; + case 1: pass_A_body(std::integral_constant{}); break; + default: __builtin_unreachable(); + } + } + else + { + static_assert(NC == 2); + switch (an) + { + case 2: pass_A_body(std::integral_constant{}); break; + case 1: pass_A_body(std::integral_constant{}); break; + default: __builtin_unreachable(); + } + } + if constexpr (!FULL_N12) + { + if (lane == 0) + { + cute::arrive_barrier(plan.bar_consumed[chunk_slot]); + } + } + + float2 reduce_pair[N_CHUNK]; +#pragma unroll + for (int n = 0; n < N_CHUNK; n++) + { + reduce_pair[n] = make_float2(sq_local[n].x + sq_local[n].y, dot_local[n].x + dot_local[n].y); + } +#pragma unroll + for (int offset = 16; offset > 0; offset >>= 1) + { +#pragma unroll + for (int n = 0; n < N_CHUNK; n++) + { + uint64_t packed = reinterpret_cast(reduce_pair[n]); + packed = __shfl_xor_sync(0xffffffff, packed, offset); + float2 other = reinterpret_cast(packed); + reduce_pair[n] = float2_add(reduce_pair[n], other); + } + } + if constexpr (FULL_N12) + { + cutlass::arch::fence_view_async_tmem_store(); + } + if (lane == 0) + { +#pragma unroll + for (int n = 0; n < N_CHUNK; n++) + { + plan.ws_stats[comp_wid][n] = reduce_pair[n]; + } + } + cutlass::arch::NamedBarrier::sync(CONSUMER_THREADS, 0); + + float local_rsig = 0.f; + float local_logit = 0.f; + auto cross_warp_tail = [&](int n) + { + float2 totals = {}; +#pragma unroll + for (int w = 0; w < CONSUMER_WARPS; w++) + { + totals = float2_add(totals, plan.ws_stats[w][n]); + } + local_rsig = rsqrtf(totals.x / H + eps_cache); + local_logit = totals.y * local_rsig; + }; + if constexpr (FULL_N12) + { + cross_warp_tail(lane & (N_CHUNK - 1)); + } + else if (lane < N_CHUNK) + { + cross_warp_tail(lane); + } + float logit_n[N_CHUNK]; +#pragma unroll + for (int n = 0; n < N_CHUNK; n++) + { + logit_n[n] = __shfl_sync(0xffffffff, local_logit, n); + } + + float m_chunk = -FLT_MAX; + if constexpr (FULL_N12) + { + float m01 = fmaxf(logit_n[0], logit_n[1]); + float m23 = fmaxf(logit_n[2], logit_n[3]); + m_chunk = fmaxf(m01, m23); + } + else + { +#pragma unroll + for (int n = 0; n < N_CHUNK; n++) + { + if (n < an) + { + m_chunk = fmaxf(m_chunk, logit_n[n]); + } + } + } + float m_new = fmaxf(m_running, m_chunk); + float corr = exp2f((m_running - m_new) * LOG2_E); + float w_n[N_CHUNK] = {}; + float w_sum = 0.f; + if constexpr (FULL_N12) + { +#pragma unroll + for (int n = 0; n < N_CHUNK; n++) + { + w_n[n] = exp2f((logit_n[n] - m_new) * LOG2_E); + } + w_sum = (w_n[0] + w_n[1]) + (w_n[2] + w_n[3]); + } + else + { +#pragma unroll + for (int n = 0; n < N_CHUNK; n++) + { + if (n < an) + { + w_n[n] = exp2f((logit_n[n] - m_new) * LOG2_E); + w_sum += w_n[n]; + } + } + } + + auto pass_B_body = [&](auto AN_TOK) + { + constexpr int AN = decltype(AN_TOK)::value; +#pragma unroll + for (int si = 0; si < SLICES_PER_GROUP; si++) + { + if constexpr (H == 7168) + { + if (si == SLICES_PER_GROUP - 1) + { + float2 corr2 = make_float2(corr, corr); + float2 a[2]; +#pragma unroll + for (int j = 0; j < 2; j++) + { + float2 old = make_float2(acc32[si * VEC + 2 * j], acc32[si * VEC + 2 * j + 1]); + a[j] = float2_mul(old, corr2); + } + float2 f_cache[AN][2]; +#pragma unroll + for (int n = 0; n < AN; n++) + { + tmem_ld_32dp32bNx<4>( + my_v_tmem + (si * N_CHUNK + n) * VEC, reinterpret_cast(f_cache[n])); + } +#pragma unroll + for (int n = 0; n < AN; n++) + { + float2 wn = make_float2(w_n[n], w_n[n]); +#pragma unroll + for (int j = 0; j < 2; j++) + { + a[j] = float2_fma(wn, f_cache[n][j], a[j]); + } + } +#pragma unroll + for (int j = 0; j < 2; j++) + { + acc32[si * VEC + 2 * j] = a[j].x; + acc32[si * VEC + 2 * j + 1] = a[j].y; + } + continue; + } + } + int dt = si * CONSUMER_GROUPS + group; + if (dt >= NHT) + continue; + float2 a[VEC / 2]; + float2 corr2 = make_float2(corr, corr); +#pragma unroll + for (int j = 0; j < VEC / 2; j++) + { + float2 old = make_float2(acc32[si * VEC + 2 * j], acc32[si * VEC + 2 * j + 1]); + a[j] = float2_mul(old, corr2); + } + float2 f_cache[AN][VEC / 2]; +#pragma unroll + for (int n = 0; n < AN; n++) + { + tmem_ld_32dp32bNx( + my_v_tmem + (si * N_CHUNK + n) * VEC, reinterpret_cast(f_cache[n])); + } +#pragma unroll + for (int n = 0; n < AN; n++) + { + float2 wn = make_float2(w_n[n], w_n[n]); +#pragma unroll + for (int j = 0; j < VEC / 2; j++) + { + a[j] = float2_fma(wn, f_cache[n][j], a[j]); + } + } +#pragma unroll + for (int j = 0; j < VEC / 2; j++) + { + acc32[si * VEC + 2 * j] = a[j].x; + acc32[si * VEC + 2 * j + 1] = a[j].y; + } + } + }; + if constexpr (FULL_N12) + { + pass_B_body(std::integral_constant{}); + } + else if constexpr (NC == 4) + { + switch (an) + { + case 4: pass_B_body(std::integral_constant{}); break; + case 3: pass_B_body(std::integral_constant{}); break; + case 2: pass_B_body(std::integral_constant{}); break; + case 1: pass_B_body(std::integral_constant{}); break; + default: __builtin_unreachable(); + } + } + else if constexpr (NC == 3) + { + switch (an) + { + case 3: pass_B_body(std::integral_constant{}); break; + case 2: pass_B_body(std::integral_constant{}); break; + case 1: pass_B_body(std::integral_constant{}); break; + default: __builtin_unreachable(); + } + } + else + { + static_assert(NC == 2); + switch (an) + { + case 2: pass_B_body(std::integral_constant{}); break; + case 1: pass_B_body(std::integral_constant{}); break; + default: __builtin_unreachable(); + } + } + + s_running = s_running * corr + w_sum; + m_running = m_new; + + if (comp_wid == 0 && lane < an) + { + int ng = ns + lane; + rsigma_out[(long long) ng * TB + tb] = local_rsig; + plan.logits_all[ng] = local_logit; + } + } + // Publish the final chunk's plan.logits_all stores before the + // cross-lane reads in consumer warp 0 below (earlier chunks are + // covered by the NamedBarrier inside the loop). + __syncwarp(); + + float inv_s = 1.f / s_running; + bf16_t* out_ptr = output + (long long) tb * H; +#pragma unroll + for (int si = 0; si < SLICES_PER_GROUP; si++) + { + if constexpr (H == 7168) + { + if (si == SLICES_PER_GROUP - 1) + { + int h_base = 6 * K_TILE + group * (K_TILE / 2) + ct_in_group * 4; + uint2 ov; + __nv_bfloat162* ov2 = reinterpret_cast<__nv_bfloat162*>(&ov); + float2 inv2 = make_float2(inv_s, inv_s); +#pragma unroll + for (int j = 0; j < 2; j++) + { + float2 old = make_float2(acc32[si * VEC + 2 * j], acc32[si * VEC + 2 * j + 1]); + ov2[j] = __float22bfloat162_rn(float2_mul(old, inv2)); + } + *reinterpret_cast(out_ptr + h_base) = ov; + continue; + } + } + int dt = si * CONSUMER_GROUPS + group; + if (dt >= NHT) + continue; + int h_base = dt * K_TILE + k_local; + uint4 ov; + __nv_bfloat162* ov2 = reinterpret_cast<__nv_bfloat162*>(&ov); + float2 inv2 = make_float2(inv_s, inv_s); +#pragma unroll + for (int j = 0; j < VEC / 2; j++) + { + float2 old = make_float2(acc32[si * VEC + 2 * j], acc32[si * VEC + 2 * j + 1]); + ov2[j] = __float22bfloat162_rn(float2_mul(old, inv2)); + } + *reinterpret_cast(out_ptr + h_base) = ov; + } + + if (comp_wid == 0 && lane < (FULL_N12 ? 12 : N)) + { + long long out_idx = (long long) lane * TB + tb; + float lg = plan.logits_all[lane]; + logits_out[out_idx] = lg; + probs_out[out_idx] = exp2f((lg - m_running) * LOG2_E) * inv_s; + } + } + } + + if (wid > 0) + { + cutlass::arch::NamedBarrier::sync(CONSUMER_THREADS, 2); + } + if (wid == 1) + { + cute::TMEM::Allocator1Sm alloc; + alloc.free(plan.tmem_base, TMEM_COLS_ALLOC); + } +#else + if (cute::thread0()) + printf("attn_res_fwd_online_v2_kernel requires sm_100a\n"); +#endif +} + +// N=1 specialization: softmax is degenerate, so output is the layer row. +// Tile multiple contiguous TB rows per CTA to reduce cp.async.bulk overhead. +template +__global__ void __launch_bounds__(BLK, 1) + attn_res_fwd_n1_ttile_kernel(bf16_t const* __restrict__ layer_res, bf16_t const* __restrict__ res_w, + bf16_t const* __restrict__ rms_w, bf16_t* __restrict__ output, float* __restrict__ rsigma_out, + float* __restrict__ probs_out, float* __restrict__ logits_out, int T, int B, float rms_eps) +{ +#if defined(__CUDA_ARCH__) && __CUDA_ARCH__ >= 1000 + constexpr int NHT = H / K_TILE; + constexpr int SLICES_PER_GROUP = (NHT + CONSUMER_GROUPS - 1) / CONSUMER_GROUPS; + constexpr int VEC = 8; + constexpr int ACC_PER_THREAD = SLICES_PER_GROUP * VEC; + static_assert(H == 4096 || H == 8192); + + int const tid = threadIdx.x; + int const wid = tid >> 5; + int const lane = tid & 31; + int const TB = T * B; + int const comp_wid = wid - 1; + int const comp_tid = tid - 32; + int const group = (comp_wid >= 4) ? 1 : 0; + int const ct_in_group = (comp_tid >= 0) ? (comp_tid & (CONSUMER_THREADS_PER_GROUP - 1)) : -1; + int const k_local = ct_in_group * VEC; + + extern __shared__ char smem_raw[]; + bf16_t* v_tiles = reinterpret_cast(smem_raw); + constexpr size_t V_BYTES = (size_t) CHUNK_DEPTH * TB_TILE * H * sizeof(bf16_t); + FwdSmemPlan<1>& plan = *reinterpret_cast*>(smem_raw + V_BYTES); + + auto phase_of = [](long long tile_i) { return (int) ((tile_i / CHUNK_DEPTH) & 1); }; + auto tile_ptr = [&](int slot, int row) -> bf16_t* { return v_tiles + ((slot * TB_TILE + row) * H); }; + + if (wid == 0 && elect_one_sync()) + { +#pragma unroll + for (int i = 0; i < CHUNK_DEPTH; i++) + { + cute::initialize_barrier(plan.bar_ready[i], 1); + cute::initialize_barrier(plan.bar_consumed[i], CONSUMER_THREADS); + } + cutlass::arch::fence_barrier_init(); + } + if (wid == 1) + { + cute::TMEM::Allocator1Sm alloc; + alloc.allocate(TMEM_Q_COLS_TOTAL, &plan.tmem_base); + if constexpr (RELEASE_TMEM) + { + alloc.release_allocation_lock(); + } + } + __syncthreads(); + + const uint32_t my_tmem = (comp_tid >= 0) ? (plan.tmem_base + ((comp_wid >= 4) ? TMEM_Q_COLS_PER_GROUP : 0)) : 0; + + if (comp_tid >= 0) + { + float q32[ACC_PER_THREAD]; +#pragma unroll + for (int si = 0; si < SLICES_PER_GROUP; si++) + { + int dt = si * CONSUMER_GROUPS + group; + if (dt >= NHT) + continue; + int h_base = dt * K_TILE + k_local; +#pragma unroll + for (int j = 0; j < VEC; j++) + { + int h = h_base + j; + q32[si * VEC + j] = __bfloat162float(rms_w[h]) * __bfloat162float(res_w[h]); + } + } +#pragma unroll + for (int si = 0; si < SLICES_PER_GROUP; si++) + { + int dt = si * CONSUMER_GROUPS + group; + if (dt >= NHT) + continue; + tmem_st_32dp32bNx(my_tmem + si * VEC, &q32[si * VEC]); + } + cutlass::arch::fence_view_async_tmem_store(); + } + __syncthreads(); + + if (wid == 0) + { + if (elect_one_sync()) + { + long long tile_i = 0; + for (int tb0 = blockIdx.x * TB_TILE; tb0 < TB; tb0 += gridDim.x * TB_TILE, tile_i++) + { + int rows = min(TB_TILE, TB - tb0); + int slot = (int) (tile_i % CHUNK_DEPTH); + int pc = phase_of(tile_i); + cute::wait_barrier(plan.bar_consumed[slot], pc ^ 1); + cute::set_barrier_transaction_bytes(plan.bar_ready[slot], rows * H * (int) sizeof(bf16_t)); + cp_async_bulk(tile_ptr(slot, 0), layer_res + (long long) tb0 * H, rows * H * sizeof(bf16_t), + plan.bar_ready[slot]); + } + } + } + else + { + long long tile_i = 0; + for (int tb0 = blockIdx.x * TB_TILE; tb0 < TB; tb0 += gridDim.x * TB_TILE, tile_i++) + { + int rows = min(TB_TILE, TB - tb0); + int slot = (int) (tile_i % CHUNK_DEPTH); + int pc = phase_of(tile_i); + cute::wait_barrier(plan.bar_ready[slot], pc); + +#pragma unroll + for (int r = 0; r < TB_TILE; r++) + { + if (r >= rows) + continue; + int tb = tb0 + r; + bf16_t* row_ptr = tile_ptr(slot, r); + bf16_t* out_ptr = output + (long long) tb * H; + float sq_local = 0.f; + float dot_local = 0.f; + +#pragma unroll + for (int si = 0; si < SLICES_PER_GROUP; si++) + { + int dt = si * CONSUMER_GROUPS + group; + if (dt >= NHT) + continue; + int h_base = dt * K_TILE + k_local; + float qv[VEC]; + tmem_ld_32dp32bNx(my_tmem + si * VEC, qv); + int4 vp = *reinterpret_cast(row_ptr + h_base); + *reinterpret_cast(out_ptr + h_base) = vp; + + __nv_bfloat162* v2 = reinterpret_cast<__nv_bfloat162*>(&vp); + float2 f0 = __bfloat1622float2(v2[0]); + float2 f1 = __bfloat1622float2(v2[1]); + float2 f2 = __bfloat1622float2(v2[2]); + float2 f3 = __bfloat1622float2(v2[3]); + sq_local += f0.x * f0.x + f0.y * f0.y + f1.x * f1.x + f1.y * f1.y + f2.x * f2.x + f2.y * f2.y + + f3.x * f3.x + f3.y * f3.y; + dot_local += f0.x * qv[0] + f0.y * qv[1] + f1.x * qv[2] + f1.y * qv[3] + f2.x * qv[4] + f2.y * qv[5] + + f3.x * qv[6] + f3.y * qv[7]; + } + +#pragma unroll + for (int offset = 16; offset > 0; offset >>= 1) + { + sq_local += __shfl_xor_sync(0xffffffff, sq_local, offset); + dot_local += __shfl_xor_sync(0xffffffff, dot_local, offset); + } + if (lane == 0) + { + plan.ws_stats[comp_wid][0] = make_float2(sq_local, dot_local); + } + cutlass::arch::NamedBarrier::sync(CONSUMER_THREADS, 0); + + if (comp_wid == 0 && lane == 0) + { + float2 totals = {}; +#pragma unroll + for (int w = 0; w < CONSUMER_WARPS; w++) + { + totals = float2_add(totals, plan.ws_stats[w][0]); + } + float rs = rsqrtf(totals.x / H + rms_eps); + rsigma_out[tb] = rs; + if (logits_out) + logits_out[tb] = totals.y * rs; + if (probs_out) + probs_out[tb] = 1.f; + } + cutlass::arch::NamedBarrier::sync(CONSUMER_THREADS, 1); + } + cute::arrive_barrier(plan.bar_consumed[slot]); + } + } + + __syncthreads(); + if (wid == 1) + { + cute::TMEM::Allocator1Sm alloc; + alloc.free(plan.tmem_base, TMEM_Q_COLS_TOTAL); + } +#else + if (cute::thread0()) + printf("attn_res_fwd_n1_ttile_kernel requires sm_100a\n"); +#endif +} + +template +static void launch_fwd(bf16_t const* block_residual, bf16_t const* layer_residual, bf16_t const* res_weight, + bf16_t const* rms_weight, bf16_t* output, float* rsigma, float* probs, float* logits, int N, int T, int B, + float rms_eps, int num_sm, cudaStream_t stream) +{ + constexpr size_t smem_size + = ((size_t) CHUNK_DEPTH * NC * H * sizeof(bf16_t) + sizeof(FwdSmemPlan) + 15) & ~size_t(15); + auto kernel = &attn_res_fwd_online_v2_kernel; + if (smem_size > 48 * 1024) + { + // cudaFuncSetAttribute applies to the current device only; set it + // once per device (per kernel instantiation). + static std::once_flag attrs_set[64]; + int dev = 0; + TLLM_CUDA_CHECK(cudaGetDevice(&dev)); + auto const set_attr = [&] + { TLLM_CUDA_CHECK(cudaFuncSetAttribute(kernel, cudaFuncAttributeMaxDynamicSharedMemorySize, smem_size)); }; + if (dev >= 0 && dev < 64) + { + std::call_once(attrs_set[dev], set_attr); + } + else + { + set_attr(); + } + } + int grid = RELEASE_TMEM ? num_sm * 2 : num_sm; + kernel<<>>( + block_residual, layer_residual, res_weight, rms_weight, output, rsigma, probs, logits, N, T, B, rms_eps); +} + +// Small-N counterpart to the Triton one-program topology. One CTA owns the +// complete token, with exactly 28 hidden elements per thread at H=7168. For +// N=2/4, packed BF16 V remains in registers across the statistics/softmax +// boundary; N=1 can write V directly because its softmax is identically one. +template +__global__ void __launch_bounds__(256, 1) + attn_res_fwd_s1_single_cta_kernel(bf16_t const* __restrict__ block_res, bf16_t const* __restrict__ layer_res, + bf16_t const* __restrict__ res_w, bf16_t const* __restrict__ rms_w, bf16_t* __restrict__ output, + float* __restrict__ rsigma_out, float* __restrict__ probs_out, float* __restrict__ logits_out, float rms_eps) +{ +#if defined(__CUDA_ARCH__) && __CUDA_ARCH__ >= 1000 + constexpr int H = 7168; + constexpr int THREADS = 256; + constexpr int WARPS = THREADS / 32; + constexpr int ITEMS = H / THREADS; + constexpr float LOG2_E = 1.4426950408889634f; + static_assert(H % THREADS == 0); + static_assert(N == 1 || N == 2 || N == 4); + + __shared__ float2 warp_stats[WARPS * N]; + __shared__ float weights[N]; + int const tid = threadIdx.x; + int const lane = tid & 31; + int const warp = tid >> 5; + + float2 stats[N] = {}; + uint32_t v_cache_bf16[ITEMS][(N + 1) / 2]; +#pragma unroll + for (int item = 0; item < ITEMS; item++) + { + int h = tid + item * THREADS; + float q = __bfloat162float(res_w[h]) * __bfloat162float(rms_w[h]); + bf16_t item_v[N]; +#pragma unroll + for (int n = 0; n < N; n++) + { + bf16_t const* row = n < N - 1 ? block_res + (size_t) n * H : layer_res; + bf16_t packed_v = row[h]; + float v = __bfloat162float(packed_v); + if constexpr (N == 1) + { + output[h] = packed_v; + } + else + { + item_v[n] = packed_v; + } + stats[n] = float2_fma(make_float2(v, v), make_float2(v, q), stats[n]); + } + if constexpr (N > 1) + { +#pragma unroll + for (int pair = 0; pair < N / 2; pair++) + { + union + { + __nv_bfloat162 bf16x2; + uint32_t bits; + } packed; + + packed.bf16x2 = __halves2bfloat162(item_v[2 * pair], item_v[2 * pair + 1]); + v_cache_bf16[item][pair] = packed.bits; + } + } + } + +#pragma unroll + for (int offset = 16; offset > 0; offset >>= 1) + { +#pragma unroll + for (int n = 0; n < N; n++) + { + uint64_t packed = reinterpret_cast(stats[n]); + packed = __shfl_down_sync(0xffffffff, packed, offset); + float2 other = reinterpret_cast(packed); + stats[n] = float2_add(stats[n], other); + } + } + if (lane == 0) + { +#pragma unroll + for (int n = 0; n < N; n++) + { + warp_stats[warp * N + n] = stats[n]; + } + } + __syncthreads(); + + if (tid < N) + { + float2 total = {}; +#pragma unroll + for (int w = 0; w < WARPS; w++) + { + total = float2_add(total, warp_stats[w * N + tid]); + } + warp_stats[tid] = total; + } + __syncthreads(); + + if (tid == 0) + { + float local_rsigma[N]; + float local_logits[N]; + float max_logit = -FLT_MAX; +#pragma unroll + for (int n = 0; n < N; n++) + { + float2 total = warp_stats[n]; + local_rsigma[n] = rsqrtf(total.x / H + rms_eps); + local_logits[n] = total.y * local_rsigma[n]; + max_logit = fmaxf(max_logit, local_logits[n]); + } + float denominator = 0.0f; +#pragma unroll + for (int n = 0; n < N; n++) + { + weights[n] = exp2f((local_logits[n] - max_logit) * LOG2_E); + denominator += weights[n]; + } + float inv_denominator = 1.0f / denominator; +#pragma unroll + for (int n = 0; n < N; n++) + { + weights[n] *= inv_denominator; + rsigma_out[n] = local_rsigma[n]; + logits_out[n] = local_logits[n]; + probs_out[n] = weights[n]; + } + } + __syncthreads(); + + if constexpr (N > 1) + { +#pragma unroll + for (int item = 0; item < ITEMS; item++) + { + float value = 0.0f; +#pragma unroll + for (int pair = 0; pair < N / 2; pair++) + { + union + { + __nv_bfloat162 bf16x2; + uint32_t bits; + } packed; + + packed.bits = v_cache_bf16[item][pair]; + float2 v = __bfloat1622float2(packed.bf16x2); + value = fmaf(weights[2 * pair], v.x, value); + value = fmaf(weights[2 * pair + 1], v.y, value); + } + int h = tid + item * THREADS; + output[h] = __float2bfloat16_rn(value); + } + } +#else + if (cute::thread0()) + { + printf("attn_res_fwd_s1_single_cta_kernel requires sm_100a\n"); + } +#endif +} + +template +static void launch_s1_single_cta(bf16_t const* block_residual, bf16_t const* layer_residual, bf16_t const* res_weight, + bf16_t const* rms_weight, bf16_t* output, float* rsigma, float* probs, float* logits, float rms_eps, + cudaStream_t stream) +{ + attn_res_fwd_s1_single_cta_kernel<<<1, 256, 0, stream>>>( + block_residual, layer_residual, res_weight, rms_weight, output, rsigma, probs, logits, rms_eps); +} + +// Single-token split-K specialization. The complete grid is one CTA cluster: +// rank g owns a disjoint H/GROUPS slice, keeps that slice of FP32 V in its +// rank-local shared memory, and exchanges only (square, dot) partials via DSM. +template +__global__ void __launch_bounds__(256, 1) + attn_res_fwd_s1_splitk_kernel(bf16_t const* __restrict__ block_res, bf16_t const* __restrict__ layer_res, + bf16_t const* __restrict__ res_w, bf16_t const* __restrict__ rms_w, bf16_t* __restrict__ output, + float* __restrict__ rsigma_out, float* __restrict__ probs_out, float* __restrict__ logits_out, float rms_eps) +{ +#if defined(__CUDA_ARCH__) && __CUDA_ARCH__ >= 1000 + namespace cg = cooperative_groups; + constexpr int H = 7168; + constexpr int K_PER_CTA = H / GROUPS; + constexpr int THREADS = 256; + constexpr int WARPS = THREADS / 32; + constexpr float LOG2_E = 1.4426950408889634f; + static_assert(H % GROUPS == 0); + + extern __shared__ char smem_raw[]; + float* v_cache = reinterpret_cast(smem_raw); + float2* warp_stats = reinterpret_cast(smem_raw + (size_t) N * K_PER_CTA * sizeof(float)); + float* weights = reinterpret_cast(warp_stats + WARPS * N); + + int const tid = threadIdx.x; + int const lane = tid & 31; + int const warp = tid >> 5; + cg::cluster_group cluster = cg::this_cluster(); + int const group = cluster.block_rank(); + int const h_begin = group * K_PER_CTA; + + float sq[N] = {}; + float dot[N] = {}; +#pragma unroll + for (int ki = tid; ki < K_PER_CTA; ki += THREADS) + { + int h = h_begin + ki; + float q = __bfloat162float(res_w[h]) * __bfloat162float(rms_w[h]); +#pragma unroll + for (int n = 0; n < N; n++) + { + bf16_t const* row = n < N - 1 ? block_res + (size_t) n * H : layer_res; + float v = __bfloat162float(row[h]); + v_cache[(size_t) n * K_PER_CTA + ki] = v; + sq[n] = fmaf(v, v, sq[n]); + dot[n] = fmaf(v, q, dot[n]); + } + } + +#pragma unroll + for (int offset = 16; offset > 0; offset >>= 1) + { +#pragma unroll + for (int n = 0; n < N; n++) + { + sq[n] += __shfl_down_sync(0xffffffff, sq[n], offset); + dot[n] += __shfl_down_sync(0xffffffff, dot[n], offset); + } + } + if (lane == 0) + { +#pragma unroll + for (int n = 0; n < N; n++) + { + warp_stats[warp * N + n] = make_float2(sq[n], dot[n]); + } + } + __syncthreads(); + + if (tid < N) + { + float2 total = {}; +#pragma unroll + for (int w = 0; w < WARPS; w++) + { + total = float2_add(total, warp_stats[w * N + tid]); + } + warp_stats[tid] = total; + } + + // Publish every rank's reduced statistics to distributed shared memory. + cluster.sync(); + + // One thread per candidate reduces across CTA ranks. Parallelizing this + // avoids making a single leader issue all GROUPS*N remote DSM reads. + if (tid < N) + { + float2 total = {}; +#pragma unroll + for (int g = 0; g < GROUPS; g++) + { + float2 const* remote_stats = cluster.map_shared_rank(warp_stats, g); + total = float2_add(total, remote_stats[tid]); + } + warp_stats[tid] = total; + } + __syncthreads(); + + if (tid == 0) + { + float local_rsigma[N]; + float local_logits[N]; + float max_logit = -FLT_MAX; +#pragma unroll + for (int n = 0; n < N; n++) + { + float2 total = warp_stats[n]; + local_rsigma[n] = rsqrtf(total.x / H + rms_eps); + local_logits[n] = total.y * local_rsigma[n]; + max_logit = fmaxf(max_logit, local_logits[n]); + } + float sum = 0.0f; +#pragma unroll + for (int n = 0; n < N; n++) + { + weights[n] = exp2f((local_logits[n] - max_logit) * LOG2_E); + sum += weights[n]; + } + float inv_sum = 1.0f / sum; +#pragma unroll + for (int n = 0; n < N; n++) + { + weights[n] *= inv_sum; + if (group == 0) + { + rsigma_out[n] = local_rsigma[n]; + logits_out[n] = local_logits[n]; + probs_out[n] = weights[n]; + } + } + } + + cluster.sync(); + +#pragma unroll + for (int ki = tid; ki < K_PER_CTA; ki += THREADS) + { + float value = 0.0f; +#pragma unroll + for (int n = 0; n < N; n++) + { + value = fmaf(weights[n], v_cache[(size_t) n * K_PER_CTA + ki], value); + } + output[h_begin + ki] = __float2bfloat16_rn(value); + } +#else + if (cute::thread0()) + { + printf("attn_res_fwd_s1_splitk_kernel requires sm_100a\n"); + } +#endif +} + +template +static void launch_s1_splitk(bf16_t const* block_residual, bf16_t const* layer_residual, bf16_t const* res_weight, + bf16_t const* rms_weight, bf16_t* output, float* rsigma, float* probs, float* logits, float rms_eps, + cudaStream_t stream) +{ + constexpr int K_PER_CTA = 7168 / GROUPS; + constexpr int WARPS = 8; + constexpr size_t smem_size + = (size_t) N * K_PER_CTA * sizeof(float) + (size_t) WARPS * N * sizeof(float2) + (size_t) N * sizeof(float); + auto kernel = &attn_res_fwd_s1_splitk_kernel; + { + // cudaFuncSetAttribute applies to the current device only; set it + // once per device (per kernel instantiation). + static std::once_flag attrs_set[64]; + int dev = 0; + TLLM_CUDA_CHECK(cudaGetDevice(&dev)); + auto const set_attr = [&] + { TLLM_CUDA_CHECK(cudaFuncSetAttribute(kernel, cudaFuncAttributeMaxDynamicSharedMemorySize, smem_size)); }; + if (dev >= 0 && dev < 64) + { + std::call_once(attrs_set[dev], set_attr); + } + else + { + set_attr(); + } + } + void* args[] = {const_cast(&block_residual), const_cast(&layer_residual), + const_cast(&res_weight), const_cast(&rms_weight), &output, &rsigma, &probs, &logits, + &rms_eps}; + cudaLaunchConfig_t config{}; + config.gridDim = dim3(GROUPS); + config.blockDim = dim3(256); + config.dynamicSmemBytes = smem_size; + config.stream = stream; + cudaLaunchAttribute attribute{}; + attribute.id = cudaLaunchAttributeClusterDimension; + attribute.val.clusterDim.x = GROUPS; + attribute.val.clusterDim.y = 1; + attribute.val.clusterDim.z = 1; + config.attrs = &attribute; + config.numAttrs = 1; + cudaLaunchKernelExC(&config, reinterpret_cast(kernel), args); +} + +template +static void launch_n1_ttile(bf16_t const* layer_residual, bf16_t const* res_weight, bf16_t const* rms_weight, + bf16_t* output, float* rsigma, float* probs, float* logits, int T, int B, float rms_eps, int num_sm, + cudaStream_t stream) +{ + constexpr size_t smem_size + = ((size_t) CHUNK_DEPTH * TB_TILE * H * sizeof(bf16_t) + sizeof(FwdSmemPlan<1>) + 15) & ~size_t(15); + auto kernel = &attn_res_fwd_n1_ttile_kernel; + if (smem_size > 48 * 1024) + { + // cudaFuncSetAttribute applies to the current device only; set it + // once per device (per kernel instantiation). + static std::once_flag attrs_set[64]; + int dev = 0; + TLLM_CUDA_CHECK(cudaGetDevice(&dev)); + auto const set_attr = [&] + { TLLM_CUDA_CHECK(cudaFuncSetAttribute(kernel, cudaFuncAttributeMaxDynamicSharedMemorySize, smem_size)); }; + if (dev >= 0 && dev < 64) + { + std::call_once(attrs_set[dev], set_attr); + } + else + { + set_attr(); + } + } + kernel<<>>( + layer_residual, res_weight, rms_weight, output, rsigma, probs, logits, T, B, rms_eps); +} + +} // namespace fwd_prod_v2 +} // namespace sm100 + +int attn_res_fwd_grid_size(int dev) +{ + static int cached_num_sm[64] = {}; + if (dev >= 0 && dev < 64 && cached_num_sm[dev] > 0) + { + return cached_num_sm[dev]; + } + int n = 0; + cudaError_t err = cudaDeviceGetAttribute(&n, cudaDevAttrMultiProcessorCount, dev); + if (err != cudaSuccess || n <= 0) + return 0; + if (dev >= 0 && dev < 64) + { + cached_num_sm[dev] = n; + } + return n; +} + +} // namespace + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels::kimiK3AttnRes +{ + +void invokeAttnResFwd(AttnResFwdParams const& params, cudaStream_t stream) +{ + using namespace sm100::fwd_prod_v2; + + bf16_t const* block_residual = params.blockResidual; + bf16_t const* layer_residual = params.layerResidual; + bf16_t const* res_weight = params.resWeight; + bf16_t const* rms_weight = params.rmsWeight; + bf16_t* output = params.output; + float* rsigma = params.rsigma; + float* probs = params.probs; + float* logits = params.logits; + int const N = params.numCandidates; + int const T = params.seqLen; + int const B = params.batchSize; + int const H = params.hiddenSize; + float const rms_eps = params.rmsEps; + + int dev = 0; + TLLM_CUDA_CHECK(cudaGetDevice(&dev)); + int num_sm = attn_res_fwd_grid_size(dev); + TLLM_CHECK_WITH_INFO(num_sm > 0, "attn_res_fwd: failed to query the SM count of device %d", dev); + TLLM_CHECK_WITH_INFO(N <= N_MAX, "attn_res_fwd: unsupported N=%d (max %d)", N, N_MAX); + + if (H == 8192) + { + if (N == 1) + { + launch_n1_ttile<8192, 2>( + layer_residual, res_weight, rms_weight, output, rsigma, probs, logits, T, B, rms_eps, num_sm, stream); + } + else if (N <= 2) + { + launch_fwd<8192, 2, true>(block_residual, layer_residual, res_weight, rms_weight, output, rsigma, probs, + logits, N, T, B, rms_eps, num_sm, stream); + } + else + { + launch_fwd<8192, 4, false>(block_residual, layer_residual, res_weight, rms_weight, output, rsigma, probs, + logits, N, T, B, rms_eps, num_sm, stream); + } + } + else if (H == 7168) + { + if (T == 1 && N == 1) + { + launch_s1_single_cta<1>( + block_residual, layer_residual, res_weight, rms_weight, output, rsigma, probs, logits, rms_eps, stream); + } + else if (T == 1 && N == 2) + { + launch_s1_single_cta<2>( + block_residual, layer_residual, res_weight, rms_weight, output, rsigma, probs, logits, rms_eps, stream); + } + else if (T == 1 && N == 4) + { + launch_s1_single_cta<4>( + block_residual, layer_residual, res_weight, rms_weight, output, rsigma, probs, logits, rms_eps, stream); + } + else if (T == 1 && N == 8) + { + launch_s1_splitk<8, 8>( + block_residual, layer_residual, res_weight, rms_weight, output, rsigma, probs, logits, rms_eps, stream); + } + else if (T == 1 && N == 12) + { + launch_s1_splitk<12, 8>( + block_residual, layer_residual, res_weight, rms_weight, output, rsigma, probs, logits, rms_eps, stream); + } + else if (N == 12 && T == 1024) + { + launch_fwd<7168, 4, false, true>(block_residual, layer_residual, res_weight, rms_weight, output, rsigma, + probs, logits, N, T, B, rms_eps, std::max(1, num_sm - 1), stream); + } + else + { + launch_fwd<7168, 4, false>(block_residual, layer_residual, res_weight, rms_weight, output, rsigma, probs, + logits, N, T, B, rms_eps, num_sm, stream); + } + } + else if (H == 6144) + { + launch_fwd<6144, 4, false>(block_residual, layer_residual, res_weight, rms_weight, output, rsigma, probs, + logits, N, T, B, rms_eps, num_sm, stream); + } + else if (H == 5120) + { + launch_fwd<5120, 4, false>(block_residual, layer_residual, res_weight, rms_weight, output, rsigma, probs, + logits, N, T, B, rms_eps, num_sm, stream); + } + else if (H == 4096) + { + if (N == 1) + { + launch_n1_ttile<4096, 4>( + layer_residual, res_weight, rms_weight, output, rsigma, probs, logits, T, B, rms_eps, num_sm, stream); + } + else if (N <= 4) + { + launch_fwd<4096, 2, true>(block_residual, layer_residual, res_weight, rms_weight, output, rsigma, probs, + logits, N, T, B, rms_eps, num_sm, stream); + } + else + { + launch_fwd<4096, 3, true>(block_residual, layer_residual, res_weight, rms_weight, output, rsigma, probs, + logits, N, T, B, rms_eps, num_sm, stream); + } + } + else + { + TLLM_CHECK_WITH_INFO(false, "attn_res_fwd: unsupported hidden size H=%d", H); + } +} + +} // namespace kernels::kimiK3AttnRes + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/kimiK3AttnRes/attnResFwd.h b/cpp/tensorrt_llm/kernels/kimiK3AttnRes/attnResFwd.h new file mode 100644 index 000000000000..65a9913f5afb --- /dev/null +++ b/cpp/tensorrt_llm/kernels/kimiK3AttnRes/attnResFwd.h @@ -0,0 +1,59 @@ +/* + * Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include "tensorrt_llm/common/config.h" + +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace kernels::kimiK3AttnRes +{ + +//! Parameters for the fused Kimi K3 attention-residual forward kernel +//! (warp-specialised online softmax + residual selection + RMSNorm, +//! Blackwell sm_100/sm_103 only). +//! +//! Contract (checked at the Torch-op bridge): B == 1, N in [1, 12], +//! T in [1, 16384], H a multiple of 1024 in [4096, 8192]; all residual +//! tensors bf16 contiguous, rsigma/probs/logits fp32 [N, T, B]. +//! blockResidual may be nullptr when N == 1. +struct AttnResFwdParams +{ + __nv_bfloat16 const* blockResidual; // [N-1, T, B, H], nullptr iff N == 1 + __nv_bfloat16 const* layerResidual; // [T, B, H] + __nv_bfloat16 const* resWeight; // [H] + __nv_bfloat16 const* rmsWeight; // [H] + __nv_bfloat16* output; // [T, B, H] + float* rsigma; // [N, T, B] + float* probs; // [N, T, B] + float* logits; // [N, T, B] + int numCandidates; // N = K + 1 + int seqLen; // T + int batchSize; // B + int hiddenSize; // H + float rmsEps; +}; + +//! Launches the fused attention-residual forward on the supplied stream. +void invokeAttnResFwd(AttnResFwdParams const& params, cudaStream_t stream); + +} // namespace kernels::kimiK3AttnRes + +TRTLLM_NAMESPACE_END diff --git a/cpp/tensorrt_llm/kernels/lora/lora.cpp b/cpp/tensorrt_llm/kernels/lora/lora.cpp index 7a2b7d330afc..14732922fcc5 100644 --- a/cpp/tensorrt_llm/kernels/lora/lora.cpp +++ b/cpp/tensorrt_llm/kernels/lora/lora.cpp @@ -1,5 +1,5 @@ /* - * SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & + * SPDX-FileCopyrightText: Copyright (c) 1993-2026 NVIDIA CORPORATION & * AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 * * Licensed under the Apache License, Version 2.0 (the "License"); @@ -99,6 +99,13 @@ void LoraImpl::setGemmConfig() mCublasWrapper->setBF16GemmConfig(); } #endif +#ifdef ENABLE_FP8 + else if (mType == tensorrt_llm::DataType::kFP8) + { + // FP8 uses the grouped GEMM path (CUTLASS 3.x), not cuBLAS. + // No cuBLAS config needed; this is a no-op. + } +#endif } int64_t getLowRankWorkSpaceSize(int64_t numTokens, int64_t maxLoraModuleNum, int64_t maxLowRank, int64_t typeSize) @@ -108,7 +115,10 @@ int64_t getLowRankWorkSpaceSize(int64_t numTokens, int64_t maxLoraModuleNum, int int64_t getGemmParamsWorkSpaceSize(int64_t nbReq) { - return std::max(getSplitkGroupedGemmParamsWorkSpaceSize(nbReq), getGroupedGemmParamsWorkSpaceSize(nbReq)); + int64_t size = std::max(getSplitkGroupedGemmParamsWorkSpaceSize(nbReq), getGroupedGemmParamsWorkSpaceSize(nbReq)); + // The FP8 path (CUTLASS 3.x) needs additional space for cute strides. + size = std::max(size, getFp8GroupedGemmParamsWorkSpaceSize(nbReq)); + return size; } int64_t getSplitkGroupedGemmWorkSpaceSize( @@ -177,6 +187,15 @@ int LoraImpl::run(int64_t numTokens, int64_t numReqs, void const* input, int32_t char* useUnifiedGemmChar = std::getenv("LORA_USE_UNIFIED_GEMM"); bool useUnifiedGemm = (useUnifiedGemmChar == nullptr || std::string(useUnifiedGemmChar) != "OFF"); +#ifdef ENABLE_FP8 + // The cuBLAS wrapper used by the unified GEMM path does not have an fp8 config. + // Force the grouped GEMM path for fp8, which uses the CUTLASS 3.x collective API. + if (mType == tensorrt_llm::DataType::kFP8) + { + useUnifiedGemm = false; + } +#endif + for (int loraModuleIdx = 0; loraModuleIdx < mNumLoraModules; loraModuleIdx++) { auto const loraRankModule = loraRanks[loraModuleIdx * numTokens]; diff --git a/cpp/tensorrt_llm/kernels/marlin/marlin.cuh b/cpp/tensorrt_llm/kernels/marlin/marlin.cuh index b04a42b66cc3..8b5c40a52dcd 100644 --- a/cpp/tensorrt_llm/kernels/marlin/marlin.cuh +++ b/cpp/tensorrt_llm/kernels/marlin/marlin.cuh @@ -41,6 +41,15 @@ #define MARLIN_NVFP4_DEVICE_SUPPORTED 0 #endif +// Dense Marlin also supports Blackwell GeForce (SM120/121). Keep this +// separate from MARLIN_NVFP4_DEVICE_SUPPORTED because Marlin MoE does not. +#if defined(__CUDA_ARCH__) \ + && ((__CUDA_ARCH__ >= 890 && __CUDA_ARCH__ < 1000) || (__CUDA_ARCH__ >= 1200 && __CUDA_ARCH__ < 1300)) +#define MARLIN_NVFP4_DENSE_DEVICE_SUPPORTED 1 +#else +#define MARLIN_NVFP4_DENSE_DEVICE_SUPPORTED 0 +#endif + namespace MARLIN_NAMESPACE_NAME { diff --git a/cpp/tensorrt_llm/kernels/marlin/marlin_nvfp4.h b/cpp/tensorrt_llm/kernels/marlin/marlin_nvfp4.h index 932baba29f63..13194fd76e29 100644 --- a/cpp/tensorrt_llm/kernels/marlin/marlin_nvfp4.h +++ b/cpp/tensorrt_llm/kernels/marlin/marlin_nvfp4.h @@ -30,6 +30,11 @@ inline bool isMarlinNvfp4SmSupported(int sm) return sm >= 89 && sm < 100; } +inline bool isMarlinNvfp4DenseSmSupported(int sm) +{ + return isMarlinNvfp4SmSupported(sm) || sm == 120 || sm == 121; +} + void dequantFp4Activations( void const* act_fp4, void const* act_sf, float const* alpha, void* act_bf16, int m, int k, cudaStream_t stream); diff --git a/cpp/tensorrt_llm/kernels/marlin/marlin_nvfp4_gemm.cu b/cpp/tensorrt_llm/kernels/marlin/marlin_nvfp4_gemm.cu index 185b6d370fb9..294d6c75f858 100644 --- a/cpp/tensorrt_llm/kernels/marlin/marlin_nvfp4_gemm.cu +++ b/cpp/tensorrt_llm/kernels/marlin/marlin_nvfp4_gemm.cu @@ -1,5 +1,5 @@ /* - * SPDX-FileCopyrightText: Copyright (c) 2022-2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-FileCopyrightText: Copyright (c) 2022-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. * SPDX-License-Identifier: Apache-2.0 * * Licensed under the Apache License, Version 2.0 (the "License"); @@ -327,8 +327,8 @@ void marlinNvfp4Gemm(void const* act_bf16, void const* weight, void* output, voi bool use_fp32_reduce, cudaStream_t stream) { int const sm = tensorrt_llm::common::getSMVersion(); - TLLM_CHECK_WITH_INFO(isMarlinNvfp4SmSupported(sm), - "Marlin NVFP4 GEMM is only supported on Ada (SM89) and Hopper (SM90-99); current SM = %d", sm); + TLLM_CHECK_WITH_INFO(isMarlinNvfp4DenseSmSupported(sm), + "Marlin NVFP4 GEMM is only supported on SM89, SM90-99, and SM120/121; current SM = %d", sm); int dev; cudaGetDevice(&dev); diff --git a/cpp/tensorrt_llm/kernels/marlin/marlin_nvfp4_template.h b/cpp/tensorrt_llm/kernels/marlin/marlin_nvfp4_template.h index 1eb5b1657d88..b2c7bbaa1f85 100644 --- a/cpp/tensorrt_llm/kernels/marlin/marlin_nvfp4_template.h +++ b/cpp/tensorrt_llm/kernels/marlin/marlin_nvfp4_template.h @@ -49,8 +49,8 @@ namespace MARLIN_NAMESPACE_NAME { -// Empty kernel stub for unsupported device passes; see marlin.cuh. -#if defined(__CUDA_ARCH__) && !MARLIN_NVFP4_DEVICE_SUPPORTED +// Empty kernel stub outside the architectures supported by dense Marlin. +#if defined(__CUDA_ARCH__) && !MARLIN_NVFP4_DENSE_DEVICE_SUPPORTED template num_sparse_topk << std::endl << "sparse_kv_cache_pool: " << this->sparse_kv_cache_pool << std::endl << "sliding_window_kv_cache_pool: " << this->sliding_window_kv_cache_pool << std::endl - << "sparse_mla_topk_lens: " << this->sparse_mla_topk_lens << std::endl + << "sparse_attn_kv_lens: " << this->sparse_attn_kv_lens << std::endl << "sparse_attn_indices_block_size: " << this->sparse_attn_indices_block_size << std::endl << "sparse_attn_indices_stride: " << this->sparse_attn_indices_stride << std::endl; return ss.str(); diff --git a/cpp/tensorrt_llm/kernels/splitkGroupGemm.cu b/cpp/tensorrt_llm/kernels/splitkGroupGemm.cu index 1f63189ec657..d1aa210665af 100644 --- a/cpp/tensorrt_llm/kernels/splitkGroupGemm.cu +++ b/cpp/tensorrt_llm/kernels/splitkGroupGemm.cu @@ -1,5 +1,5 @@ /* - * SPDX-FileCopyrightText: Copyright (c) 1993-2024 NVIDIA CORPORATION & + * SPDX-FileCopyrightText: Copyright (c) 1993-2026 NVIDIA CORPORATION & * AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 * * Licensed under the Apache License, Version 2.0 (the "License"); @@ -15,6 +15,7 @@ * limitations under the License. */ +#include "groupGemm.h" #include "splitkGroupGemm.h" #include "cutlass/cutlass.h" @@ -35,7 +36,6 @@ TRTLLM_NAMESPACE_BEGIN namespace kernels { - int64_t inline getGemmCoordSize(int64_t problemCount) { return (int64_t) (tensorrt_llm::common::divUp(problemCount * sizeof(cutlass::gemm::GemmCoord), 16) * 16); @@ -232,6 +232,18 @@ void splitkGroupedGemm(std::vector const& problemSizes bool isLoraIn, tensorrt_llm::DataType dataType, int splitKSlices, int minKN, cudaStream_t stream) { TLLM_LOG_TRACE("%s start, isLoraIn: %d, minKN = %d", __PRETTY_FUNCTION__, static_cast(isLoraIn), minKN); + +#ifdef ENABLE_FP8 + if (dataType == tensorrt_llm::DataType::kFP8) + { + // CUTLASS 3.x has no split-K grouped FP8 kernel. Reuse the regular + // cooperative implementation, which is also the intended split-K fallback. + groupedGemm(problemSizes, ptrA, ptrB, ptrC, ptrD, gemmParamsWorkSpace, gemmParamsWorkSpaceSize, gemmWorkSpace, + gemmWorkSpaceSize, isLoraIn, dataType, minKN, stream); + return; + } +#endif // ENABLE_FP8 + if (isLoraIn) { // K >> N, like K = 1024, N = 8 diff --git a/cpp/tensorrt_llm/kernels/splitkGroupGemm.h b/cpp/tensorrt_llm/kernels/splitkGroupGemm.h index bcde457db7d2..9f78215fc5bd 100644 --- a/cpp/tensorrt_llm/kernels/splitkGroupGemm.h +++ b/cpp/tensorrt_llm/kernels/splitkGroupGemm.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2019-2024, NVIDIA CORPORATION. All rights reserved. + * Copyright (c) 2019-2026, NVIDIA CORPORATION. All rights reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/KernelRunner.cpp b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/KernelRunner.cpp index 3c5265f2cc4a..782772fa28c1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/KernelRunner.cpp +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/KernelRunner.cpp @@ -1,5 +1,5 @@ /* - * Copyright (c) 2020-2025, NVIDIA CORPORATION. All rights reserved. + * Copyright (c) 2020-2026, NVIDIA CORPORATION. All rights reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -14,6 +14,7 @@ * limitations under the License. */ +#include #include #include #include @@ -61,6 +62,22 @@ constexpr bool isSMCompatible(int gpuSM, SmVersion kernelSM) return false; } +// Maps the runner-local gated ActType to the generated batchedGemm::gemmGatedAct::ActType. +// The two enums evolve independently (the generator inserted SiTuGlu before None, shifting +// values), so a value-preserving cast is not safe. Relu2/Silu are element-wise activations +// (EltwiseActType) with no fused gated kernel counterpart and must never reach this mapping. +static batchedGemm::gemmGatedAct::ActType toGemmGatedActType(ActType actType) +{ + switch (actType) + { + case ActType::SwiGlu: return batchedGemm::gemmGatedAct::ActType::SwiGlu; + case ActType::SiTu: return batchedGemm::gemmGatedAct::ActType::SiTuGlu; + case ActType::Relu2: + case ActType::Silu: break; + } + TLLM_THROW("ActType %d has no fused gated kernel mapping", static_cast(actType)); +} + static inline bool skipQuirks(BatchedGemmConfig const& config) { // Skip kernels that are known to hang/crash. Keep a record here for future reference. @@ -117,6 +134,14 @@ TrtllmGenBatchedGemmRunner::TrtllmGenBatchedGemmRunner(TrtllmGenBatchedGemmRunne rejectReason.resize(bmm.getNumBatchedGemmConfigs()); } + // Only fused gated kernels compare the gated ActType; resolve the mapping once so a + // misconfigured runner (element-wise ActType + fusedAct) fails loudly at construction. + std::optional gatedActType; + if (mOptions.fusedAct) + { + gatedActType = toGemmGatedActType(mOptions.actType); + } + int gpuSM = tensorrt_llm::common::getSMVersion(); for (size_t i = 0; i < bmm.getNumBatchedGemmConfigs(); ++i) { @@ -149,6 +174,13 @@ TrtllmGenBatchedGemmRunner::TrtllmGenBatchedGemmRunner(TrtllmGenBatchedGemmRunne continue; } + // The host runner does not wire the multicast completion barrier pointers; reject any + // future metadata that enables C multicast instead of launching with null barriers. + if (!acceptIf(!options.mUseCMultiCast, "mUseCMultiCast is not supported by the host runner")) + { + continue; + } + if (!acceptIf(dtypeMatch, fmtstr("dtypeAB mismatch (kernel: %s/%s, expected canonical: %s/%s)", tg::dtypeToString(options.mDtypeA).c_str(), tg::dtypeToString(options.mDtypeB).c_str(), @@ -235,9 +267,9 @@ TrtllmGenBatchedGemmRunner::TrtllmGenBatchedGemmRunner(TrtllmGenBatchedGemmRunne if (options.mFusedAct) { - if (!acceptIf(options.mActType == static_cast(mOptions.actType), + if (!acceptIf(options.mActType == *gatedActType, fmtstr("actType mismatch (kernel: %d, expected: %d)", static_cast(options.mActType), - static_cast(mOptions.actType)))) + static_cast(*gatedActType)))) { continue; } @@ -305,7 +337,7 @@ size_t TrtllmGenBatchedGemmRunner::getWorkspaceSizeInBytes(int32_t m, int32_t n, std::vector const& batchedTokens, int32_t numTokens, int32_t numBatches, int32_t maxNumCtasInBatchDim, int32_t configIndex) const { - BatchedGemmData gemmData; + BatchedGemmData gemmData{}; auto bmm = BatchedGemmInterface(); @@ -330,7 +362,7 @@ void TrtllmGenBatchedGemmRunner::run(int32_t m, int32_t n, int32_t k, int32_t va { auto bmm = BatchedGemmInterface(); - BatchedGemmData gemmData; + BatchedGemmData gemmData{}; auto const configs = bmm.getBatchedGemmConfigs(); @@ -489,7 +521,7 @@ std::vector TrtllmGenBatchedGemmRunner::getValidConfigIndices(int32_t m int32_t multiProcessorCount = tensorrt_llm::common::getMultiProcessorCount(); - BatchedGemmData gemmData; + BatchedGemmData gemmData{}; // Sanitize optional valid dimensions validM = validM <= 0 ? m : validM; @@ -549,7 +581,7 @@ std::vector TrtllmGenBatchedGemmRunner::getValidConfigIndices(int32_t m // prefer persistent tile scheduler. if (optionsA.mTileScheduler != optionsB.mTileScheduler) { - BatchedGemmData gemmData; + BatchedGemmData gemmData{}; setProblemDimensions(gemmData, optionsA.mTransposeMmaOutput, m, n, k, batchedTokens, numTokens, numBatches, maxNumCtasInBatchDim, validM, validN, validK); auto options = bmm.getOptionsFromConfigAndData(configs[idx0], gemmData); @@ -608,7 +640,7 @@ bool TrtllmGenBatchedGemmRunner::isValidConfigIndex(int32_t configIndex, int32_t auto const bmm = BatchedGemmInterface(); auto const configs = bmm.getBatchedGemmConfigs(); - BatchedGemmData gemmData; + BatchedGemmData gemmData{}; // Sanitize optional valid dimensions validM = validM <= 0 ? m : validM; diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/KernelRunner.h b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/KernelRunner.h index 7fb404ea34bc..3a0fdcab7bdf 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/KernelRunner.h +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/KernelRunner.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2020-2025, NVIDIA CORPORATION. All rights reserved. + * Copyright (c) 2020-2026, NVIDIA CORPORATION. All rights reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -28,8 +28,11 @@ TRTLLM_NAMESPACE_BEGIN namespace kernels { -// Keep this in sync with the ActType in -// cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/GemmGatedActOptions.h +// Backend-local activation type. The numeric values here are part of the Python/thop +// contract (see ActType_TrtllmGen in tensorrt_llm/_torch/utils.py) and are intentionally +// decoupled from the generated batchedGemm::gemmGatedAct::ActType, whose values shift +// between generator exports. KernelRunner.cpp maps gated entries explicitly; never cast +// between the two enums by value. enum class ActType { // For ActType == SwiGlu, ideally we would like to have something like @@ -43,9 +46,24 @@ enum class ActType // GatedSilu is a special case of SwiGlu where the alpha is 1.0 and the beta is 0.0. SwiGlu, Relu2, - Silu + Silu, + // SiTu gated activation (Kimi K3). Gate on x1, matching the SwiGlu convention: + // left = beta * tanh(x0 / beta) + // right = alpha * tanh(x1 / alpha) * sigmoid(x1) + // gatedAct = left * right + // alpha/beta come from the per-expert mPtrGatedActAlpha/mPtrGatedActBeta runtime + // parameters and must be > 0. + SiTu }; +// Gated activations combine gate and up projections inside the FC1 kernel; the FC1 +// logical output width is 2 * intermediateSize. Element-wise activations (Relu2/Silu) +// run on a single projection. +constexpr bool isGatedActType(ActType actType) +{ + return actType == ActType::SwiGlu || actType == ActType::SiTu; +} + // Type of the element-wise activation to apply after the Gemm enum class EltwiseActType { diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/BatchedGemmInterface.h b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/BatchedGemmInterface.h index bb5592c95cf7..18411f44c831 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/BatchedGemmInterface.h +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/BatchedGemmInterface.h @@ -163,8 +163,7 @@ struct BatchedGemmData // This is used for either: // * Per-token scaling factor quantization schemes, such as MetaFP8. The dtype is // Dtype::Float32 - // * When the routing scales are applied to the input activations (only when output is not - // transposed). The dtype is Dtype::Bfloat16 + // * One-sided per-token scaling. The dtype is Dtype::Bfloat16 by default. // // if (batchM (A is activations)): // Logical shape is [sum(divUpMul(M[bi], tileM) for bi in B)] @@ -243,8 +242,7 @@ struct BatchedGemmData // This is used for either: // * Per-token scaling factor quantization schemes, such as MetaFP8. The dtype is // Dtype::Float32 - // * When the routing scales are applied to the input activations (only when output is - // transposed). The dtype is Dtype::Bfloat16 + // * One-sided per-token scaling. The dtype is Dtype::Bfloat16 by default. // // if (batchM (B is weights)): // Logical shape is [B, divUpMul(N, tileN)] @@ -478,6 +476,35 @@ struct BatchedGemmData // [divUp(numTokens + numBatches * (tileM/N - 1), tileM/N)] int32_t const* mPtrCtaIdxXyToMnLimit; + // Map from permuted token index to expanded token index. + // Used for MoE finalize direct-store path when mUseCMultiCast is enabled. + // The dtype is int32_t. Set to nullptr when not using CMulticast. + int32_t const* mPtrPermutedIdxToExpandedIdx{nullptr}; + + ////////////////////////////////////////////////////////////////////////////////////////////////// + // + // Multicast barrier parameters + // + ////////////////////////////////////////////////////////////////////////////////////////////////// + + // When mUseCMultiCast is set, we require a barrier in global memory on each + // GPU to track completion of the multicast stores. + // + // The value of the barrier must be set to zero before the kernel is executed on any GPU. + // The barrier is self-resetting. It does not have to be reset by another kernel. + // When mUseCMultiCast is false, a nullptr may be passed. + uint32_t* mPtrMulticastCompletionBarUc; + uint32_t* mPtrMulticastCompletionBarMc; + + ////////////////////////////////////////////////////////////////////////////////////////////////// + // + // MoE finalize parameters for direct register-to-gmem store + // + ////////////////////////////////////////////////////////////////////////////////////////////////// + + // Pointer to expert weights for MoE finalize. + // The dtype depends on the use case. + void const* mPtrExpertWeightsPtr{nullptr}; // Global counter for SW-emulated dynamic tile scheduling. When dynamic scheduling is enabled, // must be initialized to gridDim.x * gridDim.y (gridDim.z is reserved for K-dim in CGA splitK) // before each kernel launch. @@ -558,8 +585,9 @@ class BatchedGemmInterface ////////////////////////////////////////////////////////////////////////////////////////////////// - BatchedGemmInterface(bool const exportsCubin = false, int32_t const numRotations = 1) - : mExportsCubin(exportsCubin) + BatchedGemmInterface(int32_t rankId = 0, bool const exportsCubin = false, int32_t const numRotations = 1) + : mRankId(rankId) + , mExportsCubin(exportsCubin) , mNumRotations(numRotations) { } @@ -655,7 +683,12 @@ class BatchedGemmInterface batchedGemmData.mInputBuffers.mPtrScaleGate, batchedGemmData.mInputBuffers.mPtrClampLimit, batchedGemmData.mInputBuffers.mPtrGatedActAlpha, batchedGemmData.mInputBuffers.mPtrGatedActBeta, batchedGemmData.mInputBuffers.mPtrRouteMap, dPtrRowMax, dPtrRowMaxBars, - batchedGemmData.mInputBuffers.mPtrNumNonExitingCtas, batchedGemmData.mInputBuffers.mPtrTotalNumPaddedTokens, + batchedGemmData.mProblemDimensions.mRank, batchedGemmData.mProblemDimensions.mWorldSize, + batchedGemmData.mInputBuffers.mPtrMulticastCompletionBarUc, + batchedGemmData.mInputBuffers.mPtrMulticastCompletionBarMc, + batchedGemmData.mInputBuffers.mPtrPermutedIdxToExpandedIdx, + batchedGemmData.mInputBuffers.mPtrExpertWeightsPtr, batchedGemmData.mInputBuffers.mPtrNumNonExitingCtas, + batchedGemmData.mInputBuffers.mPtrTotalNumPaddedTokens, batchedGemmData.mInputBuffers.mPtrCtaIdxXyToBatchIdx, batchedGemmData.mInputBuffers.mPtrCtaIdxXyToMnLimit, numCtaBatch, batchedGemmData.mInputBuffers.mPtrDynamicTileCounter); @@ -925,8 +958,14 @@ class BatchedGemmInterface auto options = getOptionsFromConfigAndData(config, data); // Check options without modifications. + // + // NOTE(TRT-LLM local fix over producer output a2ad0544-dirty): the generator + // inserted the `worldSize` parameter into checkAndUpdateBatchedGemmOptions + // without updating this call, so the literal `false` bound positionally to + // `worldSize` (=0) and `updateOptions` silently took its default `true`. + // Pass both arguments explicitly. Report upstream before the next re-export. return checkAndUpdateBatchedGemmOptions(options, config.mSm, - /* updateOptions */ false); + /* worldSize */ 1, /* updateOptions */ false); } ////////////////////////////////////////////////////////////////////////////////////////////////// @@ -1013,6 +1052,8 @@ class BatchedGemmInterface ////////////////////////////////////////////////////////////////////////////////////////////////// private: + // The rank id of the current device in the multi-gpu space. + int32_t mRankId; // Whether to export the cubin file. bool mExportsCubin; // The number of rotations. diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/BatchedGemmOptions.h b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/BatchedGemmOptions.h index a3423b611a73..26ae811ba0f4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/BatchedGemmOptions.h +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/BatchedGemmOptions.h @@ -98,8 +98,8 @@ struct BatchedGemmOptions : public gemmGatedAct::GemmGatedActOptions BatchedGemmOptions(gemm::AllReduceAlgo allReduceAlgo, tg::Dtype biasDtype, gemm::BiasType biasType, int blockK, bool clcFastDrain, int clusterDimX, int clusterDimY, int clusterDimZ, gemm::CtaSwizzleType ctaSwizzleType, tg::Dtype dtypeAcc, tg::Dtype dtypeA, tg::Dtype dtypeB, tg::Dtype dtypeC, tg::Dtype dtypeMmaA, - tg::Dtype dtypeMmaB, gemm::EltwiseActType eltwiseActType, bool enablesEarlyExit, bool enablesDelayedEarlyExit, - bool enablesGlobalPtxKnobs, int epilogueLdtmDps, int epilogueLdtmBits, int epilogueTileM, int epilogueTileN, + tg::Dtype dtypeMmaB, tg::Dtype dtypeSfC, gemm::EltwiseActType eltwiseActType, bool enablesEarlyExit, + bool enablesDelayedEarlyExit, int epilogueLdtmDps, int epilogueLdtmBits, int epilogueTileM, int epilogueTileN, int fallbackClusterDimX, int fallbackClusterDimY, int fallbackClusterDimZ, gemm::FusedBiasShuffleMode fusedBiasShuffleMode, bool fuseLoadSfTask, bool fuseUtccpWithUtcmma, bool gridTriggerSecondaryA, bool gridTriggerSecondaryB, bool gridWaitForPrimaryEarlyExit, @@ -107,17 +107,17 @@ struct BatchedGemmOptions : public gemmGatedAct::GemmGatedActOptions gemm::KernelTraits kernelTraits, gemm::MatrixLayout layoutA, gemm::MatrixLayout layoutB, int m, int mmaK, tg::MmaKind mmaKind, int mmaM, int mmaN, int mmaTileK, bool mockAllReduce, int n, int numEpilogueWarps, int numRegsCastAWarps, int numRegsCopySfLdsSttm, int numRegsCopySparsityInfo, int numRegsPerThreadEpilogueWarp, - int numRegsPerThreadNonEpilogueWarp, int numSlicesForSplitK, int numSlicesForSliceK, int numStagesA, - int numStagesB, int numStagesMma, int numStagesMmaWithinWorkTile, int numStagesMmaAcrossWorkTile, - int numStagesSmemSfA, int numStagesSmemSfB, int numStagesTmemA, int numStagesTmemSfA, int numStagesTmemSfB, - int numStagesWorkId, bool outputDebugTensors, bool patchF2fp, tg::Dtype perTokenSfDtype, - gemm::SchedHostTask schedHostTask, int32_t sfBlockSizeA, int32_t sfBlockSizeB, int32_t sfBlockSizeC, - tg::SfLayout sfLayoutA, tg::SfLayout sfLayoutB, tg::SfLayout sfLayoutC, int32_t sfReshapeFactor, bool sliceK, - tg::Sparsity sparsityA, gemm::SplitK splitK, int tileK, int tileM, int tileN, gemm::TileScheduler tileScheduler, - bool transposeMmaOutput, bool useCustomizedMma3xNvFp4, bool useCustomMmaSchedule, bool useDeepSeekFp8, - bool useFlexibleClusterDims, bool useHoistTryWaitForCustomMmaSchedule, bool useMaxTmemOverlap, - bool usePerTokenSfA, bool usePerTokenSfB, bool useShuffledMatrix, bool useTmaStore, bool useTwoTmaLoadWarps, - bool useTwoMmaWarps, bool useUnrollLoop2xForMma, int validM, int validN, int validK, int worldSize, + int numRegsPerThreadNonEpilogueWarp, int numSlicesForSplitK, int numStagesA, int numStagesB, int numStagesMma, + int numStagesMmaWithinWorkTile, int numStagesMmaAcrossWorkTile, int numStagesSmemSfA, int numStagesSmemSfB, + int numStagesTmemSfA, int numStagesTmemSfB, int numStagesWorkId, bool outputDebugTensors, bool patchF2fp, + tg::Dtype perTokenSfDtype, gemm::SchedHostTask schedHostTask, int32_t sfBlockSizeA, int32_t sfBlockSizeB, + int32_t sfBlockSizeC, tg::SfLayout sfLayoutA, tg::SfLayout sfLayoutB, tg::SfLayout sfLayoutC, + int32_t sfReshapeFactor, tg::Sparsity sparsityA, gemm::SplitK splitK, int tileK, int tileM, int tileN, + gemm::TileScheduler tileScheduler, bool transposeMmaOutput, bool useCustomizedMma3xNvFp4, + bool useCustomMmaSchedule, bool useDeepSeekFp8, bool useFlexibleClusterDims, + bool useHoistTryWaitForCustomMmaSchedule, bool useMaxTmemOverlap, bool usePerTokenSfA, bool usePerTokenSfB, + bool useShuffledMatrix, bool useTmaStore, bool useTwoTmaLoadWarps, bool useTwoMmaWarps, + bool useUnrollLoop2xForMma, int validM, int validN, int validK, int worldSize, // GemmGatedActOptions gemmGatedAct::ActType actType, bool clampBeforeAct, // BatchedGemmOptions @@ -125,26 +125,26 @@ struct BatchedGemmOptions : public gemmGatedAct::GemmGatedActOptions bool fusedAct, bool gridWaitForPrimaryRouting, bool isStaticBatch, bool isUniformNumTokensPerBatch, int numBatches, int numRegsPerThreadLoadA, int numRegsPerThreadLoadB, int numRegsPerThreadLoadSfA, int numRegsPerThreadLoadSfB, int numTokens, int numWarpsLoadA, int numWarpsLoadB, int numWarpsLoadSfA, - int numWarpsLoadSfB, RouteImpl routeImpl, std::optional routeSfsImpl, bool useTmaOobOpt, - bool prefetchB) + int numWarpsLoadSfB, RouteImpl routeImpl, std::optional routeSfsImpl, bool prefetchB, + bool useCMulticast, bool useTmaOobOpt) : gemmGatedAct::GemmGatedActOptions( gemm::GemmOptions(allReduceAlgo, biasDtype, biasType, blockK, clcFastDrain, clusterDimX, clusterDimY, - clusterDimZ, ctaSwizzleType, dtypeAcc, dtypeA, dtypeB, dtypeC, dtypeMmaA, dtypeMmaB, eltwiseActType, - enablesEarlyExit, enablesDelayedEarlyExit, enablesGlobalPtxKnobs, epilogueLdtmDps, epilogueLdtmBits, + clusterDimZ, ctaSwizzleType, dtypeAcc, dtypeA, dtypeB, dtypeC, dtypeMmaA, dtypeMmaB, dtypeSfC, + eltwiseActType, enablesEarlyExit, enablesDelayedEarlyExit, epilogueLdtmDps, epilogueLdtmBits, epilogueTileM, epilogueTileN, fallbackClusterDimX, fallbackClusterDimY, fallbackClusterDimZ, fusedBiasShuffleMode, fuseLoadSfTask, fuseUtccpWithUtcmma, gridTriggerSecondaryA, gridTriggerSecondaryB, gridWaitForPrimaryEarlyExit, gridWaitForPrimaryA, gridWaitForPrimaryB, hoistLoadTaskInit, hoistMmaTaskTryWaits, k, kernelTraits, layoutA, layoutB, m, mmaK, mmaKind, mmaM, mmaN, mmaTileK, mockAllReduce, n, numEpilogueWarps, numRegsCastAWarps, numRegsCopySfLdsSttm, numRegsCopySparsityInfo, - numRegsPerThreadEpilogueWarp, numRegsPerThreadNonEpilogueWarp, numSlicesForSplitK, numSlicesForSliceK, - numStagesA, numStagesB, numStagesMma, numStagesMmaWithinWorkTile, numStagesMmaAcrossWorkTile, - numStagesSmemSfA, numStagesSmemSfB, numStagesTmemA, numStagesTmemSfA, numStagesTmemSfB, numStagesWorkId, - outputDebugTensors, patchF2fp, perTokenSfDtype, schedHostTask, sfBlockSizeA, sfBlockSizeB, sfBlockSizeC, - sfLayoutA, sfLayoutB, sfLayoutC, sfReshapeFactor, sliceK, sparsityA, splitK, tileK, tileM, tileN, - tileScheduler, transposeMmaOutput, useCustomizedMma3xNvFp4, useCustomMmaSchedule, useDeepSeekFp8, - useFlexibleClusterDims, useHoistTryWaitForCustomMmaSchedule, useMaxTmemOverlap, usePerTokenSfA, - usePerTokenSfB, useShuffledMatrix, useTmaStore, useTwoTmaLoadWarps, useTwoMmaWarps, - useUnrollLoop2xForMma, validM, validN, validK, worldSize), + numRegsPerThreadEpilogueWarp, numRegsPerThreadNonEpilogueWarp, numSlicesForSplitK, numStagesA, + numStagesB, numStagesMma, numStagesMmaWithinWorkTile, numStagesMmaAcrossWorkTile, numStagesSmemSfA, + numStagesSmemSfB, numStagesTmemSfA, numStagesTmemSfB, numStagesWorkId, outputDebugTensors, patchF2fp, + perTokenSfDtype, schedHostTask, sfBlockSizeA, sfBlockSizeB, sfBlockSizeC, sfLayoutA, sfLayoutB, + sfLayoutC, sfReshapeFactor, sparsityA, splitK, tileK, tileM, tileN, tileScheduler, transposeMmaOutput, + useCustomizedMma3xNvFp4, useCustomMmaSchedule, useDeepSeekFp8, useFlexibleClusterDims, + useHoistTryWaitForCustomMmaSchedule, useMaxTmemOverlap, usePerTokenSfA, usePerTokenSfB, + useShuffledMatrix, useTmaStore, useTwoTmaLoadWarps, useTwoMmaWarps, useUnrollLoop2xForMma, validM, + validN, validK, worldSize), actType, clampBeforeAct) , mBatchedM(batchedM) , mBatchedN(batchedN) @@ -166,8 +166,9 @@ struct BatchedGemmOptions : public gemmGatedAct::GemmGatedActOptions , mNumWarpsLoadSfB{numWarpsLoadSfB} , mRouteImpl(routeImpl) , mRouteSfsImpl(routeSfsImpl) - , mUseTmaOobOpt(useTmaOobOpt) , mPrefetchB(prefetchB) + , mUseCMultiCast(useCMulticast) + , mUseTmaOobOpt(useTmaOobOpt) { } @@ -214,16 +215,18 @@ struct BatchedGemmOptions : public gemmGatedAct::GemmGatedActOptions std::optional mRouteSfsImpl{std::nullopt}; // Whether to use TMA out-of-bounds optimization to reduce wasted traffic. See details in // BatchedGemm/KernelParamsDecl.h. - bool mUseTmaOobOpt{false}; - // Whether to prefetch B (weights) into L2. bool mPrefetchB{false}; + // Whether to use multicast store for output C. + bool mUseCMultiCast{false}; + // Whether to use TMA out-of-bounds optimization. + bool mUseTmaOobOpt{false}; }; //////////////////////////////////////////////////////////////////////////////////////////////////// // Check if the options are valid or not. inline bool checkAndUpdateBatchedGemmOptions( - BatchedGemmOptions& options, tg::CudaArch cudaArch, bool updateOptions = true) + BatchedGemmOptions& options, tg::CudaArch cudaArch, int worldSize = 1, bool updateOptions = true) { bool isValid = true; if (options.mUseTmaOobOpt && !options.mUseTwoTmaLoadWarps) @@ -243,6 +246,7 @@ inline bool checkAndUpdateBatchedGemmOptions( if (options.mFusedAct) { // ensure that we check the fused options as well + TLLM_CHECK_ERROR(!options.mUseCMultiCast, "C Multicast is not supported with fusedAct option."); isValid = gemmGatedAct::checkAndUpdateGemmGatedActOptions(options, cudaArch, updateOptions); } else @@ -250,9 +254,13 @@ inline bool checkAndUpdateBatchedGemmOptions( TLLM_CHECK_ERROR(!gemm::usesFusedBiasReorder(options.mFusedBiasShuffleMode), "FusedBiasShuffleMode::ReorderAndShuffle is only supported for fused-act/" "gated-act paths"); - isValid = gemm::checkAndUpdateGemmOptions(options, cudaArch, 1 /* tpGrpSize */, updateOptions); + isValid = gemm::checkAndUpdateGemmOptions(options, cudaArch, worldSize, updateOptions); + } + if (options.mUseCMultiCast) + { + bool const useBlockScalingOut = tg::dtypeIsBlockFmt(options.mDtypeC); + TLLM_CHECK_ERROR(!useBlockScalingOut, "C Multicast is not supported with block scaling."); } - bool batchM = options.mBatchMode == BatchedGemmOptions::BatchMode::BatchM; if (options.mPrefetchB) { @@ -571,8 +579,9 @@ inline std::string dumpOptions(BatchedGemmOptions const& options, bool dumpRunti ss << "mRouteImpl=batchedGemm::RouteImpl(" << static_cast(options.mRouteImpl) << ")," << std::endl; ss << "mRouteSfsImpl={batchedGemm::RouteImpl(" << static_cast(options.mRouteSfsImpl.value()) << ")}," << std::endl; - ss << "mUseTmaOobOpt=" << options.mUseTmaOobOpt << "," << std::endl; - ss << "mPrefetchB=" << options.mPrefetchB << std::endl; + ss << "mPrefetchB=" << options.mPrefetchB << "," << std::endl; + ss << "mUseCMultiCast=" << options.mUseCMultiCast << "," << std::endl; + ss << "mUseTmaOobOpt=" << options.mUseTmaOobOpt << std::endl; return ss.str(); } diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/GemmGatedActOptions.h b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/GemmGatedActOptions.h index f3364f91a45c..0850af47bd47 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/GemmGatedActOptions.h +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/GemmGatedActOptions.h @@ -83,6 +83,13 @@ enum class ActType // where x0 and x1 are the raw numbers from Gemm, while scaleC and scaleGate are input scales, // beta' = beta / scaleAb, scaleC' = scaleC * scaleAb. GeGlu, + // For ActType == SiTuGlu, the formula is (gate on x1, matching SwiGlu convention): + // left = beta * tanh(x0 / beta) + // right = alpha * tanh(x1 / alpha) * sigmoid(x1) + // gatedAct = left * right + // where alpha and beta are the gatedActAlpha and gatedActBeta parameters respectively. + // Both alpha and beta must be > 0. Default values are 1.0 for both. + SiTuGlu, // Placeholder for no activation; not implemented in codegen None, }; @@ -97,8 +104,9 @@ enum class ActType return (type == ActType::actType); \ } -TLLM_ACT_TYPE_FUNCTION(SwiGlu) TLLM_ACT_TYPE_FUNCTION(GeGlu) +TLLM_ACT_TYPE_FUNCTION(SiTuGlu) +TLLM_ACT_TYPE_FUNCTION(SwiGlu) #undef TLLM_ACT_TYPE_FUNCTION @@ -110,12 +118,25 @@ inline std::string getActTypeName(ActType type) { case ActType::SwiGlu: return "SwiGlu"; case ActType::GeGlu: return "GeGlu"; + case ActType::SiTuGlu: return "SiTuGlu"; default: return "Unknown type"; } } //////////////////////////////////////////////////////////////////////////////////////////////////// +// Returns true for activations where x0 appears only in linear terms (e.g., x0+beta in SwiGlu). +// Returns false for activations where x0 feeds into nonlinearities (e.g., tanh(x0/beta) in +// SiTuGlu). +// +// Drives: scaleC placement in applyGatedAcc, test harness scaleC/beta computation. +inline bool isLinearInX0(ActType type) +{ + return isGeGlu(type) || isSwiGlu(type); +} + +//////////////////////////////////////////////////////////////////////////////////////////////////// + struct GemmGatedActOptions : public gemm::GemmOptions { GemmGatedActOptions() = default; diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/GemmOptions.h b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/GemmOptions.h index 866f3949a6d7..95555c93d86b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/GemmOptions.h +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/GemmOptions.h @@ -129,7 +129,7 @@ struct GemmOptions GemmOptions(AllReduceAlgo allReduceAlgo, tg::Dtype biasDtype, BiasType biasType, int blockK, bool clcFastDrain, int clusterDimX, int clusterDimY, int clusterDimZ, CtaSwizzleType ctaSwizzleType, tg::Dtype dtypeAcc, tg::Dtype dtypeA, tg::Dtype dtypeB, tg::Dtype dtypeC, tg::Dtype dtypeMmaA, tg::Dtype dtypeMmaB, - EltwiseActType eltwiseActType, bool enablesEarlyExit, bool enablesDelayedEarlyExit, bool enablesGlobalPtxKnobs, + tg::Dtype dtypeSfC, EltwiseActType eltwiseActType, bool enablesEarlyExit, bool enablesDelayedEarlyExit, int epilogueLdtmDps, int epilogueLdtmBits, int epilogueTileM, int epilogueTileN, int fallbackClusterDimX, int fallbackClusterDimY, int fallbackClusterDimZ, FusedBiasShuffleMode fusedBiasShuffleMode, bool fuseLoadSfTask, bool fuseUtccpWithUtcmma, bool gridTriggerSecondaryA, bool gridTriggerSecondaryB, @@ -137,18 +137,17 @@ struct GemmOptions bool hoistMmaTaskTryWaits, int k, KernelTraits kernelTraits, MatrixLayout layoutA, MatrixLayout layoutB, int m, int mmaK, tg::MmaKind mmaKind, int mmaM, int mmaN, int mmaTileK, bool mockAllReduce, int n, int numEpilogueWarps, int numRegsCastAWarps, int numRegsCopySfLdsSttm, int numRegsCopySparsityInfo, - int numRegsPerThreadEpilogueWarp, int numRegsPerThreadNonEpilogueWarp, int numSlicesForSplitK, - int numSlicesForSliceK, int numStagesA, int numStagesB, int numStagesMma, int numStagesMmaWithinWorkTile, - int numStagesMmaAcrossWorkTile, int numStagesSmemSfA, int numStagesSmemSfB, int numStagesTmemA, - int numStagesTmemSfA, int numStagesTmemSfB, int numStagesWorkId, bool outputDebugTensors, bool patchF2fp, - tg::Dtype perTokenSfDtype, SchedHostTask schedHostTask, int32_t sfBlockSizeA, int32_t sfBlockSizeB, - int32_t sfBlockSizeC, tg::SfLayout sfLayoutA, tg::SfLayout sfLayoutB, tg::SfLayout sfLayoutC, - int sfReshapeFactor, bool sliceK, tg::Sparsity sparsityA, SplitK splitK, int tileK, int tileM, int tileN, - TileScheduler tileScheduler, bool transposeMmaOutput, bool useCustomizedMma3xNvFp4, bool useCustomMmaSchedule, - bool useDeepSeekFp8, bool useFlexibleClusterDims, bool useHoistTryWaitForCustomMmaSchedule, - bool useMaxTmemOverlap, bool usePerTokenSfA, bool usePerTokenSfB, bool useShuffledMatrix, bool useTmaStore, - bool useTwoTmaLoadWarps, bool useTwoMmaWarps, bool useUnrollLoop2xForMma, int validM, int validN, int validK, - int worldSize) + int numRegsPerThreadEpilogueWarp, int numRegsPerThreadNonEpilogueWarp, int numSlicesForSplitK, int numStagesA, + int numStagesB, int numStagesMma, int numStagesMmaWithinWorkTile, int numStagesMmaAcrossWorkTile, + int numStagesSmemSfA, int numStagesSmemSfB, int numStagesTmemSfA, int numStagesTmemSfB, int numStagesWorkId, + bool outputDebugTensors, bool patchF2fp, tg::Dtype perTokenSfDtype, SchedHostTask schedHostTask, + int32_t sfBlockSizeA, int32_t sfBlockSizeB, int32_t sfBlockSizeC, tg::SfLayout sfLayoutA, + tg::SfLayout sfLayoutB, tg::SfLayout sfLayoutC, int sfReshapeFactor, tg::Sparsity sparsityA, SplitK splitK, + int tileK, int tileM, int tileN, TileScheduler tileScheduler, bool transposeMmaOutput, + bool useCustomizedMma3xNvFp4, bool useCustomMmaSchedule, bool useDeepSeekFp8, bool useFlexibleClusterDims, + bool useHoistTryWaitForCustomMmaSchedule, bool useMaxTmemOverlap, bool usePerTokenSfA, bool usePerTokenSfB, + bool useShuffledMatrix, bool useTmaStore, bool useTwoTmaLoadWarps, bool useTwoMmaWarps, + bool useUnrollLoop2xForMma, int validM, int validN, int validK, int worldSize) : mAllReduceAlgo{allReduceAlgo} , mBiasDtype{biasDtype} , mBiasType{biasType} @@ -164,10 +163,10 @@ struct GemmOptions , mDtypeC{dtypeC} , mDtypeMmaA{dtypeMmaA} , mDtypeMmaB{dtypeMmaB} + , mDtypeSfC{dtypeSfC} , mEltwiseActType{eltwiseActType} , mEnablesEarlyExit{enablesEarlyExit} , mEnablesDelayedEarlyExit{enablesDelayedEarlyExit} - , mEnablesGlobalPtxKnobs{enablesGlobalPtxKnobs} , mEpilogueLdtmDps{epilogueLdtmDps} , mEpilogueLdtmBits{epilogueLdtmBits} , mEpilogueTileM{epilogueTileM} @@ -204,7 +203,6 @@ struct GemmOptions , mNumRegsPerThreadEpilogueWarp(numRegsPerThreadEpilogueWarp) , mNumRegsPerThreadNonEpilogueWarp(numRegsPerThreadNonEpilogueWarp) , mNumSlicesForSplitK{numSlicesForSplitK} - , mNumSlicesForSliceK{numSlicesForSliceK} , mNumStagesA{numStagesA} , mNumStagesB{numStagesB} , mNumStagesMma{numStagesMma} @@ -212,7 +210,6 @@ struct GemmOptions , mNumStagesMmaAcrossWorkTile{numStagesMmaAcrossWorkTile} , mNumStagesSmemSfA{numStagesSmemSfA} , mNumStagesSmemSfB{numStagesSmemSfB} - , mNumStagesTmemA{numStagesTmemA} , mNumStagesTmemSfA{numStagesTmemSfA} , mNumStagesTmemSfB{numStagesTmemSfB} , mNumStagesWorkId{numStagesWorkId} @@ -227,7 +224,6 @@ struct GemmOptions , mSfLayoutB{sfLayoutB} , mSfLayoutC{sfLayoutC} , mSfReshapeFactor{sfReshapeFactor} - , mSliceK{sliceK} , mSparsityA{sparsityA} , mSplitK{splitK} , mTileK{tileK} @@ -285,6 +281,10 @@ struct GemmOptions tg::Dtype mDtypeMmaA{tg::Dtype::Void}; // Data type of the B matrix for the MMA, if different from the input type. tg::Dtype mDtypeMmaB{tg::Dtype::Void}; + // Data type of the block scaling factors of the output C matrix. When Void, it is auto-deduced + // from mDtypeC by checkAndUpdateGemmOptions (e.g., E4m3 for E2m1, UE8m0 for Mx formats). + // Currently supported override: Fp32 (to keep the high-precision scaling factors). + tg::Dtype mDtypeSfC{tg::Dtype::Void}; // The type of activation. EltwiseActType mEltwiseActType{EltwiseActType::None}; // Whether to enable early exit. @@ -292,8 +292,6 @@ struct GemmOptions // Whether to enable delayed early exit to overlap // numNonExitingCtas loading with the other instructions. bool mEnablesDelayedEarlyExit{false}; - // Whether to enable the global PTX knobs for guiding the compiler optimizations. - bool mEnablesGlobalPtxKnobs{true}; // The epilogue supports multiple LDTM shapes, although not every shape is applicable in every // case. In particular: // - On Hopper: must be 16dp256bit. @@ -323,9 +321,6 @@ struct GemmOptions bool mFuseLoadSfTask{false}; // Whether fuse UTCCP with UTC*MMA. bool mFuseUtccpWithUtcmma{false}; - // Hybrid SliceK: K-tile folding factor (32/mTMA for mTMA < 32, else 1). - // Computed in validation from mHybridTileHidden. - int mFundamentalKtileUnit{1}; // Whether load task A triggers the next grid. bool mGridTriggerSecondaryA{false}; // Whether load task B triggers the next grid. @@ -340,23 +335,6 @@ struct GemmOptions bool mHoistLoadTaskInit{true}; // Whether to hoist the mbarrier try_waits (e.g., mma.prodAcq, smemAb.consWait) in the MMA task. bool mHoistMmaTaskTryWaits{false}; - // Hidden/output dimension for Hybrid SliceK (M if transpose, N if not). - // Computed in validation based on transpose mode. - int32_t mHybridHiddenDim{0}; - // Token batch dimension for Hybrid SliceK (N if transpose, M if not). - // Computed in validation based on transpose mode. - int32_t mHybridNumTokens{0}; - // "Hybrid" SliceK: for mTMA >= 32, uses progressively sparser lane masks (standard - // approach). For mTMA < 32, switches to a hybrid strategy that folds K-slices into - // the M-dimension (FundamentalKtileUnit) to maintain a fixed 0x11111111 lane mask, - // avoiding the wasted TC lanes of ever-sparser masks. Supports M_tma tiles 4-128. - bool mHybridSliceK{false}; - // Tile size for hidden dimension in Hybrid SliceK (mTMA). - // tileM if transpose, tileN if not. Computed in validation. - int32_t mHybridTileHidden{0}; - // Tile size for token dimension in Hybrid SliceK (maxBS). - // tileN if transpose, tileM if not. Computed in validation. - int32_t mHybridTileTokens{0}; // The K dimension of GEMM. int mK{16 * 16}; // Traits of the kernel. @@ -399,8 +377,6 @@ struct GemmOptions // Partial results are accumulated afterwards using either GMEM or DSMEM (in CGA) // to exchange the data between CTAs. int mNumSlicesForSplitK{1}; - // Number of slices for slice-K along K dimension. - int mNumSlicesForSliceK{1}; // The depth of the pipeline for A. int mNumStagesA{0}; // The depth of the pipeline for B. @@ -416,10 +392,6 @@ struct GemmOptions int mNumStagesSmemSfA{0}; // The depth of the pipeline for the SF of B in SMEM. int mNumStagesSmemSfB{0}; - // The depth of the TMEM operand staging pipeline (Hybrid SliceK only). - // 0 means follow mNumStagesA (coupled SMEM/TMEM staging). - // Valid values: 1-6 (must be <= mNumStagesA). - int mNumStagesTmemA{0}; // The depth of the pipeline for the SF of A in TMEM. int mNumStagesTmemSfA{0}; // The depth of the pipeline for the SF of B in TMEM. @@ -435,9 +407,6 @@ struct GemmOptions tg::Dtype mPerTokenSfDtype{tg::Dtype::Void}; // Which host task the persistent scheduler is fused with (Self = dedicated scheduler task). SchedHostTask mSchedHostTask{SchedHostTask::Self}; - // Hybrid SliceK: number of K-slice iterations per MMA (128/mTMA for mTMA > 32, else 4). - // Computed in validation from mHybridTileHidden. - int mReRunTimes{1}; // Block size of A, for block-scaled types. int mSfBlockSizeA{-1}; // Block size of B, for block-scaled types. @@ -456,8 +425,6 @@ struct GemmOptions // But it reduces the number of L2 requests under the hood and potentially improves perf. // Applies to layout 8x4 only. int mSfReshapeFactor{1}; - // Slice-K implementation to use TileM dimension for TileK. - bool mSliceK{false}; // Sparsity of A. tg::Sparsity mSparsityA{tg::Sparsity::Dense}; // The location of the exchange for split-K (it's None when split-K is disabled). @@ -669,12 +636,14 @@ inline std::string dumpOptions(GemmOptions const& options, bool dumpRuntimeParam ss << "mDtypeMmaB=" << "trtllm::gen::Dtype(" << static_cast(options.mDtypeMmaB) << ")" << "," << std::endl; + ss << "mDtypeSfC=" + << "trtllm::gen::Dtype(" << static_cast(options.mDtypeSfC) << ")" + << "," << std::endl; ss << "mEltwiseActType=" << "gemm::EltwiseActType(" << static_cast(options.mEltwiseActType) << ")" << "," << std::endl; ss << "mEnablesEarlyExit=" << options.mEnablesEarlyExit << "," << std::endl; ss << "mEnablesDelayedEarlyExit=" << options.mEnablesDelayedEarlyExit << "," << std::endl; - ss << "mEnablesGlobalPtxKnobs=" << options.mEnablesGlobalPtxKnobs << "," << std::endl; ss << "mEpilogueLdtmDps=" << options.mEpilogueLdtmDps << "," << std::endl; ss << "mEpilogueLdtmBits=" << options.mEpilogueLdtmBits << "," << std::endl; ss << "mEpilogueTileM=" << options.mEpilogueTileM << "," << std::endl; @@ -727,7 +696,6 @@ inline std::string dumpOptions(GemmOptions const& options, bool dumpRuntimeParam ss << "mNumRegsPerThreadEpilogueWarp=" << options.mNumRegsPerThreadEpilogueWarp << "," << std::endl; ss << "mNumRegsPerThreadNonEpilogueWarp=" << options.mNumRegsPerThreadNonEpilogueWarp << "," << std::endl; ss << "mNumSlicesForSplitK=" << options.mNumSlicesForSplitK << "," << std::endl; - ss << "mNumSlicesForSliceK=" << options.mNumSlicesForSliceK << "," << std::endl; ss << "mNumStagesA=" << options.mNumStagesA << "," << std::endl; ss << "mNumStagesB=" << options.mNumStagesB << "," << std::endl; ss << "mNumStagesMma=" << options.mNumStagesMma << "," << std::endl; @@ -735,7 +703,6 @@ inline std::string dumpOptions(GemmOptions const& options, bool dumpRuntimeParam ss << "mNumStagesMmaAcrossWorkTile=" << options.mNumStagesMmaAcrossWorkTile << "," << std::endl; ss << "mNumStagesSmemSfA=" << options.mNumStagesSmemSfA << "," << std::endl; ss << "mNumStagesSmemSfB=" << options.mNumStagesSmemSfB << "," << std::endl; - ss << "mNumStagesTmemA=" << options.mNumStagesTmemA << "," << std::endl; ss << "mNumStagesTmemSfA=" << options.mNumStagesTmemSfA << "," << std::endl; ss << "mNumStagesTmemSfB=" << options.mNumStagesTmemSfB << "," << std::endl; ss << "mNumStagesWorkId=" << options.mNumStagesWorkId << "," << std::endl; @@ -760,7 +727,6 @@ inline std::string dumpOptions(GemmOptions const& options, bool dumpRuntimeParam << "trtllm::gen::SfLayout(" << static_cast(options.mSfLayoutC) << ")" << "," << std::endl; ss << "mSfReshapeFactor=" << options.mSfReshapeFactor << "," << std::endl; - ss << "mSliceK=" << options.mSliceK << "," << std::endl; ss << "mSparsityA=" << "trtllm::gen::Sparsity(" << static_cast(options.mSparsityA) << ")" << "," << std::endl; @@ -1198,7 +1164,6 @@ inline bool checkAndUpdateGemmOptions( "The sparsity information for one tile row must be a multiple of 16B. Use larger tileK."); TLLM_CHECK_ERROR(options.mDtypeA == options.mDtypeMmaA, "Sparsity is not supported with on-the-fly upcasting."); TLLM_CHECK_ERROR(!options.mUseDeepSeekFp8, "Sparsity is not supported with DeepSeek Fp8."); - TLLM_CHECK_ERROR(!options.mSliceK, "Sparsity is not supported with slice-k."); } if (options.mMmaKind == tg::MmaKind::Fp8Fp6Fp4) @@ -1249,9 +1214,7 @@ inline bool checkAndUpdateGemmOptions( } // Constraints for NvFp4, MxFp8, and MxFp4. - // Skip this constraint for Hybrid SliceK, which uses lane-masked MMA and supports small tiles. - if (!options.mHybridSliceK - && (options.mMmaKind == tg::MmaKind::MxFp4NvFp4 || options.mMmaKind == tg::MmaKind::MxFp8Fp6Fp4 + if ((options.mMmaKind == tg::MmaKind::MxFp4NvFp4 || options.mMmaKind == tg::MmaKind::MxFp8Fp6Fp4 || options.mDtypeC == tg::Dtype::MxE4m3) && options.mMmaM != 128) { @@ -1476,22 +1439,58 @@ inline bool checkAndUpdateGemmOptions( TLLM_CHECK_ERROR((padMultiplierB * tg::dtypeGetNumBits(options.mDtypeB) * options.mK / 8) % 16 == 0, "K dimension of B must be aligned to 16 bytes."); + // Auto-deduce the dtype of the block scaling factors of C, or validate the user's choice. + if (tg::dtypeIsBlockFmt(options.mDtypeC)) + { + auto const defaultDtypeSfC = tg::dtypeGetBlockSfType(options.mDtypeC); + if (options.mDtypeSfC == tg::Dtype::Void) + { + if (updateOptions) + { + options.mDtypeSfC = defaultDtypeSfC; + } + else + { + return false; + } + } + else + { + // Allow either the default quantized type, or Fp32 (keep the high-precision scaling factors). + TLLM_CHECK_ERROR(options.mDtypeSfC == defaultDtypeSfC || options.mDtypeSfC == tg::Dtype::Fp32 + || options.mDtypeSfC == tg::Dtype::Bfloat16, + "Unsupported dtypeSfC=", tg::dtypeToString(options.mDtypeSfC), + " for dtypeC=", tg::dtypeToString(options.mDtypeC), ". Supported: ", tg::dtypeToString(defaultDtypeSfC), + " (default) or Fp32 or Bfloat16."); + } + } + else if (options.mDtypeSfC != tg::Dtype::Void) + { + TLLM_LOG_WARNING("Got dtypeSfC=", tg::dtypeToString(options.mDtypeSfC), + " but dtypeC=", tg::dtypeToString(options.mDtypeC), " does not use block scales"); + GEMM_UPDATE_OR_ERROR(options.mDtypeSfC, tg::Dtype::Void); + } + if (tg::dtypeIsBlockFmt(options.mDtypeC)) { TLLM_CHECK_ERROR(isBlackwell, "Block scaling is only supported on Blackwell"); - TLLM_CHECK_ERROR(options.mSfLayoutC == tg::SfLayout::R128c4 || options.mSfLayoutC == tg::SfLayout::R8c4, - "Only the 128x4 and 8x4 SF layouts are supported for C."); + TLLM_CHECK_ERROR(options.mSfLayoutC == tg::SfLayout::R128c4 || options.mSfLayoutC == tg::SfLayout::R8c4 + || options.mSfLayoutC == tg::SfLayout::Linear, + "Only the 128x4, 8x4 and linear SF layouts are supported for C."); if (!options.mTransposeMmaOutput) { TLLM_CHECK_ERROR(options.mEpilogueTileN % options.mSfBlockSizeC == 0, "EpilogueTileN must be a multiple of the number of elements per SF for C"); } - int const numSfTileRowsC = options.mSfLayoutC == tg::SfLayout::R128c4 ? 128 : 8; - int const tileTokenDim = options.mTransposeMmaOutput ? options.mTileN : options.mTileM; - TLLM_CHECK_ERROR_FMT(tileTokenDim % numSfTileRowsC == 0, - "Tile%s (%d) must be a multiple of %d for C SF layout %s", options.mTransposeMmaOutput ? "N" : "M", - tileTokenDim, numSfTileRowsC, tg::sfLayoutToString(options.mSfLayoutC).c_str()); + if (options.mSfLayoutC != tg::SfLayout::Linear) + { + int const numSfTileRowsC = options.mSfLayoutC == tg::SfLayout::R128c4 ? 128 : 8; + int const tileTokenDim = options.mTransposeMmaOutput ? options.mTileN : options.mTileM; + TLLM_CHECK_ERROR_FMT(tileTokenDim % numSfTileRowsC == 0, + "Tile%s (%d) must be a multiple of %d for C SF layout %s", options.mTransposeMmaOutput ? "N" : "M", + tileTokenDim, numSfTileRowsC, tg::sfLayoutToString(options.mSfLayoutC).c_str()); + } int numEltsPerSfC = options.mSfBlockSizeC; int const hiddenDim = options.mTransposeMmaOutput ? options.mM : options.mN; @@ -1573,18 +1572,8 @@ inline bool checkAndUpdateGemmOptions( "M/validM must be a multiple of shuffle block size (", shuffleBlockSize, ") when useShuffledMatrix"); } - if (!options.mSliceK && !options.mHybridSliceK) - { - TLLM_CHECK_ERROR(options.mMmaM / (options.mClusterDimX > 1 ? 2 : 1) <= options.mEpilogueTileM, - "EpilogueTileM must be larger or equal than mmaM."); - } - else if (options.mSliceK) - { - // FIXME: this is not necessary limitation. Simply fixing num repeats in TmemSliceKA should be - // enough. - TLLM_CHECK_ERROR( - (options.mTileN & (options.mTileN - 1)) == 0, "For Slice-K TileN is required to be a power of 2"); - } + TLLM_CHECK_ERROR(options.mMmaM / (options.mClusterDimX > 1 ? 2 : 1) <= options.mEpilogueTileM, + "EpilogueTileM must be larger or equal than mmaM."); if (options.mClusterDimX >= 2) { @@ -1718,8 +1707,6 @@ inline bool checkAndUpdateGemmOptions( "EpilogueLdtmDps must be 32 when using shuffled matrix and non-transposed mma output"); TLLM_CHECK_ERROR( options.mUseTmaStore, "TMA store is required when using shuffled matrix and non-transposed mma output"); - TLLM_CHECK_ERROR( - !options.mSliceK, "Slice-K is not supported when using shuffled matrix and non-transposed mma output"); // When doing unshuffle in the epilogue, one fragment of epilogue tile must have at least one // shuffle block. auto minEpilogueTileN = getShuffleBlockSize(options.mEpilogueTileM); @@ -1837,223 +1824,6 @@ inline bool checkAndUpdateGemmOptions( TLLM_CHECK_ERROR( options.mNumEpilogueWarps == 4 || options.mNumEpilogueWarps == 8, "mNumEpilogueWarps has to be either 4 or 8."); - if (options.mSliceK) - { - TLLM_CHECK_ERROR(isBlackwell, "Slice-K is not supported on Hopper"); - - TLLM_CHECK_ERROR(!options.mUseDeepSeekFp8, "DeepSeek Fp8 GEMM is not supported for slice-K"); - TLLM_CHECK_ERROR(options.mUseTwoTmaLoadWarps, "Slice-K requires two warp load for A and B"); - TLLM_CHECK_ERROR(options.mTransposeMmaOutput, "Slice-K requires transpose mma output"); - TLLM_CHECK_ERROR(options.mUseShuffledMatrix, "Slice-K requires shuffled matrix"); - TLLM_CHECK_ERROR(options.mTileK % 128 == 0, "Slice-K requires TileK be a multiple of 128"); - TLLM_CHECK_ERROR(options.mMmaM == 128, "Slice-K requires MmaM == 128"); - TLLM_CHECK_ERROR(options.mTileN == options.mEpilogueTileN, "TileN must be equal to EpilogueTileN for slice-K"); - - TLLM_LOG_WARNING("Overwriting TileM and EpilogueTileM to 32 for slice-K"); - if (options.mTileM != 32 || options.mEpilogueTileM != 32) - { - if (updateOptions) - { - // FIXME: it is possible to remove this restriction. - options.mTileM = 32; - options.mEpilogueTileM = 32; - } - else - { - return false; - } - } - TLLM_CHECK_ERROR(options.mDtypeA == tg::Dtype::E4m3 && options.mDtypeB == tg::Dtype::E4m3, - "Slice-K requires e4m3 input dtype"); - - if (options.mNumSlicesForSliceK != 4) - { - if (updateOptions) - { - options.mNumSlicesForSliceK = 4; - } - else - { - return false; - } - } - TLLM_CHECK_ERROR((options.mTileK / options.mMmaK) % options.mNumSlicesForSliceK == 0, "TileK (", options.mTileK, - ") / MmaK (", options.mMmaK, ") must be a multiple of mNumSlicesForSliceK (", options.mNumSlicesForSliceK, - ")"); - } - - // Hybrid SliceK validation and semantic parameter computation - // Both transpose modes produce final output [tokens, hidden]: - // - transpose=true: m=hidden, n=tokens, GEMM C=[hidden,tokens], then transpose to - // [tokens,hidden] - // - transpose=false: m=tokens, n=hidden, GEMM C=[tokens,hidden], direct output - if (options.mHybridSliceK) - { - TLLM_CHECK_ERROR(isBlackwell, "Hybrid SliceK is only supported on Blackwell"); - TLLM_CHECK_ERROR(!options.mSliceK, "Cannot use both SliceK and HybridSliceK"); - TLLM_CHECK_ERROR(options.mDtypeA == tg::Dtype::E4m3 && options.mDtypeB == tg::Dtype::E4m3, - "Hybrid SliceK phase 1 requires FP8 (E4m3) input dtype"); - TLLM_CHECK_ERROR(options.mTileK % 128 == 0, "Hybrid SliceK requires TileK to be a multiple of 128"); - TLLM_CHECK_ERROR(options.mMmaM == 128, "Hybrid SliceK requires MmaM == 128"); - // TODO: 2-CTA MMA support requires fixing TMA coordinate generation in SmemHybridSliceKAb - // to properly handle loadHalfTileB and OOB optimizations when transpose=false. - TLLM_CHECK_ERROR(options.mClusterDimX == 1, "Hybrid SliceK does not yet support 2-CTA MMA (clusterDimX > 1)"); - // TODO: CGA split-K support requires implementing DSMEM epilogue path. - TLLM_CHECK_ERROR(options.mNumSlicesForSplitK <= 1, - "Hybrid SliceK does not yet support CGA split-K (numSlicesForSplitK > 1)"); - - // Compute semantic parameters based on transpose mode - // mHybridTileHidden = mTMA = tile size for hidden dimension (weight rows) - // mHybridTileTokens = maxBS = tile size for token dimension (activation batch) - if (updateOptions) - { - if (options.mTransposeMmaOutput) - { - // transpose=true: m=hidden, n=tokens - // ctaIdxM → hidden tiles, ctaIdxN → token tiles - options.mHybridNumTokens = options.mN; - options.mHybridHiddenDim = options.mM; - options.mHybridTileTokens = options.mTileN; // maxBS (N-direction = tokens) - options.mHybridTileHidden = options.mTileM; // mTMA (M-direction = hidden) - } - else - { - // transpose=false: m=tokens, n=hidden - // ctaIdxM → token tiles, ctaIdxN → hidden tiles - options.mHybridNumTokens = options.mM; - options.mHybridHiddenDim = options.mN; - options.mHybridTileTokens = options.mTileM; // maxBS (M-direction = tokens) - options.mHybridTileHidden = options.mTileN; // mTMA (N-direction = hidden) - } - } - - // Validate mTMA (hidden tile) range: 4-128 - TLLM_CHECK_ERROR(options.mHybridTileHidden >= 4 && options.mHybridTileHidden <= 128, - "Hybrid SliceK supports mTMA (hidden tile) from 4 to 128, got ", options.mHybridTileHidden); - - // Compute derived parameters based on mHybridTileHidden (mTMA) - if (updateOptions) - { - // Hybrid SliceK pipeline staging defaults (only apply if user didn't override): - // - mNumStagesA/B (SmemStage): defaults to 4, valid range 2-8 - // - mNumStagesTmemA: defaults to follow mNumStagesA, valid values 1-6 - // - MMA accum is not staged (NumStagesMma = 1) - int const defaultNumStages = 2; // GemmOptions default - if (options.mNumStagesA == defaultNumStages) - { - options.mNumStagesA = 4; // Hybrid SliceK default - options.mNumStagesB = 4; // Hybrid SliceK default - } - // mNumStagesTmemA: if not set (0), default to mNumStagesA (coupled mode) - if (options.mNumStagesTmemA == 0) - { - options.mNumStagesTmemA = std::min(options.mNumStagesA, 6); - } - // Validate stage configurations - TLLM_CHECK_ERROR(options.mNumStagesA >= 2 && options.mNumStagesA <= 8, - "Hybrid SliceK requires numStages in range [2, 8], got ", options.mNumStagesA); - TLLM_CHECK_ERROR(options.mNumStagesTmemA <= 6, - "Hybrid SliceK requires numStagesTmemA <= 6, got numStagesTmemA=", options.mNumStagesTmemA); - - // MMA staging: Hybrid SliceK requires single-stage MMA (accumulator not staged) - options.mNumStagesMma = 1; - options.mNumStagesMmaWithinWorkTile = 1; - options.mNumStagesMmaAcrossWorkTile = 1; - - // FundamentalKtileUnit: 32/mTMA for mTMA < 32, else 1 - options.mFundamentalKtileUnit = (options.mHybridTileHidden < 32) ? (32 / options.mHybridTileHidden) : 1; - // ReRunTimes: (128/mTMA) for mTMA > 32, else 4 - options.mReRunTimes = (options.mHybridTileHidden > 32) ? (128 / options.mHybridTileHidden) : 4; - - // Auto-correct tileK to satisfy: (tileK / 128) % FundamentalKtileUnit == 0 - // Minimum valid tileK = 128 * FundamentalKtileUnit - // For mTMA <= 8: FundamentalKtileUnit = 4, minTileK = 512 - // For mTMA <= 16: FundamentalKtileUnit = 2, minTileK = 256 - // For mTMA > 16: FundamentalKtileUnit = 1, minTileK = 128 - int const minTileK = 128 * options.mFundamentalKtileUnit; - int const kTile128BNum = options.mTileK / 128; - if (options.mTileK < minTileK || kTile128BNum % options.mFundamentalKtileUnit != 0) - { - int const newTileK = std::max(minTileK, divUpMul(options.mTileK, minTileK)); - TLLM_LOG_WARNING("Hybrid SliceK: Adjusting TileK from ", options.mTileK, " to ", newTileK, - " (FundamentalKtileUnit=", options.mFundamentalKtileUnit, ", minTileK=", minTileK, ")"); - options.mTileK = newTileK; - // Keep mMmaTileK in sync (it was set to the old mTileK earlier). - options.mMmaTileK = newTileK; - } - } - else - { - // Validate stage configurations for Hybrid SliceK - TLLM_CHECK_ERROR(options.mNumStagesA >= 2 && options.mNumStagesA <= 8, - "Hybrid SliceK requires numStages in range [2, 8], got ", options.mNumStagesA); - TLLM_CHECK_ERROR(options.mNumStagesTmemA > 0 && options.mNumStagesTmemA <= 6, - "Hybrid SliceK requires numStagesTmemA in range [1, 6]"); - TLLM_CHECK_ERROR(options.mNumStagesMma == 1, "Hybrid SliceK requires numStagesMma == 1 (no MMA staging)"); - TLLM_CHECK_ERROR(options.mNumStagesMmaWithinWorkTile == 1 && options.mNumStagesMmaAcrossWorkTile == 1, - "Hybrid SliceK requires numStagesMmaWithinWorkTile == 1 and " - "numStagesMmaAcrossWorkTile == 1"); - // Validate derived parameters using mHybridTileHidden - int const expectedFundamental = (options.mHybridTileHidden < 32) ? (32 / options.mHybridTileHidden) : 1; - int const expectedReRunTimes = (options.mHybridTileHidden > 32) ? (128 / options.mHybridTileHidden) : 4; - TLLM_CHECK_ERROR(options.mFundamentalKtileUnit == expectedFundamental, - "Hybrid SliceK requires FundamentalKtileUnit == ", expectedFundamental, " (got ", - options.mFundamentalKtileUnit, ")"); - TLLM_CHECK_ERROR(options.mReRunTimes == expectedReRunTimes, - "Hybrid SliceK requires ReRunTimes == ", expectedReRunTimes, " (got ", options.mReRunTimes, ")"); - - // Validate tileK constraint: (tileK / 128) % FundamentalKtileUnit == 0 - int const minTileK = 128 * expectedFundamental; - int const kTile128BNum = options.mTileK / 128; - TLLM_CHECK_ERROR(options.mTileK >= minTileK && kTile128BNum % expectedFundamental == 0, - "Hybrid SliceK requires TileK >= ", minTileK, - " and (TileK/128) divisible by FundamentalKtileUnit=", expectedFundamental, - " (got TileK=", options.mTileK, ")"); - } - - // Hybrid SliceK epilogue constraints: - // EpilogueTileM must equal TileM and EpilogueTileN must equal TileN - if (options.mEpilogueTileM != options.mTileM || options.mEpilogueTileN != options.mTileN) - { - TLLM_LOG_WARNING( - "Hybrid SliceK requires EpilogueTileM == TileM and EpilogueTileN == TileN. " - "Overwriting epilogue tiles to match main tiles."); - if (updateOptions) - { - options.mEpilogueTileM = options.mTileM; - options.mEpilogueTileN = options.mTileN; - } - else - { - return false; - } - } - - // Hybrid SliceK LDTM configuration adjustment for small epilogueTileN: - // Default 16dp256b LDTM needs numTmemCols >= 8 (i.e., epilogueTileN >= 8 for Fp32). - // For smaller tiles, switch to 32dp32b which only needs numTmemCols >= 1. - // Note: Hybrid SliceK uses custom epilogue (hybridSliceKEpilogueStore) that bypasses LDTM, - // but GmemC resource creation still triggers TmemLdstInst initialization that would SIGFPE. - int const minEpilogueTileNFor16dp256b = 8; // 16dp256b needs 8 TMEM cols per repeat - if (options.mEpilogueTileN < minEpilogueTileNFor16dp256b) - { - if (updateOptions) - { - // Switch to 32dp32b configuration which works with any epilogueTileN >= 1 - options.mEpilogueLdtmDps = 32; - options.mEpilogueLdtmBits = 32; - } - else - { - // In non-update mode, check if current LDTM config is compatible - if (options.mEpilogueLdtmDps == 16 && options.mEpilogueLdtmBits == 256) - { - return false; // Incompatible configuration - } - } - } - } - // Number of iterations in K dimension after padding. // Note the perCtaK in each CTA in the splitK group are padded to the same number of iterations. // E.g., K = 512, TileK = 128, numSlicesForSplitK = 3. Then the padded K is @@ -2195,15 +1965,10 @@ inline bool checkAndUpdateGemmOptions( "A and B dtype must be E4m3 for Meta Fp8. Found dtypeA=", tg::dtypeToString(options.mDtypeA), " dtypeB=", tg::dtypeToString(options.mDtypeB)); TLLM_CHECK_ERROR(options.mDtypeC == tg::Dtype::Fp32 || options.mDtypeC == tg::Dtype::Bfloat16 - || options.mDtypeC == tg::Dtype::Fp16, - "Only Fp32, Bfloat16, Fp16 output dtypes are supported for Meta Fp8"); - } - else - { - // RoutingScalesOnInput case - TLLM_CHECK_ERROR((options.mUsePerTokenSfA && !options.mTransposeMmaOutput) - || (options.mUsePerTokenSfB && options.mTransposeMmaOutput), - "In RoutingScalesOnInput mode, perToken scales must be used on activations"); + || options.mDtypeC == tg::Dtype::Fp16 || options.mDtypeC == tg::Dtype::E4m3 + || options.mDtypeC == tg::Dtype::E2m1, + "Only Fp32, Bfloat16, Fp16, E4m3, E2m1 output dtypes are supported for Meta " + "Fp8"); } TLLM_CHECK_ERROR(options.mPerTokenSfDtype == tg::Dtype::Fp32 || options.mPerTokenSfDtype == tg::Dtype::Bfloat16 || options.mPerTokenSfDtype == tg::Dtype::Fp16, @@ -2286,7 +2051,8 @@ inline bool checkAndUpdateGemmOptions( if (!isBiasTypeNone(options.mBiasType)) { - TLLM_CHECK_ERROR(!options.mUseDeepSeekFp8, "Bias is not supported for DeepSeek Fp8"); + TLLM_CHECK_ERROR(!options.mUseDeepSeekFp8 || isBiasTypeMn(options.mBiasType), + "Only BiasType::Mn is supported for DeepSeek Fp8"); TLLM_CHECK_ERROR(!(options.mUsePerTokenSfA && options.mUsePerTokenSfB), "Bias is not supported for Meta Fp8"); } @@ -2298,7 +2064,6 @@ inline bool checkAndUpdateGemmOptions( { TLLM_CHECK_ERROR(options.mUseTmaStore, "mUseMaxTmemOverlap only works with TMA store"); TLLM_CHECK_ERROR(options.mNumSlicesForSplitK == 1, "mUseMaxTmemOverlap does not work with splitK"); - TLLM_CHECK_ERROR(options.mNumSlicesForSliceK == 1, "mUseMaxTmemOverlap does not work with sliceK"); TLLM_CHECK_ERROR(!options.mUseDeepSeekFp8, "mUseMaxTmemOverlap does not work with mUseDeepSeekFp8"); } @@ -2307,8 +2072,6 @@ inline bool checkAndUpdateGemmOptions( TLLM_CHECK_ERROR(options.mUseTmaStore, "Using more than 4 warps for epilogue only works with TMA store"); TLLM_CHECK_ERROR( options.mNumSlicesForSplitK == 1, "Using more than 4 warps for epilogue does not work with splitK"); - TLLM_CHECK_ERROR( - options.mNumSlicesForSliceK == 1, "Using more than 4 warps for epilogue does not work with sliceK"); TLLM_CHECK_ERROR( !options.mUseDeepSeekFp8, "Using more than 4 warps for epilogue does not work with mUseDeepSeekFp8"); @@ -2320,19 +2083,17 @@ inline bool checkAndUpdateGemmOptions( if (updateOptions) { // Init kernel traits. - // For Hybrid SliceK, swap tileM/tileN so KernelTraits sees LoadA=Weight(tileN) and - // LoadB=Activation(tileM). - int32_t const tileMForTraits = options.mHybridSliceK ? options.mTileN : options.mTileM; - int32_t const tileNForTraits = options.mHybridSliceK ? options.mTileM : options.mTileN; + int32_t const tileMForTraits = options.mTileM; + int32_t const tileNForTraits = options.mTileN; options.mKernelTraits = KernelTraits(options.mDtypeA, options.mDtypeB, options.mDtypeC, options.mDtypeAcc, options.mDtypeMmaA, options.mDtypeMmaB, options.mMmaKind, options.mSparsityA, options.mMmaK, tileMForTraits, tileNForTraits, options.mTileK, options.mMmaTileK, options.mEpilogueTileM, options.mEpilogueTileN, options.mSfBlockSizeA, options.mSfBlockSizeB, options.mNumStagesA, options.mNumStagesB, options.mNumStagesMma, options.mNumStagesTmemSfA, options.mNumStagesTmemSfB, options.mNumSlicesForSplitK, - options.mNumSlicesForSliceK, options.mSplitK, options.mUseTmaStore, options.mTransposeMmaOutput, - options.mAllReduceAlgo, options.mFuseUtccpWithUtcmma, options.mUseMaxTmemOverlap, - options.mUseCustomizedMma3xNvFp4, options.mNumEpilogueWarps, isPersistentScheduler(options.mTileScheduler), - options.mUseDeepSeekFp8, options.mUsePerTokenSfA, options.mUsePerTokenSfB, + options.mSplitK, options.mUseTmaStore, options.mTransposeMmaOutput, options.mAllReduceAlgo, + options.mFuseUtccpWithUtcmma, options.mUseMaxTmemOverlap, options.mUseCustomizedMma3xNvFp4, + options.mNumEpilogueWarps, isPersistentScheduler(options.mTileScheduler), options.mUseDeepSeekFp8, + options.mUsePerTokenSfA, options.mUsePerTokenSfB, /* useTwoCtas*/ options.mClusterDimX >= 2, options.mBiasType, options.mFusedBiasShuffleMode); } @@ -2387,8 +2148,8 @@ inline CUresult loadCubinData(CUmodule* module, Config const& config) // Trtllm links the cubin into the executable while Flashinfer loads the cubin from storage. #ifdef TLLM_GEN_EXPORT_FLASHINFER #ifdef TLLM_GEN_GEMM_CUBIN_PATH - static const std::string tllm_gen_gemm_cubin_path = std::string(TLLM_GEN_GEMM_CUBIN_PATH); - const std::string sha256 = config.mHash ? config.mHash : ""; + static std::string const tllm_gen_gemm_cubin_path = std::string(TLLM_GEN_GEMM_CUBIN_PATH); + std::string const sha256 = config.mHash ? config.mHash : ""; std::string fileName = config.mFunctionName; if (!fileName.empty()) { diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/KernelMetaInfo.h b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/KernelMetaInfo.h index 6095adc6b031..49adcadb7273 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/KernelMetaInfo.h +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/KernelMetaInfo.h @@ -19,16 +19,13 @@ #include "BatchedGemmOptions.h" #include "tensorrt_llm/common/config.h" -namespace batchedGemm -{ +namespace batchedGemm { TRTLLM_NAMESPACE_BEGIN - -namespace kernels -{ +namespace kernels { // clang-format off -#define TLLM_GEN_COMMIT "71d2730e" +#define TLLM_GEN_COMMIT "a2ad0544-dirty" #define TLLM_GEN_EXPORT_VERSION "7.0.4.0.4.0" #ifndef EXCLUDE_SM_100 @@ -543,49 +540,93 @@ extern unsigned char const Bmm_Fp16_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x3 extern unsigned char const Bmm_Fp16_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f_cubin[]; extern unsigned char const Bmm_Fp16_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f_cubin[]; extern unsigned char const Bmm_Fp16_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f_cubin[]; +extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f_cubin[]; extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f_cubin[]; extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7x7x7x7x1x7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_lsfbW4_dynB_sm100f_cubin[]; +extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7x7x7x7x1x7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_siTuGlu_lbW4_lsfbW4_dynB_sm100f_cubin[]; +extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128u2_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f_cubin[]; extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128u2_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f_cubin[]; +extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f_cubin[]; extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f_cubin[]; extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4x4x4x4x1x4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_lsfbW4_dynB_sm100f_cubin[]; +extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4x4x4x4x1x4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_siTuGlu_lbW4_lsfbW4_dynB_sm100f_cubin[]; +extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256u2_s4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f_cubin[]; extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256u2_s4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f_cubin[]; +extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin[]; extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin[]; +extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin[]; extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin[]; +extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin[]; extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin[]; +extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin[]; extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin[]; +extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin[]; extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin[]; +extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin[]; extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin[]; +extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin[]; extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin[]; +extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin[]; extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin[]; extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x256x128_s6x6x6x6x1x6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f_cubin[]; +extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x256x128_s6x6x6x6x1x6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_fCp_tmOv_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f_cubin[]; extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x256x256_s3x3x3x3x1x3_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f_cubin[]; +extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x256x256_s3x3x3x3x1x3_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_fCp_tmOv_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f_cubin[]; +extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin[]; extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin[]; +extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin[]; extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin[]; +extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin[]; extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin[]; +extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin[]; extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin[]; +extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin[]; extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin[]; +extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin[]; extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin[]; +extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin[]; extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin[]; +extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin[]; extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin[]; +extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x128_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW1_lsfbW1_dynB_sm100f_cubin[]; extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x128_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW1_lsfbW1_dynB_sm100f_cubin[]; +extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x128u2_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW1_lsfbW1_dynB_sm100f_cubin[]; extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x128u2_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW1_lsfbW1_dynB_sm100f_cubin[]; +extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x256_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW1_lsfbW1_dynB_sm100f_cubin[]; extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x256_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW1_lsfbW1_dynB_sm100f_cubin[]; +extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x256u2_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW1_lsfbW1_dynB_sm100f_cubin[]; extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x256u2_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW1_lsfbW1_dynB_sm100f_cubin[]; +extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin[]; extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin[]; +extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin[]; extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin[]; +extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin[]; extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin[]; +extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin[]; extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin[]; +extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin[]; extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin[]; +extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin[]; extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin[]; +extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin[]; extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin[]; +extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin[]; extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin[]; +extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin[]; extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin[]; +extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin[]; extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin[]; +extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin[]; extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin[]; +extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin[]; extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin[]; +extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin[]; extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin[]; +extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin[]; extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin[]; +extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin[]; extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin[]; +extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin[]; extern unsigned char const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin[]; extern unsigned char const Bmm_MxE4m3_MxE4m3MxE2m1_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7_et128x64_m256x128x32_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_swiGlu_laW8_lsfaW4_dynB_sm100f_cubin[]; extern unsigned char const Bmm_MxE4m3_MxE4m3MxE2m1_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4_et128x64_m256x128x32_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_swiGlu_laW8_lsfaW4_dynB_sm100f_cubin[]; @@ -1145,49 +1186,93 @@ extern unsigned int const Bmm_Fp16_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32 extern unsigned int const Bmm_Fp16_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f_cubin_len; extern unsigned int const Bmm_Fp16_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f_cubin_len; extern unsigned int const Bmm_Fp16_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f_cubin_len; +extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f_cubin_len; extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f_cubin_len; extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7x7x7x7x1x7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_lsfbW4_dynB_sm100f_cubin_len; +extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7x7x7x7x1x7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_siTuGlu_lbW4_lsfbW4_dynB_sm100f_cubin_len; +extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128u2_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f_cubin_len; extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128u2_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f_cubin_len; +extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f_cubin_len; extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f_cubin_len; extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4x4x4x4x1x4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_lsfbW4_dynB_sm100f_cubin_len; +extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4x4x4x4x1x4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_siTuGlu_lbW4_lsfbW4_dynB_sm100f_cubin_len; +extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256u2_s4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f_cubin_len; extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256u2_s4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f_cubin_len; +extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len; extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len; +extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len; extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len; +extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len; extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len; +extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len; extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len; +extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len; extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len; +extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len; extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len; +extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len; extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len; +extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len; extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len; extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x256x128_s6x6x6x6x1x6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f_cubin_len; +extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x256x128_s6x6x6x6x1x6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_fCp_tmOv_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f_cubin_len; extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x256x256_s3x3x3x3x1x3_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f_cubin_len; +extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x256x256_s3x3x3x3x1x3_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_fCp_tmOv_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f_cubin_len; +extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len; extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len; +extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len; extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len; +extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len; extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len; +extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len; extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len; +extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len; extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len; +extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len; extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len; +extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len; extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len; +extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len; extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len; +extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x128_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW1_lsfbW1_dynB_sm100f_cubin_len; extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x128_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW1_lsfbW1_dynB_sm100f_cubin_len; +extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x128u2_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW1_lsfbW1_dynB_sm100f_cubin_len; extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x128u2_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW1_lsfbW1_dynB_sm100f_cubin_len; +extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x256_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW1_lsfbW1_dynB_sm100f_cubin_len; extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x256_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW1_lsfbW1_dynB_sm100f_cubin_len; +extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x256u2_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW1_lsfbW1_dynB_sm100f_cubin_len; extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x256u2_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW1_lsfbW1_dynB_sm100f_cubin_len; +extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len; extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len; +extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len; extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len; +extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len; extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len; +extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len; extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len; +extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len; extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len; +extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len; extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len; +extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len; extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len; +extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len; extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len; +extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len; extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len; +extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len; extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len; +extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len; extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len; +extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len; extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len; +extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len; extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len; +extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len; extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len; +extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len; extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len; +extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len; extern unsigned int const Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len; extern unsigned int const Bmm_MxE4m3_MxE4m3MxE2m1_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7_et128x64_m256x128x32_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_swiGlu_laW8_lsfaW4_dynB_sm100f_cubin_len; extern unsigned int const Bmm_MxE4m3_MxE4m3MxE2m1_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4_et128x64_m256x128x32_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_swiGlu_laW8_lsfaW4_dynB_sm100f_cubin_len; @@ -1238,7 +1323,7 @@ extern unsigned int const Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_e static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { #ifndef EXCLUDE_SM_100 -{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin_len, 116304, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a", 512, "3554cae876b9ef7995bb0651364b082d25d44de5a3da73560c96b2dbbcb7d1f5", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin_len, 116368, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a", 512, "965a6594cf88ccc3cdf8b436af029beedb2bb95e9e49fb8950cf591e679e9f18", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -1253,10 +1338,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -1293,7 +1378,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -1301,7 +1385,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -1316,7 +1399,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -1363,10 +1445,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100a}, -{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin_len, 116064, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a", 384, "c056441760848519efff34ba4a45e428eaa090bec58c7b883cd9ccd17f8b2761", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin_len, 116240, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a", 384, "744b3bdebf9b75154d3423ff5aa1a6ce819096a208badeb9c00641be98fe47cb", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -1381,10 +1464,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -1421,7 +1504,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -1429,7 +1511,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -1444,7 +1525,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -1491,10 +1571,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100a}, -{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin_len, 116304, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a", 512, "7e02e06c082378013568ccfa7ca686353f2b5a2f0734bfd8614f7cdb68adf0f8", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin_len, 116368, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a", 512, "1ea37fff7d955412fe2a7fea1144eb9e65e1c47f34c2ea4c344cd059524a265c", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -1509,10 +1590,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -1549,7 +1630,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -1557,7 +1637,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -1572,7 +1651,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -1619,10 +1697,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100a}, -{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin_len, 116064, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a", 384, "94094faf41ae6c3189d5d03da658036f7340b3b24ed6d7228e2663e2a94a7874", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin_len, 116240, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a", 384, "20a899eef3b55e5b7a159bbda0d59bd75b23d9ca7394eda138bb4a11f269d991", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -1637,10 +1716,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -1677,7 +1756,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -1685,7 +1763,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -1700,7 +1777,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -1747,10 +1823,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100a}, -{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin_len, 140880, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a", 512, "5135c6fce7525aa6b659b2ce747c42cbc5d7b8f06c49978701ae407f346fdb99", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin_len, 140944, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a", 512, "a0a6a9c48d1f6fa3fe4908f13a1fc0a36f92ca56ed5528f73df4da2059cd3d19", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -1765,10 +1842,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -1805,7 +1882,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -1813,7 +1889,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -1828,7 +1903,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -1875,10 +1949,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100a}, -{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin_len, 140640, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a", 384, "185aac7aabc4fef9b70041756edb7d74affca5f6d0c542db73610430f6bbed9d", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin_len, 140816, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a", 384, "e33c28a26a6224ccccf7df9d87c49e7709e649fbbe75c2cd70d82868fac0a18b", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -1893,10 +1968,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -1933,7 +2008,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -1941,7 +2015,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -1956,7 +2029,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -2003,10 +2075,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100a}, -{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin_len, 140880, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a", 512, "2580f3ae913efb3da341d9a4ad979d38cb239cb7d2ae762955540036e3eef0dd", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin_len, 140944, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a", 512, "96cf4dc4578b1ac8a9f16fc27d2cbdbe3b2692e832e3fcac3082b352a1d7084c", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -2021,10 +2094,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -2061,7 +2134,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -2069,7 +2141,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -2084,7 +2155,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -2131,10 +2201,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100a}, -{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin_len, 140640, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a", 384, "9cb667fc635211ae2ba1aebcff54d2a6b857b0e656196e67b747a55d34336005", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin_len, 140816, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a", 384, "8b2c5af9de42887bca74bff8436c53c5b85a3db87287ffce1a2f8572ca44af1f", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -2149,10 +2220,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -2189,7 +2260,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -2197,7 +2267,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -2212,7 +2281,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -2259,10 +2327,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100a}, -{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin_len, 157200, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a", 512, "06a578468d5b75a312fb98e819f56504b1bd51b95d55412000783cc8e9b845e4", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin_len, 157328, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a", 512, "fdab85caecfe0b99431e8dd648fe07f330f838e0514211ca0f2726d0ac0e9025", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -2277,10 +2346,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -2317,7 +2386,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 2 @@ -2325,7 +2393,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -2340,7 +2407,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -2387,10 +2453,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100a}, -{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin_len, 156960, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a", 384, "b61e5ed5e0ce9ec2fc009a5e216acf9083ba25558f9e2a8f82783ed2d6120cc5", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin_len, 157200, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a", 384, "7c413e8cc59ad3b0356c7e8c55ae5452efa0a17cdeb38ef712eb6b2618d707ac", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -2405,10 +2472,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -2445,7 +2512,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 1 @@ -2453,7 +2519,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -2468,7 +2533,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -2515,10 +2579,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100a}, -{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin_len, 157200, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a", 512, "58c7b8816a279d87cff40f49ce2fdab09b4cb16b95c3fe1fab81a90f0639a7d4", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin_len, 157328, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a", 512, "71e69d7b6228ce0a0389f45043380604934a0a6cb4be25a197eb2d3f5859e06a", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -2533,10 +2598,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -2573,7 +2638,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 2 @@ -2581,7 +2645,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -2596,7 +2659,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -2643,10 +2705,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100a}, -{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin_len, 156960, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a", 384, "1433c750db9882885abe4afdc7d259fa381c6ba15b6a98545dcc7c176568dd76", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin_len, 157200, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a", 384, "2bd9d50ccdade28e02be5827cde712400f5bab8eb377f23c0323184a1311eae1", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -2661,10 +2724,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -2701,7 +2764,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 1 @@ -2709,7 +2771,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -2724,7 +2785,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -2771,10 +2831,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100a}, -{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin_len, 104016, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a", 512, "5cab551ce866382872ed928f6105a454bb68e7d4b79f2f7bbfed22d85aea5b41", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin_len, 104080, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a", 512, "caa5b56acf9deb07a978e0b0e88bdc4e986f1447b843a9b0b873b51d0fa22c26", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -2789,10 +2850,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -2829,7 +2890,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -2837,7 +2897,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -2852,7 +2911,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -2899,10 +2957,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100a}, -{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin_len, 103776, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a", 384, "3ffb125293e2a61d261e3e346f3144e0ae3415055295a62e64374499ad5831d9", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin_len, 103952, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a", 384, "32bbacd1b75e8498c14c76f7d4e3a826e8aaf0778d10925f8bc0e0172f85b82e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -2917,10 +2976,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -2957,7 +3016,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -2965,7 +3023,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -2980,7 +3037,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -3027,10 +3083,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100a}, -{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin_len, 104016, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a", 512, "00dfb4d4f098231f275793bbcab18ba95046cab8ef8ec2107e2bd5fb9065f4a6", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin_len, 104080, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a", 512, "f84c3fcba00a7b233fea74a74d7654b000ddb5def3a6d0a2d99b1f6be5dea766", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -3045,10 +3102,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -3085,7 +3142,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -3093,7 +3149,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -3108,7 +3163,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -3155,10 +3209,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100a}, -{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin_len, 103776, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a", 384, "eccf563470a8652fb0c9ca5620303a18e876e0d689943ccc2fd021a5d00cd771", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin_len, 103952, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a", 384, "ad59d64951352b17982d2c8fc526a934c6be327bba3fa7c83aeed6ecd3da9ae7", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -3173,10 +3228,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -3213,7 +3268,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -3221,7 +3275,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -3236,7 +3289,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -3283,10 +3335,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100a}, -{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin_len, 123344, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a", 512, "adb628d8b7744068f7bd5cff7e5e7d2699008f4d67c0533c41fc811d0a56570a", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin_len, 123536, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a", 512, "ff03a360a0d7a7dad7a81307ebe809879f3ae0fc7c0d0b3e4f4761594c845d98", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -3301,10 +3354,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -3341,7 +3394,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 2 @@ -3349,7 +3401,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -3364,7 +3415,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -3411,10 +3461,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100a}, -{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin_len, 123104, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a", 384, "21ef5268be1180adcc5428ac66d62e21bce1ad95f30b92311088d485c72ea282", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin_len, 123152, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a", 384, "6b429ecb0da0ca2616e024d5163401298272ee935d9f3f2a279212196b83d79d", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -3429,10 +3480,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -3469,7 +3520,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 1 @@ -3477,7 +3527,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -3492,7 +3541,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -3539,10 +3587,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100a}, -{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin_len, 123344, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a", 512, "c863e57b9c6fd7a4626524a920b5dae9c4151027a384d7a3356f81f4de8b3a38", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin_len, 123536, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a", 512, "116243ee6ae2bc3b7c66f71c36b70bbe87d8f2698ec3366faf00a67f8f5e9256", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -3557,10 +3606,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -3597,7 +3646,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 2 @@ -3605,7 +3653,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -3620,7 +3667,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -3667,10 +3713,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100a}, -{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin_len, 123104, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a", 384, "ed75b64c16501e065f2b1e8a281a1167b38e17cfbcb85fc9d94fa59745b1013b", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a_cubin_len, 123152, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a", 384, "efb3849f33afe9af7a03eecf149becf892cdd0ac3c82137ea72d946c1d99b029", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -3685,10 +3732,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -3725,7 +3772,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 1 @@ -3733,7 +3779,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -3748,7 +3793,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -3795,10 +3839,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100a}, -{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin_len, 114256, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a", 512, "590b4099a67713f1c56cbc35911dd3f99c34f746bb0b6ec43a4fb4ff218c90fa", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin_len, 114320, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a", 512, "cd84bfe0a7d03fa0ccf4db6c60a06cda92813ae8e253672487d0f1da87aed5e6", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -3813,10 +3858,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -3853,7 +3898,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -3861,7 +3905,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -3876,7 +3919,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -3923,10 +3965,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100a}, -{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin_len, 114016, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a", 512, "2b685c4c692ae69e94326c26e3e0877480138df7d62bc84e0553a76d08eac29b", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin_len, 114192, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a", 512, "9c4ed0b86d09d5b8c35793477324651c3023f0b634c23373b1d2e804b1811ac5", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -3941,10 +3984,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -3981,7 +4024,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -3989,7 +4031,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -4004,7 +4045,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -4051,10 +4091,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100a}, -{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin_len, 114256, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a", 512, "7ccd07dd8c4e4d63ec4c071d83f3d7b22ffb66035fd79273a671d4c84cd0056a", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin_len, 114320, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a", 512, "3c9a421b78042cf35a6f5fb515e95c20d4947f111e9d21585213a2222372fb9a", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -4069,10 +4110,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -4109,7 +4150,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -4117,7 +4157,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -4132,7 +4171,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -4179,10 +4217,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100a}, -{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin_len, 114016, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a", 512, "14e5778b79e494450ad3a3c3210d0211c8ef578dae2ba2405aa37c9036df50f8", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin_len, 114192, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a", 512, "229b2b98791c60c27647748ccc5b81cf0efbe969dff9dbf44f4ad14830004693", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -4197,10 +4236,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -4237,7 +4276,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -4245,7 +4283,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -4260,7 +4297,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -4307,10 +4343,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100a}, -{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin_len, 136784, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a", 512, "134ced5e1c6816987705de3c982bec13e746c9e03016095ac2755f7c80528057", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin_len, 136848, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a", 512, "e35c45da758ccc295101ae3b81e3798f1f241b4e10fd188de1b2cc8b6c343004", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -4325,10 +4362,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -4365,7 +4402,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -4373,7 +4409,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -4388,7 +4423,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -4435,10 +4469,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100a}, -{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin_len, 136544, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a", 512, "9905ccf7bf5f1914b0729bf460e672e202bae4e9d7cd0721e2c882587dcac0bb", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin_len, 136720, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a", 512, "075f09d0e0adb46bbd9a2a330a22f914ae893777181e5c348e198e7d14aebffb", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -4453,10 +4488,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -4493,7 +4528,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -4501,7 +4535,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -4516,7 +4549,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -4563,10 +4595,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100a}, -{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin_len, 136784, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a", 512, "956dfe6f29ff35d3aafd9d6200ca909959588e0cd57c518260f825b8b71cd68a", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin_len, 136848, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a", 512, "8da4142dda1c730c25048a5adb85dda7862fa32f5b81c3d75c09a313661bfa1a", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -4581,10 +4614,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -4621,7 +4654,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -4629,7 +4661,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -4644,7 +4675,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -4691,10 +4721,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100a}, -{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin_len, 136544, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a", 512, "dfebc486ed0c59cd939c7b1f35260db70e90913052d60d0c16479a842cbbfc82", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin_len, 136720, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a", 512, "1f10c947c51c8c1ff0685f2a69492e9f3c8f5326bce838c317e5a346aac44775", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -4709,10 +4740,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -4749,7 +4780,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -4757,7 +4787,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -4772,7 +4801,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -4819,10 +4847,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100a}, -{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin_len, 149008, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a", 512, "0584ca763d8669661d23d079c430190ecb638df764a9eb5a7812e7c6c093959d", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin_len, 149136, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a", 512, "e4ad20bc661353d925eed9fbcf2c6cb162a303646f37a93830e22bb12ca2e68d", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -4837,10 +4866,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -4877,7 +4906,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 2 @@ -4885,7 +4913,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -4900,7 +4927,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -4947,10 +4973,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100a}, -{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin_len, 148768, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a", 512, "7022b9859131709b0313a7a68ea527e9ab840e3a1c507277e6147fff3b90195a", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin_len, 149008, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a", 512, "b044e59c128a0379b26a1e2da332d41e523aad3a37b27ff72ed513db18026f5c", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -4965,10 +4992,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -5005,7 +5032,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 1 @@ -5013,7 +5039,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -5028,7 +5053,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -5075,10 +5099,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100a}, -{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin_len, 149008, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a", 512, "8d66c9acdf280690c6e72250a1d0ebc9ac4fc1b66c037a7f7c1ab08474d15644", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin_len, 149136, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a", 512, "c15033f19958a9505d57d7fe1315dbb64d355830f6103df4a4924ef4600ea50b", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -5093,10 +5118,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -5133,7 +5158,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 2 @@ -5141,7 +5165,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -5156,7 +5179,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -5203,10 +5225,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100a}, -{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin_len, 148768, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a", 512, "8d04ff31bf8c6d82eb4133d1a9cbbd49391ecdf14812ea4423df02b5a275a130", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin_len, 149008, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a", 512, "80183b38838e754cac2a2b2413cae34e1c556512d2a2fe9cf07b37743d3e4516", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -5221,10 +5244,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -5261,7 +5284,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 1 @@ -5269,7 +5291,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -5284,7 +5305,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -5331,10 +5351,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100a}, -{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin_len, 102992, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a", 512, "e110c2ec096005fac84d12499fbcbb4698fb84d51b7a94d1241e345e8cf15215", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin_len, 103056, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a", 512, "ee31f269ef5f142e18c8080068882dbf4565c4fabf4223da369b1ec236bbd593", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -5349,10 +5370,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -5389,7 +5410,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -5397,7 +5417,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -5412,7 +5431,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -5459,10 +5477,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100a}, -{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin_len, 102752, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a", 512, "06a26011aa0ec1280601819f5b7f307740740d05d1e436a781bc384235dc7855", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin_len, 102928, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a", 512, "c68a34b135d156f1a0da6833f9a6f9697da87a1feead83f10902faa9a882980d", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -5477,10 +5496,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -5517,7 +5536,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -5525,7 +5543,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -5540,7 +5557,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -5587,10 +5603,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100a}, -{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin_len, 102992, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a", 512, "4039902886ad5a450b3cdd7a42609d44d79d30bcd8c4d2cb44f245769cd68aa3", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin_len, 103056, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a", 512, "ef26d6c75598e85bca1f98b0c8dfe1631b175333e3d665adab59198fcfa102b2", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -5605,10 +5622,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -5645,7 +5662,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -5653,7 +5669,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -5668,7 +5683,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -5715,10 +5729,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100a}, -{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin_len, 102752, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a", 512, "5be9b6f45e6b8389072975a694ee8958a107467764160c705f2b3410cadb8dad", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin_len, 102928, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a", 512, "c60ebb86795820f14f7915e83a865a9e4ab20b62d7fcdbdeb0a44fbe8a0984dd", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -5733,10 +5748,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -5773,7 +5788,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -5781,7 +5795,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -5796,7 +5809,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -5843,10 +5855,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100a}, -{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin_len, 122320, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a", 512, "c1fb49e3daf868ba5c7192be2d9ec49e1aacacd1db742ac3dc7ba0223c98db02", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin_len, 122512, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a", 512, "2e70ffe6e5f5013b1101409cd4740822ccf5d71cecd228fc7641810df2624f42", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -5861,10 +5874,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -5901,7 +5914,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 2 @@ -5909,7 +5921,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -5924,7 +5935,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -5971,10 +5981,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100a}, -{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin_len, 122080, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a", 512, "be4a9c7835489038d007f99beb5d61613a6b41c0e42fdaa07ab7baba84ef8e7b", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin_len, 122128, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a", 512, "b9dc80efc5787e83e18dd660ee5fe3e0a22270cd5b4e3f60d815b3d534c959bc", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -5989,10 +6000,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -6029,7 +6040,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 1 @@ -6037,7 +6047,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -6052,7 +6061,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -6099,10 +6107,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100a}, -{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin_len, 122320, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a", 512, "40ecf69b70d717268cbdaa0877abc185b6f7e074d55b41e9162ac0e8fbbf4518", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin_len, 122512, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a", 512, "4dcc53a09ad5dd1a23579c9041ea50369285530abf786d3bbd99bd3d8f1b417a", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -6117,10 +6126,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -6157,7 +6166,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 2 @@ -6165,7 +6173,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -6180,7 +6187,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -6227,10 +6233,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100a}, -{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin_len, 122080, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a", 512, "9e8033201e9d7205837268dc43a01aab41688c7cc1dd8ad38cdd9b4ba897a1e3", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a_cubin_len, 122128, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a", 512, "4851b4d886236f65e5973797a30356c447865358d19dd56c9e8f823414759c2f", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -6245,10 +6252,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -6285,7 +6292,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 1 @@ -6293,7 +6299,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -6308,7 +6313,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -6355,12 +6359,13 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100a}, #endif // EXCLUDE_SM_100 #ifndef EXCLUDE_SM_100F -{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x128x256_s6_et128x128_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x128x256_s6_et128x128_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 211704, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x128x256_s6_et128x128_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "b07a7b1b3c3215b3cddbe722db5cc5cbf9ebee2d25a307fcad58b147e4e49aa3", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x128x256_s6_et128x128_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x128x256_s6_et128x128_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 211864, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x128x256_s6_et128x128_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "a937714d994d7eb7f2a67f608c7a5f61b84727bf6f02e0905bfaf94710a598e2", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -6375,10 +6380,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -6415,7 +6420,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -6423,7 +6427,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -6438,7 +6441,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -6485,10 +6487,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x128x256_s6_et128x128_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x128x256_s6_et128x128_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_rgTma_clmp_dynB_sm100f_cubin_len, 209656, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x128x256_s6_et128x128_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_rgTma_clmp_dynB_sm100f", 384, "b3b96f9c605c7de2c5692a8fcfb435ff8e081e0cbb5f9d73348311c31b04c176", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x128x256_s6_et128x128_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x128x256_s6_et128x128_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_rgTma_clmp_dynB_sm100f_cubin_len, 209816, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x128x256_s6_et128x128_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_rgTma_clmp_dynB_sm100f", 384, "a1234531cbc2d51b88d3d8b9ed05f9c5e48d443d71110b7035613da56cbc02b0", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(2) , /* mBlockK */ -1 @@ -6503,10 +6506,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 32 , /* mEpilogueLdtmBits */ 32 , /* mEpilogueTileM */ 128 @@ -6543,7 +6546,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -6551,7 +6553,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -6566,7 +6567,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -6613,10 +6613,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x128x256u2_s6_et128x128_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x128x256u2_s6_et128x128_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 211704, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x128x256u2_s6_et128x128_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "727fbfac29ee56da9bd58e55290472e5e9cacf24c83017ee0bff6bc39e59b7f7", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x128x256u2_s6_et128x128_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x128x256u2_s6_et128x128_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 211864, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x128x256u2_s6_et128x128_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "3269239b9f8ef17955867ae1ed5da6a342e8389a3119ae78488a091c5219c7cd", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -6631,10 +6632,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -6671,7 +6672,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -6679,7 +6679,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -6694,7 +6693,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -6741,10 +6739,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 196720, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "426b07a3e3c28b0bdacb566776caf04b9fcddf8b5a7529132f127fbb5a229e1b", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 196752, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "9e527f6dd9cddf757bc0fb387ad0e369aba6be0b17a961c1cde9f7161ce8185b", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -6759,10 +6758,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -6799,7 +6798,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 2 @@ -6807,7 +6805,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -6822,7 +6819,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -6869,10 +6865,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 196480, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "376d188901a6db713bae362bf0f21915e0ccc466ec5b304bf33c4b733f52a3c8", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 196624, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "40dc42950c24cefdc74415ac415e07385f77d029e3f968a0b29b7eb6bb2b1dbe", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -6887,10 +6884,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -6927,7 +6924,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 1 @@ -6935,7 +6931,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -6950,7 +6945,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -6997,10 +6991,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 196720, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "68b66e7aa018f89f0485d63e1628915b31f7deebe4072c4a6d56f0aaec0d2cd1", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 196752, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "6aa49c46ae1d785cbacbcebd799286866e3b1f71128bd4e6b5d9c5ffb8b0d3c7", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -7015,10 +7010,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -7055,7 +7050,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 2 @@ -7063,7 +7057,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -7078,7 +7071,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -7125,10 +7117,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 196480, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "1dafee29eea0375db16a8eb236b50508fe02dafe94877e3c76bf1ac34d949778", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 196624, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "97747defd698fe5c67c9a1508e88180ffa51bba2eab166ec1f924034bd7c401e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -7143,10 +7136,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -7183,7 +7176,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 1 @@ -7191,7 +7183,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -7206,7 +7197,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -7253,10 +7243,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 206584, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "432c6432a1b3585e683606fb7297dc74e62784b17b00d0e0e2fd67ee99cf7a6f", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 206744, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "c46b942dc80dad27e0e99b610b07162b203adf5ccfb34f0db3177787ea0da8a6", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -7271,10 +7262,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -7311,7 +7302,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -7319,7 +7309,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -7334,7 +7323,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -7381,10 +7369,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 206344, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "02e206527351e84b97d1e004af3a7688728310582e2b648d8e280bb27dd5c272", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 206360, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "fc528fa511d3e00ebdbafa822540a0e6e0f2ba138b5d54deedbbe43b1269bc55", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -7399,10 +7388,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -7439,7 +7428,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -7447,7 +7435,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -7462,7 +7449,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -7509,10 +7495,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 206584, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "1e73c9b77c05473af76fec15e95ace6ce85aeae6a7f336ccfcfd74e39dc10da4", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 206744, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "3d93107643439f1533a0bbb6cefbafd9ff85b2d4de12a9f7dca922aa64a056cc", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -7527,10 +7514,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -7567,7 +7554,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -7575,7 +7561,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -7590,7 +7575,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -7637,10 +7621,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 206344, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "821cc71bf9d8740ede63d9ff13c7947a5d3c38baa197e83d92fa4d2594e6bab2", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 206360, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "2856549f44988ddad33810ba27c713a26036fef5dab5362f2d00494220fe1832", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -7655,10 +7640,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -7695,7 +7680,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -7703,7 +7687,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -7718,7 +7701,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -7765,10 +7747,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x256x256_s5x5x5x5x1x1_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x256x256_s5x5x5x5x1x1_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_rgTma_clmp_dynB_sm100f_cubin_len, 219736, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x256x256_s5x5x5x5x1x1_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_rgTma_clmp_dynB_sm100f", 384, "bc4051ad67e9c05c3c634b4b12787050ec87a0eb780c06fcf73c3582b925f1d9", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x256x256_s5x5x5x5x1x1_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x256x256_s5x5x5x5x1x1_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_rgTma_clmp_dynB_sm100f_cubin_len, 219800, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x256x256_s5x5x5x5x1x1_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_rgTma_clmp_dynB_sm100f", 384, "b19c23c1da645d6db5146d84d685f60c6660eef6241449ad8f8a55167bf6e41b", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -7783,10 +7766,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -7823,7 +7806,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -7831,7 +7813,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 1 , /* mNumStagesTmemSfB */ 1 , /* mNumStagesWorkId */ 3 @@ -7846,7 +7827,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -7893,10 +7873,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x256x256_s5x5x5x5x1x1_et128x64_m256x256x64_c2x1x1_rN_TN_schPdx3_biasFp32N_fCp_tmOv_bM_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x256x256_s5x5x5x5x1x1_et128x64_m256x256x64_c2x1x1_rN_TN_schPdx3_biasFp32N_fCp_tmOv_bM_rgTma_clmp_dynB_sm100f_cubin_len, 215640, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x256x256_s5x5x5x5x1x1_et128x64_m256x256x64_c2x1x1_rN_TN_schPdx3_biasFp32N_fCp_tmOv_bM_rgTma_clmp_dynB_sm100f", 384, "659921858310fca0942fca2f874168f63481b8bdc1e3feaa867cd8938fbb6d96", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x256x256_s5x5x5x5x1x1_et128x64_m256x256x64_c2x1x1_rN_TN_schPdx3_biasFp32N_fCp_tmOv_bM_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x256x256_s5x5x5x5x1x1_et128x64_m256x256x64_c2x1x1_rN_TN_schPdx3_biasFp32N_fCp_tmOv_bM_rgTma_clmp_dynB_sm100f_cubin_len, 215704, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x256x256_s5x5x5x5x1x1_et128x64_m256x256x64_c2x1x1_rN_TN_schPdx3_biasFp32N_fCp_tmOv_bM_rgTma_clmp_dynB_sm100f", 384, "ae82e8f732b917cb54d6bf7d98ca2630b634ccbf5eee1ca68767f46f4380a371", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(2) , /* mBlockK */ -1 @@ -7911,10 +7892,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 32 , /* mEpilogueLdtmBits */ 32 , /* mEpilogueTileM */ 128 @@ -7951,7 +7932,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -7959,7 +7939,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 1 , /* mNumStagesTmemSfB */ 1 , /* mNumStagesWorkId */ 3 @@ -7974,7 +7953,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -8021,10 +7999,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 221296, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "cb5193e99bed46c42b57324d97f16d47f41b3c118611f2b30647f2e12ae04185", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 221328, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "970f2a89428aa5c550a746b54cfc62ef22b12a662b4716b26e24b4a3fec6ad97", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -8039,10 +8018,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -8079,7 +8058,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 2 @@ -8087,7 +8065,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -8102,7 +8079,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -8149,10 +8125,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 221056, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "45d074818044372d1b18c83b58269044f95c6d96ee1623657e651ecc812d762d", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 221200, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "c4d8067a82a110d34f7b4b172ad9f4dc559e072392f8a8e33d7279f48c91400f", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -8167,10 +8144,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -8207,7 +8184,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 1 @@ -8215,7 +8191,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -8230,7 +8205,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -8277,10 +8251,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 221296, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "277f568f16dfc4e4d4e3619f9983eef34678156d1570af9e3556b813961a688e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 221328, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "5ddcd34ac1ca8c6c67178a3bf209fa424cccf12bc0bedef3be2b5d995d92909b", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -8295,10 +8270,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -8335,7 +8310,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 2 @@ -8343,7 +8317,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -8358,7 +8331,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -8405,10 +8377,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 221056, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "b6cb044c2d680dfdaf83ce519e0f21ae0d340dc1086aec39ff8947674382c459", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 221200, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "61826677f58e6051bdfaf73ae663da6b9e10dc2b928c1bb8490d1a9b15b03a28", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -8423,10 +8396,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -8463,7 +8436,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 1 @@ -8471,7 +8443,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -8486,7 +8457,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -8533,10 +8503,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 222968, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "89eedf651dc48a04a3b289f73ac9f65daac604de9e2154813894405e3cf4ecb5", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 223128, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "97571478a6b3d803b4b6cdec98a144e6fd333299e7bb30f7c14a1d53cbc24709", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -8551,10 +8522,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -8591,7 +8562,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -8599,7 +8569,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -8614,7 +8583,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -8661,10 +8629,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 222728, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "533d94708c6c50f6f18cc2bf25baf27984490cc81d80982945ff67746d80601b", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 222744, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "18e909d98a5681eaf4f7b90a583a7549d6237548aafde795afe059d729f95e68", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -8679,10 +8648,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -8719,7 +8688,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -8727,7 +8695,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -8742,7 +8709,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -8789,10 +8755,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 222968, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "aa9614df7e6e58080afcff23c914fd034e26ee9e41f63e3d9c4f4cdbad28bfd3", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 223128, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "9835859f8e625856adca1d653adbff667e5a3f23c060667f570be2c4b09b569a", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -8807,10 +8774,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -8847,7 +8814,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -8855,7 +8821,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -8870,7 +8835,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -8917,10 +8881,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 222728, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "6f74111ac6153013380619e400222f08222670475d66fc946a39d37f995dcbfc", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 222744, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "67e981ad966bbc438e979010cdb1e39b28d1ea9c0bf9b61edb470a8076abdd15", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -8935,10 +8900,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -8975,7 +8940,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -8983,7 +8947,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -8998,7 +8961,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -9045,10 +9007,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x64x512_s4_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x64x512_s4_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 210584, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x64x512_s4_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "79150f9d624cb68acedd26a360bb153a664111b5cc02ec3d4458c1d390259d0d", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x64x512_s4_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x64x512_s4_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 210584, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x64x512_s4_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "35480670ac9f9f05f2a40cce7036467da7e6b09aa7bd9cd3186bbb7ec21fb05f", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -9063,10 +9026,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -9103,7 +9066,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 2 @@ -9111,7 +9073,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -9126,7 +9087,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -9173,10 +9133,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x64x512u2_s4_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x64x512u2_s4_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 210584, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x64x512u2_s4_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "a9c3c3488b93f483ff214a15aa704ab9037f5bc779ee521c80614901b7ccebd5", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x64x512u2_s4_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x64x512u2_s4_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 210584, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x64x512u2_s4_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "a6c29be6b1772a5d01708c04f36cafa9625b8e98bccbb7a1ce59d70ae3946aa1", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -9191,10 +9152,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -9231,7 +9192,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 2 @@ -9239,7 +9199,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -9254,7 +9213,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -9301,10 +9259,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 184432, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "e6b1c6020f36ff02b8ef12644ff7124eb5a0e130c05a81a32235e657216acb4e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 184464, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "dadcbdfcd871ee40b1d1a8bb02874a54e01af6808d2cdabacb34bd28f589aba0", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -9319,10 +9278,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -9359,7 +9318,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 2 @@ -9367,7 +9325,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -9382,7 +9339,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -9429,10 +9385,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 184192, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "85e4802f9c3707141d298f60f92b8cc5bf75da479c5e556e9867d25038e79a88", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 184336, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "d19b8baa2ea768aa1a545d84b09288e1f9b0d50ca5f6767686f7a4fca42028eb", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -9447,10 +9404,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -9487,7 +9444,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 1 @@ -9495,7 +9451,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -9510,7 +9465,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -9557,10 +9511,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 184432, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "df9e25cc1222d830d5e2dd51a7bc9172b54a4cf02170feba017ad858917709f5", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 184464, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "74f4a6fbb9cec92c3730bc474b091c0f0ab960bb239d141e571dcec4dc59ed18", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -9575,10 +9530,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -9615,7 +9570,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 2 @@ -9623,7 +9577,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -9638,7 +9591,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -9685,10 +9637,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 184192, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "561fe7006ec94bb14055f1480c5ffd0fd525d74c237fca3ca2e6fb0e91001b38", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 184336, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "3c9604d0c49a2428b29d1928759018f3a43aea5779e36513571ce4526629487e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -9703,10 +9656,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -9743,7 +9696,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 1 @@ -9751,7 +9703,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -9766,7 +9717,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -9813,10 +9763,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f_cubin_len, 163232, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f", 512, "e76d6c68f92d435d40cc3a9d574f728e080457b0d206bf881ba4040cb4b4100c", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f_cubin_len, 163344, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f", 512, "18e6c258af654bb21bcd3ee7dd56a57440bc851cd3962e31c55312c08a740740", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -9831,10 +9782,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 0 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -9871,7 +9822,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 1 @@ -9879,7 +9829,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -9894,7 +9843,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -9941,10 +9889,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 203504, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "155a4d159b9bcc60d99c6b10633dd5a724574d1cee53b0e14828920b71ae76f0", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 203664, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "5fb85a2f4cf4a0ea644ad5f5644dac8efc7f0d161db696972ad7db8f2bb5f18a", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -9959,10 +9908,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -9999,7 +9948,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -10007,7 +9955,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -10022,7 +9969,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -10069,10 +10015,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 203264, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "de54dde3891caf000c9482323dbe8b14217cbe925d4c3328e1bc17b4b12aab2c", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 203280, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "cbeddea34380981d827339a7b74a63c31d8fb111e0b5c8d948d0f23ffb65a045", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -10087,10 +10034,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -10127,7 +10074,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -10135,7 +10081,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -10150,7 +10095,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -10197,10 +10141,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x2_rM_splitK2_TN_transOut_schPdx3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x2_rM_splitK2_TN_transOut_schPdx3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 209640, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x2_rM_splitK2_TN_transOut_schPdx3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "37b994245f956d1eb0657320767fcb3c4f18f5603de6bde15052045a63655c6c", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x2_rM_splitK2_TN_transOut_schPdx3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x2_rM_splitK2_TN_transOut_schPdx3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 209816, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x2_rM_splitK2_TN_transOut_schPdx3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "ce1bedeb6391dca56a5f425f5fa987958162e039276d213a10f07b449b150cf7", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -10215,10 +10160,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -10255,7 +10200,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 2 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -10263,7 +10207,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -10278,7 +10221,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(2) , /* mTileK */ 512 @@ -10325,10 +10267,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x3_rM_splitK3_TN_transOut_schPdx3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x3_rM_splitK3_TN_transOut_schPdx3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 213736, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x3_rM_splitK3_TN_transOut_schPdx3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "ecacca24e4d88775a965be2b14959968f86b340413e7da177d48aaefcb20b5da", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x3_rM_splitK3_TN_transOut_schPdx3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x3_rM_splitK3_TN_transOut_schPdx3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 213912, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x3_rM_splitK3_TN_transOut_schPdx3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "a36ea163e919cb2d0673ac22d99f37cc1c096201cfc6b630d2398a58d8fbd160", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -10343,10 +10286,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -10383,7 +10326,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 3 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -10391,7 +10333,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -10406,7 +10347,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(2) , /* mTileK */ 512 @@ -10453,10 +10393,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x4_rM_splitK4_TN_transOut_schPdx3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x4_rM_splitK4_TN_transOut_schPdx3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 217832, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x4_rM_splitK4_TN_transOut_schPdx3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "1a3478389169d0bfa88a86986c5c94476ba0b27e2c16ce093f310a8fdd5a8aef", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x4_rM_splitK4_TN_transOut_schPdx3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x4_rM_splitK4_TN_transOut_schPdx3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 218008, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x4_rM_splitK4_TN_transOut_schPdx3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "020f440af5828a27f01ef43dd8ab42e209da8481fa16a4a4f1b03fd5c18f6e47", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -10471,10 +10412,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -10511,7 +10452,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 4 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -10519,7 +10459,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -10534,7 +10473,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(2) , /* mTileK */ 512 @@ -10581,10 +10519,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512u2_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512u2_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f_cubin_len, 163232, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512u2_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f", 512, "90b7d19f0752243e604c4aed85ceefe96651bcbc6031239eeb6c9ea04e714f4d", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512u2_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512u2_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f_cubin_len, 163344, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512u2_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f", 512, "0999d64d09b451a2d56b500acd036ad93ca69c2d2b87c740e42bf4743653f13e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -10599,10 +10538,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 0 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -10639,7 +10578,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 1 @@ -10647,7 +10585,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -10662,7 +10599,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -10709,10 +10645,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 203504, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "1b46fa91dbfd9b7272335bb408a6137176bda5701b438339bfebc49d2e2e22dd", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 203664, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "eb14d82ae6fbb552bd01994ffccb44deef33083b059cab26535cfcfa5739ddda", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -10727,10 +10664,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -10767,7 +10704,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -10775,7 +10711,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -10790,7 +10725,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -10837,10 +10771,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 203264, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "40981e6a465b6c7be3c7fc97717d4de7e3a11c6285ef13a3200eb42912e71d08", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 203280, "bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "29069189319ef6c2569e22ffb49284750acea524254be1320edb51f81890f8c1", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -10855,10 +10790,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -10895,7 +10830,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -10903,7 +10837,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -10918,7 +10851,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -10965,10 +10897,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f_cubin_len, 206976, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f", 512, "3f8e9385dc86c9753e3f8e6ddcb4a9915427c1d3ff2561c1980c6c541e1f82ca", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f_cubin_len, 207024, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f", 512, "c7076a07becce38262bb2bf1ec8c5372717bf40fd309a8a5e1adda00c1fb2eb3", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -10983,10 +10916,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 64 @@ -11023,7 +10956,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 160 , /* mNumRegsPerThreadNonEpilogueWarp */ 64 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 4 @@ -11031,7 +10963,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -11046,7 +10977,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -11093,10 +11023,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f_cubin_len, 206688, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f", 384, "45b6492ac095797e10a4d8e46a4e22278e261c5ad5e61660c98b7d12ae6f01f4", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f_cubin_len, 206896, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f", 384, "15ab98b956990c81ee07240b644ec12ab4ad83c40b79a955a44136e34e579215", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -11111,10 +11042,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 64 @@ -11151,7 +11082,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 160 , /* mNumRegsPerThreadNonEpilogueWarp */ 64 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -11159,7 +11089,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -11174,7 +11103,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -11221,10 +11149,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f_cubin_len, 185848, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f", 256, "0bf3d7a10e6698a21e56536008133af03050b103d12560467150bb6836f7d539", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f_cubin_len, 186008, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f", 256, "7128edf9fd2f4dd3079e611b0c60e6e5b25853b2a5975c1c82baad0118121bb6", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -11239,10 +11168,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -11279,7 +11208,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 128 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 7 , /* mNumStagesB */ 7 , /* mNumStagesMma */ 2 @@ -11287,7 +11215,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 7 , /* mNumStagesSmemSfB */ 7 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 7 , /* mNumStagesTmemSfB */ 7 , /* mNumStagesWorkId */ 3 @@ -11302,7 +11229,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -11349,10 +11275,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128u2_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128u2_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f_cubin_len, 206976, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128u2_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f", 512, "0063a4edcb479bd181d618599a650bc4d4cd66024c67f6e818952facb1049ad2", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128u2_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128u2_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f_cubin_len, 207024, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128u2_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f", 512, "5a277b429f57e417b12e05a2d1c099c85935bb5200e67374d3453208b2ed9e6f", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -11367,10 +11294,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 64 @@ -11407,7 +11334,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 160 , /* mNumRegsPerThreadNonEpilogueWarp */ 64 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 4 @@ -11415,7 +11341,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -11430,7 +11355,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -11477,10 +11401,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128u2_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128u2_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f_cubin_len, 206688, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128u2_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f", 384, "90c1602e49fdbea1438a70a85b36ff7e5de16c501bf25e4377f9a32f4bdcb548", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128u2_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128u2_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f_cubin_len, 206896, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128u2_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f", 384, "e8a33eb0b61711aaf19375b2e05af2350dd7b2ac2606e89fb76527b7ae66a563", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -11495,10 +11420,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 64 @@ -11535,7 +11460,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 160 , /* mNumRegsPerThreadNonEpilogueWarp */ 64 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -11543,7 +11467,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -11558,7 +11481,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -11605,10 +11527,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128u2_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128u2_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f_cubin_len, 185848, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128u2_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f", 256, "f2e4a1d7ed7167e08671c2cfe408e3a0b316f7349bac88741f3e0e5b2fb420a2", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128u2_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128u2_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f_cubin_len, 186008, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128u2_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f", 256, "5f247addc9aefed3366663918da1784dc280f41c2ccf8283e12ca068c0535065", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -11623,10 +11546,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -11663,7 +11586,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 128 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 7 , /* mNumStagesB */ 7 , /* mNumStagesMma */ 2 @@ -11671,7 +11593,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 7 , /* mNumStagesSmemSfB */ 7 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 7 , /* mNumStagesTmemSfB */ 7 , /* mNumStagesWorkId */ 3 @@ -11686,7 +11607,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -11733,10 +11653,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x128_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x128_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f_cubin_len, 121264, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x128_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f", 512, "eddc8624b6c1c57cdc575c4a690f483b59f6f2e9b18ab5ccfaf674522db9904a", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x128_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x128_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f_cubin_len, 121264, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x128_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f", 512, "3bdc7309ac3672e853be5350f2dd2ed5fa6cacee2a94bd76ce683c541db5492d", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -11751,10 +11672,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 64 @@ -11791,7 +11712,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 160 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 4 @@ -11799,7 +11719,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -11814,7 +11733,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -11861,10 +11779,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x128_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x128_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f_cubin_len, 120976, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x128_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f", 384, "e063655422b6097cbed9cbb505c29ba9bc97df74a5ef27e4aa8be18e74e57a03", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x128_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x128_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f_cubin_len, 121136, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x128_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f", 384, "063f448ad6b7ce2b273dadacd5192e7616e5aba2949283428d01c315bfef6b19", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -11879,10 +11798,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 64 @@ -11919,7 +11838,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 160 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -11927,7 +11845,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -11942,7 +11859,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -11989,10 +11905,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x128u2_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x128u2_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f_cubin_len, 121264, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x128u2_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f", 512, "09cf3339727cb0b69f4299e8ca71f3f1b8048efa032f7b72fe303c4bf02ec29b", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x128u2_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x128u2_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f_cubin_len, 121264, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x128u2_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f", 512, "f3e75c31605f056fad394a33127715c119804b85dbe8c7be9a58dc998d94972d", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -12007,10 +11924,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 64 @@ -12047,7 +11964,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 160 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 4 @@ -12055,7 +11971,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -12070,7 +11985,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -12117,10 +12031,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x128u2_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x128u2_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f_cubin_len, 120976, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x128u2_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f", 384, "2572f590640d7ddadd2db750461e53cb42b1d45d81f7f45dd24017dce2438dd3", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x128u2_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x128u2_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f_cubin_len, 121136, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x128u2_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f", 384, "978130f8eca8c21e1a63f29cc87b2af31a3887e8a82d8a0608546f94af4d9aba", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -12135,10 +12050,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 64 @@ -12175,7 +12090,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 160 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -12183,7 +12097,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -12198,7 +12111,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -12245,10 +12157,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f_cubin_len, 229840, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f", 256, "cd67dad42d6001ce71ef243db4b0cf5f5ce5dee609ee649f6405e63afb279e31", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f_cubin_len, 230032, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f", 256, "b8cb10a32f8da3c06f9b8c0183e22f31a7b348745222558ef08b3ad921abe25e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -12263,10 +12176,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -12303,7 +12216,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -12311,7 +12223,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -12326,7 +12237,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -12373,10 +12283,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f_cubin_len, 229600, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f", 256, "fcacc7a005a71dcfa3ca3b34beee2b6abed386c913881b541288a51159025c1e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f_cubin_len, 229648, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f", 256, "41fc31c7c4a1f4ee4ae1e9e90ca2317ea8ede1af2b95e9928af29f8b0493e034", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -12391,10 +12302,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -12431,7 +12342,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 1 @@ -12439,7 +12349,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -12454,7 +12363,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -12501,10 +12409,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f_cubin_len, 229840, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f", 256, "c28d9c05801124a90922f8d58d8ecdf627e63020624ec665f1106c26d82b7eb0", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f_cubin_len, 230032, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f", 256, "df16da2a0aa0c9499d92f1b12bfb1865fb1a4c34d208fa75d82120190273195c", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -12519,10 +12428,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -12559,7 +12468,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -12567,7 +12475,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -12582,7 +12489,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -12629,10 +12535,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f_cubin_len, 229600, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f", 256, "aee2e551c96312ed95054b8198cacdddc3f76eb02a574a0b7eeed73714ce7005", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f_cubin_len, 229648, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f", 256, "cc5bb964f490c1e86ec87a3a5828356ba4af6098ce643358d5d550db54a96f29", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -12647,10 +12554,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -12687,7 +12594,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 1 @@ -12695,7 +12601,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -12710,7 +12615,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -12757,10 +12661,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x192x128_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x192x128_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f_cubin_len, 215544, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x192x128_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f", 256, "bb0c3558729e88131c3fc7c235d0bc3ee61531383b62d5227481b03fb57acdb0", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x192x128_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x192x128_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f_cubin_len, 215704, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x192x128_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f", 256, "96fcefdb8923f824dacb266a360daafa89a026362b515e06862ec82e57ae6161", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -12775,10 +12680,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -12815,7 +12720,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 128 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 7 , /* mNumStagesB */ 7 , /* mNumStagesMma */ 2 @@ -12823,7 +12727,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 7 , /* mNumStagesSmemSfB */ 7 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 7 , /* mNumStagesTmemSfB */ 7 , /* mNumStagesWorkId */ 3 @@ -12838,7 +12741,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -12885,10 +12787,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x192x128u2_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x192x128u2_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f_cubin_len, 215544, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x192x128u2_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f", 256, "063d7412e97d2c6664b2aacfafc320ceb0269aa4acd23cab8e2538c1147b4339", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x192x128u2_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x192x128u2_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f_cubin_len, 215704, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x192x128u2_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f", 256, "823fc84fa1124a449729875de22a9f3a8d780fb08e0694e3a7b3fe7c55b55438", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -12903,10 +12806,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -12943,7 +12846,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 128 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 7 , /* mNumStagesB */ 7 , /* mNumStagesMma */ 2 @@ -12951,7 +12853,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 7 , /* mNumStagesSmemSfB */ 7 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 7 , /* mNumStagesTmemSfB */ 7 , /* mNumStagesWorkId */ 3 @@ -12966,7 +12867,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -13013,10 +12913,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x256x128_s6_et128x32_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_eW8_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x256x128_s6_et128x32_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_eW8_bN_rgTma_clmp_dynB_sm100f_cubin_len, 220632, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x256x128_s6_et128x32_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_eW8_bN_rgTma_clmp_dynB_sm100f", 384, "8efb0cb358a277ff04e9a8af7435bce54740676f242b4224b50fbf802e335690", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x256x128_s6_et128x32_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_eW8_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x256x128_s6_et128x32_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_eW8_bN_rgTma_clmp_dynB_sm100f_cubin_len, 220824, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x256x128_s6_et128x32_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_eW8_bN_rgTma_clmp_dynB_sm100f", 384, "fc7e65a2f48d463d8f1b33bb8a76f836fbeb5cb408e66bc7ac577c0af6c4e022", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -13031,10 +12932,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -13071,7 +12972,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 128 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -13079,7 +12979,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -13094,7 +12993,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -13141,10 +13039,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x256x128u2_s6_et128x32_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_eW8_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x256x128u2_s6_et128x32_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_eW8_bN_rgTma_clmp_dynB_sm100f_cubin_len, 220632, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x256x128u2_s6_et128x32_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_eW8_bN_rgTma_clmp_dynB_sm100f", 384, "bf8afc854967360b69f719bf7363e6c0a855a70374326c3f14d9844a648c039e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x256x128u2_s6_et128x32_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_eW8_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x256x128u2_s6_et128x32_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_eW8_bN_rgTma_clmp_dynB_sm100f_cubin_len, 220824, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x256x128u2_s6_et128x32_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_eW8_bN_rgTma_clmp_dynB_sm100f", 384, "cd4e9fd0ec7a5012bde5d32982ec000aafc6ba2dad0f7c688132d705dd81814e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -13159,10 +13058,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -13199,7 +13098,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 128 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -13207,7 +13105,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -13222,7 +13119,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -13269,10 +13165,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x128_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x128_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f_cubin_len, 137648, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x128_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f", 512, "a643ee20f857b218d8eefd6293c18d6d3344e2c0bbdcf960783a70c3ae6bf4ee", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x128_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x128_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f_cubin_len, 137648, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x128_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f", 512, "5dad3f69616bdb3866e8275af5cedc50966b3609d87c5eb940acc53ee9d22aa2", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -13287,10 +13184,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 64 @@ -13327,7 +13224,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 160 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 4 @@ -13335,7 +13231,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -13350,7 +13245,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -13397,10 +13291,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x128_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x128_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f_cubin_len, 137360, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x128_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f", 384, "ed918177993fc76129ded33af3622e3e1bba88ebe07866a271a45bb627f401fe", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x128_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x128_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f_cubin_len, 137520, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x128_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f", 384, "adbc8fa5ffca70c872173d9344eef2e4860339895fcb0ecc4f333075b367276a", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -13415,10 +13310,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 64 @@ -13455,7 +13350,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 160 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -13463,7 +13357,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -13478,7 +13371,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -13525,10 +13417,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x128u2_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x128u2_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f_cubin_len, 137648, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x128u2_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f", 512, "af85edc460946ba8a503cfd86e4a28812160303c89759935c1f24541f2ba4651", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x128u2_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x128u2_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f_cubin_len, 137648, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x128u2_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f", 512, "9092d9f971d793b5d4334f70a3c6c42a7b52593715b9f71768d1fdcf3554b8ca", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -13543,10 +13436,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 64 @@ -13583,7 +13476,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 160 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 4 @@ -13591,7 +13483,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -13606,7 +13497,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -13653,10 +13543,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x128u2_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x128u2_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f_cubin_len, 137360, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x128u2_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f", 384, "38c06c8985cc10c4b810f07f9539bf0debc273bffca54e2f2409f2576a9d1091", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x128u2_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x128u2_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f_cubin_len, 137520, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x128u2_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f", 384, "1f79b16012bc62aa3540a52e86809ab248eb38699d5680ead9cb18222e8c0736", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -13671,10 +13562,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 64 @@ -13711,7 +13602,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 160 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -13719,7 +13609,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -13734,7 +13623,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -13781,10 +13669,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f_cubin_len, 217520, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f", 256, "bbe62ca322668e6bde507412e9c649184e0aa3cde7db8da8c1dd0bb9d08620f1", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f_cubin_len, 217744, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f", 256, "123f8e4e64460410f5626f4e1666555687ca3708c9df505a0b2d5c77e8de7707", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -13799,10 +13688,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -13839,7 +13728,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -13847,7 +13735,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -13862,7 +13749,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -13909,10 +13795,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f_cubin_len, 217280, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f", 256, "e825583c76319f727ad49b0a7337be0d6a2e41cbc9665fd9baf0680a13d29ead", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f_cubin_len, 217360, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f", 256, "73b74b3e2ef44c643cff9751a519834ff76f0f0e53b26d883dc4b92df016960d", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -13927,10 +13814,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -13967,7 +13854,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -13975,7 +13861,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -13990,7 +13875,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -14037,10 +13921,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f_cubin_len, 217520, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f", 256, "5d94fd1827ac79362b76f8127e888ac06ccc0df53e8b3e77d8136dfe5f9856cf", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f_cubin_len, 217744, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f", 256, "cbe65be83568be1a3efeca3c64a4c39b60f1f14775a2406185828bb0b882a6fb", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -14055,10 +13940,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -14095,7 +13980,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -14103,7 +13987,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -14118,7 +14001,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -14165,10 +14047,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f_cubin_len, 217280, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f", 256, "1b8df965c2d57a6155672190f8b88b48780a88f39ca6b1be0a09d6ed62183a62", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f_cubin_len, 217360, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f", 256, "8b523256bd34c4db92dfecc924ba4a86f689cef40e4e977ca38031d32cc765b8", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -14183,10 +14066,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -14223,7 +14106,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -14231,7 +14113,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -14246,7 +14127,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -14293,10 +14173,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f_cubin_len, 172208, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f", 512, "78db32ce34970e0624cdda1d6631ded3841c1d5fbdc6010b325104e7cef2b51f", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f_cubin_len, 172208, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f", 512, "ecad6e81b5296824f6b69f5b3c9a500b9d88c9ea5f8487df417454c7b7c9bfae", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -14311,10 +14192,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 64 @@ -14351,7 +14232,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 160 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 4 @@ -14359,7 +14239,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -14374,7 +14253,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -14421,10 +14299,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f_cubin_len, 171920, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f", 384, "ffb28ef7a393e12b1f2c7b47cce10793ee09ed646514ae939a336754ee3fb204", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f_cubin_len, 172080, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f", 384, "412e8d953f355174014bd67340fcdf4d9e272d281264344237a39f379d25f51f", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -14439,10 +14318,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 64 @@ -14479,7 +14358,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 160 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -14487,7 +14365,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -14502,7 +14379,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -14549,10 +14425,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128_s8_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128_s8_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f_cubin_len, 184856, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128_s8_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f", 256, "0d0c4aa73e6e487388811ba17e774528060596ad477327a27b279d45d9a9daf1", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128_s8_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128_s8_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f_cubin_len, 184984, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128_s8_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f", 256, "a0fbbe73d9fde39161bce440dd2c8ce7ca567254e7cfbce64c96f1242092a9c0", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -14567,10 +14444,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -14607,7 +14484,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 128 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 8 , /* mNumStagesB */ 8 , /* mNumStagesMma */ 2 @@ -14615,7 +14491,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 8 , /* mNumStagesSmemSfB */ 8 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 8 , /* mNumStagesTmemSfB */ 8 , /* mNumStagesWorkId */ 3 @@ -14630,7 +14505,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -14677,10 +14551,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128u2_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128u2_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f_cubin_len, 172208, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128u2_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f", 512, "dd563f9a350e6c1a7a5b3a4a1bf3ecd107e86797e0474cf9c8d6ea5c2329eefe", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128u2_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128u2_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f_cubin_len, 172208, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128u2_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f", 512, "c5bd27a6f77ee9aee22e127672bd368498e256d0830d65b77f0b9da5dd0356e1", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -14695,10 +14570,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 64 @@ -14735,7 +14610,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 160 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 4 @@ -14743,7 +14617,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -14758,7 +14631,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -14805,10 +14677,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128u2_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128u2_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f_cubin_len, 171920, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128u2_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f", 384, "928676dbeda0edf821f9ba83d03f0aa9fd4d4c5fe826277a7eb8aaef6dfabef6", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128u2_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128u2_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f_cubin_len, 172080, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128u2_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f", 384, "1df246bc938e50dd2139d56137eac86d65b1f16342e75843e631c3623bca268e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -14823,10 +14696,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 64 @@ -14863,7 +14736,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 160 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -14871,7 +14743,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -14886,7 +14757,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -14933,10 +14803,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128u2_s8_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128u2_s8_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f_cubin_len, 184856, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128u2_s8_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f", 256, "68724258b272dc63794d8eb99960c75d994206fe3459a6d38afe8f2de04d3a90", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128u2_s8_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128u2_s8_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f_cubin_len, 184984, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128u2_s8_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f", 256, "d093cc5e98a84a4b3e44bc970fa5e0046b8abd59bb968c8ba67cb255d5c753c5", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -14951,10 +14822,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -14991,7 +14862,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 128 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 8 , /* mNumStagesB */ 8 , /* mNumStagesMma */ 2 @@ -14999,7 +14869,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 8 , /* mNumStagesSmemSfB */ 8 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 8 , /* mNumStagesTmemSfB */ 8 , /* mNumStagesWorkId */ 3 @@ -15014,7 +14883,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -15061,10 +14929,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f_cubin_len, 77600, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f", 384, "f2b20b1cba65b6b61630ff29924a71017acc2ee3d9ccf2d43afc1779866f6bb7", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f_cubin_len, 77600, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f", 384, "f13e492a388eae9128c7c8157ad2934e01510e61e0c828717d89dfceb533cf43", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -15079,10 +14948,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 0 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 64 @@ -15119,7 +14988,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 2 @@ -15127,7 +14995,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -15142,7 +15009,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -15189,10 +15055,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f_cubin_len, 148240, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f", 512, "042dc620db25956a69bd231ea59607980e94151e2e9d2d0e85b34c8dae3bebc9", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f_cubin_len, 148400, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f", 512, "86d1dc59a81f8e1562d76bf25414224e5263782be23a93a1ead9f9a03b8c3512", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -15207,10 +15074,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 64 @@ -15247,7 +15114,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 160 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 8 , /* mNumStagesB */ 8 , /* mNumStagesMma */ 4 @@ -15255,7 +15121,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 8 , /* mNumStagesSmemSfB */ 8 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 8 , /* mNumStagesTmemSfB */ 8 , /* mNumStagesWorkId */ 3 @@ -15270,7 +15135,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -15317,10 +15181,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f_cubin_len, 147952, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f", 384, "8239c0d2ad90d07dfe20a150481449920820d323cd4ef61efc9da861b933b193", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f_cubin_len, 148016, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f", 384, "4fbb0ef2471b2ce9a1cb7b7da27e94fee9064893d0df03ee93eab12817904271", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -15335,10 +15200,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 64 @@ -15375,7 +15240,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 160 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 8 , /* mNumStagesB */ 8 , /* mNumStagesMma */ 2 @@ -15383,7 +15247,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 8 , /* mNumStagesSmemSfB */ 8 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 8 , /* mNumStagesTmemSfB */ 8 , /* mNumStagesWorkId */ 3 @@ -15398,7 +15261,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -15445,10 +15307,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128u2_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128u2_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f_cubin_len, 77600, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128u2_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f", 384, "4c54ed1d11041524e2908fa2aae1e345141f757808877f426c09918abd093824", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128u2_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128u2_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f_cubin_len, 77600, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128u2_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f", 384, "3b4fe97a110b534bf76c21e7bcced520c5869a589cc1f58820fb3ba499fbff24", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -15463,10 +15326,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 0 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 64 @@ -15503,7 +15366,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 2 @@ -15511,7 +15373,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -15526,7 +15387,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -15573,10 +15433,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128u2_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128u2_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f_cubin_len, 148240, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128u2_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f", 512, "8bf694fb2592b455649ab99507a67c6e361604c9d12fbc915b919945d092dad9", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128u2_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128u2_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f_cubin_len, 148400, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128u2_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f", 512, "9c69aef2a3b75fe49487be1b9fdbdc3af022ba6ed6d718c3246265b155718f0f", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -15591,10 +15452,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 64 @@ -15631,7 +15492,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 160 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 8 , /* mNumStagesB */ 8 , /* mNumStagesMma */ 4 @@ -15639,7 +15499,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 8 , /* mNumStagesSmemSfB */ 8 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 8 , /* mNumStagesTmemSfB */ 8 , /* mNumStagesWorkId */ 3 @@ -15654,7 +15513,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -15701,10 +15559,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128u2_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128u2_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f_cubin_len, 147952, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128u2_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f", 384, "d814712de14b3941d8b639f4deb05b3e96347d001b4065054c7f1f29b8376163", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128u2_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128u2_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f_cubin_len, 148016, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128u2_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f", 384, "402ee0070a0f7f72b24b565de39142673957368e8f285d58b3d252e1fbc346c8", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -15719,10 +15578,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 64 @@ -15759,7 +15618,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 160 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 8 , /* mNumStagesB */ 8 , /* mNumStagesMma */ 2 @@ -15767,7 +15625,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 8 , /* mNumStagesSmemSfB */ 8 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 8 , /* mNumStagesTmemSfB */ 8 , /* mNumStagesWorkId */ 3 @@ -15782,7 +15639,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -15829,10 +15685,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f_cubin_len, 215504, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f", 256, "59276b4f5c9ee030bc8f0fbf38ebf40d6a3327ddb933924baf1a5163580e362f", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f_cubin_len, 215696, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f", 256, "9f8dd9abadfd78f4c73a894d53ba1ea9c43baaf264297fc3f645d6e62b9d43c3", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -15847,10 +15704,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -15887,7 +15744,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -15895,7 +15751,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -15910,7 +15765,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -15957,10 +15811,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f_cubin_len, 215264, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f", 256, "93c878f7793819717d6ac0164cf7b227ba98706cab8c2e2b0fadc4c05439c666", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f_cubin_len, 215312, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f", 256, "7d74759953228b33013cc809f620a6b9501abdc6b3644b4afa12c278f7e64550", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -15975,10 +15830,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -16015,7 +15870,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 1 @@ -16023,7 +15877,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -16038,7 +15891,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -16085,10 +15937,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f_cubin_len, 215504, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f", 256, "c2d1ec63c2bc4fbe941106672c13db7c041587fbe3afcaac3a5d97cce58e8edf", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f_cubin_len, 215696, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f", 256, "c78c311d7bff9aad6c6e3f1e5edccbecde9c0a7ec075af298f7a04ba28357eed", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -16103,10 +15956,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -16143,7 +15996,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -16151,7 +16003,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -16166,7 +16017,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -16213,10 +16063,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f_cubin_len, 215264, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f", 256, "ba596b2c4d449bac7d719d4e6a12ce4cf048305a1884144dfd8ecc4620221fb2", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f_cubin_len, 215312, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f", 256, "0fe5d9694eb04ca489b916af2227c0c9b823ef6e7825ea8259ed7094b0915555", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -16231,10 +16082,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -16271,7 +16122,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 1 @@ -16279,7 +16129,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -16294,7 +16143,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -16341,10 +16189,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f_cubin_len, 215168, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f", 256, "f12b9b2f7ae15d6b30d43b5d1675a214fd392fe3d7b973482fc47018f19e10e3", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f_cubin_len, 215312, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f", 256, "d0f8da72cc009b09b440bc5739a69ceaa82eeef1b1efab8b061a14ec034bfc88", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -16359,10 +16208,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 0 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -16399,7 +16248,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 1 @@ -16407,7 +16255,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -16422,7 +16269,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -16469,10 +16315,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f_cubin_len, 215168, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f", 256, "57bda14f4186f35585bd5ba9aa92bb8dae31fef9f0e7549728493eec533b2ec7", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f_cubin_len, 215312, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f", 256, "5f4a91492fbe02049074f669da1031f0163326afa11dd3cf7e32c5eed43220c8", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -16487,10 +16334,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 0 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -16527,7 +16374,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 1 @@ -16535,7 +16381,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -16550,7 +16395,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -16597,10 +16441,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f_cubin_len, 215168, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f", 256, "8ed58ef7c0e708886ba3ebbdf0666a4da932bf5811dc691152a4fd9cbb0a1ff3", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f_cubin_len, 215312, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f", 256, "bb1b7c893b276c17111879b59c88f0cf8b277da1b8053f52d86307f3736b93c0", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -16615,10 +16460,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 0 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -16655,7 +16500,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 1 @@ -16663,7 +16507,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -16678,7 +16521,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -16725,10 +16567,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f_cubin_len, 215168, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f", 256, "2fc40c49b02088076898bc9cb5fda898a11365006e955b26350e8fc299e05707", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f_cubin, Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f_cubin_len, 215312, "bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f", 256, "871632efcd4f975a16251bf51e9fe93f14cf9ec1856e9039106f81edde3ca50d", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -16743,10 +16586,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 0 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -16783,7 +16626,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 1 @@ -16791,7 +16633,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -16806,7 +16647,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -16853,10 +16693,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x128_s5_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x128_s5_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 73296, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x128_s5_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "9cc83b16d06d45d5ee317a43087587279692a1181c32583e84599d9cf2cc3ec8", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x128_s5_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x128_s5_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 73360, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x128_s5_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "8155544a5404b9347d9222113c0124d74318e7faf9cfa4cd3f9d5e8150f61b07", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -16871,10 +16712,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaB */ trtllm::gen::Dtype(1052672) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -16911,7 +16752,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -16919,7 +16759,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -16934,7 +16773,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -16981,10 +16819,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x128_s5_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x128_s5_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 73056, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x128_s5_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "43ccb4b975174484df81e8044d0ff6292f9b2698b1014f291237028dc0699187", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x128_s5_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x128_s5_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 73232, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x128_s5_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "f7b979cca3e86e66d3f7b47d61f7ded7bce8d6192e0604477d333019cb072c88", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -16999,10 +16838,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaB */ trtllm::gen::Dtype(1052672) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -17039,7 +16878,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -17047,7 +16885,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -17062,7 +16899,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -17109,10 +16945,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x128u2_s5_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x128u2_s5_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 73296, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x128u2_s5_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "1bd9e094812eb9f15f3f21f548d9cda88c76fedbfecb13acf067f7c40ecc03d4", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x128u2_s5_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x128u2_s5_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 73360, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x128u2_s5_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "ae623552d7edd0359eb2164cb7f90850d653e416db2f87efab13d42d1f523126", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -17127,10 +16964,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaB */ trtllm::gen::Dtype(1052672) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -17167,7 +17004,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -17175,7 +17011,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -17190,7 +17025,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -17237,10 +17071,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x128u2_s5_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x128u2_s5_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 73056, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x128u2_s5_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "750d100c0cae7591e37c69cf2d082e5d1ac73d4eb6914f16018c2d8c388e2269", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x128u2_s5_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x128u2_s5_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 73232, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x128u2_s5_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "05e19f6bab9609b1f39da63fa1c85362b9c777fd11ce1fdf6a7563892e784a2f", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -17255,10 +17090,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaB */ trtllm::gen::Dtype(1052672) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -17295,7 +17130,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -17303,7 +17137,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -17318,7 +17151,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -17365,10 +17197,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 85456, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "ed9b06abc13de8fc15543eec31ac8a25a89032fb6d80c7d9b47a05932e9d0173", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 85648, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "9c9575fb25aef17bbb0bc48aa218ac5c8d44cbab0f50a49e40ff7bf408ca2312", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -17383,10 +17216,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaB */ trtllm::gen::Dtype(1052672) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -17423,7 +17256,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 2 @@ -17431,7 +17263,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -17446,7 +17277,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -17493,10 +17323,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 85456, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "b1cf0c4a3af7ac78d349ce2eb1fbd4418b4a44fc6446da6d2e9b42e0bbd744ad", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 85648, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "c53987bd8fbcb51fac898808fbd942378c4a04aa2bd264c5c0b738c27f2902e7", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -17511,10 +17342,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaB */ trtllm::gen::Dtype(1052672) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -17551,7 +17382,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 2 @@ -17559,7 +17389,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -17574,7 +17403,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -17621,10 +17449,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 85216, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "e6ed0e14eaa68a93422518f42fbafeb210eafa1f85b264b12fdb4d145916a18f", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 85264, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "2d7e836ad7d69a994058c8a9c2616a34e1f043f053539a210825ab2e90f0c118", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -17639,10 +17468,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaB */ trtllm::gen::Dtype(1052672) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -17679,7 +17508,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 1 @@ -17687,7 +17515,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -17702,7 +17529,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -17749,10 +17575,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 85216, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "868add156a732c6bdd9a066e2a0f1df2f4e18d7fbf17ee94a79137d59bbbde86", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 85264, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "55246a68d69c49a705bfd786d35c179cf34909b652e49adf0ce00c39d4d8fbaa", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -17767,10 +17594,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaB */ trtllm::gen::Dtype(1052672) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -17807,7 +17634,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 1 @@ -17815,7 +17641,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -17830,7 +17655,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -17877,10 +17701,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256u2_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256u2_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 85456, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256u2_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "66fbf001f52b397cb4eed1ea3bd1ef52083cbf40bf7426c545e0e7ae3f1d7b3f", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256u2_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256u2_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 85648, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256u2_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "7eda59dc95153b78b261b213517f6324c9f64eb74e980add148cc1b665712b99", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -17895,10 +17720,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaB */ trtllm::gen::Dtype(1052672) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -17935,7 +17760,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 2 @@ -17943,7 +17767,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -17958,7 +17781,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -18005,10 +17827,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256u2_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256u2_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 85456, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256u2_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "1e8027ac444314bfb87cef0634960658d7c776f928be8828f624f2a3e75ecc2d", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256u2_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256u2_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 85648, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256u2_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "73ed5fad6dca98def881df3294ce03fd5a2144aa6196c13b40daff931739db2f", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -18023,10 +17846,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaB */ trtllm::gen::Dtype(1052672) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -18063,7 +17886,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 2 @@ -18071,7 +17893,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -18086,7 +17907,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -18133,10 +17953,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256u2_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256u2_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 85216, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256u2_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "a239ea039282cf99881218230f8e705ae6d750aa67ea83294f89010d4f999a11", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256u2_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256u2_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 85264, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256u2_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "381324bfe07f0a90efe932008c44e7c964bfb825a5f6a0999524921ebbe74aa8", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -18151,10 +17972,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaB */ trtllm::gen::Dtype(1052672) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -18191,7 +18012,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 1 @@ -18199,7 +18019,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -18214,7 +18033,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -18261,10 +18079,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256u2_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256u2_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 85216, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256u2_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "479c4092337e50fd7ff0cb789fe2f4ab7de7ef56e0d40d18db38d6a9ecd36701", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256u2_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256u2_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 85264, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256u2_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "ceb014566d6e415152dca433499c7bb8ff23dae899ccd622829084e20d27874b", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -18279,10 +18098,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaB */ trtllm::gen::Dtype(1052672) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -18319,7 +18138,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 1 @@ -18327,7 +18145,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -18342,7 +18159,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -18389,10 +18205,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x128_s5_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x128_s5_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 97872, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x128_s5_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "a1c95dc1446869181267f51ea3e0f6aa74b9d6c20a77538fb0435111bb348b78", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x128_s5_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x128_s5_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 97936, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x128_s5_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "6196af0d95f7581c85052aa556b4fd4cf06ae35cd4cb17f651caad20a05eb0aa", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -18407,10 +18224,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaB */ trtllm::gen::Dtype(1052672) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -18447,7 +18264,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -18455,7 +18271,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -18470,7 +18285,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -18517,10 +18331,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x128_s5_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x128_s5_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 97632, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x128_s5_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "bb567912462e01995b562140aa2117d3c53d5f056c3cf896381a2d57ef5d9f33", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x128_s5_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x128_s5_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 97808, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x128_s5_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "bf0249e2b97c4f96888c902b015eebfd15ca78e846fc0ad687fac2a8daf1df05", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -18535,10 +18350,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaB */ trtllm::gen::Dtype(1052672) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -18575,7 +18390,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -18583,7 +18397,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -18598,7 +18411,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -18645,10 +18457,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x128u2_s5_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x128u2_s5_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 97872, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x128u2_s5_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "503b8b9c294c81359a07c1a79cfc08ea230f503a1bdf8b4e9fdc89c46f161c51", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x128u2_s5_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x128u2_s5_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 97936, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x128u2_s5_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "372b5ed01052af0eb8b8c7780deef1d915b0dd6d413bc8ae23e96d9e5583263f", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -18663,10 +18476,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaB */ trtllm::gen::Dtype(1052672) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -18703,7 +18516,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -18711,7 +18523,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -18726,7 +18537,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -18773,10 +18583,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x128u2_s5_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x128u2_s5_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 97632, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x128u2_s5_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "f404be18bba8014d3087b8a61088b0910dec8f3ca1c80e0bdae32cb2994bf7c3", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x128u2_s5_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x128u2_s5_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 97808, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x128u2_s5_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "ac259a3feaee0830a400aa0a61b718069a0f7441f364ef24686c370cdf5f3500", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -18791,10 +18602,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaB */ trtllm::gen::Dtype(1052672) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -18831,7 +18642,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -18839,7 +18649,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -18854,7 +18663,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -18901,10 +18709,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 114128, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "b8cc881bef03dd23ba2a1ee5c71b15ea41e9043e3658bd67067036cab9d6fb39", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 114320, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "1c37123698499cf0914588fe7d287dc3985fad556ce7b0057d304f4d0ddd64fb", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -18919,10 +18728,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaB */ trtllm::gen::Dtype(1052672) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -18959,7 +18768,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 2 @@ -18967,7 +18775,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -18982,7 +18789,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -19029,10 +18835,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 114128, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "b09cdacf8b295bd19afe5723ff788c85d9e2dc52e59bbcb43062b26e290f3133", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 114320, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "e4c6c9437626ff54ddd4b61823d585eb14296a6be6b8f873e12bffedfaa49e08", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -19047,10 +18854,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaB */ trtllm::gen::Dtype(1052672) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -19087,7 +18894,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 2 @@ -19095,7 +18901,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -19110,7 +18915,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -19157,10 +18961,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 113888, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "dd109b062eeadbb554b405d5a5232fe45bce474880fafe82f2da4b383e239ca9", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 113936, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "82a473b1494e512b666f3683175dbb7a1b6aa8236d767b1820a8a8f6dd8cc1f2", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -19175,10 +18980,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaB */ trtllm::gen::Dtype(1052672) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -19215,7 +19020,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 1 @@ -19223,7 +19027,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -19238,7 +19041,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -19285,10 +19087,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 113888, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "440e772d9a73fcbc3ccca701cf0707682b2ca67a2350d891092ee0f327b15803", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 113936, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "1c3e601b4cebd1cc0e8c6ea1d7450a413108a07f4f01be89161ac54773b12f94", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -19303,10 +19106,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaB */ trtllm::gen::Dtype(1052672) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -19343,7 +19146,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 1 @@ -19351,7 +19153,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -19366,7 +19167,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -19413,10 +19213,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256u2_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256u2_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 114128, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256u2_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "0b0d9bba87da41cf6367a6ded8d7bac6951e4381d03281934151d1e40e4560ad", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256u2_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256u2_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 114320, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256u2_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "ddc17d1506b0d4f4ef2c7891f6c9bd12023a302d555f011deff5c4ccb4339c6b", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -19431,10 +19232,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaB */ trtllm::gen::Dtype(1052672) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -19471,7 +19272,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 2 @@ -19479,7 +19279,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -19494,7 +19293,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -19541,10 +19339,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256u2_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256u2_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 114128, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256u2_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "450f4254c6ba549bcfa76ec99d5f5e44b7ea029f611a88954f22eca4f02e3769", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256u2_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256u2_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 114320, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256u2_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "fc5b93748711eb953296b599e1b73cbdf1a9a680bf89fe9e4fe1a758ea554db4", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -19559,10 +19358,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaB */ trtllm::gen::Dtype(1052672) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -19599,7 +19398,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 2 @@ -19607,7 +19405,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -19622,7 +19419,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -19669,10 +19465,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256u2_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256u2_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 113888, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256u2_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "39224cb59412b451a0d11070452b4f0ea43e82f4c9df6996b1c5f914720d12c5", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256u2_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256u2_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 113936, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256u2_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "3fa3039e7dc64b9fffffac9bc4b580364c786f4642044fc5a241c126de21753d", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -19687,10 +19484,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaB */ trtllm::gen::Dtype(1052672) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -19727,7 +19524,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 1 @@ -19735,7 +19531,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -19750,7 +19545,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -19797,10 +19591,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256u2_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256u2_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 113888, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256u2_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "27539f98019a0d1de7b1f631c8cd96bcb2280df990e4be7702d3b9dcec54d8d5", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256u2_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256u2_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 113936, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256u2_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "74de757c9f7e73169519b557ba72de0f17742ff845b451ea2c18ca1d9e2a082d", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -19815,10 +19610,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaB */ trtllm::gen::Dtype(1052672) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -19855,7 +19650,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 1 @@ -19863,7 +19657,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -19878,7 +19671,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -19925,10 +19717,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x128_s5_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x128_s5_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 148048, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x128_s5_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "4ab36980d439db425b0a63c75cb4c716e87ed7dd6a6bf578465ac13754c1ef1c", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x128_s5_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x128_s5_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 148112, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x128_s5_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "69b2370bd2aeef420b5f0a197610c47fe5032f42983d302ffe2739472d337188", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -19943,10 +19736,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaB */ trtllm::gen::Dtype(1052672) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -19983,7 +19776,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -19991,7 +19783,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -20006,7 +19797,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -20053,10 +19843,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x128_s5_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x128_s5_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 147808, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x128_s5_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "7f308abe1899fb0359e4e578df585428f5dee5d22c92bffce73ec413e034421c", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x128_s5_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x128_s5_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 147984, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x128_s5_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "0dbc78db5d4d7a5ea65bbc394d6c0c1b19207b6847836e0bb8f6c27d53d9e93a", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -20071,10 +19862,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaB */ trtllm::gen::Dtype(1052672) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -20111,7 +19902,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -20119,7 +19909,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -20134,7 +19923,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -20181,10 +19969,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x128u2_s5_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x128u2_s5_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 148048, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x128u2_s5_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "fdba996bfc4033e98073c38c1c9a9e2b2c18b7e615600a712531d973035375fa", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x128u2_s5_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x128u2_s5_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 148112, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x128u2_s5_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "a3873db946b227b9d52bcb6a94066c1bc41c46274857c983c4f69f9c31bc7314", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -20199,10 +19988,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaB */ trtllm::gen::Dtype(1052672) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -20239,7 +20028,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -20247,7 +20035,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -20262,7 +20049,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -20309,10 +20095,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x128u2_s5_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x128u2_s5_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 147808, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x128u2_s5_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "8037304c85ca12a25a547ab5924ea25c70bc879a41ab640b3fbae42db7b1279d", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x128u2_s5_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x128u2_s5_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 147984, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x128u2_s5_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "8621e794901da057609e5403daaf1773a4824eb9e27225d7a5132e47aac14276", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -20327,10 +20114,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaB */ trtllm::gen::Dtype(1052672) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -20367,7 +20154,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -20375,7 +20161,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -20390,7 +20175,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -20437,10 +20221,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 172496, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "bca6d92d61fde55845dfc73b22407c84eed62ba97660610ffb859cd4afcd77b9", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 172688, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "b03fb8f83a7e7af9d003dfe59b8053190fde8d54c0e55aa50b3968ef1f672aec", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -20455,10 +20240,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaB */ trtllm::gen::Dtype(1052672) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -20495,7 +20280,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 2 @@ -20503,7 +20287,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -20518,7 +20301,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -20565,10 +20347,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 172496, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "d1ef2409fe1ff8333fcae5394f83fd8471454cbe3bc17d4b49027f9f21ba4b37", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 172688, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "1c743b6b0917bcbf6e17850ce1c9b77a3f193e26e11f5355afc7be2b60912d9e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -20583,10 +20366,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaB */ trtllm::gen::Dtype(1052672) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -20623,7 +20406,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 2 @@ -20631,7 +20413,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -20646,7 +20427,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -20693,10 +20473,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 172256, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "349b07c1f20861026022c9cc334b51f0e35416521867e98e49259ed07468de2a", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 172304, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "ceec67dd522dbec05896349ad717e20899af8258675d9f45b6ef175203517686", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -20711,10 +20492,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaB */ trtllm::gen::Dtype(1052672) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -20751,7 +20532,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 1 @@ -20759,7 +20539,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -20774,7 +20553,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -20821,10 +20599,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 172256, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "92eebe8d95124c97d6f026c06f18b14b14fe06f0b44f4b53229388be9ccd6c80", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 172304, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "5ab61542f6fd0fc3d4e66bedd205adb3347af5e6b7dbb532e241319565ea199f", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -20839,10 +20618,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaB */ trtllm::gen::Dtype(1052672) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -20879,7 +20658,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 1 @@ -20887,7 +20665,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -20902,7 +20679,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -20949,10 +20725,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256u2_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256u2_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 172496, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256u2_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "5836a80df333d6b999cb60f0e90486132899332f4e8b856db4b7e0afa5085905", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256u2_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256u2_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 172688, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256u2_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "939a16cfcd63913d99e60fa38a77ea0b558e81cddaa27b5e8783a44174335005", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -20967,10 +20744,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaB */ trtllm::gen::Dtype(1052672) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -21007,7 +20784,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 2 @@ -21015,7 +20791,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -21030,7 +20805,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -21077,10 +20851,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256u2_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256u2_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 172496, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256u2_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "6cac90f1e4e782eb950262f47d884b1af3f8d6fc6ed4b026ce015440e724e887", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256u2_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256u2_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 172688, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256u2_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "277e2e34995314a3aa82446e285ca6b9796daf3b257418ceabf8ce9d85df5e2c", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -21095,10 +20870,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaB */ trtllm::gen::Dtype(1052672) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -21135,7 +20910,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 2 @@ -21143,7 +20917,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -21158,7 +20931,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -21205,10 +20977,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256u2_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256u2_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 172256, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256u2_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "7469145be905ad08a6f937f89fab67ab73e09edf1d3aff0b692781b2faed0167", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256u2_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256u2_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 172304, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256u2_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "9fc3083ae35ace85c9b7168cb8eebe7a3fe21dfda58dc7c67be4e691c128e9d5", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -21223,10 +20996,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaB */ trtllm::gen::Dtype(1052672) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -21263,7 +21036,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 1 @@ -21271,7 +21043,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -21286,7 +21057,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -21333,10 +21103,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256u2_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256u2_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 172256, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256u2_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "5ed8aa0edc5cb3b5e095b6d3e273f4b9ca235a971bc81e359da90d94bf803653", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256u2_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256u2_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 172304, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256u2_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "d32f9270035b983dcea1c341f16528f0ab6bbdccc373687a273861c86f6a06fa", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -21351,10 +21122,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaB */ trtllm::gen::Dtype(1052672) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -21391,7 +21162,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 1 @@ -21399,7 +21169,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -21414,7 +21183,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -21461,10 +21229,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x128_s5_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x128_s5_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 61008, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x128_s5_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "4c52eda6ab91f2bd710113e726ab22150fa5fc11ba97c1a85fbbaf3b94b7c1af", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x128_s5_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x128_s5_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 61072, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x128_s5_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "cbc52c49c31203e700a6a270ef2ab70e61a7b640840e2c357a58f5168513a514", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -21479,10 +21248,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaB */ trtllm::gen::Dtype(1052672) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -21519,7 +21288,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -21527,7 +21295,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -21542,7 +21309,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -21589,10 +21355,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x128_s5_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x128_s5_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 60768, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x128_s5_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "de5b65c173fff472a0868dcb5f2c4424356afa2ac4c5015ba95f2e3e7b6926ed", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x128_s5_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x128_s5_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 60944, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x128_s5_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "c2117a13246a6d06f495b53eaf841afce884033088413ae55fe6a89758edbfd4", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -21607,10 +21374,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaB */ trtllm::gen::Dtype(1052672) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -21647,7 +21414,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -21655,7 +21421,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -21670,7 +21435,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -21717,10 +21481,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x128u2_s5_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x128u2_s5_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 61008, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x128u2_s5_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "60b15601fcaed9034a726fce39e8e58991d235033fbf300168d81a702b32e3fc", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x128u2_s5_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x128u2_s5_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 61072, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x128u2_s5_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "50c64cc351b55f77dbdcd6258b30818028c23e1467940cf2a377d311e86074d5", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -21735,10 +21500,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaB */ trtllm::gen::Dtype(1052672) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -21775,7 +21540,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -21783,7 +21547,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -21798,7 +21561,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -21845,10 +21607,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x128u2_s5_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x128u2_s5_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 60768, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x128u2_s5_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "0775ec9326a4456847e53afe3eb27517089c23b4a7bb24f21e9038045156964d", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x128u2_s5_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x128u2_s5_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 60944, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x128u2_s5_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "0f87305e47db77e2deb5ede7a708f7cdaece04cedbeccf8c57ff9c868ecee2c6", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -21863,10 +21626,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaB */ trtllm::gen::Dtype(1052672) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -21903,7 +21666,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -21911,7 +21673,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -21926,7 +21687,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -21973,10 +21733,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 71120, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "ae57689935d06e19de9b32b64ef02aebc2688af0fa773eea0c32a1ba76e79d09", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 71312, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "4692c832bd1df78af88c6c5e05b8c88cf004a585e355ae2215045627ee7b5818", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -21991,10 +21752,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaB */ trtllm::gen::Dtype(1052672) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -22031,7 +21792,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 2 @@ -22039,7 +21799,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -22054,7 +21813,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -22101,10 +21859,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 71120, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "f977d72b13a174f729b5154f33392bd226a073abb52212a8285942ddde76bf77", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 71312, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "a846c94052b732e65e766b62dce7e3f5f8b34dbc6e7024956591ccf6e870571e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -22119,10 +21878,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaB */ trtllm::gen::Dtype(1052672) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -22159,7 +21918,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 2 @@ -22167,7 +21925,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -22182,7 +21939,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -22229,10 +21985,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 70880, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "0cda58ce3f09e68edee231d9ab329253f7bcbe646ce2a2666980d5018c836fc5", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 70928, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "b7c9ca44c8565bfc4878a22b0836caad3c77154e20869f1ba1ce5546ffcc7895", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -22247,10 +22004,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaB */ trtllm::gen::Dtype(1052672) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -22287,7 +22044,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 1 @@ -22295,7 +22051,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -22310,7 +22065,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -22357,10 +22111,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 70880, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "2a93a93c088e0f1e8b20efaf854fbac0799530220b3a2bd8818f4495a3c34a33", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 70928, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "71aa86d00c4109f2c1898e53df0a23db4374ec983586ecb84151648e998b118d", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -22375,10 +22130,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaB */ trtllm::gen::Dtype(1052672) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -22415,7 +22170,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 1 @@ -22423,7 +22177,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -22438,7 +22191,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -22485,10 +22237,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256u2_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256u2_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 71120, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256u2_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "dab172f6cf3b599605e8865c32d981cd3eaf95ce53cc1b9a68b367454dc83f2e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256u2_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256u2_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 71312, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256u2_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "59be3d62b3fee330d59883eb8ed69d0f81ecbf0e2bcb48035187cdc749fa09b2", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -22503,10 +22256,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaB */ trtllm::gen::Dtype(1052672) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -22543,7 +22296,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 2 @@ -22551,7 +22303,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -22566,7 +22317,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -22613,10 +22363,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256u2_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256u2_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 71120, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256u2_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "a39907c65da7a7cace29b9f4b2d3b12fea0542a4911ca96ce477411060b4e380", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256u2_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256u2_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 71312, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256u2_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "754d23850a2187f06f7b970084e111044dd18c3a67e2ece83794d0cc8d5ec098", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -22631,10 +22382,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaB */ trtllm::gen::Dtype(1052672) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -22671,7 +22422,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 2 @@ -22679,7 +22429,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -22694,7 +22443,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -22741,10 +22489,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256u2_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256u2_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 70880, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256u2_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "44780dffb95b36d92cadce836144ac1842ea786690bd71046c3504ee70e7f565", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256u2_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256u2_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 70928, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256u2_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "d014728ad17d0ef3dcc9cc12f1b911a9cbd1ad1f588be9fac9a909ca0ec78c6c", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -22759,10 +22508,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaB */ trtllm::gen::Dtype(1052672) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -22799,7 +22548,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 1 @@ -22807,7 +22555,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -22822,7 +22569,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -22869,10 +22615,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256u2_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256u2_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 70880, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256u2_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "f290422cfb9e0a00a0347590ddda11c76c26e715d891cdd4dcf6dc66afea234e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256u2_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256u2_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 70928, "bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256u2_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "aa1dd791baf0417ac058380060a6d80a19fd2d60764b1bb7d65a9fc9fab17e33", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -22887,10 +22634,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaB */ trtllm::gen::Dtype(1052672) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -22927,7 +22674,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 1 @@ -22935,7 +22681,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -22950,7 +22695,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -22997,10 +22741,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 199248, "bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "1310c4245c39847f7dbb21cf83b463b7395fb188cb574bf4d30e9e679507237b", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 199312, "bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "0fece5d6f509a8d23732bd1b72749d9335219122be976c0c2115129824b87eb2", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -23015,10 +22760,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -23055,7 +22800,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -23063,7 +22807,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -23078,7 +22821,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -23125,10 +22867,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 199008, "bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "4964777cb780d80828df2d59be50bdf56faae972b6805b21de5deef6eab533cd", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 199184, "bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "5d9a0cb1d8be81b95293ca8fe6b051c3ff25448d071d88217eb5c8e321f84048", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -23143,10 +22886,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -23183,7 +22926,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -23191,7 +22933,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -23206,7 +22947,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -23253,10 +22993,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 199248, "bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "d1912dc6f30d07e1d25638d2a8fb6ebba6bf3c3bce0fd9dc1df3784b7a6059fd", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 199312, "bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "4df694a67874b6f3fe70fa3af962bb3f4aff3b358646f0693af1cdc865b23b3b", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -23271,10 +23012,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -23311,7 +23052,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -23319,7 +23059,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -23334,7 +23073,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -23381,10 +23119,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 199008, "bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "c4e8e6905c1e448d36daf639f0d3a732ef523a56425c0d4c342243b6078ab44c", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 199184, "bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "41b1a6b2fbd0b259bc45418b2e830238d1c098cb5aef835460cb0119b390d84e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -23399,10 +23138,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -23439,7 +23178,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -23447,7 +23185,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -23462,7 +23199,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -23509,10 +23245,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 223824, "bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "dbbc04946ef4bfa80f7e20902603ceaadb1149b1e6a187b3dc5dff93cd9ef72e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 223888, "bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "e8032168d0443fa360c2d035bba29e25f0fd30f57bcaaef99ea902fe9d0f3774", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -23527,10 +23264,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -23567,7 +23304,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -23575,7 +23311,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -23590,7 +23325,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -23637,10 +23371,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 223584, "bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "fd8209584269dbe80c620fc52d05c15ac5d3a88c6935df802648180fbfece4e1", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 223760, "bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "303214d601ba22b816d76bd3eed255cf17248025dfbb3202700591e86ec642b5", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -23655,10 +23390,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -23695,7 +23430,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -23703,7 +23437,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -23718,7 +23451,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -23765,10 +23497,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 223824, "bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "4b084b9012732b2cc6b1fa27938cca284d3d151ae59a03abebc341b573bc3032", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 223888, "bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "9c526f23697b2f36ee00a72167eeccfd82f76474938514fff86bc499de894502", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -23783,10 +23516,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -23823,7 +23556,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -23831,7 +23563,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -23846,7 +23577,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -23893,10 +23623,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 223584, "bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "5111a085adebb73a6074b190c97e94a0e6f4fbf16bb24b0414be835927a78e50", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 223760, "bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "efd919aac9d0bfad0763c822b4816be0c113bf7bcacb941077c3274ac6ed7ae7", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -23911,10 +23642,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -23951,7 +23682,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -23959,7 +23689,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -23974,7 +23703,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -24021,10 +23749,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 222736, "bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "2edf41c22604c32be237a4d6f6b8d3466f35819aef589bbb2863a9c73b912a0c", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 222864, "bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "752ecbfa4180314bf8980137341dbd4df672de072a8770ebab578ce1886595b6", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -24039,10 +23768,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -24079,7 +23808,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 2 @@ -24087,7 +23815,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -24102,7 +23829,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -24149,10 +23875,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 222496, "bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "80adb574d5a61a296e92c19e40a7f06dd1ed36c2ad6de572c0b9185a47bd2cf3", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 222736, "bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "e6b24c65a1f13b932131b92edce42430790840432732462c9e9f5257cec476ec", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -24167,10 +23894,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -24207,7 +23934,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 1 @@ -24215,7 +23941,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -24230,7 +23955,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -24277,10 +24001,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 222736, "bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "1b08d9e4c256ca904a77d16c1e3a4b290714cd9b1e1749ef3eb0df525d2b2416", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 222864, "bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "79f312523d5fd0a446821a14e4fdf8e7ce12cb1335220db769440b0e00dbc7df", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -24295,10 +24020,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -24335,7 +24060,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 2 @@ -24343,7 +24067,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -24358,7 +24081,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -24405,10 +24127,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 222496, "bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "f138c736593e168bbb54a7d9c1eb854decdbd7b7b721d74f38877d172f6ed251", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 222736, "bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "d4d78dccca24e96674512f431eb0826605456b69e13e8aa2b483d33345b28e58", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -24423,10 +24146,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -24463,7 +24186,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 1 @@ -24471,7 +24193,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -24486,7 +24207,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -24533,10 +24253,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 186960, "bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "67dce85aaa44edc30325e4fe41cc9d46105009dae194e71598a0124807cc60da", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 187024, "bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "7846842f19e8a0f8fc85ad57196c0761030f839e6f4b1b19025e3ae00d11d952", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -24551,10 +24272,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -24591,7 +24312,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -24599,7 +24319,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -24614,7 +24333,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -24661,10 +24379,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 186720, "bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "89263c587e1c701af1696850bf3a407f331e129e8a75c24c9a93a6c3fcb6fddd", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 186896, "bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "25b6c7f2ac8d2d44a54b24f8817831cfeaba1e5d4b33ea2b4f1090e2879403a6", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -24679,10 +24398,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -24719,7 +24438,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -24727,7 +24445,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -24742,7 +24459,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -24789,10 +24505,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 186960, "bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "f203cbe891f346ef9f1f825ea9d72abab66b949c1ff3d5613cfefa753e313ffd", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 187024, "bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "809453ee91c1e234f1549a4f9135f2486bddb093e6bdf840fd2bc35593efacd3", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -24807,10 +24524,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -24847,7 +24564,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -24855,7 +24571,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -24870,7 +24585,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -24917,10 +24631,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 186720, "bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "a650eaaf1e869969aecf899fc1781e33d06803e1d51be63d5a2aa715e1157695", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 186896, "bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "f8dd75e6b16eec6ff9fcf939dd7e081a416900d3b8d30b027ef0ea01dfe078a6", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -24935,10 +24650,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -24975,7 +24690,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -24983,7 +24697,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -24998,7 +24711,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -25045,10 +24757,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 222672, "bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "f996cef4b275ce5eaf82deceab5d26f4a73fc53784f50bb02eaf8c39da162a56", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 222864, "bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "52e54012399a798c495fe2db4799bbe0e99027757fb17ff906ddfd5fdeea3eee", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -25063,10 +24776,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -25103,7 +24816,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 2 @@ -25111,7 +24823,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -25126,7 +24837,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -25173,10 +24883,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 222432, "bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "3bc8f07679eea92ad50ae9b6d4cfe1aac73d5c0f3147f8b964cd6c70434fc0d6", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 222480, "bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "b8c4647ce371c232b919af211c7e9382d154f9c1623a67cb679fadf837d31384", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -25191,10 +24902,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -25231,7 +24942,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 1 @@ -25239,7 +24949,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -25254,7 +24963,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -25301,10 +25009,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 222672, "bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "9300d647defeb6412f4753b3dc46c347d362fb779f9d919b455de6ef298716b1", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 222864, "bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "967becb2177e400001a46507a544c8b1154570c5bd50e6fbc721177b819737fc", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -25319,10 +25028,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -25359,7 +25068,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 2 @@ -25367,7 +25075,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -25382,7 +25089,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -25429,10 +25135,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 222432, "bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "e53947d20567e1aafd80ebb6793ab4b8e06efdb1b49be867dc4dc24b55fa8c08", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 222480, "bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "4a2bd8ba8e59a0aeb406c3571021e80495b1b788a1320b1ca8d94a5df34c106b", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -25447,10 +25154,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -25487,7 +25194,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 1 @@ -25495,7 +25201,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -25510,7 +25215,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -25557,10 +25261,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x128x128_s7_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x128x128_s7_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 203592, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x128x128_s7_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "3ccabcf11541beee2ee46713053cf103a31a607a531aef6df9872ae0fae58f77", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x128x128_s7_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x128x128_s7_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 203672, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x128x128_s7_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "070afe2bbd0b267f1cae24aa8ed2f7884c9ba230a24edde8371b6cf557a2ce04", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -25575,10 +25280,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -25615,7 +25320,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 168 , /* mNumRegsPerThreadNonEpilogueWarp */ 96 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 7 , /* mNumStagesB */ 7 , /* mNumStagesMma */ 2 @@ -25623,7 +25327,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 7 , /* mNumStagesSmemSfB */ 7 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 7 , /* mNumStagesTmemSfB */ 7 , /* mNumStagesWorkId */ 3 @@ -25638,7 +25341,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -25685,10 +25387,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x128x128u2_s7_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x128x128u2_s7_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 203592, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x128x128u2_s7_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "fc0981f0041030b03b719fd9bcdc1d716f9aa0a969df8cf2231e85614ae7e71e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x128x128u2_s7_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x128x128u2_s7_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 203672, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x128x128u2_s7_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "4b069cc38bbe35c38d7daf5e566a8b54475879b34aa1af6e8971dd5ffac212fa", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -25703,10 +25406,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -25743,7 +25446,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 168 , /* mNumRegsPerThreadNonEpilogueWarp */ 96 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 7 , /* mNumStagesB */ 7 , /* mNumStagesMma */ 2 @@ -25751,7 +25453,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 7 , /* mNumStagesSmemSfB */ 7 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 7 , /* mNumStagesTmemSfB */ 7 , /* mNumStagesWorkId */ 3 @@ -25766,7 +25467,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -25813,10 +25513,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x128x256_s4_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x128x256_s4_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 227928, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x128x256_s4_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "ed7c7d1642d345d2b3243080a186440704863b5cd141addd44dbfec1e3c01a8e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x128x256_s4_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x128x256_s4_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 227992, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x128x256_s4_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "8e75d79fff0a703862b424845fd73efe70a9841592eca697da98a0a4e795da2b", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -25831,10 +25532,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -25871,7 +25572,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 168 , /* mNumRegsPerThreadNonEpilogueWarp */ 96 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 2 @@ -25879,7 +25579,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -25894,7 +25593,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -25941,10 +25639,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x128x256u2_s4_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x128x256u2_s4_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 227928, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x128x256u2_s4_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "63772c766be2021661a78ad11dd13050bd160e21f260fb47ce5081341ef2439f", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x128x256u2_s4_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x128x256u2_s4_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 227992, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x128x256u2_s4_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 384, "ce75f0426d8fdd04e3e84a5054e7272f9fc1c5e16f3ac14d67031543d0b46306", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -25959,10 +25658,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -25999,7 +25698,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 168 , /* mNumRegsPerThreadNonEpilogueWarp */ 96 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 2 @@ -26007,7 +25705,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -26022,7 +25719,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -26069,10 +25765,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 199408, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "0287abe28420f921a00f56c6d035c654eb38aa9e02b4e00386ae72856c6fe809", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 199568, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "766084e4b4325e0d531c9980418b0c23bd9a3e0c8b9e5833840ec7e01c240498", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -26087,10 +25784,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -26127,7 +25824,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -26135,7 +25831,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -26150,7 +25845,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -26197,10 +25891,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 199168, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "123437e4ff210deb16d1fdce792915a35a40c517a0def80b42f1dccd4466ba14", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 199184, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "0784dbcc594d03700e334cd29f3a8f1e683bc262650e887b77818b63379d3340", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -26215,10 +25910,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -26255,7 +25950,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -26263,7 +25957,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -26278,7 +25971,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -26325,138 +26017,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 - }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 225112, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "164170cf97c696c38017ea6cc11618563c46308e5c4bc148ed1104c97fe5eb2f", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) -, /* mBiasDtype */ trtllm::gen::Dtype(1056776) -, /* mBiasType */ gemm::BiasType(1) -, /* mBlockK */ -1 -, /* mClcFastDrain */ 1 -, /* mClusterDimX */ 2 -, /* mClusterDimY */ 1 -, /* mClusterDimZ */ 1 -, /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) -, /* mDtypeAcc */ trtllm::gen::Dtype(1056776) -, /* mDtypeA */ trtllm::gen::Dtype(17826828) -, /* mDtypeB */ trtllm::gen::Dtype(17827853) -, /* mDtypeC */ trtllm::gen::Dtype(1052672) -, /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) -, /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) -, /* mEltwiseActType */ gemm::EltwiseActType(0) -, /* mEnablesEarlyExit */ 1 -, /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 -, /* mEpilogueLdtmDps */ 16 -, /* mEpilogueLdtmBits */ 256 -, /* mEpilogueTileM */ 128 -, /* mEpilogueTileN */ 16 -, /* mFallbackClusterDimX */ 1 -, /* mFallbackClusterDimY */ 1 -, /* mFallbackClusterDimZ */ 1 -, /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) -, /* mFuseLoadSfTask */ 0 -, /* mFuseUtccpWithUtcmma */ 0 -, /* mGridTriggerSecondaryA */ 0 -, /* mGridTriggerSecondaryB */ 1 -, /* mGridWaitForPrimaryEarlyExit */ 1 -, /* mGridWaitForPrimaryA */ 0 -, /* mGridWaitForPrimaryB */ 1 -, /* mHoistLoadTaskInit */ 1 -, /* mHoistMmaTaskTryWaits */ 0 -, /* mK */ 256 -, /* mKernelTraits */ {} -, /* mLayoutA */ gemm::MatrixLayout(0) -, /* mLayoutB */ gemm::MatrixLayout(0) -, /* mM */ 256 -, /* mMmaK */ 32 -, /* mMmaKind */ trtllm::gen::MmaKind(5) -, /* mMmaM */ 256 -, /* mMmaN */ 16 -, /* mMmaTileK */ 256 -, /* mMockAllReduce */ 0 -, /* mN */ 256 -, /* mNumEpilogueWarps */ 4 -, /* mNumRegsCastAWarps */ 0 -, /* mNumRegsCopySfLdsSttm */ 0 -, /* mNumRegsCopySparsityInfo */ 0 -, /* mNumRegsPerThreadEpilogueWarp */ 0 -, /* mNumRegsPerThreadNonEpilogueWarp */ 0 -, /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 -, /* mNumStagesA */ 6 -, /* mNumStagesB */ 6 -, /* mNumStagesMma */ 2 -, /* mNumStagesMmaWithinWorkTile */ 1 -, /* mNumStagesMmaAcrossWorkTile */ 2 -, /* mNumStagesSmemSfA */ 6 -, /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 -, /* mNumStagesTmemSfA */ 6 -, /* mNumStagesTmemSfB */ 6 -, /* mNumStagesWorkId */ 3 -, /* mOutputDebugTensors */ 0 -, /* mPatchF2fp */ 0 -, /* mPerTokenSfDtype */ trtllm::gen::Dtype(1052672) -, /* mSchedHostTask */ gemm::SchedHostTask(0) -, /* mSfBlockSizeA */ 32 -, /* mSfBlockSizeB */ 32 -, /* mSfBlockSizeC */ -1 -, /* mSfLayoutA */ trtllm::gen::SfLayout(3) -, /* mSfLayoutB */ trtllm::gen::SfLayout(1) -, /* mSfLayoutC */ trtllm::gen::SfLayout(1) -, /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 -, /* mSparsityA */ trtllm::gen::Sparsity(0) -, /* mSplitK */ gemm::SplitK(0) -, /* mTileK */ 256 -, /* mTileM */ 128 -, /* mTileN */ 16 -, /* mTileScheduler */ gemm::TileScheduler(1) -, /* mTransposeMmaOutput */ 1 -, /* mUseCustomizedMma3xNvFp4 */ 0 -, /* mUseCustomMmaSchedule */ 1 -, /* mUseDeepSeekFp8 */ 0 -, /* mUseFlexibleClusterDims */ 0 -, /* mUseHoistTryWaitForCustomMmaSchedule */ 0 -, /* mUseMaxTmemOverlap */ 0 -, /* mUsePerTokenSfA */ 0 -, /* mUsePerTokenSfB */ 0 -, /* mUseShuffledMatrix */ 1 -, /* mUseTmaStore */ 1 -, /* mUseTwoTmaLoadWarps */ 1 -, /* mUseTwoMmaWarps */ 0 -, /* mUseUnrollLoop2xForMma */ 0 -, /* mValidM */ 256 -, /* mValidN */ 256 -, /* mValidK */ 256 -, /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(0) -, /* mClampBeforeAct */ 1 -, /* mBatchedM */ {} -, /* mBatchedN */ {} -, /* mBatchMode */ batchedGemm::BatchedGemmOptions::BatchMode(1) -, /* mBatchStrideInTokens */ -1 -, /* mFusedAct */ 0 -, /* mGridWaitForPrimaryRouting */ 0 -, /* mIsStaticBatch */ 0 -, /* mIsUniformNumTokensPerBatch */ 0 -, /* mNumBatches */ 128 -, /* mNumRegsPerThreadLoadA */ 0 -, /* mNumRegsPerThreadLoadB */ 0 -, /* mNumRegsPerThreadLoadSfA */ 0 -, /* mNumRegsPerThreadLoadSfB */ 0 -, /* mNumTokens */ 2 -, /* mNumWarpsLoadA */ 0 -, /* mNumWarpsLoadB */ 0 -, /* mNumWarpsLoadSfA */ 0 -, /* mNumWarpsLoadSfB */ 0 -, /* mRouteImpl */ batchedGemm::RouteImpl(0) -, /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} +, /* mUseCMultiCast */ 0 , /* mUseTmaOobOpt */ 1 -, /* mPrefetchB */ 0 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 224872, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "4d3b9fb39728655bc83dc16cfb936af5d4431740c62f02f1343c07eb9b010752", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 225176, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "5305503fd91f80b270f9cb379ad50a3d34c0dc6dd3f23be9fc699b42be441531", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -26471,10 +26036,136 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) +, /* mEltwiseActType */ gemm::EltwiseActType(0) +, /* mEnablesEarlyExit */ 1 +, /* mEnablesDelayedEarlyExit */ 0 +, /* mEpilogueLdtmDps */ 16 +, /* mEpilogueLdtmBits */ 256 +, /* mEpilogueTileM */ 128 +, /* mEpilogueTileN */ 16 +, /* mFallbackClusterDimX */ 1 +, /* mFallbackClusterDimY */ 1 +, /* mFallbackClusterDimZ */ 1 +, /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) +, /* mFuseLoadSfTask */ 0 +, /* mFuseUtccpWithUtcmma */ 0 +, /* mGridTriggerSecondaryA */ 0 +, /* mGridTriggerSecondaryB */ 1 +, /* mGridWaitForPrimaryEarlyExit */ 1 +, /* mGridWaitForPrimaryA */ 0 +, /* mGridWaitForPrimaryB */ 1 +, /* mHoistLoadTaskInit */ 1 +, /* mHoistMmaTaskTryWaits */ 0 +, /* mK */ 256 +, /* mKernelTraits */ {} +, /* mLayoutA */ gemm::MatrixLayout(0) +, /* mLayoutB */ gemm::MatrixLayout(0) +, /* mM */ 256 +, /* mMmaK */ 32 +, /* mMmaKind */ trtllm::gen::MmaKind(5) +, /* mMmaM */ 256 +, /* mMmaN */ 16 +, /* mMmaTileK */ 256 +, /* mMockAllReduce */ 0 +, /* mN */ 256 +, /* mNumEpilogueWarps */ 4 +, /* mNumRegsCastAWarps */ 0 +, /* mNumRegsCopySfLdsSttm */ 0 +, /* mNumRegsCopySparsityInfo */ 0 +, /* mNumRegsPerThreadEpilogueWarp */ 0 +, /* mNumRegsPerThreadNonEpilogueWarp */ 0 +, /* mNumSlicesForSplitK */ 1 +, /* mNumStagesA */ 6 +, /* mNumStagesB */ 6 +, /* mNumStagesMma */ 2 +, /* mNumStagesMmaWithinWorkTile */ 1 +, /* mNumStagesMmaAcrossWorkTile */ 2 +, /* mNumStagesSmemSfA */ 6 +, /* mNumStagesSmemSfB */ 6 +, /* mNumStagesTmemSfA */ 6 +, /* mNumStagesTmemSfB */ 6 +, /* mNumStagesWorkId */ 3 +, /* mOutputDebugTensors */ 0 +, /* mPatchF2fp */ 0 +, /* mPerTokenSfDtype */ trtllm::gen::Dtype(1052672) +, /* mSchedHostTask */ gemm::SchedHostTask(0) +, /* mSfBlockSizeA */ 32 +, /* mSfBlockSizeB */ 32 +, /* mSfBlockSizeC */ -1 +, /* mSfLayoutA */ trtllm::gen::SfLayout(3) +, /* mSfLayoutB */ trtllm::gen::SfLayout(1) +, /* mSfLayoutC */ trtllm::gen::SfLayout(1) +, /* mSfReshapeFactor */ 1 +, /* mSparsityA */ trtllm::gen::Sparsity(0) +, /* mSplitK */ gemm::SplitK(0) +, /* mTileK */ 256 +, /* mTileM */ 128 +, /* mTileN */ 16 +, /* mTileScheduler */ gemm::TileScheduler(1) +, /* mTransposeMmaOutput */ 1 +, /* mUseCustomizedMma3xNvFp4 */ 0 +, /* mUseCustomMmaSchedule */ 1 +, /* mUseDeepSeekFp8 */ 0 +, /* mUseFlexibleClusterDims */ 0 +, /* mUseHoistTryWaitForCustomMmaSchedule */ 0 +, /* mUseMaxTmemOverlap */ 0 +, /* mUsePerTokenSfA */ 0 +, /* mUsePerTokenSfB */ 0 +, /* mUseShuffledMatrix */ 1 +, /* mUseTmaStore */ 1 +, /* mUseTwoTmaLoadWarps */ 1 +, /* mUseTwoMmaWarps */ 0 +, /* mUseUnrollLoop2xForMma */ 0 +, /* mValidM */ 256 +, /* mValidN */ 256 +, /* mValidK */ 256 +, /* mWorldSize */ 1 +, /* mActType */ gemmGatedAct::ActType(0) +, /* mClampBeforeAct */ 1 +, /* mBatchedM */ {} +, /* mBatchedN */ {} +, /* mBatchMode */ batchedGemm::BatchedGemmOptions::BatchMode(1) +, /* mBatchStrideInTokens */ -1 +, /* mFusedAct */ 0 +, /* mGridWaitForPrimaryRouting */ 0 +, /* mIsStaticBatch */ 0 +, /* mIsUniformNumTokensPerBatch */ 0 +, /* mNumBatches */ 128 +, /* mNumRegsPerThreadLoadA */ 0 +, /* mNumRegsPerThreadLoadB */ 0 +, /* mNumRegsPerThreadLoadSfA */ 0 +, /* mNumRegsPerThreadLoadSfB */ 0 +, /* mNumTokens */ 2 +, /* mNumWarpsLoadA */ 0 +, /* mNumWarpsLoadB */ 0 +, /* mNumWarpsLoadSfA */ 0 +, /* mNumWarpsLoadSfB */ 0 +, /* mRouteImpl */ batchedGemm::RouteImpl(0) +, /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} +, /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 + }, gemm::SmVersion::Sm100f}, +{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 225048, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "4240c86f6a715de3b5a7449530faed7b247890b7772c625d4914dc6dafb07cde", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +, /* mBiasDtype */ trtllm::gen::Dtype(1056776) +, /* mBiasType */ gemm::BiasType(1) +, /* mBlockK */ -1 +, /* mClcFastDrain */ 1 +, /* mClusterDimX */ 2 +, /* mClusterDimY */ 1 +, /* mClusterDimZ */ 1 +, /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) +, /* mDtypeAcc */ trtllm::gen::Dtype(1056776) +, /* mDtypeA */ trtllm::gen::Dtype(17826828) +, /* mDtypeB */ trtllm::gen::Dtype(17827853) +, /* mDtypeC */ trtllm::gen::Dtype(1052672) +, /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) +, /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -26511,7 +26202,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 1 @@ -26519,7 +26209,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -26534,7 +26223,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -26581,10 +26269,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 199408, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "44c4aaaad67ef26130eb7b694e958780cae4fb98cdf111e909e27b31683f7f2b", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 199568, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "2741aefca2063a5260988819b60b3120b4c60cc3fae9694752f3308ac4f06843", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -26599,10 +26288,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -26639,7 +26328,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -26647,7 +26335,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -26662,7 +26349,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -26709,10 +26395,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 199168, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "ba532300dcb14c4822323639c8b7f006bd190fdfa1cebe88f0e2414266c97d6d", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 199184, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "7a6f797718e4e568d6fae1b968d7145c084409d7d177276993b8d122c4ab69e7", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -26727,10 +26414,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -26767,7 +26454,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -26775,7 +26461,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -26790,7 +26475,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -26837,10 +26521,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 225112, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "8529331ff757ce04698f4696cdb9dd7778f918a0fb44fa530f3039ec52c18d35", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 225176, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "808ab62b00db1333027c7318b4af19019d27102cccd338d72003c444f177f886", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -26855,10 +26540,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -26895,7 +26580,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -26903,7 +26587,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -26918,7 +26601,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -26965,10 +26647,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 224872, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "b85a2ad6fd4244914849cbeac4e98488960b8884b5011dcf89b2e60645191d3c", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 225048, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "8347d3631183c4953125c91dfb536629e5185faf2a8949e4632a314b20a73009", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -26983,10 +26666,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -27023,7 +26706,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 1 @@ -27031,7 +26713,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -27046,7 +26727,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -27093,10 +26773,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x256x128_s5x5x5x5x1x1_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x256x128_s5x5x5x5x1x1_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_rgTma_clmp_dynB_sm100f_cubin_len, 197208, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x256x128_s5x5x5x5x1x1_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_rgTma_clmp_dynB_sm100f", 384, "8476871fe798fb3b1b298b49645bd798191ee40475d084d9dfb335809119c941", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x256x128_s5x5x5x5x1x1_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x256x128_s5x5x5x5x1x1_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_rgTma_clmp_dynB_sm100f_cubin_len, 197272, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x256x128_s5x5x5x5x1x1_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_rgTma_clmp_dynB_sm100f", 384, "6a69ef3d32e12702fff7f7cc59f3f351db5c75ad041be6250c88769d505c47fd", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -27111,10 +26792,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -27151,7 +26832,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 144 , /* mNumRegsPerThreadNonEpilogueWarp */ 96 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -27159,7 +26839,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 1 , /* mNumStagesTmemSfB */ 1 , /* mNumStagesWorkId */ 3 @@ -27174,7 +26853,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -27221,10 +26899,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 225008, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "c53594c1b59c1c720de312336dfa6b689cc1fa4d290428f1fca1cc7baaf136f1", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 225168, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "b82e9d73de3da8b0637bf2d9b65352ec7f6461143b6e27e84b5596ab9c71f628", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -27239,10 +26918,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -27279,7 +26958,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -27287,7 +26965,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -27302,7 +26979,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -27349,10 +27025,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 224768, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "54cc6ebfd299097c9f4c8d0a134fff46385a4d3d4e29ddaf479a5e61d7a88903", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 224784, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "0073994dbb06b866e0b84394c124df8b81e54958143f966a3b701aeeaa5e65a2", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -27367,10 +27044,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -27407,7 +27084,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -27415,7 +27091,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -27430,7 +27105,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -27477,10 +27151,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 204536, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "47c01b8b1b3d11373a4b6a4b6d5df79c137c6db5fcdd012d1c00e5d2ccf1c9ec", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 204696, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "2ab3f64cdfafe899103ee5fcba1ec7de6e07987dcd504a1e072aaa9258b07ae9", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -27495,10 +27170,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -27535,7 +27210,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -27543,7 +27217,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -27558,7 +27231,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -27605,10 +27277,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 204296, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "b8320eea147ae1a9f99a5d6a0bdb72b8c2782b3d0689e1694dfdc47d0b973983", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 204312, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "2247acbfa1966fd0e317aad06a76724f408acd8bab63794a65884eaf3689964c", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -27623,10 +27296,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -27663,7 +27336,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -27671,7 +27343,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -27686,7 +27357,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -27733,10 +27403,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 225008, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "9af9fa56e8615b649189fa56e804acc2036cefcafd11b6643a8ca62f450d5878", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 225168, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "42af25cc67a24d4eef5558b3a45dd1eeaaaf07d1e9fd6e19a1c9d2b32bc844c7", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -27751,10 +27422,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -27791,7 +27462,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -27799,7 +27469,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -27814,7 +27483,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -27861,10 +27529,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 224768, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "1c224d95a1462414f0c1ce9b429c8ce6e02422e51c24f5b878be58a10433823a", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 224784, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "05b41b78aca20e1e40716ccd9194e796331c0ee2e59c869db4f58ff7596eb96e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -27879,10 +27548,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -27919,7 +27588,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -27927,7 +27595,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -27942,7 +27609,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -27989,10 +27655,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 204536, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "4131f97371a1d7f6f789242626d3629673c9b347c47c226529899f6c632d6999", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 204696, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "037254fa03d539fea3355e2d65f2980e1d8c41d1439c50c999e7cb68f121adc1", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -28007,10 +27674,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -28047,7 +27714,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -28055,7 +27721,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -28070,7 +27735,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -28117,10 +27781,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 204296, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "464b0eece9c557c7f75590211de79732bb54bca7bc726f464bfd7aee6a6583e9", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 204312, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "6d200fd364a2aa515a18e3fdc6e1a533530cee1e96320a459170f9ca2437de77", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -28135,10 +27800,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -28175,7 +27840,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -28183,7 +27847,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -28198,7 +27861,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -28245,10 +27907,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x64x128_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x64x128_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 171960, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x64x128_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "ef223e1a41c20627ebaeaaca3509929c51da99618357fe167ddf8d3da4d285c4", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x64x128_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x64x128_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 172184, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x64x128_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "c82fd85a8857028b6ca5900f23b5bd02e3e78332bbb86f5f0df88c33772087be", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -28263,10 +27926,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -28303,7 +27966,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 168 , /* mNumRegsPerThreadNonEpilogueWarp */ 96 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 7 , /* mNumStagesB */ 7 , /* mNumStagesMma */ 2 @@ -28311,7 +27973,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 7 , /* mNumStagesSmemSfB */ 7 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 7 , /* mNumStagesTmemSfB */ 7 , /* mNumStagesWorkId */ 3 @@ -28326,7 +27987,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -28373,10 +28033,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x64x128u2_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x64x128u2_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 171960, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x64x128u2_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "0227515a28512aa3fdd2cff090d129873da845e2a77e9978a2f8cbe0e0984dd9", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x64x128u2_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x64x128u2_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 172184, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x64x128u2_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "40fba5048ce64389188520de43993ae8154aee079bfa7265212ad3f06f4bf9fa", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -28391,10 +28052,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -28431,7 +28092,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 168 , /* mNumRegsPerThreadNonEpilogueWarp */ 96 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 7 , /* mNumStagesB */ 7 , /* mNumStagesMma */ 2 @@ -28439,7 +28099,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 7 , /* mNumStagesSmemSfB */ 7 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 7 , /* mNumStagesTmemSfB */ 7 , /* mNumStagesWorkId */ 3 @@ -28454,7 +28113,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -28501,10 +28159,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x64x256_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x64x256_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 192152, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x64x256_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "e1264836e63e7b888c46757a0b61308893e49d985f1b43cfe1cc060f31c44e6c", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x64x256_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x64x256_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 192152, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x64x256_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "ffc9e87aa41ba330df7fe046d78fe2b6491718aab7e07335cd7da4929c109992", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -28519,10 +28178,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -28559,7 +28218,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 168 , /* mNumRegsPerThreadNonEpilogueWarp */ 96 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 2 @@ -28567,7 +28225,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -28582,7 +28239,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -28629,10 +28285,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x64x256u2_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x64x256u2_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 192152, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x64x256u2_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "9c0cea6766f871c3978b6919c8c6fae1ba599ecc1d9a83b5964fed3214a9ff66", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x64x256u2_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x64x256u2_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 192152, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x64x256u2_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "584096941de4322c66db15e04b305d540d96eb561421a9a7821c79ff3e531a3c", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -28647,10 +28304,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -28687,7 +28344,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 168 , /* mNumRegsPerThreadNonEpilogueWarp */ 96 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 2 @@ -28695,7 +28351,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -28710,7 +28365,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -28757,10 +28411,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 223056, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "0b894ae6ca6dc39334a541c6358e623cffc22eecabbf0d5cffb477312e59bb7d", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 223120, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "089ef7628c51440444d43c32289fb25012e4f148addc1321d75c7aa7b97bcbe1", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -28775,10 +28430,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -28815,7 +28470,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -28823,7 +28477,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -28838,7 +28491,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -28885,10 +28537,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 222816, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "c2e85c07b624ed334245a465388554786e697aef80f68b92f9fd7d3ab35ab7d9", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 222992, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "c77fecd8c9fb42b38714597c4912bc2dfca702067f0e23af5107f4fb8fc3c8cb", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -28903,10 +28556,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -28943,7 +28596,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 1 @@ -28951,7 +28603,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -28966,7 +28617,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -29013,10 +28663,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 229208, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "02134017fb2ecded25370f7f66d081c6fc6f9cf5eef2c398241da2f82bd4ff4e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 229272, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "85f0e4e97a768cdb9334e11806946f8c44d401ce1507952f848637fd2e5a8038", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -29031,10 +28682,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -29071,7 +28722,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 2 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -29079,7 +28729,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -29094,7 +28743,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(2) , /* mTileK */ 256 @@ -29141,10 +28789,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 220776, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "0cfe31ef9b5c1f9fcc2be3834f975eff19493108bf72f1e78e098a6627aba907", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 220952, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "b3d6f9ccdd1374455ceb9c9b15dd32783eb0d13dc4a71548a44e3bc04b12f144", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -29159,10 +28808,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -29199,7 +28848,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 2 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 1 @@ -29207,7 +28855,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -29222,7 +28869,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(2) , /* mTileK */ 256 @@ -29269,10 +28915,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 223056, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "9b3fd2faa7b1c9f960ae812199de80e4f1b07519f9fbeea174bdce6926992785", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 223120, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "c29b7e964315c40deeec4d62b1cc2e2686b8717353b0dc47dfa4100a1e7b4014", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -29287,10 +28934,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -29327,7 +28974,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -29335,7 +28981,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -29350,7 +28995,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -29397,10 +29041,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 222816, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "178b7d81a558735162745ff383a05918b31c28ba4c96484542a83fb988570d07", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 222992, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "38fd2e9cf9797c59ca9c4001e5c2a384a19814e3da0d68bb8b90c660d0b4750e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -29415,10 +29060,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -29455,7 +29100,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 1 @@ -29463,7 +29107,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -29478,7 +29121,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -29525,10 +29167,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 229208, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "611538b53fdb0360fdc4052bc99c71e538d9df07ad3ebbbe4ca618f85bf8232d", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 229272, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "b22a179046c59f62468931ab30e0719d25c027156c326058e532caafba6617de", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -29543,10 +29186,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -29583,7 +29226,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 2 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -29591,7 +29233,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -29606,7 +29247,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(2) , /* mTileK */ 256 @@ -29653,10 +29293,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 220776, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "060f6533e5b191c716a634d9af2dd54bd3b77f45d554d6e0f6553d03e6e910e8", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 220952, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "b1fc27759fc7df0d3f32e30c871ab5bc4707b2b835184a4a1dcde9b80618396a", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -29671,10 +29312,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -29711,7 +29352,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 2 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 1 @@ -29719,7 +29359,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -29734,7 +29373,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(2) , /* mTileK */ 256 @@ -29781,10 +29419,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 222768, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "b2360af57008a016e4f0476183b9a5bf4a867089eeda513871690aa1ae30d565", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 222864, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "171c7bcff2c3f1157619e80fd29932b365028525b453accb5d7f77cde17798c9", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -29799,10 +29438,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -29839,7 +29478,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 2 @@ -29847,7 +29485,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -29862,7 +29499,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -29909,10 +29545,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 222528, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "b1ee8013de810f9a23c5740c6369e115d8ce93e274969bf195cce5b1766a9be6", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 222736, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "773fc7e3ece7872af040624075f86ea2a5fefa1a5d0820886cbbca62d82400a0", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -29927,10 +29564,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -29967,7 +29604,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 1 @@ -29975,7 +29611,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -29990,7 +29625,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -30037,10 +29671,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 228920, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "6d2e198928d0bf309bd1aa5a19ba0851673ac53e5ec8abb4d99138d48e014832", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 229016, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "a1cfcb3d01af4ae7e6cb78712a7bdbef6f3f7e44c5beb58b9922befe3dbaabd2", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -30055,10 +29690,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -30095,7 +29730,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 2 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 2 @@ -30103,7 +29737,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -30118,7 +29751,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(2) , /* mTileK */ 512 @@ -30165,10 +29797,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 220488, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "a24347e9ed7589cef35fb543230532a20064e82d4c01ec72e0fdec407ca87679", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 220696, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "d95a4232c6682463b883dba0480c7093cf0d51de983517eb0c6195bff509ed99", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -30183,10 +29816,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -30223,7 +29856,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 2 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 1 @@ -30231,7 +29863,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -30246,7 +29877,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(2) , /* mTileK */ 512 @@ -30293,10 +29923,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 222768, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "6a7ed625ae56787ad284f575da62464bc7d235db8b0a8614c84ef62963b318dd", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 222864, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "38e9c7c6f32c97afe32a742500ba5d42de0bafda5110335aae5b56afdd72a626", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -30311,10 +29942,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -30351,7 +29982,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 2 @@ -30359,7 +29989,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -30374,7 +30003,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -30421,10 +30049,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 222528, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "4351f37ca4acfe34d11333c7eac845357de135dbdd7144b71d99f5bb64f0a125", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 222736, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "ef880a9e4748b5c2d920f5212abba6a1e2b2552a62a6a6ea66e1933d0dc722bc", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -30439,10 +30068,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -30479,7 +30108,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 1 @@ -30487,7 +30115,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -30502,7 +30129,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -30549,10 +30175,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 228920, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "126058c35dc61b1d3484468cb9117f9a952cdd6d03170dc69b92d7ee1b05ae0c", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 229016, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "d60e6c1591fba4d6f36b85b067a3ff2df4e2f3f9d1b3f5a9ab15c3b79b33c613", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -30567,10 +30194,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -30607,7 +30234,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 2 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 2 @@ -30615,7 +30241,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -30630,7 +30255,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(2) , /* mTileK */ 512 @@ -30677,10 +30301,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 220488, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "d9dfd369af0a7d10be79c16e070bbcf39389c1f5c98b7ed2b0a113de2337db9d", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin, Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f_cubin_len, 220696, "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f", 512, "b860d3462e4dd862078eb62bb2f86f76459c65db8cf77ccd07d3408a729fc291", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -30695,10 +30320,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -30735,7 +30360,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 2 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 1 @@ -30743,7 +30367,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -30758,7 +30381,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(2) , /* mTileK */ 512 @@ -30805,10 +30427,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 180984, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f", 512, "48a34e92bee32094be67d617e1637804514ae458f7cf2f692d7d33ea1d480665", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 181144, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f", 512, "0800fc4f4c9fb56d1876a4065af3b7311ef88970a9c132737b68af7d20a5eafe", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -30823,10 +30446,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -30863,7 +30486,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -30871,7 +30493,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -30886,7 +30507,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -30933,10 +30553,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 180984, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "a5cedf64a1faecb802d0f6f604a3de95d969f709d2405060f0fcd507107632e2", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 181144, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "ea04344109bd50d6c9d53011c692f104aa619b2987275bbf49b1cf17e7e072b3", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -30951,10 +30572,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -30991,7 +30612,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -30999,7 +30619,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -31014,7 +30633,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -31061,10 +30679,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 180984, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 512, "8062990052fcbadc776aed3b1972f9d46bb6b5e071c91965c158255b7395ec53", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 181144, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 512, "c39768a2366c66f142773fe526a4c93992313590a4cc3fc7a8dc78f35957c157", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -31079,10 +30698,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -31119,7 +30738,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -31127,7 +30745,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -31142,7 +30759,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -31167,7 +30783,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 256 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -31189,10 +30805,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 180984, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 512, "480ec18fb86c63ce66f4b66f7753d8fcb5307a2baaab0fb0ec11227f62a8cb9f", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 181144, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 512, "d5aba136f449f01ceed6b1e32e73f25801972c52091d59b1c34faa3f9cf71460", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -31207,10 +30824,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -31247,7 +30864,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -31255,7 +30871,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -31270,7 +30885,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -31295,7 +30909,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 256 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -31317,10 +30931,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_geGlu_lsfaW2_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_geGlu_lsfaW2_dynB_sm100f_cubin_len, 180984, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_geGlu_lsfaW2_dynB_sm100f", 640, "3a8e828aa39ba002d12ab05f513174df96b74637fca3e4e036e408994e9012cf", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_geGlu_lsfaW2_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_geGlu_lsfaW2_dynB_sm100f_cubin_len, 181144, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_geGlu_lsfaW2_dynB_sm100f", 640, "80980e42090347b6027b71115b985f08349e6f99ed7ad1de4b59a253b3fa11b5", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(2) , /* mBlockK */ -1 @@ -31335,10 +30950,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 32 , /* mEpilogueLdtmBits */ 32 , /* mEpilogueTileM */ 128 @@ -31375,7 +30990,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -31383,7 +30997,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -31398,7 +31011,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -31445,10 +31057,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_geGlu_lsfaW4_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_geGlu_lsfaW4_dynB_sm100f_cubin_len, 180984, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_geGlu_lsfaW4_dynB_sm100f", 768, "28d1d153ab330957313c5631a3acf089ab973d8fab92275dac6734bc51604cca", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_geGlu_lsfaW4_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_geGlu_lsfaW4_dynB_sm100f_cubin_len, 181144, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_geGlu_lsfaW4_dynB_sm100f", 768, "7af3bf3259318bf008181d564333c637138879849f48cffc37b372fe7b508696", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(2) , /* mBlockK */ -1 @@ -31463,10 +31076,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 32 , /* mEpilogueLdtmBits */ 32 , /* mEpilogueTileM */ 128 @@ -31503,7 +31116,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -31511,7 +31123,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -31526,7 +31137,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -31573,10 +31183,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_swiGlu_lsfaW2_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_swiGlu_lsfaW2_dynB_sm100f_cubin_len, 180984, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_swiGlu_lsfaW2_dynB_sm100f", 640, "d9178a6c84fc65afa2f00e64a91570e73406dd2764ae6a605586d10e3763c524", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_swiGlu_lsfaW2_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_swiGlu_lsfaW2_dynB_sm100f_cubin_len, 181144, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_swiGlu_lsfaW2_dynB_sm100f", 640, "88adedea3da89a8cb12b299ab09d83e5b2f6fc13732a6f3040f8e7e057b1571e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(2) , /* mBlockK */ -1 @@ -31591,10 +31202,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 32 , /* mEpilogueLdtmBits */ 32 , /* mEpilogueTileM */ 128 @@ -31631,7 +31242,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -31639,7 +31249,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -31654,7 +31263,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -31701,10 +31309,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_swiGlu_lsfaW4_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_swiGlu_lsfaW4_dynB_sm100f_cubin_len, 180984, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_swiGlu_lsfaW4_dynB_sm100f", 768, "f3428a247f3cbec66ab05814c7856b5225fbd246f90e587b83526cef43a1bb7b", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_swiGlu_lsfaW4_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_swiGlu_lsfaW4_dynB_sm100f_cubin_len, 181144, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_swiGlu_lsfaW4_dynB_sm100f", 768, "f14cf6b79fdd401509237e8ecf6106bc3f588debe831205d096d5ebb95f30759", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(2) , /* mBlockK */ -1 @@ -31719,10 +31328,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 32 , /* mEpilogueLdtmBits */ 32 , /* mEpilogueTileM */ 128 @@ -31759,7 +31368,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -31767,7 +31375,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -31782,7 +31389,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -31829,10 +31435,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_relu2_bM_tma_ldgstsSf_rgTma_clmp_lsfaW2_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_relu2_bM_tma_ldgstsSf_rgTma_clmp_lsfaW2_dynB_sm100f_cubin_len, 180984, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_relu2_bM_tma_ldgstsSf_rgTma_clmp_lsfaW2_dynB_sm100f", 640, "6204b1a7be36d502d35c6ec9455c9bf0a321600b3ecc8c665844d254867baeff", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_relu2_bM_tma_ldgstsSf_rgTma_clmp_lsfaW2_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_relu2_bM_tma_ldgstsSf_rgTma_clmp_lsfaW2_dynB_sm100f_cubin_len, 181144, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_relu2_bM_tma_ldgstsSf_rgTma_clmp_lsfaW2_dynB_sm100f", 640, "a12e57b1bc8caff82de0ebe8d70864ac6625ec76b6c26b20d1e71f434cc00f51", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(2) , /* mBlockK */ -1 @@ -31847,10 +31454,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 32 , /* mEpilogueLdtmBits */ 32 , /* mEpilogueTileM */ 128 @@ -31887,7 +31494,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -31895,7 +31501,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -31910,7 +31515,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -31935,7 +31539,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 256 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -31957,10 +31561,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_relu2_bM_tma_ldgstsSf_rgTma_clmp_lsfaW4_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_relu2_bM_tma_ldgstsSf_rgTma_clmp_lsfaW4_dynB_sm100f_cubin_len, 180984, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_relu2_bM_tma_ldgstsSf_rgTma_clmp_lsfaW4_dynB_sm100f", 768, "023aa95d497cd7e62e4dd9cdda5e3b846add46a04fdc8700acd655c1b28cf64f", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_relu2_bM_tma_ldgstsSf_rgTma_clmp_lsfaW4_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_relu2_bM_tma_ldgstsSf_rgTma_clmp_lsfaW4_dynB_sm100f_cubin_len, 181144, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_relu2_bM_tma_ldgstsSf_rgTma_clmp_lsfaW4_dynB_sm100f", 768, "67b98f12a4385da35c54feb24004852afcca33ed648cf8a0c1e660b03f7b951e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(2) , /* mBlockK */ -1 @@ -31975,10 +31580,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 32 , /* mEpilogueLdtmBits */ 32 , /* mEpilogueTileM */ 128 @@ -32015,7 +31620,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -32023,7 +31627,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -32038,7 +31641,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -32063,7 +31665,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 256 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -32085,10 +31687,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_silu_bM_tma_ldgstsSf_rgTma_clmp_lsfaW2_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_silu_bM_tma_ldgstsSf_rgTma_clmp_lsfaW2_dynB_sm100f_cubin_len, 180984, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_silu_bM_tma_ldgstsSf_rgTma_clmp_lsfaW2_dynB_sm100f", 640, "2d599265b3e1d4880a89348f4baccc339761cdc6a45545c7917e52ee8588c4d1", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_silu_bM_tma_ldgstsSf_rgTma_clmp_lsfaW2_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_silu_bM_tma_ldgstsSf_rgTma_clmp_lsfaW2_dynB_sm100f_cubin_len, 181144, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_silu_bM_tma_ldgstsSf_rgTma_clmp_lsfaW2_dynB_sm100f", 640, "f2c8ed7d7382b08f06b23df4bb4727f235c34f7c385486feede6c6b7d35af3e6", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(2) , /* mBlockK */ -1 @@ -32103,10 +31706,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 32 , /* mEpilogueLdtmBits */ 32 , /* mEpilogueTileM */ 128 @@ -32143,7 +31746,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -32151,7 +31753,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -32166,7 +31767,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -32191,7 +31791,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 256 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -32213,10 +31813,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_silu_bM_tma_ldgstsSf_rgTma_clmp_lsfaW4_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_silu_bM_tma_ldgstsSf_rgTma_clmp_lsfaW4_dynB_sm100f_cubin_len, 180984, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_silu_bM_tma_ldgstsSf_rgTma_clmp_lsfaW4_dynB_sm100f", 768, "54a86050952469687274eba45d6b68f6b953eb47160cc03c1b3c8ef0e87d299f", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_silu_bM_tma_ldgstsSf_rgTma_clmp_lsfaW4_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_silu_bM_tma_ldgstsSf_rgTma_clmp_lsfaW4_dynB_sm100f_cubin_len, 181144, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_silu_bM_tma_ldgstsSf_rgTma_clmp_lsfaW4_dynB_sm100f", 768, "0f84220dbb7f4a72c0bd464c842f1c4b1ef749c996525c7344713fa4dd27bd5d", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(2) , /* mBlockK */ -1 @@ -32231,10 +31832,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 32 , /* mEpilogueLdtmBits */ 32 , /* mEpilogueTileM */ 128 @@ -32271,7 +31872,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -32279,7 +31879,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -32294,7 +31893,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -32319,7 +31917,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 256 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -32341,10 +31939,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256u2_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256u2_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 180984, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256u2_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f", 512, "b0ba14dd58d542ae7cda3317994da0a2f95d35037de6da9bc4d16b45b5e5e2c6", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256u2_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256u2_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 181144, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256u2_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f", 512, "5139e257fd12c8f8b14e961a68909326f016f4f67f6df7ab35c082eda101a43e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -32359,10 +31958,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -32399,7 +31998,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -32407,7 +32005,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -32422,7 +32019,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -32469,10 +32065,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256u2_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256u2_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 180984, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256u2_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "03185d1d3a31462d5c2c5ec284c8ec8aab692ae7c3d1aeaef7d42c21194a3e6f", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256u2_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256u2_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 181144, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256u2_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "fc5842520859a98bd1674c5941d8a6025633759156b776f95d5622da0df6847c", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -32487,10 +32084,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -32527,7 +32124,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -32535,7 +32131,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -32550,7 +32145,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -32597,10 +32191,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256u2_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256u2_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 180984, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256u2_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 512, "7d116db9850593e0b3020a4dd8157022bea36622b89f8177bb01a7d5ea9ef11a", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256u2_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256u2_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 181144, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256u2_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 512, "353c7ac930d2f30d773d27c86e747c038572e31a9fd05f996422cb9e3d03f230", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -32615,10 +32210,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -32655,7 +32250,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -32663,7 +32257,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -32678,7 +32271,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -32703,7 +32295,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 512 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -32725,10 +32317,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256u2_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256u2_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 180984, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256u2_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 512, "b35fd5e165c3b65b6a3e153b6fbed90c5db0ccaa7deb03fcf9a119d53e35ffb8", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256u2_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256u2_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 181144, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256u2_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 512, "1cbbcdab1ba2744c69f303e899beb4e2ec7b284619360170d6b957cfccf6b8c6", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -32743,10 +32336,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -32783,7 +32376,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -32791,7 +32383,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -32806,7 +32397,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -32831,7 +32421,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 512 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -32853,10 +32443,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x512_s3x3x3x3x1x3_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_geGlu_lbW4_lsfbW4_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x512_s3x3x3x3x1x3_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_geGlu_lbW4_lsfbW4_dynB_sm100f_cubin_len, 180712, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x512_s3x3x3x3x1x3_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_geGlu_lbW4_lsfbW4_dynB_sm100f", 384, "5b988ac5dfabf308524e3703da110239e410012391f8a917a8c9c9c3f73af043", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x512_s3x3x3x3x1x3_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_geGlu_lbW4_lsfbW4_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x512_s3x3x3x3x1x3_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_geGlu_lbW4_lsfbW4_dynB_sm100f_cubin_len, 180888, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x512_s3x3x3x3x1x3_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_geGlu_lbW4_lsfbW4_dynB_sm100f", 384, "74f91d4c838c4367e2b7a20cdb5a35bf6d986d51e4665af7975f1859b57f3209", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -32871,10 +32462,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -32911,7 +32502,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 2 @@ -32919,7 +32509,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 1 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -32934,7 +32523,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -32981,10 +32569,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 4 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x512_s3x3x3x3x1x3_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_lsfbW4_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x512_s3x3x3x3x1x3_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_lsfbW4_dynB_sm100f_cubin_len, 180712, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x512_s3x3x3x3x1x3_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_lsfbW4_dynB_sm100f", 384, "dda490284c8b67ec11f03c0611510b6c2b77f4a66d83ef68f39e14db317f7c99", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x512_s3x3x3x3x1x3_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_lsfbW4_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x512_s3x3x3x3x1x3_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_lsfbW4_dynB_sm100f_cubin_len, 180888, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x512_s3x3x3x3x1x3_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_lsfbW4_dynB_sm100f", 384, "47585f807ef67c2231b1ea0e29fe88455afd4b388e8dca10cf9a53110f3a7f7f", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -32999,10 +32588,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -33039,7 +32628,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 2 @@ -33047,7 +32635,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 1 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -33062,7 +32649,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -33109,10 +32695,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 4 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x512_s3x3x3x3x1x3_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_lbW4_lsfbW4_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x512_s3x3x3x3x1x3_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_lbW4_lsfbW4_dynB_sm100f_cubin_len, 180712, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x512_s3x3x3x3x1x3_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_lbW4_lsfbW4_dynB_sm100f", 384, "aa00a688019678b29cc2c006206eb773a88416e1b711437b2ec53c89e8a1dd9e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x512_s3x3x3x3x1x3_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_lbW4_lsfbW4_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x512_s3x3x3x3x1x3_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_lbW4_lsfbW4_dynB_sm100f_cubin_len, 180888, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x512_s3x3x3x3x1x3_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_lbW4_lsfbW4_dynB_sm100f", 384, "a073296e8a3594c07ca2eae044c8f6f57f7b05964a0177890fe55dd906d190b4", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -33127,10 +32714,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -33167,7 +32754,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 2 @@ -33175,7 +32761,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 1 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -33190,7 +32775,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -33215,7 +32799,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 512 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -33237,10 +32821,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 4 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x512_s3x3x3x3x1x3_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_lbW4_lsfbW4_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x512_s3x3x3x3x1x3_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_lbW4_lsfbW4_dynB_sm100f_cubin_len, 180712, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x512_s3x3x3x3x1x3_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_lbW4_lsfbW4_dynB_sm100f", 384, "41c5060b205e01a870969d51cf9d937675aa6b3d834c5d6ddb8f7492047bd1d9", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x512_s3x3x3x3x1x3_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_lbW4_lsfbW4_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x512_s3x3x3x3x1x3_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_lbW4_lsfbW4_dynB_sm100f_cubin_len, 180888, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x512_s3x3x3x3x1x3_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_lbW4_lsfbW4_dynB_sm100f", 384, "7bb1b3e4502bb18a56a33816595dc9fbbaa1945fcc020cdbfbd2f4c5f3c56cbd", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -33255,10 +32840,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -33295,7 +32880,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 2 @@ -33303,7 +32887,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 1 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -33318,7 +32901,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -33343,7 +32925,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 512 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -33365,10 +32947,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 4 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 193648, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f", 640, "24debd58ef1d65a7c1af7b16e17fea255f1da965704edbfa1584dc40eab5a224", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 193680, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f", 640, "eb8141c7f37171079a8357474ef61385e2d7a7dc6e345ab1bc50781ff852f7e4", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -33383,10 +32966,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -33423,7 +33006,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 2 @@ -33431,7 +33013,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -33446,7 +33027,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -33493,10 +33073,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 193648, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "80ae6ce2ee12824baeaf9d2769efeece28b1f700138dab6208c7188f7bac970d", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 193680, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "273ab8585f04e734258fd6b7e1b5f9c202b7ea98660caecd5a8e5d0cca858e94", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -33511,10 +33092,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -33551,7 +33132,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 2 @@ -33559,7 +33139,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -33574,7 +33153,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -33621,10 +33199,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 193648, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 640, "3d8c76a0a6988bac3a75aee639dae4f4adf7389a5514702f4532bf123667e319", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 193680, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 640, "c882d703d1b37ee920ad62a8e21d220528217d40eb16f7e588da72f2c1802cec", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -33639,10 +33218,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -33679,7 +33258,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 2 @@ -33687,7 +33265,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -33702,7 +33279,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -33749,10 +33325,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 193648, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 640, "650e89a2e03e523095e72dfa8960c0d70bc16a4b0c6f8d902542123165f81891", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 193680, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 640, "9e68c6379e718205316a78127f10f9f81db1a810b88fb599f12068b8a3b87552", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -33767,10 +33344,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -33807,7 +33384,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 2 @@ -33815,7 +33391,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -33830,7 +33405,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -33855,7 +33429,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 256 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -33877,10 +33451,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 193408, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f", 640, "b3602c57aeae0bf33bafc2e8007d83708f03886a9becb7227b3534ae835c0d15", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 193552, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f", 640, "9eae9d82cb1311ff1531ea613a5b20a57722c929a007c2fe4dcf5c94a42f47ed", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -33895,10 +33470,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -33935,7 +33510,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 1 @@ -33943,7 +33517,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -33958,7 +33531,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -34005,10 +33577,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 193408, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "18e98e7224ea93488a9952bf5fdf68d3aacdfcb1daa9be46e4a9bd1147599881", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 193552, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "f6fb6301cfea4c746627d1f6cf3d53c1ff9b7bf4efadac74d1e285f80434060f", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -34023,10 +33596,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -34063,7 +33636,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 1 @@ -34071,7 +33643,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -34086,7 +33657,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -34133,10 +33703,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 193408, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 640, "08aeaf1512ca2ff06350696818042391642059695e4218614ce19b2c4e7edfe8", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 193552, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 640, "5e5594af545c789303c934eeb9a5a0848659cab8c70a4588801c0bb3f4bdd6fe", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -34151,10 +33722,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -34191,7 +33762,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 1 @@ -34199,7 +33769,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -34214,7 +33783,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -34261,10 +33829,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 193408, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 640, "5134b146edacf7b80dd05cfca9eeed8c2c93604017c1b5a50134bbac3f1aa7bf", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 193552, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 640, "b7e70fea1bd4a252d05a5012c635700a49abcac098df118569b40243cce8cd28", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -34279,10 +33848,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -34319,7 +33888,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 1 @@ -34327,7 +33895,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -34342,7 +33909,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -34367,7 +33933,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 256 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -34389,10 +33955,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 193648, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f", 640, "a8a2e705812a5d018fc8ce16a34e4fdecb51094078b70915f82a3d84f82b4667", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 193680, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f", 640, "435269db3fa1e4dea3377d39b04bd32c91ff7bc1a9942a85a055abe816b0b45c", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -34407,10 +33974,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -34447,7 +34014,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 2 @@ -34455,7 +34021,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -34470,7 +34035,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -34517,10 +34081,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 193648, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "9ee4a77cd8e15e2e6faeee958cb0515b181c0d4016ea36c0897b4eb096c3a708", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 193680, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "5e1a564b47d3260bb3463826966c834a4660d9a09450da8c118b39a7be333e80", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -34535,10 +34100,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -34575,7 +34140,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 2 @@ -34583,7 +34147,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -34598,7 +34161,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -34645,10 +34207,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 193648, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 640, "d4e6e859bb962d1df3eb78f6a9edf1721abe918d602c9c7c548bd5269e257420", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 193680, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 640, "38fabc266c4bbb35fb91e8af53a323a523ca74748f76cafff69dd6d01a6d9fa8", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -34663,10 +34226,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -34703,7 +34266,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 2 @@ -34711,7 +34273,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -34726,7 +34287,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -34773,10 +34333,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 193648, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 640, "8482d133789556ca46d7468f626fdf7d2774f91d749f83433b49199773677435", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 193680, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 640, "64eb52449bdc7abf6251e912f2a82aeb67d67186fa46ba283dec0bd25d8382a9", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -34791,10 +34352,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -34831,7 +34392,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 2 @@ -34839,7 +34399,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -34854,7 +34413,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -34879,7 +34437,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 512 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -34901,10 +34459,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 193408, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f", 640, "054a741bf71764e576342e2520cdedae9cf5c7fd2ceed553a66ab88af2f85b38", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 193552, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f", 640, "db3561d9cc85b49282b0ee4eac0a5671c178e1fc1140f9d1a7f94046f6307fc6", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -34919,10 +34478,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -34959,7 +34518,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 1 @@ -34967,7 +34525,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -34982,7 +34539,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -35029,10 +34585,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 193408, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "9a565029a74c0731e927fc7911162abd3a260710d372b0b5be9143aca69d8aa2", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 193552, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "18e2a53492e8246c8d957d53c95d0fc6c421031f9956fcd48452e549e0e934a2", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -35047,10 +34604,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -35087,7 +34644,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 1 @@ -35095,7 +34651,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -35110,7 +34665,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -35157,10 +34711,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 193408, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 640, "f284ddbe117a72c8f59671305677b35be78205afc1f5e386114601e6d5ea2777", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 193552, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 640, "0a7a832b7f94d26c2454e0aed22100ec2bd1f959295643fc328ffcce445280e2", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -35175,10 +34730,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -35215,7 +34770,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 1 @@ -35223,7 +34777,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -35238,7 +34791,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -35285,10 +34837,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 193408, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 640, "de864d509f9db6bfa9d35944b506a3a29925b773f4292fb9556dfd811076c63c", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 193552, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 640, "021a5f047a1ce19631906b61b68a1ab597148f681da461a28072c470c2efa39e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -35303,10 +34856,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -35343,7 +34896,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 1 @@ -35351,7 +34903,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -35366,7 +34917,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -35391,7 +34941,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 512 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -35413,10 +34963,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 195224, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f", 768, "d36daa9bdbfee25f829fdb2930c96164bb234b8cd17fa96708bc065b8c35b436", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 195224, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f", 768, "60a5e6e439a2467f048ef4fb7abcf80779bdeeae963bab57c5570adc31f7aff9", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -35431,10 +34982,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -35471,7 +35022,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 3 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 2 @@ -35479,7 +35029,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -35494,7 +35043,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(2) , /* mTileK */ 512 @@ -35541,10 +35089,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 195224, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 768, "e5ac6b40f8cf33d50f52df23cf2514a557a0b6e4dc9a17d03f55af5f36451d3f", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 195224, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 768, "e42455edae8bcd32c39b50b233423200a750c923f75e19462ef0c2c38eda96d8", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -35559,10 +35108,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -35599,7 +35148,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 3 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 2 @@ -35607,7 +35155,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -35622,7 +35169,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(2) , /* mTileK */ 512 @@ -35647,7 +35193,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 1536 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -35669,10 +35215,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 195224, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 768, "037d8af45df4458b766865239111e4e422b35172d81c5e7ea8326323dacc8dad", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 195224, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 768, "95d1039fe1e4862bf55c7069e3ac601b293e3e9e0048c533b530d3e877a6b7c9", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -35687,10 +35234,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -35727,7 +35274,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 3 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 2 @@ -35735,7 +35281,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -35750,7 +35295,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(2) , /* mTileK */ 512 @@ -35775,7 +35319,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 1536 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -35797,10 +35341,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 203416, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f", 768, "909a02b148122ba1c239d4904e6b2e87038c36a5259f5f6ecd06540e34472e3e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 203416, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f", 768, "0daf2823ede553a9c49a56afe46d31d537e424f54fa505d225a90cd92b19b097", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -35815,10 +35360,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -35855,7 +35400,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 4 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 2 @@ -35863,7 +35407,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -35878,7 +35421,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(2) , /* mTileK */ 512 @@ -35925,10 +35467,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 203416, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 768, "eabc577bb15ebfacadb0ab90a5f81eec01c751962b735fbb7799039e93adb94a", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 203416, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 768, "6d91b78fc01924e6f0c7ba11bc0d9ef0cc80f013757f29984871052e7a34dde7", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -35943,10 +35486,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -35983,7 +35526,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 4 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 2 @@ -35991,7 +35533,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -36006,7 +35547,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(2) , /* mTileK */ 512 @@ -36031,7 +35571,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 2048 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -36053,10 +35593,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 203416, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 768, "78ac1f784e92413305476f16839926f56aca1da154180e5039baba18d03ac1fb", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 203416, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 768, "7c945cb964673ab6fb447c9f19a503d196cd09d24ffb8aa4900a68224de346af", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -36071,10 +35612,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -36111,7 +35652,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 4 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 2 @@ -36119,7 +35659,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -36134,7 +35673,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(2) , /* mTileK */ 512 @@ -36159,7 +35697,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 2048 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -36181,10 +35719,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 203512, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f", 640, "22a0c69a3c8cb98269eee093f22e5339e2bb0129567b99e6d4674a4f791985b6", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 203672, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f", 640, "0023fb5528293e1372784e4f4f5eb17f3e5b876f56b6508f3b2316d85b4f6b4f", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -36199,10 +35738,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -36239,7 +35778,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -36247,7 +35785,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -36262,7 +35799,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -36309,10 +35845,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 203512, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "d98c28121021a47179b74c736e6a143165393157e2530b7863638a36a10ea953", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 203672, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "822bc7094959e6e5f648c445167e377d2b9b9386ee09dcb0ae5b33e9d069f0a7", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -36327,10 +35864,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -36367,7 +35904,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -36375,7 +35911,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -36390,7 +35925,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -36437,10 +35971,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 203512, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 640, "389814eb85fc3bc402e4212b73d12081cd1c980fd2591c51fbfe33c2ec010a6d", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 203672, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 640, "5892ded9cf2e5d0795195e5e36a01537472c12b24d89b02d85f27dca51ec0d33", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -36455,10 +35990,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -36495,7 +36030,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -36503,7 +36037,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -36518,7 +36051,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -36565,10 +36097,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 203512, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 640, "42d6f973553a95d32b6ec111247777672d3b5f57db8325bba908aee19e165ee6", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 203672, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 640, "23776927d9444a8f84f8b2d706171fb1792b1565d929661066b143a7a1b38131", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -36583,10 +36116,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -36623,7 +36156,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -36631,7 +36163,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -36646,7 +36177,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -36671,7 +36201,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 512 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -36693,10 +36223,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 203272, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f", 640, "36f8cd1192c36d92923ee8edd2167af4ac51b4215b62708baf2b109ae2688fba", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 203288, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f", 640, "82eb7aa2c5e511d6aa8419b6082dcf65f5ed647509bb7e07a5a4ab1c8c6d0a77", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -36711,10 +36242,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -36751,7 +36282,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -36759,7 +36289,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -36774,7 +36303,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -36821,10 +36349,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 203272, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "0f142ab3e5eb23a8fe71289a78c15cd4ffd753e6dd3fc50d955ddd63bf858c25", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 203288, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "f7731647196df15ade7ba953bccd68db614269b2d2d59f4d9efd5f0591fd567d", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -36839,10 +36368,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -36879,7 +36408,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -36887,7 +36415,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -36902,7 +36429,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -36949,10 +36475,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 203272, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 640, "df602f013ef72cc66e07c2f6eba28286eb3f441014b64606271e2de163e778e3", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 203288, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 640, "78a0e608204b168fe65dbffd09b19705cfabf8724458410d2b6d64e58afd7813", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -36967,10 +36494,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -37007,7 +36534,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -37015,7 +36541,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -37030,7 +36555,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -37077,10 +36601,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 203272, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 640, "e7189d7293c00a540851a78188a36feea6f486ed9eb3e03dd76e9ce8859662cb", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 203288, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 640, "bf39a381651feea10ab26c855601d6b6ecb919d05a1f1a627f722a49437d3d2c", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -37095,10 +36620,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -37135,7 +36660,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -37143,7 +36667,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -37158,7 +36681,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -37183,7 +36705,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 512 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -37205,10 +36727,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 203512, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f", 640, "c98c7c7a7f259968f52afecb1878c4511fef3e4c0e0e1627d205a802c44b7f1e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 203672, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f", 640, "43d0a3004a82ae69cefb205731030068b632eb29dd882d3792fd0ff207c887a2", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -37223,10 +36746,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -37263,7 +36786,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -37271,7 +36793,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -37286,7 +36807,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -37333,10 +36853,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 203512, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "96cd1878a765f0fff65a2943f3e93d2f4097a5bb17cfb6cc4a22f97cc3f07c75", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 203672, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "ea45593a1b79998942c457f2882e3efb784e9b312b0e3d65401dc680a8709f0f", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -37351,10 +36872,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -37391,7 +36912,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -37399,7 +36919,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -37414,7 +36933,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -37461,10 +36979,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 203512, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 640, "a7efe52ca7d8db1a256664a8162fb6dda4b2553af8d6bb3951096b6556799058", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 203672, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 640, "edc06af8e38e340af8a960b20672e9f2524cc3c8e259bde0d5d964ab4827be3a", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -37479,10 +36998,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -37519,7 +37038,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -37527,7 +37045,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -37542,7 +37059,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -37589,10 +37105,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 203512, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 640, "ed87d24c53375448af0cb0f7c5d95653b8fb4393702bcd624c455db9f1bbc823", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 203672, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 640, "839bd79351712d5cf5cb0699beb499a0df9993da454fd870fc0ce17d15381868", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -37607,10 +37124,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -37647,7 +37164,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -37655,7 +37171,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -37670,7 +37185,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -37695,7 +37209,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 1024 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -37717,10 +37231,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 203272, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f", 640, "41d2ddd16b20ea5a69140fbe07f07d1520c6555f51be0cbac49292fd7856005b", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 203288, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f", 640, "c3a003d82263aef19d9588e4989936eb3ac0e7daf7c6944aced00ad1d984433c", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -37735,10 +37250,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -37775,7 +37290,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -37783,7 +37297,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -37798,7 +37311,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -37845,10 +37357,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 203272, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "3c9e08784fa8c306ab31a4651770f1a13ad8611abbf29ed6935531b611ba56a2", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 203288, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "d35e2ab5fee7114cf71e0e1055a1122c1b672d5e49518993fd6fcf7d82b7b821", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -37863,10 +37376,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -37903,7 +37416,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -37911,7 +37423,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -37926,7 +37437,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -37973,10 +37483,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 203272, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 640, "5f7a71e1090ee9c5147c27aca0a454833a13f562f58afa5f678267258b080b61", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 203288, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 640, "b3576dc1a038c18ae9957bd385f917e9c64b3605ef888f5b9c8a92509c1ac6b8", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -37991,10 +37502,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -38031,7 +37542,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -38039,7 +37549,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -38054,7 +37563,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -38101,10 +37609,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 203272, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 640, "b97bb9826b429e20b7e370cfbe47546d1370164613be47bfb7b361559ee8f15a", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 203288, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 640, "d834b9b5635d9b77c2273a7a0c0c032ee37d0eceadba6b7b9a70264ce94bbdc9", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -38119,10 +37628,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -38159,7 +37668,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -38167,7 +37675,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -38182,7 +37689,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -38207,7 +37713,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 1024 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -38229,10 +37735,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_eW8_fCp_tmOv_bN_tma_ldgSf_rgTma_clmp_geGlu_lbW8_lsfbW4_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_eW8_fCp_tmOv_bN_tma_ldgSf_rgTma_clmp_geGlu_lbW8_lsfbW4_dynB_sm100f_cubin_len, 172568, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_eW8_fCp_tmOv_bN_tma_ldgSf_rgTma_clmp_geGlu_lbW8_lsfbW4_dynB_sm100f", 768, "75c7f09042faf923a19a42dd73fddfbdf54bf3039f1470301e1fc40f601de347", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_eW8_fCp_tmOv_bN_tma_ldgSf_rgTma_clmp_geGlu_lbW8_lsfbW4_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_eW8_fCp_tmOv_bN_tma_ldgSf_rgTma_clmp_geGlu_lbW8_lsfbW4_dynB_sm100f_cubin_len, 172696, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_eW8_fCp_tmOv_bN_tma_ldgSf_rgTma_clmp_geGlu_lbW8_lsfbW4_dynB_sm100f", 768, "ecb5360a5e4c1f001fbf9d6f03ff30a79c127926c2fddaa757a14eb23333d8cf", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -38247,10 +37754,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -38287,7 +37794,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 144 , /* mNumRegsPerThreadNonEpilogueWarp */ 88 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 1 @@ -38295,7 +37801,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 1 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -38310,7 +37815,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -38357,10 +37861,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 4 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(3)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_eW8_fCp_tmOv_bN_tma_ldgSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_eW8_fCp_tmOv_bN_tma_ldgSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f_cubin_len, 172568, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_eW8_fCp_tmOv_bN_tma_ldgSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f", 768, "b8c1b00e29d784ce53298a33318b4b4614dae800ac71131341af25cee4c1e4fd", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_eW8_fCp_tmOv_bN_tma_ldgSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_eW8_fCp_tmOv_bN_tma_ldgSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f_cubin_len, 172696, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_eW8_fCp_tmOv_bN_tma_ldgSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f", 768, "210a5300774b5d1bdcbc8e1184964a9346fa83049ceea15e07b871b3444e0e67", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -38375,10 +37880,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -38415,7 +37920,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 144 , /* mNumRegsPerThreadNonEpilogueWarp */ 88 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 1 @@ -38423,7 +37927,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 1 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -38438,7 +37941,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -38485,10 +37987,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 4 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(3)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_ldgsts_ldgstsSf_rgTma_clmp_geGlu_lbW4_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_ldgsts_ldgstsSf_rgTma_clmp_geGlu_lbW4_dynB_sm100f_cubin_len, 168472, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_ldgsts_ldgstsSf_rgTma_clmp_geGlu_lbW4_dynB_sm100f", 384, "89b36fb71c6dccf92929181a24524feaf7ee239995eee1f63134d91a4f0b516c", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_ldgsts_ldgstsSf_rgTma_clmp_geGlu_lbW4_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_ldgsts_ldgstsSf_rgTma_clmp_geGlu_lbW4_dynB_sm100f_cubin_len, 168600, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_ldgsts_ldgstsSf_rgTma_clmp_geGlu_lbW4_dynB_sm100f", 384, "754e25643de5a5bba6cce4516cd4a9744a7460eaadedfd6a3df40b763f4966bf", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -38503,10 +38006,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -38543,7 +38046,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 232 , /* mNumRegsPerThreadNonEpilogueWarp */ 88 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 1 @@ -38551,7 +38053,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 1 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -38566,7 +38067,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -38613,10 +38113,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_dynB_sm100f_cubin_len, 168472, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_dynB_sm100f", 384, "8289ad411f89cfe348b03a920dbb7ffcf5d8904325d859ea02b580f1cdc728b9", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_dynB_sm100f_cubin_len, 168600, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_dynB_sm100f", 384, "54766d60c6b1f2826639722e0e8bf981ff8df71116db358dd7ef2d767802b03e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -38631,10 +38132,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -38671,7 +38172,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 232 , /* mNumRegsPerThreadNonEpilogueWarp */ 88 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 1 @@ -38679,7 +38179,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 1 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -38694,7 +38193,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -38741,10 +38239,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_relu2_eW8_fCp_tmOv_bN_tma_ldgSf_rgTma_clmp_lbW8_lsfbW4_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_relu2_eW8_fCp_tmOv_bN_tma_ldgSf_rgTma_clmp_lbW8_lsfbW4_dynB_sm100f_cubin_len, 172568, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_relu2_eW8_fCp_tmOv_bN_tma_ldgSf_rgTma_clmp_lbW8_lsfbW4_dynB_sm100f", 768, "78316d11d1b07ce27f5d7b7b54d416f6c87e430b718611a36f1df3cd10a01197", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_relu2_eW8_fCp_tmOv_bN_tma_ldgSf_rgTma_clmp_lbW8_lsfbW4_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_relu2_eW8_fCp_tmOv_bN_tma_ldgSf_rgTma_clmp_lbW8_lsfbW4_dynB_sm100f_cubin_len, 172696, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_relu2_eW8_fCp_tmOv_bN_tma_ldgSf_rgTma_clmp_lbW8_lsfbW4_dynB_sm100f", 768, "6ed22eb842af4c21ccf604ac0bba7feb5aa9dee683e79380c9f6b51d89062da4", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -38759,10 +38258,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -38799,7 +38298,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 144 , /* mNumRegsPerThreadNonEpilogueWarp */ 88 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 1 @@ -38807,7 +38305,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 1 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -38822,7 +38319,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -38847,7 +38343,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 256 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -38869,10 +38365,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 4 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(3)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_relu2_fCp_tmOv_bN_ldgsts_ldgstsSf_rgTma_clmp_lbW4_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_relu2_fCp_tmOv_bN_ldgsts_ldgstsSf_rgTma_clmp_lbW4_dynB_sm100f_cubin_len, 168472, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_relu2_fCp_tmOv_bN_ldgsts_ldgstsSf_rgTma_clmp_lbW4_dynB_sm100f", 384, "005cf3d7f664f3c35fdb52cf00cfab60c999348226c02e3aa319d7cd9505062b", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_relu2_fCp_tmOv_bN_ldgsts_ldgstsSf_rgTma_clmp_lbW4_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_relu2_fCp_tmOv_bN_ldgsts_ldgstsSf_rgTma_clmp_lbW4_dynB_sm100f_cubin_len, 168600, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_relu2_fCp_tmOv_bN_ldgsts_ldgstsSf_rgTma_clmp_lbW4_dynB_sm100f", 384, "a275d63ffe577a954792672c7061a25560a982806973924ebf623e51f42ce86f", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -38887,10 +38384,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -38927,7 +38424,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 232 , /* mNumRegsPerThreadNonEpilogueWarp */ 88 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 1 @@ -38935,7 +38431,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 1 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -38950,7 +38445,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -38975,7 +38469,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 256 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -38997,10 +38491,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_silu_eW8_fCp_tmOv_bN_tma_ldgSf_rgTma_clmp_lbW8_lsfbW4_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_silu_eW8_fCp_tmOv_bN_tma_ldgSf_rgTma_clmp_lbW8_lsfbW4_dynB_sm100f_cubin_len, 172568, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_silu_eW8_fCp_tmOv_bN_tma_ldgSf_rgTma_clmp_lbW8_lsfbW4_dynB_sm100f", 768, "d2a8cccdb1247ae5f95963233f3fc0ac08927b9ad233e3b41f21fe1ef41b03f4", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_silu_eW8_fCp_tmOv_bN_tma_ldgSf_rgTma_clmp_lbW8_lsfbW4_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_silu_eW8_fCp_tmOv_bN_tma_ldgSf_rgTma_clmp_lbW8_lsfbW4_dynB_sm100f_cubin_len, 172696, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_silu_eW8_fCp_tmOv_bN_tma_ldgSf_rgTma_clmp_lbW8_lsfbW4_dynB_sm100f", 768, "9edb847256c8ba18cad06eb83d7317d98470992688f6dc9c8c740d4f81771a58", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -39015,10 +38510,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -39055,7 +38550,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 144 , /* mNumRegsPerThreadNonEpilogueWarp */ 88 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 1 @@ -39063,7 +38557,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 1 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -39078,7 +38571,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -39103,7 +38595,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 256 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -39125,10 +38617,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 4 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(3)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_silu_fCp_tmOv_bN_ldgsts_ldgstsSf_rgTma_clmp_lbW4_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_silu_fCp_tmOv_bN_ldgsts_ldgstsSf_rgTma_clmp_lbW4_dynB_sm100f_cubin_len, 168472, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_silu_fCp_tmOv_bN_ldgsts_ldgstsSf_rgTma_clmp_lbW4_dynB_sm100f", 384, "5205f8b9a6c7252f2377d6507c1832b71941123a10f1ecf3bf87612f216f4c88", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_silu_fCp_tmOv_bN_ldgsts_ldgstsSf_rgTma_clmp_lbW4_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_silu_fCp_tmOv_bN_ldgsts_ldgstsSf_rgTma_clmp_lbW4_dynB_sm100f_cubin_len, 168600, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_silu_fCp_tmOv_bN_ldgsts_ldgstsSf_rgTma_clmp_lbW4_dynB_sm100f", 384, "644b8da7b105b67fdb91a25c3b7778db1cf317daa3ab18deacbbab5cbf418b62", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -39143,10 +38636,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -39183,7 +38676,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 232 , /* mNumRegsPerThreadNonEpilogueWarp */ 88 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 1 @@ -39191,7 +38683,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 1 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -39206,7 +38697,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -39231,7 +38721,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 256 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -39253,10 +38743,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s5x5x5x5x5x1_et128x64_m256x256x64_c2x1x1_rN_TN_schPdx3_biasFp32N_fCp_tmOv_bM_tma_ldgSf_rgTma_clmp_geGlu_laW8_lsfaW4_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s5x5x5x5x5x1_et128x64_m256x256x64_c2x1x1_rN_TN_schPdx3_biasFp32N_fCp_tmOv_bM_tma_ldgSf_rgTma_clmp_geGlu_laW8_lsfaW4_dynB_sm100f_cubin_len, 203416, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s5x5x5x5x5x1_et128x64_m256x256x64_c2x1x1_rN_TN_schPdx3_biasFp32N_fCp_tmOv_bM_tma_ldgSf_rgTma_clmp_geGlu_laW8_lsfaW4_dynB_sm100f", 640, "bb3d10bf68d0b1363a51e265f0eb989f43996df9000208b61be9ff8d1ccbf373", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s5x5x5x5x5x1_et128x64_m256x256x64_c2x1x1_rN_TN_schPdx3_biasFp32N_fCp_tmOv_bM_tma_ldgSf_rgTma_clmp_geGlu_laW8_lsfaW4_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s5x5x5x5x5x1_et128x64_m256x256x64_c2x1x1_rN_TN_schPdx3_biasFp32N_fCp_tmOv_bM_tma_ldgSf_rgTma_clmp_geGlu_laW8_lsfaW4_dynB_sm100f_cubin_len, 203416, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s5x5x5x5x5x1_et128x64_m256x256x64_c2x1x1_rN_TN_schPdx3_biasFp32N_fCp_tmOv_bM_tma_ldgSf_rgTma_clmp_geGlu_laW8_lsfaW4_dynB_sm100f", 640, "a86c8dee26a642c62a097020e049304873a22aaec872ecb6c85f96ba5596e562", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(2) , /* mBlockK */ -1 @@ -39271,10 +38762,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 32 , /* mEpilogueLdtmBits */ 32 , /* mEpilogueTileM */ 128 @@ -39311,7 +38802,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 256 , /* mNumRegsPerThreadNonEpilogueWarp */ 104 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -39319,7 +38809,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 1 , /* mNumStagesWorkId */ 3 @@ -39334,7 +38823,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -39381,10 +38869,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(3)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s5x5x5x5x5x1_et128x64_m256x256x64_c2x1x1_rN_TN_schPdx3_biasFp32N_fCp_tmOv_bM_tma_ldgSf_rgTma_clmp_swiGlu_laW8_lsfaW4_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s5x5x5x5x5x1_et128x64_m256x256x64_c2x1x1_rN_TN_schPdx3_biasFp32N_fCp_tmOv_bM_tma_ldgSf_rgTma_clmp_swiGlu_laW8_lsfaW4_dynB_sm100f_cubin_len, 203416, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s5x5x5x5x5x1_et128x64_m256x256x64_c2x1x1_rN_TN_schPdx3_biasFp32N_fCp_tmOv_bM_tma_ldgSf_rgTma_clmp_swiGlu_laW8_lsfaW4_dynB_sm100f", 640, "2e728ee61ed0436d6fc7183040347e81333505356ab6865cfc740674dbfbfed0", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s5x5x5x5x5x1_et128x64_m256x256x64_c2x1x1_rN_TN_schPdx3_biasFp32N_fCp_tmOv_bM_tma_ldgSf_rgTma_clmp_swiGlu_laW8_lsfaW4_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s5x5x5x5x5x1_et128x64_m256x256x64_c2x1x1_rN_TN_schPdx3_biasFp32N_fCp_tmOv_bM_tma_ldgSf_rgTma_clmp_swiGlu_laW8_lsfaW4_dynB_sm100f_cubin_len, 203416, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s5x5x5x5x5x1_et128x64_m256x256x64_c2x1x1_rN_TN_schPdx3_biasFp32N_fCp_tmOv_bM_tma_ldgSf_rgTma_clmp_swiGlu_laW8_lsfaW4_dynB_sm100f", 640, "d116020528832ac1471b8ece21a3989029f00bf9c954c8290a1239cdfaefc502", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(2) , /* mBlockK */ -1 @@ -39399,10 +38888,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 32 , /* mEpilogueLdtmBits */ 32 , /* mEpilogueTileM */ 128 @@ -39439,7 +38928,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 256 , /* mNumRegsPerThreadNonEpilogueWarp */ 104 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -39447,7 +38935,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 1 , /* mNumStagesWorkId */ 3 @@ -39462,7 +38949,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -39509,10 +38995,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(3)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s5x5x5x5x5x1_et128x64_m256x256x64_c2x1x1_rN_TN_schPdx3_biasFp32N_relu2_fCp_tmOv_bM_tma_ldgSf_rgTma_clmp_laW8_lsfaW4_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s5x5x5x5x5x1_et128x64_m256x256x64_c2x1x1_rN_TN_schPdx3_biasFp32N_relu2_fCp_tmOv_bM_tma_ldgSf_rgTma_clmp_laW8_lsfaW4_dynB_sm100f_cubin_len, 203416, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s5x5x5x5x5x1_et128x64_m256x256x64_c2x1x1_rN_TN_schPdx3_biasFp32N_relu2_fCp_tmOv_bM_tma_ldgSf_rgTma_clmp_laW8_lsfaW4_dynB_sm100f", 640, "08822b44cdfeb0fd7dd44121901b8e2e1bd70af14208e242d84959b1aa144c38", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s5x5x5x5x5x1_et128x64_m256x256x64_c2x1x1_rN_TN_schPdx3_biasFp32N_relu2_fCp_tmOv_bM_tma_ldgSf_rgTma_clmp_laW8_lsfaW4_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s5x5x5x5x5x1_et128x64_m256x256x64_c2x1x1_rN_TN_schPdx3_biasFp32N_relu2_fCp_tmOv_bM_tma_ldgSf_rgTma_clmp_laW8_lsfaW4_dynB_sm100f_cubin_len, 203416, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s5x5x5x5x5x1_et128x64_m256x256x64_c2x1x1_rN_TN_schPdx3_biasFp32N_relu2_fCp_tmOv_bM_tma_ldgSf_rgTma_clmp_laW8_lsfaW4_dynB_sm100f", 640, "0e233531b1ab78c69085cc5c28a8aa376609729d512bd5201dd82d507f1256fe", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(2) , /* mBlockK */ -1 @@ -39527,10 +39014,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 32 , /* mEpilogueLdtmBits */ 32 , /* mEpilogueTileM */ 128 @@ -39567,7 +39054,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 256 , /* mNumRegsPerThreadNonEpilogueWarp */ 104 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -39575,7 +39061,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 1 , /* mNumStagesWorkId */ 3 @@ -39590,7 +39075,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -39615,7 +39099,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 256 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -39637,10 +39121,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(3)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s5x5x5x5x5x1_et128x64_m256x256x64_c2x1x1_rN_TN_schPdx3_biasFp32N_silu_fCp_tmOv_bM_tma_ldgSf_rgTma_clmp_laW8_lsfaW4_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s5x5x5x5x5x1_et128x64_m256x256x64_c2x1x1_rN_TN_schPdx3_biasFp32N_silu_fCp_tmOv_bM_tma_ldgSf_rgTma_clmp_laW8_lsfaW4_dynB_sm100f_cubin_len, 203416, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s5x5x5x5x5x1_et128x64_m256x256x64_c2x1x1_rN_TN_schPdx3_biasFp32N_silu_fCp_tmOv_bM_tma_ldgSf_rgTma_clmp_laW8_lsfaW4_dynB_sm100f", 640, "d57cb4dc9a25158a099a6d9144a03909c97fbe5c81dc89b41f270f30cd876096", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s5x5x5x5x5x1_et128x64_m256x256x64_c2x1x1_rN_TN_schPdx3_biasFp32N_silu_fCp_tmOv_bM_tma_ldgSf_rgTma_clmp_laW8_lsfaW4_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s5x5x5x5x5x1_et128x64_m256x256x64_c2x1x1_rN_TN_schPdx3_biasFp32N_silu_fCp_tmOv_bM_tma_ldgSf_rgTma_clmp_laW8_lsfaW4_dynB_sm100f_cubin_len, 203416, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s5x5x5x5x5x1_et128x64_m256x256x64_c2x1x1_rN_TN_schPdx3_biasFp32N_silu_fCp_tmOv_bM_tma_ldgSf_rgTma_clmp_laW8_lsfaW4_dynB_sm100f", 640, "ea3b2a39ddf253f7d9d7d9d57cceb030d32665ca5d4cf02bbfd64bc5fc727e8c", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(2) , /* mBlockK */ -1 @@ -39655,10 +39140,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 32 , /* mEpilogueLdtmBits */ 32 , /* mEpilogueTileM */ 128 @@ -39695,7 +39180,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 256 , /* mNumRegsPerThreadNonEpilogueWarp */ 104 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -39703,7 +39187,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 1 , /* mNumStagesWorkId */ 3 @@ -39718,7 +39201,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -39743,7 +39225,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 256 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -39765,10 +39247,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(3)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 215152, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f", 640, "2ea13440352e11906672febc7ea4def46353e9a90c0f7f264df37d2a5051bc6a", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 215184, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f", 640, "3fa46629bbcf794937dcfed33fdd12b19a1d15ef0c61d18ba5e827eccb52bfd5", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -39783,10 +39266,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -39823,7 +39306,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 2 @@ -39831,7 +39313,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -39846,7 +39327,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -39893,10 +39373,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 215152, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "877c87e13afc91e9a5b4f57c87fdc9f47ef261336487ce75d4444765790e8414", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 215184, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "e31bc1891ac54cc71a79de912b1642d0cf3e6db14218108041d5cb0e68ff4bc8", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -39911,10 +39392,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -39951,7 +39432,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 2 @@ -39959,7 +39439,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -39974,7 +39453,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -40021,10 +39499,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 215152, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 640, "106b47e68a86a5d515e395bf5b7a5444b8007e982635d5d423389295c81c9db7", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 215184, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 640, "29a4121f549f323fee4a2a3bf79f54f8aa72d2a3000368d7b714f38882eee224", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -40039,10 +39518,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -40079,7 +39558,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 2 @@ -40087,7 +39565,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -40102,7 +39579,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -40149,10 +39625,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 215152, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 640, "e526447732f059f8b73ebb0a35cb901a30ad7f0f386d9b56eec111e9d10e54f6", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 215184, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 640, "b443a38efe304820dadb35a3a5523a068487ee6d02fd5cd7cf563747b220b645", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -40167,10 +39644,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -40207,7 +39684,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 2 @@ -40215,7 +39691,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -40230,7 +39705,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -40255,7 +39729,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 256 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -40277,10 +39751,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 214912, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f", 640, "7e4955c0eb1379b5d4bf19b4390a6ef6df6b09039c99640e74a3c18e88717ed9", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 215056, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f", 640, "1cf3a4604b0cfcc3c259db26c34eab0606cc9703bdd554298e8b69f6bf22eec4", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -40295,10 +39770,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -40335,7 +39810,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 1 @@ -40343,7 +39817,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -40358,7 +39831,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -40405,10 +39877,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 214912, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "1ff2c73c6a43f7002f0179ebe2f31eafbe5e6391860a8a1f8df5fc1c2e781987", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 215056, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "f6b9c68880afe76421606c898c10500ae3e2468ee15b2bc382b4cf2995acc280", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -40423,10 +39896,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -40463,7 +39936,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 1 @@ -40471,7 +39943,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -40486,7 +39957,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -40533,10 +40003,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 214912, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 640, "a7f2cf60aa400797b065948e26147906ff25ece81f0ebaa02d1c91222cb6d1ee", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 215056, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 640, "f93cd33d47aafcccfd5979a2ec457ec545bcc7dc12cf7d5bac5f4ba2c58e02b5", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -40551,10 +40022,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -40591,7 +40062,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 1 @@ -40599,7 +40069,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -40614,7 +40083,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -40661,10 +40129,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 214912, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 640, "0ac16f361375ff41be3b95ed621589dc79c312fd40ad155b47e1f175eb98e0d4", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 215056, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 640, "e25b71344195ba8d447662915f1591ea0b0ecb0b1ea02f11823ef84fcf407ba5", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -40679,10 +40148,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -40719,7 +40188,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 1 @@ -40727,7 +40195,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -40742,7 +40209,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -40767,7 +40233,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 256 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -40789,10 +40255,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 215152, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f", 640, "50fa3fce0b470c864a44b4214a6e7d42f666a04831304b7d70771a08e77e3384", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 215184, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f", 640, "432cc5c5d5c2e6164b46d917fda5e06514b9d62b8a14e31102cdbb9c19e7077a", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -40807,10 +40274,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -40847,7 +40314,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 2 @@ -40855,7 +40321,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -40870,7 +40335,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -40917,10 +40381,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 215152, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "f1dd0bbee4745310175c5d2cf09214b1651d618681c037283bfdf6ec5938b511", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 215184, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "6cc28befcb46faad5bb875dae905e03bbc2335052a8c71531aa06b292b4871ae", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -40935,10 +40400,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -40975,7 +40440,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 2 @@ -40983,7 +40447,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -40998,7 +40461,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -41045,10 +40507,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 215152, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 640, "749ec2b2397c208590241ba4cda990f82b1523a8d5110fa4a3062481d292c4bc", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 215184, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 640, "f333d0dce2e1704471c5890f277d62cce2f38120a8d5a4a3434c1addcd1bfb8e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -41063,10 +40526,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -41103,7 +40566,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 2 @@ -41111,7 +40573,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -41126,7 +40587,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -41173,10 +40633,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 215152, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 640, "037dadc1bcc4c7f613b08d75045562f450c4c5f5a4bdd88b0033e45a27324755", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 215184, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 640, "2b53fadc2d3706885737e4940c44ae3e996a2cff734ec20de580f14dbd5fb992", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -41191,10 +40652,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -41231,7 +40692,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 2 @@ -41239,7 +40699,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -41254,7 +40713,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -41279,7 +40737,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 512 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -41301,10 +40759,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 214912, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f", 640, "0ba5ed8e6f618d2fc9f32deca26449d293205372312962d895035bd98f4bc2da", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 215056, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f", 640, "757e46a221011bceeea41ab9c950e766f4dada1eeb2a1327c7c0cbab53431e8b", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -41319,10 +40778,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -41359,7 +40818,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 1 @@ -41367,7 +40825,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -41382,7 +40839,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -41429,10 +40885,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 214912, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "573fef5b71087a115ed0d0669b0a1396ab537bb1774a97f3388a18215831378c", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 215056, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "41c968f3fbd4acd910b9b3f64405860cd5b4a3bf927f28bd62626227de2dc89d", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -41447,10 +40904,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -41487,7 +40944,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 1 @@ -41495,7 +40951,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -41510,7 +40965,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -41557,10 +41011,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 214912, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 640, "8a4d386f911a4c3e8597544f5af4ee083e76dba7fd1a0469e1adb913d909e305", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 215056, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 640, "0da1fb14827844701f93372b22d1f1847e730ba8f41bb0863acacc9b9b74baf2", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -41575,10 +41030,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -41615,7 +41070,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 1 @@ -41623,7 +41077,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -41638,7 +41091,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -41685,10 +41137,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 214912, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 640, "6a97999f445be4b703a886e9d49b9c4215538c11625d6034020bddb820ad1c81", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 215056, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 640, "f9b27ab662a0bb14e48d66481075bebfe9e35a6b1c82e78de576292fc9bb1876", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -41703,10 +41156,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -41743,7 +41196,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 1 @@ -41751,7 +41203,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -41766,7 +41217,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -41791,7 +41241,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 512 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -41813,10 +41263,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s3_et128x32_m128x32x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s3_et128x32_m128x32x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 175672, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s3_et128x32_m128x32x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f", 896, "673ce7440c78699649faa35cb3c0dfdddb70ff5057f607ec520d3211866dd8af", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s3_et128x32_m128x32x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s3_et128x32_m128x32x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 175768, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s3_et128x32_m128x32x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f", 896, "a0f856b937942695890b99f63190a45d749d83a3a3515a69e807f87798277bb6", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -41831,10 +41282,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -41871,7 +41322,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 2 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 2 @@ -41879,7 +41329,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -41894,7 +41343,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(2) , /* mTileK */ 512 @@ -41941,10 +41389,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s3_et128x32_m128x32x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s3_et128x32_m128x32x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 175672, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s3_et128x32_m128x32x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 896, "8e70bf81c794ea433223bf8be0cb74791a819198f8176110f7e16a55066e9129", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s3_et128x32_m128x32x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s3_et128x32_m128x32x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 175768, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s3_et128x32_m128x32x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 896, "a79089c8dafb298893b4af40453ba24c0c395179dfc6103089e3d04ab9d0e955", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -41959,10 +41408,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -41999,7 +41448,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 2 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 2 @@ -42007,7 +41455,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -42022,7 +41469,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(2) , /* mTileK */ 512 @@ -42047,7 +41493,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 1024 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -42069,10 +41515,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s3_et128x32_m128x32x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s3_et128x32_m128x32x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 175672, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s3_et128x32_m128x32x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 896, "cf0c949ad67f26e2b8621c654952135276639cbdd8b98a899ed2d3f736c3a388", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s3_et128x32_m128x32x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s3_et128x32_m128x32x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 175768, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s3_et128x32_m128x32x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 896, "b89ff6ddd24f8d90e97639991c0d1058679efa4913802c5accdb661ea59d818b", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -42087,10 +41534,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -42127,7 +41574,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 2 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 2 @@ -42135,7 +41581,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -42150,7 +41595,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(2) , /* mTileK */ 512 @@ -42175,7 +41619,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 1024 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -42197,10 +41641,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 216824, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f", 896, "a989ef6380a2007579ad30e96d8577f6ec53ff928121fcf75ff7782bbce14b4c", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 216984, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f", 896, "1db4be9e0ebe9c6b7dfe23880a2a16115aee8614dc5b11197c474c5fc5cfa5db", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -42215,10 +41660,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -42255,7 +41700,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -42263,7 +41707,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -42278,7 +41721,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -42325,10 +41767,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 216824, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 896, "57703dd1f3c5e00d21d780ce3bb6f7a31255c4844c42b8d3ff31c23bbd65124e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 216984, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 896, "85a869d77ed64db2ad22f80f0dc7f78a25def7a7cb68ef8885d8e5986efdbb41", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -42343,10 +41786,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -42383,7 +41826,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -42391,7 +41833,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -42406,7 +41847,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -42453,10 +41893,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 216824, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 896, "7f413ab08191787334eef578d7ffd4d5b336ed3099fa27efb1cd6db149980856", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 216984, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 896, "5590da7579c4042a2d76a446db9246d70f0048ac98e188e730c8b9365962fd87", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -42471,10 +41912,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -42511,7 +41952,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -42519,7 +41959,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -42534,7 +41973,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -42581,10 +42019,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 216824, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 896, "fc53da4a521059a78d549022fbfc133f9bfe492a1da2a14d8f83842725d69b14", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 216984, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 896, "29bfcacbe0656a1b44c9edebf04a10740d4eaeb6be3f55cf9b11997bb2a506f6", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -42599,10 +42038,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -42639,7 +42078,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -42647,7 +42085,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -42662,7 +42099,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -42687,7 +42123,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 512 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -42709,10 +42145,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 216584, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f", 768, "e60fc1831e9829a6f8c62b5cb83b073850b24860589d56278edd0347c568168b", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 216600, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f", 768, "99e879b8ed8c8bb5fd1719f2bdb3472893778a978525ad31f5fc6a596f7cf6e4", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -42727,10 +42164,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -42767,7 +42204,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -42775,7 +42211,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -42790,7 +42225,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -42837,10 +42271,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 216584, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 768, "96e568ee3903e88a4b06c8eb88d3ccc1e139e11ef298736419ca91d5fbbb8722", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 216600, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 768, "4daf61574a0f60ef270e43398d75bfc0911b710f217668f6ff265a4d2a558cfc", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -42855,10 +42290,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -42895,7 +42330,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -42903,7 +42337,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -42918,7 +42351,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -42965,10 +42397,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 216584, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 768, "173576359ca96dff9b58e21e940b12242682545ccceb14a38e8146221f62e710", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 216600, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 768, "ab56ae46fb46980898000d681f492bbe88e237438ee945a13836774b37224c7e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -42983,10 +42416,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -43023,7 +42456,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -43031,7 +42463,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -43046,7 +42477,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -43093,10 +42523,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 216584, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 768, "e653e5fa51ca7df9bb1110e6dae1c93f1fd8c33a0035fb277f1afda314ddda8f", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 216600, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 768, "630f4be5c4efaa7813b64baf8992f19e6d6bc128c92be4f8e743b660a7380aa9", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -43111,10 +42542,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -43151,7 +42582,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -43159,7 +42589,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -43174,7 +42603,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -43199,7 +42627,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 512 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -43221,10 +42649,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 216824, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f", 896, "4d4b12ce769049a61102e606b93c46866293f41f7add2d967ecb32104de495bf", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 216984, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f", 896, "4de88bd855155f2357e30c64a5ec6367dd1ef7fcac71a3b6755e0ff5eb698a30", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -43239,10 +42668,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -43279,7 +42708,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -43287,7 +42715,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -43302,7 +42729,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -43349,10 +42775,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 216824, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 896, "02f45be868c336dfba3daca0dd5bb98c236f5e2462e4d8cb318f2624a5ff5760", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 216984, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 896, "77a54feeb2e3e1e78e5d21e0f9a38620e8930ed96322e7c641098fd3969dc50c", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -43367,10 +42794,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -43407,7 +42834,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -43415,7 +42841,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -43430,7 +42855,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -43477,10 +42901,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 216824, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 896, "d28310a63dc79a08e945489fa394c4ef9df5e9dc166e741b098f416fc3ce4e3a", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 216984, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 896, "6dd592dcae0c2e8fcad78ddca9ec04879ce787be1167debe2bdbb8657f1d7dc9", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -43495,10 +42920,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -43535,7 +42960,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -43543,7 +42967,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -43558,7 +42981,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -43605,10 +43027,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 216824, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 896, "9ea4cafa90615ec8b6be24982637eb993419d0d7981d606a5a0ffa4e84fb129d", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 216984, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 896, "3dda856f910bce58b174bfe891f8cfe00826ec9930427ac03c3a72ecb8b3ecbe", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -43623,10 +43046,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -43663,7 +43086,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -43671,7 +43093,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -43686,7 +43107,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -43711,7 +43131,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 1024 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -43733,10 +43153,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 216584, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f", 768, "3f1869514e10520eec54b58bf7a6e9f8ce660c2c662e4402f0da4642a907ed92", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 216600, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f", 768, "4fe840d8ed8840ea59b230b5cc72e019daef839cb13b02abcbc85ee47bcb9754", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -43751,10 +43172,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -43791,7 +43212,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -43799,7 +43219,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -43814,7 +43233,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -43861,10 +43279,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 216584, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 768, "1076392ba39b1bba3683887dbc27fdbe006072e18ae062368cb1e4bd3b8699ce", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 216600, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 768, "8c9cb5052e1810977bb09cbf8490a9bbd523f3c9d594c391e7cbd6b21668cd46", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -43879,10 +43298,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -43919,7 +43338,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -43927,7 +43345,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -43942,7 +43359,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -43989,10 +43405,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 216584, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 768, "ee47b52ebe8beba788e8f5abe6960fc73be5cbd4c6250a0c693e483c62b469fd", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 216600, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 768, "8657274862f83ddf1c031bd3767342e05d357f15cbe9579097aae221369f4b7d", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -44007,10 +43424,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -44047,7 +43464,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -44055,7 +43471,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -44070,7 +43485,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -44117,10 +43531,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 216584, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 768, "31d0dce075c3f50f64538a66946390b929532fdb5fb9ac6c1a303cbc35913794", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 216600, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 768, "0cac5280ee597bfccc719993c7f3a04aa1c3306db481922d22f141a2fda81791", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -44135,10 +43550,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -44175,7 +43590,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -44183,7 +43597,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -44198,7 +43611,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -44223,7 +43635,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 1024 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -44245,10 +43657,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 151384, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f", 640, "dc473634c9da57fecd60c962475a62ef62f90bb6dba8964d9a695df9f841a73b", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 151448, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f", 640, "49ed9329080255d7d7dfd8926e40ce55102b05eff1ec4a302cba798261c1e52a", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -44263,10 +43676,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -44303,7 +43716,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -44311,7 +43723,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -44326,7 +43737,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -44373,10 +43783,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 151384, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "bccc093614a5cc5f8674868aeb8eb29338dda4a358eb2e6a85cd70f02b7a4df9", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 151448, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "eca147c07d5ce270e57cf83db557d6bc4e384876a776dd025d9af52caaa1c17e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -44391,10 +43802,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -44431,7 +43842,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -44439,7 +43849,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -44454,7 +43863,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -44501,10 +43909,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 151384, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 640, "f53879a5aed4848fd806ff4026a13227f6f52d1ab4d4e8d30ec6f01f4634bb70", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 151448, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 640, "a811661feae910a1198dc330f1ecf549bc38f89d74017163776cfd21ca2fbede", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -44519,10 +43928,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -44559,7 +43968,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -44567,7 +43975,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -44582,7 +43989,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -44607,7 +44013,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 256 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -44629,10 +44035,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 151384, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 640, "3d705a50dab96517b5f97901a3b030f51cab725795e79fca38376e96665bc664", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 151448, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 640, "afa1068f89495f40462f9cd960a99ad7acda322dd66632c02a1c5a1fb0453333", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -44647,10 +44054,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -44687,7 +44094,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -44695,7 +44101,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -44710,7 +44115,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -44735,7 +44139,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 256 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -44757,10 +44161,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256u2_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256u2_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 151384, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256u2_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f", 640, "874b84e931f973af09ddead340bc251cc8f2cf164c73e694a5869e54aa9c27c8", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256u2_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256u2_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 151448, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256u2_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f", 640, "3fe9134ec5299e8516ab1544ade686faf7887f62a23a8ffe1bde57eaa6cd9855", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -44775,10 +44180,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -44815,7 +44220,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -44823,7 +44227,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -44838,7 +44241,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -44885,10 +44287,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256u2_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256u2_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 151384, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256u2_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "09c955c621035e842f7946baa9d7c431081e58499ab1394c7f01deca930f0573", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256u2_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256u2_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 151448, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256u2_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "15b02b5ff92bbdf7f5668e58929cb75676e68dbbca63b30f646bdfd57c786cf9", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -44903,10 +44306,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -44943,7 +44346,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -44951,7 +44353,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -44966,7 +44367,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -45013,10 +44413,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256u2_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256u2_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 151384, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256u2_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 640, "f5d4c327255d5f7a5ff5d92ebab50d3eb9f585be9736280a98ed5734ef66704a", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256u2_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256u2_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 151448, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256u2_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 640, "18dceee0abf90ec7b76dfe61f7e0bc7dad09ccd398853119a3f2553754ed4dbc", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -45031,10 +44432,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -45071,7 +44472,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -45079,7 +44479,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -45094,7 +44493,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -45119,7 +44517,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 512 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -45141,10 +44539,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256u2_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256u2_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 151384, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256u2_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 640, "52b38f637e417f0b3035c3fc4ec6f1d74f363085c1429776523cceb1ccdb9338", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256u2_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256u2_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 151448, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256u2_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 640, "1fffb302b9c73c6ef33ad9eede6b63830ff3384de515b55a29c1dc232d9916a7", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -45159,10 +44558,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -45199,7 +44598,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -45207,7 +44605,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -45222,7 +44619,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -45247,7 +44643,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 512 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -45269,10 +44665,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 196248, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f", 896, "61095307fd380a8a527edaf4c8d4ffdb505b3fb42d802c754a2ec388e8424300", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 196248, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f", 896, "a5ff5c5231d93ce9715272178b4699ab1085f82a2b7621d872343e0f3f626c98", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -45287,10 +44684,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -45327,7 +44724,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 2 @@ -45335,7 +44731,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -45350,7 +44745,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -45397,10 +44791,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 196248, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 896, "e4ea2da9938b9f052de04c042f2840bad9c8daa27d7c70dbd4bc700b0bd72cb7", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 196248, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 896, "a2895d97cd26211bb3714a1ad919a2fb87b4bebbb9eafce8edc3e6bb1621779a", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -45415,10 +44810,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -45455,7 +44850,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 2 @@ -45463,7 +44857,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -45478,7 +44871,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -45525,10 +44917,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 196248, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 896, "a53591d7ce01b1583ef536ba489e17f6583dfc13becfeed412d4953c94325b3f", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 196248, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 896, "2ca55f92d11ed8b10f429929dd1246e91e25589eae6a2fbc9d4c5e37b33ab4a4", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -45543,10 +44936,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -45583,7 +44976,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 2 @@ -45591,7 +44983,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -45606,7 +44997,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -45631,7 +45021,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 512 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -45653,10 +45043,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 196248, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 896, "3d9202f84bd4bfd2812ada4ffba09dc197e68d4af88a0a6769e4ddc96e61af06", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 196248, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 896, "1870628d0c59ffe92c2dd22f42c52c87f14f4ad614fcc64d8be28c9b03268cc6", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -45671,10 +45062,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -45711,7 +45102,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 2 @@ -45719,7 +45109,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -45734,7 +45123,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -45759,7 +45147,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 512 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -45781,10 +45169,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512u2_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512u2_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 196248, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512u2_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f", 896, "a7914dae4dcaf2b3f6af4926d8cbba59d25bb5f25dfd6392b4b3789bd013764d", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512u2_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512u2_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 196248, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512u2_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f", 896, "55b8297bccfdb00c698b21aad6e52ac1100e0e92d1dbfee1bc04fc43b9cf727d", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -45799,10 +45188,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -45839,7 +45228,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 2 @@ -45847,7 +45235,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -45862,7 +45249,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -45909,10 +45295,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512u2_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512u2_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 196248, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512u2_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 896, "9faae1bf9d27f5fc1158eb785437c2ef40d87942114b190c7ecd76a00cecfb1e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512u2_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512u2_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 196248, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512u2_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 896, "3926328a24d9c0345c271e1c1abe8788f115f28ffda89d63f959f1eda0232327", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -45927,10 +45314,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -45967,7 +45354,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 2 @@ -45975,7 +45361,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -45990,7 +45375,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -46037,10 +45421,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512u2_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512u2_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 196248, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512u2_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 896, "6b2d9ed85052382c790be84ff830403b07213fd9ad3db9a7032d0a491e34d887", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512u2_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512u2_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 196248, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512u2_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 896, "d68a779a16ef7e5112e6cf9178ba4c90a9cafd33f3ac3cc473bded7b4f12b38c", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -46055,10 +45440,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -46095,7 +45480,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 2 @@ -46103,7 +45487,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -46118,7 +45501,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -46143,7 +45525,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 1024 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -46165,10 +45547,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512u2_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512u2_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 196248, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512u2_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 896, "043245228b674ed5148883de37d9f8333f337445a472c8edc21408bc8b6c53bd", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512u2_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512u2_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 196248, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512u2_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 896, "2de609606d699ad746d292251fe04f2bcfd725517405ca8584e54d4183e3c72f", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -46183,10 +45566,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -46223,7 +45606,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 2 @@ -46231,7 +45613,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -46246,7 +45627,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -46271,7 +45651,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 1024 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -46293,10 +45673,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 183408, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f", 512, "bc843d09fd317e10dcdc138f25495695e22243cbc46181dc1e0a1fa28f2b2b17", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 183440, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f", 512, "8e29c1dce23de3e3a78bb3a3de3624feb581f37141b2116ead5abb6dc8b8d756", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -46311,10 +45692,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -46351,7 +45732,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 2 @@ -46359,7 +45739,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -46374,7 +45753,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -46421,10 +45799,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 183408, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "a627d8d9c09226f5efc9f0c1d955daf73f89c85f2185ca8394a6a6eda1a4c6b3", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 183440, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "71fea993e95fdcad9b243119b403ade3224f3bb3f592b301690b63ce070bb3e7", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -46439,10 +45818,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -46479,7 +45858,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 2 @@ -46487,7 +45865,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -46502,7 +45879,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -46549,10 +45925,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 183408, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 512, "055d16699f1d2772d056bd5fd9b532dbe648483e7b7dccb7782de8bce274fe02", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 183440, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 512, "1d9ec1d73577ce1239a44fb4c42f38370e7b213751fac7026e285710065feab2", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -46567,10 +45944,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -46607,7 +45984,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 2 @@ -46615,7 +45991,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -46630,7 +46005,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -46677,10 +46051,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 183408, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 512, "d66b6ff562ada6a2e6a4d143df5f2fc3ab57eb728cdd8408855c965a0b9fa044", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 183440, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 512, "6d45a517f003dbda07c4561c3b2074e0534109f9e5ffacdd48c0c418b54adc62", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -46695,10 +46070,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -46735,7 +46110,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 2 @@ -46743,7 +46117,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -46758,7 +46131,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -46783,7 +46155,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 256 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -46805,10 +46177,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 183168, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f", 512, "ba2972805eedb9e2e16b986854683cd737f6f8a5bda400ee7101281e8ac5bc80", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 183312, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f", 512, "6bf7414821ee2bd06588ffd1105454750ea587b4d818367d51e9c0a0e1b6288f", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -46823,10 +46196,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -46863,7 +46236,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 1 @@ -46871,7 +46243,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -46886,7 +46257,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -46933,10 +46303,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 183168, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "28d7854ee9e3558aa1b02e0aad3ae71dcda3f4709b3125a603eab03112c4564a", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 183312, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "e03d90cd77399704927d22f3cb3078e2e6b83800aeb5868345abcd5ae999d21c", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -46951,10 +46322,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -46991,7 +46362,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 1 @@ -46999,7 +46369,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -47014,7 +46383,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -47061,10 +46429,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 183168, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 512, "e512489db11439a9e53e22eeae6bf6e9bb5b789ef28e6385a752fae6a559f648", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 183312, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 512, "f2ecc35a2129d90cb3a1bf9277385d3a25aead39553b7e7f0542a256da497e5e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -47079,10 +46448,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -47119,7 +46488,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 1 @@ -47127,7 +46495,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -47142,7 +46509,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -47189,10 +46555,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 183168, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 512, "bbab5a387838e88cccfb52aca9fc754f23ba8e91f67378785e4f400aeb577718", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 183312, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 512, "4a973cfde8c9634215edbbede3bdbdca99049e07aedbe40e4f0d70bb7d107fc7", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -47207,10 +46574,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -47247,7 +46614,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 1 @@ -47255,7 +46621,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -47270,7 +46635,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -47295,7 +46659,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 256 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -47317,10 +46681,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 183408, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f", 512, "d1cfca6eac35cc3fa54018222de415e1cae68b09f49f002ebe37c076dd050e02", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 183440, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f", 512, "65be3280bd48adfaa047f33bcd3da29f208aca32eba52d2155d00385e0a99db1", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -47335,10 +46700,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -47375,7 +46740,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 2 @@ -47383,7 +46747,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -47398,7 +46761,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -47445,10 +46807,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 183408, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "066bf729f9ff04f0ed6642372c278a73e95ff8e9de07c7d6824f1ab409c83bd6", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 183440, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "6d4a57d9cbfe434b5c3fa5ec7105366557b0409eeead03654552ee451e8c3345", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -47463,10 +46826,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -47503,7 +46866,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 2 @@ -47511,7 +46873,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -47526,7 +46887,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -47573,10 +46933,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 183408, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 512, "2be6865036d57ff7432dee5c9fbb21d5ed5aa5c6447dd086f5da4ec185e194b1", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 183440, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 512, "f4f6d78fbf0b9370f07da3d5054aad54596f132f6d7239e90f3255d362b1ba3e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -47591,10 +46952,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -47631,7 +46992,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 2 @@ -47639,7 +46999,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -47654,7 +47013,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -47701,10 +47059,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 183408, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 512, "28d6192bc83dc8f484f4e4c472bb7547f4209691dd666a0ec19f0ce3a63f0e08", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 183440, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 512, "1d135437b451de27407e1e3e2f2c94cae5319acc37d78e16467b91d5593dfef7", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -47719,10 +47078,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -47759,7 +47118,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 2 @@ -47767,7 +47125,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -47782,7 +47139,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -47807,7 +47163,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 512 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -47829,10 +47185,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 183168, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f", 512, "6b6a7a9a988a0682657382373124507d5de4e00fb24dbfcfebb6587f6ca360e1", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 183312, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f", 512, "323d7117879c4856e8303c6436e4260b4cf67488231b908df2e1b6fdd6f376e7", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -47847,10 +47204,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -47887,7 +47244,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 1 @@ -47895,7 +47251,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -47910,7 +47265,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -47957,10 +47311,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 183168, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "c2e23f95b7c43e076dedbac82d8fc42fb55d3d6dd0890b71e859152c07862a65", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 183312, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "9a3a54df273167d5f485cdd4e94b109885d31185aaf8c38cd6f1306fbebcfbe9", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -47975,10 +47330,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -48015,7 +47370,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 1 @@ -48023,7 +47377,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -48038,7 +47391,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -48085,10 +47437,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 183168, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 512, "0b59ac7add1da407e3c42dc07ac2a962c5860472af585c5ad089861f575d3bc2", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 183312, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 512, "ab992021edc350e300aaafd13a2647b99d717033df48915de033cfe1129db856", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -48103,10 +47456,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -48143,7 +47496,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 1 @@ -48151,7 +47503,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -48166,7 +47517,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -48213,10 +47563,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 183168, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 512, "3cca1432ea055440b6b6f23dbecb8f2124eb6a41f42a047d2640aa09fb0ac2b6", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f_cubin_len, 183312, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f", 512, "62d288482b3da6e09c46764ca604e47bc373e61b1ef64f70d51f985c7bfb94e9", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -48231,10 +47582,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -48271,7 +47622,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 9 , /* mNumStagesB */ 9 , /* mNumStagesMma */ 1 @@ -48279,7 +47629,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 9 , /* mNumStagesSmemSfB */ 9 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 9 , /* mNumStagesTmemSfB */ 9 , /* mNumStagesWorkId */ 3 @@ -48294,7 +47643,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -48319,7 +47667,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 512 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -48341,10 +47689,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f_cubin_len, 162208, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f", 512, "315dae74ce804f87ac379137eeb27d5ff9901feeb2d191bebb146946202677a5", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f_cubin_len, 162320, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f", 512, "86a4157a091c5f8bfd084c5d21ff497ae15b4d456e8f53630fe8ef98195e336d", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -48359,10 +47708,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 0 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -48399,7 +47748,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 1 @@ -48407,7 +47755,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -48422,7 +47769,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -48469,10 +47815,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 202480, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f", 640, "97c3632ffc4ba8b20c6cc2428d0c100a2e3d9ac4d8931f1f3475956836fbc918", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 202640, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f", 640, "d5ad91f85bca4a1626dc91840c080c0a3c33ca779167e95c4805eab520b85463", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -48487,10 +47834,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -48527,7 +47874,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -48535,7 +47881,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -48550,7 +47895,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -48597,10 +47941,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 202480, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "372d5ba0a83438939ef8b751a35e45bba828bb25549c60c2ca674e934f65c757", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 202640, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "5f1ba6239f0d92f906ca94935f2236a04bcd8255a36038f4486a302c045c2afe", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -48615,10 +47960,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -48655,7 +48000,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -48663,7 +48007,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -48678,7 +48021,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -48725,10 +48067,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 202480, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 640, "ddbf99702ca99490a5159b4561a6df1fe60b0042c1240d50def20a9145b6d00f", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 202640, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 640, "a6bde4cefa7444d67921e3bc58976f9a74ced058bd43170bd5fe0aa89608c428", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -48743,10 +48086,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -48783,7 +48126,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -48791,7 +48133,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -48806,7 +48147,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -48853,10 +48193,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 202480, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 640, "040746c657b79696a795ab1d496633ac244b23d1a81bc407ed7ff33c5bb4b789", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 202640, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 640, "9329e4104a13301d2dcc7ce1ed4f076eace049e25914bc6ff54eeba028b90437", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -48871,10 +48212,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -48911,7 +48252,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -48919,7 +48259,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -48934,7 +48273,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -48959,7 +48297,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 512 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -48981,10 +48319,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 202240, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f", 512, "caff45b58c0b5a649740d1fe7f90b7f64353d827a5e514acdc6d5930b6baea46", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 202256, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f", 512, "0733341d8eb04862927c527f4d85447f2fe327409241ec4130716da44edfb0a2", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -48999,10 +48338,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -49039,7 +48378,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -49047,7 +48385,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -49062,7 +48399,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -49109,10 +48445,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 202240, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "b156ac0ff09d9f197b72eaed5d59913554f3559db53b7c8ac146d28a77c8e5f3", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 202256, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "5dbd34e5ab514329e2aefda61b5a2bea7f0582baddd263e1926ed1e19c59d774", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -49127,10 +48464,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -49167,7 +48504,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -49175,7 +48511,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -49190,7 +48525,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -49237,10 +48571,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 202240, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 512, "96830e50e20d133687aa161e81ae11125ec3ed782eba2d75b0bd8609f70d9638", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 202256, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 512, "e6c882a43186ddf025da419f2a04ae45c8ee8352b2d294db7e0c8d11a785c2d6", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -49255,10 +48590,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -49295,7 +48630,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -49303,7 +48637,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -49318,7 +48651,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -49365,10 +48697,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 202240, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 512, "8fc310df8752500aa3673c6d725e28a9f91d1b39168532413f1da6c79731ef22", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 202256, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 512, "f9c1f126bf693a5dd8d08839c9b5618b6ffa59ab212a8e227d4fbefbd99c45b5", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -49383,10 +48716,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -49423,7 +48756,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -49431,7 +48763,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -49446,7 +48777,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -49471,7 +48801,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 512 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -49493,10 +48823,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 209656, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f", 640, "a8dbab304097be76a1dd20f7f164a96e933b8193c95d6d177cda048f5191df65", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 209816, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f", 640, "fd6a812279a8ba46eb54d090b36f4a754a66af1d25212e78d73bbbbb35c111ba", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -49511,10 +48842,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -49551,7 +48882,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 2 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -49559,7 +48889,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -49574,7 +48903,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(2) , /* mTileK */ 512 @@ -49621,10 +48949,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 209656, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 640, "e82d0423815eaa9d2248b5081de66531f287dd68cdbd1d09ab62da7afe62802d", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 209816, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 640, "f96be89c9b175fb8d261cd71e8f545c99300399289492d04fe01eecfc7e6a23b", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -49639,10 +48968,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -49679,7 +49008,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 2 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -49687,7 +49015,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -49702,7 +49029,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(2) , /* mTileK */ 512 @@ -49727,7 +49053,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 1024 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -49749,10 +49075,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 209656, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 640, "0505f3bf2802dda54ddbcc67d1c7f453f5721596432a7f1c0ebf27fb686673ac", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 209816, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 640, "a8c2b9a87305e4db9bd307b211ce9a7e1b8de669d42bc1cbb0472494062b27f7", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -49767,10 +49094,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -49807,7 +49134,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 2 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -49815,7 +49141,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -49830,7 +49155,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(2) , /* mTileK */ 512 @@ -49855,7 +49179,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 1024 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -49877,10 +49201,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 213752, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f", 640, "83293942af25740508913c5bf6ea3c4e1aa650ff7c0ba7a6086074be096d50a1", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 213912, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f", 640, "b11734bba44e6f5ed79d22dd39435b13822d5c77328b1fcc4012196e03e28600", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -49895,10 +49220,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -49935,7 +49260,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 3 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -49943,7 +49267,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -49958,7 +49281,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(2) , /* mTileK */ 512 @@ -50005,10 +49327,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 213752, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 640, "25bb92c1c037851f85d344af9d39da46b46acfa3518f43d249d51dd82085f0bf", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 213912, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 640, "2270c140cdec40bf2431760668882ff95c6b0fe7a0c973be4c1a2b8cc20e7ae6", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -50023,10 +49346,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -50063,7 +49386,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 3 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -50071,7 +49393,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -50086,7 +49407,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(2) , /* mTileK */ 512 @@ -50111,7 +49431,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 1536 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -50133,10 +49453,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 213752, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 640, "e66625d6b941a9b10381b4adcf8c7ccc1f7910cd10d4f643815b62bb35d6fd1c", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 213912, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 640, "49d964be5275b46c2c12714bb37cd6231592551373ea2a9ee224b19918d7ffeb", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -50151,10 +49472,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -50191,7 +49512,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 3 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -50199,7 +49519,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -50214,7 +49533,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(2) , /* mTileK */ 512 @@ -50239,7 +49557,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 1536 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -50261,10 +49579,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 217848, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f", 640, "c75668690056a6a6a77d91c015201a54d51bbed602b9b3578b6eeb4199a0ebee", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 218008, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f", 640, "0f9d41f79f0b633dc49b53969d065fd5c45b76bf23889b1c5fe38e87a040c3cb", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -50279,10 +49598,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -50319,7 +49638,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 4 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -50327,7 +49645,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -50342,7 +49659,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(2) , /* mTileK */ 512 @@ -50389,10 +49705,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 217848, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 640, "562301f96337ffda41c0f8996ca61356573747f5477a0384dd1296a8af5e0e51", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 218008, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 640, "ed0aa035dc9cd85be6daff01a91ca206df76da4cc225fa1d7cce96f15e0bb2b5", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -50407,10 +49724,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -50447,7 +49764,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 4 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -50455,7 +49771,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -50470,7 +49785,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(2) , /* mTileK */ 512 @@ -50495,7 +49809,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 2048 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -50517,10 +49831,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 217848, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 640, "e95892d7fc904375bbb9d613a6c7af163b731443be6aa235dc2c6a5ba13cb9bb", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 218008, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 640, "844ad567857930d54e3495dabe4ba7f5579e2dfcec7b00279d101211bcd1d32c", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -50535,10 +49850,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -50575,7 +49890,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 4 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -50583,7 +49897,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -50598,7 +49911,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(2) , /* mTileK */ 512 @@ -50623,7 +49935,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 2048 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -50645,10 +49957,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f_cubin_len, 162208, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f", 512, "ccab8de374f94d9710567d8bfbf25f96afd3ce5b75c1a963e46617e683272182", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f_cubin_len, 162320, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f", 512, "a6e8ad6d9deafbc1681352a2d7a1f1014e22936657050cde2fe7ad1aba04642a", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -50663,10 +49976,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 0 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -50703,7 +50016,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 1 @@ -50711,7 +50023,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -50726,7 +50037,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -50773,10 +50083,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 202480, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f", 640, "08aa921a6bb7083d0560622cc51c5e40f7c3db675094fea8d3b41b7469d9a876", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 202640, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f", 640, "d455ef510ce22e1e5d0da160d8fa843140837fab205b2ee344848e67c584dc5e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -50791,10 +50102,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -50831,7 +50142,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -50839,7 +50149,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -50854,7 +50163,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -50901,10 +50209,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 202480, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "ae713ef91a76db33c9498cc332365893657639df0d48c3f7cfe6d50d493a3927", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 202640, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "577cd363c3fe17e885d6e989221c2408c9af34f164be4fe257b66e8dca289207", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -50919,10 +50228,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -50959,7 +50268,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -50967,7 +50275,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -50982,7 +50289,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -51029,10 +50335,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 202480, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 640, "e5e905dc9c6ff0e9439475af39a308ed3142993505bc4a524887e9d854a92cec", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 202640, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 640, "61ab3c3eb6daa69463f1083f89dbbdee8cb194173b7a236a52c8216d4de57914", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -51047,10 +50354,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -51087,7 +50394,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -51095,7 +50401,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -51110,7 +50415,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -51157,10 +50461,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 202480, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 640, "3d8b757a79eb9795a428d6c0bfc00af3e3562965667fc9ffcaabc3c5c864fbd4", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 202640, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 640, "a193bafea5abf0e74bef83f59f93ab63d3319a7c106b4b479d58c10cb2f135ce", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -51175,10 +50480,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -51215,7 +50520,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -51223,7 +50527,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -51238,7 +50541,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -51263,7 +50565,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 1024 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -51285,10 +50587,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 202240, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f", 512, "593248a55435cba04ec97865ad5abe0ac8527a8288f68ef2ab5f6bd3b880fced", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f_cubin_len, 202256, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f", 512, "5e6eafbc9ad680c492939a0232e620663c296966bc4abdbbe168ee3210b1cdeb", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -51303,10 +50606,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -51343,7 +50646,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -51351,7 +50653,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -51366,7 +50667,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -51413,10 +50713,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 202240, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "0465744b62696d595fe4a8649bf6881322b926508799b42f3c849c7e3bd3d020", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 202256, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "10ce69b50d326f27acf8e8f4b693cfd93828d58126aaa6875af68eaa1660b870", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -51431,10 +50732,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -51471,7 +50772,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -51479,7 +50779,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -51494,7 +50793,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -51541,10 +50839,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 202240, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 512, "323497aad36a3e9c8330b994755aff48819934cb104042874f1c7662e70a0850", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 202256, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 512, "fa350271b7344b051e3272de089634f2da917e2bf0824eed396438ac3c83a917", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -51559,10 +50858,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -51599,7 +50898,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -51607,7 +50905,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -51622,7 +50919,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -51669,10 +50965,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 202240, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 512, "ca586c318580a1f594c7a3e54a0162515867f0c1a71998ef0a6f5d250c9a0903", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 202256, "bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 512, "0bc68323c7b397db47d89002662187052228e685a82b5225601587c0f6018d82", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -51687,10 +50984,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1050629) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -51727,7 +51024,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -51735,7 +51031,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -51750,7 +51045,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -51775,7 +51069,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 1024 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(2) +, /* mActType */ gemmGatedAct::ActType(3) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -51797,10 +51091,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f_cubin_len, 211480, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f", 512, "a70737484e57615237edbe8015cdd37d70ae1027357436564934e8a48a31e1ac", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f_cubin_len, 211608, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f", 512, "fc51c0a95cfd3e802b38209cdbf66dcf6a38b386d0c471353b2f19b5e29f6054", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -51815,10 +51110,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -51855,7 +51150,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 128 , /* mNumRegsPerThreadNonEpilogueWarp */ 56 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 8 , /* mNumStagesB */ 8 , /* mNumStagesMma */ 2 @@ -51863,7 +51157,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 8 , /* mNumStagesSmemSfB */ 8 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 8 , /* mNumStagesTmemSfB */ 8 , /* mNumStagesWorkId */ 3 @@ -51878,7 +51171,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -51925,10 +51217,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin_len, 211480, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f", 512, "00e98927aba1eb97333c7e40bec099f5b21cb7323b18e032fe1013993f5a568b", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin_len, 211608, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f", 512, "6afdb9bdba1a85243cd120e70fa559be922e164cf400d832b0888452f9087da3", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -51943,10 +51236,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -51983,7 +51276,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 128 , /* mNumRegsPerThreadNonEpilogueWarp */ 56 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 8 , /* mNumStagesB */ 8 , /* mNumStagesMma */ 2 @@ -51991,7 +51283,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 8 , /* mNumStagesSmemSfB */ 8 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 8 , /* mNumStagesTmemSfB */ 8 , /* mNumStagesWorkId */ 3 @@ -52006,7 +51297,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -52053,10 +51343,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin_len, 211480, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f", 512, "9071fcfb6f8c99cc66291f316f2c188002eb17570a112e2c7f3bb7b63eee5a66", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin_len, 211608, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f", 512, "7a3f93f23989911d91a1a4ab50371583bb8759ba59200ada92df327054e83c20", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -52071,10 +51362,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -52111,7 +51402,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 128 , /* mNumRegsPerThreadNonEpilogueWarp */ 56 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 8 , /* mNumStagesB */ 8 , /* mNumStagesMma */ 2 @@ -52119,7 +51409,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 8 , /* mNumStagesSmemSfB */ 8 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 8 , /* mNumStagesTmemSfB */ 8 , /* mNumStagesWorkId */ 3 @@ -52134,7 +51423,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -52181,10 +51469,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128u2_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128u2_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f_cubin_len, 211480, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128u2_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f", 512, "0763b619142b48d422f62bea32499757036195e42cf68faae65447400fb35600", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128u2_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128u2_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f_cubin_len, 211608, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128u2_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f", 512, "c09b3b2295588859844185cdef51519eb3ecac84bd9da0a30554a7c073e3081e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -52199,10 +51488,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -52239,7 +51528,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 128 , /* mNumRegsPerThreadNonEpilogueWarp */ 56 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 8 , /* mNumStagesB */ 8 , /* mNumStagesMma */ 2 @@ -52247,7 +51535,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 8 , /* mNumStagesSmemSfB */ 8 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 8 , /* mNumStagesTmemSfB */ 8 , /* mNumStagesWorkId */ 3 @@ -52262,7 +51549,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -52309,10 +51595,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128u2_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128u2_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin_len, 211480, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128u2_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f", 512, "bb446662943c7fce3ac41f7d2b7100be07aa54148c4e9b2a739119c9b9f4e4d3", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128u2_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128u2_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin_len, 211608, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128u2_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f", 512, "a8ad26148d175ab05b90dd1a2ed864478c3e73275fa3e97b3ab575ed4a038937", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -52327,10 +51614,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -52367,7 +51654,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 128 , /* mNumRegsPerThreadNonEpilogueWarp */ 56 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 8 , /* mNumStagesB */ 8 , /* mNumStagesMma */ 2 @@ -52375,7 +51661,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 8 , /* mNumStagesSmemSfB */ 8 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 8 , /* mNumStagesTmemSfB */ 8 , /* mNumStagesWorkId */ 3 @@ -52390,7 +51675,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -52437,10 +51721,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128u2_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128u2_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin_len, 211480, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128u2_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f", 512, "2f634a78b46c1ee24a12558fe7daa2939e138d8c6067fe54b90d7a6321ec4dd3", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128u2_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128u2_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin_len, 211608, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128u2_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f", 512, "150f8c85ce34d0b9cb1beb3cce40c6d03089e283146cbb72690576ec22a95a5f", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -52455,10 +51740,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -52495,7 +51780,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 128 , /* mNumRegsPerThreadNonEpilogueWarp */ 56 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 8 , /* mNumStagesB */ 8 , /* mNumStagesMma */ 2 @@ -52503,7 +51787,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 8 , /* mNumStagesSmemSfB */ 8 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 8 , /* mNumStagesTmemSfB */ 8 , /* mNumStagesWorkId */ 3 @@ -52518,7 +51801,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -52565,10 +51847,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 227792, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "43de8d4aa54da3caef1adf6fb423e7e2af3655f50c97cb4748434b6b0164bd1d", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 227984, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "8f76f0096f094e61312a1e8d51ce214a0f49e99002c9b69fcbe3e0e944ae1f37", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -52583,10 +51866,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -52623,7 +51906,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -52631,7 +51913,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -52646,7 +51927,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -52693,10 +51973,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 227792, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 384, "9097a18a4006ff2c299c88bbfda9cfc0b00acaeccee4264044fe5f0995f22270", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 227984, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 384, "7d614055d7f11311509b2fbbf7242ef282e03e0e5c1c2ef8d5de7b097b6b58c7", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -52711,10 +51992,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -52751,7 +52032,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -52759,7 +52039,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -52774,7 +52053,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -52821,10 +52099,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 227792, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 384, "400d0ec04047ada963106e6976baebdf9d3091fc4c2c941d597a514e83cfa51a", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 227984, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 384, "80cf474086aa5b979ceb07f952810ad94cf47b70305ec21f8fcf02c87a0d5738", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -52839,10 +52118,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -52879,7 +52158,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -52887,7 +52165,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -52902,7 +52179,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -52949,10 +52225,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 227552, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "2f055fe6118b6f70d80a0558a3f0487e07ac6745798443b6ff9e90fc2eadc363", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 227600, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "39245cc67700b334e47c548603fa467a4556d276c4e6390194d5622ae476cbda", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -52967,10 +52244,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -53007,7 +52284,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 1 @@ -53015,7 +52291,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -53030,7 +52305,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -53077,10 +52351,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 227552, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 384, "e11d1462994e4ae7dd4677ff535cccbc14ef21e4f29ffe325e8c65fe2d39d374", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 227600, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 384, "2266fb7136ce1b15805f753f65f58efa66a3a1c8176ccc9f383288d39485f128", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -53095,10 +52370,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -53135,7 +52410,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 1 @@ -53143,7 +52417,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -53158,7 +52431,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -53205,10 +52477,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 227552, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 384, "5fbd2616583bfdadfd1c68d71e43191d22b99061589422da98db4d00ff3d91c3", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 227600, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 384, "d7320c17701cd6fe804d88afbd5c75717148f510ce1aebab28b2ead935ce89d3", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -53223,10 +52496,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -53263,7 +52536,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 1 @@ -53271,7 +52543,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -53286,7 +52557,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -53333,10 +52603,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 227792, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "7c5291664e93b602262d8ab54d2dc04c8310933a7759456b07e23a75d61574b5", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 227984, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "ddf919e2754b6656a8073293f7f76bf35e71f11f343a25892022daade06ef55c", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -53351,10 +52622,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -53391,7 +52662,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -53399,7 +52669,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -53414,7 +52683,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -53461,10 +52729,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 227792, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 384, "97bbecc2eddc0ba9195b36bb28956dea946bf6e771beeee5bfd43fe736d4f2e8", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 227984, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 384, "49c8cb5b367c213d2e46b0a058e076081a4ac3a04312234879159a36463935e2", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -53479,10 +52748,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -53519,7 +52788,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -53527,7 +52795,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -53542,7 +52809,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -53589,10 +52855,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 227792, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 384, "93dfc51b23a061243f1721985725ba2edfe1f5cf5882ee24f9744beee28f6290", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 227984, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 384, "d07e26cf4e81049355cc54424b570fc8ca0c5db5c8b4b5e4352f6a7fa3d03893", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -53607,10 +52874,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -53647,7 +52914,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -53655,7 +52921,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -53670,7 +52935,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -53717,10 +52981,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 227552, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "ee60caaac9262b26658bbe114dd34fbd30f180c5c30bca894177ffe29fcf2cc6", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 227600, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "cab262e5c0325a9e883e1009861383edc174b3188b82bbd7737efabde477e002", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -53735,10 +53000,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -53775,7 +53040,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 1 @@ -53783,7 +53047,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -53798,7 +53061,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -53845,10 +53107,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 227552, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 384, "f05dc1b23a2800ca260d0e518df40f94ed7af70d33df7b9926b85c896b348ff6", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 227600, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 384, "36a9a420bf4979f362dd6cd6ebaaa1c716b4e5d4d66cfca88673df202fb12273", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -53863,10 +53126,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -53903,7 +53166,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 1 @@ -53911,7 +53173,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -53926,7 +53187,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -53973,10 +53233,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 227552, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 384, "ee2030bf4a4c6ec1e065ca0def02d25d63a26d88d1a1235b6b4900531c9f6ac8", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 227600, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 384, "4a3fcf2da0bc5c5e77f9bfa9593570a7fc4a8d4da9c893014d1b67e5b0b182cb", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -53991,10 +53252,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -54031,7 +53292,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 1 @@ -54039,7 +53299,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -54054,7 +53313,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -54101,10 +53359,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f_cubin_len, 212472, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f", 512, "90349fb4641ae963ad00754d02acbbf42ba486caa05f75775e05367328cd7e06", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f_cubin_len, 212632, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f", 512, "76d4d29c78f890ddbb84efaedd7f5ad78c6ddb87e800326c47c195a084ac523b", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -54119,10 +53378,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -54159,7 +53418,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 128 , /* mNumRegsPerThreadNonEpilogueWarp */ 56 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 7 , /* mNumStagesB */ 7 , /* mNumStagesMma */ 2 @@ -54167,7 +53425,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 7 , /* mNumStagesSmemSfB */ 7 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 7 , /* mNumStagesTmemSfB */ 7 , /* mNumStagesWorkId */ 3 @@ -54182,7 +53439,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -54229,10 +53485,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin_len, 212472, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f", 512, "5d4fbf4ae19a8b991651d1d5ae251896e50f72b38ced3a33154f143f581e9da3", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin_len, 212632, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f", 512, "03772bb38101eb93b72db615f5b2f5a33942df0af665b37f229b34f72b67ae5f", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -54247,10 +53504,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -54287,7 +53544,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 128 , /* mNumRegsPerThreadNonEpilogueWarp */ 56 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 7 , /* mNumStagesB */ 7 , /* mNumStagesMma */ 2 @@ -54295,7 +53551,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 7 , /* mNumStagesSmemSfB */ 7 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 7 , /* mNumStagesTmemSfB */ 7 , /* mNumStagesWorkId */ 3 @@ -54310,7 +53565,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -54357,10 +53611,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin_len, 212472, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f", 512, "0205e9ab961c5343b67ad356ca8c29859f08c6064b9ba8ffab69cb2f9430db95", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin_len, 212632, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f", 512, "946fe09d34006b24361bcaf97718bd15a74cad663b31ce93d5d1d4dfcdd29fe0", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -54375,10 +53630,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -54415,7 +53670,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 128 , /* mNumRegsPerThreadNonEpilogueWarp */ 56 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 7 , /* mNumStagesB */ 7 , /* mNumStagesMma */ 2 @@ -54423,7 +53677,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 7 , /* mNumStagesSmemSfB */ 7 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 7 , /* mNumStagesTmemSfB */ 7 , /* mNumStagesWorkId */ 3 @@ -54438,7 +53691,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -54485,10 +53737,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128u2_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128u2_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f_cubin_len, 212472, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128u2_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f", 512, "4cb0abb6fe46c96d2f7c3d52eb7da9069b586658fd61657cf153219dd44815a0", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128u2_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128u2_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f_cubin_len, 212632, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128u2_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f", 512, "7077c6bc9f285fa109bda99fa5d2ee717fb5d80bead361d82ab5f581e410e8ac", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -54503,10 +53756,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -54543,7 +53796,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 128 , /* mNumRegsPerThreadNonEpilogueWarp */ 56 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 7 , /* mNumStagesB */ 7 , /* mNumStagesMma */ 2 @@ -54551,7 +53803,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 7 , /* mNumStagesSmemSfB */ 7 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 7 , /* mNumStagesTmemSfB */ 7 , /* mNumStagesWorkId */ 3 @@ -54566,7 +53817,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -54613,10 +53863,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128u2_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128u2_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin_len, 212472, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128u2_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f", 512, "94376e795717fc6fb5368fe36c2c1a7890539eeb9a69bbd884fe0b125ea661a0", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128u2_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128u2_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin_len, 212632, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128u2_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f", 512, "bc52798304db4812a0b753052fc106b1a68c187d0b33c00477f2e964314e95bc", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -54631,10 +53882,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -54671,7 +53922,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 128 , /* mNumRegsPerThreadNonEpilogueWarp */ 56 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 7 , /* mNumStagesB */ 7 , /* mNumStagesMma */ 2 @@ -54679,7 +53929,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 7 , /* mNumStagesSmemSfB */ 7 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 7 , /* mNumStagesTmemSfB */ 7 , /* mNumStagesWorkId */ 3 @@ -54694,7 +53943,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -54741,10 +53989,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128u2_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128u2_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin_len, 212472, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128u2_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f", 512, "4908ecc8a0e4f1f9d434957a1f73e2b259fb74d98e60439dd15c13ba3eea9220", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128u2_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128u2_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin_len, 212632, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128u2_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f", 512, "65f82e2bf3233efd02611a1ae7fea44c55f20ea1d0f502c73b0d922e45b2b9c0", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -54759,10 +54008,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -54799,7 +54048,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 128 , /* mNumRegsPerThreadNonEpilogueWarp */ 56 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 7 , /* mNumStagesB */ 7 , /* mNumStagesMma */ 2 @@ -54807,7 +54055,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 7 , /* mNumStagesSmemSfB */ 7 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 7 , /* mNumStagesTmemSfB */ 7 , /* mNumStagesWorkId */ 3 @@ -54822,7 +54069,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -54869,10 +54115,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_eW8_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_eW8_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f_cubin_len, 221656, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_eW8_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f", 640, "5854b224599ed3555cc4f0e961f6345b4c47b99d828a3c3983d213517a1bc7cc", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_eW8_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_eW8_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f_cubin_len, 221848, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_eW8_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f", 640, "fdc9014b785bcfac8cfd0e26b78b17c04cb9741c6e4b3b385856db58ae7305d6", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -54887,10 +54134,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -54927,7 +54174,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 128 , /* mNumRegsPerThreadNonEpilogueWarp */ 56 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -54935,7 +54181,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -54950,7 +54195,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -54997,10 +54241,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_eW8_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_eW8_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin_len, 221656, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_eW8_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f", 640, "f016620bb767d01e5169f8e215963ebb9b996b5ec2e7e1d729080660818bcfda", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_eW8_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_eW8_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin_len, 221848, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_eW8_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f", 640, "2e92fe6a367588a5d8c92808e8ef1efdd97df734cbf25b59cbca3d8dabe2ba32", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -55015,10 +54260,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -55055,7 +54300,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 128 , /* mNumRegsPerThreadNonEpilogueWarp */ 56 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -55063,7 +54307,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -55078,7 +54321,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -55125,10 +54367,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_eW8_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_eW8_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin_len, 221656, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_eW8_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f", 640, "5a7bf3e84a405467eebb3cb34c4d7cb03cd9fae9e9948ab4f69e5e4f2abe5d2a", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_eW8_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_eW8_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin_len, 221848, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_eW8_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f", 640, "61dc36f33872049253d07fb80460cb5f698979981e0a6f2b8644c430aab03e5a", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -55143,10 +54386,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -55183,7 +54426,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 128 , /* mNumRegsPerThreadNonEpilogueWarp */ 56 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -55191,7 +54433,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -55206,7 +54447,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -55253,10 +54493,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128u2_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_eW8_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128u2_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_eW8_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f_cubin_len, 221656, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128u2_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_eW8_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f", 640, "997f88bfe5f2a0d521006714e239f1e2f89ed0571c400a3d99d982f779560a35", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128u2_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_eW8_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128u2_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_eW8_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f_cubin_len, 221848, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128u2_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_eW8_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f", 640, "7e2b4908512566eeeaed1597de374d362963fe051d87561731a06683a22c9541", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -55271,10 +54512,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -55311,7 +54552,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 128 , /* mNumRegsPerThreadNonEpilogueWarp */ 56 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -55319,7 +54559,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -55334,7 +54573,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -55381,10 +54619,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128u2_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_eW8_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128u2_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_eW8_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin_len, 221656, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128u2_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_eW8_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f", 640, "f4946affc9ef27c28d47db06e1ce5acfa9c73ada0c4ef75d7d42987c5d32551a", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128u2_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_eW8_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128u2_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_eW8_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin_len, 221848, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128u2_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_eW8_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f", 640, "ad9eef10e895de07089df2a669414eb185839e868bfa85b0a8796dfa4eda0921", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -55399,10 +54638,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -55439,7 +54678,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 128 , /* mNumRegsPerThreadNonEpilogueWarp */ 56 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -55447,7 +54685,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -55462,7 +54699,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -55509,10 +54745,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128u2_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_eW8_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128u2_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_eW8_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin_len, 221656, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128u2_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_eW8_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f", 640, "e47fe16c29d9d54a098c4e0ffd8412b1c3f38ce274d347a59c55c4b03619108b", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128u2_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_eW8_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128u2_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_eW8_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin_len, 221848, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128u2_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_eW8_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f", 640, "24e99ea5bd232c97435d1b68f404f2e5d986668f5073b2035ebcd93c8fe6c975", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -55527,10 +54764,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -55567,7 +54804,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 128 , /* mNumRegsPerThreadNonEpilogueWarp */ 56 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -55575,7 +54811,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -55590,7 +54825,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -55637,10 +54871,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 213424, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "ad5b6d096ff7057a8995754ee6ab2a39c4648b9a645c0eeefcded0c3a57d82f0", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 213648, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "37cf107cd46d1eab1a55a1bcdd54b60c63b21964fa2c79bfc0942247235d6dfb", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -55655,10 +54890,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -55695,7 +54930,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -55703,7 +54937,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -55718,7 +54951,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -55765,10 +54997,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 213424, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 384, "c261b7be96cff90e92ddb5bfa2b6ac5b9c116db8e6858a1629e77a32815341e1", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 213648, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 384, "56d855c64f058519843810ecc01b67b5870da43b227ecd480ad673bc7fa19a26", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -55783,10 +55016,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -55823,7 +55056,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -55831,7 +55063,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -55846,7 +55077,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -55893,10 +55123,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 213424, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 384, "926a534fe64ecae6375a778c32b8936c090d92c28626225a3f9818c3f2eb299d", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 213648, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 384, "c1e2d42ecaa83eddd697b58768f3213e74674ba4c3fb25d75fa24ec9bd86dec8", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -55911,10 +55142,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -55951,7 +55182,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -55959,7 +55189,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -55974,7 +55203,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -56021,10 +55249,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 213184, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "ed30e114c78653f0749be56c3454ddfee5414a82ba41db47dfd6800cd107b8ba", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 213264, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "e7610fcd50a47baedf228bc2a5ae1275112b24d887dee5cc986f8a6f14b740a4", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -56039,10 +55268,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -56079,7 +55308,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -56087,7 +55315,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -56102,7 +55329,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -56149,10 +55375,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 213184, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 384, "df3a177268a2c5b1040fb97f27f07f5b6f10ee2de9e7ffb2dd4027cf9f548e44", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 213264, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 384, "4e807026b0aa8b698a50227a3f7943b07656ee840b130780771719a39849257c", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -56167,10 +55394,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -56207,7 +55434,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -56215,7 +55441,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -56230,7 +55455,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -56277,10 +55501,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 213184, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 384, "93a2999454802899f105c696498f67eae86b7a56fbdf7cad5f76dc955fea7612", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 213264, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 384, "9085c9f40f11b2a693022c9fca020ae3c9dc603992c2ad6cd5b975a9c390323d", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -56295,10 +55520,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -56335,7 +55560,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -56343,7 +55567,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -56358,7 +55581,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -56405,10 +55627,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 213424, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "2516c8ed533cd0116dc4f68d20c27ceb7d6220ccbc2d1462495344c76e525c44", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 213648, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "5fc2e7e21046490b05d9af031aefc0951e439372b8781ac89079628dbfabaa49", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -56423,10 +55646,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -56463,7 +55686,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -56471,7 +55693,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -56486,7 +55707,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -56533,10 +55753,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 213424, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 384, "e04d625a7adfd66b394b46e720507612042a519db10d2e21dc18bdd5e90d6260", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 213648, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 384, "a7eb061f80e9a9de17ee0143ec8575bd495bf8918737153fa86bc967357487eb", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -56551,10 +55772,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -56591,7 +55812,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -56599,7 +55819,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -56614,7 +55833,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -56661,10 +55879,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 213424, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 384, "5232890fc8af645c27f0e334e5cd650edfb54ffc6da764889a1371b6041c6119", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 213648, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 384, "d876bb0bad5856bb485155efd9452ab69002f2dd19cae0e12bd27eddc501ab50", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -56679,10 +55898,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -56719,7 +55938,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -56727,7 +55945,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -56742,7 +55959,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -56789,10 +56005,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 213184, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "eb87e1ac053c5997b64b39c0ee7f72872279706716518c27478ecf77e7a5dcfe", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 213264, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "9e4d2fd79819ff453e8d0e9001d48f7dd613b490050ce6038e0d350ce778cbbd", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -56807,10 +56024,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -56847,7 +56064,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -56855,7 +56071,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -56870,7 +56085,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -56917,10 +56131,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 213184, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 384, "d03d2dc0e38bb6ea1a80278f94b094dc01c1140c6b11247539771a41c6044c10", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 213264, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 384, "79aec59ac0d1b0ee3164f5380d097aef638b2fe9804b2987c7b7b7d3b38815a1", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -56935,10 +56150,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -56975,7 +56190,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -56983,7 +56197,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -56998,7 +56211,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -57045,10 +56257,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 213184, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 384, "93175de8ce110ff2659eb87428766304a4c24ce50337d13a460ddbeffb48d119", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 213264, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 384, "391deb3926e48e02d7430b62ff2eda73879708ad737af9510de4df0959a08499", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -57063,10 +56276,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -57103,7 +56316,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -57111,7 +56323,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -57126,7 +56337,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -57173,10 +56383,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f_cubin_len, 218552, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f", 512, "85d113dd205839b15c2821e84b4c75bf461e69b14cd70a7c0830673828693903", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f_cubin_len, 218776, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f", 512, "a12ebc1ebd821213c4dee283035d6fa83d431b2c30e0597483c36c3adbb88e7a", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -57191,10 +56402,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -57231,7 +56442,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 128 , /* mNumRegsPerThreadNonEpilogueWarp */ 56 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -57239,7 +56449,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -57254,7 +56463,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -57301,10 +56509,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin_len, 218552, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f", 512, "27f17231e30d6e04d1915f308b7e060a8b48048de132e00288becfec60b92544", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin_len, 218776, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f", 512, "31a1f45c0a05a017d0cfd239b803ac09b76590dc11da7cae293aaa5c3bde3296", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -57319,10 +56528,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -57359,7 +56568,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 128 , /* mNumRegsPerThreadNonEpilogueWarp */ 56 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -57367,7 +56575,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -57382,7 +56589,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -57429,10 +56635,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin_len, 218552, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f", 512, "9d4c32cbfec3a3d47148efd47b508e20da1f38f204358ca469d9aade9bc8aee0", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin_len, 218776, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f", 512, "cac912408944b824f9c609f91900eca3896c8d3f58c3db0853f8383014c81ed6", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -57447,10 +56654,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -57487,7 +56694,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 128 , /* mNumRegsPerThreadNonEpilogueWarp */ 56 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -57495,7 +56701,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -57510,7 +56715,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -57557,10 +56761,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256u2_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256u2_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f_cubin_len, 218552, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256u2_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f", 512, "4d689f513fb461206a44e0134681051438ed3ca21d46a96c6bf94d1e2c44623e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256u2_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256u2_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f_cubin_len, 218776, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256u2_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f", 512, "857d48b24d6eee9f9182b8126072dfb795f86cb03e4d6c7f79355606eeabbeed", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -57575,10 +56780,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -57615,7 +56820,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 128 , /* mNumRegsPerThreadNonEpilogueWarp */ 56 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -57623,7 +56827,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -57638,7 +56841,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -57685,10 +56887,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256u2_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256u2_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin_len, 218552, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256u2_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f", 512, "fe0b6f952b2d088eff842e8c9445bcf7de2a6801d55f435ed792213c0decf793", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256u2_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256u2_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin_len, 218776, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256u2_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f", 512, "aa4a8ee86a152a34253e141ef00d0ff4e03b8ddaf38b3f54777a0266d71a48de", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -57703,10 +56906,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -57743,7 +56946,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 128 , /* mNumRegsPerThreadNonEpilogueWarp */ 56 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -57751,7 +56953,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -57766,7 +56967,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -57813,10 +57013,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256u2_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256u2_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin_len, 218552, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256u2_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f", 512, "5127d79c1e6a552ee68df7b6328b692226c1da351484c7a5dd4d55b626e394f7", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256u2_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256u2_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f_cubin_len, 218776, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256u2_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f", 512, "077a992d85bf706fb7077cda80522956e818b51e5167e6f60a999af8789dfc71", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -57831,10 +57032,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -57871,7 +57072,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 128 , /* mNumRegsPerThreadNonEpilogueWarp */ 56 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -57879,7 +57079,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -57894,7 +57093,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -57941,10 +57139,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 214480, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "7d1542be66203892263ed283a46647be334fe8dba7987719e0e405de55e9125b", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 214672, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "5b261a22492fce7866eced26de55c54eeb0809b7bff68af57ccc042cb571e851", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -57959,10 +57158,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -57999,7 +57198,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -58007,7 +57205,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -58022,7 +57219,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -58069,10 +57265,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 214480, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 384, "198dc2dfc8da682c45ae262ac358fa22208a119364547dd8f19eef54140a5d57", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 214672, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 384, "d100e6c020454b2778d3aa48d37e7add2ac953b18dee78c11bb46248319378df", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -58087,10 +57284,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -58127,7 +57324,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -58135,7 +57331,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -58150,7 +57345,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -58197,10 +57391,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 214480, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 384, "4680ccc53024fe35b421fcf59aff6d9b50d941fc3a15f4010f446fa5ebd7f0cf", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 214672, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 384, "a57a01790bedb57d817f0d3fcf757a76e76d506adc4ca96bdbe17a0518de498d", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -58215,10 +57410,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -58255,7 +57450,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -58263,7 +57457,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -58278,7 +57471,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -58325,10 +57517,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 214240, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 256, "471541e8e16af63f65c4e12394adf04047a598f2437338b5354387e98f89cb44", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 214288, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 256, "b41ba6ad67e121f7f585d846cfa115e68b5f1a66a277503f559fa44734c51aec", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -58343,10 +57536,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -58383,7 +57576,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 1 @@ -58391,7 +57583,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -58406,7 +57597,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -58453,10 +57643,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 214240, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 256, "2f32653fb93732b7688bc6d15ccea224676ecabac1ccd599c1a639f651789bd6", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 214288, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 256, "da869d348ad807a58b05e17e15d7161a2aae734d298fd1fcedeab0f29d9e7a6e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -58471,10 +57662,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -58511,7 +57702,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 1 @@ -58519,7 +57709,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -58534,7 +57723,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -58581,10 +57769,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 214240, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 256, "6c78d04ec980020c2a0b58293a3c34bcf1b6f703a019126cfe36ce0c6800df6f", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 214288, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 256, "614e84f7ae5b54079c93f17a3eb5cf9646fa16cd9fc148cae71e3d8d34ff8114", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -58599,10 +57788,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -58639,7 +57828,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 1 @@ -58647,7 +57835,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -58662,7 +57849,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -58709,10 +57895,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 214480, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "6ea0fd45712992e2dcb61600e75c72a009c99215e82353c74666044f8cb6da34", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 214672, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "7d2f20893e0a8290d0550f4082eaa70a13ce896a16da3007cb5c3d3e4b2f6f5c", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -58727,10 +57914,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -58767,7 +57954,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -58775,7 +57961,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -58790,7 +57975,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -58837,10 +58021,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 214480, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 384, "6add274a7a50230dea02e4dcefdecaca42008d31951c63e1ed7d8bec0e3b6444", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 214672, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 384, "7f7882e82b41eb02631f79ef08d5f602d1ae1a1a0f555badaee7a21debb1d351", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -58855,10 +58040,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -58895,7 +58080,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -58903,7 +58087,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -58918,7 +58101,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -58965,10 +58147,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 214480, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 384, "5244f2b4ac8b5d1817157eab327432101d0b6837ecb7b11b6bdb7d31d04efa6e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 214672, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 384, "2aeb1112f5aa9c28c2dcb15c07a8750e14c397da6723f50a4167c4c3632dee66", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -58983,10 +58166,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -59023,7 +58206,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -59031,7 +58213,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -59046,7 +58227,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -59093,10 +58273,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 214240, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 256, "33f7e9e4b2ae916c8cdab7a7a90ca14175716b410a400be14dceea501218caa7", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 214288, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f", 256, "2bfaf6596001e5cee1146b9d505637ec817a747467e86083133ba01f138cef3d", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -59111,10 +58292,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -59151,7 +58332,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 1 @@ -59159,7 +58339,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -59174,7 +58353,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -59221,10 +58399,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 214240, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 256, "36952441fb8fc16979cca9962f16762d79455262c3ed3a888f14a905a38535bd", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 214288, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 256, "340995ff338f29ba51877545102e40f17e675c2abeb004c96721e493c259e626", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -59239,10 +58418,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -59279,7 +58458,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 1 @@ -59287,7 +58465,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -59302,7 +58479,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -59349,10 +58525,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 214240, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 256, "8b2c05e69d4044f9972c314c8bac9201452cb6526fa4891e4f87293b33043af6", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f_cubin_len, 214288, "bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f", 256, "bd71e58feb4eaadc7a21b8f48e13856a5429488a6778e446b3c0bd856ae9f076", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -59367,10 +58544,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(3) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -59407,7 +58584,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 1 @@ -59415,7 +58591,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -59430,7 +58605,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -59477,10 +58651,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_t128x128x128_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x128x128_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f_cubin_len, 190592, "bmm_E4m3_E4m3E4m3_Fp32_t128x128x128_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f", 512, "4bf0f2821ef43ec0f034dbe71e3f37b9465e69ad61474d8c15312253a2fafd65", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_t128x128x128_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x128x128_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f_cubin_len, 190640, "bmm_E4m3_E4m3E4m3_Fp32_t128x128x128_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f", 512, "2e9ce134618efa3af5bf0a262c6a4f3f6ca0418e5ddb62df58e67e977cc9e66e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -59495,10 +58670,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 64 @@ -59535,7 +58710,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 152 , /* mNumRegsPerThreadNonEpilogueWarp */ 80 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 4 @@ -59543,7 +58717,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -59558,7 +58731,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -59605,10 +58777,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_t128x128x128_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x128x128_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f_cubin_len, 190304, "bmm_E4m3_E4m3E4m3_Fp32_t128x128x128_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f", 512, "c3af5aa4d4e1f8fdba65d5685e29a7388aac4d19759b913c518f7991218a40fd", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_t128x128x128_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x128x128_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f_cubin_len, 190512, "bmm_E4m3_E4m3E4m3_Fp32_t128x128x128_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f", 512, "2d78d641c3f5ce6db17758e95bfb385fed331c52c99736571275f37e14a36d85", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -59623,10 +58796,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 64 @@ -59663,7 +58836,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 152 , /* mNumRegsPerThreadNonEpilogueWarp */ 80 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -59671,7 +58843,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -59686,7 +58857,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -59733,10 +58903,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_t128x128x128u2_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x128x128u2_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f_cubin_len, 190592, "bmm_E4m3_E4m3E4m3_Fp32_t128x128x128u2_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f", 512, "6a3e0455f9c716ae5106999083ecf7f6c26d18583f3db06ffc004f74931a22c4", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_t128x128x128u2_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x128x128u2_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f_cubin_len, 190640, "bmm_E4m3_E4m3E4m3_Fp32_t128x128x128u2_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f", 512, "693e46559617e718787478386e64aaa2e724e0d08737512d9865422ab53fa4a4", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -59751,10 +58922,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 64 @@ -59791,7 +58962,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 152 , /* mNumRegsPerThreadNonEpilogueWarp */ 80 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 4 @@ -59799,7 +58969,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -59814,7 +58983,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -59861,10 +59029,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_t128x128x128u2_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x128x128u2_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f_cubin_len, 190304, "bmm_E4m3_E4m3E4m3_Fp32_t128x128x128u2_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f", 512, "fcde425af486134b2a5a7bbf0f02a1ecbf4053fc14b2830bc723e88fe01bf0e1", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_t128x128x128u2_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x128x128u2_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f_cubin_len, 190512, "bmm_E4m3_E4m3E4m3_Fp32_t128x128x128u2_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f", 512, "0cf25b4921cc625dcf8969549bdc3f98c6d9374385ce00c36d1814ee37ec2897", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -59879,10 +59048,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 64 @@ -59919,7 +59088,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 152 , /* mNumRegsPerThreadNonEpilogueWarp */ 80 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -59927,7 +59095,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -59942,7 +59109,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -59989,10 +59155,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_t128x16x128_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x16x128_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f_cubin_len, 119216, "bmm_E4m3_E4m3E4m3_Fp32_t128x16x128_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f", 512, "b2880589a6d3bfb2310d64fdf327ca9abc89625f8e5be386b644f48c3e078405", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_t128x16x128_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x16x128_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f_cubin_len, 119216, "bmm_E4m3_E4m3E4m3_Fp32_t128x16x128_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f", 512, "1122f6b890a381a4c89f559f1f705eae99e60a08e90ac30b9740a20bf7e27457", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -60007,10 +59174,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 64 @@ -60047,7 +59214,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 160 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 4 @@ -60055,7 +59221,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -60070,7 +59235,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -60117,10 +59281,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_t128x16x128_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x16x128_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f_cubin_len, 118928, "bmm_E4m3_E4m3E4m3_Fp32_t128x16x128_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f", 512, "39212d57a844768d1211c3d3b5919c489c4cc3effd4cbf4ab2c04c7ad0abcc00", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_t128x16x128_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x16x128_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f_cubin_len, 119088, "bmm_E4m3_E4m3E4m3_Fp32_t128x16x128_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f", 512, "fad190b3f3d5bfb8afe036f0191034dc013c42f416c389d884268fac776acb55", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -60135,10 +59300,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 64 @@ -60175,7 +59340,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 160 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -60183,7 +59347,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -60198,7 +59361,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -60245,10 +59407,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_t128x16x128u2_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x16x128u2_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f_cubin_len, 119216, "bmm_E4m3_E4m3E4m3_Fp32_t128x16x128u2_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f", 512, "ce30ca817e8f6b0648b190bff72030bef689b7978c67c024a5f3111e5432c70a", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_t128x16x128u2_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x16x128u2_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f_cubin_len, 119216, "bmm_E4m3_E4m3E4m3_Fp32_t128x16x128u2_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f", 512, "bd93e2de6838acf1452469e616e630aec20516501b208f59e3954f35ca2fdf67", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -60263,10 +59426,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 64 @@ -60303,7 +59466,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 160 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 4 @@ -60311,7 +59473,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -60326,7 +59487,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -60373,10 +59533,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_t128x16x128u2_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x16x128u2_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f_cubin_len, 118928, "bmm_E4m3_E4m3E4m3_Fp32_t128x16x128u2_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f", 512, "731474663a6905b3685a15974df7700a1d93be3b877d8e31538b8811303e9e85", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_t128x16x128u2_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x16x128u2_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f_cubin_len, 119088, "bmm_E4m3_E4m3E4m3_Fp32_t128x16x128u2_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f", 512, "07bf927187a4a1ea04c898725c464573d462196dafe7b602dea0fa59c6fed7a4", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -60391,10 +59552,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 64 @@ -60431,7 +59592,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 160 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -60439,7 +59599,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -60454,7 +59613,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -60501,10 +59659,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_t128x32x128_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x32x128_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f_cubin_len, 133552, "bmm_E4m3_E4m3E4m3_Fp32_t128x32x128_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f", 512, "73a20c7c817712d9368937895aa2008a8e92b4f3e91558c9b81962de0c92459f", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_t128x32x128_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x32x128_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f_cubin_len, 133552, "bmm_E4m3_E4m3E4m3_Fp32_t128x32x128_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f", 512, "b6e0ebf9c8018d828c165c7508a5908000392a1eb5fbb3cc39b02c2ceae513b2", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -60519,10 +59678,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 64 @@ -60559,7 +59718,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 160 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 4 @@ -60567,7 +59725,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -60582,7 +59739,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -60629,10 +59785,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_t128x32x128_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x32x128_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f_cubin_len, 133264, "bmm_E4m3_E4m3E4m3_Fp32_t128x32x128_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f", 512, "cee9332a1dbee31b5d367bd87d2d403e9ebacf16c04b5cb66bcb9830aad56629", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_t128x32x128_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x32x128_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f_cubin_len, 133424, "bmm_E4m3_E4m3E4m3_Fp32_t128x32x128_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f", 512, "169f826d3a4120b737d7062c92efa80371c4159cedfedd5e007368126cea5774", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -60647,10 +59804,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 64 @@ -60687,7 +59844,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 160 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -60695,7 +59851,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -60710,7 +59865,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -60757,10 +59911,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_t128x32x128u2_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x32x128u2_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f_cubin_len, 133552, "bmm_E4m3_E4m3E4m3_Fp32_t128x32x128u2_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f", 512, "7500dd08b65984b5b0d817d968a462b9a2845a9453885c132ae3857206709395", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_t128x32x128u2_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x32x128u2_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f_cubin_len, 133552, "bmm_E4m3_E4m3E4m3_Fp32_t128x32x128u2_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f", 512, "19beda82bfcdd907cdc2f86073509944facbdf980f649ae8c81cfb954b80bfe9", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -60775,10 +59930,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 64 @@ -60815,7 +59970,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 160 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 4 @@ -60823,7 +59977,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -60838,7 +59991,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -60885,10 +60037,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_t128x32x128u2_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x32x128u2_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f_cubin_len, 133264, "bmm_E4m3_E4m3E4m3_Fp32_t128x32x128u2_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f", 512, "9e60ddd3c95cbe07108ab5fee7d40ce03c26ec57d0e659be638e1bf6caba50aa", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_t128x32x128u2_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x32x128u2_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f_cubin_len, 133424, "bmm_E4m3_E4m3E4m3_Fp32_t128x32x128u2_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f", 512, "ba7f46aa74978bed02fa28f71ea0a0aedebcda0d065d8c881918d453f20b42b9", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -60903,10 +60056,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 64 @@ -60943,7 +60096,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 160 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -60951,7 +60103,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -60966,7 +60117,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -61013,10 +60163,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_t128x64x128_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x64x128_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f_cubin_len, 164016, "bmm_E4m3_E4m3E4m3_Fp32_t128x64x128_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f", 512, "e27148f040c5e36f713f416ef7f8c34fe77774ff683507e8b3c0da94566e2293", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_t128x64x128_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x64x128_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f_cubin_len, 164016, "bmm_E4m3_E4m3E4m3_Fp32_t128x64x128_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f", 512, "c304b5eb779f1b0ed2e7e76ed21286f81ddf55fb61546ae2d1f69c7348e7df04", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -61031,10 +60182,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 64 @@ -61071,7 +60222,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 160 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 4 @@ -61079,7 +60229,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -61094,7 +60243,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -61141,10 +60289,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_t128x64x128_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x64x128_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f_cubin_len, 163728, "bmm_E4m3_E4m3E4m3_Fp32_t128x64x128_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f", 512, "4b9c72aaa15c9adc34841d229362e7180f19ef83f8f8ddc8c7a5e73c2046a665", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_t128x64x128_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x64x128_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f_cubin_len, 163888, "bmm_E4m3_E4m3E4m3_Fp32_t128x64x128_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f", 512, "106e90cda78af26a70325c0342df7e7e26908621f334c1dd3a82f13b4b411f56", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -61159,10 +60308,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 64 @@ -61199,7 +60348,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 160 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -61207,7 +60355,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -61222,7 +60369,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -61269,10 +60415,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_t128x64x128u2_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x64x128u2_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f_cubin_len, 164016, "bmm_E4m3_E4m3E4m3_Fp32_t128x64x128u2_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f", 512, "c4536a724a2353bd9d2a8e509a04c62ad19bee94670a85ecd8c1f972bcfc6d40", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_t128x64x128u2_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x64x128u2_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f_cubin_len, 164016, "bmm_E4m3_E4m3E4m3_Fp32_t128x64x128u2_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f", 512, "80091b9c1df2163cd4e911356c5e901c199e652fde7a42f33b79fb88bcbf66ad", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -61287,10 +60434,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 64 @@ -61327,7 +60474,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 160 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 4 @@ -61335,7 +60481,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -61350,7 +60495,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -61397,10 +60541,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_t128x64x128u2_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x64x128u2_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f_cubin_len, 163728, "bmm_E4m3_E4m3E4m3_Fp32_t128x64x128u2_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f", 512, "d4c9bd4d227a72d689e9accccaed6934662283fa49217e41cbe6938cedaefbfa", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_t128x64x128u2_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x64x128u2_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f_cubin_len, 163888, "bmm_E4m3_E4m3E4m3_Fp32_t128x64x128u2_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f", 512, "46bb858d70bff24998ab9fac472f3a02371abc3f85a2f637169aaa206e56e740", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -61415,10 +60560,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 64 @@ -61455,7 +60600,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 160 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -61463,7 +60607,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -61478,7 +60621,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -61525,10 +60667,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_t128x8x128_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x8x128_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f_cubin_len, 77600, "bmm_E4m3_E4m3E4m3_Fp32_t128x8x128_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f", 384, "d5d110953879306c4c16dd1f0fa0f608f6f15474bd49ec4870bf4be025fd1a21", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_t128x8x128_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x8x128_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f_cubin_len, 77600, "bmm_E4m3_E4m3E4m3_Fp32_t128x8x128_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f", 384, "bf97736c6eca1938d1aa6b138934e6f0af4630121c94daef32a02f694297aab1", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -61543,10 +60686,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 0 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 64 @@ -61583,7 +60726,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 2 @@ -61591,7 +60733,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -61606,7 +60747,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -61653,10 +60793,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_t128x8x128_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x8x128_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f_cubin_len, 148240, "bmm_E4m3_E4m3E4m3_Fp32_t128x8x128_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f", 512, "b563efbd432a6e7176ca4d7cf6fedfa9867f77de40122b97c71c4bc139ce0a7b", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_t128x8x128_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x8x128_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f_cubin_len, 148400, "bmm_E4m3_E4m3E4m3_Fp32_t128x8x128_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f", 512, "9735f4620f7a49872a1d8fbc70121e44772d31b501c7de756194cb908475f19c", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -61671,10 +60812,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 64 @@ -61711,7 +60852,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 160 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 8 , /* mNumStagesB */ 8 , /* mNumStagesMma */ 4 @@ -61719,7 +60859,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 8 , /* mNumStagesSmemSfB */ 8 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 8 , /* mNumStagesTmemSfB */ 8 , /* mNumStagesWorkId */ 3 @@ -61734,7 +60873,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -61781,10 +60919,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_t128x8x128_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x8x128_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f_cubin_len, 147952, "bmm_E4m3_E4m3E4m3_Fp32_t128x8x128_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f", 512, "7b262ce7526fe0b54262c7c59ffbac66a31f7415129b8af5ea7590af1a935648", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_t128x8x128_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x8x128_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f_cubin_len, 148016, "bmm_E4m3_E4m3E4m3_Fp32_t128x8x128_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f", 512, "af89772453aa3810c357ec97cc78993fbc0f4795ef8f1c571f8bdd575ffd278f", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -61799,10 +60938,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 64 @@ -61839,7 +60978,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 160 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 8 , /* mNumStagesB */ 8 , /* mNumStagesMma */ 2 @@ -61847,7 +60985,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 8 , /* mNumStagesSmemSfB */ 8 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 8 , /* mNumStagesTmemSfB */ 8 , /* mNumStagesWorkId */ 3 @@ -61862,7 +60999,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -61909,10 +61045,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_t128x8x128u2_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x8x128u2_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f_cubin_len, 77600, "bmm_E4m3_E4m3E4m3_Fp32_t128x8x128u2_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f", 384, "83ea18de890b24f20545fb6b9ad845cb4e037a06f4e7611fa4f88d419f63cdc1", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_t128x8x128u2_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x8x128u2_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f_cubin_len, 77600, "bmm_E4m3_E4m3E4m3_Fp32_t128x8x128u2_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f", 384, "f8896c75af18524fc878cff4d1b88ab07e930a8705136b8446d6c9d116cd8b29", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -61927,10 +61064,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 0 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 64 @@ -61967,7 +61104,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 2 @@ -61975,7 +61111,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -61990,7 +61125,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -62037,10 +61171,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_t128x8x128u2_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x8x128u2_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f_cubin_len, 148240, "bmm_E4m3_E4m3E4m3_Fp32_t128x8x128u2_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f", 512, "e60f3a81ebc8039300bc11da4fab7604df50415b10a454e1d5ae5e466887756b", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_t128x8x128u2_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x8x128u2_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f_cubin_len, 148400, "bmm_E4m3_E4m3E4m3_Fp32_t128x8x128u2_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f", 512, "794eac7e536e460d8bbade89b6a887cb9b9dcbaf58df7376272b1620ac78e154", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -62055,10 +61190,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 64 @@ -62095,7 +61230,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 160 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 8 , /* mNumStagesB */ 8 , /* mNumStagesMma */ 4 @@ -62103,7 +61237,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 8 , /* mNumStagesSmemSfB */ 8 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 8 , /* mNumStagesTmemSfB */ 8 , /* mNumStagesWorkId */ 3 @@ -62118,7 +61251,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -62165,10 +61297,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_t128x8x128u2_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x8x128u2_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f_cubin_len, 147952, "bmm_E4m3_E4m3E4m3_Fp32_t128x8x128u2_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f", 512, "089c8b68d6d3152c141da36aced85797d5269aac2b81e737f9f5fb374d091fb5", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_t128x8x128u2_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x8x128u2_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f_cubin_len, 148016, "bmm_E4m3_E4m3E4m3_Fp32_t128x8x128u2_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f", 512, "74130c217679c26d03c899ff8001748e72369ffd8ae52a35d9119204d913bbc9", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -62183,10 +61316,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 64 @@ -62223,7 +61356,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 160 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 8 , /* mNumStagesB */ 8 , /* mNumStagesMma */ 2 @@ -62231,7 +61363,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 8 , /* mNumStagesSmemSfB */ 8 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 8 , /* mNumStagesTmemSfB */ 8 , /* mNumStagesWorkId */ 3 @@ -62246,7 +61377,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -62293,10 +61423,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(2)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f_cubin_len, 214144, "bmm_E4m3_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f", 256, "7e1919c11be6e6d9bd4bf8cd3025be1dbaf479726616b9df4880b827083f19a6", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f_cubin_len, 214288, "bmm_E4m3_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f", 256, "10b00a2d507f6376c0f3a7f5b07e226989986d1b8508b7940e651fcbf0e14093", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -62311,10 +61442,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 0 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -62351,7 +61482,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 1 @@ -62359,7 +61489,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -62374,7 +61503,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -62421,10 +61549,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f_cubin_len, 214144, "bmm_E4m3_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f", 256, "db66a7d04b21f175b58ddd2bbfdecb9032676d84b8d4e3240038d9959d22d60e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f_cubin_len, 214288, "bmm_E4m3_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f", 256, "5f2b48d498411d03a85bb972301f0ccf8f5161539db326e5ffde98bc07e6d029", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -62439,10 +61568,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 0 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -62479,7 +61608,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 1 @@ -62487,7 +61615,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -62502,7 +61629,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -62549,10 +61675,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f_cubin_len, 214144, "bmm_E4m3_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f", 256, "5d8c94ac1ddc487323b053bdbb7c72b9e92ec5c04b9a11b06f7a4d6f5f967c7c", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f_cubin_len, 214288, "bmm_E4m3_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f", 256, "98015ea893543e8f48f8f30312c8da20e9e9cc026757d270937740ba1045eb36", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -62567,10 +61694,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 0 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -62607,7 +61734,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 1 @@ -62615,7 +61741,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -62630,7 +61755,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -62677,10 +61801,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f_cubin_len, 214144, "bmm_E4m3_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f", 256, "1fb58718bce60477b71413d0cdcdbbd4fd39a3f833915b8cbe4a89613782a3f7", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f_cubin, Bmm_E4m3_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f_cubin_len, 214288, "bmm_E4m3_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f", 256, "f847eb6bc3ba1b77ffe51b830f6a3ff46fd5b7d504c7f9255e77acd3814d4296", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -62695,10 +61820,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 0 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -62735,7 +61860,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 1 @@ -62743,7 +61867,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -62758,7 +61881,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -62805,10 +61927,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 197200, "bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "39939ee051465a3568435f53dfa0cdc43c938a13ea2e2de0ef7a65736d242f6f", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 197264, "bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "b9176ed1e8ecbb4ec9fd34c66dcec28263393cd67b4cf01f44e49fbe435dbf52", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -62823,10 +61946,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -62863,7 +61986,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -62871,7 +61993,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -62886,7 +62007,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -62933,10 +62053,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 196960, "bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "c3aa1c2a548ae03d3939c3e94643296064fd194ed19da84b5ae7da3e97554202", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 197136, "bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "7efafb8e5d6a0f891efd6ec4d0e6b99a335cc463b21ddf3abc7938a49d659880", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -62951,10 +62072,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -62991,7 +62112,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -62999,7 +62119,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -63014,7 +62133,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -63061,10 +62179,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 197200, "bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "31321e7913692f8b7449bcd5074fc582ef0e59e75d35ab49a14ef49d881a5117", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 197264, "bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "a9c6b4468d9118a9d2cd3704723b92fb995d40f32b2d01aefb8455d82a76f625", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -63079,10 +62198,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -63119,7 +62238,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -63127,7 +62245,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -63142,7 +62259,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -63189,10 +62305,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 196960, "bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "abeb65258d5f3f6b27eaa1033d729a9333f842fefbd67e0ed38617d9e04e97f4", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 197136, "bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "d1daec13fd45ee77f5e8ca27fc45dbd6afd4b4ee9e928bda3171758595665fe7", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -63207,10 +62324,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -63247,7 +62364,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -63255,7 +62371,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -63270,7 +62385,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -63317,10 +62431,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 219728, "bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "ce583630401a3019168559faf76cdf0dd648e7b19494374c98d72e55ff23891a", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 219792, "bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "fc1ca5433f76492353e9f716a2c691ff15e48aeb156d6312783bd4e754284125", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -63335,10 +62450,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -63375,7 +62490,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -63383,7 +62497,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -63398,7 +62511,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -63445,10 +62557,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 219488, "bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "c472c805b51dd13b6dde943cedf2b1bf601edf7d2a6aee258079efba9717f7b7", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 219664, "bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "abbdff57e604e3a3438e53d567a0ab5bb8f848f0e7d213a8c6e648ac0af303b6", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -63463,10 +62576,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -63503,7 +62616,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -63511,7 +62623,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -63526,7 +62637,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -63573,10 +62683,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 219728, "bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "149e9f683ae4e254f3542203f1e1c6a391dba64f1f84daf87456c71d5f82b316", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 219792, "bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "ff2349beeecd60b8a4f70866ddb80e58372d603a1972f97bc9d9add585072fc2", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -63591,10 +62702,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -63631,7 +62742,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -63639,7 +62749,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -63654,7 +62763,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -63701,10 +62809,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 219488, "bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "6f95a86a88314f9479f94d22a0b7a8977f8605d5b936f52b79aaecef930d253e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 219664, "bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "01a39e60531aa46ccce80294035a8eb9eb623a880a30df2e5e8d0a4cba653eb6", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -63719,10 +62828,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -63759,7 +62868,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -63767,7 +62875,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -63782,7 +62889,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -63829,10 +62935,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 214544, "bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "2a5b31e2ba41fca2dbd428769634857213fc893ca1aea3243088908be1c8ecac", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 214672, "bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "43b89cdd540a4d30470b8a68383b70bb45ed8eab0f86cbc98955f4b96bb485f4", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -63847,10 +62954,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -63887,7 +62994,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 2 @@ -63895,7 +63001,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -63910,7 +63015,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -63957,10 +63061,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 214304, "bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "d0f2cfe67b4363f8638bb083332f54da9b0b192530f17227b0f50d791071703c", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 214544, "bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "087ef901e031017bbe42c1887b044ed0c58ba3fde9f141cf6bfefa44bd442983", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -63975,10 +63080,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -64015,7 +63120,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 1 @@ -64023,7 +63127,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -64038,7 +63141,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -64085,10 +63187,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 214544, "bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "9a2a5b5146e99a32dfd817fd82f874f9028548d9c91a14b68d226972ddcf10e4", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 214672, "bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "e66f6fd87e374d25e54b3f663ecbff37326993cddf1209f7e56dce6102322609", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -64103,10 +63206,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -64143,7 +63246,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 2 @@ -64151,7 +63253,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -64166,7 +63267,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -64213,10 +63313,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 214304, "bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "08ae3d590a4e3c3722bc2fddade8b9dbf06d5992ed3e3da96cb965e88d687934", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 214544, "bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "421cdeb05cdd8841d98b131d81013246d3c41d26dd4610ea79e6929bdb2ac46a", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -64231,10 +63332,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -64271,7 +63372,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 1 @@ -64279,7 +63379,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -64294,7 +63393,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -64341,10 +63439,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 185936, "bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "5479bcbe739224c55d81e0eef5ad842f8fe0a1c2f473ecbdbe8164072f4a8dab", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 186000, "bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "38f7a35eb3071576c8646ec5d8015ac07637414551823239de8febc2eda23030", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -64359,10 +63458,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -64399,7 +63498,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -64407,7 +63505,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -64422,7 +63519,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -64469,10 +63565,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 185696, "bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "144fa71dfc69d0fcc2edbb1182e9fb3284a390fd9f1a993cc1b83dca1d87a4f8", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 185872, "bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "b59a6f402ad5d671e3171f3a93000a5b81adf11710e91c487800352a15c84340", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -64487,10 +63584,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -64527,7 +63624,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -64535,7 +63631,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -64550,7 +63645,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -64597,10 +63691,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 185936, "bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "482c7e22f20c40c7f040203228008b848653131507212678d4e194f02ed6c561", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 186000, "bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "c22415f5c02ce88e5e50580b1c316a63d8fc1af0ee0d0318e48d7cec8713d66e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -64615,10 +63710,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -64655,7 +63750,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -64663,7 +63757,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -64678,7 +63771,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -64725,10 +63817,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 185696, "bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "53fe488ccbc8077f8cfc4de8cfe3efb0980d3097422fed671802decd0240a742", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 185872, "bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "b24161e6870621828c26b892fa4b3eb25e2961d30f2a7fb58803122804063153", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -64743,10 +63836,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -64783,7 +63876,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -64791,7 +63883,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -64806,7 +63897,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -64853,10 +63943,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 221648, "bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "c4d44d11931611e99d6c95caeb8a33428290053da582302996c58848f68b2209", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 221840, "bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "cc33375f30da72c6adcb1fa1d01af12d374c4b73e35147484ade96e45e7a7b7f", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -64871,10 +63962,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -64911,7 +64002,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 2 @@ -64919,7 +64009,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -64934,7 +64023,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -64981,10 +64069,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 221408, "bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "4e062c016f7e1765a1f240a1efeab9dba8dd3ab9d8eb126adc0a6e30b3daa954", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 221456, "bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "ecf9077063770fdfe2e8400bca946e4652e993645769b078b85dc9bcf44bb7c3", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -64999,10 +64088,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -65039,7 +64128,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 1 @@ -65047,7 +64135,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -65062,7 +64149,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -65109,10 +64195,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 221648, "bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "027f8c99de5a722d0d3021da4b4d1f8b48b8e311d85001b4a152cf7f894b871c", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 221840, "bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "bd27f8e5b0267e76dd774f5d1341c9a0ac9c5a7ead094bff30b6c603dc224c3a", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -65127,10 +64214,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -65167,7 +64254,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 2 @@ -65175,7 +64261,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -65190,7 +64275,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -65237,10 +64321,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 221408, "bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "5e8ffa085c4d113db33fe95d791859554b78526197ae37dacef274a4e592ed04", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 221456, "bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 384, "b33396900a11a3f382088e91d069ff4c73570afed9457946e6b80e999aa069ba", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -65255,10 +64340,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -65295,7 +64380,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 1 @@ -65303,7 +64387,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -65318,7 +64401,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -65365,10 +64447,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Fp16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f_cubin, Bmm_Fp16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f_cubin_len, 163232, "bmm_Fp16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f", 512, "7583c006aae553af7f5cc470cb7d8e624cecabba8402682dbaf056f5722848e5", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Fp16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f_cubin, Bmm_Fp16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f_cubin_len, 163344, "bmm_Fp16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f", 512, "c6f451d32798efa5176013225f5f16fb47871368a7f5fcf6136e01e87fa77f05", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -65383,10 +64466,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052679) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 0 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -65423,7 +64506,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 1 @@ -65431,7 +64513,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -65446,7 +64527,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -65493,10 +64573,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Fp16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512u2_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f_cubin, Bmm_Fp16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512u2_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f_cubin_len, 163232, "bmm_Fp16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512u2_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f", 512, "68db539847709fdda29800918d1c5eb6503d1b612851d4d585ceef782d000959", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Fp16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512u2_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f_cubin, Bmm_Fp16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512u2_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f_cubin_len, 163344, "bmm_Fp16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512u2_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f", 512, "f30a522a3b2611527238e547699ebc4bafecf067fb2aa9b686f555f098d5e8ed", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -65511,10 +64592,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052679) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826818) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826818) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 0 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -65551,7 +64632,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 1 @@ -65559,7 +64639,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -65574,7 +64653,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -65621,10 +64699,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Fp16_E4m3E4m3_Fp32_t128x8x128_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f_cubin, Bmm_Fp16_E4m3E4m3_Fp32_t128x8x128_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f_cubin_len, 77600, "bmm_Fp16_E4m3E4m3_Fp32_t128x8x128_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f", 384, "d36101573a7fb44d8e8222d2dd440f7afabdd87d27527bfbcd172eb4603402b7", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Fp16_E4m3E4m3_Fp32_t128x8x128_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f_cubin, Bmm_Fp16_E4m3E4m3_Fp32_t128x8x128_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f_cubin_len, 77600, "bmm_Fp16_E4m3E4m3_Fp32_t128x8x128_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f", 384, "222a8605d41afbf8544ef43605dd3db148c674733cea55792716898133ab0836", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -65639,10 +64718,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052679) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 0 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 64 @@ -65679,7 +64758,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 2 @@ -65687,7 +64765,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -65702,7 +64779,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -65749,10 +64825,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Fp16_E4m3E4m3_Fp32_t128x8x128u2_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f_cubin, Bmm_Fp16_E4m3E4m3_Fp32_t128x8x128u2_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f_cubin_len, 77600, "bmm_Fp16_E4m3E4m3_Fp32_t128x8x128u2_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f", 384, "bca128be7e1884218d31d282297db307ca76f24968bffb96707bc6dcd9aad7bc", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Fp16_E4m3E4m3_Fp32_t128x8x128u2_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f_cubin, Bmm_Fp16_E4m3E4m3_Fp32_t128x8x128u2_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f_cubin_len, 77600, "bmm_Fp16_E4m3E4m3_Fp32_t128x8x128u2_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f", 384, "60e6793839ae1faf6e1586a798ce2c23a03681099f5f5de6d3f34718a151c94b", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -65767,10 +64844,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052679) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 0 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 64 @@ -65807,7 +64884,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 2 @@ -65815,7 +64891,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -65830,7 +64905,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -65877,10 +64951,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Fp16_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f_cubin, Bmm_Fp16_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f_cubin_len, 215168, "bmm_Fp16_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f", 256, "57bb2e0455d45b6fdfbcda7a53ba7eec4ce04427c601104e724587c87e55cda2", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Fp16_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f_cubin, Bmm_Fp16_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f_cubin_len, 215312, "bmm_Fp16_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f", 256, "d4c81d413ee0008dfab9876f9fa187fb491ebba0658f05d0808bb71d7c7ee841", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -65895,10 +64970,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052679) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 0 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -65935,7 +65010,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 1 @@ -65943,7 +65017,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -65958,7 +65031,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -66005,10 +65077,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Fp16_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f_cubin, Bmm_Fp16_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f_cubin_len, 215168, "bmm_Fp16_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f", 256, "1eb118aef3b95b941781953496b2c9e419edbf4a1bc4f6616989a76ddedd9831", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Fp16_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f_cubin, Bmm_Fp16_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f_cubin_len, 215312, "bmm_Fp16_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f", 256, "770e5c35da7a98e7fe15407e614f52755dc8173f0b83aabbdfe6556f278d0be8", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -66023,10 +65096,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052679) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 0 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -66063,7 +65136,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 1 @@ -66071,7 +65143,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -66086,7 +65157,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -66133,10 +65203,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Fp16_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f_cubin, Bmm_Fp16_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f_cubin_len, 215168, "bmm_Fp16_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f", 256, "2b5282256d476ee7f178620b17160d15b5bfa1fc0d63b6140e6037e5fdd95343", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Fp16_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f_cubin, Bmm_Fp16_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f_cubin_len, 215312, "bmm_Fp16_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f", 256, "5b6536b38365461fdcddd19a9efc0f10a92e410b4274c6cdfcb84e5673e6b63c", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -66151,10 +65222,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052679) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 0 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -66191,7 +65262,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 1 @@ -66199,7 +65269,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -66214,7 +65283,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -66261,10 +65329,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_Fp16_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f_cubin, Bmm_Fp16_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f_cubin_len, 215168, "bmm_Fp16_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f", 256, "e087c3ecd88764d2040fddd9fa9836a71eac8a0d56ef6ee16ce1101974410222", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Fp16_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f_cubin, Bmm_Fp16_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f_cubin_len, 215312, "bmm_Fp16_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f", 256, "a2b98a99fc23fac19dfc19c7b71cf3061d2715d69c6db8b189830eca751ff87a", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 @@ -66279,10 +65348,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052679) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(2) , /* mEnablesEarlyExit */ 0 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -66319,7 +65388,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 1 @@ -66327,7 +65395,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -66342,7 +65409,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -66389,10 +65455,137 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} +, /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 , /* mUseTmaOobOpt */ 1 + }, gemm::SmVersion::Sm100f}, +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f_cubin_len, 190360, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f", 768, "41c49b5c29da4462a33edd8af0ea689fbce4fdba08f0c02fb989ea37abb7a6c0", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +, /* mBiasDtype */ trtllm::gen::Dtype(1056776) +, /* mBiasType */ gemm::BiasType(0) +, /* mBlockK */ -1 +, /* mClcFastDrain */ 1 +, /* mClusterDimX */ 2 +, /* mClusterDimY */ 1 +, /* mClusterDimZ */ 1 +, /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) +, /* mDtypeAcc */ trtllm::gen::Dtype(1056776) +, /* mDtypeA */ trtllm::gen::Dtype(17826828) +, /* mDtypeB */ trtllm::gen::Dtype(17827853) +, /* mDtypeC */ trtllm::gen::Dtype(17827853) +, /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) +, /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) +, /* mEltwiseActType */ gemm::EltwiseActType(0) +, /* mEnablesEarlyExit */ 1 +, /* mEnablesDelayedEarlyExit */ 0 +, /* mEpilogueLdtmDps */ 16 +, /* mEpilogueLdtmBits */ 256 +, /* mEpilogueTileM */ 128 +, /* mEpilogueTileN */ 32 +, /* mFallbackClusterDimX */ 1 +, /* mFallbackClusterDimY */ 1 +, /* mFallbackClusterDimZ */ 1 +, /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) +, /* mFuseLoadSfTask */ 0 +, /* mFuseUtccpWithUtcmma */ 0 +, /* mGridTriggerSecondaryA */ 0 +, /* mGridTriggerSecondaryB */ 1 +, /* mGridWaitForPrimaryEarlyExit */ 1 +, /* mGridWaitForPrimaryA */ 0 +, /* mGridWaitForPrimaryB */ 1 +, /* mHoistLoadTaskInit */ 1 +, /* mHoistMmaTaskTryWaits */ 0 +, /* mK */ 128 +, /* mKernelTraits */ {} +, /* mLayoutA */ gemm::MatrixLayout(0) +, /* mLayoutB */ gemm::MatrixLayout(0) +, /* mM */ 256 +, /* mMmaK */ 32 +, /* mMmaKind */ trtllm::gen::MmaKind(5) +, /* mMmaM */ 256 +, /* mMmaN */ 128 +, /* mMmaTileK */ 128 +, /* mMockAllReduce */ 0 +, /* mN */ 256 +, /* mNumEpilogueWarps */ 4 +, /* mNumRegsCastAWarps */ 0 +, /* mNumRegsCopySfLdsSttm */ 0 +, /* mNumRegsCopySparsityInfo */ 0 +, /* mNumRegsPerThreadEpilogueWarp */ 152 +, /* mNumRegsPerThreadNonEpilogueWarp */ 48 +, /* mNumSlicesForSplitK */ 1 +, /* mNumStagesA */ 7 +, /* mNumStagesB */ 7 +, /* mNumStagesMma */ 2 +, /* mNumStagesMmaWithinWorkTile */ 1 +, /* mNumStagesMmaAcrossWorkTile */ 2 +, /* mNumStagesSmemSfA */ 7 +, /* mNumStagesSmemSfB */ 7 +, /* mNumStagesTmemSfA */ 7 +, /* mNumStagesTmemSfB */ 7 +, /* mNumStagesWorkId */ 3 +, /* mOutputDebugTensors */ 0 +, /* mPatchF2fp */ 0 +, /* mPerTokenSfDtype */ trtllm::gen::Dtype(1052672) +, /* mSchedHostTask */ gemm::SchedHostTask(0) +, /* mSfBlockSizeA */ 32 +, /* mSfBlockSizeB */ 32 +, /* mSfBlockSizeC */ 32 +, /* mSfLayoutA */ trtllm::gen::SfLayout(3) +, /* mSfLayoutB */ trtllm::gen::SfLayout(0) +, /* mSfLayoutC */ trtllm::gen::SfLayout(3) +, /* mSfReshapeFactor */ 1 +, /* mSparsityA */ trtllm::gen::Sparsity(0) +, /* mSplitK */ gemm::SplitK(0) +, /* mTileK */ 128 +, /* mTileM */ 128 +, /* mTileN */ 128 +, /* mTileScheduler */ gemm::TileScheduler(1) +, /* mTransposeMmaOutput */ 1 +, /* mUseCustomizedMma3xNvFp4 */ 0 +, /* mUseCustomMmaSchedule */ 1 +, /* mUseDeepSeekFp8 */ 0 +, /* mUseFlexibleClusterDims */ 0 +, /* mUseHoistTryWaitForCustomMmaSchedule */ 0 +, /* mUseMaxTmemOverlap */ 0 +, /* mUsePerTokenSfA */ 0 +, /* mUsePerTokenSfB */ 0 +, /* mUseShuffledMatrix */ 1 +, /* mUseTmaStore */ 1 +, /* mUseTwoTmaLoadWarps */ 1 +, /* mUseTwoMmaWarps */ 0 +, /* mUseUnrollLoop2xForMma */ 0 +, /* mValidM */ 256 +, /* mValidN */ 256 +, /* mValidK */ 128 +, /* mWorldSize */ 1 +, /* mActType */ gemmGatedAct::ActType(2) +, /* mClampBeforeAct */ 1 +, /* mBatchedM */ {} +, /* mBatchedN */ {} +, /* mBatchMode */ batchedGemm::BatchedGemmOptions::BatchMode(1) +, /* mBatchStrideInTokens */ -1 +, /* mFusedAct */ 1 +, /* mGridWaitForPrimaryRouting */ 1 +, /* mIsStaticBatch */ 0 +, /* mIsUniformNumTokensPerBatch */ 0 +, /* mNumBatches */ 128 +, /* mNumRegsPerThreadLoadA */ 0 +, /* mNumRegsPerThreadLoadB */ 48 +, /* mNumRegsPerThreadLoadSfA */ 0 +, /* mNumRegsPerThreadLoadSfB */ 48 +, /* mNumTokens */ 2 +, /* mNumWarpsLoadA */ 0 +, /* mNumWarpsLoadB */ 8 +, /* mNumWarpsLoadSfA */ 0 +, /* mNumWarpsLoadSfB */ 4 +, /* mRouteImpl */ batchedGemm::RouteImpl(2) +, /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f_cubin_len, 191304, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f", 768, "94eb99818cad0b43d89cb5c1a46df177ae987e66fbb3c9c51c6b9b1c88bc8967", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f_cubin_len, 191384, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f", 768, "8d1bbde613872a43a2b7280c74d3b31a0fd4e23dd8f9df5533c983bb7d066897", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -66407,10 +65600,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17827853) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -66447,7 +65640,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 152 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 7 , /* mNumStagesB */ 7 , /* mNumStagesMma */ 2 @@ -66455,7 +65647,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 7 , /* mNumStagesSmemSfB */ 7 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 7 , /* mNumStagesTmemSfB */ 7 , /* mNumStagesWorkId */ 3 @@ -66470,7 +65661,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -66517,10 +65707,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 4 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7x7x7x7x1x7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_lsfbW4_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7x7x7x7x1x7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_lsfbW4_dynB_sm100f_cubin_len, 191208, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7x7x7x7x1x7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_lsfbW4_dynB_sm100f", 384, "4f87a83c775d6386e39665006c019874214af608552daaacdf1117d3bf0d29e8", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7x7x7x7x1x7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_lsfbW4_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7x7x7x7x1x7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_lsfbW4_dynB_sm100f_cubin_len, 191384, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7x7x7x7x1x7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_lsfbW4_dynB_sm100f", 384, "e523ecebd69225c3d675a4c823b046cbb3483d9cc926b3dd8ded662ba5893dcb", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -66535,10 +65726,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17827853) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -66575,7 +65766,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 168 , /* mNumRegsPerThreadNonEpilogueWarp */ 96 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 7 , /* mNumStagesB */ 7 , /* mNumStagesMma */ 2 @@ -66583,7 +65773,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 7 , /* mNumStagesSmemSfB */ 7 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 1 , /* mNumStagesTmemSfB */ 7 , /* mNumStagesWorkId */ 3 @@ -66598,7 +65787,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -66645,12 +65833,13 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 4 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128u2_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128u2_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f_cubin_len, 191304, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128u2_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f", 768, "7babca269132b870014344132e2bc62271c7fb3c37d7209168e508240a2ddc43", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7x7x7x7x1x7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_siTuGlu_lbW4_lsfbW4_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7x7x7x7x1x7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_siTuGlu_lbW4_lsfbW4_dynB_sm100f_cubin_len, 190360, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7x7x7x7x1x7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_siTuGlu_lbW4_lsfbW4_dynB_sm100f", 384, "54dcb526aadfa88f5546f868cda57f955efdee3e00f98d22b0e363de09d29287", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) -, /* mBiasType */ gemm::BiasType(1) +, /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 , /* mClcFastDrain */ 1 , /* mClusterDimX */ 2 @@ -66663,10 +65852,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17827853) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -66675,8 +65864,8 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mFallbackClusterDimY */ 1 , /* mFallbackClusterDimZ */ 1 , /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) -, /* mFuseLoadSfTask */ 0 -, /* mFuseUtccpWithUtcmma */ 0 +, /* mFuseLoadSfTask */ 1 +, /* mFuseUtccpWithUtcmma */ 1 , /* mGridTriggerSecondaryA */ 0 , /* mGridTriggerSecondaryB */ 1 , /* mGridWaitForPrimaryEarlyExit */ 1 @@ -66684,7 +65873,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mGridWaitForPrimaryB */ 1 , /* mHoistLoadTaskInit */ 1 , /* mHoistMmaTaskTryWaits */ 0 -, /* mK */ 256 +, /* mK */ 128 , /* mKernelTraits */ {} , /* mLayoutA */ gemm::MatrixLayout(0) , /* mLayoutB */ gemm::MatrixLayout(0) @@ -66700,10 +65889,9 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsCastAWarps */ 0 , /* mNumRegsCopySfLdsSttm */ 0 , /* mNumRegsCopySparsityInfo */ 0 -, /* mNumRegsPerThreadEpilogueWarp */ 152 -, /* mNumRegsPerThreadNonEpilogueWarp */ 48 +, /* mNumRegsPerThreadEpilogueWarp */ 168 +, /* mNumRegsPerThreadNonEpilogueWarp */ 96 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 7 , /* mNumStagesB */ 7 , /* mNumStagesMma */ 2 @@ -66711,8 +65899,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 7 , /* mNumStagesSmemSfB */ 7 -, /* mNumStagesTmemA */ 0 -, /* mNumStagesTmemSfA */ 7 +, /* mNumStagesTmemSfA */ 1 , /* mNumStagesTmemSfB */ 7 , /* mNumStagesWorkId */ 3 , /* mOutputDebugTensors */ 0 @@ -66726,7 +65913,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -66746,12 +65932,12 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mUseTmaStore */ 1 , /* mUseTwoTmaLoadWarps */ 1 , /* mUseTwoMmaWarps */ 0 -, /* mUseUnrollLoop2xForMma */ 1 +, /* mUseUnrollLoop2xForMma */ 0 , /* mValidM */ 256 , /* mValidN */ 256 -, /* mValidK */ 256 +, /* mValidK */ 128 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(0) +, /* mActType */ gemmGatedAct::ActType(2) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -66763,22 +65949,23 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mIsUniformNumTokensPerBatch */ 0 , /* mNumBatches */ 128 , /* mNumRegsPerThreadLoadA */ 0 -, /* mNumRegsPerThreadLoadB */ 48 +, /* mNumRegsPerThreadLoadB */ 128 , /* mNumRegsPerThreadLoadSfA */ 0 -, /* mNumRegsPerThreadLoadSfB */ 48 +, /* mNumRegsPerThreadLoadSfB */ 0 , /* mNumTokens */ 2 , /* mNumWarpsLoadA */ 0 -, /* mNumWarpsLoadB */ 8 +, /* mNumWarpsLoadB */ 4 , /* mNumWarpsLoadSfA */ 0 , /* mNumWarpsLoadSfB */ 4 -, /* mRouteImpl */ batchedGemm::RouteImpl(2) +, /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f_cubin_len, 215640, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f", 768, "878005eeea6136efd8513a6f20969b67d14d90b48a19c7e9107e25876077ea4e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128u2_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128u2_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f_cubin_len, 190360, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128u2_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f", 768, "98b5cd405aa5825a01083fa5ac9717eea6795764772ea4bdf41b4f692e7438a7", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) -, /* mBiasType */ gemm::BiasType(1) +, /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 , /* mClcFastDrain */ 1 , /* mClusterDimX */ 2 @@ -66791,10 +65978,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17827853) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -66821,7 +66008,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mMmaKind */ trtllm::gen::MmaKind(5) , /* mMmaM */ 256 , /* mMmaN */ 128 -, /* mMmaTileK */ 256 +, /* mMmaTileK */ 128 , /* mMockAllReduce */ 0 , /* mN */ 256 , /* mNumEpilogueWarps */ 4 @@ -66831,17 +66018,15 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 152 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 -, /* mNumStagesA */ 4 -, /* mNumStagesB */ 4 +, /* mNumStagesA */ 7 +, /* mNumStagesB */ 7 , /* mNumStagesMma */ 2 , /* mNumStagesMmaWithinWorkTile */ 1 , /* mNumStagesMmaAcrossWorkTile */ 2 -, /* mNumStagesSmemSfA */ 4 -, /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 -, /* mNumStagesTmemSfA */ 4 -, /* mNumStagesTmemSfB */ 4 +, /* mNumStagesSmemSfA */ 7 +, /* mNumStagesSmemSfB */ 7 +, /* mNumStagesTmemSfA */ 7 +, /* mNumStagesTmemSfB */ 7 , /* mNumStagesWorkId */ 3 , /* mOutputDebugTensors */ 0 , /* mPatchF2fp */ 0 @@ -66854,10 +66039,9 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) -, /* mTileK */ 256 +, /* mTileK */ 128 , /* mTileM */ 128 , /* mTileN */ 128 , /* mTileScheduler */ gemm::TileScheduler(1) @@ -66874,12 +66058,12 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mUseTmaStore */ 1 , /* mUseTwoTmaLoadWarps */ 1 , /* mUseTwoMmaWarps */ 0 -, /* mUseUnrollLoop2xForMma */ 0 +, /* mUseUnrollLoop2xForMma */ 1 , /* mValidM */ 256 , /* mValidN */ 256 , /* mValidK */ 256 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(0) +, /* mActType */ gemmGatedAct::ActType(2) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -66901,10 +66085,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 4 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4x4x4x4x1x4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_lsfbW4_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4x4x4x4x1x4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_lsfbW4_dynB_sm100f_cubin_len, 215592, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4x4x4x4x1x4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_lsfbW4_dynB_sm100f", 384, "1bbf5b78b6379da00a497a85c85059f6cc76f7968e691f582cdd107cb25376d1", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128u2_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128u2_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f_cubin_len, 191384, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128u2_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f", 768, "a84b7e5b283c22af508aed4e3b1c57c7b95dbf3a10f21a44c9b05141f66de78c", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -66919,10 +66104,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17827853) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -66931,8 +66116,8 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mFallbackClusterDimY */ 1 , /* mFallbackClusterDimZ */ 1 , /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) -, /* mFuseLoadSfTask */ 1 -, /* mFuseUtccpWithUtcmma */ 1 +, /* mFuseLoadSfTask */ 0 +, /* mFuseUtccpWithUtcmma */ 0 , /* mGridTriggerSecondaryA */ 0 , /* mGridTriggerSecondaryB */ 1 , /* mGridWaitForPrimaryEarlyExit */ 1 @@ -66949,27 +66134,25 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mMmaKind */ trtllm::gen::MmaKind(5) , /* mMmaM */ 256 , /* mMmaN */ 128 -, /* mMmaTileK */ 256 +, /* mMmaTileK */ 128 , /* mMockAllReduce */ 0 , /* mN */ 256 , /* mNumEpilogueWarps */ 4 , /* mNumRegsCastAWarps */ 0 , /* mNumRegsCopySfLdsSttm */ 0 , /* mNumRegsCopySparsityInfo */ 0 -, /* mNumRegsPerThreadEpilogueWarp */ 168 -, /* mNumRegsPerThreadNonEpilogueWarp */ 96 +, /* mNumRegsPerThreadEpilogueWarp */ 152 +, /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 -, /* mNumStagesA */ 4 -, /* mNumStagesB */ 4 +, /* mNumStagesA */ 7 +, /* mNumStagesB */ 7 , /* mNumStagesMma */ 2 , /* mNumStagesMmaWithinWorkTile */ 1 , /* mNumStagesMmaAcrossWorkTile */ 2 -, /* mNumStagesSmemSfA */ 4 -, /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 -, /* mNumStagesTmemSfA */ 1 -, /* mNumStagesTmemSfB */ 4 +, /* mNumStagesSmemSfA */ 7 +, /* mNumStagesSmemSfB */ 7 +, /* mNumStagesTmemSfA */ 7 +, /* mNumStagesTmemSfB */ 7 , /* mNumStagesWorkId */ 3 , /* mOutputDebugTensors */ 0 , /* mPatchF2fp */ 0 @@ -66982,10 +66165,9 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) -, /* mTileK */ 256 +, /* mTileK */ 128 , /* mTileM */ 128 , /* mTileN */ 128 , /* mTileScheduler */ gemm::TileScheduler(1) @@ -67002,7 +66184,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mUseTmaStore */ 1 , /* mUseTwoTmaLoadWarps */ 1 , /* mUseTwoMmaWarps */ 0 -, /* mUseUnrollLoop2xForMma */ 0 +, /* mUseUnrollLoop2xForMma */ 1 , /* mValidM */ 256 , /* mValidN */ 256 , /* mValidK */ 256 @@ -67019,22 +66201,23 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mIsUniformNumTokensPerBatch */ 0 , /* mNumBatches */ 128 , /* mNumRegsPerThreadLoadA */ 0 -, /* mNumRegsPerThreadLoadB */ 128 +, /* mNumRegsPerThreadLoadB */ 48 , /* mNumRegsPerThreadLoadSfA */ 0 -, /* mNumRegsPerThreadLoadSfB */ 0 +, /* mNumRegsPerThreadLoadSfB */ 48 , /* mNumTokens */ 2 , /* mNumWarpsLoadA */ 0 -, /* mNumWarpsLoadB */ 4 +, /* mNumWarpsLoadB */ 8 , /* mNumWarpsLoadSfA */ 0 , /* mNumWarpsLoadSfB */ 4 -, /* mRouteImpl */ batchedGemm::RouteImpl(1) +, /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256u2_s4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256u2_s4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f_cubin_len, 215640, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256u2_s4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f", 768, "00417708d7966a9aed85ac708e79e9336402e9f4401121bbdf4984917576e36f", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f_cubin_len, 214680, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f", 768, "2def5a9167d42081c328ff38c2078b1a56f22b9ba95ef6ee15076560e4fc0121", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) -, /* mBiasType */ gemm::BiasType(1) +, /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 , /* mClcFastDrain */ 1 , /* mClusterDimX */ 2 @@ -67047,10 +66230,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17827853) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -67068,7 +66251,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mGridWaitForPrimaryB */ 1 , /* mHoistLoadTaskInit */ 1 , /* mHoistMmaTaskTryWaits */ 0 -, /* mK */ 512 +, /* mK */ 256 , /* mKernelTraits */ {} , /* mLayoutA */ gemm::MatrixLayout(0) , /* mLayoutB */ gemm::MatrixLayout(0) @@ -67087,7 +66270,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 152 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 2 @@ -67095,7 +66277,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -67110,7 +66291,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -67130,12 +66310,12 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mUseTmaStore */ 1 , /* mUseTwoTmaLoadWarps */ 1 , /* mUseTwoMmaWarps */ 0 -, /* mUseUnrollLoop2xForMma */ 1 +, /* mUseUnrollLoop2xForMma */ 0 , /* mValidM */ 256 , /* mValidN */ 256 -, /* mValidK */ 512 +, /* mValidK */ 256 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(0) +, /* mActType */ gemmGatedAct::ActType(2) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -67157,15 +66337,16 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 4 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 197360, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "c0f9fa70dafe149c12ddc957064225c40a93cb620ac59bcbda0c9d8ebf380316", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f_cubin_len, 215704, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f", 768, "1024acf28e0bb05e3aed6e64a9775f0e2290fd028c328be6e84efe217a1c59e6", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 , /* mClcFastDrain */ 1 -, /* mClusterDimX */ 1 +, /* mClusterDimX */ 2 , /* mClusterDimY */ 1 , /* mClusterDimZ */ 1 , /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) @@ -67175,14 +66356,14 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17827853) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 -, /* mEpilogueTileN */ 16 +, /* mEpilogueTileN */ 32 , /* mFallbackClusterDimX */ 1 , /* mFallbackClusterDimY */ 1 , /* mFallbackClusterDimZ */ 1 @@ -67203,8 +66384,8 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mM */ 256 , /* mMmaK */ 32 , /* mMmaKind */ trtllm::gen::MmaKind(5) -, /* mMmaM */ 128 -, /* mMmaN */ 16 +, /* mMmaM */ 256 +, /* mMmaN */ 128 , /* mMmaTileK */ 256 , /* mMockAllReduce */ 0 , /* mN */ 256 @@ -67212,20 +66393,18 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsCastAWarps */ 0 , /* mNumRegsCopySfLdsSttm */ 0 , /* mNumRegsCopySparsityInfo */ 0 -, /* mNumRegsPerThreadEpilogueWarp */ 0 -, /* mNumRegsPerThreadNonEpilogueWarp */ 0 +, /* mNumRegsPerThreadEpilogueWarp */ 152 +, /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 -, /* mNumStagesA */ 5 -, /* mNumStagesB */ 5 +, /* mNumStagesA */ 4 +, /* mNumStagesB */ 4 , /* mNumStagesMma */ 2 , /* mNumStagesMmaWithinWorkTile */ 1 , /* mNumStagesMmaAcrossWorkTile */ 2 -, /* mNumStagesSmemSfA */ 5 -, /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 -, /* mNumStagesTmemSfA */ 5 -, /* mNumStagesTmemSfB */ 5 +, /* mNumStagesSmemSfA */ 4 +, /* mNumStagesSmemSfB */ 4 +, /* mNumStagesTmemSfA */ 4 +, /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 , /* mOutputDebugTensors */ 0 , /* mPatchF2fp */ 0 @@ -67236,14 +66415,13 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfBlockSizeC */ 32 , /* mSfLayoutA */ trtllm::gen::SfLayout(3) , /* mSfLayoutB */ trtllm::gen::SfLayout(0) -, /* mSfLayoutC */ trtllm::gen::SfLayout(1) +, /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 , /* mTileM */ 128 -, /* mTileN */ 16 +, /* mTileN */ 128 , /* mTileScheduler */ gemm::TileScheduler(1) , /* mTransposeMmaOutput */ 1 , /* mUseCustomizedMma3xNvFp4 */ 0 @@ -67275,25 +66453,26 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mIsUniformNumTokensPerBatch */ 0 , /* mNumBatches */ 128 , /* mNumRegsPerThreadLoadA */ 0 -, /* mNumRegsPerThreadLoadB */ 0 +, /* mNumRegsPerThreadLoadB */ 48 , /* mNumRegsPerThreadLoadSfA */ 0 -, /* mNumRegsPerThreadLoadSfB */ 0 +, /* mNumRegsPerThreadLoadSfB */ 48 , /* mNumTokens */ 2 , /* mNumWarpsLoadA */ 0 -, /* mNumWarpsLoadB */ 0 +, /* mNumWarpsLoadB */ 8 , /* mNumWarpsLoadSfA */ 0 -, /* mNumWarpsLoadSfB */ 0 +, /* mNumWarpsLoadSfB */ 4 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 197120, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "f60e3582675dc397ce43e0be81a5c742e45ec3ebd2f4dc5487073a1cb03ec65c", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4x4x4x4x1x4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_lsfbW4_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4x4x4x4x1x4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_lsfbW4_dynB_sm100f_cubin_len, 215704, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4x4x4x4x1x4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_lsfbW4_dynB_sm100f", 384, "1ebdfe613e3ac9f3e2bab6fd2aecebecdf19414dedd2374248175cc1c2b1b0cc", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 , /* mClcFastDrain */ 1 -, /* mClusterDimX */ 1 +, /* mClusterDimX */ 2 , /* mClusterDimY */ 1 , /* mClusterDimZ */ 1 , /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) @@ -67303,20 +66482,20 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17827853) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 -, /* mEpilogueTileN */ 16 +, /* mEpilogueTileN */ 32 , /* mFallbackClusterDimX */ 1 , /* mFallbackClusterDimY */ 1 , /* mFallbackClusterDimZ */ 1 , /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) -, /* mFuseLoadSfTask */ 0 -, /* mFuseUtccpWithUtcmma */ 0 +, /* mFuseLoadSfTask */ 1 +, /* mFuseUtccpWithUtcmma */ 1 , /* mGridTriggerSecondaryA */ 0 , /* mGridTriggerSecondaryB */ 1 , /* mGridWaitForPrimaryEarlyExit */ 1 @@ -67331,8 +66510,8 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mM */ 256 , /* mMmaK */ 32 , /* mMmaKind */ trtllm::gen::MmaKind(5) -, /* mMmaM */ 128 -, /* mMmaN */ 16 +, /* mMmaM */ 256 +, /* mMmaN */ 128 , /* mMmaTileK */ 256 , /* mMockAllReduce */ 0 , /* mN */ 256 @@ -67340,20 +66519,18 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsCastAWarps */ 0 , /* mNumRegsCopySfLdsSttm */ 0 , /* mNumRegsCopySparsityInfo */ 0 -, /* mNumRegsPerThreadEpilogueWarp */ 0 -, /* mNumRegsPerThreadNonEpilogueWarp */ 0 +, /* mNumRegsPerThreadEpilogueWarp */ 168 +, /* mNumRegsPerThreadNonEpilogueWarp */ 96 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 -, /* mNumStagesA */ 5 -, /* mNumStagesB */ 5 -, /* mNumStagesMma */ 1 +, /* mNumStagesA */ 4 +, /* mNumStagesB */ 4 +, /* mNumStagesMma */ 2 , /* mNumStagesMmaWithinWorkTile */ 1 -, /* mNumStagesMmaAcrossWorkTile */ 1 -, /* mNumStagesSmemSfA */ 5 -, /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 -, /* mNumStagesTmemSfA */ 5 -, /* mNumStagesTmemSfB */ 5 +, /* mNumStagesMmaAcrossWorkTile */ 2 +, /* mNumStagesSmemSfA */ 4 +, /* mNumStagesSmemSfB */ 4 +, /* mNumStagesTmemSfA */ 1 +, /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 , /* mOutputDebugTensors */ 0 , /* mPatchF2fp */ 0 @@ -67364,15 +66541,14 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfBlockSizeC */ 32 , /* mSfLayoutA */ trtllm::gen::SfLayout(3) , /* mSfLayoutB */ trtllm::gen::SfLayout(0) -, /* mSfLayoutC */ trtllm::gen::SfLayout(1) +, /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 , /* mTileM */ 128 -, /* mTileN */ 16 -, /* mTileScheduler */ gemm::TileScheduler(0) +, /* mTileN */ 128 +, /* mTileScheduler */ gemm::TileScheduler(1) , /* mTransposeMmaOutput */ 1 , /* mUseCustomizedMma3xNvFp4 */ 0 , /* mUseCustomMmaSchedule */ 1 @@ -67403,22 +66579,23 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mIsUniformNumTokensPerBatch */ 0 , /* mNumBatches */ 128 , /* mNumRegsPerThreadLoadA */ 0 -, /* mNumRegsPerThreadLoadB */ 0 +, /* mNumRegsPerThreadLoadB */ 128 , /* mNumRegsPerThreadLoadSfA */ 0 , /* mNumRegsPerThreadLoadSfB */ 0 , /* mNumTokens */ 2 , /* mNumWarpsLoadA */ 0 -, /* mNumWarpsLoadB */ 0 +, /* mNumWarpsLoadB */ 4 , /* mNumWarpsLoadSfA */ 0 -, /* mNumWarpsLoadSfB */ 0 -, /* mRouteImpl */ batchedGemm::RouteImpl(2) +, /* mNumWarpsLoadSfB */ 4 +, /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 223064, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "40f21ceb11d2eef4a3dd8d4f94f8caf74fe0c02dbca16cb33b00d907baeebf17", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4x4x4x4x1x4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_siTuGlu_lbW4_lsfbW4_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4x4x4x4x1x4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_siTuGlu_lbW4_lsfbW4_dynB_sm100f_cubin_len, 214680, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4x4x4x4x1x4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_siTuGlu_lbW4_lsfbW4_dynB_sm100f", 384, "fed222b86391c7c34a24f149a76f924a0226e34b3e450040a49c97e82fb9334d", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) -, /* mBiasType */ gemm::BiasType(1) +, /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 , /* mClcFastDrain */ 1 , /* mClusterDimX */ 2 @@ -67431,20 +66608,20 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17827853) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 -, /* mEpilogueTileN */ 16 +, /* mEpilogueTileN */ 32 , /* mFallbackClusterDimX */ 1 , /* mFallbackClusterDimY */ 1 , /* mFallbackClusterDimZ */ 1 , /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) -, /* mFuseLoadSfTask */ 0 -, /* mFuseUtccpWithUtcmma */ 0 +, /* mFuseLoadSfTask */ 1 +, /* mFuseUtccpWithUtcmma */ 1 , /* mGridTriggerSecondaryA */ 0 , /* mGridTriggerSecondaryB */ 1 , /* mGridWaitForPrimaryEarlyExit */ 1 @@ -67460,7 +66637,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mMmaK */ 32 , /* mMmaKind */ trtllm::gen::MmaKind(5) , /* mMmaM */ 256 -, /* mMmaN */ 16 +, /* mMmaN */ 128 , /* mMmaTileK */ 256 , /* mMockAllReduce */ 0 , /* mN */ 256 @@ -67468,20 +66645,18 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsCastAWarps */ 0 , /* mNumRegsCopySfLdsSttm */ 0 , /* mNumRegsCopySparsityInfo */ 0 -, /* mNumRegsPerThreadEpilogueWarp */ 0 -, /* mNumRegsPerThreadNonEpilogueWarp */ 0 +, /* mNumRegsPerThreadEpilogueWarp */ 168 +, /* mNumRegsPerThreadNonEpilogueWarp */ 96 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 -, /* mNumStagesA */ 6 -, /* mNumStagesB */ 6 +, /* mNumStagesA */ 4 +, /* mNumStagesB */ 4 , /* mNumStagesMma */ 2 , /* mNumStagesMmaWithinWorkTile */ 1 , /* mNumStagesMmaAcrossWorkTile */ 2 -, /* mNumStagesSmemSfA */ 6 -, /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 -, /* mNumStagesTmemSfA */ 6 -, /* mNumStagesTmemSfB */ 6 +, /* mNumStagesSmemSfA */ 4 +, /* mNumStagesSmemSfB */ 4 +, /* mNumStagesTmemSfA */ 1 +, /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 , /* mOutputDebugTensors */ 0 , /* mPatchF2fp */ 0 @@ -67492,14 +66667,13 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfBlockSizeC */ 32 , /* mSfLayoutA */ trtllm::gen::SfLayout(3) , /* mSfLayoutB */ trtllm::gen::SfLayout(0) -, /* mSfLayoutC */ trtllm::gen::SfLayout(1) +, /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 , /* mTileM */ 128 -, /* mTileN */ 16 +, /* mTileN */ 128 , /* mTileScheduler */ gemm::TileScheduler(1) , /* mTransposeMmaOutput */ 1 , /* mUseCustomizedMma3xNvFp4 */ 0 @@ -67519,7 +66693,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 256 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(0) +, /* mActType */ gemmGatedAct::ActType(2) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -67531,22 +66705,23 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mIsUniformNumTokensPerBatch */ 0 , /* mNumBatches */ 128 , /* mNumRegsPerThreadLoadA */ 0 -, /* mNumRegsPerThreadLoadB */ 0 +, /* mNumRegsPerThreadLoadB */ 128 , /* mNumRegsPerThreadLoadSfA */ 0 , /* mNumRegsPerThreadLoadSfB */ 0 , /* mNumTokens */ 2 , /* mNumWarpsLoadA */ 0 -, /* mNumWarpsLoadB */ 0 +, /* mNumWarpsLoadB */ 4 , /* mNumWarpsLoadSfA */ 0 -, /* mNumWarpsLoadSfB */ 0 -, /* mRouteImpl */ batchedGemm::RouteImpl(2) +, /* mNumWarpsLoadSfB */ 4 +, /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 222824, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "60bc1f5256377074fc1ff6329159c877718045b32fb9ab4516390576e9204e3a", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256u2_s4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256u2_s4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f_cubin_len, 214680, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256u2_s4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f", 768, "37af651f4b874418585f208a21d8f89bb69d211cb4ffe37ddb09b16e74a0946c", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) -, /* mBiasType */ gemm::BiasType(1) +, /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 , /* mClcFastDrain */ 1 , /* mClusterDimX */ 2 @@ -67559,14 +66734,14 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17827853) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 -, /* mEpilogueTileN */ 16 +, /* mEpilogueTileN */ 32 , /* mFallbackClusterDimX */ 1 , /* mFallbackClusterDimY */ 1 , /* mFallbackClusterDimZ */ 1 @@ -67580,7 +66755,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mGridWaitForPrimaryB */ 1 , /* mHoistLoadTaskInit */ 1 , /* mHoistMmaTaskTryWaits */ 0 -, /* mK */ 256 +, /* mK */ 512 , /* mKernelTraits */ {} , /* mLayoutA */ gemm::MatrixLayout(0) , /* mLayoutB */ gemm::MatrixLayout(0) @@ -67588,7 +66763,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mMmaK */ 32 , /* mMmaKind */ trtllm::gen::MmaKind(5) , /* mMmaM */ 256 -, /* mMmaN */ 16 +, /* mMmaN */ 128 , /* mMmaTileK */ 256 , /* mMockAllReduce */ 0 , /* mN */ 256 @@ -67596,20 +66771,18 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsCastAWarps */ 0 , /* mNumRegsCopySfLdsSttm */ 0 , /* mNumRegsCopySparsityInfo */ 0 -, /* mNumRegsPerThreadEpilogueWarp */ 0 -, /* mNumRegsPerThreadNonEpilogueWarp */ 0 +, /* mNumRegsPerThreadEpilogueWarp */ 152 +, /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 -, /* mNumStagesA */ 6 -, /* mNumStagesB */ 6 -, /* mNumStagesMma */ 1 +, /* mNumStagesA */ 4 +, /* mNumStagesB */ 4 +, /* mNumStagesMma */ 2 , /* mNumStagesMmaWithinWorkTile */ 1 -, /* mNumStagesMmaAcrossWorkTile */ 1 -, /* mNumStagesSmemSfA */ 6 -, /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 -, /* mNumStagesTmemSfA */ 6 -, /* mNumStagesTmemSfB */ 6 +, /* mNumStagesMmaAcrossWorkTile */ 2 +, /* mNumStagesSmemSfA */ 4 +, /* mNumStagesSmemSfB */ 4 +, /* mNumStagesTmemSfA */ 4 +, /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 , /* mOutputDebugTensors */ 0 , /* mPatchF2fp */ 0 @@ -67620,15 +66793,14 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfBlockSizeC */ 32 , /* mSfLayoutA */ trtllm::gen::SfLayout(3) , /* mSfLayoutB */ trtllm::gen::SfLayout(0) -, /* mSfLayoutC */ trtllm::gen::SfLayout(1) +, /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 , /* mTileM */ 128 -, /* mTileN */ 16 -, /* mTileScheduler */ gemm::TileScheduler(0) +, /* mTileN */ 128 +, /* mTileScheduler */ gemm::TileScheduler(1) , /* mTransposeMmaOutput */ 1 , /* mUseCustomizedMma3xNvFp4 */ 0 , /* mUseCustomMmaSchedule */ 1 @@ -67642,12 +66814,12 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mUseTmaStore */ 1 , /* mUseTwoTmaLoadWarps */ 1 , /* mUseTwoMmaWarps */ 0 -, /* mUseUnrollLoop2xForMma */ 0 +, /* mUseUnrollLoop2xForMma */ 1 , /* mValidM */ 256 , /* mValidN */ 256 -, /* mValidK */ 256 +, /* mValidK */ 512 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(0) +, /* mActType */ gemmGatedAct::ActType(2) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -67659,25 +66831,26 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mIsUniformNumTokensPerBatch */ 0 , /* mNumBatches */ 128 , /* mNumRegsPerThreadLoadA */ 0 -, /* mNumRegsPerThreadLoadB */ 0 +, /* mNumRegsPerThreadLoadB */ 48 , /* mNumRegsPerThreadLoadSfA */ 0 -, /* mNumRegsPerThreadLoadSfB */ 0 +, /* mNumRegsPerThreadLoadSfB */ 48 , /* mNumTokens */ 2 , /* mNumWarpsLoadA */ 0 -, /* mNumWarpsLoadB */ 0 +, /* mNumWarpsLoadB */ 8 , /* mNumWarpsLoadSfA */ 0 -, /* mNumWarpsLoadSfB */ 0 +, /* mNumWarpsLoadSfB */ 4 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 197360, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "63ed536aeab6f6b4a971a27e9e0ac9f76fc14c283538b91fcba96a847175b4f6", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256u2_s4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256u2_s4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f_cubin_len, 215704, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256u2_s4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f", 768, "3aeb51137ba1f1f595d15df7e6cbda62f715d5eceab772f9b35e52c2f8a6aeac", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 , /* mClcFastDrain */ 1 -, /* mClusterDimX */ 1 +, /* mClusterDimX */ 2 , /* mClusterDimY */ 1 , /* mClusterDimZ */ 1 , /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) @@ -67687,14 +66860,14 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17827853) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 -, /* mEpilogueTileN */ 16 +, /* mEpilogueTileN */ 32 , /* mFallbackClusterDimX */ 1 , /* mFallbackClusterDimY */ 1 , /* mFallbackClusterDimZ */ 1 @@ -67715,8 +66888,8 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mM */ 256 , /* mMmaK */ 32 , /* mMmaKind */ trtllm::gen::MmaKind(5) -, /* mMmaM */ 128 -, /* mMmaN */ 16 +, /* mMmaM */ 256 +, /* mMmaN */ 128 , /* mMmaTileK */ 256 , /* mMockAllReduce */ 0 , /* mN */ 256 @@ -67724,20 +66897,18 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsCastAWarps */ 0 , /* mNumRegsCopySfLdsSttm */ 0 , /* mNumRegsCopySparsityInfo */ 0 -, /* mNumRegsPerThreadEpilogueWarp */ 0 -, /* mNumRegsPerThreadNonEpilogueWarp */ 0 +, /* mNumRegsPerThreadEpilogueWarp */ 152 +, /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 -, /* mNumStagesA */ 5 -, /* mNumStagesB */ 5 +, /* mNumStagesA */ 4 +, /* mNumStagesB */ 4 , /* mNumStagesMma */ 2 , /* mNumStagesMmaWithinWorkTile */ 1 , /* mNumStagesMmaAcrossWorkTile */ 2 -, /* mNumStagesSmemSfA */ 5 -, /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 -, /* mNumStagesTmemSfA */ 5 -, /* mNumStagesTmemSfB */ 5 +, /* mNumStagesSmemSfA */ 4 +, /* mNumStagesSmemSfB */ 4 +, /* mNumStagesTmemSfA */ 4 +, /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 , /* mOutputDebugTensors */ 0 , /* mPatchF2fp */ 0 @@ -67748,14 +66919,13 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfBlockSizeC */ 32 , /* mSfLayoutA */ trtllm::gen::SfLayout(3) , /* mSfLayoutB */ trtllm::gen::SfLayout(0) -, /* mSfLayoutC */ trtllm::gen::SfLayout(1) +, /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 , /* mTileM */ 128 -, /* mTileN */ 16 +, /* mTileN */ 128 , /* mTileScheduler */ gemm::TileScheduler(1) , /* mTransposeMmaOutput */ 1 , /* mUseCustomizedMma3xNvFp4 */ 0 @@ -67787,22 +66957,23 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mIsUniformNumTokensPerBatch */ 0 , /* mNumBatches */ 128 , /* mNumRegsPerThreadLoadA */ 0 -, /* mNumRegsPerThreadLoadB */ 0 +, /* mNumRegsPerThreadLoadB */ 48 , /* mNumRegsPerThreadLoadSfA */ 0 -, /* mNumRegsPerThreadLoadSfB */ 0 +, /* mNumRegsPerThreadLoadSfB */ 48 , /* mNumTokens */ 2 , /* mNumWarpsLoadA */ 0 -, /* mNumWarpsLoadB */ 0 +, /* mNumWarpsLoadB */ 8 , /* mNumWarpsLoadSfA */ 0 -, /* mNumWarpsLoadSfB */ 0 +, /* mNumWarpsLoadSfB */ 4 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 197120, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "44cdce21f9fc0918e13ce2799ab9d38d5289ef1bb7c48455202db45f3133cb92", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len, 197520, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f", 640, "5bfdd8cd202d3f33f62029a0c0008ad3c2b682be38575a8f6384a68dfe3c0d8b", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) -, /* mBiasType */ gemm::BiasType(1) +, /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 , /* mClcFastDrain */ 1 , /* mClusterDimX */ 1 @@ -67815,10 +66986,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17827853) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -67836,7 +67007,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mGridWaitForPrimaryB */ 1 , /* mHoistLoadTaskInit */ 1 , /* mHoistMmaTaskTryWaits */ 0 -, /* mK */ 512 +, /* mK */ 256 , /* mKernelTraits */ {} , /* mLayoutA */ gemm::MatrixLayout(0) , /* mLayoutB */ gemm::MatrixLayout(0) @@ -67855,15 +67026,13 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 -, /* mNumStagesMma */ 1 +, /* mNumStagesMma */ 2 , /* mNumStagesMmaWithinWorkTile */ 1 -, /* mNumStagesMmaAcrossWorkTile */ 1 +, /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -67878,13 +67047,12 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 , /* mTileM */ 128 , /* mTileN */ 16 -, /* mTileScheduler */ gemm::TileScheduler(0) +, /* mTileScheduler */ gemm::TileScheduler(1) , /* mTransposeMmaOutput */ 1 , /* mUseCustomizedMma3xNvFp4 */ 0 , /* mUseCustomMmaSchedule */ 1 @@ -67898,12 +67066,12 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mUseTmaStore */ 1 , /* mUseTwoTmaLoadWarps */ 1 , /* mUseTwoMmaWarps */ 0 -, /* mUseUnrollLoop2xForMma */ 1 +, /* mUseUnrollLoop2xForMma */ 0 , /* mValidM */ 256 , /* mValidN */ 256 -, /* mValidK */ 512 +, /* mValidK */ 256 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(0) +, /* mActType */ gemmGatedAct::ActType(2) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -67925,15 +67093,16 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 223064, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "2504d15af54e2492742798aae3becaf79524e009b5e471f94da07f091060a671", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 197520, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "0245ae9b677c31722373472f7dce52595865aff37079c15f4354aa0fed8ffc38", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 , /* mClcFastDrain */ 1 -, /* mClusterDimX */ 2 +, /* mClusterDimX */ 1 , /* mClusterDimY */ 1 , /* mClusterDimZ */ 1 , /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) @@ -67943,10 +67112,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17827853) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -67964,14 +67133,14 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mGridWaitForPrimaryB */ 1 , /* mHoistLoadTaskInit */ 1 , /* mHoistMmaTaskTryWaits */ 0 -, /* mK */ 512 +, /* mK */ 256 , /* mKernelTraits */ {} , /* mLayoutA */ gemm::MatrixLayout(0) , /* mLayoutB */ gemm::MatrixLayout(0) , /* mM */ 256 , /* mMmaK */ 32 , /* mMmaKind */ trtllm::gen::MmaKind(5) -, /* mMmaM */ 256 +, /* mMmaM */ 128 , /* mMmaN */ 16 , /* mMmaTileK */ 256 , /* mMockAllReduce */ 0 @@ -67983,17 +67152,15 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 -, /* mNumStagesA */ 6 -, /* mNumStagesB */ 6 +, /* mNumStagesA */ 5 +, /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 , /* mNumStagesMmaWithinWorkTile */ 1 , /* mNumStagesMmaAcrossWorkTile */ 2 -, /* mNumStagesSmemSfA */ 6 -, /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 -, /* mNumStagesTmemSfA */ 6 -, /* mNumStagesTmemSfB */ 6 +, /* mNumStagesSmemSfA */ 5 +, /* mNumStagesSmemSfB */ 5 +, /* mNumStagesTmemSfA */ 5 +, /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 , /* mOutputDebugTensors */ 0 , /* mPatchF2fp */ 0 @@ -68006,7 +67173,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -68026,10 +67192,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mUseTmaStore */ 1 , /* mUseTwoTmaLoadWarps */ 1 , /* mUseTwoMmaWarps */ 0 -, /* mUseUnrollLoop2xForMma */ 1 +, /* mUseUnrollLoop2xForMma */ 0 , /* mValidM */ 256 , /* mValidN */ 256 -, /* mValidK */ 512 +, /* mValidK */ 256 , /* mWorldSize */ 1 , /* mActType */ gemmGatedAct::ActType(0) , /* mClampBeforeAct */ 1 @@ -68053,15 +67219,16 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 222824, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "2a872442c36df7c48ae4e5170e6d9e2b948dfcf21fc2e6bdf3f899c6dbc235b2", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len, 197136, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f", 640, "b807bf53f5272a0c48fffb35a55c4364a1d9a0aeb94b880ab6ca688149971111", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) -, /* mBiasType */ gemm::BiasType(1) +, /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 , /* mClcFastDrain */ 1 -, /* mClusterDimX */ 2 +, /* mClusterDimX */ 1 , /* mClusterDimY */ 1 , /* mClusterDimZ */ 1 , /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) @@ -68071,10 +67238,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17827853) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -68092,14 +67259,14 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mGridWaitForPrimaryB */ 1 , /* mHoistLoadTaskInit */ 1 , /* mHoistMmaTaskTryWaits */ 0 -, /* mK */ 512 +, /* mK */ 256 , /* mKernelTraits */ {} , /* mLayoutA */ gemm::MatrixLayout(0) , /* mLayoutB */ gemm::MatrixLayout(0) , /* mM */ 256 , /* mMmaK */ 32 , /* mMmaKind */ trtllm::gen::MmaKind(5) -, /* mMmaM */ 256 +, /* mMmaM */ 128 , /* mMmaN */ 16 , /* mMmaTileK */ 256 , /* mMockAllReduce */ 0 @@ -68111,17 +67278,15 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 -, /* mNumStagesA */ 6 -, /* mNumStagesB */ 6 +, /* mNumStagesA */ 5 +, /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 , /* mNumStagesMmaWithinWorkTile */ 1 , /* mNumStagesMmaAcrossWorkTile */ 1 -, /* mNumStagesSmemSfA */ 6 -, /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 -, /* mNumStagesTmemSfA */ 6 -, /* mNumStagesTmemSfB */ 6 +, /* mNumStagesSmemSfA */ 5 +, /* mNumStagesSmemSfB */ 5 +, /* mNumStagesTmemSfA */ 5 +, /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 , /* mOutputDebugTensors */ 0 , /* mPatchF2fp */ 0 @@ -68134,7 +67299,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -68154,12 +67318,12 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mUseTmaStore */ 1 , /* mUseTwoTmaLoadWarps */ 1 , /* mUseTwoMmaWarps */ 0 -, /* mUseUnrollLoop2xForMma */ 1 +, /* mUseUnrollLoop2xForMma */ 0 , /* mValidM */ 256 , /* mValidN */ 256 -, /* mValidK */ 512 +, /* mValidK */ 256 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(0) +, /* mActType */ gemmGatedAct::ActType(2) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -68181,143 +67345,16 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 - }, gemm::SmVersion::Sm100f}, -{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x256x128_s6x6x6x6x1x6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x256x128_s6x6x6x6x1x6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f_cubin_len, 222872, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x256x128_s6x6x6x6x1x6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f", 640, "d04ab7da46bcc6cc9c217caeb998106f9e6b11c05166de1f47fc007997f37210", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) -, /* mBiasDtype */ trtllm::gen::Dtype(1056776) -, /* mBiasType */ gemm::BiasType(1) -, /* mBlockK */ -1 -, /* mClcFastDrain */ 1 -, /* mClusterDimX */ 2 -, /* mClusterDimY */ 1 -, /* mClusterDimZ */ 1 -, /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) -, /* mDtypeAcc */ trtllm::gen::Dtype(1056776) -, /* mDtypeA */ trtllm::gen::Dtype(17826828) -, /* mDtypeB */ trtllm::gen::Dtype(17827853) -, /* mDtypeC */ trtllm::gen::Dtype(17827853) -, /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) -, /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) -, /* mEltwiseActType */ gemm::EltwiseActType(0) -, /* mEnablesEarlyExit */ 1 -, /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 -, /* mEpilogueLdtmDps */ 16 -, /* mEpilogueLdtmBits */ 256 -, /* mEpilogueTileM */ 128 -, /* mEpilogueTileN */ 64 -, /* mFallbackClusterDimX */ 1 -, /* mFallbackClusterDimY */ 1 -, /* mFallbackClusterDimZ */ 1 -, /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) -, /* mFuseLoadSfTask */ 0 -, /* mFuseUtccpWithUtcmma */ 1 -, /* mGridTriggerSecondaryA */ 0 -, /* mGridTriggerSecondaryB */ 1 -, /* mGridWaitForPrimaryEarlyExit */ 1 -, /* mGridWaitForPrimaryA */ 0 -, /* mGridWaitForPrimaryB */ 1 -, /* mHoistLoadTaskInit */ 1 -, /* mHoistMmaTaskTryWaits */ 0 -, /* mK */ 128 -, /* mKernelTraits */ {} -, /* mLayoutA */ gemm::MatrixLayout(0) -, /* mLayoutB */ gemm::MatrixLayout(0) -, /* mM */ 256 -, /* mMmaK */ 32 -, /* mMmaKind */ trtllm::gen::MmaKind(5) -, /* mMmaM */ 256 -, /* mMmaN */ 256 -, /* mMmaTileK */ 128 -, /* mMockAllReduce */ 0 -, /* mN */ 256 -, /* mNumEpilogueWarps */ 4 -, /* mNumRegsCastAWarps */ 0 -, /* mNumRegsCopySfLdsSttm */ 0 -, /* mNumRegsCopySparsityInfo */ 0 -, /* mNumRegsPerThreadEpilogueWarp */ 144 -, /* mNumRegsPerThreadNonEpilogueWarp */ 88 -, /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 -, /* mNumStagesA */ 6 -, /* mNumStagesB */ 6 -, /* mNumStagesMma */ 1 -, /* mNumStagesMmaWithinWorkTile */ 1 -, /* mNumStagesMmaAcrossWorkTile */ 1 -, /* mNumStagesSmemSfA */ 6 -, /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 -, /* mNumStagesTmemSfA */ 1 -, /* mNumStagesTmemSfB */ 6 -, /* mNumStagesWorkId */ 3 -, /* mOutputDebugTensors */ 0 -, /* mPatchF2fp */ 0 -, /* mPerTokenSfDtype */ trtllm::gen::Dtype(1052672) -, /* mSchedHostTask */ gemm::SchedHostTask(0) -, /* mSfBlockSizeA */ 32 -, /* mSfBlockSizeB */ 32 -, /* mSfBlockSizeC */ 32 -, /* mSfLayoutA */ trtllm::gen::SfLayout(3) -, /* mSfLayoutB */ trtllm::gen::SfLayout(0) -, /* mSfLayoutC */ trtllm::gen::SfLayout(3) -, /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 -, /* mSparsityA */ trtllm::gen::Sparsity(0) -, /* mSplitK */ gemm::SplitK(0) -, /* mTileK */ 128 -, /* mTileM */ 128 -, /* mTileN */ 256 -, /* mTileScheduler */ gemm::TileScheduler(1) -, /* mTransposeMmaOutput */ 1 -, /* mUseCustomizedMma3xNvFp4 */ 0 -, /* mUseCustomMmaSchedule */ 1 -, /* mUseDeepSeekFp8 */ 0 -, /* mUseFlexibleClusterDims */ 0 -, /* mUseHoistTryWaitForCustomMmaSchedule */ 0 -, /* mUseMaxTmemOverlap */ 1 -, /* mUsePerTokenSfA */ 0 -, /* mUsePerTokenSfB */ 0 -, /* mUseShuffledMatrix */ 1 -, /* mUseTmaStore */ 1 -, /* mUseTwoTmaLoadWarps */ 1 -, /* mUseTwoMmaWarps */ 0 -, /* mUseUnrollLoop2xForMma */ 0 -, /* mValidM */ 256 -, /* mValidN */ 256 -, /* mValidK */ 128 -, /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(0) -, /* mClampBeforeAct */ 1 -, /* mBatchedM */ {} -, /* mBatchedN */ {} -, /* mBatchMode */ batchedGemm::BatchedGemmOptions::BatchMode(1) -, /* mBatchStrideInTokens */ -1 -, /* mFusedAct */ 1 -, /* mGridWaitForPrimaryRouting */ 1 -, /* mIsStaticBatch */ 0 -, /* mIsUniformNumTokensPerBatch */ 0 -, /* mNumBatches */ 128 -, /* mNumRegsPerThreadLoadA */ 0 -, /* mNumRegsPerThreadLoadB */ 56 -, /* mNumRegsPerThreadLoadSfA */ 0 -, /* mNumRegsPerThreadLoadSfB */ 56 -, /* mNumTokens */ 2 -, /* mNumWarpsLoadA */ 0 -, /* mNumWarpsLoadB */ 8 -, /* mNumWarpsLoadSfA */ 0 -, /* mNumWarpsLoadSfB */ 4 -, /* mRouteImpl */ batchedGemm::RouteImpl(2) -, /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} +, /* mUseCMultiCast */ 0 , /* mUseTmaOobOpt */ 1 -, /* mPrefetchB */ 0 }, gemm::SmVersion::Sm100f}, -{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x256x256_s3x3x3x3x1x3_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x256x256_s3x3x3x3x1x3_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f_cubin_len, 222680, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x256x256_s3x3x3x3x1x3_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f", 640, "2c7ebc6aafba9a3558cf9b9582afc7ff69f203aee036f8aedf2e71a521c347a8", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 197136, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "e7c3707ecc45a56252b7a6412916d0e311441fd351864e9803c1f62fae248d0c", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 , /* mClcFastDrain */ 1 -, /* mClusterDimX */ 2 +, /* mClusterDimX */ 1 , /* mClusterDimY */ 1 , /* mClusterDimZ */ 1 , /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) @@ -68327,20 +67364,20 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17827853) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 -, /* mEpilogueTileN */ 64 +, /* mEpilogueTileN */ 16 , /* mFallbackClusterDimX */ 1 , /* mFallbackClusterDimY */ 1 , /* mFallbackClusterDimZ */ 1 , /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) , /* mFuseLoadSfTask */ 0 -, /* mFuseUtccpWithUtcmma */ 1 +, /* mFuseUtccpWithUtcmma */ 0 , /* mGridTriggerSecondaryA */ 0 , /* mGridTriggerSecondaryB */ 1 , /* mGridWaitForPrimaryEarlyExit */ 1 @@ -68355,8 +67392,8 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mM */ 256 , /* mMmaK */ 32 , /* mMmaKind */ trtllm::gen::MmaKind(5) -, /* mMmaM */ 256 -, /* mMmaN */ 256 +, /* mMmaM */ 128 +, /* mMmaN */ 16 , /* mMmaTileK */ 256 , /* mMockAllReduce */ 0 , /* mN */ 256 @@ -68364,20 +67401,18 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsCastAWarps */ 0 , /* mNumRegsCopySfLdsSttm */ 0 , /* mNumRegsCopySparsityInfo */ 0 -, /* mNumRegsPerThreadEpilogueWarp */ 144 -, /* mNumRegsPerThreadNonEpilogueWarp */ 88 +, /* mNumRegsPerThreadEpilogueWarp */ 0 +, /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 -, /* mNumStagesA */ 3 -, /* mNumStagesB */ 3 +, /* mNumStagesA */ 5 +, /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 , /* mNumStagesMmaWithinWorkTile */ 1 , /* mNumStagesMmaAcrossWorkTile */ 1 -, /* mNumStagesSmemSfA */ 3 -, /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 -, /* mNumStagesTmemSfA */ 1 -, /* mNumStagesTmemSfB */ 3 +, /* mNumStagesSmemSfA */ 5 +, /* mNumStagesSmemSfB */ 5 +, /* mNumStagesTmemSfA */ 5 +, /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 , /* mOutputDebugTensors */ 0 , /* mPatchF2fp */ 0 @@ -68388,22 +67423,21 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfBlockSizeC */ 32 , /* mSfLayoutA */ trtllm::gen::SfLayout(3) , /* mSfLayoutB */ trtllm::gen::SfLayout(0) -, /* mSfLayoutC */ trtllm::gen::SfLayout(3) +, /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 , /* mTileM */ 128 -, /* mTileN */ 256 -, /* mTileScheduler */ gemm::TileScheduler(1) +, /* mTileN */ 16 +, /* mTileScheduler */ gemm::TileScheduler(0) , /* mTransposeMmaOutput */ 1 , /* mUseCustomizedMma3xNvFp4 */ 0 , /* mUseCustomMmaSchedule */ 1 , /* mUseDeepSeekFp8 */ 0 , /* mUseFlexibleClusterDims */ 0 , /* mUseHoistTryWaitForCustomMmaSchedule */ 0 -, /* mUseMaxTmemOverlap */ 1 +, /* mUseMaxTmemOverlap */ 0 , /* mUsePerTokenSfA */ 0 , /* mUsePerTokenSfB */ 0 , /* mUseShuffledMatrix */ 1 @@ -68427,25 +67461,26 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mIsUniformNumTokensPerBatch */ 0 , /* mNumBatches */ 128 , /* mNumRegsPerThreadLoadA */ 0 -, /* mNumRegsPerThreadLoadB */ 56 +, /* mNumRegsPerThreadLoadB */ 0 , /* mNumRegsPerThreadLoadSfA */ 0 -, /* mNumRegsPerThreadLoadSfB */ 56 +, /* mNumRegsPerThreadLoadSfB */ 0 , /* mNumTokens */ 2 , /* mNumWarpsLoadA */ 0 -, /* mNumWarpsLoadB */ 8 +, /* mNumWarpsLoadB */ 0 , /* mNumWarpsLoadSfA */ 0 -, /* mNumWarpsLoadSfB */ 4 +, /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 220912, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "5d2ee179317ac92383d46782cf6760d4381b13b2279cca2823541a849278b477", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len, 223128, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f", 512, "3f2a93dba4bfc40006c609808e12329f0c33a01963e97d72c6657b1e2e70e302", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) -, /* mBiasType */ gemm::BiasType(1) +, /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 , /* mClcFastDrain */ 1 -, /* mClusterDimX */ 1 +, /* mClusterDimX */ 2 , /* mClusterDimY */ 1 , /* mClusterDimZ */ 1 , /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) @@ -68455,14 +67490,14 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17827853) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 -, /* mEpilogueTileN */ 32 +, /* mEpilogueTileN */ 16 , /* mFallbackClusterDimX */ 1 , /* mFallbackClusterDimY */ 1 , /* mFallbackClusterDimZ */ 1 @@ -68483,8 +67518,8 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mM */ 256 , /* mMmaK */ 32 , /* mMmaKind */ trtllm::gen::MmaKind(5) -, /* mMmaM */ 128 -, /* mMmaN */ 32 +, /* mMmaM */ 256 +, /* mMmaN */ 16 , /* mMmaTileK */ 256 , /* mMockAllReduce */ 0 , /* mN */ 256 @@ -68495,17 +67530,15 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 -, /* mNumStagesA */ 5 -, /* mNumStagesB */ 5 +, /* mNumStagesA */ 6 +, /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 , /* mNumStagesMmaWithinWorkTile */ 1 , /* mNumStagesMmaAcrossWorkTile */ 2 -, /* mNumStagesSmemSfA */ 5 -, /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 -, /* mNumStagesTmemSfA */ 5 -, /* mNumStagesTmemSfB */ 5 +, /* mNumStagesSmemSfA */ 6 +, /* mNumStagesSmemSfB */ 6 +, /* mNumStagesTmemSfA */ 6 +, /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 , /* mOutputDebugTensors */ 0 , /* mPatchF2fp */ 0 @@ -68518,12 +67551,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 , /* mTileM */ 128 -, /* mTileN */ 32 +, /* mTileN */ 16 , /* mTileScheduler */ gemm::TileScheduler(1) , /* mTransposeMmaOutput */ 1 , /* mUseCustomizedMma3xNvFp4 */ 0 @@ -68543,7 +67575,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 256 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(0) +, /* mActType */ gemmGatedAct::ActType(2) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -68565,15 +67597,16 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 220672, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "c2bf0a08790e440abc6b09c9801c5fc893d873b41ec333ea3a4388b0efa25baf", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 223128, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "563a2d87e62b4629843aea9e92838883779557940c42039e5623a18dd1ad91e2", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 , /* mClcFastDrain */ 1 -, /* mClusterDimX */ 1 +, /* mClusterDimX */ 2 , /* mClusterDimY */ 1 , /* mClusterDimZ */ 1 , /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) @@ -68583,14 +67616,14 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17827853) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 -, /* mEpilogueTileN */ 32 +, /* mEpilogueTileN */ 16 , /* mFallbackClusterDimX */ 1 , /* mFallbackClusterDimY */ 1 , /* mFallbackClusterDimZ */ 1 @@ -68611,8 +67644,8 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mM */ 256 , /* mMmaK */ 32 , /* mMmaKind */ trtllm::gen::MmaKind(5) -, /* mMmaM */ 128 -, /* mMmaN */ 32 +, /* mMmaM */ 256 +, /* mMmaN */ 16 , /* mMmaTileK */ 256 , /* mMockAllReduce */ 0 , /* mN */ 256 @@ -68623,17 +67656,15 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 -, /* mNumStagesA */ 5 -, /* mNumStagesB */ 5 -, /* mNumStagesMma */ 1 +, /* mNumStagesA */ 6 +, /* mNumStagesB */ 6 +, /* mNumStagesMma */ 2 , /* mNumStagesMmaWithinWorkTile */ 1 -, /* mNumStagesMmaAcrossWorkTile */ 1 -, /* mNumStagesSmemSfA */ 5 -, /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 -, /* mNumStagesTmemSfA */ 5 -, /* mNumStagesTmemSfB */ 5 +, /* mNumStagesMmaAcrossWorkTile */ 2 +, /* mNumStagesSmemSfA */ 6 +, /* mNumStagesSmemSfB */ 6 +, /* mNumStagesTmemSfA */ 6 +, /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 , /* mOutputDebugTensors */ 0 , /* mPatchF2fp */ 0 @@ -68646,13 +67677,12 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 , /* mTileM */ 128 -, /* mTileN */ 32 -, /* mTileScheduler */ gemm::TileScheduler(0) +, /* mTileN */ 16 +, /* mTileScheduler */ gemm::TileScheduler(1) , /* mTransposeMmaOutput */ 1 , /* mUseCustomizedMma3xNvFp4 */ 0 , /* mUseCustomMmaSchedule */ 1 @@ -68693,12 +67723,13 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 200440, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "fa171165a44c941279e2b0e7faa3ac5d85e88f1035abc0bb8d00dabc2ae0c2f9", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len, 223000, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f", 512, "e53f7bdc5beb779215da83ed06f8b2ff7fdd371093c036e1c228dc7520132bb5", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) -, /* mBiasType */ gemm::BiasType(1) +, /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 , /* mClcFastDrain */ 1 , /* mClusterDimX */ 2 @@ -68711,14 +67742,14 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17827853) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 -, /* mEpilogueTileN */ 32 +, /* mEpilogueTileN */ 16 , /* mFallbackClusterDimX */ 1 , /* mFallbackClusterDimY */ 1 , /* mFallbackClusterDimZ */ 1 @@ -68740,7 +67771,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mMmaK */ 32 , /* mMmaKind */ trtllm::gen::MmaKind(5) , /* mMmaM */ 256 -, /* mMmaN */ 32 +, /* mMmaN */ 16 , /* mMmaTileK */ 256 , /* mMockAllReduce */ 0 , /* mN */ 256 @@ -68751,17 +67782,15 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 -, /* mNumStagesA */ 5 -, /* mNumStagesB */ 5 -, /* mNumStagesMma */ 2 +, /* mNumStagesA */ 6 +, /* mNumStagesB */ 6 +, /* mNumStagesMma */ 1 , /* mNumStagesMmaWithinWorkTile */ 1 -, /* mNumStagesMmaAcrossWorkTile */ 2 -, /* mNumStagesSmemSfA */ 5 -, /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 -, /* mNumStagesTmemSfA */ 5 -, /* mNumStagesTmemSfB */ 5 +, /* mNumStagesMmaAcrossWorkTile */ 1 +, /* mNumStagesSmemSfA */ 6 +, /* mNumStagesSmemSfB */ 6 +, /* mNumStagesTmemSfA */ 6 +, /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 , /* mOutputDebugTensors */ 0 , /* mPatchF2fp */ 0 @@ -68774,13 +67803,12 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 , /* mTileM */ 128 -, /* mTileN */ 32 -, /* mTileScheduler */ gemm::TileScheduler(1) +, /* mTileN */ 16 +, /* mTileScheduler */ gemm::TileScheduler(0) , /* mTransposeMmaOutput */ 1 , /* mUseCustomizedMma3xNvFp4 */ 0 , /* mUseCustomMmaSchedule */ 1 @@ -68799,7 +67827,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 256 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(0) +, /* mActType */ gemmGatedAct::ActType(2) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -68821,10 +67849,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 200200, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "acad4fdce5e38654daa2779c33cb031b5d53e5ef4f1882de8810d62a64ff7d45", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 223000, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "2e106095c57d7750c2d4e3b8b0c1ae240dfdbbfcc2b5af6f8a0f8c96477d6e94", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -68839,14 +67868,14 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17827853) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 -, /* mEpilogueTileN */ 32 +, /* mEpilogueTileN */ 16 , /* mFallbackClusterDimX */ 1 , /* mFallbackClusterDimY */ 1 , /* mFallbackClusterDimZ */ 1 @@ -68868,7 +67897,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mMmaK */ 32 , /* mMmaKind */ trtllm::gen::MmaKind(5) , /* mMmaM */ 256 -, /* mMmaN */ 32 +, /* mMmaN */ 16 , /* mMmaTileK */ 256 , /* mMockAllReduce */ 0 , /* mN */ 256 @@ -68879,17 +67908,15 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 -, /* mNumStagesA */ 5 -, /* mNumStagesB */ 5 +, /* mNumStagesA */ 6 +, /* mNumStagesB */ 6 , /* mNumStagesMma */ 1 , /* mNumStagesMmaWithinWorkTile */ 1 , /* mNumStagesMmaAcrossWorkTile */ 1 -, /* mNumStagesSmemSfA */ 5 -, /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 -, /* mNumStagesTmemSfA */ 5 -, /* mNumStagesTmemSfB */ 5 +, /* mNumStagesSmemSfA */ 6 +, /* mNumStagesSmemSfB */ 6 +, /* mNumStagesTmemSfA */ 6 +, /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 , /* mOutputDebugTensors */ 0 , /* mPatchF2fp */ 0 @@ -68902,12 +67929,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 , /* mTileM */ 128 -, /* mTileN */ 32 +, /* mTileN */ 16 , /* mTileScheduler */ gemm::TileScheduler(0) , /* mTransposeMmaOutput */ 1 , /* mUseCustomizedMma3xNvFp4 */ 0 @@ -68949,12 +67975,13 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 220912, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "6e678562fa83ad5a8887b1deb0d2fa1c8da95cfd353baefee28d53ebe8c22d9d", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len, 197520, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f", 640, "7fd2d8906fc0c13a304612162c55e76360c2dbbf412d424e42b7de9f511f44e3", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) -, /* mBiasType */ gemm::BiasType(1) +, /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 , /* mClcFastDrain */ 1 , /* mClusterDimX */ 1 @@ -68967,14 +67994,14 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17827853) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 -, /* mEpilogueTileN */ 32 +, /* mEpilogueTileN */ 16 , /* mFallbackClusterDimX */ 1 , /* mFallbackClusterDimY */ 1 , /* mFallbackClusterDimZ */ 1 @@ -68996,7 +68023,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mMmaK */ 32 , /* mMmaKind */ trtllm::gen::MmaKind(5) , /* mMmaM */ 128 -, /* mMmaN */ 32 +, /* mMmaN */ 16 , /* mMmaTileK */ 256 , /* mMockAllReduce */ 0 , /* mN */ 256 @@ -69007,7 +68034,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -69015,7 +68041,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -69030,12 +68055,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 , /* mTileM */ 128 -, /* mTileN */ 32 +, /* mTileN */ 16 , /* mTileScheduler */ gemm::TileScheduler(1) , /* mTransposeMmaOutput */ 1 , /* mUseCustomizedMma3xNvFp4 */ 0 @@ -69055,7 +68079,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 512 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(0) +, /* mActType */ gemmGatedAct::ActType(2) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -69077,10 +68101,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 220672, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "a7f4224ab48e43ff9dba901cd86b1c258c3179fe279dbcd19bfdbda7da1dcab8", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 197520, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "891acad6ccbea3f0047c96feee6b142333cbfcbf511a2a452f8f9a1df541c96b", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -69095,14 +68120,14 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17827853) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 -, /* mEpilogueTileN */ 32 +, /* mEpilogueTileN */ 16 , /* mFallbackClusterDimX */ 1 , /* mFallbackClusterDimY */ 1 , /* mFallbackClusterDimZ */ 1 @@ -69124,7 +68149,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mMmaK */ 32 , /* mMmaKind */ trtllm::gen::MmaKind(5) , /* mMmaM */ 128 -, /* mMmaN */ 32 +, /* mMmaN */ 16 , /* mMmaTileK */ 256 , /* mMockAllReduce */ 0 , /* mN */ 256 @@ -69135,15 +68160,13 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 -, /* mNumStagesMma */ 1 +, /* mNumStagesMma */ 2 , /* mNumStagesMmaWithinWorkTile */ 1 -, /* mNumStagesMmaAcrossWorkTile */ 1 +, /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -69158,13 +68181,12 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 , /* mTileM */ 128 -, /* mTileN */ 32 -, /* mTileScheduler */ gemm::TileScheduler(0) +, /* mTileN */ 16 +, /* mTileScheduler */ gemm::TileScheduler(1) , /* mTransposeMmaOutput */ 1 , /* mUseCustomizedMma3xNvFp4 */ 0 , /* mUseCustomMmaSchedule */ 1 @@ -69205,15 +68227,16 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 200440, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "14507bc995f04e42bf6d3b0cc7c3781431e1dc277e5a26e8fb2ac7ff4ee0918e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len, 197136, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f", 640, "798ac5b1f13b50f3326a7de1988b2b918344d1f3b7bbf97082a1783d4094a4af", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) -, /* mBiasType */ gemm::BiasType(1) +, /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 , /* mClcFastDrain */ 1 -, /* mClusterDimX */ 2 +, /* mClusterDimX */ 1 , /* mClusterDimY */ 1 , /* mClusterDimZ */ 1 , /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) @@ -69223,14 +68246,14 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17827853) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 -, /* mEpilogueTileN */ 32 +, /* mEpilogueTileN */ 16 , /* mFallbackClusterDimX */ 1 , /* mFallbackClusterDimY */ 1 , /* mFallbackClusterDimZ */ 1 @@ -69251,8 +68274,8 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mM */ 256 , /* mMmaK */ 32 , /* mMmaKind */ trtllm::gen::MmaKind(5) -, /* mMmaM */ 256 -, /* mMmaN */ 32 +, /* mMmaM */ 128 +, /* mMmaN */ 16 , /* mMmaTileK */ 256 , /* mMockAllReduce */ 0 , /* mN */ 256 @@ -69263,15 +68286,13 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 -, /* mNumStagesMma */ 2 +, /* mNumStagesMma */ 1 , /* mNumStagesMmaWithinWorkTile */ 1 -, /* mNumStagesMmaAcrossWorkTile */ 2 +, /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -69286,13 +68307,12 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 , /* mTileM */ 128 -, /* mTileN */ 32 -, /* mTileScheduler */ gemm::TileScheduler(1) +, /* mTileN */ 16 +, /* mTileScheduler */ gemm::TileScheduler(0) , /* mTransposeMmaOutput */ 1 , /* mUseCustomizedMma3xNvFp4 */ 0 , /* mUseCustomMmaSchedule */ 1 @@ -69311,7 +68331,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mValidN */ 256 , /* mValidK */ 512 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(0) +, /* mActType */ gemmGatedAct::ActType(2) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -69333,15 +68353,16 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 200200, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "53fa70cf2d7c706ebfd61185c56f11b06f65db62843036559e96f42412ffa3fb", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 197136, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "489e8f4b6743eaea6911e8258dc64c4d962b9558ed91b793dce6faa3554d2a1f", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 , /* mClcFastDrain */ 1 -, /* mClusterDimX */ 2 +, /* mClusterDimX */ 1 , /* mClusterDimY */ 1 , /* mClusterDimZ */ 1 , /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) @@ -69351,14 +68372,14 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17827853) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 -, /* mEpilogueTileN */ 32 +, /* mEpilogueTileN */ 16 , /* mFallbackClusterDimX */ 1 , /* mFallbackClusterDimY */ 1 , /* mFallbackClusterDimZ */ 1 @@ -69379,8 +68400,8 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mM */ 256 , /* mMmaK */ 32 , /* mMmaKind */ trtllm::gen::MmaKind(5) -, /* mMmaM */ 256 -, /* mMmaN */ 32 +, /* mMmaM */ 128 +, /* mMmaN */ 16 , /* mMmaTileK */ 256 , /* mMockAllReduce */ 0 , /* mN */ 256 @@ -69391,7 +68412,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -69399,7 +68419,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -69414,12 +68433,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 , /* mTileM */ 128 -, /* mTileN */ 32 +, /* mTileN */ 16 , /* mTileScheduler */ gemm::TileScheduler(0) , /* mTransposeMmaOutput */ 1 , /* mUseCustomizedMma3xNvFp4 */ 0 @@ -69461,12 +68479,13 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x128_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW1_lsfbW1_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x128_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW1_lsfbW1_dynB_sm100f_cubin_len, 163768, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x128_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW1_lsfbW1_dynB_sm100f", 512, "13e08de3e2c00877a774f817e3e3ea32c287270da68751adc7fd2cd1e275997c", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len, 223128, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f", 512, "3ab8eca687f3f073b82ced294d3d9fe3f04f03779ba53d7c8dee35df0df6f9d0", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) -, /* mBiasType */ gemm::BiasType(1) +, /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 , /* mClcFastDrain */ 1 , /* mClusterDimX */ 2 @@ -69479,14 +68498,14 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17827853) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 -, /* mEpilogueTileN */ 64 +, /* mEpilogueTileN */ 16 , /* mFallbackClusterDimX */ 1 , /* mFallbackClusterDimY */ 1 , /* mFallbackClusterDimZ */ 1 @@ -69500,7 +68519,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mGridWaitForPrimaryB */ 1 , /* mHoistLoadTaskInit */ 1 , /* mHoistMmaTaskTryWaits */ 0 -, /* mK */ 128 +, /* mK */ 512 , /* mKernelTraits */ {} , /* mLayoutA */ gemm::MatrixLayout(0) , /* mLayoutB */ gemm::MatrixLayout(0) @@ -69508,28 +68527,26 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mMmaK */ 32 , /* mMmaKind */ trtllm::gen::MmaKind(5) , /* mMmaM */ 256 -, /* mMmaN */ 64 -, /* mMmaTileK */ 128 +, /* mMmaN */ 16 +, /* mMmaTileK */ 256 , /* mMockAllReduce */ 0 , /* mN */ 256 , /* mNumEpilogueWarps */ 4 , /* mNumRegsCastAWarps */ 0 , /* mNumRegsCopySfLdsSttm */ 0 , /* mNumRegsCopySparsityInfo */ 0 -, /* mNumRegsPerThreadEpilogueWarp */ 168 -, /* mNumRegsPerThreadNonEpilogueWarp */ 48 +, /* mNumRegsPerThreadEpilogueWarp */ 0 +, /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 -, /* mNumStagesA */ 7 -, /* mNumStagesB */ 7 +, /* mNumStagesA */ 6 +, /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 , /* mNumStagesMmaWithinWorkTile */ 1 , /* mNumStagesMmaAcrossWorkTile */ 2 -, /* mNumStagesSmemSfA */ 7 -, /* mNumStagesSmemSfB */ 7 -, /* mNumStagesTmemA */ 0 -, /* mNumStagesTmemSfA */ 7 -, /* mNumStagesTmemSfB */ 7 +, /* mNumStagesSmemSfA */ 6 +, /* mNumStagesSmemSfB */ 6 +, /* mNumStagesTmemSfA */ 6 +, /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 , /* mOutputDebugTensors */ 0 , /* mPatchF2fp */ 0 @@ -69542,12 +68559,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) -, /* mTileK */ 128 +, /* mTileK */ 256 , /* mTileM */ 128 -, /* mTileN */ 64 +, /* mTileN */ 16 , /* mTileScheduler */ gemm::TileScheduler(1) , /* mTransposeMmaOutput */ 1 , /* mUseCustomizedMma3xNvFp4 */ 0 @@ -69562,12 +68578,12 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mUseTmaStore */ 1 , /* mUseTwoTmaLoadWarps */ 1 , /* mUseTwoMmaWarps */ 0 -, /* mUseUnrollLoop2xForMma */ 0 +, /* mUseUnrollLoop2xForMma */ 1 , /* mValidM */ 256 , /* mValidN */ 256 -, /* mValidK */ 128 +, /* mValidK */ 512 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(0) +, /* mActType */ gemmGatedAct::ActType(2) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -69579,20 +68595,21 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mIsUniformNumTokensPerBatch */ 0 , /* mNumBatches */ 128 , /* mNumRegsPerThreadLoadA */ 0 -, /* mNumRegsPerThreadLoadB */ 48 +, /* mNumRegsPerThreadLoadB */ 0 , /* mNumRegsPerThreadLoadSfA */ 0 -, /* mNumRegsPerThreadLoadSfB */ 48 +, /* mNumRegsPerThreadLoadSfB */ 0 , /* mNumTokens */ 2 , /* mNumWarpsLoadA */ 0 -, /* mNumWarpsLoadB */ 1 +, /* mNumWarpsLoadB */ 0 , /* mNumWarpsLoadSfA */ 0 -, /* mNumWarpsLoadSfB */ 1 +, /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x128u2_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW1_lsfbW1_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x128u2_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW1_lsfbW1_dynB_sm100f_cubin_len, 163768, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x128u2_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW1_lsfbW1_dynB_sm100f", 512, "bf0d44a31544910a2541d938abaddf7e89386ff8a5cbb3661c247f188cb9ba61", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 223128, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "efd313e300227160bea6f87373b7cca358acfcac4a9c904fd835321f405b29a0", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -69607,14 +68624,14 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17827853) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 -, /* mEpilogueTileN */ 64 +, /* mEpilogueTileN */ 16 , /* mFallbackClusterDimX */ 1 , /* mFallbackClusterDimY */ 1 , /* mFallbackClusterDimZ */ 1 @@ -69628,7 +68645,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mGridWaitForPrimaryB */ 1 , /* mHoistLoadTaskInit */ 1 , /* mHoistMmaTaskTryWaits */ 0 -, /* mK */ 256 +, /* mK */ 512 , /* mKernelTraits */ {} , /* mLayoutA */ gemm::MatrixLayout(0) , /* mLayoutB */ gemm::MatrixLayout(0) @@ -69636,28 +68653,26 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mMmaK */ 32 , /* mMmaKind */ trtllm::gen::MmaKind(5) , /* mMmaM */ 256 -, /* mMmaN */ 64 -, /* mMmaTileK */ 128 +, /* mMmaN */ 16 +, /* mMmaTileK */ 256 , /* mMockAllReduce */ 0 , /* mN */ 256 , /* mNumEpilogueWarps */ 4 , /* mNumRegsCastAWarps */ 0 , /* mNumRegsCopySfLdsSttm */ 0 , /* mNumRegsCopySparsityInfo */ 0 -, /* mNumRegsPerThreadEpilogueWarp */ 168 -, /* mNumRegsPerThreadNonEpilogueWarp */ 48 +, /* mNumRegsPerThreadEpilogueWarp */ 0 +, /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 -, /* mNumStagesA */ 7 -, /* mNumStagesB */ 7 +, /* mNumStagesA */ 6 +, /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 , /* mNumStagesMmaWithinWorkTile */ 1 , /* mNumStagesMmaAcrossWorkTile */ 2 -, /* mNumStagesSmemSfA */ 7 -, /* mNumStagesSmemSfB */ 7 -, /* mNumStagesTmemA */ 0 -, /* mNumStagesTmemSfA */ 7 -, /* mNumStagesTmemSfB */ 7 +, /* mNumStagesSmemSfA */ 6 +, /* mNumStagesSmemSfB */ 6 +, /* mNumStagesTmemSfA */ 6 +, /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 , /* mOutputDebugTensors */ 0 , /* mPatchF2fp */ 0 @@ -69670,12 +68685,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) -, /* mTileK */ 128 +, /* mTileK */ 256 , /* mTileM */ 128 -, /* mTileN */ 64 +, /* mTileN */ 16 , /* mTileScheduler */ gemm::TileScheduler(1) , /* mTransposeMmaOutput */ 1 , /* mUseCustomizedMma3xNvFp4 */ 0 @@ -69693,7 +68707,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mUseUnrollLoop2xForMma */ 1 , /* mValidM */ 256 , /* mValidN */ 256 -, /* mValidK */ 256 +, /* mValidK */ 512 , /* mWorldSize */ 1 , /* mActType */ gemmGatedAct::ActType(0) , /* mClampBeforeAct */ 1 @@ -69707,22 +68721,23 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mIsUniformNumTokensPerBatch */ 0 , /* mNumBatches */ 128 , /* mNumRegsPerThreadLoadA */ 0 -, /* mNumRegsPerThreadLoadB */ 48 +, /* mNumRegsPerThreadLoadB */ 0 , /* mNumRegsPerThreadLoadSfA */ 0 -, /* mNumRegsPerThreadLoadSfB */ 48 +, /* mNumRegsPerThreadLoadSfB */ 0 , /* mNumTokens */ 2 , /* mNumWarpsLoadA */ 0 -, /* mNumWarpsLoadB */ 1 +, /* mNumWarpsLoadB */ 0 , /* mNumWarpsLoadSfA */ 0 -, /* mNumWarpsLoadSfB */ 1 +, /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x256_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW1_lsfbW1_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x256_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW1_lsfbW1_dynB_sm100f_cubin_len, 183960, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x256_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW1_lsfbW1_dynB_sm100f", 512, "42c6bc55abe5ab84d8be44600fdc2b1f92c089b65f3719f6b3a9aaa1e5f85b25", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len, 223000, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f", 512, "b1bd0c055b1673061d8edb54df5d0a664c9e7d39ccfd2382952eaaf6ba665f96", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) -, /* mBiasType */ gemm::BiasType(1) +, /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 , /* mClcFastDrain */ 1 , /* mClusterDimX */ 2 @@ -69735,14 +68750,14 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17827853) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 -, /* mEpilogueTileN */ 64 +, /* mEpilogueTileN */ 16 , /* mFallbackClusterDimX */ 1 , /* mFallbackClusterDimY */ 1 , /* mFallbackClusterDimZ */ 1 @@ -69756,7 +68771,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mGridWaitForPrimaryB */ 1 , /* mHoistLoadTaskInit */ 1 , /* mHoistMmaTaskTryWaits */ 0 -, /* mK */ 256 +, /* mK */ 512 , /* mKernelTraits */ {} , /* mLayoutA */ gemm::MatrixLayout(0) , /* mLayoutB */ gemm::MatrixLayout(0) @@ -69764,7 +68779,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mMmaK */ 32 , /* mMmaKind */ trtllm::gen::MmaKind(5) , /* mMmaM */ 256 -, /* mMmaN */ 64 +, /* mMmaN */ 16 , /* mMmaTileK */ 256 , /* mMockAllReduce */ 0 , /* mN */ 256 @@ -69772,20 +68787,18 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsCastAWarps */ 0 , /* mNumRegsCopySfLdsSttm */ 0 , /* mNumRegsCopySparsityInfo */ 0 -, /* mNumRegsPerThreadEpilogueWarp */ 168 -, /* mNumRegsPerThreadNonEpilogueWarp */ 48 +, /* mNumRegsPerThreadEpilogueWarp */ 0 +, /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 -, /* mNumStagesA */ 4 -, /* mNumStagesB */ 4 -, /* mNumStagesMma */ 2 +, /* mNumStagesA */ 6 +, /* mNumStagesB */ 6 +, /* mNumStagesMma */ 1 , /* mNumStagesMmaWithinWorkTile */ 1 -, /* mNumStagesMmaAcrossWorkTile */ 2 -, /* mNumStagesSmemSfA */ 4 -, /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 -, /* mNumStagesTmemSfA */ 4 -, /* mNumStagesTmemSfB */ 4 +, /* mNumStagesMmaAcrossWorkTile */ 1 +, /* mNumStagesSmemSfA */ 6 +, /* mNumStagesSmemSfB */ 6 +, /* mNumStagesTmemSfA */ 6 +, /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 , /* mOutputDebugTensors */ 0 , /* mPatchF2fp */ 0 @@ -69798,13 +68811,12 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 , /* mTileM */ 128 -, /* mTileN */ 64 -, /* mTileScheduler */ gemm::TileScheduler(1) +, /* mTileN */ 16 +, /* mTileScheduler */ gemm::TileScheduler(0) , /* mTransposeMmaOutput */ 1 , /* mUseCustomizedMma3xNvFp4 */ 0 , /* mUseCustomMmaSchedule */ 1 @@ -69818,12 +68830,12 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mUseTmaStore */ 1 , /* mUseTwoTmaLoadWarps */ 1 , /* mUseTwoMmaWarps */ 0 -, /* mUseUnrollLoop2xForMma */ 0 +, /* mUseUnrollLoop2xForMma */ 1 , /* mValidM */ 256 , /* mValidN */ 256 -, /* mValidK */ 256 +, /* mValidK */ 512 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(0) +, /* mActType */ gemmGatedAct::ActType(2) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -69835,20 +68847,21 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mIsUniformNumTokensPerBatch */ 0 , /* mNumBatches */ 128 , /* mNumRegsPerThreadLoadA */ 0 -, /* mNumRegsPerThreadLoadB */ 48 +, /* mNumRegsPerThreadLoadB */ 0 , /* mNumRegsPerThreadLoadSfA */ 0 -, /* mNumRegsPerThreadLoadSfB */ 48 +, /* mNumRegsPerThreadLoadSfB */ 0 , /* mNumTokens */ 2 , /* mNumWarpsLoadA */ 0 -, /* mNumWarpsLoadB */ 1 +, /* mNumWarpsLoadB */ 0 , /* mNumWarpsLoadSfA */ 0 -, /* mNumWarpsLoadSfB */ 1 +, /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x256u2_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW1_lsfbW1_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x256u2_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW1_lsfbW1_dynB_sm100f_cubin_len, 183960, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x256u2_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW1_lsfbW1_dynB_sm100f", 512, "1160bf8ba93ffaf20f398f9cb940a7cd3cbb54d7c3a54f0b1002c3e3dfba2455", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 223000, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "3b6582cdb0bca52a8541b65362a697223d603e2ef2b5ed575d52375c151d3d8e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -69863,14 +68876,14 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17827853) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 -, /* mEpilogueTileN */ 64 +, /* mEpilogueTileN */ 16 , /* mFallbackClusterDimX */ 1 , /* mFallbackClusterDimY */ 1 , /* mFallbackClusterDimZ */ 1 @@ -69892,7 +68905,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mMmaK */ 32 , /* mMmaKind */ trtllm::gen::MmaKind(5) , /* mMmaM */ 256 -, /* mMmaN */ 64 +, /* mMmaN */ 16 , /* mMmaTileK */ 256 , /* mMockAllReduce */ 0 , /* mN */ 256 @@ -69900,20 +68913,18 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsCastAWarps */ 0 , /* mNumRegsCopySfLdsSttm */ 0 , /* mNumRegsCopySparsityInfo */ 0 -, /* mNumRegsPerThreadEpilogueWarp */ 168 -, /* mNumRegsPerThreadNonEpilogueWarp */ 48 +, /* mNumRegsPerThreadEpilogueWarp */ 0 +, /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 -, /* mNumStagesA */ 4 -, /* mNumStagesB */ 4 -, /* mNumStagesMma */ 2 +, /* mNumStagesA */ 6 +, /* mNumStagesB */ 6 +, /* mNumStagesMma */ 1 , /* mNumStagesMmaWithinWorkTile */ 1 -, /* mNumStagesMmaAcrossWorkTile */ 2 -, /* mNumStagesSmemSfA */ 4 -, /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 -, /* mNumStagesTmemSfA */ 4 -, /* mNumStagesTmemSfB */ 4 +, /* mNumStagesMmaAcrossWorkTile */ 1 +, /* mNumStagesSmemSfA */ 6 +, /* mNumStagesSmemSfB */ 6 +, /* mNumStagesTmemSfA */ 6 +, /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 , /* mOutputDebugTensors */ 0 , /* mPatchF2fp */ 0 @@ -69926,13 +68937,3540 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 , /* mTileM */ 128 -, /* mTileN */ 64 -, /* mTileScheduler */ gemm::TileScheduler(1) +, /* mTileN */ 16 +, /* mTileScheduler */ gemm::TileScheduler(0) +, /* mTransposeMmaOutput */ 1 +, /* mUseCustomizedMma3xNvFp4 */ 0 +, /* mUseCustomMmaSchedule */ 1 +, /* mUseDeepSeekFp8 */ 0 +, /* mUseFlexibleClusterDims */ 0 +, /* mUseHoistTryWaitForCustomMmaSchedule */ 0 +, /* mUseMaxTmemOverlap */ 0 +, /* mUsePerTokenSfA */ 0 +, /* mUsePerTokenSfB */ 0 +, /* mUseShuffledMatrix */ 1 +, /* mUseTmaStore */ 1 +, /* mUseTwoTmaLoadWarps */ 1 +, /* mUseTwoMmaWarps */ 0 +, /* mUseUnrollLoop2xForMma */ 1 +, /* mValidM */ 256 +, /* mValidN */ 256 +, /* mValidK */ 512 +, /* mWorldSize */ 1 +, /* mActType */ gemmGatedAct::ActType(0) +, /* mClampBeforeAct */ 1 +, /* mBatchedM */ {} +, /* mBatchedN */ {} +, /* mBatchMode */ batchedGemm::BatchedGemmOptions::BatchMode(1) +, /* mBatchStrideInTokens */ -1 +, /* mFusedAct */ 1 +, /* mGridWaitForPrimaryRouting */ 1 +, /* mIsStaticBatch */ 0 +, /* mIsUniformNumTokensPerBatch */ 0 +, /* mNumBatches */ 128 +, /* mNumRegsPerThreadLoadA */ 0 +, /* mNumRegsPerThreadLoadB */ 0 +, /* mNumRegsPerThreadLoadSfA */ 0 +, /* mNumRegsPerThreadLoadSfB */ 0 +, /* mNumTokens */ 2 +, /* mNumWarpsLoadA */ 0 +, /* mNumWarpsLoadB */ 0 +, /* mNumWarpsLoadSfA */ 0 +, /* mNumWarpsLoadSfB */ 0 +, /* mRouteImpl */ batchedGemm::RouteImpl(2) +, /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} +, /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 + }, gemm::SmVersion::Sm100f}, +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x256x128_s6x6x6x6x1x6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x256x128_s6x6x6x6x1x6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f_cubin_len, 222872, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x256x128_s6x6x6x6x1x6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f", 640, "453174759486d5cf69905d999afdc218ca339cb24ad2a7f90146751588222c3b", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +, /* mBiasDtype */ trtllm::gen::Dtype(1056776) +, /* mBiasType */ gemm::BiasType(1) +, /* mBlockK */ -1 +, /* mClcFastDrain */ 1 +, /* mClusterDimX */ 2 +, /* mClusterDimY */ 1 +, /* mClusterDimZ */ 1 +, /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) +, /* mDtypeAcc */ trtllm::gen::Dtype(1056776) +, /* mDtypeA */ trtllm::gen::Dtype(17826828) +, /* mDtypeB */ trtllm::gen::Dtype(17827853) +, /* mDtypeC */ trtllm::gen::Dtype(17827853) +, /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) +, /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) +, /* mEltwiseActType */ gemm::EltwiseActType(0) +, /* mEnablesEarlyExit */ 1 +, /* mEnablesDelayedEarlyExit */ 0 +, /* mEpilogueLdtmDps */ 16 +, /* mEpilogueLdtmBits */ 256 +, /* mEpilogueTileM */ 128 +, /* mEpilogueTileN */ 64 +, /* mFallbackClusterDimX */ 1 +, /* mFallbackClusterDimY */ 1 +, /* mFallbackClusterDimZ */ 1 +, /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) +, /* mFuseLoadSfTask */ 0 +, /* mFuseUtccpWithUtcmma */ 1 +, /* mGridTriggerSecondaryA */ 0 +, /* mGridTriggerSecondaryB */ 1 +, /* mGridWaitForPrimaryEarlyExit */ 1 +, /* mGridWaitForPrimaryA */ 0 +, /* mGridWaitForPrimaryB */ 1 +, /* mHoistLoadTaskInit */ 1 +, /* mHoistMmaTaskTryWaits */ 0 +, /* mK */ 128 +, /* mKernelTraits */ {} +, /* mLayoutA */ gemm::MatrixLayout(0) +, /* mLayoutB */ gemm::MatrixLayout(0) +, /* mM */ 256 +, /* mMmaK */ 32 +, /* mMmaKind */ trtllm::gen::MmaKind(5) +, /* mMmaM */ 256 +, /* mMmaN */ 256 +, /* mMmaTileK */ 128 +, /* mMockAllReduce */ 0 +, /* mN */ 256 +, /* mNumEpilogueWarps */ 4 +, /* mNumRegsCastAWarps */ 0 +, /* mNumRegsCopySfLdsSttm */ 0 +, /* mNumRegsCopySparsityInfo */ 0 +, /* mNumRegsPerThreadEpilogueWarp */ 144 +, /* mNumRegsPerThreadNonEpilogueWarp */ 88 +, /* mNumSlicesForSplitK */ 1 +, /* mNumStagesA */ 6 +, /* mNumStagesB */ 6 +, /* mNumStagesMma */ 1 +, /* mNumStagesMmaWithinWorkTile */ 1 +, /* mNumStagesMmaAcrossWorkTile */ 1 +, /* mNumStagesSmemSfA */ 6 +, /* mNumStagesSmemSfB */ 6 +, /* mNumStagesTmemSfA */ 1 +, /* mNumStagesTmemSfB */ 6 +, /* mNumStagesWorkId */ 3 +, /* mOutputDebugTensors */ 0 +, /* mPatchF2fp */ 0 +, /* mPerTokenSfDtype */ trtllm::gen::Dtype(1052672) +, /* mSchedHostTask */ gemm::SchedHostTask(0) +, /* mSfBlockSizeA */ 32 +, /* mSfBlockSizeB */ 32 +, /* mSfBlockSizeC */ 32 +, /* mSfLayoutA */ trtllm::gen::SfLayout(3) +, /* mSfLayoutB */ trtllm::gen::SfLayout(0) +, /* mSfLayoutC */ trtllm::gen::SfLayout(3) +, /* mSfReshapeFactor */ 1 +, /* mSparsityA */ trtllm::gen::Sparsity(0) +, /* mSplitK */ gemm::SplitK(0) +, /* mTileK */ 128 +, /* mTileM */ 128 +, /* mTileN */ 256 +, /* mTileScheduler */ gemm::TileScheduler(1) +, /* mTransposeMmaOutput */ 1 +, /* mUseCustomizedMma3xNvFp4 */ 0 +, /* mUseCustomMmaSchedule */ 1 +, /* mUseDeepSeekFp8 */ 0 +, /* mUseFlexibleClusterDims */ 0 +, /* mUseHoistTryWaitForCustomMmaSchedule */ 0 +, /* mUseMaxTmemOverlap */ 1 +, /* mUsePerTokenSfA */ 0 +, /* mUsePerTokenSfB */ 0 +, /* mUseShuffledMatrix */ 1 +, /* mUseTmaStore */ 1 +, /* mUseTwoTmaLoadWarps */ 1 +, /* mUseTwoMmaWarps */ 0 +, /* mUseUnrollLoop2xForMma */ 0 +, /* mValidM */ 256 +, /* mValidN */ 256 +, /* mValidK */ 128 +, /* mWorldSize */ 1 +, /* mActType */ gemmGatedAct::ActType(0) +, /* mClampBeforeAct */ 1 +, /* mBatchedM */ {} +, /* mBatchedN */ {} +, /* mBatchMode */ batchedGemm::BatchedGemmOptions::BatchMode(1) +, /* mBatchStrideInTokens */ -1 +, /* mFusedAct */ 1 +, /* mGridWaitForPrimaryRouting */ 1 +, /* mIsStaticBatch */ 0 +, /* mIsUniformNumTokensPerBatch */ 0 +, /* mNumBatches */ 128 +, /* mNumRegsPerThreadLoadA */ 0 +, /* mNumRegsPerThreadLoadB */ 56 +, /* mNumRegsPerThreadLoadSfA */ 0 +, /* mNumRegsPerThreadLoadSfB */ 56 +, /* mNumTokens */ 2 +, /* mNumWarpsLoadA */ 0 +, /* mNumWarpsLoadB */ 8 +, /* mNumWarpsLoadSfA */ 0 +, /* mNumWarpsLoadSfB */ 4 +, /* mRouteImpl */ batchedGemm::RouteImpl(2) +, /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} +, /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 + }, gemm::SmVersion::Sm100f}, +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x256x128_s6x6x6x6x1x6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_fCp_tmOv_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x256x128_s6x6x6x6x1x6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_fCp_tmOv_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f_cubin_len, 221848, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x256x128_s6x6x6x6x1x6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_fCp_tmOv_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f", 640, "eb804c975c9070017f70d777a78da26c1ddb7f24f0c40c349a0867256d06e65f", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +, /* mBiasDtype */ trtllm::gen::Dtype(1056776) +, /* mBiasType */ gemm::BiasType(0) +, /* mBlockK */ -1 +, /* mClcFastDrain */ 1 +, /* mClusterDimX */ 2 +, /* mClusterDimY */ 1 +, /* mClusterDimZ */ 1 +, /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) +, /* mDtypeAcc */ trtllm::gen::Dtype(1056776) +, /* mDtypeA */ trtllm::gen::Dtype(17826828) +, /* mDtypeB */ trtllm::gen::Dtype(17827853) +, /* mDtypeC */ trtllm::gen::Dtype(17827853) +, /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) +, /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) +, /* mEltwiseActType */ gemm::EltwiseActType(0) +, /* mEnablesEarlyExit */ 1 +, /* mEnablesDelayedEarlyExit */ 0 +, /* mEpilogueLdtmDps */ 16 +, /* mEpilogueLdtmBits */ 256 +, /* mEpilogueTileM */ 128 +, /* mEpilogueTileN */ 64 +, /* mFallbackClusterDimX */ 1 +, /* mFallbackClusterDimY */ 1 +, /* mFallbackClusterDimZ */ 1 +, /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) +, /* mFuseLoadSfTask */ 0 +, /* mFuseUtccpWithUtcmma */ 1 +, /* mGridTriggerSecondaryA */ 0 +, /* mGridTriggerSecondaryB */ 1 +, /* mGridWaitForPrimaryEarlyExit */ 1 +, /* mGridWaitForPrimaryA */ 0 +, /* mGridWaitForPrimaryB */ 1 +, /* mHoistLoadTaskInit */ 1 +, /* mHoistMmaTaskTryWaits */ 0 +, /* mK */ 128 +, /* mKernelTraits */ {} +, /* mLayoutA */ gemm::MatrixLayout(0) +, /* mLayoutB */ gemm::MatrixLayout(0) +, /* mM */ 256 +, /* mMmaK */ 32 +, /* mMmaKind */ trtllm::gen::MmaKind(5) +, /* mMmaM */ 256 +, /* mMmaN */ 256 +, /* mMmaTileK */ 128 +, /* mMockAllReduce */ 0 +, /* mN */ 256 +, /* mNumEpilogueWarps */ 4 +, /* mNumRegsCastAWarps */ 0 +, /* mNumRegsCopySfLdsSttm */ 0 +, /* mNumRegsCopySparsityInfo */ 0 +, /* mNumRegsPerThreadEpilogueWarp */ 144 +, /* mNumRegsPerThreadNonEpilogueWarp */ 88 +, /* mNumSlicesForSplitK */ 1 +, /* mNumStagesA */ 6 +, /* mNumStagesB */ 6 +, /* mNumStagesMma */ 1 +, /* mNumStagesMmaWithinWorkTile */ 1 +, /* mNumStagesMmaAcrossWorkTile */ 1 +, /* mNumStagesSmemSfA */ 6 +, /* mNumStagesSmemSfB */ 6 +, /* mNumStagesTmemSfA */ 1 +, /* mNumStagesTmemSfB */ 6 +, /* mNumStagesWorkId */ 3 +, /* mOutputDebugTensors */ 0 +, /* mPatchF2fp */ 0 +, /* mPerTokenSfDtype */ trtllm::gen::Dtype(1052672) +, /* mSchedHostTask */ gemm::SchedHostTask(0) +, /* mSfBlockSizeA */ 32 +, /* mSfBlockSizeB */ 32 +, /* mSfBlockSizeC */ 32 +, /* mSfLayoutA */ trtllm::gen::SfLayout(3) +, /* mSfLayoutB */ trtllm::gen::SfLayout(0) +, /* mSfLayoutC */ trtllm::gen::SfLayout(3) +, /* mSfReshapeFactor */ 1 +, /* mSparsityA */ trtllm::gen::Sparsity(0) +, /* mSplitK */ gemm::SplitK(0) +, /* mTileK */ 128 +, /* mTileM */ 128 +, /* mTileN */ 256 +, /* mTileScheduler */ gemm::TileScheduler(1) +, /* mTransposeMmaOutput */ 1 +, /* mUseCustomizedMma3xNvFp4 */ 0 +, /* mUseCustomMmaSchedule */ 1 +, /* mUseDeepSeekFp8 */ 0 +, /* mUseFlexibleClusterDims */ 0 +, /* mUseHoistTryWaitForCustomMmaSchedule */ 0 +, /* mUseMaxTmemOverlap */ 1 +, /* mUsePerTokenSfA */ 0 +, /* mUsePerTokenSfB */ 0 +, /* mUseShuffledMatrix */ 1 +, /* mUseTmaStore */ 1 +, /* mUseTwoTmaLoadWarps */ 1 +, /* mUseTwoMmaWarps */ 0 +, /* mUseUnrollLoop2xForMma */ 0 +, /* mValidM */ 256 +, /* mValidN */ 256 +, /* mValidK */ 128 +, /* mWorldSize */ 1 +, /* mActType */ gemmGatedAct::ActType(2) +, /* mClampBeforeAct */ 1 +, /* mBatchedM */ {} +, /* mBatchedN */ {} +, /* mBatchMode */ batchedGemm::BatchedGemmOptions::BatchMode(1) +, /* mBatchStrideInTokens */ -1 +, /* mFusedAct */ 1 +, /* mGridWaitForPrimaryRouting */ 1 +, /* mIsStaticBatch */ 0 +, /* mIsUniformNumTokensPerBatch */ 0 +, /* mNumBatches */ 128 +, /* mNumRegsPerThreadLoadA */ 0 +, /* mNumRegsPerThreadLoadB */ 56 +, /* mNumRegsPerThreadLoadSfA */ 0 +, /* mNumRegsPerThreadLoadSfB */ 56 +, /* mNumTokens */ 2 +, /* mNumWarpsLoadA */ 0 +, /* mNumWarpsLoadB */ 8 +, /* mNumWarpsLoadSfA */ 0 +, /* mNumWarpsLoadSfB */ 4 +, /* mRouteImpl */ batchedGemm::RouteImpl(2) +, /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} +, /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 + }, gemm::SmVersion::Sm100f}, +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x256x256_s3x3x3x3x1x3_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x256x256_s3x3x3x3x1x3_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f_cubin_len, 222872, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x256x256_s3x3x3x3x1x3_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f", 640, "e387feae1e1ca9b2c93bc72d54cd035b4f9796b6380c458a3d615a82141633e8", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +, /* mBiasDtype */ trtllm::gen::Dtype(1056776) +, /* mBiasType */ gemm::BiasType(1) +, /* mBlockK */ -1 +, /* mClcFastDrain */ 1 +, /* mClusterDimX */ 2 +, /* mClusterDimY */ 1 +, /* mClusterDimZ */ 1 +, /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) +, /* mDtypeAcc */ trtllm::gen::Dtype(1056776) +, /* mDtypeA */ trtllm::gen::Dtype(17826828) +, /* mDtypeB */ trtllm::gen::Dtype(17827853) +, /* mDtypeC */ trtllm::gen::Dtype(17827853) +, /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) +, /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) +, /* mEltwiseActType */ gemm::EltwiseActType(0) +, /* mEnablesEarlyExit */ 1 +, /* mEnablesDelayedEarlyExit */ 0 +, /* mEpilogueLdtmDps */ 16 +, /* mEpilogueLdtmBits */ 256 +, /* mEpilogueTileM */ 128 +, /* mEpilogueTileN */ 64 +, /* mFallbackClusterDimX */ 1 +, /* mFallbackClusterDimY */ 1 +, /* mFallbackClusterDimZ */ 1 +, /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) +, /* mFuseLoadSfTask */ 0 +, /* mFuseUtccpWithUtcmma */ 1 +, /* mGridTriggerSecondaryA */ 0 +, /* mGridTriggerSecondaryB */ 1 +, /* mGridWaitForPrimaryEarlyExit */ 1 +, /* mGridWaitForPrimaryA */ 0 +, /* mGridWaitForPrimaryB */ 1 +, /* mHoistLoadTaskInit */ 1 +, /* mHoistMmaTaskTryWaits */ 0 +, /* mK */ 256 +, /* mKernelTraits */ {} +, /* mLayoutA */ gemm::MatrixLayout(0) +, /* mLayoutB */ gemm::MatrixLayout(0) +, /* mM */ 256 +, /* mMmaK */ 32 +, /* mMmaKind */ trtllm::gen::MmaKind(5) +, /* mMmaM */ 256 +, /* mMmaN */ 256 +, /* mMmaTileK */ 256 +, /* mMockAllReduce */ 0 +, /* mN */ 256 +, /* mNumEpilogueWarps */ 4 +, /* mNumRegsCastAWarps */ 0 +, /* mNumRegsCopySfLdsSttm */ 0 +, /* mNumRegsCopySparsityInfo */ 0 +, /* mNumRegsPerThreadEpilogueWarp */ 144 +, /* mNumRegsPerThreadNonEpilogueWarp */ 88 +, /* mNumSlicesForSplitK */ 1 +, /* mNumStagesA */ 3 +, /* mNumStagesB */ 3 +, /* mNumStagesMma */ 1 +, /* mNumStagesMmaWithinWorkTile */ 1 +, /* mNumStagesMmaAcrossWorkTile */ 1 +, /* mNumStagesSmemSfA */ 3 +, /* mNumStagesSmemSfB */ 3 +, /* mNumStagesTmemSfA */ 1 +, /* mNumStagesTmemSfB */ 3 +, /* mNumStagesWorkId */ 3 +, /* mOutputDebugTensors */ 0 +, /* mPatchF2fp */ 0 +, /* mPerTokenSfDtype */ trtllm::gen::Dtype(1052672) +, /* mSchedHostTask */ gemm::SchedHostTask(0) +, /* mSfBlockSizeA */ 32 +, /* mSfBlockSizeB */ 32 +, /* mSfBlockSizeC */ 32 +, /* mSfLayoutA */ trtllm::gen::SfLayout(3) +, /* mSfLayoutB */ trtllm::gen::SfLayout(0) +, /* mSfLayoutC */ trtllm::gen::SfLayout(3) +, /* mSfReshapeFactor */ 1 +, /* mSparsityA */ trtllm::gen::Sparsity(0) +, /* mSplitK */ gemm::SplitK(0) +, /* mTileK */ 256 +, /* mTileM */ 128 +, /* mTileN */ 256 +, /* mTileScheduler */ gemm::TileScheduler(1) +, /* mTransposeMmaOutput */ 1 +, /* mUseCustomizedMma3xNvFp4 */ 0 +, /* mUseCustomMmaSchedule */ 1 +, /* mUseDeepSeekFp8 */ 0 +, /* mUseFlexibleClusterDims */ 0 +, /* mUseHoistTryWaitForCustomMmaSchedule */ 0 +, /* mUseMaxTmemOverlap */ 1 +, /* mUsePerTokenSfA */ 0 +, /* mUsePerTokenSfB */ 0 +, /* mUseShuffledMatrix */ 1 +, /* mUseTmaStore */ 1 +, /* mUseTwoTmaLoadWarps */ 1 +, /* mUseTwoMmaWarps */ 0 +, /* mUseUnrollLoop2xForMma */ 0 +, /* mValidM */ 256 +, /* mValidN */ 256 +, /* mValidK */ 256 +, /* mWorldSize */ 1 +, /* mActType */ gemmGatedAct::ActType(0) +, /* mClampBeforeAct */ 1 +, /* mBatchedM */ {} +, /* mBatchedN */ {} +, /* mBatchMode */ batchedGemm::BatchedGemmOptions::BatchMode(1) +, /* mBatchStrideInTokens */ -1 +, /* mFusedAct */ 1 +, /* mGridWaitForPrimaryRouting */ 1 +, /* mIsStaticBatch */ 0 +, /* mIsUniformNumTokensPerBatch */ 0 +, /* mNumBatches */ 128 +, /* mNumRegsPerThreadLoadA */ 0 +, /* mNumRegsPerThreadLoadB */ 56 +, /* mNumRegsPerThreadLoadSfA */ 0 +, /* mNumRegsPerThreadLoadSfB */ 56 +, /* mNumTokens */ 2 +, /* mNumWarpsLoadA */ 0 +, /* mNumWarpsLoadB */ 8 +, /* mNumWarpsLoadSfA */ 0 +, /* mNumWarpsLoadSfB */ 4 +, /* mRouteImpl */ batchedGemm::RouteImpl(2) +, /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} +, /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 + }, gemm::SmVersion::Sm100f}, +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x256x256_s3x3x3x3x1x3_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_fCp_tmOv_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x256x256_s3x3x3x3x1x3_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_fCp_tmOv_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f_cubin_len, 221848, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x256x256_s3x3x3x3x1x3_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_fCp_tmOv_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f", 640, "43daaae59700c6261cfff6fae7cbe91056f4e61452170608db2f80b625b6090d", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +, /* mBiasDtype */ trtllm::gen::Dtype(1056776) +, /* mBiasType */ gemm::BiasType(0) +, /* mBlockK */ -1 +, /* mClcFastDrain */ 1 +, /* mClusterDimX */ 2 +, /* mClusterDimY */ 1 +, /* mClusterDimZ */ 1 +, /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) +, /* mDtypeAcc */ trtllm::gen::Dtype(1056776) +, /* mDtypeA */ trtllm::gen::Dtype(17826828) +, /* mDtypeB */ trtllm::gen::Dtype(17827853) +, /* mDtypeC */ trtllm::gen::Dtype(17827853) +, /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) +, /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) +, /* mEltwiseActType */ gemm::EltwiseActType(0) +, /* mEnablesEarlyExit */ 1 +, /* mEnablesDelayedEarlyExit */ 0 +, /* mEpilogueLdtmDps */ 16 +, /* mEpilogueLdtmBits */ 256 +, /* mEpilogueTileM */ 128 +, /* mEpilogueTileN */ 64 +, /* mFallbackClusterDimX */ 1 +, /* mFallbackClusterDimY */ 1 +, /* mFallbackClusterDimZ */ 1 +, /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) +, /* mFuseLoadSfTask */ 0 +, /* mFuseUtccpWithUtcmma */ 1 +, /* mGridTriggerSecondaryA */ 0 +, /* mGridTriggerSecondaryB */ 1 +, /* mGridWaitForPrimaryEarlyExit */ 1 +, /* mGridWaitForPrimaryA */ 0 +, /* mGridWaitForPrimaryB */ 1 +, /* mHoistLoadTaskInit */ 1 +, /* mHoistMmaTaskTryWaits */ 0 +, /* mK */ 256 +, /* mKernelTraits */ {} +, /* mLayoutA */ gemm::MatrixLayout(0) +, /* mLayoutB */ gemm::MatrixLayout(0) +, /* mM */ 256 +, /* mMmaK */ 32 +, /* mMmaKind */ trtllm::gen::MmaKind(5) +, /* mMmaM */ 256 +, /* mMmaN */ 256 +, /* mMmaTileK */ 256 +, /* mMockAllReduce */ 0 +, /* mN */ 256 +, /* mNumEpilogueWarps */ 4 +, /* mNumRegsCastAWarps */ 0 +, /* mNumRegsCopySfLdsSttm */ 0 +, /* mNumRegsCopySparsityInfo */ 0 +, /* mNumRegsPerThreadEpilogueWarp */ 144 +, /* mNumRegsPerThreadNonEpilogueWarp */ 88 +, /* mNumSlicesForSplitK */ 1 +, /* mNumStagesA */ 3 +, /* mNumStagesB */ 3 +, /* mNumStagesMma */ 1 +, /* mNumStagesMmaWithinWorkTile */ 1 +, /* mNumStagesMmaAcrossWorkTile */ 1 +, /* mNumStagesSmemSfA */ 3 +, /* mNumStagesSmemSfB */ 3 +, /* mNumStagesTmemSfA */ 1 +, /* mNumStagesTmemSfB */ 3 +, /* mNumStagesWorkId */ 3 +, /* mOutputDebugTensors */ 0 +, /* mPatchF2fp */ 0 +, /* mPerTokenSfDtype */ trtllm::gen::Dtype(1052672) +, /* mSchedHostTask */ gemm::SchedHostTask(0) +, /* mSfBlockSizeA */ 32 +, /* mSfBlockSizeB */ 32 +, /* mSfBlockSizeC */ 32 +, /* mSfLayoutA */ trtllm::gen::SfLayout(3) +, /* mSfLayoutB */ trtllm::gen::SfLayout(0) +, /* mSfLayoutC */ trtllm::gen::SfLayout(3) +, /* mSfReshapeFactor */ 1 +, /* mSparsityA */ trtllm::gen::Sparsity(0) +, /* mSplitK */ gemm::SplitK(0) +, /* mTileK */ 256 +, /* mTileM */ 128 +, /* mTileN */ 256 +, /* mTileScheduler */ gemm::TileScheduler(1) +, /* mTransposeMmaOutput */ 1 +, /* mUseCustomizedMma3xNvFp4 */ 0 +, /* mUseCustomMmaSchedule */ 1 +, /* mUseDeepSeekFp8 */ 0 +, /* mUseFlexibleClusterDims */ 0 +, /* mUseHoistTryWaitForCustomMmaSchedule */ 0 +, /* mUseMaxTmemOverlap */ 1 +, /* mUsePerTokenSfA */ 0 +, /* mUsePerTokenSfB */ 0 +, /* mUseShuffledMatrix */ 1 +, /* mUseTmaStore */ 1 +, /* mUseTwoTmaLoadWarps */ 1 +, /* mUseTwoMmaWarps */ 0 +, /* mUseUnrollLoop2xForMma */ 0 +, /* mValidM */ 256 +, /* mValidN */ 256 +, /* mValidK */ 256 +, /* mWorldSize */ 1 +, /* mActType */ gemmGatedAct::ActType(2) +, /* mClampBeforeAct */ 1 +, /* mBatchedM */ {} +, /* mBatchedN */ {} +, /* mBatchMode */ batchedGemm::BatchedGemmOptions::BatchMode(1) +, /* mBatchStrideInTokens */ -1 +, /* mFusedAct */ 1 +, /* mGridWaitForPrimaryRouting */ 1 +, /* mIsStaticBatch */ 0 +, /* mIsUniformNumTokensPerBatch */ 0 +, /* mNumBatches */ 128 +, /* mNumRegsPerThreadLoadA */ 0 +, /* mNumRegsPerThreadLoadB */ 56 +, /* mNumRegsPerThreadLoadSfA */ 0 +, /* mNumRegsPerThreadLoadSfB */ 56 +, /* mNumTokens */ 2 +, /* mNumWarpsLoadA */ 0 +, /* mNumWarpsLoadB */ 8 +, /* mNumWarpsLoadSfA */ 0 +, /* mNumWarpsLoadSfB */ 4 +, /* mRouteImpl */ batchedGemm::RouteImpl(2) +, /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} +, /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 + }, gemm::SmVersion::Sm100f}, +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len, 221072, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f", 640, "509722e9ec11421b2513f7cd2d3faff5bc473434b1cbb0da19c1f7c019950237", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +, /* mBiasDtype */ trtllm::gen::Dtype(1056776) +, /* mBiasType */ gemm::BiasType(0) +, /* mBlockK */ -1 +, /* mClcFastDrain */ 1 +, /* mClusterDimX */ 1 +, /* mClusterDimY */ 1 +, /* mClusterDimZ */ 1 +, /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) +, /* mDtypeAcc */ trtllm::gen::Dtype(1056776) +, /* mDtypeA */ trtllm::gen::Dtype(17826828) +, /* mDtypeB */ trtllm::gen::Dtype(17827853) +, /* mDtypeC */ trtllm::gen::Dtype(17827853) +, /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) +, /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) +, /* mEltwiseActType */ gemm::EltwiseActType(0) +, /* mEnablesEarlyExit */ 1 +, /* mEnablesDelayedEarlyExit */ 0 +, /* mEpilogueLdtmDps */ 16 +, /* mEpilogueLdtmBits */ 256 +, /* mEpilogueTileM */ 128 +, /* mEpilogueTileN */ 32 +, /* mFallbackClusterDimX */ 1 +, /* mFallbackClusterDimY */ 1 +, /* mFallbackClusterDimZ */ 1 +, /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) +, /* mFuseLoadSfTask */ 0 +, /* mFuseUtccpWithUtcmma */ 0 +, /* mGridTriggerSecondaryA */ 0 +, /* mGridTriggerSecondaryB */ 1 +, /* mGridWaitForPrimaryEarlyExit */ 1 +, /* mGridWaitForPrimaryA */ 0 +, /* mGridWaitForPrimaryB */ 1 +, /* mHoistLoadTaskInit */ 1 +, /* mHoistMmaTaskTryWaits */ 0 +, /* mK */ 256 +, /* mKernelTraits */ {} +, /* mLayoutA */ gemm::MatrixLayout(0) +, /* mLayoutB */ gemm::MatrixLayout(0) +, /* mM */ 256 +, /* mMmaK */ 32 +, /* mMmaKind */ trtllm::gen::MmaKind(5) +, /* mMmaM */ 128 +, /* mMmaN */ 32 +, /* mMmaTileK */ 256 +, /* mMockAllReduce */ 0 +, /* mN */ 256 +, /* mNumEpilogueWarps */ 4 +, /* mNumRegsCastAWarps */ 0 +, /* mNumRegsCopySfLdsSttm */ 0 +, /* mNumRegsCopySparsityInfo */ 0 +, /* mNumRegsPerThreadEpilogueWarp */ 0 +, /* mNumRegsPerThreadNonEpilogueWarp */ 0 +, /* mNumSlicesForSplitK */ 1 +, /* mNumStagesA */ 5 +, /* mNumStagesB */ 5 +, /* mNumStagesMma */ 2 +, /* mNumStagesMmaWithinWorkTile */ 1 +, /* mNumStagesMmaAcrossWorkTile */ 2 +, /* mNumStagesSmemSfA */ 5 +, /* mNumStagesSmemSfB */ 5 +, /* mNumStagesTmemSfA */ 5 +, /* mNumStagesTmemSfB */ 5 +, /* mNumStagesWorkId */ 3 +, /* mOutputDebugTensors */ 0 +, /* mPatchF2fp */ 0 +, /* mPerTokenSfDtype */ trtllm::gen::Dtype(1052672) +, /* mSchedHostTask */ gemm::SchedHostTask(0) +, /* mSfBlockSizeA */ 32 +, /* mSfBlockSizeB */ 32 +, /* mSfBlockSizeC */ 32 +, /* mSfLayoutA */ trtllm::gen::SfLayout(3) +, /* mSfLayoutB */ trtllm::gen::SfLayout(0) +, /* mSfLayoutC */ trtllm::gen::SfLayout(1) +, /* mSfReshapeFactor */ 1 +, /* mSparsityA */ trtllm::gen::Sparsity(0) +, /* mSplitK */ gemm::SplitK(0) +, /* mTileK */ 256 +, /* mTileM */ 128 +, /* mTileN */ 32 +, /* mTileScheduler */ gemm::TileScheduler(1) +, /* mTransposeMmaOutput */ 1 +, /* mUseCustomizedMma3xNvFp4 */ 0 +, /* mUseCustomMmaSchedule */ 1 +, /* mUseDeepSeekFp8 */ 0 +, /* mUseFlexibleClusterDims */ 0 +, /* mUseHoistTryWaitForCustomMmaSchedule */ 0 +, /* mUseMaxTmemOverlap */ 0 +, /* mUsePerTokenSfA */ 0 +, /* mUsePerTokenSfB */ 0 +, /* mUseShuffledMatrix */ 1 +, /* mUseTmaStore */ 1 +, /* mUseTwoTmaLoadWarps */ 1 +, /* mUseTwoMmaWarps */ 0 +, /* mUseUnrollLoop2xForMma */ 0 +, /* mValidM */ 256 +, /* mValidN */ 256 +, /* mValidK */ 256 +, /* mWorldSize */ 1 +, /* mActType */ gemmGatedAct::ActType(2) +, /* mClampBeforeAct */ 1 +, /* mBatchedM */ {} +, /* mBatchedN */ {} +, /* mBatchMode */ batchedGemm::BatchedGemmOptions::BatchMode(1) +, /* mBatchStrideInTokens */ -1 +, /* mFusedAct */ 1 +, /* mGridWaitForPrimaryRouting */ 1 +, /* mIsStaticBatch */ 0 +, /* mIsUniformNumTokensPerBatch */ 0 +, /* mNumBatches */ 128 +, /* mNumRegsPerThreadLoadA */ 0 +, /* mNumRegsPerThreadLoadB */ 0 +, /* mNumRegsPerThreadLoadSfA */ 0 +, /* mNumRegsPerThreadLoadSfB */ 0 +, /* mNumTokens */ 2 +, /* mNumWarpsLoadA */ 0 +, /* mNumWarpsLoadB */ 0 +, /* mNumWarpsLoadSfA */ 0 +, /* mNumWarpsLoadSfB */ 0 +, /* mRouteImpl */ batchedGemm::RouteImpl(2) +, /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} +, /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 + }, gemm::SmVersion::Sm100f}, +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 221072, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "535f3fe8da9d47dd1988bd950f780b4fdaca568757b2b3a41cd2347f21e08292", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +, /* mBiasDtype */ trtllm::gen::Dtype(1056776) +, /* mBiasType */ gemm::BiasType(1) +, /* mBlockK */ -1 +, /* mClcFastDrain */ 1 +, /* mClusterDimX */ 1 +, /* mClusterDimY */ 1 +, /* mClusterDimZ */ 1 +, /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) +, /* mDtypeAcc */ trtllm::gen::Dtype(1056776) +, /* mDtypeA */ trtllm::gen::Dtype(17826828) +, /* mDtypeB */ trtllm::gen::Dtype(17827853) +, /* mDtypeC */ trtllm::gen::Dtype(17827853) +, /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) +, /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) +, /* mEltwiseActType */ gemm::EltwiseActType(0) +, /* mEnablesEarlyExit */ 1 +, /* mEnablesDelayedEarlyExit */ 0 +, /* mEpilogueLdtmDps */ 16 +, /* mEpilogueLdtmBits */ 256 +, /* mEpilogueTileM */ 128 +, /* mEpilogueTileN */ 32 +, /* mFallbackClusterDimX */ 1 +, /* mFallbackClusterDimY */ 1 +, /* mFallbackClusterDimZ */ 1 +, /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) +, /* mFuseLoadSfTask */ 0 +, /* mFuseUtccpWithUtcmma */ 0 +, /* mGridTriggerSecondaryA */ 0 +, /* mGridTriggerSecondaryB */ 1 +, /* mGridWaitForPrimaryEarlyExit */ 1 +, /* mGridWaitForPrimaryA */ 0 +, /* mGridWaitForPrimaryB */ 1 +, /* mHoistLoadTaskInit */ 1 +, /* mHoistMmaTaskTryWaits */ 0 +, /* mK */ 256 +, /* mKernelTraits */ {} +, /* mLayoutA */ gemm::MatrixLayout(0) +, /* mLayoutB */ gemm::MatrixLayout(0) +, /* mM */ 256 +, /* mMmaK */ 32 +, /* mMmaKind */ trtllm::gen::MmaKind(5) +, /* mMmaM */ 128 +, /* mMmaN */ 32 +, /* mMmaTileK */ 256 +, /* mMockAllReduce */ 0 +, /* mN */ 256 +, /* mNumEpilogueWarps */ 4 +, /* mNumRegsCastAWarps */ 0 +, /* mNumRegsCopySfLdsSttm */ 0 +, /* mNumRegsCopySparsityInfo */ 0 +, /* mNumRegsPerThreadEpilogueWarp */ 0 +, /* mNumRegsPerThreadNonEpilogueWarp */ 0 +, /* mNumSlicesForSplitK */ 1 +, /* mNumStagesA */ 5 +, /* mNumStagesB */ 5 +, /* mNumStagesMma */ 2 +, /* mNumStagesMmaWithinWorkTile */ 1 +, /* mNumStagesMmaAcrossWorkTile */ 2 +, /* mNumStagesSmemSfA */ 5 +, /* mNumStagesSmemSfB */ 5 +, /* mNumStagesTmemSfA */ 5 +, /* mNumStagesTmemSfB */ 5 +, /* mNumStagesWorkId */ 3 +, /* mOutputDebugTensors */ 0 +, /* mPatchF2fp */ 0 +, /* mPerTokenSfDtype */ trtllm::gen::Dtype(1052672) +, /* mSchedHostTask */ gemm::SchedHostTask(0) +, /* mSfBlockSizeA */ 32 +, /* mSfBlockSizeB */ 32 +, /* mSfBlockSizeC */ 32 +, /* mSfLayoutA */ trtllm::gen::SfLayout(3) +, /* mSfLayoutB */ trtllm::gen::SfLayout(0) +, /* mSfLayoutC */ trtllm::gen::SfLayout(1) +, /* mSfReshapeFactor */ 1 +, /* mSparsityA */ trtllm::gen::Sparsity(0) +, /* mSplitK */ gemm::SplitK(0) +, /* mTileK */ 256 +, /* mTileM */ 128 +, /* mTileN */ 32 +, /* mTileScheduler */ gemm::TileScheduler(1) +, /* mTransposeMmaOutput */ 1 +, /* mUseCustomizedMma3xNvFp4 */ 0 +, /* mUseCustomMmaSchedule */ 1 +, /* mUseDeepSeekFp8 */ 0 +, /* mUseFlexibleClusterDims */ 0 +, /* mUseHoistTryWaitForCustomMmaSchedule */ 0 +, /* mUseMaxTmemOverlap */ 0 +, /* mUsePerTokenSfA */ 0 +, /* mUsePerTokenSfB */ 0 +, /* mUseShuffledMatrix */ 1 +, /* mUseTmaStore */ 1 +, /* mUseTwoTmaLoadWarps */ 1 +, /* mUseTwoMmaWarps */ 0 +, /* mUseUnrollLoop2xForMma */ 0 +, /* mValidM */ 256 +, /* mValidN */ 256 +, /* mValidK */ 256 +, /* mWorldSize */ 1 +, /* mActType */ gemmGatedAct::ActType(0) +, /* mClampBeforeAct */ 1 +, /* mBatchedM */ {} +, /* mBatchedN */ {} +, /* mBatchMode */ batchedGemm::BatchedGemmOptions::BatchMode(1) +, /* mBatchStrideInTokens */ -1 +, /* mFusedAct */ 1 +, /* mGridWaitForPrimaryRouting */ 1 +, /* mIsStaticBatch */ 0 +, /* mIsUniformNumTokensPerBatch */ 0 +, /* mNumBatches */ 128 +, /* mNumRegsPerThreadLoadA */ 0 +, /* mNumRegsPerThreadLoadB */ 0 +, /* mNumRegsPerThreadLoadSfA */ 0 +, /* mNumRegsPerThreadLoadSfB */ 0 +, /* mNumTokens */ 2 +, /* mNumWarpsLoadA */ 0 +, /* mNumWarpsLoadB */ 0 +, /* mNumWarpsLoadSfA */ 0 +, /* mNumWarpsLoadSfB */ 0 +, /* mRouteImpl */ batchedGemm::RouteImpl(2) +, /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} +, /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 + }, gemm::SmVersion::Sm100f}, +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len, 220688, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f", 640, "e1de93fa95104b4050c53238c35ad3be46eac580d026df11dddae77c1165816b", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +, /* mBiasDtype */ trtllm::gen::Dtype(1056776) +, /* mBiasType */ gemm::BiasType(0) +, /* mBlockK */ -1 +, /* mClcFastDrain */ 1 +, /* mClusterDimX */ 1 +, /* mClusterDimY */ 1 +, /* mClusterDimZ */ 1 +, /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) +, /* mDtypeAcc */ trtllm::gen::Dtype(1056776) +, /* mDtypeA */ trtllm::gen::Dtype(17826828) +, /* mDtypeB */ trtllm::gen::Dtype(17827853) +, /* mDtypeC */ trtllm::gen::Dtype(17827853) +, /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) +, /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) +, /* mEltwiseActType */ gemm::EltwiseActType(0) +, /* mEnablesEarlyExit */ 1 +, /* mEnablesDelayedEarlyExit */ 0 +, /* mEpilogueLdtmDps */ 16 +, /* mEpilogueLdtmBits */ 256 +, /* mEpilogueTileM */ 128 +, /* mEpilogueTileN */ 32 +, /* mFallbackClusterDimX */ 1 +, /* mFallbackClusterDimY */ 1 +, /* mFallbackClusterDimZ */ 1 +, /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) +, /* mFuseLoadSfTask */ 0 +, /* mFuseUtccpWithUtcmma */ 0 +, /* mGridTriggerSecondaryA */ 0 +, /* mGridTriggerSecondaryB */ 1 +, /* mGridWaitForPrimaryEarlyExit */ 1 +, /* mGridWaitForPrimaryA */ 0 +, /* mGridWaitForPrimaryB */ 1 +, /* mHoistLoadTaskInit */ 1 +, /* mHoistMmaTaskTryWaits */ 0 +, /* mK */ 256 +, /* mKernelTraits */ {} +, /* mLayoutA */ gemm::MatrixLayout(0) +, /* mLayoutB */ gemm::MatrixLayout(0) +, /* mM */ 256 +, /* mMmaK */ 32 +, /* mMmaKind */ trtllm::gen::MmaKind(5) +, /* mMmaM */ 128 +, /* mMmaN */ 32 +, /* mMmaTileK */ 256 +, /* mMockAllReduce */ 0 +, /* mN */ 256 +, /* mNumEpilogueWarps */ 4 +, /* mNumRegsCastAWarps */ 0 +, /* mNumRegsCopySfLdsSttm */ 0 +, /* mNumRegsCopySparsityInfo */ 0 +, /* mNumRegsPerThreadEpilogueWarp */ 0 +, /* mNumRegsPerThreadNonEpilogueWarp */ 0 +, /* mNumSlicesForSplitK */ 1 +, /* mNumStagesA */ 5 +, /* mNumStagesB */ 5 +, /* mNumStagesMma */ 1 +, /* mNumStagesMmaWithinWorkTile */ 1 +, /* mNumStagesMmaAcrossWorkTile */ 1 +, /* mNumStagesSmemSfA */ 5 +, /* mNumStagesSmemSfB */ 5 +, /* mNumStagesTmemSfA */ 5 +, /* mNumStagesTmemSfB */ 5 +, /* mNumStagesWorkId */ 3 +, /* mOutputDebugTensors */ 0 +, /* mPatchF2fp */ 0 +, /* mPerTokenSfDtype */ trtllm::gen::Dtype(1052672) +, /* mSchedHostTask */ gemm::SchedHostTask(0) +, /* mSfBlockSizeA */ 32 +, /* mSfBlockSizeB */ 32 +, /* mSfBlockSizeC */ 32 +, /* mSfLayoutA */ trtllm::gen::SfLayout(3) +, /* mSfLayoutB */ trtllm::gen::SfLayout(0) +, /* mSfLayoutC */ trtllm::gen::SfLayout(1) +, /* mSfReshapeFactor */ 1 +, /* mSparsityA */ trtllm::gen::Sparsity(0) +, /* mSplitK */ gemm::SplitK(0) +, /* mTileK */ 256 +, /* mTileM */ 128 +, /* mTileN */ 32 +, /* mTileScheduler */ gemm::TileScheduler(0) +, /* mTransposeMmaOutput */ 1 +, /* mUseCustomizedMma3xNvFp4 */ 0 +, /* mUseCustomMmaSchedule */ 1 +, /* mUseDeepSeekFp8 */ 0 +, /* mUseFlexibleClusterDims */ 0 +, /* mUseHoistTryWaitForCustomMmaSchedule */ 0 +, /* mUseMaxTmemOverlap */ 0 +, /* mUsePerTokenSfA */ 0 +, /* mUsePerTokenSfB */ 0 +, /* mUseShuffledMatrix */ 1 +, /* mUseTmaStore */ 1 +, /* mUseTwoTmaLoadWarps */ 1 +, /* mUseTwoMmaWarps */ 0 +, /* mUseUnrollLoop2xForMma */ 0 +, /* mValidM */ 256 +, /* mValidN */ 256 +, /* mValidK */ 256 +, /* mWorldSize */ 1 +, /* mActType */ gemmGatedAct::ActType(2) +, /* mClampBeforeAct */ 1 +, /* mBatchedM */ {} +, /* mBatchedN */ {} +, /* mBatchMode */ batchedGemm::BatchedGemmOptions::BatchMode(1) +, /* mBatchStrideInTokens */ -1 +, /* mFusedAct */ 1 +, /* mGridWaitForPrimaryRouting */ 1 +, /* mIsStaticBatch */ 0 +, /* mIsUniformNumTokensPerBatch */ 0 +, /* mNumBatches */ 128 +, /* mNumRegsPerThreadLoadA */ 0 +, /* mNumRegsPerThreadLoadB */ 0 +, /* mNumRegsPerThreadLoadSfA */ 0 +, /* mNumRegsPerThreadLoadSfB */ 0 +, /* mNumTokens */ 2 +, /* mNumWarpsLoadA */ 0 +, /* mNumWarpsLoadB */ 0 +, /* mNumWarpsLoadSfA */ 0 +, /* mNumWarpsLoadSfB */ 0 +, /* mRouteImpl */ batchedGemm::RouteImpl(2) +, /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} +, /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 + }, gemm::SmVersion::Sm100f}, +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 220688, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "68ed4c77bda741a7fd9e5200e945a0686d69bc27d39f88c09b640fe65433f058", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +, /* mBiasDtype */ trtllm::gen::Dtype(1056776) +, /* mBiasType */ gemm::BiasType(1) +, /* mBlockK */ -1 +, /* mClcFastDrain */ 1 +, /* mClusterDimX */ 1 +, /* mClusterDimY */ 1 +, /* mClusterDimZ */ 1 +, /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) +, /* mDtypeAcc */ trtllm::gen::Dtype(1056776) +, /* mDtypeA */ trtllm::gen::Dtype(17826828) +, /* mDtypeB */ trtllm::gen::Dtype(17827853) +, /* mDtypeC */ trtllm::gen::Dtype(17827853) +, /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) +, /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) +, /* mEltwiseActType */ gemm::EltwiseActType(0) +, /* mEnablesEarlyExit */ 1 +, /* mEnablesDelayedEarlyExit */ 0 +, /* mEpilogueLdtmDps */ 16 +, /* mEpilogueLdtmBits */ 256 +, /* mEpilogueTileM */ 128 +, /* mEpilogueTileN */ 32 +, /* mFallbackClusterDimX */ 1 +, /* mFallbackClusterDimY */ 1 +, /* mFallbackClusterDimZ */ 1 +, /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) +, /* mFuseLoadSfTask */ 0 +, /* mFuseUtccpWithUtcmma */ 0 +, /* mGridTriggerSecondaryA */ 0 +, /* mGridTriggerSecondaryB */ 1 +, /* mGridWaitForPrimaryEarlyExit */ 1 +, /* mGridWaitForPrimaryA */ 0 +, /* mGridWaitForPrimaryB */ 1 +, /* mHoistLoadTaskInit */ 1 +, /* mHoistMmaTaskTryWaits */ 0 +, /* mK */ 256 +, /* mKernelTraits */ {} +, /* mLayoutA */ gemm::MatrixLayout(0) +, /* mLayoutB */ gemm::MatrixLayout(0) +, /* mM */ 256 +, /* mMmaK */ 32 +, /* mMmaKind */ trtllm::gen::MmaKind(5) +, /* mMmaM */ 128 +, /* mMmaN */ 32 +, /* mMmaTileK */ 256 +, /* mMockAllReduce */ 0 +, /* mN */ 256 +, /* mNumEpilogueWarps */ 4 +, /* mNumRegsCastAWarps */ 0 +, /* mNumRegsCopySfLdsSttm */ 0 +, /* mNumRegsCopySparsityInfo */ 0 +, /* mNumRegsPerThreadEpilogueWarp */ 0 +, /* mNumRegsPerThreadNonEpilogueWarp */ 0 +, /* mNumSlicesForSplitK */ 1 +, /* mNumStagesA */ 5 +, /* mNumStagesB */ 5 +, /* mNumStagesMma */ 1 +, /* mNumStagesMmaWithinWorkTile */ 1 +, /* mNumStagesMmaAcrossWorkTile */ 1 +, /* mNumStagesSmemSfA */ 5 +, /* mNumStagesSmemSfB */ 5 +, /* mNumStagesTmemSfA */ 5 +, /* mNumStagesTmemSfB */ 5 +, /* mNumStagesWorkId */ 3 +, /* mOutputDebugTensors */ 0 +, /* mPatchF2fp */ 0 +, /* mPerTokenSfDtype */ trtllm::gen::Dtype(1052672) +, /* mSchedHostTask */ gemm::SchedHostTask(0) +, /* mSfBlockSizeA */ 32 +, /* mSfBlockSizeB */ 32 +, /* mSfBlockSizeC */ 32 +, /* mSfLayoutA */ trtllm::gen::SfLayout(3) +, /* mSfLayoutB */ trtllm::gen::SfLayout(0) +, /* mSfLayoutC */ trtllm::gen::SfLayout(1) +, /* mSfReshapeFactor */ 1 +, /* mSparsityA */ trtllm::gen::Sparsity(0) +, /* mSplitK */ gemm::SplitK(0) +, /* mTileK */ 256 +, /* mTileM */ 128 +, /* mTileN */ 32 +, /* mTileScheduler */ gemm::TileScheduler(0) +, /* mTransposeMmaOutput */ 1 +, /* mUseCustomizedMma3xNvFp4 */ 0 +, /* mUseCustomMmaSchedule */ 1 +, /* mUseDeepSeekFp8 */ 0 +, /* mUseFlexibleClusterDims */ 0 +, /* mUseHoistTryWaitForCustomMmaSchedule */ 0 +, /* mUseMaxTmemOverlap */ 0 +, /* mUsePerTokenSfA */ 0 +, /* mUsePerTokenSfB */ 0 +, /* mUseShuffledMatrix */ 1 +, /* mUseTmaStore */ 1 +, /* mUseTwoTmaLoadWarps */ 1 +, /* mUseTwoMmaWarps */ 0 +, /* mUseUnrollLoop2xForMma */ 0 +, /* mValidM */ 256 +, /* mValidN */ 256 +, /* mValidK */ 256 +, /* mWorldSize */ 1 +, /* mActType */ gemmGatedAct::ActType(0) +, /* mClampBeforeAct */ 1 +, /* mBatchedM */ {} +, /* mBatchedN */ {} +, /* mBatchMode */ batchedGemm::BatchedGemmOptions::BatchMode(1) +, /* mBatchStrideInTokens */ -1 +, /* mFusedAct */ 1 +, /* mGridWaitForPrimaryRouting */ 1 +, /* mIsStaticBatch */ 0 +, /* mIsUniformNumTokensPerBatch */ 0 +, /* mNumBatches */ 128 +, /* mNumRegsPerThreadLoadA */ 0 +, /* mNumRegsPerThreadLoadB */ 0 +, /* mNumRegsPerThreadLoadSfA */ 0 +, /* mNumRegsPerThreadLoadSfB */ 0 +, /* mNumTokens */ 2 +, /* mNumWarpsLoadA */ 0 +, /* mNumWarpsLoadB */ 0 +, /* mNumWarpsLoadSfA */ 0 +, /* mNumWarpsLoadSfB */ 0 +, /* mRouteImpl */ batchedGemm::RouteImpl(2) +, /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} +, /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 + }, gemm::SmVersion::Sm100f}, +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len, 200600, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f", 640, "446a3b442242e4524b8db27b529c31a78bd0a5284c75b1e110c867008496f8bf", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +, /* mBiasDtype */ trtllm::gen::Dtype(1056776) +, /* mBiasType */ gemm::BiasType(0) +, /* mBlockK */ -1 +, /* mClcFastDrain */ 1 +, /* mClusterDimX */ 2 +, /* mClusterDimY */ 1 +, /* mClusterDimZ */ 1 +, /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) +, /* mDtypeAcc */ trtllm::gen::Dtype(1056776) +, /* mDtypeA */ trtllm::gen::Dtype(17826828) +, /* mDtypeB */ trtllm::gen::Dtype(17827853) +, /* mDtypeC */ trtllm::gen::Dtype(17827853) +, /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) +, /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) +, /* mEltwiseActType */ gemm::EltwiseActType(0) +, /* mEnablesEarlyExit */ 1 +, /* mEnablesDelayedEarlyExit */ 0 +, /* mEpilogueLdtmDps */ 16 +, /* mEpilogueLdtmBits */ 256 +, /* mEpilogueTileM */ 128 +, /* mEpilogueTileN */ 32 +, /* mFallbackClusterDimX */ 1 +, /* mFallbackClusterDimY */ 1 +, /* mFallbackClusterDimZ */ 1 +, /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) +, /* mFuseLoadSfTask */ 0 +, /* mFuseUtccpWithUtcmma */ 0 +, /* mGridTriggerSecondaryA */ 0 +, /* mGridTriggerSecondaryB */ 1 +, /* mGridWaitForPrimaryEarlyExit */ 1 +, /* mGridWaitForPrimaryA */ 0 +, /* mGridWaitForPrimaryB */ 1 +, /* mHoistLoadTaskInit */ 1 +, /* mHoistMmaTaskTryWaits */ 0 +, /* mK */ 256 +, /* mKernelTraits */ {} +, /* mLayoutA */ gemm::MatrixLayout(0) +, /* mLayoutB */ gemm::MatrixLayout(0) +, /* mM */ 256 +, /* mMmaK */ 32 +, /* mMmaKind */ trtllm::gen::MmaKind(5) +, /* mMmaM */ 256 +, /* mMmaN */ 32 +, /* mMmaTileK */ 256 +, /* mMockAllReduce */ 0 +, /* mN */ 256 +, /* mNumEpilogueWarps */ 4 +, /* mNumRegsCastAWarps */ 0 +, /* mNumRegsCopySfLdsSttm */ 0 +, /* mNumRegsCopySparsityInfo */ 0 +, /* mNumRegsPerThreadEpilogueWarp */ 0 +, /* mNumRegsPerThreadNonEpilogueWarp */ 0 +, /* mNumSlicesForSplitK */ 1 +, /* mNumStagesA */ 5 +, /* mNumStagesB */ 5 +, /* mNumStagesMma */ 2 +, /* mNumStagesMmaWithinWorkTile */ 1 +, /* mNumStagesMmaAcrossWorkTile */ 2 +, /* mNumStagesSmemSfA */ 5 +, /* mNumStagesSmemSfB */ 5 +, /* mNumStagesTmemSfA */ 5 +, /* mNumStagesTmemSfB */ 5 +, /* mNumStagesWorkId */ 3 +, /* mOutputDebugTensors */ 0 +, /* mPatchF2fp */ 0 +, /* mPerTokenSfDtype */ trtllm::gen::Dtype(1052672) +, /* mSchedHostTask */ gemm::SchedHostTask(0) +, /* mSfBlockSizeA */ 32 +, /* mSfBlockSizeB */ 32 +, /* mSfBlockSizeC */ 32 +, /* mSfLayoutA */ trtllm::gen::SfLayout(3) +, /* mSfLayoutB */ trtllm::gen::SfLayout(0) +, /* mSfLayoutC */ trtllm::gen::SfLayout(1) +, /* mSfReshapeFactor */ 1 +, /* mSparsityA */ trtllm::gen::Sparsity(0) +, /* mSplitK */ gemm::SplitK(0) +, /* mTileK */ 256 +, /* mTileM */ 128 +, /* mTileN */ 32 +, /* mTileScheduler */ gemm::TileScheduler(1) +, /* mTransposeMmaOutput */ 1 +, /* mUseCustomizedMma3xNvFp4 */ 0 +, /* mUseCustomMmaSchedule */ 1 +, /* mUseDeepSeekFp8 */ 0 +, /* mUseFlexibleClusterDims */ 0 +, /* mUseHoistTryWaitForCustomMmaSchedule */ 0 +, /* mUseMaxTmemOverlap */ 0 +, /* mUsePerTokenSfA */ 0 +, /* mUsePerTokenSfB */ 0 +, /* mUseShuffledMatrix */ 1 +, /* mUseTmaStore */ 1 +, /* mUseTwoTmaLoadWarps */ 1 +, /* mUseTwoMmaWarps */ 0 +, /* mUseUnrollLoop2xForMma */ 0 +, /* mValidM */ 256 +, /* mValidN */ 256 +, /* mValidK */ 256 +, /* mWorldSize */ 1 +, /* mActType */ gemmGatedAct::ActType(2) +, /* mClampBeforeAct */ 1 +, /* mBatchedM */ {} +, /* mBatchedN */ {} +, /* mBatchMode */ batchedGemm::BatchedGemmOptions::BatchMode(1) +, /* mBatchStrideInTokens */ -1 +, /* mFusedAct */ 1 +, /* mGridWaitForPrimaryRouting */ 1 +, /* mIsStaticBatch */ 0 +, /* mIsUniformNumTokensPerBatch */ 0 +, /* mNumBatches */ 128 +, /* mNumRegsPerThreadLoadA */ 0 +, /* mNumRegsPerThreadLoadB */ 0 +, /* mNumRegsPerThreadLoadSfA */ 0 +, /* mNumRegsPerThreadLoadSfB */ 0 +, /* mNumTokens */ 2 +, /* mNumWarpsLoadA */ 0 +, /* mNumWarpsLoadB */ 0 +, /* mNumWarpsLoadSfA */ 0 +, /* mNumWarpsLoadSfB */ 0 +, /* mRouteImpl */ batchedGemm::RouteImpl(2) +, /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} +, /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 + }, gemm::SmVersion::Sm100f}, +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 200600, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "79b4e3f1a0c50d0de7243a468b27d5a5d19fea41d1de624427712eef10430d7b", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +, /* mBiasDtype */ trtllm::gen::Dtype(1056776) +, /* mBiasType */ gemm::BiasType(1) +, /* mBlockK */ -1 +, /* mClcFastDrain */ 1 +, /* mClusterDimX */ 2 +, /* mClusterDimY */ 1 +, /* mClusterDimZ */ 1 +, /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) +, /* mDtypeAcc */ trtllm::gen::Dtype(1056776) +, /* mDtypeA */ trtllm::gen::Dtype(17826828) +, /* mDtypeB */ trtllm::gen::Dtype(17827853) +, /* mDtypeC */ trtllm::gen::Dtype(17827853) +, /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) +, /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) +, /* mEltwiseActType */ gemm::EltwiseActType(0) +, /* mEnablesEarlyExit */ 1 +, /* mEnablesDelayedEarlyExit */ 0 +, /* mEpilogueLdtmDps */ 16 +, /* mEpilogueLdtmBits */ 256 +, /* mEpilogueTileM */ 128 +, /* mEpilogueTileN */ 32 +, /* mFallbackClusterDimX */ 1 +, /* mFallbackClusterDimY */ 1 +, /* mFallbackClusterDimZ */ 1 +, /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) +, /* mFuseLoadSfTask */ 0 +, /* mFuseUtccpWithUtcmma */ 0 +, /* mGridTriggerSecondaryA */ 0 +, /* mGridTriggerSecondaryB */ 1 +, /* mGridWaitForPrimaryEarlyExit */ 1 +, /* mGridWaitForPrimaryA */ 0 +, /* mGridWaitForPrimaryB */ 1 +, /* mHoistLoadTaskInit */ 1 +, /* mHoistMmaTaskTryWaits */ 0 +, /* mK */ 256 +, /* mKernelTraits */ {} +, /* mLayoutA */ gemm::MatrixLayout(0) +, /* mLayoutB */ gemm::MatrixLayout(0) +, /* mM */ 256 +, /* mMmaK */ 32 +, /* mMmaKind */ trtllm::gen::MmaKind(5) +, /* mMmaM */ 256 +, /* mMmaN */ 32 +, /* mMmaTileK */ 256 +, /* mMockAllReduce */ 0 +, /* mN */ 256 +, /* mNumEpilogueWarps */ 4 +, /* mNumRegsCastAWarps */ 0 +, /* mNumRegsCopySfLdsSttm */ 0 +, /* mNumRegsCopySparsityInfo */ 0 +, /* mNumRegsPerThreadEpilogueWarp */ 0 +, /* mNumRegsPerThreadNonEpilogueWarp */ 0 +, /* mNumSlicesForSplitK */ 1 +, /* mNumStagesA */ 5 +, /* mNumStagesB */ 5 +, /* mNumStagesMma */ 2 +, /* mNumStagesMmaWithinWorkTile */ 1 +, /* mNumStagesMmaAcrossWorkTile */ 2 +, /* mNumStagesSmemSfA */ 5 +, /* mNumStagesSmemSfB */ 5 +, /* mNumStagesTmemSfA */ 5 +, /* mNumStagesTmemSfB */ 5 +, /* mNumStagesWorkId */ 3 +, /* mOutputDebugTensors */ 0 +, /* mPatchF2fp */ 0 +, /* mPerTokenSfDtype */ trtllm::gen::Dtype(1052672) +, /* mSchedHostTask */ gemm::SchedHostTask(0) +, /* mSfBlockSizeA */ 32 +, /* mSfBlockSizeB */ 32 +, /* mSfBlockSizeC */ 32 +, /* mSfLayoutA */ trtllm::gen::SfLayout(3) +, /* mSfLayoutB */ trtllm::gen::SfLayout(0) +, /* mSfLayoutC */ trtllm::gen::SfLayout(1) +, /* mSfReshapeFactor */ 1 +, /* mSparsityA */ trtllm::gen::Sparsity(0) +, /* mSplitK */ gemm::SplitK(0) +, /* mTileK */ 256 +, /* mTileM */ 128 +, /* mTileN */ 32 +, /* mTileScheduler */ gemm::TileScheduler(1) +, /* mTransposeMmaOutput */ 1 +, /* mUseCustomizedMma3xNvFp4 */ 0 +, /* mUseCustomMmaSchedule */ 1 +, /* mUseDeepSeekFp8 */ 0 +, /* mUseFlexibleClusterDims */ 0 +, /* mUseHoistTryWaitForCustomMmaSchedule */ 0 +, /* mUseMaxTmemOverlap */ 0 +, /* mUsePerTokenSfA */ 0 +, /* mUsePerTokenSfB */ 0 +, /* mUseShuffledMatrix */ 1 +, /* mUseTmaStore */ 1 +, /* mUseTwoTmaLoadWarps */ 1 +, /* mUseTwoMmaWarps */ 0 +, /* mUseUnrollLoop2xForMma */ 0 +, /* mValidM */ 256 +, /* mValidN */ 256 +, /* mValidK */ 256 +, /* mWorldSize */ 1 +, /* mActType */ gemmGatedAct::ActType(0) +, /* mClampBeforeAct */ 1 +, /* mBatchedM */ {} +, /* mBatchedN */ {} +, /* mBatchMode */ batchedGemm::BatchedGemmOptions::BatchMode(1) +, /* mBatchStrideInTokens */ -1 +, /* mFusedAct */ 1 +, /* mGridWaitForPrimaryRouting */ 1 +, /* mIsStaticBatch */ 0 +, /* mIsUniformNumTokensPerBatch */ 0 +, /* mNumBatches */ 128 +, /* mNumRegsPerThreadLoadA */ 0 +, /* mNumRegsPerThreadLoadB */ 0 +, /* mNumRegsPerThreadLoadSfA */ 0 +, /* mNumRegsPerThreadLoadSfB */ 0 +, /* mNumTokens */ 2 +, /* mNumWarpsLoadA */ 0 +, /* mNumWarpsLoadB */ 0 +, /* mNumWarpsLoadSfA */ 0 +, /* mNumWarpsLoadSfB */ 0 +, /* mRouteImpl */ batchedGemm::RouteImpl(2) +, /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} +, /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 + }, gemm::SmVersion::Sm100f}, +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len, 200216, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f", 640, "84976dbf9331b270bedc382984a5529d903072f96e98316228b9868bf29326bb", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +, /* mBiasDtype */ trtllm::gen::Dtype(1056776) +, /* mBiasType */ gemm::BiasType(0) +, /* mBlockK */ -1 +, /* mClcFastDrain */ 1 +, /* mClusterDimX */ 2 +, /* mClusterDimY */ 1 +, /* mClusterDimZ */ 1 +, /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) +, /* mDtypeAcc */ trtllm::gen::Dtype(1056776) +, /* mDtypeA */ trtllm::gen::Dtype(17826828) +, /* mDtypeB */ trtllm::gen::Dtype(17827853) +, /* mDtypeC */ trtllm::gen::Dtype(17827853) +, /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) +, /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) +, /* mEltwiseActType */ gemm::EltwiseActType(0) +, /* mEnablesEarlyExit */ 1 +, /* mEnablesDelayedEarlyExit */ 0 +, /* mEpilogueLdtmDps */ 16 +, /* mEpilogueLdtmBits */ 256 +, /* mEpilogueTileM */ 128 +, /* mEpilogueTileN */ 32 +, /* mFallbackClusterDimX */ 1 +, /* mFallbackClusterDimY */ 1 +, /* mFallbackClusterDimZ */ 1 +, /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) +, /* mFuseLoadSfTask */ 0 +, /* mFuseUtccpWithUtcmma */ 0 +, /* mGridTriggerSecondaryA */ 0 +, /* mGridTriggerSecondaryB */ 1 +, /* mGridWaitForPrimaryEarlyExit */ 1 +, /* mGridWaitForPrimaryA */ 0 +, /* mGridWaitForPrimaryB */ 1 +, /* mHoistLoadTaskInit */ 1 +, /* mHoistMmaTaskTryWaits */ 0 +, /* mK */ 256 +, /* mKernelTraits */ {} +, /* mLayoutA */ gemm::MatrixLayout(0) +, /* mLayoutB */ gemm::MatrixLayout(0) +, /* mM */ 256 +, /* mMmaK */ 32 +, /* mMmaKind */ trtllm::gen::MmaKind(5) +, /* mMmaM */ 256 +, /* mMmaN */ 32 +, /* mMmaTileK */ 256 +, /* mMockAllReduce */ 0 +, /* mN */ 256 +, /* mNumEpilogueWarps */ 4 +, /* mNumRegsCastAWarps */ 0 +, /* mNumRegsCopySfLdsSttm */ 0 +, /* mNumRegsCopySparsityInfo */ 0 +, /* mNumRegsPerThreadEpilogueWarp */ 0 +, /* mNumRegsPerThreadNonEpilogueWarp */ 0 +, /* mNumSlicesForSplitK */ 1 +, /* mNumStagesA */ 5 +, /* mNumStagesB */ 5 +, /* mNumStagesMma */ 1 +, /* mNumStagesMmaWithinWorkTile */ 1 +, /* mNumStagesMmaAcrossWorkTile */ 1 +, /* mNumStagesSmemSfA */ 5 +, /* mNumStagesSmemSfB */ 5 +, /* mNumStagesTmemSfA */ 5 +, /* mNumStagesTmemSfB */ 5 +, /* mNumStagesWorkId */ 3 +, /* mOutputDebugTensors */ 0 +, /* mPatchF2fp */ 0 +, /* mPerTokenSfDtype */ trtllm::gen::Dtype(1052672) +, /* mSchedHostTask */ gemm::SchedHostTask(0) +, /* mSfBlockSizeA */ 32 +, /* mSfBlockSizeB */ 32 +, /* mSfBlockSizeC */ 32 +, /* mSfLayoutA */ trtllm::gen::SfLayout(3) +, /* mSfLayoutB */ trtllm::gen::SfLayout(0) +, /* mSfLayoutC */ trtllm::gen::SfLayout(1) +, /* mSfReshapeFactor */ 1 +, /* mSparsityA */ trtllm::gen::Sparsity(0) +, /* mSplitK */ gemm::SplitK(0) +, /* mTileK */ 256 +, /* mTileM */ 128 +, /* mTileN */ 32 +, /* mTileScheduler */ gemm::TileScheduler(0) +, /* mTransposeMmaOutput */ 1 +, /* mUseCustomizedMma3xNvFp4 */ 0 +, /* mUseCustomMmaSchedule */ 1 +, /* mUseDeepSeekFp8 */ 0 +, /* mUseFlexibleClusterDims */ 0 +, /* mUseHoistTryWaitForCustomMmaSchedule */ 0 +, /* mUseMaxTmemOverlap */ 0 +, /* mUsePerTokenSfA */ 0 +, /* mUsePerTokenSfB */ 0 +, /* mUseShuffledMatrix */ 1 +, /* mUseTmaStore */ 1 +, /* mUseTwoTmaLoadWarps */ 1 +, /* mUseTwoMmaWarps */ 0 +, /* mUseUnrollLoop2xForMma */ 0 +, /* mValidM */ 256 +, /* mValidN */ 256 +, /* mValidK */ 256 +, /* mWorldSize */ 1 +, /* mActType */ gemmGatedAct::ActType(2) +, /* mClampBeforeAct */ 1 +, /* mBatchedM */ {} +, /* mBatchedN */ {} +, /* mBatchMode */ batchedGemm::BatchedGemmOptions::BatchMode(1) +, /* mBatchStrideInTokens */ -1 +, /* mFusedAct */ 1 +, /* mGridWaitForPrimaryRouting */ 1 +, /* mIsStaticBatch */ 0 +, /* mIsUniformNumTokensPerBatch */ 0 +, /* mNumBatches */ 128 +, /* mNumRegsPerThreadLoadA */ 0 +, /* mNumRegsPerThreadLoadB */ 0 +, /* mNumRegsPerThreadLoadSfA */ 0 +, /* mNumRegsPerThreadLoadSfB */ 0 +, /* mNumTokens */ 2 +, /* mNumWarpsLoadA */ 0 +, /* mNumWarpsLoadB */ 0 +, /* mNumWarpsLoadSfA */ 0 +, /* mNumWarpsLoadSfB */ 0 +, /* mRouteImpl */ batchedGemm::RouteImpl(2) +, /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} +, /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 + }, gemm::SmVersion::Sm100f}, +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 200216, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "0df5949855db1a0d57d07f1000b22df5dd15fece821da9875b8c8bee950fb7b9", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +, /* mBiasDtype */ trtllm::gen::Dtype(1056776) +, /* mBiasType */ gemm::BiasType(1) +, /* mBlockK */ -1 +, /* mClcFastDrain */ 1 +, /* mClusterDimX */ 2 +, /* mClusterDimY */ 1 +, /* mClusterDimZ */ 1 +, /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) +, /* mDtypeAcc */ trtllm::gen::Dtype(1056776) +, /* mDtypeA */ trtllm::gen::Dtype(17826828) +, /* mDtypeB */ trtllm::gen::Dtype(17827853) +, /* mDtypeC */ trtllm::gen::Dtype(17827853) +, /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) +, /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) +, /* mEltwiseActType */ gemm::EltwiseActType(0) +, /* mEnablesEarlyExit */ 1 +, /* mEnablesDelayedEarlyExit */ 0 +, /* mEpilogueLdtmDps */ 16 +, /* mEpilogueLdtmBits */ 256 +, /* mEpilogueTileM */ 128 +, /* mEpilogueTileN */ 32 +, /* mFallbackClusterDimX */ 1 +, /* mFallbackClusterDimY */ 1 +, /* mFallbackClusterDimZ */ 1 +, /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) +, /* mFuseLoadSfTask */ 0 +, /* mFuseUtccpWithUtcmma */ 0 +, /* mGridTriggerSecondaryA */ 0 +, /* mGridTriggerSecondaryB */ 1 +, /* mGridWaitForPrimaryEarlyExit */ 1 +, /* mGridWaitForPrimaryA */ 0 +, /* mGridWaitForPrimaryB */ 1 +, /* mHoistLoadTaskInit */ 1 +, /* mHoistMmaTaskTryWaits */ 0 +, /* mK */ 256 +, /* mKernelTraits */ {} +, /* mLayoutA */ gemm::MatrixLayout(0) +, /* mLayoutB */ gemm::MatrixLayout(0) +, /* mM */ 256 +, /* mMmaK */ 32 +, /* mMmaKind */ trtllm::gen::MmaKind(5) +, /* mMmaM */ 256 +, /* mMmaN */ 32 +, /* mMmaTileK */ 256 +, /* mMockAllReduce */ 0 +, /* mN */ 256 +, /* mNumEpilogueWarps */ 4 +, /* mNumRegsCastAWarps */ 0 +, /* mNumRegsCopySfLdsSttm */ 0 +, /* mNumRegsCopySparsityInfo */ 0 +, /* mNumRegsPerThreadEpilogueWarp */ 0 +, /* mNumRegsPerThreadNonEpilogueWarp */ 0 +, /* mNumSlicesForSplitK */ 1 +, /* mNumStagesA */ 5 +, /* mNumStagesB */ 5 +, /* mNumStagesMma */ 1 +, /* mNumStagesMmaWithinWorkTile */ 1 +, /* mNumStagesMmaAcrossWorkTile */ 1 +, /* mNumStagesSmemSfA */ 5 +, /* mNumStagesSmemSfB */ 5 +, /* mNumStagesTmemSfA */ 5 +, /* mNumStagesTmemSfB */ 5 +, /* mNumStagesWorkId */ 3 +, /* mOutputDebugTensors */ 0 +, /* mPatchF2fp */ 0 +, /* mPerTokenSfDtype */ trtllm::gen::Dtype(1052672) +, /* mSchedHostTask */ gemm::SchedHostTask(0) +, /* mSfBlockSizeA */ 32 +, /* mSfBlockSizeB */ 32 +, /* mSfBlockSizeC */ 32 +, /* mSfLayoutA */ trtllm::gen::SfLayout(3) +, /* mSfLayoutB */ trtllm::gen::SfLayout(0) +, /* mSfLayoutC */ trtllm::gen::SfLayout(1) +, /* mSfReshapeFactor */ 1 +, /* mSparsityA */ trtllm::gen::Sparsity(0) +, /* mSplitK */ gemm::SplitK(0) +, /* mTileK */ 256 +, /* mTileM */ 128 +, /* mTileN */ 32 +, /* mTileScheduler */ gemm::TileScheduler(0) +, /* mTransposeMmaOutput */ 1 +, /* mUseCustomizedMma3xNvFp4 */ 0 +, /* mUseCustomMmaSchedule */ 1 +, /* mUseDeepSeekFp8 */ 0 +, /* mUseFlexibleClusterDims */ 0 +, /* mUseHoistTryWaitForCustomMmaSchedule */ 0 +, /* mUseMaxTmemOverlap */ 0 +, /* mUsePerTokenSfA */ 0 +, /* mUsePerTokenSfB */ 0 +, /* mUseShuffledMatrix */ 1 +, /* mUseTmaStore */ 1 +, /* mUseTwoTmaLoadWarps */ 1 +, /* mUseTwoMmaWarps */ 0 +, /* mUseUnrollLoop2xForMma */ 0 +, /* mValidM */ 256 +, /* mValidN */ 256 +, /* mValidK */ 256 +, /* mWorldSize */ 1 +, /* mActType */ gemmGatedAct::ActType(0) +, /* mClampBeforeAct */ 1 +, /* mBatchedM */ {} +, /* mBatchedN */ {} +, /* mBatchMode */ batchedGemm::BatchedGemmOptions::BatchMode(1) +, /* mBatchStrideInTokens */ -1 +, /* mFusedAct */ 1 +, /* mGridWaitForPrimaryRouting */ 1 +, /* mIsStaticBatch */ 0 +, /* mIsUniformNumTokensPerBatch */ 0 +, /* mNumBatches */ 128 +, /* mNumRegsPerThreadLoadA */ 0 +, /* mNumRegsPerThreadLoadB */ 0 +, /* mNumRegsPerThreadLoadSfA */ 0 +, /* mNumRegsPerThreadLoadSfB */ 0 +, /* mNumTokens */ 2 +, /* mNumWarpsLoadA */ 0 +, /* mNumWarpsLoadB */ 0 +, /* mNumWarpsLoadSfA */ 0 +, /* mNumWarpsLoadSfB */ 0 +, /* mRouteImpl */ batchedGemm::RouteImpl(2) +, /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} +, /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 + }, gemm::SmVersion::Sm100f}, +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len, 221072, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f", 640, "5f57e2b74c8c324aa1f3de982b3c67cfd0237a59aacca7308c1dabca68359952", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +, /* mBiasDtype */ trtllm::gen::Dtype(1056776) +, /* mBiasType */ gemm::BiasType(0) +, /* mBlockK */ -1 +, /* mClcFastDrain */ 1 +, /* mClusterDimX */ 1 +, /* mClusterDimY */ 1 +, /* mClusterDimZ */ 1 +, /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) +, /* mDtypeAcc */ trtllm::gen::Dtype(1056776) +, /* mDtypeA */ trtllm::gen::Dtype(17826828) +, /* mDtypeB */ trtllm::gen::Dtype(17827853) +, /* mDtypeC */ trtllm::gen::Dtype(17827853) +, /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) +, /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) +, /* mEltwiseActType */ gemm::EltwiseActType(0) +, /* mEnablesEarlyExit */ 1 +, /* mEnablesDelayedEarlyExit */ 0 +, /* mEpilogueLdtmDps */ 16 +, /* mEpilogueLdtmBits */ 256 +, /* mEpilogueTileM */ 128 +, /* mEpilogueTileN */ 32 +, /* mFallbackClusterDimX */ 1 +, /* mFallbackClusterDimY */ 1 +, /* mFallbackClusterDimZ */ 1 +, /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) +, /* mFuseLoadSfTask */ 0 +, /* mFuseUtccpWithUtcmma */ 0 +, /* mGridTriggerSecondaryA */ 0 +, /* mGridTriggerSecondaryB */ 1 +, /* mGridWaitForPrimaryEarlyExit */ 1 +, /* mGridWaitForPrimaryA */ 0 +, /* mGridWaitForPrimaryB */ 1 +, /* mHoistLoadTaskInit */ 1 +, /* mHoistMmaTaskTryWaits */ 0 +, /* mK */ 512 +, /* mKernelTraits */ {} +, /* mLayoutA */ gemm::MatrixLayout(0) +, /* mLayoutB */ gemm::MatrixLayout(0) +, /* mM */ 256 +, /* mMmaK */ 32 +, /* mMmaKind */ trtllm::gen::MmaKind(5) +, /* mMmaM */ 128 +, /* mMmaN */ 32 +, /* mMmaTileK */ 256 +, /* mMockAllReduce */ 0 +, /* mN */ 256 +, /* mNumEpilogueWarps */ 4 +, /* mNumRegsCastAWarps */ 0 +, /* mNumRegsCopySfLdsSttm */ 0 +, /* mNumRegsCopySparsityInfo */ 0 +, /* mNumRegsPerThreadEpilogueWarp */ 0 +, /* mNumRegsPerThreadNonEpilogueWarp */ 0 +, /* mNumSlicesForSplitK */ 1 +, /* mNumStagesA */ 5 +, /* mNumStagesB */ 5 +, /* mNumStagesMma */ 2 +, /* mNumStagesMmaWithinWorkTile */ 1 +, /* mNumStagesMmaAcrossWorkTile */ 2 +, /* mNumStagesSmemSfA */ 5 +, /* mNumStagesSmemSfB */ 5 +, /* mNumStagesTmemSfA */ 5 +, /* mNumStagesTmemSfB */ 5 +, /* mNumStagesWorkId */ 3 +, /* mOutputDebugTensors */ 0 +, /* mPatchF2fp */ 0 +, /* mPerTokenSfDtype */ trtllm::gen::Dtype(1052672) +, /* mSchedHostTask */ gemm::SchedHostTask(0) +, /* mSfBlockSizeA */ 32 +, /* mSfBlockSizeB */ 32 +, /* mSfBlockSizeC */ 32 +, /* mSfLayoutA */ trtllm::gen::SfLayout(3) +, /* mSfLayoutB */ trtllm::gen::SfLayout(0) +, /* mSfLayoutC */ trtllm::gen::SfLayout(1) +, /* mSfReshapeFactor */ 1 +, /* mSparsityA */ trtllm::gen::Sparsity(0) +, /* mSplitK */ gemm::SplitK(0) +, /* mTileK */ 256 +, /* mTileM */ 128 +, /* mTileN */ 32 +, /* mTileScheduler */ gemm::TileScheduler(1) +, /* mTransposeMmaOutput */ 1 +, /* mUseCustomizedMma3xNvFp4 */ 0 +, /* mUseCustomMmaSchedule */ 1 +, /* mUseDeepSeekFp8 */ 0 +, /* mUseFlexibleClusterDims */ 0 +, /* mUseHoistTryWaitForCustomMmaSchedule */ 0 +, /* mUseMaxTmemOverlap */ 0 +, /* mUsePerTokenSfA */ 0 +, /* mUsePerTokenSfB */ 0 +, /* mUseShuffledMatrix */ 1 +, /* mUseTmaStore */ 1 +, /* mUseTwoTmaLoadWarps */ 1 +, /* mUseTwoMmaWarps */ 0 +, /* mUseUnrollLoop2xForMma */ 1 +, /* mValidM */ 256 +, /* mValidN */ 256 +, /* mValidK */ 512 +, /* mWorldSize */ 1 +, /* mActType */ gemmGatedAct::ActType(2) +, /* mClampBeforeAct */ 1 +, /* mBatchedM */ {} +, /* mBatchedN */ {} +, /* mBatchMode */ batchedGemm::BatchedGemmOptions::BatchMode(1) +, /* mBatchStrideInTokens */ -1 +, /* mFusedAct */ 1 +, /* mGridWaitForPrimaryRouting */ 1 +, /* mIsStaticBatch */ 0 +, /* mIsUniformNumTokensPerBatch */ 0 +, /* mNumBatches */ 128 +, /* mNumRegsPerThreadLoadA */ 0 +, /* mNumRegsPerThreadLoadB */ 0 +, /* mNumRegsPerThreadLoadSfA */ 0 +, /* mNumRegsPerThreadLoadSfB */ 0 +, /* mNumTokens */ 2 +, /* mNumWarpsLoadA */ 0 +, /* mNumWarpsLoadB */ 0 +, /* mNumWarpsLoadSfA */ 0 +, /* mNumWarpsLoadSfB */ 0 +, /* mRouteImpl */ batchedGemm::RouteImpl(2) +, /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} +, /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 + }, gemm::SmVersion::Sm100f}, +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 221072, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "2030724ccadf3ea36dde52962cc9e00ee9aaca201659a010e8cd1d9dd6580eb3", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +, /* mBiasDtype */ trtllm::gen::Dtype(1056776) +, /* mBiasType */ gemm::BiasType(1) +, /* mBlockK */ -1 +, /* mClcFastDrain */ 1 +, /* mClusterDimX */ 1 +, /* mClusterDimY */ 1 +, /* mClusterDimZ */ 1 +, /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) +, /* mDtypeAcc */ trtllm::gen::Dtype(1056776) +, /* mDtypeA */ trtllm::gen::Dtype(17826828) +, /* mDtypeB */ trtllm::gen::Dtype(17827853) +, /* mDtypeC */ trtllm::gen::Dtype(17827853) +, /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) +, /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) +, /* mEltwiseActType */ gemm::EltwiseActType(0) +, /* mEnablesEarlyExit */ 1 +, /* mEnablesDelayedEarlyExit */ 0 +, /* mEpilogueLdtmDps */ 16 +, /* mEpilogueLdtmBits */ 256 +, /* mEpilogueTileM */ 128 +, /* mEpilogueTileN */ 32 +, /* mFallbackClusterDimX */ 1 +, /* mFallbackClusterDimY */ 1 +, /* mFallbackClusterDimZ */ 1 +, /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) +, /* mFuseLoadSfTask */ 0 +, /* mFuseUtccpWithUtcmma */ 0 +, /* mGridTriggerSecondaryA */ 0 +, /* mGridTriggerSecondaryB */ 1 +, /* mGridWaitForPrimaryEarlyExit */ 1 +, /* mGridWaitForPrimaryA */ 0 +, /* mGridWaitForPrimaryB */ 1 +, /* mHoistLoadTaskInit */ 1 +, /* mHoistMmaTaskTryWaits */ 0 +, /* mK */ 512 +, /* mKernelTraits */ {} +, /* mLayoutA */ gemm::MatrixLayout(0) +, /* mLayoutB */ gemm::MatrixLayout(0) +, /* mM */ 256 +, /* mMmaK */ 32 +, /* mMmaKind */ trtllm::gen::MmaKind(5) +, /* mMmaM */ 128 +, /* mMmaN */ 32 +, /* mMmaTileK */ 256 +, /* mMockAllReduce */ 0 +, /* mN */ 256 +, /* mNumEpilogueWarps */ 4 +, /* mNumRegsCastAWarps */ 0 +, /* mNumRegsCopySfLdsSttm */ 0 +, /* mNumRegsCopySparsityInfo */ 0 +, /* mNumRegsPerThreadEpilogueWarp */ 0 +, /* mNumRegsPerThreadNonEpilogueWarp */ 0 +, /* mNumSlicesForSplitK */ 1 +, /* mNumStagesA */ 5 +, /* mNumStagesB */ 5 +, /* mNumStagesMma */ 2 +, /* mNumStagesMmaWithinWorkTile */ 1 +, /* mNumStagesMmaAcrossWorkTile */ 2 +, /* mNumStagesSmemSfA */ 5 +, /* mNumStagesSmemSfB */ 5 +, /* mNumStagesTmemSfA */ 5 +, /* mNumStagesTmemSfB */ 5 +, /* mNumStagesWorkId */ 3 +, /* mOutputDebugTensors */ 0 +, /* mPatchF2fp */ 0 +, /* mPerTokenSfDtype */ trtllm::gen::Dtype(1052672) +, /* mSchedHostTask */ gemm::SchedHostTask(0) +, /* mSfBlockSizeA */ 32 +, /* mSfBlockSizeB */ 32 +, /* mSfBlockSizeC */ 32 +, /* mSfLayoutA */ trtllm::gen::SfLayout(3) +, /* mSfLayoutB */ trtllm::gen::SfLayout(0) +, /* mSfLayoutC */ trtllm::gen::SfLayout(1) +, /* mSfReshapeFactor */ 1 +, /* mSparsityA */ trtllm::gen::Sparsity(0) +, /* mSplitK */ gemm::SplitK(0) +, /* mTileK */ 256 +, /* mTileM */ 128 +, /* mTileN */ 32 +, /* mTileScheduler */ gemm::TileScheduler(1) +, /* mTransposeMmaOutput */ 1 +, /* mUseCustomizedMma3xNvFp4 */ 0 +, /* mUseCustomMmaSchedule */ 1 +, /* mUseDeepSeekFp8 */ 0 +, /* mUseFlexibleClusterDims */ 0 +, /* mUseHoistTryWaitForCustomMmaSchedule */ 0 +, /* mUseMaxTmemOverlap */ 0 +, /* mUsePerTokenSfA */ 0 +, /* mUsePerTokenSfB */ 0 +, /* mUseShuffledMatrix */ 1 +, /* mUseTmaStore */ 1 +, /* mUseTwoTmaLoadWarps */ 1 +, /* mUseTwoMmaWarps */ 0 +, /* mUseUnrollLoop2xForMma */ 1 +, /* mValidM */ 256 +, /* mValidN */ 256 +, /* mValidK */ 512 +, /* mWorldSize */ 1 +, /* mActType */ gemmGatedAct::ActType(0) +, /* mClampBeforeAct */ 1 +, /* mBatchedM */ {} +, /* mBatchedN */ {} +, /* mBatchMode */ batchedGemm::BatchedGemmOptions::BatchMode(1) +, /* mBatchStrideInTokens */ -1 +, /* mFusedAct */ 1 +, /* mGridWaitForPrimaryRouting */ 1 +, /* mIsStaticBatch */ 0 +, /* mIsUniformNumTokensPerBatch */ 0 +, /* mNumBatches */ 128 +, /* mNumRegsPerThreadLoadA */ 0 +, /* mNumRegsPerThreadLoadB */ 0 +, /* mNumRegsPerThreadLoadSfA */ 0 +, /* mNumRegsPerThreadLoadSfB */ 0 +, /* mNumTokens */ 2 +, /* mNumWarpsLoadA */ 0 +, /* mNumWarpsLoadB */ 0 +, /* mNumWarpsLoadSfA */ 0 +, /* mNumWarpsLoadSfB */ 0 +, /* mRouteImpl */ batchedGemm::RouteImpl(2) +, /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} +, /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 + }, gemm::SmVersion::Sm100f}, +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len, 220688, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f", 640, "000296a9befea9ec01b16d1bc984d9d9d858fec1471696e9410434b333fce0f9", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +, /* mBiasDtype */ trtllm::gen::Dtype(1056776) +, /* mBiasType */ gemm::BiasType(0) +, /* mBlockK */ -1 +, /* mClcFastDrain */ 1 +, /* mClusterDimX */ 1 +, /* mClusterDimY */ 1 +, /* mClusterDimZ */ 1 +, /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) +, /* mDtypeAcc */ trtllm::gen::Dtype(1056776) +, /* mDtypeA */ trtllm::gen::Dtype(17826828) +, /* mDtypeB */ trtllm::gen::Dtype(17827853) +, /* mDtypeC */ trtllm::gen::Dtype(17827853) +, /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) +, /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) +, /* mEltwiseActType */ gemm::EltwiseActType(0) +, /* mEnablesEarlyExit */ 1 +, /* mEnablesDelayedEarlyExit */ 0 +, /* mEpilogueLdtmDps */ 16 +, /* mEpilogueLdtmBits */ 256 +, /* mEpilogueTileM */ 128 +, /* mEpilogueTileN */ 32 +, /* mFallbackClusterDimX */ 1 +, /* mFallbackClusterDimY */ 1 +, /* mFallbackClusterDimZ */ 1 +, /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) +, /* mFuseLoadSfTask */ 0 +, /* mFuseUtccpWithUtcmma */ 0 +, /* mGridTriggerSecondaryA */ 0 +, /* mGridTriggerSecondaryB */ 1 +, /* mGridWaitForPrimaryEarlyExit */ 1 +, /* mGridWaitForPrimaryA */ 0 +, /* mGridWaitForPrimaryB */ 1 +, /* mHoistLoadTaskInit */ 1 +, /* mHoistMmaTaskTryWaits */ 0 +, /* mK */ 512 +, /* mKernelTraits */ {} +, /* mLayoutA */ gemm::MatrixLayout(0) +, /* mLayoutB */ gemm::MatrixLayout(0) +, /* mM */ 256 +, /* mMmaK */ 32 +, /* mMmaKind */ trtllm::gen::MmaKind(5) +, /* mMmaM */ 128 +, /* mMmaN */ 32 +, /* mMmaTileK */ 256 +, /* mMockAllReduce */ 0 +, /* mN */ 256 +, /* mNumEpilogueWarps */ 4 +, /* mNumRegsCastAWarps */ 0 +, /* mNumRegsCopySfLdsSttm */ 0 +, /* mNumRegsCopySparsityInfo */ 0 +, /* mNumRegsPerThreadEpilogueWarp */ 0 +, /* mNumRegsPerThreadNonEpilogueWarp */ 0 +, /* mNumSlicesForSplitK */ 1 +, /* mNumStagesA */ 5 +, /* mNumStagesB */ 5 +, /* mNumStagesMma */ 1 +, /* mNumStagesMmaWithinWorkTile */ 1 +, /* mNumStagesMmaAcrossWorkTile */ 1 +, /* mNumStagesSmemSfA */ 5 +, /* mNumStagesSmemSfB */ 5 +, /* mNumStagesTmemSfA */ 5 +, /* mNumStagesTmemSfB */ 5 +, /* mNumStagesWorkId */ 3 +, /* mOutputDebugTensors */ 0 +, /* mPatchF2fp */ 0 +, /* mPerTokenSfDtype */ trtllm::gen::Dtype(1052672) +, /* mSchedHostTask */ gemm::SchedHostTask(0) +, /* mSfBlockSizeA */ 32 +, /* mSfBlockSizeB */ 32 +, /* mSfBlockSizeC */ 32 +, /* mSfLayoutA */ trtllm::gen::SfLayout(3) +, /* mSfLayoutB */ trtllm::gen::SfLayout(0) +, /* mSfLayoutC */ trtllm::gen::SfLayout(1) +, /* mSfReshapeFactor */ 1 +, /* mSparsityA */ trtllm::gen::Sparsity(0) +, /* mSplitK */ gemm::SplitK(0) +, /* mTileK */ 256 +, /* mTileM */ 128 +, /* mTileN */ 32 +, /* mTileScheduler */ gemm::TileScheduler(0) +, /* mTransposeMmaOutput */ 1 +, /* mUseCustomizedMma3xNvFp4 */ 0 +, /* mUseCustomMmaSchedule */ 1 +, /* mUseDeepSeekFp8 */ 0 +, /* mUseFlexibleClusterDims */ 0 +, /* mUseHoistTryWaitForCustomMmaSchedule */ 0 +, /* mUseMaxTmemOverlap */ 0 +, /* mUsePerTokenSfA */ 0 +, /* mUsePerTokenSfB */ 0 +, /* mUseShuffledMatrix */ 1 +, /* mUseTmaStore */ 1 +, /* mUseTwoTmaLoadWarps */ 1 +, /* mUseTwoMmaWarps */ 0 +, /* mUseUnrollLoop2xForMma */ 1 +, /* mValidM */ 256 +, /* mValidN */ 256 +, /* mValidK */ 512 +, /* mWorldSize */ 1 +, /* mActType */ gemmGatedAct::ActType(2) +, /* mClampBeforeAct */ 1 +, /* mBatchedM */ {} +, /* mBatchedN */ {} +, /* mBatchMode */ batchedGemm::BatchedGemmOptions::BatchMode(1) +, /* mBatchStrideInTokens */ -1 +, /* mFusedAct */ 1 +, /* mGridWaitForPrimaryRouting */ 1 +, /* mIsStaticBatch */ 0 +, /* mIsUniformNumTokensPerBatch */ 0 +, /* mNumBatches */ 128 +, /* mNumRegsPerThreadLoadA */ 0 +, /* mNumRegsPerThreadLoadB */ 0 +, /* mNumRegsPerThreadLoadSfA */ 0 +, /* mNumRegsPerThreadLoadSfB */ 0 +, /* mNumTokens */ 2 +, /* mNumWarpsLoadA */ 0 +, /* mNumWarpsLoadB */ 0 +, /* mNumWarpsLoadSfA */ 0 +, /* mNumWarpsLoadSfB */ 0 +, /* mRouteImpl */ batchedGemm::RouteImpl(2) +, /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} +, /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 + }, gemm::SmVersion::Sm100f}, +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 220688, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "32c35df4172c523d329b3fef001e6d531937b67b9152c7577ca9d8ab3dec4d78", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +, /* mBiasDtype */ trtllm::gen::Dtype(1056776) +, /* mBiasType */ gemm::BiasType(1) +, /* mBlockK */ -1 +, /* mClcFastDrain */ 1 +, /* mClusterDimX */ 1 +, /* mClusterDimY */ 1 +, /* mClusterDimZ */ 1 +, /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) +, /* mDtypeAcc */ trtllm::gen::Dtype(1056776) +, /* mDtypeA */ trtllm::gen::Dtype(17826828) +, /* mDtypeB */ trtllm::gen::Dtype(17827853) +, /* mDtypeC */ trtllm::gen::Dtype(17827853) +, /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) +, /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) +, /* mEltwiseActType */ gemm::EltwiseActType(0) +, /* mEnablesEarlyExit */ 1 +, /* mEnablesDelayedEarlyExit */ 0 +, /* mEpilogueLdtmDps */ 16 +, /* mEpilogueLdtmBits */ 256 +, /* mEpilogueTileM */ 128 +, /* mEpilogueTileN */ 32 +, /* mFallbackClusterDimX */ 1 +, /* mFallbackClusterDimY */ 1 +, /* mFallbackClusterDimZ */ 1 +, /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) +, /* mFuseLoadSfTask */ 0 +, /* mFuseUtccpWithUtcmma */ 0 +, /* mGridTriggerSecondaryA */ 0 +, /* mGridTriggerSecondaryB */ 1 +, /* mGridWaitForPrimaryEarlyExit */ 1 +, /* mGridWaitForPrimaryA */ 0 +, /* mGridWaitForPrimaryB */ 1 +, /* mHoistLoadTaskInit */ 1 +, /* mHoistMmaTaskTryWaits */ 0 +, /* mK */ 512 +, /* mKernelTraits */ {} +, /* mLayoutA */ gemm::MatrixLayout(0) +, /* mLayoutB */ gemm::MatrixLayout(0) +, /* mM */ 256 +, /* mMmaK */ 32 +, /* mMmaKind */ trtllm::gen::MmaKind(5) +, /* mMmaM */ 128 +, /* mMmaN */ 32 +, /* mMmaTileK */ 256 +, /* mMockAllReduce */ 0 +, /* mN */ 256 +, /* mNumEpilogueWarps */ 4 +, /* mNumRegsCastAWarps */ 0 +, /* mNumRegsCopySfLdsSttm */ 0 +, /* mNumRegsCopySparsityInfo */ 0 +, /* mNumRegsPerThreadEpilogueWarp */ 0 +, /* mNumRegsPerThreadNonEpilogueWarp */ 0 +, /* mNumSlicesForSplitK */ 1 +, /* mNumStagesA */ 5 +, /* mNumStagesB */ 5 +, /* mNumStagesMma */ 1 +, /* mNumStagesMmaWithinWorkTile */ 1 +, /* mNumStagesMmaAcrossWorkTile */ 1 +, /* mNumStagesSmemSfA */ 5 +, /* mNumStagesSmemSfB */ 5 +, /* mNumStagesTmemSfA */ 5 +, /* mNumStagesTmemSfB */ 5 +, /* mNumStagesWorkId */ 3 +, /* mOutputDebugTensors */ 0 +, /* mPatchF2fp */ 0 +, /* mPerTokenSfDtype */ trtllm::gen::Dtype(1052672) +, /* mSchedHostTask */ gemm::SchedHostTask(0) +, /* mSfBlockSizeA */ 32 +, /* mSfBlockSizeB */ 32 +, /* mSfBlockSizeC */ 32 +, /* mSfLayoutA */ trtllm::gen::SfLayout(3) +, /* mSfLayoutB */ trtllm::gen::SfLayout(0) +, /* mSfLayoutC */ trtllm::gen::SfLayout(1) +, /* mSfReshapeFactor */ 1 +, /* mSparsityA */ trtllm::gen::Sparsity(0) +, /* mSplitK */ gemm::SplitK(0) +, /* mTileK */ 256 +, /* mTileM */ 128 +, /* mTileN */ 32 +, /* mTileScheduler */ gemm::TileScheduler(0) +, /* mTransposeMmaOutput */ 1 +, /* mUseCustomizedMma3xNvFp4 */ 0 +, /* mUseCustomMmaSchedule */ 1 +, /* mUseDeepSeekFp8 */ 0 +, /* mUseFlexibleClusterDims */ 0 +, /* mUseHoistTryWaitForCustomMmaSchedule */ 0 +, /* mUseMaxTmemOverlap */ 0 +, /* mUsePerTokenSfA */ 0 +, /* mUsePerTokenSfB */ 0 +, /* mUseShuffledMatrix */ 1 +, /* mUseTmaStore */ 1 +, /* mUseTwoTmaLoadWarps */ 1 +, /* mUseTwoMmaWarps */ 0 +, /* mUseUnrollLoop2xForMma */ 1 +, /* mValidM */ 256 +, /* mValidN */ 256 +, /* mValidK */ 512 +, /* mWorldSize */ 1 +, /* mActType */ gemmGatedAct::ActType(0) +, /* mClampBeforeAct */ 1 +, /* mBatchedM */ {} +, /* mBatchedN */ {} +, /* mBatchMode */ batchedGemm::BatchedGemmOptions::BatchMode(1) +, /* mBatchStrideInTokens */ -1 +, /* mFusedAct */ 1 +, /* mGridWaitForPrimaryRouting */ 1 +, /* mIsStaticBatch */ 0 +, /* mIsUniformNumTokensPerBatch */ 0 +, /* mNumBatches */ 128 +, /* mNumRegsPerThreadLoadA */ 0 +, /* mNumRegsPerThreadLoadB */ 0 +, /* mNumRegsPerThreadLoadSfA */ 0 +, /* mNumRegsPerThreadLoadSfB */ 0 +, /* mNumTokens */ 2 +, /* mNumWarpsLoadA */ 0 +, /* mNumWarpsLoadB */ 0 +, /* mNumWarpsLoadSfA */ 0 +, /* mNumWarpsLoadSfB */ 0 +, /* mRouteImpl */ batchedGemm::RouteImpl(2) +, /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} +, /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 + }, gemm::SmVersion::Sm100f}, +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len, 200600, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f", 640, "3122cf721fc10ab2547d5df61f8951db840322e0a3e245d7e08cab99a28b7d6f", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +, /* mBiasDtype */ trtllm::gen::Dtype(1056776) +, /* mBiasType */ gemm::BiasType(0) +, /* mBlockK */ -1 +, /* mClcFastDrain */ 1 +, /* mClusterDimX */ 2 +, /* mClusterDimY */ 1 +, /* mClusterDimZ */ 1 +, /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) +, /* mDtypeAcc */ trtllm::gen::Dtype(1056776) +, /* mDtypeA */ trtllm::gen::Dtype(17826828) +, /* mDtypeB */ trtllm::gen::Dtype(17827853) +, /* mDtypeC */ trtllm::gen::Dtype(17827853) +, /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) +, /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) +, /* mEltwiseActType */ gemm::EltwiseActType(0) +, /* mEnablesEarlyExit */ 1 +, /* mEnablesDelayedEarlyExit */ 0 +, /* mEpilogueLdtmDps */ 16 +, /* mEpilogueLdtmBits */ 256 +, /* mEpilogueTileM */ 128 +, /* mEpilogueTileN */ 32 +, /* mFallbackClusterDimX */ 1 +, /* mFallbackClusterDimY */ 1 +, /* mFallbackClusterDimZ */ 1 +, /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) +, /* mFuseLoadSfTask */ 0 +, /* mFuseUtccpWithUtcmma */ 0 +, /* mGridTriggerSecondaryA */ 0 +, /* mGridTriggerSecondaryB */ 1 +, /* mGridWaitForPrimaryEarlyExit */ 1 +, /* mGridWaitForPrimaryA */ 0 +, /* mGridWaitForPrimaryB */ 1 +, /* mHoistLoadTaskInit */ 1 +, /* mHoistMmaTaskTryWaits */ 0 +, /* mK */ 512 +, /* mKernelTraits */ {} +, /* mLayoutA */ gemm::MatrixLayout(0) +, /* mLayoutB */ gemm::MatrixLayout(0) +, /* mM */ 256 +, /* mMmaK */ 32 +, /* mMmaKind */ trtllm::gen::MmaKind(5) +, /* mMmaM */ 256 +, /* mMmaN */ 32 +, /* mMmaTileK */ 256 +, /* mMockAllReduce */ 0 +, /* mN */ 256 +, /* mNumEpilogueWarps */ 4 +, /* mNumRegsCastAWarps */ 0 +, /* mNumRegsCopySfLdsSttm */ 0 +, /* mNumRegsCopySparsityInfo */ 0 +, /* mNumRegsPerThreadEpilogueWarp */ 0 +, /* mNumRegsPerThreadNonEpilogueWarp */ 0 +, /* mNumSlicesForSplitK */ 1 +, /* mNumStagesA */ 5 +, /* mNumStagesB */ 5 +, /* mNumStagesMma */ 2 +, /* mNumStagesMmaWithinWorkTile */ 1 +, /* mNumStagesMmaAcrossWorkTile */ 2 +, /* mNumStagesSmemSfA */ 5 +, /* mNumStagesSmemSfB */ 5 +, /* mNumStagesTmemSfA */ 5 +, /* mNumStagesTmemSfB */ 5 +, /* mNumStagesWorkId */ 3 +, /* mOutputDebugTensors */ 0 +, /* mPatchF2fp */ 0 +, /* mPerTokenSfDtype */ trtllm::gen::Dtype(1052672) +, /* mSchedHostTask */ gemm::SchedHostTask(0) +, /* mSfBlockSizeA */ 32 +, /* mSfBlockSizeB */ 32 +, /* mSfBlockSizeC */ 32 +, /* mSfLayoutA */ trtllm::gen::SfLayout(3) +, /* mSfLayoutB */ trtllm::gen::SfLayout(0) +, /* mSfLayoutC */ trtllm::gen::SfLayout(1) +, /* mSfReshapeFactor */ 1 +, /* mSparsityA */ trtllm::gen::Sparsity(0) +, /* mSplitK */ gemm::SplitK(0) +, /* mTileK */ 256 +, /* mTileM */ 128 +, /* mTileN */ 32 +, /* mTileScheduler */ gemm::TileScheduler(1) +, /* mTransposeMmaOutput */ 1 +, /* mUseCustomizedMma3xNvFp4 */ 0 +, /* mUseCustomMmaSchedule */ 1 +, /* mUseDeepSeekFp8 */ 0 +, /* mUseFlexibleClusterDims */ 0 +, /* mUseHoistTryWaitForCustomMmaSchedule */ 0 +, /* mUseMaxTmemOverlap */ 0 +, /* mUsePerTokenSfA */ 0 +, /* mUsePerTokenSfB */ 0 +, /* mUseShuffledMatrix */ 1 +, /* mUseTmaStore */ 1 +, /* mUseTwoTmaLoadWarps */ 1 +, /* mUseTwoMmaWarps */ 0 +, /* mUseUnrollLoop2xForMma */ 1 +, /* mValidM */ 256 +, /* mValidN */ 256 +, /* mValidK */ 512 +, /* mWorldSize */ 1 +, /* mActType */ gemmGatedAct::ActType(2) +, /* mClampBeforeAct */ 1 +, /* mBatchedM */ {} +, /* mBatchedN */ {} +, /* mBatchMode */ batchedGemm::BatchedGemmOptions::BatchMode(1) +, /* mBatchStrideInTokens */ -1 +, /* mFusedAct */ 1 +, /* mGridWaitForPrimaryRouting */ 1 +, /* mIsStaticBatch */ 0 +, /* mIsUniformNumTokensPerBatch */ 0 +, /* mNumBatches */ 128 +, /* mNumRegsPerThreadLoadA */ 0 +, /* mNumRegsPerThreadLoadB */ 0 +, /* mNumRegsPerThreadLoadSfA */ 0 +, /* mNumRegsPerThreadLoadSfB */ 0 +, /* mNumTokens */ 2 +, /* mNumWarpsLoadA */ 0 +, /* mNumWarpsLoadB */ 0 +, /* mNumWarpsLoadSfA */ 0 +, /* mNumWarpsLoadSfB */ 0 +, /* mRouteImpl */ batchedGemm::RouteImpl(2) +, /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} +, /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 + }, gemm::SmVersion::Sm100f}, +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 200600, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "d6a0b34d4655de8521e8361f05351431d9d3e5c0537d3f1ee1594ea04200740a", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +, /* mBiasDtype */ trtllm::gen::Dtype(1056776) +, /* mBiasType */ gemm::BiasType(1) +, /* mBlockK */ -1 +, /* mClcFastDrain */ 1 +, /* mClusterDimX */ 2 +, /* mClusterDimY */ 1 +, /* mClusterDimZ */ 1 +, /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) +, /* mDtypeAcc */ trtllm::gen::Dtype(1056776) +, /* mDtypeA */ trtllm::gen::Dtype(17826828) +, /* mDtypeB */ trtllm::gen::Dtype(17827853) +, /* mDtypeC */ trtllm::gen::Dtype(17827853) +, /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) +, /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) +, /* mEltwiseActType */ gemm::EltwiseActType(0) +, /* mEnablesEarlyExit */ 1 +, /* mEnablesDelayedEarlyExit */ 0 +, /* mEpilogueLdtmDps */ 16 +, /* mEpilogueLdtmBits */ 256 +, /* mEpilogueTileM */ 128 +, /* mEpilogueTileN */ 32 +, /* mFallbackClusterDimX */ 1 +, /* mFallbackClusterDimY */ 1 +, /* mFallbackClusterDimZ */ 1 +, /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) +, /* mFuseLoadSfTask */ 0 +, /* mFuseUtccpWithUtcmma */ 0 +, /* mGridTriggerSecondaryA */ 0 +, /* mGridTriggerSecondaryB */ 1 +, /* mGridWaitForPrimaryEarlyExit */ 1 +, /* mGridWaitForPrimaryA */ 0 +, /* mGridWaitForPrimaryB */ 1 +, /* mHoistLoadTaskInit */ 1 +, /* mHoistMmaTaskTryWaits */ 0 +, /* mK */ 512 +, /* mKernelTraits */ {} +, /* mLayoutA */ gemm::MatrixLayout(0) +, /* mLayoutB */ gemm::MatrixLayout(0) +, /* mM */ 256 +, /* mMmaK */ 32 +, /* mMmaKind */ trtllm::gen::MmaKind(5) +, /* mMmaM */ 256 +, /* mMmaN */ 32 +, /* mMmaTileK */ 256 +, /* mMockAllReduce */ 0 +, /* mN */ 256 +, /* mNumEpilogueWarps */ 4 +, /* mNumRegsCastAWarps */ 0 +, /* mNumRegsCopySfLdsSttm */ 0 +, /* mNumRegsCopySparsityInfo */ 0 +, /* mNumRegsPerThreadEpilogueWarp */ 0 +, /* mNumRegsPerThreadNonEpilogueWarp */ 0 +, /* mNumSlicesForSplitK */ 1 +, /* mNumStagesA */ 5 +, /* mNumStagesB */ 5 +, /* mNumStagesMma */ 2 +, /* mNumStagesMmaWithinWorkTile */ 1 +, /* mNumStagesMmaAcrossWorkTile */ 2 +, /* mNumStagesSmemSfA */ 5 +, /* mNumStagesSmemSfB */ 5 +, /* mNumStagesTmemSfA */ 5 +, /* mNumStagesTmemSfB */ 5 +, /* mNumStagesWorkId */ 3 +, /* mOutputDebugTensors */ 0 +, /* mPatchF2fp */ 0 +, /* mPerTokenSfDtype */ trtllm::gen::Dtype(1052672) +, /* mSchedHostTask */ gemm::SchedHostTask(0) +, /* mSfBlockSizeA */ 32 +, /* mSfBlockSizeB */ 32 +, /* mSfBlockSizeC */ 32 +, /* mSfLayoutA */ trtllm::gen::SfLayout(3) +, /* mSfLayoutB */ trtllm::gen::SfLayout(0) +, /* mSfLayoutC */ trtllm::gen::SfLayout(1) +, /* mSfReshapeFactor */ 1 +, /* mSparsityA */ trtllm::gen::Sparsity(0) +, /* mSplitK */ gemm::SplitK(0) +, /* mTileK */ 256 +, /* mTileM */ 128 +, /* mTileN */ 32 +, /* mTileScheduler */ gemm::TileScheduler(1) +, /* mTransposeMmaOutput */ 1 +, /* mUseCustomizedMma3xNvFp4 */ 0 +, /* mUseCustomMmaSchedule */ 1 +, /* mUseDeepSeekFp8 */ 0 +, /* mUseFlexibleClusterDims */ 0 +, /* mUseHoistTryWaitForCustomMmaSchedule */ 0 +, /* mUseMaxTmemOverlap */ 0 +, /* mUsePerTokenSfA */ 0 +, /* mUsePerTokenSfB */ 0 +, /* mUseShuffledMatrix */ 1 +, /* mUseTmaStore */ 1 +, /* mUseTwoTmaLoadWarps */ 1 +, /* mUseTwoMmaWarps */ 0 +, /* mUseUnrollLoop2xForMma */ 1 +, /* mValidM */ 256 +, /* mValidN */ 256 +, /* mValidK */ 512 +, /* mWorldSize */ 1 +, /* mActType */ gemmGatedAct::ActType(0) +, /* mClampBeforeAct */ 1 +, /* mBatchedM */ {} +, /* mBatchedN */ {} +, /* mBatchMode */ batchedGemm::BatchedGemmOptions::BatchMode(1) +, /* mBatchStrideInTokens */ -1 +, /* mFusedAct */ 1 +, /* mGridWaitForPrimaryRouting */ 1 +, /* mIsStaticBatch */ 0 +, /* mIsUniformNumTokensPerBatch */ 0 +, /* mNumBatches */ 128 +, /* mNumRegsPerThreadLoadA */ 0 +, /* mNumRegsPerThreadLoadB */ 0 +, /* mNumRegsPerThreadLoadSfA */ 0 +, /* mNumRegsPerThreadLoadSfB */ 0 +, /* mNumTokens */ 2 +, /* mNumWarpsLoadA */ 0 +, /* mNumWarpsLoadB */ 0 +, /* mNumWarpsLoadSfA */ 0 +, /* mNumWarpsLoadSfB */ 0 +, /* mRouteImpl */ batchedGemm::RouteImpl(2) +, /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} +, /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 + }, gemm::SmVersion::Sm100f}, +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len, 200216, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f", 640, "72cba0eebe2d931ddfcdd1419bb57f7dfad07cce74c665170a793a46046bb0e6", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +, /* mBiasDtype */ trtllm::gen::Dtype(1056776) +, /* mBiasType */ gemm::BiasType(0) +, /* mBlockK */ -1 +, /* mClcFastDrain */ 1 +, /* mClusterDimX */ 2 +, /* mClusterDimY */ 1 +, /* mClusterDimZ */ 1 +, /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) +, /* mDtypeAcc */ trtllm::gen::Dtype(1056776) +, /* mDtypeA */ trtllm::gen::Dtype(17826828) +, /* mDtypeB */ trtllm::gen::Dtype(17827853) +, /* mDtypeC */ trtllm::gen::Dtype(17827853) +, /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) +, /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) +, /* mEltwiseActType */ gemm::EltwiseActType(0) +, /* mEnablesEarlyExit */ 1 +, /* mEnablesDelayedEarlyExit */ 0 +, /* mEpilogueLdtmDps */ 16 +, /* mEpilogueLdtmBits */ 256 +, /* mEpilogueTileM */ 128 +, /* mEpilogueTileN */ 32 +, /* mFallbackClusterDimX */ 1 +, /* mFallbackClusterDimY */ 1 +, /* mFallbackClusterDimZ */ 1 +, /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) +, /* mFuseLoadSfTask */ 0 +, /* mFuseUtccpWithUtcmma */ 0 +, /* mGridTriggerSecondaryA */ 0 +, /* mGridTriggerSecondaryB */ 1 +, /* mGridWaitForPrimaryEarlyExit */ 1 +, /* mGridWaitForPrimaryA */ 0 +, /* mGridWaitForPrimaryB */ 1 +, /* mHoistLoadTaskInit */ 1 +, /* mHoistMmaTaskTryWaits */ 0 +, /* mK */ 512 +, /* mKernelTraits */ {} +, /* mLayoutA */ gemm::MatrixLayout(0) +, /* mLayoutB */ gemm::MatrixLayout(0) +, /* mM */ 256 +, /* mMmaK */ 32 +, /* mMmaKind */ trtllm::gen::MmaKind(5) +, /* mMmaM */ 256 +, /* mMmaN */ 32 +, /* mMmaTileK */ 256 +, /* mMockAllReduce */ 0 +, /* mN */ 256 +, /* mNumEpilogueWarps */ 4 +, /* mNumRegsCastAWarps */ 0 +, /* mNumRegsCopySfLdsSttm */ 0 +, /* mNumRegsCopySparsityInfo */ 0 +, /* mNumRegsPerThreadEpilogueWarp */ 0 +, /* mNumRegsPerThreadNonEpilogueWarp */ 0 +, /* mNumSlicesForSplitK */ 1 +, /* mNumStagesA */ 5 +, /* mNumStagesB */ 5 +, /* mNumStagesMma */ 1 +, /* mNumStagesMmaWithinWorkTile */ 1 +, /* mNumStagesMmaAcrossWorkTile */ 1 +, /* mNumStagesSmemSfA */ 5 +, /* mNumStagesSmemSfB */ 5 +, /* mNumStagesTmemSfA */ 5 +, /* mNumStagesTmemSfB */ 5 +, /* mNumStagesWorkId */ 3 +, /* mOutputDebugTensors */ 0 +, /* mPatchF2fp */ 0 +, /* mPerTokenSfDtype */ trtllm::gen::Dtype(1052672) +, /* mSchedHostTask */ gemm::SchedHostTask(0) +, /* mSfBlockSizeA */ 32 +, /* mSfBlockSizeB */ 32 +, /* mSfBlockSizeC */ 32 +, /* mSfLayoutA */ trtllm::gen::SfLayout(3) +, /* mSfLayoutB */ trtllm::gen::SfLayout(0) +, /* mSfLayoutC */ trtllm::gen::SfLayout(1) +, /* mSfReshapeFactor */ 1 +, /* mSparsityA */ trtllm::gen::Sparsity(0) +, /* mSplitK */ gemm::SplitK(0) +, /* mTileK */ 256 +, /* mTileM */ 128 +, /* mTileN */ 32 +, /* mTileScheduler */ gemm::TileScheduler(0) +, /* mTransposeMmaOutput */ 1 +, /* mUseCustomizedMma3xNvFp4 */ 0 +, /* mUseCustomMmaSchedule */ 1 +, /* mUseDeepSeekFp8 */ 0 +, /* mUseFlexibleClusterDims */ 0 +, /* mUseHoistTryWaitForCustomMmaSchedule */ 0 +, /* mUseMaxTmemOverlap */ 0 +, /* mUsePerTokenSfA */ 0 +, /* mUsePerTokenSfB */ 0 +, /* mUseShuffledMatrix */ 1 +, /* mUseTmaStore */ 1 +, /* mUseTwoTmaLoadWarps */ 1 +, /* mUseTwoMmaWarps */ 0 +, /* mUseUnrollLoop2xForMma */ 1 +, /* mValidM */ 256 +, /* mValidN */ 256 +, /* mValidK */ 512 +, /* mWorldSize */ 1 +, /* mActType */ gemmGatedAct::ActType(2) +, /* mClampBeforeAct */ 1 +, /* mBatchedM */ {} +, /* mBatchedN */ {} +, /* mBatchMode */ batchedGemm::BatchedGemmOptions::BatchMode(1) +, /* mBatchStrideInTokens */ -1 +, /* mFusedAct */ 1 +, /* mGridWaitForPrimaryRouting */ 1 +, /* mIsStaticBatch */ 0 +, /* mIsUniformNumTokensPerBatch */ 0 +, /* mNumBatches */ 128 +, /* mNumRegsPerThreadLoadA */ 0 +, /* mNumRegsPerThreadLoadB */ 0 +, /* mNumRegsPerThreadLoadSfA */ 0 +, /* mNumRegsPerThreadLoadSfB */ 0 +, /* mNumTokens */ 2 +, /* mNumWarpsLoadA */ 0 +, /* mNumWarpsLoadB */ 0 +, /* mNumWarpsLoadSfA */ 0 +, /* mNumWarpsLoadSfB */ 0 +, /* mRouteImpl */ batchedGemm::RouteImpl(2) +, /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} +, /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 + }, gemm::SmVersion::Sm100f}, +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 200216, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 640, "b1a3443eacc4496bd1757ee2a7da68a37068b3eb002ecfcea9a953c9f6ff2045", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +, /* mBiasDtype */ trtllm::gen::Dtype(1056776) +, /* mBiasType */ gemm::BiasType(1) +, /* mBlockK */ -1 +, /* mClcFastDrain */ 1 +, /* mClusterDimX */ 2 +, /* mClusterDimY */ 1 +, /* mClusterDimZ */ 1 +, /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) +, /* mDtypeAcc */ trtllm::gen::Dtype(1056776) +, /* mDtypeA */ trtllm::gen::Dtype(17826828) +, /* mDtypeB */ trtllm::gen::Dtype(17827853) +, /* mDtypeC */ trtllm::gen::Dtype(17827853) +, /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) +, /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) +, /* mEltwiseActType */ gemm::EltwiseActType(0) +, /* mEnablesEarlyExit */ 1 +, /* mEnablesDelayedEarlyExit */ 0 +, /* mEpilogueLdtmDps */ 16 +, /* mEpilogueLdtmBits */ 256 +, /* mEpilogueTileM */ 128 +, /* mEpilogueTileN */ 32 +, /* mFallbackClusterDimX */ 1 +, /* mFallbackClusterDimY */ 1 +, /* mFallbackClusterDimZ */ 1 +, /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) +, /* mFuseLoadSfTask */ 0 +, /* mFuseUtccpWithUtcmma */ 0 +, /* mGridTriggerSecondaryA */ 0 +, /* mGridTriggerSecondaryB */ 1 +, /* mGridWaitForPrimaryEarlyExit */ 1 +, /* mGridWaitForPrimaryA */ 0 +, /* mGridWaitForPrimaryB */ 1 +, /* mHoistLoadTaskInit */ 1 +, /* mHoistMmaTaskTryWaits */ 0 +, /* mK */ 512 +, /* mKernelTraits */ {} +, /* mLayoutA */ gemm::MatrixLayout(0) +, /* mLayoutB */ gemm::MatrixLayout(0) +, /* mM */ 256 +, /* mMmaK */ 32 +, /* mMmaKind */ trtllm::gen::MmaKind(5) +, /* mMmaM */ 256 +, /* mMmaN */ 32 +, /* mMmaTileK */ 256 +, /* mMockAllReduce */ 0 +, /* mN */ 256 +, /* mNumEpilogueWarps */ 4 +, /* mNumRegsCastAWarps */ 0 +, /* mNumRegsCopySfLdsSttm */ 0 +, /* mNumRegsCopySparsityInfo */ 0 +, /* mNumRegsPerThreadEpilogueWarp */ 0 +, /* mNumRegsPerThreadNonEpilogueWarp */ 0 +, /* mNumSlicesForSplitK */ 1 +, /* mNumStagesA */ 5 +, /* mNumStagesB */ 5 +, /* mNumStagesMma */ 1 +, /* mNumStagesMmaWithinWorkTile */ 1 +, /* mNumStagesMmaAcrossWorkTile */ 1 +, /* mNumStagesSmemSfA */ 5 +, /* mNumStagesSmemSfB */ 5 +, /* mNumStagesTmemSfA */ 5 +, /* mNumStagesTmemSfB */ 5 +, /* mNumStagesWorkId */ 3 +, /* mOutputDebugTensors */ 0 +, /* mPatchF2fp */ 0 +, /* mPerTokenSfDtype */ trtllm::gen::Dtype(1052672) +, /* mSchedHostTask */ gemm::SchedHostTask(0) +, /* mSfBlockSizeA */ 32 +, /* mSfBlockSizeB */ 32 +, /* mSfBlockSizeC */ 32 +, /* mSfLayoutA */ trtllm::gen::SfLayout(3) +, /* mSfLayoutB */ trtllm::gen::SfLayout(0) +, /* mSfLayoutC */ trtllm::gen::SfLayout(1) +, /* mSfReshapeFactor */ 1 +, /* mSparsityA */ trtllm::gen::Sparsity(0) +, /* mSplitK */ gemm::SplitK(0) +, /* mTileK */ 256 +, /* mTileM */ 128 +, /* mTileN */ 32 +, /* mTileScheduler */ gemm::TileScheduler(0) +, /* mTransposeMmaOutput */ 1 +, /* mUseCustomizedMma3xNvFp4 */ 0 +, /* mUseCustomMmaSchedule */ 1 +, /* mUseDeepSeekFp8 */ 0 +, /* mUseFlexibleClusterDims */ 0 +, /* mUseHoistTryWaitForCustomMmaSchedule */ 0 +, /* mUseMaxTmemOverlap */ 0 +, /* mUsePerTokenSfA */ 0 +, /* mUsePerTokenSfB */ 0 +, /* mUseShuffledMatrix */ 1 +, /* mUseTmaStore */ 1 +, /* mUseTwoTmaLoadWarps */ 1 +, /* mUseTwoMmaWarps */ 0 +, /* mUseUnrollLoop2xForMma */ 1 +, /* mValidM */ 256 +, /* mValidN */ 256 +, /* mValidK */ 512 +, /* mWorldSize */ 1 +, /* mActType */ gemmGatedAct::ActType(0) +, /* mClampBeforeAct */ 1 +, /* mBatchedM */ {} +, /* mBatchedN */ {} +, /* mBatchMode */ batchedGemm::BatchedGemmOptions::BatchMode(1) +, /* mBatchStrideInTokens */ -1 +, /* mFusedAct */ 1 +, /* mGridWaitForPrimaryRouting */ 1 +, /* mIsStaticBatch */ 0 +, /* mIsUniformNumTokensPerBatch */ 0 +, /* mNumBatches */ 128 +, /* mNumRegsPerThreadLoadA */ 0 +, /* mNumRegsPerThreadLoadB */ 0 +, /* mNumRegsPerThreadLoadSfA */ 0 +, /* mNumRegsPerThreadLoadSfB */ 0 +, /* mNumTokens */ 2 +, /* mNumWarpsLoadA */ 0 +, /* mNumWarpsLoadB */ 0 +, /* mNumWarpsLoadSfA */ 0 +, /* mNumWarpsLoadSfB */ 0 +, /* mRouteImpl */ batchedGemm::RouteImpl(2) +, /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} +, /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 + }, gemm::SmVersion::Sm100f}, +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x128_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW1_lsfbW1_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x128_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW1_lsfbW1_dynB_sm100f_cubin_len, 162968, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x128_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW1_lsfbW1_dynB_sm100f", 512, "c10896220d4932d9d0a6b88694203f537050af37b542c2ed2cf57093ac36d54b", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +, /* mBiasDtype */ trtllm::gen::Dtype(1056776) +, /* mBiasType */ gemm::BiasType(0) +, /* mBlockK */ -1 +, /* mClcFastDrain */ 1 +, /* mClusterDimX */ 2 +, /* mClusterDimY */ 1 +, /* mClusterDimZ */ 1 +, /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) +, /* mDtypeAcc */ trtllm::gen::Dtype(1056776) +, /* mDtypeA */ trtllm::gen::Dtype(17826828) +, /* mDtypeB */ trtllm::gen::Dtype(17827853) +, /* mDtypeC */ trtllm::gen::Dtype(17827853) +, /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) +, /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) +, /* mEltwiseActType */ gemm::EltwiseActType(0) +, /* mEnablesEarlyExit */ 1 +, /* mEnablesDelayedEarlyExit */ 0 +, /* mEpilogueLdtmDps */ 16 +, /* mEpilogueLdtmBits */ 256 +, /* mEpilogueTileM */ 128 +, /* mEpilogueTileN */ 64 +, /* mFallbackClusterDimX */ 1 +, /* mFallbackClusterDimY */ 1 +, /* mFallbackClusterDimZ */ 1 +, /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) +, /* mFuseLoadSfTask */ 0 +, /* mFuseUtccpWithUtcmma */ 0 +, /* mGridTriggerSecondaryA */ 0 +, /* mGridTriggerSecondaryB */ 1 +, /* mGridWaitForPrimaryEarlyExit */ 1 +, /* mGridWaitForPrimaryA */ 0 +, /* mGridWaitForPrimaryB */ 1 +, /* mHoistLoadTaskInit */ 1 +, /* mHoistMmaTaskTryWaits */ 0 +, /* mK */ 128 +, /* mKernelTraits */ {} +, /* mLayoutA */ gemm::MatrixLayout(0) +, /* mLayoutB */ gemm::MatrixLayout(0) +, /* mM */ 256 +, /* mMmaK */ 32 +, /* mMmaKind */ trtllm::gen::MmaKind(5) +, /* mMmaM */ 256 +, /* mMmaN */ 64 +, /* mMmaTileK */ 128 +, /* mMockAllReduce */ 0 +, /* mN */ 256 +, /* mNumEpilogueWarps */ 4 +, /* mNumRegsCastAWarps */ 0 +, /* mNumRegsCopySfLdsSttm */ 0 +, /* mNumRegsCopySparsityInfo */ 0 +, /* mNumRegsPerThreadEpilogueWarp */ 168 +, /* mNumRegsPerThreadNonEpilogueWarp */ 48 +, /* mNumSlicesForSplitK */ 1 +, /* mNumStagesA */ 7 +, /* mNumStagesB */ 7 +, /* mNumStagesMma */ 2 +, /* mNumStagesMmaWithinWorkTile */ 1 +, /* mNumStagesMmaAcrossWorkTile */ 2 +, /* mNumStagesSmemSfA */ 7 +, /* mNumStagesSmemSfB */ 7 +, /* mNumStagesTmemSfA */ 7 +, /* mNumStagesTmemSfB */ 7 +, /* mNumStagesWorkId */ 3 +, /* mOutputDebugTensors */ 0 +, /* mPatchF2fp */ 0 +, /* mPerTokenSfDtype */ trtllm::gen::Dtype(1052672) +, /* mSchedHostTask */ gemm::SchedHostTask(0) +, /* mSfBlockSizeA */ 32 +, /* mSfBlockSizeB */ 32 +, /* mSfBlockSizeC */ 32 +, /* mSfLayoutA */ trtllm::gen::SfLayout(3) +, /* mSfLayoutB */ trtllm::gen::SfLayout(0) +, /* mSfLayoutC */ trtllm::gen::SfLayout(1) +, /* mSfReshapeFactor */ 1 +, /* mSparsityA */ trtllm::gen::Sparsity(0) +, /* mSplitK */ gemm::SplitK(0) +, /* mTileK */ 128 +, /* mTileM */ 128 +, /* mTileN */ 64 +, /* mTileScheduler */ gemm::TileScheduler(1) +, /* mTransposeMmaOutput */ 1 +, /* mUseCustomizedMma3xNvFp4 */ 0 +, /* mUseCustomMmaSchedule */ 1 +, /* mUseDeepSeekFp8 */ 0 +, /* mUseFlexibleClusterDims */ 0 +, /* mUseHoistTryWaitForCustomMmaSchedule */ 0 +, /* mUseMaxTmemOverlap */ 0 +, /* mUsePerTokenSfA */ 0 +, /* mUsePerTokenSfB */ 0 +, /* mUseShuffledMatrix */ 1 +, /* mUseTmaStore */ 1 +, /* mUseTwoTmaLoadWarps */ 1 +, /* mUseTwoMmaWarps */ 0 +, /* mUseUnrollLoop2xForMma */ 0 +, /* mValidM */ 256 +, /* mValidN */ 256 +, /* mValidK */ 128 +, /* mWorldSize */ 1 +, /* mActType */ gemmGatedAct::ActType(2) +, /* mClampBeforeAct */ 1 +, /* mBatchedM */ {} +, /* mBatchedN */ {} +, /* mBatchMode */ batchedGemm::BatchedGemmOptions::BatchMode(1) +, /* mBatchStrideInTokens */ -1 +, /* mFusedAct */ 1 +, /* mGridWaitForPrimaryRouting */ 1 +, /* mIsStaticBatch */ 0 +, /* mIsUniformNumTokensPerBatch */ 0 +, /* mNumBatches */ 128 +, /* mNumRegsPerThreadLoadA */ 0 +, /* mNumRegsPerThreadLoadB */ 48 +, /* mNumRegsPerThreadLoadSfA */ 0 +, /* mNumRegsPerThreadLoadSfB */ 48 +, /* mNumTokens */ 2 +, /* mNumWarpsLoadA */ 0 +, /* mNumWarpsLoadB */ 1 +, /* mNumWarpsLoadSfA */ 0 +, /* mNumWarpsLoadSfB */ 1 +, /* mRouteImpl */ batchedGemm::RouteImpl(2) +, /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} +, /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 + }, gemm::SmVersion::Sm100f}, +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x128_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW1_lsfbW1_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x128_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW1_lsfbW1_dynB_sm100f_cubin_len, 163992, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x128_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW1_lsfbW1_dynB_sm100f", 512, "34b8f2bf422d8a04996c43ca98d2d34ee40492c0f412e99d7a1d5ef811139358", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +, /* mBiasDtype */ trtllm::gen::Dtype(1056776) +, /* mBiasType */ gemm::BiasType(1) +, /* mBlockK */ -1 +, /* mClcFastDrain */ 1 +, /* mClusterDimX */ 2 +, /* mClusterDimY */ 1 +, /* mClusterDimZ */ 1 +, /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) +, /* mDtypeAcc */ trtllm::gen::Dtype(1056776) +, /* mDtypeA */ trtllm::gen::Dtype(17826828) +, /* mDtypeB */ trtllm::gen::Dtype(17827853) +, /* mDtypeC */ trtllm::gen::Dtype(17827853) +, /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) +, /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) +, /* mEltwiseActType */ gemm::EltwiseActType(0) +, /* mEnablesEarlyExit */ 1 +, /* mEnablesDelayedEarlyExit */ 0 +, /* mEpilogueLdtmDps */ 16 +, /* mEpilogueLdtmBits */ 256 +, /* mEpilogueTileM */ 128 +, /* mEpilogueTileN */ 64 +, /* mFallbackClusterDimX */ 1 +, /* mFallbackClusterDimY */ 1 +, /* mFallbackClusterDimZ */ 1 +, /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) +, /* mFuseLoadSfTask */ 0 +, /* mFuseUtccpWithUtcmma */ 0 +, /* mGridTriggerSecondaryA */ 0 +, /* mGridTriggerSecondaryB */ 1 +, /* mGridWaitForPrimaryEarlyExit */ 1 +, /* mGridWaitForPrimaryA */ 0 +, /* mGridWaitForPrimaryB */ 1 +, /* mHoistLoadTaskInit */ 1 +, /* mHoistMmaTaskTryWaits */ 0 +, /* mK */ 128 +, /* mKernelTraits */ {} +, /* mLayoutA */ gemm::MatrixLayout(0) +, /* mLayoutB */ gemm::MatrixLayout(0) +, /* mM */ 256 +, /* mMmaK */ 32 +, /* mMmaKind */ trtllm::gen::MmaKind(5) +, /* mMmaM */ 256 +, /* mMmaN */ 64 +, /* mMmaTileK */ 128 +, /* mMockAllReduce */ 0 +, /* mN */ 256 +, /* mNumEpilogueWarps */ 4 +, /* mNumRegsCastAWarps */ 0 +, /* mNumRegsCopySfLdsSttm */ 0 +, /* mNumRegsCopySparsityInfo */ 0 +, /* mNumRegsPerThreadEpilogueWarp */ 168 +, /* mNumRegsPerThreadNonEpilogueWarp */ 48 +, /* mNumSlicesForSplitK */ 1 +, /* mNumStagesA */ 7 +, /* mNumStagesB */ 7 +, /* mNumStagesMma */ 2 +, /* mNumStagesMmaWithinWorkTile */ 1 +, /* mNumStagesMmaAcrossWorkTile */ 2 +, /* mNumStagesSmemSfA */ 7 +, /* mNumStagesSmemSfB */ 7 +, /* mNumStagesTmemSfA */ 7 +, /* mNumStagesTmemSfB */ 7 +, /* mNumStagesWorkId */ 3 +, /* mOutputDebugTensors */ 0 +, /* mPatchF2fp */ 0 +, /* mPerTokenSfDtype */ trtllm::gen::Dtype(1052672) +, /* mSchedHostTask */ gemm::SchedHostTask(0) +, /* mSfBlockSizeA */ 32 +, /* mSfBlockSizeB */ 32 +, /* mSfBlockSizeC */ 32 +, /* mSfLayoutA */ trtllm::gen::SfLayout(3) +, /* mSfLayoutB */ trtllm::gen::SfLayout(0) +, /* mSfLayoutC */ trtllm::gen::SfLayout(1) +, /* mSfReshapeFactor */ 1 +, /* mSparsityA */ trtllm::gen::Sparsity(0) +, /* mSplitK */ gemm::SplitK(0) +, /* mTileK */ 128 +, /* mTileM */ 128 +, /* mTileN */ 64 +, /* mTileScheduler */ gemm::TileScheduler(1) +, /* mTransposeMmaOutput */ 1 +, /* mUseCustomizedMma3xNvFp4 */ 0 +, /* mUseCustomMmaSchedule */ 1 +, /* mUseDeepSeekFp8 */ 0 +, /* mUseFlexibleClusterDims */ 0 +, /* mUseHoistTryWaitForCustomMmaSchedule */ 0 +, /* mUseMaxTmemOverlap */ 0 +, /* mUsePerTokenSfA */ 0 +, /* mUsePerTokenSfB */ 0 +, /* mUseShuffledMatrix */ 1 +, /* mUseTmaStore */ 1 +, /* mUseTwoTmaLoadWarps */ 1 +, /* mUseTwoMmaWarps */ 0 +, /* mUseUnrollLoop2xForMma */ 0 +, /* mValidM */ 256 +, /* mValidN */ 256 +, /* mValidK */ 128 +, /* mWorldSize */ 1 +, /* mActType */ gemmGatedAct::ActType(0) +, /* mClampBeforeAct */ 1 +, /* mBatchedM */ {} +, /* mBatchedN */ {} +, /* mBatchMode */ batchedGemm::BatchedGemmOptions::BatchMode(1) +, /* mBatchStrideInTokens */ -1 +, /* mFusedAct */ 1 +, /* mGridWaitForPrimaryRouting */ 1 +, /* mIsStaticBatch */ 0 +, /* mIsUniformNumTokensPerBatch */ 0 +, /* mNumBatches */ 128 +, /* mNumRegsPerThreadLoadA */ 0 +, /* mNumRegsPerThreadLoadB */ 48 +, /* mNumRegsPerThreadLoadSfA */ 0 +, /* mNumRegsPerThreadLoadSfB */ 48 +, /* mNumTokens */ 2 +, /* mNumWarpsLoadA */ 0 +, /* mNumWarpsLoadB */ 1 +, /* mNumWarpsLoadSfA */ 0 +, /* mNumWarpsLoadSfB */ 1 +, /* mRouteImpl */ batchedGemm::RouteImpl(2) +, /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} +, /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 + }, gemm::SmVersion::Sm100f}, +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x128u2_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW1_lsfbW1_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x128u2_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW1_lsfbW1_dynB_sm100f_cubin_len, 162968, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x128u2_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW1_lsfbW1_dynB_sm100f", 512, "9a1046b197f953ef89caa22a96791b12f7fbf3fd9e9e88505c8fa9f1f17207ca", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +, /* mBiasDtype */ trtllm::gen::Dtype(1056776) +, /* mBiasType */ gemm::BiasType(0) +, /* mBlockK */ -1 +, /* mClcFastDrain */ 1 +, /* mClusterDimX */ 2 +, /* mClusterDimY */ 1 +, /* mClusterDimZ */ 1 +, /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) +, /* mDtypeAcc */ trtllm::gen::Dtype(1056776) +, /* mDtypeA */ trtllm::gen::Dtype(17826828) +, /* mDtypeB */ trtllm::gen::Dtype(17827853) +, /* mDtypeC */ trtllm::gen::Dtype(17827853) +, /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) +, /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) +, /* mEltwiseActType */ gemm::EltwiseActType(0) +, /* mEnablesEarlyExit */ 1 +, /* mEnablesDelayedEarlyExit */ 0 +, /* mEpilogueLdtmDps */ 16 +, /* mEpilogueLdtmBits */ 256 +, /* mEpilogueTileM */ 128 +, /* mEpilogueTileN */ 64 +, /* mFallbackClusterDimX */ 1 +, /* mFallbackClusterDimY */ 1 +, /* mFallbackClusterDimZ */ 1 +, /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) +, /* mFuseLoadSfTask */ 0 +, /* mFuseUtccpWithUtcmma */ 0 +, /* mGridTriggerSecondaryA */ 0 +, /* mGridTriggerSecondaryB */ 1 +, /* mGridWaitForPrimaryEarlyExit */ 1 +, /* mGridWaitForPrimaryA */ 0 +, /* mGridWaitForPrimaryB */ 1 +, /* mHoistLoadTaskInit */ 1 +, /* mHoistMmaTaskTryWaits */ 0 +, /* mK */ 256 +, /* mKernelTraits */ {} +, /* mLayoutA */ gemm::MatrixLayout(0) +, /* mLayoutB */ gemm::MatrixLayout(0) +, /* mM */ 256 +, /* mMmaK */ 32 +, /* mMmaKind */ trtllm::gen::MmaKind(5) +, /* mMmaM */ 256 +, /* mMmaN */ 64 +, /* mMmaTileK */ 128 +, /* mMockAllReduce */ 0 +, /* mN */ 256 +, /* mNumEpilogueWarps */ 4 +, /* mNumRegsCastAWarps */ 0 +, /* mNumRegsCopySfLdsSttm */ 0 +, /* mNumRegsCopySparsityInfo */ 0 +, /* mNumRegsPerThreadEpilogueWarp */ 168 +, /* mNumRegsPerThreadNonEpilogueWarp */ 48 +, /* mNumSlicesForSplitK */ 1 +, /* mNumStagesA */ 7 +, /* mNumStagesB */ 7 +, /* mNumStagesMma */ 2 +, /* mNumStagesMmaWithinWorkTile */ 1 +, /* mNumStagesMmaAcrossWorkTile */ 2 +, /* mNumStagesSmemSfA */ 7 +, /* mNumStagesSmemSfB */ 7 +, /* mNumStagesTmemSfA */ 7 +, /* mNumStagesTmemSfB */ 7 +, /* mNumStagesWorkId */ 3 +, /* mOutputDebugTensors */ 0 +, /* mPatchF2fp */ 0 +, /* mPerTokenSfDtype */ trtllm::gen::Dtype(1052672) +, /* mSchedHostTask */ gemm::SchedHostTask(0) +, /* mSfBlockSizeA */ 32 +, /* mSfBlockSizeB */ 32 +, /* mSfBlockSizeC */ 32 +, /* mSfLayoutA */ trtllm::gen::SfLayout(3) +, /* mSfLayoutB */ trtllm::gen::SfLayout(0) +, /* mSfLayoutC */ trtllm::gen::SfLayout(1) +, /* mSfReshapeFactor */ 1 +, /* mSparsityA */ trtllm::gen::Sparsity(0) +, /* mSplitK */ gemm::SplitK(0) +, /* mTileK */ 128 +, /* mTileM */ 128 +, /* mTileN */ 64 +, /* mTileScheduler */ gemm::TileScheduler(1) +, /* mTransposeMmaOutput */ 1 +, /* mUseCustomizedMma3xNvFp4 */ 0 +, /* mUseCustomMmaSchedule */ 1 +, /* mUseDeepSeekFp8 */ 0 +, /* mUseFlexibleClusterDims */ 0 +, /* mUseHoistTryWaitForCustomMmaSchedule */ 0 +, /* mUseMaxTmemOverlap */ 0 +, /* mUsePerTokenSfA */ 0 +, /* mUsePerTokenSfB */ 0 +, /* mUseShuffledMatrix */ 1 +, /* mUseTmaStore */ 1 +, /* mUseTwoTmaLoadWarps */ 1 +, /* mUseTwoMmaWarps */ 0 +, /* mUseUnrollLoop2xForMma */ 1 +, /* mValidM */ 256 +, /* mValidN */ 256 +, /* mValidK */ 256 +, /* mWorldSize */ 1 +, /* mActType */ gemmGatedAct::ActType(2) +, /* mClampBeforeAct */ 1 +, /* mBatchedM */ {} +, /* mBatchedN */ {} +, /* mBatchMode */ batchedGemm::BatchedGemmOptions::BatchMode(1) +, /* mBatchStrideInTokens */ -1 +, /* mFusedAct */ 1 +, /* mGridWaitForPrimaryRouting */ 1 +, /* mIsStaticBatch */ 0 +, /* mIsUniformNumTokensPerBatch */ 0 +, /* mNumBatches */ 128 +, /* mNumRegsPerThreadLoadA */ 0 +, /* mNumRegsPerThreadLoadB */ 48 +, /* mNumRegsPerThreadLoadSfA */ 0 +, /* mNumRegsPerThreadLoadSfB */ 48 +, /* mNumTokens */ 2 +, /* mNumWarpsLoadA */ 0 +, /* mNumWarpsLoadB */ 1 +, /* mNumWarpsLoadSfA */ 0 +, /* mNumWarpsLoadSfB */ 1 +, /* mRouteImpl */ batchedGemm::RouteImpl(2) +, /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} +, /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 + }, gemm::SmVersion::Sm100f}, +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x128u2_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW1_lsfbW1_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x128u2_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW1_lsfbW1_dynB_sm100f_cubin_len, 163992, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x128u2_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW1_lsfbW1_dynB_sm100f", 512, "bb64c69a2995d59f55d5a4e9ff876c6afcb99860e94743454e14a766db5752d3", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +, /* mBiasDtype */ trtllm::gen::Dtype(1056776) +, /* mBiasType */ gemm::BiasType(1) +, /* mBlockK */ -1 +, /* mClcFastDrain */ 1 +, /* mClusterDimX */ 2 +, /* mClusterDimY */ 1 +, /* mClusterDimZ */ 1 +, /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) +, /* mDtypeAcc */ trtllm::gen::Dtype(1056776) +, /* mDtypeA */ trtllm::gen::Dtype(17826828) +, /* mDtypeB */ trtllm::gen::Dtype(17827853) +, /* mDtypeC */ trtllm::gen::Dtype(17827853) +, /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) +, /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) +, /* mEltwiseActType */ gemm::EltwiseActType(0) +, /* mEnablesEarlyExit */ 1 +, /* mEnablesDelayedEarlyExit */ 0 +, /* mEpilogueLdtmDps */ 16 +, /* mEpilogueLdtmBits */ 256 +, /* mEpilogueTileM */ 128 +, /* mEpilogueTileN */ 64 +, /* mFallbackClusterDimX */ 1 +, /* mFallbackClusterDimY */ 1 +, /* mFallbackClusterDimZ */ 1 +, /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) +, /* mFuseLoadSfTask */ 0 +, /* mFuseUtccpWithUtcmma */ 0 +, /* mGridTriggerSecondaryA */ 0 +, /* mGridTriggerSecondaryB */ 1 +, /* mGridWaitForPrimaryEarlyExit */ 1 +, /* mGridWaitForPrimaryA */ 0 +, /* mGridWaitForPrimaryB */ 1 +, /* mHoistLoadTaskInit */ 1 +, /* mHoistMmaTaskTryWaits */ 0 +, /* mK */ 256 +, /* mKernelTraits */ {} +, /* mLayoutA */ gemm::MatrixLayout(0) +, /* mLayoutB */ gemm::MatrixLayout(0) +, /* mM */ 256 +, /* mMmaK */ 32 +, /* mMmaKind */ trtllm::gen::MmaKind(5) +, /* mMmaM */ 256 +, /* mMmaN */ 64 +, /* mMmaTileK */ 128 +, /* mMockAllReduce */ 0 +, /* mN */ 256 +, /* mNumEpilogueWarps */ 4 +, /* mNumRegsCastAWarps */ 0 +, /* mNumRegsCopySfLdsSttm */ 0 +, /* mNumRegsCopySparsityInfo */ 0 +, /* mNumRegsPerThreadEpilogueWarp */ 168 +, /* mNumRegsPerThreadNonEpilogueWarp */ 48 +, /* mNumSlicesForSplitK */ 1 +, /* mNumStagesA */ 7 +, /* mNumStagesB */ 7 +, /* mNumStagesMma */ 2 +, /* mNumStagesMmaWithinWorkTile */ 1 +, /* mNumStagesMmaAcrossWorkTile */ 2 +, /* mNumStagesSmemSfA */ 7 +, /* mNumStagesSmemSfB */ 7 +, /* mNumStagesTmemSfA */ 7 +, /* mNumStagesTmemSfB */ 7 +, /* mNumStagesWorkId */ 3 +, /* mOutputDebugTensors */ 0 +, /* mPatchF2fp */ 0 +, /* mPerTokenSfDtype */ trtllm::gen::Dtype(1052672) +, /* mSchedHostTask */ gemm::SchedHostTask(0) +, /* mSfBlockSizeA */ 32 +, /* mSfBlockSizeB */ 32 +, /* mSfBlockSizeC */ 32 +, /* mSfLayoutA */ trtllm::gen::SfLayout(3) +, /* mSfLayoutB */ trtllm::gen::SfLayout(0) +, /* mSfLayoutC */ trtllm::gen::SfLayout(1) +, /* mSfReshapeFactor */ 1 +, /* mSparsityA */ trtllm::gen::Sparsity(0) +, /* mSplitK */ gemm::SplitK(0) +, /* mTileK */ 128 +, /* mTileM */ 128 +, /* mTileN */ 64 +, /* mTileScheduler */ gemm::TileScheduler(1) +, /* mTransposeMmaOutput */ 1 +, /* mUseCustomizedMma3xNvFp4 */ 0 +, /* mUseCustomMmaSchedule */ 1 +, /* mUseDeepSeekFp8 */ 0 +, /* mUseFlexibleClusterDims */ 0 +, /* mUseHoistTryWaitForCustomMmaSchedule */ 0 +, /* mUseMaxTmemOverlap */ 0 +, /* mUsePerTokenSfA */ 0 +, /* mUsePerTokenSfB */ 0 +, /* mUseShuffledMatrix */ 1 +, /* mUseTmaStore */ 1 +, /* mUseTwoTmaLoadWarps */ 1 +, /* mUseTwoMmaWarps */ 0 +, /* mUseUnrollLoop2xForMma */ 1 +, /* mValidM */ 256 +, /* mValidN */ 256 +, /* mValidK */ 256 +, /* mWorldSize */ 1 +, /* mActType */ gemmGatedAct::ActType(0) +, /* mClampBeforeAct */ 1 +, /* mBatchedM */ {} +, /* mBatchedN */ {} +, /* mBatchMode */ batchedGemm::BatchedGemmOptions::BatchMode(1) +, /* mBatchStrideInTokens */ -1 +, /* mFusedAct */ 1 +, /* mGridWaitForPrimaryRouting */ 1 +, /* mIsStaticBatch */ 0 +, /* mIsUniformNumTokensPerBatch */ 0 +, /* mNumBatches */ 128 +, /* mNumRegsPerThreadLoadA */ 0 +, /* mNumRegsPerThreadLoadB */ 48 +, /* mNumRegsPerThreadLoadSfA */ 0 +, /* mNumRegsPerThreadLoadSfB */ 48 +, /* mNumTokens */ 2 +, /* mNumWarpsLoadA */ 0 +, /* mNumWarpsLoadB */ 1 +, /* mNumWarpsLoadSfA */ 0 +, /* mNumWarpsLoadSfB */ 1 +, /* mRouteImpl */ batchedGemm::RouteImpl(2) +, /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} +, /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 + }, gemm::SmVersion::Sm100f}, +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x256_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW1_lsfbW1_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x256_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW1_lsfbW1_dynB_sm100f_cubin_len, 182936, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x256_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW1_lsfbW1_dynB_sm100f", 512, "4cb6e06e25c2958b33903d82eb25de5951b2ec1f9065e976fbff6e76e995096c", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +, /* mBiasDtype */ trtllm::gen::Dtype(1056776) +, /* mBiasType */ gemm::BiasType(0) +, /* mBlockK */ -1 +, /* mClcFastDrain */ 1 +, /* mClusterDimX */ 2 +, /* mClusterDimY */ 1 +, /* mClusterDimZ */ 1 +, /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) +, /* mDtypeAcc */ trtllm::gen::Dtype(1056776) +, /* mDtypeA */ trtllm::gen::Dtype(17826828) +, /* mDtypeB */ trtllm::gen::Dtype(17827853) +, /* mDtypeC */ trtllm::gen::Dtype(17827853) +, /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) +, /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) +, /* mEltwiseActType */ gemm::EltwiseActType(0) +, /* mEnablesEarlyExit */ 1 +, /* mEnablesDelayedEarlyExit */ 0 +, /* mEpilogueLdtmDps */ 16 +, /* mEpilogueLdtmBits */ 256 +, /* mEpilogueTileM */ 128 +, /* mEpilogueTileN */ 64 +, /* mFallbackClusterDimX */ 1 +, /* mFallbackClusterDimY */ 1 +, /* mFallbackClusterDimZ */ 1 +, /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) +, /* mFuseLoadSfTask */ 0 +, /* mFuseUtccpWithUtcmma */ 0 +, /* mGridTriggerSecondaryA */ 0 +, /* mGridTriggerSecondaryB */ 1 +, /* mGridWaitForPrimaryEarlyExit */ 1 +, /* mGridWaitForPrimaryA */ 0 +, /* mGridWaitForPrimaryB */ 1 +, /* mHoistLoadTaskInit */ 1 +, /* mHoistMmaTaskTryWaits */ 0 +, /* mK */ 256 +, /* mKernelTraits */ {} +, /* mLayoutA */ gemm::MatrixLayout(0) +, /* mLayoutB */ gemm::MatrixLayout(0) +, /* mM */ 256 +, /* mMmaK */ 32 +, /* mMmaKind */ trtllm::gen::MmaKind(5) +, /* mMmaM */ 256 +, /* mMmaN */ 64 +, /* mMmaTileK */ 256 +, /* mMockAllReduce */ 0 +, /* mN */ 256 +, /* mNumEpilogueWarps */ 4 +, /* mNumRegsCastAWarps */ 0 +, /* mNumRegsCopySfLdsSttm */ 0 +, /* mNumRegsCopySparsityInfo */ 0 +, /* mNumRegsPerThreadEpilogueWarp */ 168 +, /* mNumRegsPerThreadNonEpilogueWarp */ 48 +, /* mNumSlicesForSplitK */ 1 +, /* mNumStagesA */ 4 +, /* mNumStagesB */ 4 +, /* mNumStagesMma */ 2 +, /* mNumStagesMmaWithinWorkTile */ 1 +, /* mNumStagesMmaAcrossWorkTile */ 2 +, /* mNumStagesSmemSfA */ 4 +, /* mNumStagesSmemSfB */ 4 +, /* mNumStagesTmemSfA */ 4 +, /* mNumStagesTmemSfB */ 4 +, /* mNumStagesWorkId */ 3 +, /* mOutputDebugTensors */ 0 +, /* mPatchF2fp */ 0 +, /* mPerTokenSfDtype */ trtllm::gen::Dtype(1052672) +, /* mSchedHostTask */ gemm::SchedHostTask(0) +, /* mSfBlockSizeA */ 32 +, /* mSfBlockSizeB */ 32 +, /* mSfBlockSizeC */ 32 +, /* mSfLayoutA */ trtllm::gen::SfLayout(3) +, /* mSfLayoutB */ trtllm::gen::SfLayout(0) +, /* mSfLayoutC */ trtllm::gen::SfLayout(1) +, /* mSfReshapeFactor */ 1 +, /* mSparsityA */ trtllm::gen::Sparsity(0) +, /* mSplitK */ gemm::SplitK(0) +, /* mTileK */ 256 +, /* mTileM */ 128 +, /* mTileN */ 64 +, /* mTileScheduler */ gemm::TileScheduler(1) +, /* mTransposeMmaOutput */ 1 +, /* mUseCustomizedMma3xNvFp4 */ 0 +, /* mUseCustomMmaSchedule */ 1 +, /* mUseDeepSeekFp8 */ 0 +, /* mUseFlexibleClusterDims */ 0 +, /* mUseHoistTryWaitForCustomMmaSchedule */ 0 +, /* mUseMaxTmemOverlap */ 0 +, /* mUsePerTokenSfA */ 0 +, /* mUsePerTokenSfB */ 0 +, /* mUseShuffledMatrix */ 1 +, /* mUseTmaStore */ 1 +, /* mUseTwoTmaLoadWarps */ 1 +, /* mUseTwoMmaWarps */ 0 +, /* mUseUnrollLoop2xForMma */ 0 +, /* mValidM */ 256 +, /* mValidN */ 256 +, /* mValidK */ 256 +, /* mWorldSize */ 1 +, /* mActType */ gemmGatedAct::ActType(2) +, /* mClampBeforeAct */ 1 +, /* mBatchedM */ {} +, /* mBatchedN */ {} +, /* mBatchMode */ batchedGemm::BatchedGemmOptions::BatchMode(1) +, /* mBatchStrideInTokens */ -1 +, /* mFusedAct */ 1 +, /* mGridWaitForPrimaryRouting */ 1 +, /* mIsStaticBatch */ 0 +, /* mIsUniformNumTokensPerBatch */ 0 +, /* mNumBatches */ 128 +, /* mNumRegsPerThreadLoadA */ 0 +, /* mNumRegsPerThreadLoadB */ 48 +, /* mNumRegsPerThreadLoadSfA */ 0 +, /* mNumRegsPerThreadLoadSfB */ 48 +, /* mNumTokens */ 2 +, /* mNumWarpsLoadA */ 0 +, /* mNumWarpsLoadB */ 1 +, /* mNumWarpsLoadSfA */ 0 +, /* mNumWarpsLoadSfB */ 1 +, /* mRouteImpl */ batchedGemm::RouteImpl(2) +, /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} +, /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 + }, gemm::SmVersion::Sm100f}, +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x256_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW1_lsfbW1_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x256_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW1_lsfbW1_dynB_sm100f_cubin_len, 183960, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x256_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW1_lsfbW1_dynB_sm100f", 512, "bd21ef8a85152bffc3bf9d532431d8bd7c43e7e06f0aa73206b4c4456a8f7e55", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +, /* mBiasDtype */ trtllm::gen::Dtype(1056776) +, /* mBiasType */ gemm::BiasType(1) +, /* mBlockK */ -1 +, /* mClcFastDrain */ 1 +, /* mClusterDimX */ 2 +, /* mClusterDimY */ 1 +, /* mClusterDimZ */ 1 +, /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) +, /* mDtypeAcc */ trtllm::gen::Dtype(1056776) +, /* mDtypeA */ trtllm::gen::Dtype(17826828) +, /* mDtypeB */ trtllm::gen::Dtype(17827853) +, /* mDtypeC */ trtllm::gen::Dtype(17827853) +, /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) +, /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) +, /* mEltwiseActType */ gemm::EltwiseActType(0) +, /* mEnablesEarlyExit */ 1 +, /* mEnablesDelayedEarlyExit */ 0 +, /* mEpilogueLdtmDps */ 16 +, /* mEpilogueLdtmBits */ 256 +, /* mEpilogueTileM */ 128 +, /* mEpilogueTileN */ 64 +, /* mFallbackClusterDimX */ 1 +, /* mFallbackClusterDimY */ 1 +, /* mFallbackClusterDimZ */ 1 +, /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) +, /* mFuseLoadSfTask */ 0 +, /* mFuseUtccpWithUtcmma */ 0 +, /* mGridTriggerSecondaryA */ 0 +, /* mGridTriggerSecondaryB */ 1 +, /* mGridWaitForPrimaryEarlyExit */ 1 +, /* mGridWaitForPrimaryA */ 0 +, /* mGridWaitForPrimaryB */ 1 +, /* mHoistLoadTaskInit */ 1 +, /* mHoistMmaTaskTryWaits */ 0 +, /* mK */ 256 +, /* mKernelTraits */ {} +, /* mLayoutA */ gemm::MatrixLayout(0) +, /* mLayoutB */ gemm::MatrixLayout(0) +, /* mM */ 256 +, /* mMmaK */ 32 +, /* mMmaKind */ trtllm::gen::MmaKind(5) +, /* mMmaM */ 256 +, /* mMmaN */ 64 +, /* mMmaTileK */ 256 +, /* mMockAllReduce */ 0 +, /* mN */ 256 +, /* mNumEpilogueWarps */ 4 +, /* mNumRegsCastAWarps */ 0 +, /* mNumRegsCopySfLdsSttm */ 0 +, /* mNumRegsCopySparsityInfo */ 0 +, /* mNumRegsPerThreadEpilogueWarp */ 168 +, /* mNumRegsPerThreadNonEpilogueWarp */ 48 +, /* mNumSlicesForSplitK */ 1 +, /* mNumStagesA */ 4 +, /* mNumStagesB */ 4 +, /* mNumStagesMma */ 2 +, /* mNumStagesMmaWithinWorkTile */ 1 +, /* mNumStagesMmaAcrossWorkTile */ 2 +, /* mNumStagesSmemSfA */ 4 +, /* mNumStagesSmemSfB */ 4 +, /* mNumStagesTmemSfA */ 4 +, /* mNumStagesTmemSfB */ 4 +, /* mNumStagesWorkId */ 3 +, /* mOutputDebugTensors */ 0 +, /* mPatchF2fp */ 0 +, /* mPerTokenSfDtype */ trtllm::gen::Dtype(1052672) +, /* mSchedHostTask */ gemm::SchedHostTask(0) +, /* mSfBlockSizeA */ 32 +, /* mSfBlockSizeB */ 32 +, /* mSfBlockSizeC */ 32 +, /* mSfLayoutA */ trtllm::gen::SfLayout(3) +, /* mSfLayoutB */ trtllm::gen::SfLayout(0) +, /* mSfLayoutC */ trtllm::gen::SfLayout(1) +, /* mSfReshapeFactor */ 1 +, /* mSparsityA */ trtllm::gen::Sparsity(0) +, /* mSplitK */ gemm::SplitK(0) +, /* mTileK */ 256 +, /* mTileM */ 128 +, /* mTileN */ 64 +, /* mTileScheduler */ gemm::TileScheduler(1) +, /* mTransposeMmaOutput */ 1 +, /* mUseCustomizedMma3xNvFp4 */ 0 +, /* mUseCustomMmaSchedule */ 1 +, /* mUseDeepSeekFp8 */ 0 +, /* mUseFlexibleClusterDims */ 0 +, /* mUseHoistTryWaitForCustomMmaSchedule */ 0 +, /* mUseMaxTmemOverlap */ 0 +, /* mUsePerTokenSfA */ 0 +, /* mUsePerTokenSfB */ 0 +, /* mUseShuffledMatrix */ 1 +, /* mUseTmaStore */ 1 +, /* mUseTwoTmaLoadWarps */ 1 +, /* mUseTwoMmaWarps */ 0 +, /* mUseUnrollLoop2xForMma */ 0 +, /* mValidM */ 256 +, /* mValidN */ 256 +, /* mValidK */ 256 +, /* mWorldSize */ 1 +, /* mActType */ gemmGatedAct::ActType(0) +, /* mClampBeforeAct */ 1 +, /* mBatchedM */ {} +, /* mBatchedN */ {} +, /* mBatchMode */ batchedGemm::BatchedGemmOptions::BatchMode(1) +, /* mBatchStrideInTokens */ -1 +, /* mFusedAct */ 1 +, /* mGridWaitForPrimaryRouting */ 1 +, /* mIsStaticBatch */ 0 +, /* mIsUniformNumTokensPerBatch */ 0 +, /* mNumBatches */ 128 +, /* mNumRegsPerThreadLoadA */ 0 +, /* mNumRegsPerThreadLoadB */ 48 +, /* mNumRegsPerThreadLoadSfA */ 0 +, /* mNumRegsPerThreadLoadSfB */ 48 +, /* mNumTokens */ 2 +, /* mNumWarpsLoadA */ 0 +, /* mNumWarpsLoadB */ 1 +, /* mNumWarpsLoadSfA */ 0 +, /* mNumWarpsLoadSfB */ 1 +, /* mRouteImpl */ batchedGemm::RouteImpl(2) +, /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} +, /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 + }, gemm::SmVersion::Sm100f}, +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x256u2_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW1_lsfbW1_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x256u2_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW1_lsfbW1_dynB_sm100f_cubin_len, 182936, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x256u2_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW1_lsfbW1_dynB_sm100f", 512, "8d7c80646237114d2a5ca5c8dfb032f3da8038aa8fbadf703c591ffd505eae50", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +, /* mBiasDtype */ trtllm::gen::Dtype(1056776) +, /* mBiasType */ gemm::BiasType(0) +, /* mBlockK */ -1 +, /* mClcFastDrain */ 1 +, /* mClusterDimX */ 2 +, /* mClusterDimY */ 1 +, /* mClusterDimZ */ 1 +, /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) +, /* mDtypeAcc */ trtllm::gen::Dtype(1056776) +, /* mDtypeA */ trtllm::gen::Dtype(17826828) +, /* mDtypeB */ trtllm::gen::Dtype(17827853) +, /* mDtypeC */ trtllm::gen::Dtype(17827853) +, /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) +, /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) +, /* mEltwiseActType */ gemm::EltwiseActType(0) +, /* mEnablesEarlyExit */ 1 +, /* mEnablesDelayedEarlyExit */ 0 +, /* mEpilogueLdtmDps */ 16 +, /* mEpilogueLdtmBits */ 256 +, /* mEpilogueTileM */ 128 +, /* mEpilogueTileN */ 64 +, /* mFallbackClusterDimX */ 1 +, /* mFallbackClusterDimY */ 1 +, /* mFallbackClusterDimZ */ 1 +, /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) +, /* mFuseLoadSfTask */ 0 +, /* mFuseUtccpWithUtcmma */ 0 +, /* mGridTriggerSecondaryA */ 0 +, /* mGridTriggerSecondaryB */ 1 +, /* mGridWaitForPrimaryEarlyExit */ 1 +, /* mGridWaitForPrimaryA */ 0 +, /* mGridWaitForPrimaryB */ 1 +, /* mHoistLoadTaskInit */ 1 +, /* mHoistMmaTaskTryWaits */ 0 +, /* mK */ 512 +, /* mKernelTraits */ {} +, /* mLayoutA */ gemm::MatrixLayout(0) +, /* mLayoutB */ gemm::MatrixLayout(0) +, /* mM */ 256 +, /* mMmaK */ 32 +, /* mMmaKind */ trtllm::gen::MmaKind(5) +, /* mMmaM */ 256 +, /* mMmaN */ 64 +, /* mMmaTileK */ 256 +, /* mMockAllReduce */ 0 +, /* mN */ 256 +, /* mNumEpilogueWarps */ 4 +, /* mNumRegsCastAWarps */ 0 +, /* mNumRegsCopySfLdsSttm */ 0 +, /* mNumRegsCopySparsityInfo */ 0 +, /* mNumRegsPerThreadEpilogueWarp */ 168 +, /* mNumRegsPerThreadNonEpilogueWarp */ 48 +, /* mNumSlicesForSplitK */ 1 +, /* mNumStagesA */ 4 +, /* mNumStagesB */ 4 +, /* mNumStagesMma */ 2 +, /* mNumStagesMmaWithinWorkTile */ 1 +, /* mNumStagesMmaAcrossWorkTile */ 2 +, /* mNumStagesSmemSfA */ 4 +, /* mNumStagesSmemSfB */ 4 +, /* mNumStagesTmemSfA */ 4 +, /* mNumStagesTmemSfB */ 4 +, /* mNumStagesWorkId */ 3 +, /* mOutputDebugTensors */ 0 +, /* mPatchF2fp */ 0 +, /* mPerTokenSfDtype */ trtllm::gen::Dtype(1052672) +, /* mSchedHostTask */ gemm::SchedHostTask(0) +, /* mSfBlockSizeA */ 32 +, /* mSfBlockSizeB */ 32 +, /* mSfBlockSizeC */ 32 +, /* mSfLayoutA */ trtllm::gen::SfLayout(3) +, /* mSfLayoutB */ trtllm::gen::SfLayout(0) +, /* mSfLayoutC */ trtllm::gen::SfLayout(1) +, /* mSfReshapeFactor */ 1 +, /* mSparsityA */ trtllm::gen::Sparsity(0) +, /* mSplitK */ gemm::SplitK(0) +, /* mTileK */ 256 +, /* mTileM */ 128 +, /* mTileN */ 64 +, /* mTileScheduler */ gemm::TileScheduler(1) +, /* mTransposeMmaOutput */ 1 +, /* mUseCustomizedMma3xNvFp4 */ 0 +, /* mUseCustomMmaSchedule */ 1 +, /* mUseDeepSeekFp8 */ 0 +, /* mUseFlexibleClusterDims */ 0 +, /* mUseHoistTryWaitForCustomMmaSchedule */ 0 +, /* mUseMaxTmemOverlap */ 0 +, /* mUsePerTokenSfA */ 0 +, /* mUsePerTokenSfB */ 0 +, /* mUseShuffledMatrix */ 1 +, /* mUseTmaStore */ 1 +, /* mUseTwoTmaLoadWarps */ 1 +, /* mUseTwoMmaWarps */ 0 +, /* mUseUnrollLoop2xForMma */ 1 +, /* mValidM */ 256 +, /* mValidN */ 256 +, /* mValidK */ 512 +, /* mWorldSize */ 1 +, /* mActType */ gemmGatedAct::ActType(2) +, /* mClampBeforeAct */ 1 +, /* mBatchedM */ {} +, /* mBatchedN */ {} +, /* mBatchMode */ batchedGemm::BatchedGemmOptions::BatchMode(1) +, /* mBatchStrideInTokens */ -1 +, /* mFusedAct */ 1 +, /* mGridWaitForPrimaryRouting */ 1 +, /* mIsStaticBatch */ 0 +, /* mIsUniformNumTokensPerBatch */ 0 +, /* mNumBatches */ 128 +, /* mNumRegsPerThreadLoadA */ 0 +, /* mNumRegsPerThreadLoadB */ 48 +, /* mNumRegsPerThreadLoadSfA */ 0 +, /* mNumRegsPerThreadLoadSfB */ 48 +, /* mNumTokens */ 2 +, /* mNumWarpsLoadA */ 0 +, /* mNumWarpsLoadB */ 1 +, /* mNumWarpsLoadSfA */ 0 +, /* mNumWarpsLoadSfB */ 1 +, /* mRouteImpl */ batchedGemm::RouteImpl(2) +, /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} +, /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 + }, gemm::SmVersion::Sm100f}, +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x256u2_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW1_lsfbW1_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x256u2_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW1_lsfbW1_dynB_sm100f_cubin_len, 183960, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x256u2_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW1_lsfbW1_dynB_sm100f", 512, "b990550c50bc69dced298f4bea3542c265729a3973713ac7499e26c2a82755ed", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +, /* mBiasDtype */ trtllm::gen::Dtype(1056776) +, /* mBiasType */ gemm::BiasType(1) +, /* mBlockK */ -1 +, /* mClcFastDrain */ 1 +, /* mClusterDimX */ 2 +, /* mClusterDimY */ 1 +, /* mClusterDimZ */ 1 +, /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) +, /* mDtypeAcc */ trtllm::gen::Dtype(1056776) +, /* mDtypeA */ trtllm::gen::Dtype(17826828) +, /* mDtypeB */ trtllm::gen::Dtype(17827853) +, /* mDtypeC */ trtllm::gen::Dtype(17827853) +, /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) +, /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) +, /* mEltwiseActType */ gemm::EltwiseActType(0) +, /* mEnablesEarlyExit */ 1 +, /* mEnablesDelayedEarlyExit */ 0 +, /* mEpilogueLdtmDps */ 16 +, /* mEpilogueLdtmBits */ 256 +, /* mEpilogueTileM */ 128 +, /* mEpilogueTileN */ 64 +, /* mFallbackClusterDimX */ 1 +, /* mFallbackClusterDimY */ 1 +, /* mFallbackClusterDimZ */ 1 +, /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) +, /* mFuseLoadSfTask */ 0 +, /* mFuseUtccpWithUtcmma */ 0 +, /* mGridTriggerSecondaryA */ 0 +, /* mGridTriggerSecondaryB */ 1 +, /* mGridWaitForPrimaryEarlyExit */ 1 +, /* mGridWaitForPrimaryA */ 0 +, /* mGridWaitForPrimaryB */ 1 +, /* mHoistLoadTaskInit */ 1 +, /* mHoistMmaTaskTryWaits */ 0 +, /* mK */ 512 +, /* mKernelTraits */ {} +, /* mLayoutA */ gemm::MatrixLayout(0) +, /* mLayoutB */ gemm::MatrixLayout(0) +, /* mM */ 256 +, /* mMmaK */ 32 +, /* mMmaKind */ trtllm::gen::MmaKind(5) +, /* mMmaM */ 256 +, /* mMmaN */ 64 +, /* mMmaTileK */ 256 +, /* mMockAllReduce */ 0 +, /* mN */ 256 +, /* mNumEpilogueWarps */ 4 +, /* mNumRegsCastAWarps */ 0 +, /* mNumRegsCopySfLdsSttm */ 0 +, /* mNumRegsCopySparsityInfo */ 0 +, /* mNumRegsPerThreadEpilogueWarp */ 168 +, /* mNumRegsPerThreadNonEpilogueWarp */ 48 +, /* mNumSlicesForSplitK */ 1 +, /* mNumStagesA */ 4 +, /* mNumStagesB */ 4 +, /* mNumStagesMma */ 2 +, /* mNumStagesMmaWithinWorkTile */ 1 +, /* mNumStagesMmaAcrossWorkTile */ 2 +, /* mNumStagesSmemSfA */ 4 +, /* mNumStagesSmemSfB */ 4 +, /* mNumStagesTmemSfA */ 4 +, /* mNumStagesTmemSfB */ 4 +, /* mNumStagesWorkId */ 3 +, /* mOutputDebugTensors */ 0 +, /* mPatchF2fp */ 0 +, /* mPerTokenSfDtype */ trtllm::gen::Dtype(1052672) +, /* mSchedHostTask */ gemm::SchedHostTask(0) +, /* mSfBlockSizeA */ 32 +, /* mSfBlockSizeB */ 32 +, /* mSfBlockSizeC */ 32 +, /* mSfLayoutA */ trtllm::gen::SfLayout(3) +, /* mSfLayoutB */ trtllm::gen::SfLayout(0) +, /* mSfLayoutC */ trtllm::gen::SfLayout(1) +, /* mSfReshapeFactor */ 1 +, /* mSparsityA */ trtllm::gen::Sparsity(0) +, /* mSplitK */ gemm::SplitK(0) +, /* mTileK */ 256 +, /* mTileM */ 128 +, /* mTileN */ 64 +, /* mTileScheduler */ gemm::TileScheduler(1) , /* mTransposeMmaOutput */ 1 , /* mUseCustomizedMma3xNvFp4 */ 0 , /* mUseCustomMmaSchedule */ 1 @@ -69965,18 +72503,1405 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadLoadA */ 0 , /* mNumRegsPerThreadLoadB */ 48 , /* mNumRegsPerThreadLoadSfA */ 0 -, /* mNumRegsPerThreadLoadSfB */ 48 +, /* mNumRegsPerThreadLoadSfB */ 48 +, /* mNumTokens */ 2 +, /* mNumWarpsLoadA */ 0 +, /* mNumWarpsLoadB */ 1 +, /* mNumWarpsLoadSfA */ 0 +, /* mNumWarpsLoadSfB */ 1 +, /* mRouteImpl */ batchedGemm::RouteImpl(2) +, /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} +, /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 + }, gemm::SmVersion::Sm100f}, +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len, 222096, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f", 512, "a71a666d4eb660092b018280d41d6d3abf3c172d4c4aa9c4dc35045a7c0ce305", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +, /* mBiasDtype */ trtllm::gen::Dtype(1056776) +, /* mBiasType */ gemm::BiasType(0) +, /* mBlockK */ -1 +, /* mClcFastDrain */ 1 +, /* mClusterDimX */ 1 +, /* mClusterDimY */ 1 +, /* mClusterDimZ */ 1 +, /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) +, /* mDtypeAcc */ trtllm::gen::Dtype(1056776) +, /* mDtypeA */ trtllm::gen::Dtype(17826828) +, /* mDtypeB */ trtllm::gen::Dtype(17827853) +, /* mDtypeC */ trtllm::gen::Dtype(17827853) +, /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) +, /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) +, /* mEltwiseActType */ gemm::EltwiseActType(0) +, /* mEnablesEarlyExit */ 1 +, /* mEnablesDelayedEarlyExit */ 0 +, /* mEpilogueLdtmDps */ 16 +, /* mEpilogueLdtmBits */ 256 +, /* mEpilogueTileM */ 128 +, /* mEpilogueTileN */ 8 +, /* mFallbackClusterDimX */ 1 +, /* mFallbackClusterDimY */ 1 +, /* mFallbackClusterDimZ */ 1 +, /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) +, /* mFuseLoadSfTask */ 0 +, /* mFuseUtccpWithUtcmma */ 0 +, /* mGridTriggerSecondaryA */ 0 +, /* mGridTriggerSecondaryB */ 1 +, /* mGridWaitForPrimaryEarlyExit */ 1 +, /* mGridWaitForPrimaryA */ 0 +, /* mGridWaitForPrimaryB */ 1 +, /* mHoistLoadTaskInit */ 1 +, /* mHoistMmaTaskTryWaits */ 0 +, /* mK */ 256 +, /* mKernelTraits */ {} +, /* mLayoutA */ gemm::MatrixLayout(0) +, /* mLayoutB */ gemm::MatrixLayout(0) +, /* mM */ 256 +, /* mMmaK */ 32 +, /* mMmaKind */ trtllm::gen::MmaKind(5) +, /* mMmaM */ 128 +, /* mMmaN */ 8 +, /* mMmaTileK */ 256 +, /* mMockAllReduce */ 0 +, /* mN */ 256 +, /* mNumEpilogueWarps */ 4 +, /* mNumRegsCastAWarps */ 0 +, /* mNumRegsCopySfLdsSttm */ 0 +, /* mNumRegsCopySparsityInfo */ 0 +, /* mNumRegsPerThreadEpilogueWarp */ 0 +, /* mNumRegsPerThreadNonEpilogueWarp */ 0 +, /* mNumSlicesForSplitK */ 1 +, /* mNumStagesA */ 6 +, /* mNumStagesB */ 6 +, /* mNumStagesMma */ 2 +, /* mNumStagesMmaWithinWorkTile */ 1 +, /* mNumStagesMmaAcrossWorkTile */ 2 +, /* mNumStagesSmemSfA */ 6 +, /* mNumStagesSmemSfB */ 6 +, /* mNumStagesTmemSfA */ 6 +, /* mNumStagesTmemSfB */ 6 +, /* mNumStagesWorkId */ 3 +, /* mOutputDebugTensors */ 0 +, /* mPatchF2fp */ 0 +, /* mPerTokenSfDtype */ trtllm::gen::Dtype(1052672) +, /* mSchedHostTask */ gemm::SchedHostTask(0) +, /* mSfBlockSizeA */ 32 +, /* mSfBlockSizeB */ 32 +, /* mSfBlockSizeC */ 32 +, /* mSfLayoutA */ trtllm::gen::SfLayout(3) +, /* mSfLayoutB */ trtllm::gen::SfLayout(0) +, /* mSfLayoutC */ trtllm::gen::SfLayout(1) +, /* mSfReshapeFactor */ 1 +, /* mSparsityA */ trtllm::gen::Sparsity(0) +, /* mSplitK */ gemm::SplitK(0) +, /* mTileK */ 256 +, /* mTileM */ 128 +, /* mTileN */ 8 +, /* mTileScheduler */ gemm::TileScheduler(1) +, /* mTransposeMmaOutput */ 1 +, /* mUseCustomizedMma3xNvFp4 */ 0 +, /* mUseCustomMmaSchedule */ 1 +, /* mUseDeepSeekFp8 */ 0 +, /* mUseFlexibleClusterDims */ 0 +, /* mUseHoistTryWaitForCustomMmaSchedule */ 0 +, /* mUseMaxTmemOverlap */ 0 +, /* mUsePerTokenSfA */ 0 +, /* mUsePerTokenSfB */ 0 +, /* mUseShuffledMatrix */ 1 +, /* mUseTmaStore */ 1 +, /* mUseTwoTmaLoadWarps */ 1 +, /* mUseTwoMmaWarps */ 0 +, /* mUseUnrollLoop2xForMma */ 0 +, /* mValidM */ 256 +, /* mValidN */ 256 +, /* mValidK */ 256 +, /* mWorldSize */ 1 +, /* mActType */ gemmGatedAct::ActType(2) +, /* mClampBeforeAct */ 1 +, /* mBatchedM */ {} +, /* mBatchedN */ {} +, /* mBatchMode */ batchedGemm::BatchedGemmOptions::BatchMode(1) +, /* mBatchStrideInTokens */ -1 +, /* mFusedAct */ 1 +, /* mGridWaitForPrimaryRouting */ 1 +, /* mIsStaticBatch */ 0 +, /* mIsUniformNumTokensPerBatch */ 0 +, /* mNumBatches */ 128 +, /* mNumRegsPerThreadLoadA */ 0 +, /* mNumRegsPerThreadLoadB */ 0 +, /* mNumRegsPerThreadLoadSfA */ 0 +, /* mNumRegsPerThreadLoadSfB */ 0 +, /* mNumTokens */ 2 +, /* mNumWarpsLoadA */ 0 +, /* mNumWarpsLoadB */ 0 +, /* mNumWarpsLoadSfA */ 0 +, /* mNumWarpsLoadSfB */ 0 +, /* mRouteImpl */ batchedGemm::RouteImpl(2) +, /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} +, /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 + }, gemm::SmVersion::Sm100f}, +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 222096, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "07a055ef85e904d22689ec7e248731ec9358e84799d4bc60b26872137d72f9b8", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +, /* mBiasDtype */ trtllm::gen::Dtype(1056776) +, /* mBiasType */ gemm::BiasType(1) +, /* mBlockK */ -1 +, /* mClcFastDrain */ 1 +, /* mClusterDimX */ 1 +, /* mClusterDimY */ 1 +, /* mClusterDimZ */ 1 +, /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) +, /* mDtypeAcc */ trtllm::gen::Dtype(1056776) +, /* mDtypeA */ trtllm::gen::Dtype(17826828) +, /* mDtypeB */ trtllm::gen::Dtype(17827853) +, /* mDtypeC */ trtllm::gen::Dtype(17827853) +, /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) +, /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) +, /* mEltwiseActType */ gemm::EltwiseActType(0) +, /* mEnablesEarlyExit */ 1 +, /* mEnablesDelayedEarlyExit */ 0 +, /* mEpilogueLdtmDps */ 16 +, /* mEpilogueLdtmBits */ 256 +, /* mEpilogueTileM */ 128 +, /* mEpilogueTileN */ 8 +, /* mFallbackClusterDimX */ 1 +, /* mFallbackClusterDimY */ 1 +, /* mFallbackClusterDimZ */ 1 +, /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) +, /* mFuseLoadSfTask */ 0 +, /* mFuseUtccpWithUtcmma */ 0 +, /* mGridTriggerSecondaryA */ 0 +, /* mGridTriggerSecondaryB */ 1 +, /* mGridWaitForPrimaryEarlyExit */ 1 +, /* mGridWaitForPrimaryA */ 0 +, /* mGridWaitForPrimaryB */ 1 +, /* mHoistLoadTaskInit */ 1 +, /* mHoistMmaTaskTryWaits */ 0 +, /* mK */ 256 +, /* mKernelTraits */ {} +, /* mLayoutA */ gemm::MatrixLayout(0) +, /* mLayoutB */ gemm::MatrixLayout(0) +, /* mM */ 256 +, /* mMmaK */ 32 +, /* mMmaKind */ trtllm::gen::MmaKind(5) +, /* mMmaM */ 128 +, /* mMmaN */ 8 +, /* mMmaTileK */ 256 +, /* mMockAllReduce */ 0 +, /* mN */ 256 +, /* mNumEpilogueWarps */ 4 +, /* mNumRegsCastAWarps */ 0 +, /* mNumRegsCopySfLdsSttm */ 0 +, /* mNumRegsCopySparsityInfo */ 0 +, /* mNumRegsPerThreadEpilogueWarp */ 0 +, /* mNumRegsPerThreadNonEpilogueWarp */ 0 +, /* mNumSlicesForSplitK */ 1 +, /* mNumStagesA */ 6 +, /* mNumStagesB */ 6 +, /* mNumStagesMma */ 2 +, /* mNumStagesMmaWithinWorkTile */ 1 +, /* mNumStagesMmaAcrossWorkTile */ 2 +, /* mNumStagesSmemSfA */ 6 +, /* mNumStagesSmemSfB */ 6 +, /* mNumStagesTmemSfA */ 6 +, /* mNumStagesTmemSfB */ 6 +, /* mNumStagesWorkId */ 3 +, /* mOutputDebugTensors */ 0 +, /* mPatchF2fp */ 0 +, /* mPerTokenSfDtype */ trtllm::gen::Dtype(1052672) +, /* mSchedHostTask */ gemm::SchedHostTask(0) +, /* mSfBlockSizeA */ 32 +, /* mSfBlockSizeB */ 32 +, /* mSfBlockSizeC */ 32 +, /* mSfLayoutA */ trtllm::gen::SfLayout(3) +, /* mSfLayoutB */ trtllm::gen::SfLayout(0) +, /* mSfLayoutC */ trtllm::gen::SfLayout(1) +, /* mSfReshapeFactor */ 1 +, /* mSparsityA */ trtllm::gen::Sparsity(0) +, /* mSplitK */ gemm::SplitK(0) +, /* mTileK */ 256 +, /* mTileM */ 128 +, /* mTileN */ 8 +, /* mTileScheduler */ gemm::TileScheduler(1) +, /* mTransposeMmaOutput */ 1 +, /* mUseCustomizedMma3xNvFp4 */ 0 +, /* mUseCustomMmaSchedule */ 1 +, /* mUseDeepSeekFp8 */ 0 +, /* mUseFlexibleClusterDims */ 0 +, /* mUseHoistTryWaitForCustomMmaSchedule */ 0 +, /* mUseMaxTmemOverlap */ 0 +, /* mUsePerTokenSfA */ 0 +, /* mUsePerTokenSfB */ 0 +, /* mUseShuffledMatrix */ 1 +, /* mUseTmaStore */ 1 +, /* mUseTwoTmaLoadWarps */ 1 +, /* mUseTwoMmaWarps */ 0 +, /* mUseUnrollLoop2xForMma */ 0 +, /* mValidM */ 256 +, /* mValidN */ 256 +, /* mValidK */ 256 +, /* mWorldSize */ 1 +, /* mActType */ gemmGatedAct::ActType(0) +, /* mClampBeforeAct */ 1 +, /* mBatchedM */ {} +, /* mBatchedN */ {} +, /* mBatchMode */ batchedGemm::BatchedGemmOptions::BatchMode(1) +, /* mBatchStrideInTokens */ -1 +, /* mFusedAct */ 1 +, /* mGridWaitForPrimaryRouting */ 1 +, /* mIsStaticBatch */ 0 +, /* mIsUniformNumTokensPerBatch */ 0 +, /* mNumBatches */ 128 +, /* mNumRegsPerThreadLoadA */ 0 +, /* mNumRegsPerThreadLoadB */ 0 +, /* mNumRegsPerThreadLoadSfA */ 0 +, /* mNumRegsPerThreadLoadSfB */ 0 +, /* mNumTokens */ 2 +, /* mNumWarpsLoadA */ 0 +, /* mNumWarpsLoadB */ 0 +, /* mNumWarpsLoadSfA */ 0 +, /* mNumWarpsLoadSfB */ 0 +, /* mRouteImpl */ batchedGemm::RouteImpl(2) +, /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} +, /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 + }, gemm::SmVersion::Sm100f}, +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len, 221968, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f", 512, "424faa689f530844e9916d7124b0e21d81766a8e78b256816972309795766182", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +, /* mBiasDtype */ trtllm::gen::Dtype(1056776) +, /* mBiasType */ gemm::BiasType(0) +, /* mBlockK */ -1 +, /* mClcFastDrain */ 1 +, /* mClusterDimX */ 1 +, /* mClusterDimY */ 1 +, /* mClusterDimZ */ 1 +, /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) +, /* mDtypeAcc */ trtllm::gen::Dtype(1056776) +, /* mDtypeA */ trtllm::gen::Dtype(17826828) +, /* mDtypeB */ trtllm::gen::Dtype(17827853) +, /* mDtypeC */ trtllm::gen::Dtype(17827853) +, /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) +, /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) +, /* mEltwiseActType */ gemm::EltwiseActType(0) +, /* mEnablesEarlyExit */ 1 +, /* mEnablesDelayedEarlyExit */ 0 +, /* mEpilogueLdtmDps */ 16 +, /* mEpilogueLdtmBits */ 256 +, /* mEpilogueTileM */ 128 +, /* mEpilogueTileN */ 8 +, /* mFallbackClusterDimX */ 1 +, /* mFallbackClusterDimY */ 1 +, /* mFallbackClusterDimZ */ 1 +, /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) +, /* mFuseLoadSfTask */ 0 +, /* mFuseUtccpWithUtcmma */ 0 +, /* mGridTriggerSecondaryA */ 0 +, /* mGridTriggerSecondaryB */ 1 +, /* mGridWaitForPrimaryEarlyExit */ 1 +, /* mGridWaitForPrimaryA */ 0 +, /* mGridWaitForPrimaryB */ 1 +, /* mHoistLoadTaskInit */ 1 +, /* mHoistMmaTaskTryWaits */ 0 +, /* mK */ 256 +, /* mKernelTraits */ {} +, /* mLayoutA */ gemm::MatrixLayout(0) +, /* mLayoutB */ gemm::MatrixLayout(0) +, /* mM */ 256 +, /* mMmaK */ 32 +, /* mMmaKind */ trtllm::gen::MmaKind(5) +, /* mMmaM */ 128 +, /* mMmaN */ 8 +, /* mMmaTileK */ 256 +, /* mMockAllReduce */ 0 +, /* mN */ 256 +, /* mNumEpilogueWarps */ 4 +, /* mNumRegsCastAWarps */ 0 +, /* mNumRegsCopySfLdsSttm */ 0 +, /* mNumRegsCopySparsityInfo */ 0 +, /* mNumRegsPerThreadEpilogueWarp */ 0 +, /* mNumRegsPerThreadNonEpilogueWarp */ 0 +, /* mNumSlicesForSplitK */ 1 +, /* mNumStagesA */ 6 +, /* mNumStagesB */ 6 +, /* mNumStagesMma */ 1 +, /* mNumStagesMmaWithinWorkTile */ 1 +, /* mNumStagesMmaAcrossWorkTile */ 1 +, /* mNumStagesSmemSfA */ 6 +, /* mNumStagesSmemSfB */ 6 +, /* mNumStagesTmemSfA */ 6 +, /* mNumStagesTmemSfB */ 6 +, /* mNumStagesWorkId */ 3 +, /* mOutputDebugTensors */ 0 +, /* mPatchF2fp */ 0 +, /* mPerTokenSfDtype */ trtllm::gen::Dtype(1052672) +, /* mSchedHostTask */ gemm::SchedHostTask(0) +, /* mSfBlockSizeA */ 32 +, /* mSfBlockSizeB */ 32 +, /* mSfBlockSizeC */ 32 +, /* mSfLayoutA */ trtllm::gen::SfLayout(3) +, /* mSfLayoutB */ trtllm::gen::SfLayout(0) +, /* mSfLayoutC */ trtllm::gen::SfLayout(1) +, /* mSfReshapeFactor */ 1 +, /* mSparsityA */ trtllm::gen::Sparsity(0) +, /* mSplitK */ gemm::SplitK(0) +, /* mTileK */ 256 +, /* mTileM */ 128 +, /* mTileN */ 8 +, /* mTileScheduler */ gemm::TileScheduler(0) +, /* mTransposeMmaOutput */ 1 +, /* mUseCustomizedMma3xNvFp4 */ 0 +, /* mUseCustomMmaSchedule */ 1 +, /* mUseDeepSeekFp8 */ 0 +, /* mUseFlexibleClusterDims */ 0 +, /* mUseHoistTryWaitForCustomMmaSchedule */ 0 +, /* mUseMaxTmemOverlap */ 0 +, /* mUsePerTokenSfA */ 0 +, /* mUsePerTokenSfB */ 0 +, /* mUseShuffledMatrix */ 1 +, /* mUseTmaStore */ 1 +, /* mUseTwoTmaLoadWarps */ 1 +, /* mUseTwoMmaWarps */ 0 +, /* mUseUnrollLoop2xForMma */ 0 +, /* mValidM */ 256 +, /* mValidN */ 256 +, /* mValidK */ 256 +, /* mWorldSize */ 1 +, /* mActType */ gemmGatedAct::ActType(2) +, /* mClampBeforeAct */ 1 +, /* mBatchedM */ {} +, /* mBatchedN */ {} +, /* mBatchMode */ batchedGemm::BatchedGemmOptions::BatchMode(1) +, /* mBatchStrideInTokens */ -1 +, /* mFusedAct */ 1 +, /* mGridWaitForPrimaryRouting */ 1 +, /* mIsStaticBatch */ 0 +, /* mIsUniformNumTokensPerBatch */ 0 +, /* mNumBatches */ 128 +, /* mNumRegsPerThreadLoadA */ 0 +, /* mNumRegsPerThreadLoadB */ 0 +, /* mNumRegsPerThreadLoadSfA */ 0 +, /* mNumRegsPerThreadLoadSfB */ 0 +, /* mNumTokens */ 2 +, /* mNumWarpsLoadA */ 0 +, /* mNumWarpsLoadB */ 0 +, /* mNumWarpsLoadSfA */ 0 +, /* mNumWarpsLoadSfB */ 0 +, /* mRouteImpl */ batchedGemm::RouteImpl(2) +, /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} +, /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 + }, gemm::SmVersion::Sm100f}, +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 221968, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "d6962e99fde2cf9211a083f11a062664dcca57194d46b6abb5d0b9f8c31bd32b", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +, /* mBiasDtype */ trtllm::gen::Dtype(1056776) +, /* mBiasType */ gemm::BiasType(1) +, /* mBlockK */ -1 +, /* mClcFastDrain */ 1 +, /* mClusterDimX */ 1 +, /* mClusterDimY */ 1 +, /* mClusterDimZ */ 1 +, /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) +, /* mDtypeAcc */ trtllm::gen::Dtype(1056776) +, /* mDtypeA */ trtllm::gen::Dtype(17826828) +, /* mDtypeB */ trtllm::gen::Dtype(17827853) +, /* mDtypeC */ trtllm::gen::Dtype(17827853) +, /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) +, /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) +, /* mEltwiseActType */ gemm::EltwiseActType(0) +, /* mEnablesEarlyExit */ 1 +, /* mEnablesDelayedEarlyExit */ 0 +, /* mEpilogueLdtmDps */ 16 +, /* mEpilogueLdtmBits */ 256 +, /* mEpilogueTileM */ 128 +, /* mEpilogueTileN */ 8 +, /* mFallbackClusterDimX */ 1 +, /* mFallbackClusterDimY */ 1 +, /* mFallbackClusterDimZ */ 1 +, /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) +, /* mFuseLoadSfTask */ 0 +, /* mFuseUtccpWithUtcmma */ 0 +, /* mGridTriggerSecondaryA */ 0 +, /* mGridTriggerSecondaryB */ 1 +, /* mGridWaitForPrimaryEarlyExit */ 1 +, /* mGridWaitForPrimaryA */ 0 +, /* mGridWaitForPrimaryB */ 1 +, /* mHoistLoadTaskInit */ 1 +, /* mHoistMmaTaskTryWaits */ 0 +, /* mK */ 256 +, /* mKernelTraits */ {} +, /* mLayoutA */ gemm::MatrixLayout(0) +, /* mLayoutB */ gemm::MatrixLayout(0) +, /* mM */ 256 +, /* mMmaK */ 32 +, /* mMmaKind */ trtllm::gen::MmaKind(5) +, /* mMmaM */ 128 +, /* mMmaN */ 8 +, /* mMmaTileK */ 256 +, /* mMockAllReduce */ 0 +, /* mN */ 256 +, /* mNumEpilogueWarps */ 4 +, /* mNumRegsCastAWarps */ 0 +, /* mNumRegsCopySfLdsSttm */ 0 +, /* mNumRegsCopySparsityInfo */ 0 +, /* mNumRegsPerThreadEpilogueWarp */ 0 +, /* mNumRegsPerThreadNonEpilogueWarp */ 0 +, /* mNumSlicesForSplitK */ 1 +, /* mNumStagesA */ 6 +, /* mNumStagesB */ 6 +, /* mNumStagesMma */ 1 +, /* mNumStagesMmaWithinWorkTile */ 1 +, /* mNumStagesMmaAcrossWorkTile */ 1 +, /* mNumStagesSmemSfA */ 6 +, /* mNumStagesSmemSfB */ 6 +, /* mNumStagesTmemSfA */ 6 +, /* mNumStagesTmemSfB */ 6 +, /* mNumStagesWorkId */ 3 +, /* mOutputDebugTensors */ 0 +, /* mPatchF2fp */ 0 +, /* mPerTokenSfDtype */ trtllm::gen::Dtype(1052672) +, /* mSchedHostTask */ gemm::SchedHostTask(0) +, /* mSfBlockSizeA */ 32 +, /* mSfBlockSizeB */ 32 +, /* mSfBlockSizeC */ 32 +, /* mSfLayoutA */ trtllm::gen::SfLayout(3) +, /* mSfLayoutB */ trtllm::gen::SfLayout(0) +, /* mSfLayoutC */ trtllm::gen::SfLayout(1) +, /* mSfReshapeFactor */ 1 +, /* mSparsityA */ trtllm::gen::Sparsity(0) +, /* mSplitK */ gemm::SplitK(0) +, /* mTileK */ 256 +, /* mTileM */ 128 +, /* mTileN */ 8 +, /* mTileScheduler */ gemm::TileScheduler(0) +, /* mTransposeMmaOutput */ 1 +, /* mUseCustomizedMma3xNvFp4 */ 0 +, /* mUseCustomMmaSchedule */ 1 +, /* mUseDeepSeekFp8 */ 0 +, /* mUseFlexibleClusterDims */ 0 +, /* mUseHoistTryWaitForCustomMmaSchedule */ 0 +, /* mUseMaxTmemOverlap */ 0 +, /* mUsePerTokenSfA */ 0 +, /* mUsePerTokenSfB */ 0 +, /* mUseShuffledMatrix */ 1 +, /* mUseTmaStore */ 1 +, /* mUseTwoTmaLoadWarps */ 1 +, /* mUseTwoMmaWarps */ 0 +, /* mUseUnrollLoop2xForMma */ 0 +, /* mValidM */ 256 +, /* mValidN */ 256 +, /* mValidK */ 256 +, /* mWorldSize */ 1 +, /* mActType */ gemmGatedAct::ActType(0) +, /* mClampBeforeAct */ 1 +, /* mBatchedM */ {} +, /* mBatchedN */ {} +, /* mBatchMode */ batchedGemm::BatchedGemmOptions::BatchMode(1) +, /* mBatchStrideInTokens */ -1 +, /* mFusedAct */ 1 +, /* mGridWaitForPrimaryRouting */ 1 +, /* mIsStaticBatch */ 0 +, /* mIsUniformNumTokensPerBatch */ 0 +, /* mNumBatches */ 128 +, /* mNumRegsPerThreadLoadA */ 0 +, /* mNumRegsPerThreadLoadB */ 0 +, /* mNumRegsPerThreadLoadSfA */ 0 +, /* mNumRegsPerThreadLoadSfB */ 0 +, /* mNumTokens */ 2 +, /* mNumWarpsLoadA */ 0 +, /* mNumWarpsLoadB */ 0 +, /* mNumWarpsLoadSfA */ 0 +, /* mNumWarpsLoadSfB */ 0 +, /* mRouteImpl */ batchedGemm::RouteImpl(2) +, /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} +, /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 + }, gemm::SmVersion::Sm100f}, +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len, 229272, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f", 512, "d0c04ad40bed365b29dc419528d0ace66b60d05911da1b404a7499c271d2f3f6", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +, /* mBiasDtype */ trtllm::gen::Dtype(1056776) +, /* mBiasType */ gemm::BiasType(0) +, /* mBlockK */ -1 +, /* mClcFastDrain */ 1 +, /* mClusterDimX */ 1 +, /* mClusterDimY */ 1 +, /* mClusterDimZ */ 2 +, /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) +, /* mDtypeAcc */ trtllm::gen::Dtype(1056776) +, /* mDtypeA */ trtllm::gen::Dtype(17826828) +, /* mDtypeB */ trtllm::gen::Dtype(17827853) +, /* mDtypeC */ trtllm::gen::Dtype(17827853) +, /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) +, /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) +, /* mEltwiseActType */ gemm::EltwiseActType(0) +, /* mEnablesEarlyExit */ 1 +, /* mEnablesDelayedEarlyExit */ 0 +, /* mEpilogueLdtmDps */ 16 +, /* mEpilogueLdtmBits */ 256 +, /* mEpilogueTileM */ 128 +, /* mEpilogueTileN */ 8 +, /* mFallbackClusterDimX */ 1 +, /* mFallbackClusterDimY */ 1 +, /* mFallbackClusterDimZ */ 1 +, /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) +, /* mFuseLoadSfTask */ 0 +, /* mFuseUtccpWithUtcmma */ 0 +, /* mGridTriggerSecondaryA */ 0 +, /* mGridTriggerSecondaryB */ 1 +, /* mGridWaitForPrimaryEarlyExit */ 1 +, /* mGridWaitForPrimaryA */ 0 +, /* mGridWaitForPrimaryB */ 1 +, /* mHoistLoadTaskInit */ 1 +, /* mHoistMmaTaskTryWaits */ 0 +, /* mK */ 512 +, /* mKernelTraits */ {} +, /* mLayoutA */ gemm::MatrixLayout(0) +, /* mLayoutB */ gemm::MatrixLayout(0) +, /* mM */ 256 +, /* mMmaK */ 32 +, /* mMmaKind */ trtllm::gen::MmaKind(5) +, /* mMmaM */ 128 +, /* mMmaN */ 8 +, /* mMmaTileK */ 256 +, /* mMockAllReduce */ 0 +, /* mN */ 256 +, /* mNumEpilogueWarps */ 4 +, /* mNumRegsCastAWarps */ 0 +, /* mNumRegsCopySfLdsSttm */ 0 +, /* mNumRegsCopySparsityInfo */ 0 +, /* mNumRegsPerThreadEpilogueWarp */ 0 +, /* mNumRegsPerThreadNonEpilogueWarp */ 0 +, /* mNumSlicesForSplitK */ 2 +, /* mNumStagesA */ 6 +, /* mNumStagesB */ 6 +, /* mNumStagesMma */ 2 +, /* mNumStagesMmaWithinWorkTile */ 1 +, /* mNumStagesMmaAcrossWorkTile */ 2 +, /* mNumStagesSmemSfA */ 6 +, /* mNumStagesSmemSfB */ 6 +, /* mNumStagesTmemSfA */ 6 +, /* mNumStagesTmemSfB */ 6 +, /* mNumStagesWorkId */ 3 +, /* mOutputDebugTensors */ 0 +, /* mPatchF2fp */ 0 +, /* mPerTokenSfDtype */ trtllm::gen::Dtype(1052672) +, /* mSchedHostTask */ gemm::SchedHostTask(0) +, /* mSfBlockSizeA */ 32 +, /* mSfBlockSizeB */ 32 +, /* mSfBlockSizeC */ 32 +, /* mSfLayoutA */ trtllm::gen::SfLayout(3) +, /* mSfLayoutB */ trtllm::gen::SfLayout(0) +, /* mSfLayoutC */ trtllm::gen::SfLayout(1) +, /* mSfReshapeFactor */ 1 +, /* mSparsityA */ trtllm::gen::Sparsity(0) +, /* mSplitK */ gemm::SplitK(2) +, /* mTileK */ 256 +, /* mTileM */ 128 +, /* mTileN */ 8 +, /* mTileScheduler */ gemm::TileScheduler(1) +, /* mTransposeMmaOutput */ 1 +, /* mUseCustomizedMma3xNvFp4 */ 0 +, /* mUseCustomMmaSchedule */ 1 +, /* mUseDeepSeekFp8 */ 0 +, /* mUseFlexibleClusterDims */ 0 +, /* mUseHoistTryWaitForCustomMmaSchedule */ 0 +, /* mUseMaxTmemOverlap */ 0 +, /* mUsePerTokenSfA */ 0 +, /* mUsePerTokenSfB */ 0 +, /* mUseShuffledMatrix */ 1 +, /* mUseTmaStore */ 1 +, /* mUseTwoTmaLoadWarps */ 1 +, /* mUseTwoMmaWarps */ 0 +, /* mUseUnrollLoop2xForMma */ 0 +, /* mValidM */ 256 +, /* mValidN */ 256 +, /* mValidK */ 512 +, /* mWorldSize */ 1 +, /* mActType */ gemmGatedAct::ActType(2) +, /* mClampBeforeAct */ 1 +, /* mBatchedM */ {} +, /* mBatchedN */ {} +, /* mBatchMode */ batchedGemm::BatchedGemmOptions::BatchMode(1) +, /* mBatchStrideInTokens */ -1 +, /* mFusedAct */ 1 +, /* mGridWaitForPrimaryRouting */ 1 +, /* mIsStaticBatch */ 0 +, /* mIsUniformNumTokensPerBatch */ 0 +, /* mNumBatches */ 128 +, /* mNumRegsPerThreadLoadA */ 0 +, /* mNumRegsPerThreadLoadB */ 0 +, /* mNumRegsPerThreadLoadSfA */ 0 +, /* mNumRegsPerThreadLoadSfB */ 0 +, /* mNumTokens */ 2 +, /* mNumWarpsLoadA */ 0 +, /* mNumWarpsLoadB */ 0 +, /* mNumWarpsLoadSfA */ 0 +, /* mNumWarpsLoadSfB */ 0 +, /* mRouteImpl */ batchedGemm::RouteImpl(2) +, /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} +, /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 + }, gemm::SmVersion::Sm100f}, +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 229272, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "b2da343403a27e0bcf97bf7afd44bc7babe463661df8002d4f802d10f17bf83e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +, /* mBiasDtype */ trtllm::gen::Dtype(1056776) +, /* mBiasType */ gemm::BiasType(1) +, /* mBlockK */ -1 +, /* mClcFastDrain */ 1 +, /* mClusterDimX */ 1 +, /* mClusterDimY */ 1 +, /* mClusterDimZ */ 2 +, /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) +, /* mDtypeAcc */ trtllm::gen::Dtype(1056776) +, /* mDtypeA */ trtllm::gen::Dtype(17826828) +, /* mDtypeB */ trtllm::gen::Dtype(17827853) +, /* mDtypeC */ trtllm::gen::Dtype(17827853) +, /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) +, /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) +, /* mEltwiseActType */ gemm::EltwiseActType(0) +, /* mEnablesEarlyExit */ 1 +, /* mEnablesDelayedEarlyExit */ 0 +, /* mEpilogueLdtmDps */ 16 +, /* mEpilogueLdtmBits */ 256 +, /* mEpilogueTileM */ 128 +, /* mEpilogueTileN */ 8 +, /* mFallbackClusterDimX */ 1 +, /* mFallbackClusterDimY */ 1 +, /* mFallbackClusterDimZ */ 1 +, /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) +, /* mFuseLoadSfTask */ 0 +, /* mFuseUtccpWithUtcmma */ 0 +, /* mGridTriggerSecondaryA */ 0 +, /* mGridTriggerSecondaryB */ 1 +, /* mGridWaitForPrimaryEarlyExit */ 1 +, /* mGridWaitForPrimaryA */ 0 +, /* mGridWaitForPrimaryB */ 1 +, /* mHoistLoadTaskInit */ 1 +, /* mHoistMmaTaskTryWaits */ 0 +, /* mK */ 512 +, /* mKernelTraits */ {} +, /* mLayoutA */ gemm::MatrixLayout(0) +, /* mLayoutB */ gemm::MatrixLayout(0) +, /* mM */ 256 +, /* mMmaK */ 32 +, /* mMmaKind */ trtllm::gen::MmaKind(5) +, /* mMmaM */ 128 +, /* mMmaN */ 8 +, /* mMmaTileK */ 256 +, /* mMockAllReduce */ 0 +, /* mN */ 256 +, /* mNumEpilogueWarps */ 4 +, /* mNumRegsCastAWarps */ 0 +, /* mNumRegsCopySfLdsSttm */ 0 +, /* mNumRegsCopySparsityInfo */ 0 +, /* mNumRegsPerThreadEpilogueWarp */ 0 +, /* mNumRegsPerThreadNonEpilogueWarp */ 0 +, /* mNumSlicesForSplitK */ 2 +, /* mNumStagesA */ 6 +, /* mNumStagesB */ 6 +, /* mNumStagesMma */ 2 +, /* mNumStagesMmaWithinWorkTile */ 1 +, /* mNumStagesMmaAcrossWorkTile */ 2 +, /* mNumStagesSmemSfA */ 6 +, /* mNumStagesSmemSfB */ 6 +, /* mNumStagesTmemSfA */ 6 +, /* mNumStagesTmemSfB */ 6 +, /* mNumStagesWorkId */ 3 +, /* mOutputDebugTensors */ 0 +, /* mPatchF2fp */ 0 +, /* mPerTokenSfDtype */ trtllm::gen::Dtype(1052672) +, /* mSchedHostTask */ gemm::SchedHostTask(0) +, /* mSfBlockSizeA */ 32 +, /* mSfBlockSizeB */ 32 +, /* mSfBlockSizeC */ 32 +, /* mSfLayoutA */ trtllm::gen::SfLayout(3) +, /* mSfLayoutB */ trtllm::gen::SfLayout(0) +, /* mSfLayoutC */ trtllm::gen::SfLayout(1) +, /* mSfReshapeFactor */ 1 +, /* mSparsityA */ trtllm::gen::Sparsity(0) +, /* mSplitK */ gemm::SplitK(2) +, /* mTileK */ 256 +, /* mTileM */ 128 +, /* mTileN */ 8 +, /* mTileScheduler */ gemm::TileScheduler(1) +, /* mTransposeMmaOutput */ 1 +, /* mUseCustomizedMma3xNvFp4 */ 0 +, /* mUseCustomMmaSchedule */ 1 +, /* mUseDeepSeekFp8 */ 0 +, /* mUseFlexibleClusterDims */ 0 +, /* mUseHoistTryWaitForCustomMmaSchedule */ 0 +, /* mUseMaxTmemOverlap */ 0 +, /* mUsePerTokenSfA */ 0 +, /* mUsePerTokenSfB */ 0 +, /* mUseShuffledMatrix */ 1 +, /* mUseTmaStore */ 1 +, /* mUseTwoTmaLoadWarps */ 1 +, /* mUseTwoMmaWarps */ 0 +, /* mUseUnrollLoop2xForMma */ 0 +, /* mValidM */ 256 +, /* mValidN */ 256 +, /* mValidK */ 512 +, /* mWorldSize */ 1 +, /* mActType */ gemmGatedAct::ActType(0) +, /* mClampBeforeAct */ 1 +, /* mBatchedM */ {} +, /* mBatchedN */ {} +, /* mBatchMode */ batchedGemm::BatchedGemmOptions::BatchMode(1) +, /* mBatchStrideInTokens */ -1 +, /* mFusedAct */ 1 +, /* mGridWaitForPrimaryRouting */ 1 +, /* mIsStaticBatch */ 0 +, /* mIsUniformNumTokensPerBatch */ 0 +, /* mNumBatches */ 128 +, /* mNumRegsPerThreadLoadA */ 0 +, /* mNumRegsPerThreadLoadB */ 0 +, /* mNumRegsPerThreadLoadSfA */ 0 +, /* mNumRegsPerThreadLoadSfB */ 0 +, /* mNumTokens */ 2 +, /* mNumWarpsLoadA */ 0 +, /* mNumWarpsLoadB */ 0 +, /* mNumWarpsLoadSfA */ 0 +, /* mNumWarpsLoadSfB */ 0 +, /* mRouteImpl */ batchedGemm::RouteImpl(2) +, /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} +, /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 + }, gemm::SmVersion::Sm100f}, +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len, 220952, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f", 512, "7e5e80b0a30d4aa40f847a4ad2e032cb4e9545516407b5bbceab25f21acb4e29", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +, /* mBiasDtype */ trtllm::gen::Dtype(1056776) +, /* mBiasType */ gemm::BiasType(0) +, /* mBlockK */ -1 +, /* mClcFastDrain */ 1 +, /* mClusterDimX */ 1 +, /* mClusterDimY */ 1 +, /* mClusterDimZ */ 2 +, /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) +, /* mDtypeAcc */ trtllm::gen::Dtype(1056776) +, /* mDtypeA */ trtllm::gen::Dtype(17826828) +, /* mDtypeB */ trtllm::gen::Dtype(17827853) +, /* mDtypeC */ trtllm::gen::Dtype(17827853) +, /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) +, /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) +, /* mEltwiseActType */ gemm::EltwiseActType(0) +, /* mEnablesEarlyExit */ 1 +, /* mEnablesDelayedEarlyExit */ 0 +, /* mEpilogueLdtmDps */ 16 +, /* mEpilogueLdtmBits */ 256 +, /* mEpilogueTileM */ 128 +, /* mEpilogueTileN */ 8 +, /* mFallbackClusterDimX */ 1 +, /* mFallbackClusterDimY */ 1 +, /* mFallbackClusterDimZ */ 1 +, /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) +, /* mFuseLoadSfTask */ 0 +, /* mFuseUtccpWithUtcmma */ 0 +, /* mGridTriggerSecondaryA */ 0 +, /* mGridTriggerSecondaryB */ 1 +, /* mGridWaitForPrimaryEarlyExit */ 1 +, /* mGridWaitForPrimaryA */ 0 +, /* mGridWaitForPrimaryB */ 1 +, /* mHoistLoadTaskInit */ 1 +, /* mHoistMmaTaskTryWaits */ 0 +, /* mK */ 512 +, /* mKernelTraits */ {} +, /* mLayoutA */ gemm::MatrixLayout(0) +, /* mLayoutB */ gemm::MatrixLayout(0) +, /* mM */ 256 +, /* mMmaK */ 32 +, /* mMmaKind */ trtllm::gen::MmaKind(5) +, /* mMmaM */ 128 +, /* mMmaN */ 8 +, /* mMmaTileK */ 256 +, /* mMockAllReduce */ 0 +, /* mN */ 256 +, /* mNumEpilogueWarps */ 4 +, /* mNumRegsCastAWarps */ 0 +, /* mNumRegsCopySfLdsSttm */ 0 +, /* mNumRegsCopySparsityInfo */ 0 +, /* mNumRegsPerThreadEpilogueWarp */ 0 +, /* mNumRegsPerThreadNonEpilogueWarp */ 0 +, /* mNumSlicesForSplitK */ 2 +, /* mNumStagesA */ 6 +, /* mNumStagesB */ 6 +, /* mNumStagesMma */ 1 +, /* mNumStagesMmaWithinWorkTile */ 1 +, /* mNumStagesMmaAcrossWorkTile */ 1 +, /* mNumStagesSmemSfA */ 6 +, /* mNumStagesSmemSfB */ 6 +, /* mNumStagesTmemSfA */ 6 +, /* mNumStagesTmemSfB */ 6 +, /* mNumStagesWorkId */ 3 +, /* mOutputDebugTensors */ 0 +, /* mPatchF2fp */ 0 +, /* mPerTokenSfDtype */ trtllm::gen::Dtype(1052672) +, /* mSchedHostTask */ gemm::SchedHostTask(0) +, /* mSfBlockSizeA */ 32 +, /* mSfBlockSizeB */ 32 +, /* mSfBlockSizeC */ 32 +, /* mSfLayoutA */ trtllm::gen::SfLayout(3) +, /* mSfLayoutB */ trtllm::gen::SfLayout(0) +, /* mSfLayoutC */ trtllm::gen::SfLayout(1) +, /* mSfReshapeFactor */ 1 +, /* mSparsityA */ trtllm::gen::Sparsity(0) +, /* mSplitK */ gemm::SplitK(2) +, /* mTileK */ 256 +, /* mTileM */ 128 +, /* mTileN */ 8 +, /* mTileScheduler */ gemm::TileScheduler(0) +, /* mTransposeMmaOutput */ 1 +, /* mUseCustomizedMma3xNvFp4 */ 0 +, /* mUseCustomMmaSchedule */ 1 +, /* mUseDeepSeekFp8 */ 0 +, /* mUseFlexibleClusterDims */ 0 +, /* mUseHoistTryWaitForCustomMmaSchedule */ 0 +, /* mUseMaxTmemOverlap */ 0 +, /* mUsePerTokenSfA */ 0 +, /* mUsePerTokenSfB */ 0 +, /* mUseShuffledMatrix */ 1 +, /* mUseTmaStore */ 1 +, /* mUseTwoTmaLoadWarps */ 1 +, /* mUseTwoMmaWarps */ 0 +, /* mUseUnrollLoop2xForMma */ 0 +, /* mValidM */ 256 +, /* mValidN */ 256 +, /* mValidK */ 512 +, /* mWorldSize */ 1 +, /* mActType */ gemmGatedAct::ActType(2) +, /* mClampBeforeAct */ 1 +, /* mBatchedM */ {} +, /* mBatchedN */ {} +, /* mBatchMode */ batchedGemm::BatchedGemmOptions::BatchMode(1) +, /* mBatchStrideInTokens */ -1 +, /* mFusedAct */ 1 +, /* mGridWaitForPrimaryRouting */ 1 +, /* mIsStaticBatch */ 0 +, /* mIsUniformNumTokensPerBatch */ 0 +, /* mNumBatches */ 128 +, /* mNumRegsPerThreadLoadA */ 0 +, /* mNumRegsPerThreadLoadB */ 0 +, /* mNumRegsPerThreadLoadSfA */ 0 +, /* mNumRegsPerThreadLoadSfB */ 0 +, /* mNumTokens */ 2 +, /* mNumWarpsLoadA */ 0 +, /* mNumWarpsLoadB */ 0 +, /* mNumWarpsLoadSfA */ 0 +, /* mNumWarpsLoadSfB */ 0 +, /* mRouteImpl */ batchedGemm::RouteImpl(2) +, /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} +, /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 + }, gemm::SmVersion::Sm100f}, +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 220952, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "7c062c8271b765c785821e7a8a2f6c0f97dffe8e46b8b288e348ca69881ff5df", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +, /* mBiasDtype */ trtllm::gen::Dtype(1056776) +, /* mBiasType */ gemm::BiasType(1) +, /* mBlockK */ -1 +, /* mClcFastDrain */ 1 +, /* mClusterDimX */ 1 +, /* mClusterDimY */ 1 +, /* mClusterDimZ */ 2 +, /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) +, /* mDtypeAcc */ trtllm::gen::Dtype(1056776) +, /* mDtypeA */ trtllm::gen::Dtype(17826828) +, /* mDtypeB */ trtllm::gen::Dtype(17827853) +, /* mDtypeC */ trtllm::gen::Dtype(17827853) +, /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) +, /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) +, /* mEltwiseActType */ gemm::EltwiseActType(0) +, /* mEnablesEarlyExit */ 1 +, /* mEnablesDelayedEarlyExit */ 0 +, /* mEpilogueLdtmDps */ 16 +, /* mEpilogueLdtmBits */ 256 +, /* mEpilogueTileM */ 128 +, /* mEpilogueTileN */ 8 +, /* mFallbackClusterDimX */ 1 +, /* mFallbackClusterDimY */ 1 +, /* mFallbackClusterDimZ */ 1 +, /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) +, /* mFuseLoadSfTask */ 0 +, /* mFuseUtccpWithUtcmma */ 0 +, /* mGridTriggerSecondaryA */ 0 +, /* mGridTriggerSecondaryB */ 1 +, /* mGridWaitForPrimaryEarlyExit */ 1 +, /* mGridWaitForPrimaryA */ 0 +, /* mGridWaitForPrimaryB */ 1 +, /* mHoistLoadTaskInit */ 1 +, /* mHoistMmaTaskTryWaits */ 0 +, /* mK */ 512 +, /* mKernelTraits */ {} +, /* mLayoutA */ gemm::MatrixLayout(0) +, /* mLayoutB */ gemm::MatrixLayout(0) +, /* mM */ 256 +, /* mMmaK */ 32 +, /* mMmaKind */ trtllm::gen::MmaKind(5) +, /* mMmaM */ 128 +, /* mMmaN */ 8 +, /* mMmaTileK */ 256 +, /* mMockAllReduce */ 0 +, /* mN */ 256 +, /* mNumEpilogueWarps */ 4 +, /* mNumRegsCastAWarps */ 0 +, /* mNumRegsCopySfLdsSttm */ 0 +, /* mNumRegsCopySparsityInfo */ 0 +, /* mNumRegsPerThreadEpilogueWarp */ 0 +, /* mNumRegsPerThreadNonEpilogueWarp */ 0 +, /* mNumSlicesForSplitK */ 2 +, /* mNumStagesA */ 6 +, /* mNumStagesB */ 6 +, /* mNumStagesMma */ 1 +, /* mNumStagesMmaWithinWorkTile */ 1 +, /* mNumStagesMmaAcrossWorkTile */ 1 +, /* mNumStagesSmemSfA */ 6 +, /* mNumStagesSmemSfB */ 6 +, /* mNumStagesTmemSfA */ 6 +, /* mNumStagesTmemSfB */ 6 +, /* mNumStagesWorkId */ 3 +, /* mOutputDebugTensors */ 0 +, /* mPatchF2fp */ 0 +, /* mPerTokenSfDtype */ trtllm::gen::Dtype(1052672) +, /* mSchedHostTask */ gemm::SchedHostTask(0) +, /* mSfBlockSizeA */ 32 +, /* mSfBlockSizeB */ 32 +, /* mSfBlockSizeC */ 32 +, /* mSfLayoutA */ trtllm::gen::SfLayout(3) +, /* mSfLayoutB */ trtllm::gen::SfLayout(0) +, /* mSfLayoutC */ trtllm::gen::SfLayout(1) +, /* mSfReshapeFactor */ 1 +, /* mSparsityA */ trtllm::gen::Sparsity(0) +, /* mSplitK */ gemm::SplitK(2) +, /* mTileK */ 256 +, /* mTileM */ 128 +, /* mTileN */ 8 +, /* mTileScheduler */ gemm::TileScheduler(0) +, /* mTransposeMmaOutput */ 1 +, /* mUseCustomizedMma3xNvFp4 */ 0 +, /* mUseCustomMmaSchedule */ 1 +, /* mUseDeepSeekFp8 */ 0 +, /* mUseFlexibleClusterDims */ 0 +, /* mUseHoistTryWaitForCustomMmaSchedule */ 0 +, /* mUseMaxTmemOverlap */ 0 +, /* mUsePerTokenSfA */ 0 +, /* mUsePerTokenSfB */ 0 +, /* mUseShuffledMatrix */ 1 +, /* mUseTmaStore */ 1 +, /* mUseTwoTmaLoadWarps */ 1 +, /* mUseTwoMmaWarps */ 0 +, /* mUseUnrollLoop2xForMma */ 0 +, /* mValidM */ 256 +, /* mValidN */ 256 +, /* mValidK */ 512 +, /* mWorldSize */ 1 +, /* mActType */ gemmGatedAct::ActType(0) +, /* mClampBeforeAct */ 1 +, /* mBatchedM */ {} +, /* mBatchedN */ {} +, /* mBatchMode */ batchedGemm::BatchedGemmOptions::BatchMode(1) +, /* mBatchStrideInTokens */ -1 +, /* mFusedAct */ 1 +, /* mGridWaitForPrimaryRouting */ 1 +, /* mIsStaticBatch */ 0 +, /* mIsUniformNumTokensPerBatch */ 0 +, /* mNumBatches */ 128 +, /* mNumRegsPerThreadLoadA */ 0 +, /* mNumRegsPerThreadLoadB */ 0 +, /* mNumRegsPerThreadLoadSfA */ 0 +, /* mNumRegsPerThreadLoadSfB */ 0 +, /* mNumTokens */ 2 +, /* mNumWarpsLoadA */ 0 +, /* mNumWarpsLoadB */ 0 +, /* mNumWarpsLoadSfA */ 0 +, /* mNumWarpsLoadSfB */ 0 +, /* mRouteImpl */ batchedGemm::RouteImpl(2) +, /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} +, /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 + }, gemm::SmVersion::Sm100f}, +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len, 222096, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f", 512, "7abdb9582e2d1fc00828ae028dc9f6d37619c49bd9f9fd2021350b076ecb1c7a", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +, /* mBiasDtype */ trtllm::gen::Dtype(1056776) +, /* mBiasType */ gemm::BiasType(0) +, /* mBlockK */ -1 +, /* mClcFastDrain */ 1 +, /* mClusterDimX */ 1 +, /* mClusterDimY */ 1 +, /* mClusterDimZ */ 1 +, /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) +, /* mDtypeAcc */ trtllm::gen::Dtype(1056776) +, /* mDtypeA */ trtllm::gen::Dtype(17826828) +, /* mDtypeB */ trtllm::gen::Dtype(17827853) +, /* mDtypeC */ trtllm::gen::Dtype(17827853) +, /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) +, /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) +, /* mEltwiseActType */ gemm::EltwiseActType(0) +, /* mEnablesEarlyExit */ 1 +, /* mEnablesDelayedEarlyExit */ 0 +, /* mEpilogueLdtmDps */ 16 +, /* mEpilogueLdtmBits */ 256 +, /* mEpilogueTileM */ 128 +, /* mEpilogueTileN */ 8 +, /* mFallbackClusterDimX */ 1 +, /* mFallbackClusterDimY */ 1 +, /* mFallbackClusterDimZ */ 1 +, /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) +, /* mFuseLoadSfTask */ 0 +, /* mFuseUtccpWithUtcmma */ 0 +, /* mGridTriggerSecondaryA */ 0 +, /* mGridTriggerSecondaryB */ 1 +, /* mGridWaitForPrimaryEarlyExit */ 1 +, /* mGridWaitForPrimaryA */ 0 +, /* mGridWaitForPrimaryB */ 1 +, /* mHoistLoadTaskInit */ 1 +, /* mHoistMmaTaskTryWaits */ 0 +, /* mK */ 512 +, /* mKernelTraits */ {} +, /* mLayoutA */ gemm::MatrixLayout(0) +, /* mLayoutB */ gemm::MatrixLayout(0) +, /* mM */ 256 +, /* mMmaK */ 32 +, /* mMmaKind */ trtllm::gen::MmaKind(5) +, /* mMmaM */ 128 +, /* mMmaN */ 8 +, /* mMmaTileK */ 256 +, /* mMockAllReduce */ 0 +, /* mN */ 256 +, /* mNumEpilogueWarps */ 4 +, /* mNumRegsCastAWarps */ 0 +, /* mNumRegsCopySfLdsSttm */ 0 +, /* mNumRegsCopySparsityInfo */ 0 +, /* mNumRegsPerThreadEpilogueWarp */ 0 +, /* mNumRegsPerThreadNonEpilogueWarp */ 0 +, /* mNumSlicesForSplitK */ 1 +, /* mNumStagesA */ 6 +, /* mNumStagesB */ 6 +, /* mNumStagesMma */ 2 +, /* mNumStagesMmaWithinWorkTile */ 1 +, /* mNumStagesMmaAcrossWorkTile */ 2 +, /* mNumStagesSmemSfA */ 6 +, /* mNumStagesSmemSfB */ 6 +, /* mNumStagesTmemSfA */ 6 +, /* mNumStagesTmemSfB */ 6 +, /* mNumStagesWorkId */ 3 +, /* mOutputDebugTensors */ 0 +, /* mPatchF2fp */ 0 +, /* mPerTokenSfDtype */ trtllm::gen::Dtype(1052672) +, /* mSchedHostTask */ gemm::SchedHostTask(0) +, /* mSfBlockSizeA */ 32 +, /* mSfBlockSizeB */ 32 +, /* mSfBlockSizeC */ 32 +, /* mSfLayoutA */ trtllm::gen::SfLayout(3) +, /* mSfLayoutB */ trtllm::gen::SfLayout(0) +, /* mSfLayoutC */ trtllm::gen::SfLayout(1) +, /* mSfReshapeFactor */ 1 +, /* mSparsityA */ trtllm::gen::Sparsity(0) +, /* mSplitK */ gemm::SplitK(0) +, /* mTileK */ 256 +, /* mTileM */ 128 +, /* mTileN */ 8 +, /* mTileScheduler */ gemm::TileScheduler(1) +, /* mTransposeMmaOutput */ 1 +, /* mUseCustomizedMma3xNvFp4 */ 0 +, /* mUseCustomMmaSchedule */ 1 +, /* mUseDeepSeekFp8 */ 0 +, /* mUseFlexibleClusterDims */ 0 +, /* mUseHoistTryWaitForCustomMmaSchedule */ 0 +, /* mUseMaxTmemOverlap */ 0 +, /* mUsePerTokenSfA */ 0 +, /* mUsePerTokenSfB */ 0 +, /* mUseShuffledMatrix */ 1 +, /* mUseTmaStore */ 1 +, /* mUseTwoTmaLoadWarps */ 1 +, /* mUseTwoMmaWarps */ 0 +, /* mUseUnrollLoop2xForMma */ 1 +, /* mValidM */ 256 +, /* mValidN */ 256 +, /* mValidK */ 512 +, /* mWorldSize */ 1 +, /* mActType */ gemmGatedAct::ActType(2) +, /* mClampBeforeAct */ 1 +, /* mBatchedM */ {} +, /* mBatchedN */ {} +, /* mBatchMode */ batchedGemm::BatchedGemmOptions::BatchMode(1) +, /* mBatchStrideInTokens */ -1 +, /* mFusedAct */ 1 +, /* mGridWaitForPrimaryRouting */ 1 +, /* mIsStaticBatch */ 0 +, /* mIsUniformNumTokensPerBatch */ 0 +, /* mNumBatches */ 128 +, /* mNumRegsPerThreadLoadA */ 0 +, /* mNumRegsPerThreadLoadB */ 0 +, /* mNumRegsPerThreadLoadSfA */ 0 +, /* mNumRegsPerThreadLoadSfB */ 0 +, /* mNumTokens */ 2 +, /* mNumWarpsLoadA */ 0 +, /* mNumWarpsLoadB */ 0 +, /* mNumWarpsLoadSfA */ 0 +, /* mNumWarpsLoadSfB */ 0 +, /* mRouteImpl */ batchedGemm::RouteImpl(2) +, /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} +, /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 + }, gemm::SmVersion::Sm100f}, +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 222096, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "53474c90b1714cf9ed273c68e43c1b2be5ae9b4612f523f10be7e88b577bd30d", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +, /* mBiasDtype */ trtllm::gen::Dtype(1056776) +, /* mBiasType */ gemm::BiasType(1) +, /* mBlockK */ -1 +, /* mClcFastDrain */ 1 +, /* mClusterDimX */ 1 +, /* mClusterDimY */ 1 +, /* mClusterDimZ */ 1 +, /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) +, /* mDtypeAcc */ trtllm::gen::Dtype(1056776) +, /* mDtypeA */ trtllm::gen::Dtype(17826828) +, /* mDtypeB */ trtllm::gen::Dtype(17827853) +, /* mDtypeC */ trtllm::gen::Dtype(17827853) +, /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) +, /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) +, /* mEltwiseActType */ gemm::EltwiseActType(0) +, /* mEnablesEarlyExit */ 1 +, /* mEnablesDelayedEarlyExit */ 0 +, /* mEpilogueLdtmDps */ 16 +, /* mEpilogueLdtmBits */ 256 +, /* mEpilogueTileM */ 128 +, /* mEpilogueTileN */ 8 +, /* mFallbackClusterDimX */ 1 +, /* mFallbackClusterDimY */ 1 +, /* mFallbackClusterDimZ */ 1 +, /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) +, /* mFuseLoadSfTask */ 0 +, /* mFuseUtccpWithUtcmma */ 0 +, /* mGridTriggerSecondaryA */ 0 +, /* mGridTriggerSecondaryB */ 1 +, /* mGridWaitForPrimaryEarlyExit */ 1 +, /* mGridWaitForPrimaryA */ 0 +, /* mGridWaitForPrimaryB */ 1 +, /* mHoistLoadTaskInit */ 1 +, /* mHoistMmaTaskTryWaits */ 0 +, /* mK */ 512 +, /* mKernelTraits */ {} +, /* mLayoutA */ gemm::MatrixLayout(0) +, /* mLayoutB */ gemm::MatrixLayout(0) +, /* mM */ 256 +, /* mMmaK */ 32 +, /* mMmaKind */ trtllm::gen::MmaKind(5) +, /* mMmaM */ 128 +, /* mMmaN */ 8 +, /* mMmaTileK */ 256 +, /* mMockAllReduce */ 0 +, /* mN */ 256 +, /* mNumEpilogueWarps */ 4 +, /* mNumRegsCastAWarps */ 0 +, /* mNumRegsCopySfLdsSttm */ 0 +, /* mNumRegsCopySparsityInfo */ 0 +, /* mNumRegsPerThreadEpilogueWarp */ 0 +, /* mNumRegsPerThreadNonEpilogueWarp */ 0 +, /* mNumSlicesForSplitK */ 1 +, /* mNumStagesA */ 6 +, /* mNumStagesB */ 6 +, /* mNumStagesMma */ 2 +, /* mNumStagesMmaWithinWorkTile */ 1 +, /* mNumStagesMmaAcrossWorkTile */ 2 +, /* mNumStagesSmemSfA */ 6 +, /* mNumStagesSmemSfB */ 6 +, /* mNumStagesTmemSfA */ 6 +, /* mNumStagesTmemSfB */ 6 +, /* mNumStagesWorkId */ 3 +, /* mOutputDebugTensors */ 0 +, /* mPatchF2fp */ 0 +, /* mPerTokenSfDtype */ trtllm::gen::Dtype(1052672) +, /* mSchedHostTask */ gemm::SchedHostTask(0) +, /* mSfBlockSizeA */ 32 +, /* mSfBlockSizeB */ 32 +, /* mSfBlockSizeC */ 32 +, /* mSfLayoutA */ trtllm::gen::SfLayout(3) +, /* mSfLayoutB */ trtllm::gen::SfLayout(0) +, /* mSfLayoutC */ trtllm::gen::SfLayout(1) +, /* mSfReshapeFactor */ 1 +, /* mSparsityA */ trtllm::gen::Sparsity(0) +, /* mSplitK */ gemm::SplitK(0) +, /* mTileK */ 256 +, /* mTileM */ 128 +, /* mTileN */ 8 +, /* mTileScheduler */ gemm::TileScheduler(1) +, /* mTransposeMmaOutput */ 1 +, /* mUseCustomizedMma3xNvFp4 */ 0 +, /* mUseCustomMmaSchedule */ 1 +, /* mUseDeepSeekFp8 */ 0 +, /* mUseFlexibleClusterDims */ 0 +, /* mUseHoistTryWaitForCustomMmaSchedule */ 0 +, /* mUseMaxTmemOverlap */ 0 +, /* mUsePerTokenSfA */ 0 +, /* mUsePerTokenSfB */ 0 +, /* mUseShuffledMatrix */ 1 +, /* mUseTmaStore */ 1 +, /* mUseTwoTmaLoadWarps */ 1 +, /* mUseTwoMmaWarps */ 0 +, /* mUseUnrollLoop2xForMma */ 1 +, /* mValidM */ 256 +, /* mValidN */ 256 +, /* mValidK */ 512 +, /* mWorldSize */ 1 +, /* mActType */ gemmGatedAct::ActType(0) +, /* mClampBeforeAct */ 1 +, /* mBatchedM */ {} +, /* mBatchedN */ {} +, /* mBatchMode */ batchedGemm::BatchedGemmOptions::BatchMode(1) +, /* mBatchStrideInTokens */ -1 +, /* mFusedAct */ 1 +, /* mGridWaitForPrimaryRouting */ 1 +, /* mIsStaticBatch */ 0 +, /* mIsUniformNumTokensPerBatch */ 0 +, /* mNumBatches */ 128 +, /* mNumRegsPerThreadLoadA */ 0 +, /* mNumRegsPerThreadLoadB */ 0 +, /* mNumRegsPerThreadLoadSfA */ 0 +, /* mNumRegsPerThreadLoadSfB */ 0 +, /* mNumTokens */ 2 +, /* mNumWarpsLoadA */ 0 +, /* mNumWarpsLoadB */ 0 +, /* mNumWarpsLoadSfA */ 0 +, /* mNumWarpsLoadSfB */ 0 +, /* mRouteImpl */ batchedGemm::RouteImpl(2) +, /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} +, /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 + }, gemm::SmVersion::Sm100f}, +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len, 221968, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f", 512, "e07a060965d5f7483ef97c78aaa350603acee827b350da4ef5a4e7be011c5935", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +, /* mBiasDtype */ trtllm::gen::Dtype(1056776) +, /* mBiasType */ gemm::BiasType(0) +, /* mBlockK */ -1 +, /* mClcFastDrain */ 1 +, /* mClusterDimX */ 1 +, /* mClusterDimY */ 1 +, /* mClusterDimZ */ 1 +, /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) +, /* mDtypeAcc */ trtllm::gen::Dtype(1056776) +, /* mDtypeA */ trtllm::gen::Dtype(17826828) +, /* mDtypeB */ trtllm::gen::Dtype(17827853) +, /* mDtypeC */ trtllm::gen::Dtype(17827853) +, /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) +, /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) +, /* mEltwiseActType */ gemm::EltwiseActType(0) +, /* mEnablesEarlyExit */ 1 +, /* mEnablesDelayedEarlyExit */ 0 +, /* mEpilogueLdtmDps */ 16 +, /* mEpilogueLdtmBits */ 256 +, /* mEpilogueTileM */ 128 +, /* mEpilogueTileN */ 8 +, /* mFallbackClusterDimX */ 1 +, /* mFallbackClusterDimY */ 1 +, /* mFallbackClusterDimZ */ 1 +, /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) +, /* mFuseLoadSfTask */ 0 +, /* mFuseUtccpWithUtcmma */ 0 +, /* mGridTriggerSecondaryA */ 0 +, /* mGridTriggerSecondaryB */ 1 +, /* mGridWaitForPrimaryEarlyExit */ 1 +, /* mGridWaitForPrimaryA */ 0 +, /* mGridWaitForPrimaryB */ 1 +, /* mHoistLoadTaskInit */ 1 +, /* mHoistMmaTaskTryWaits */ 0 +, /* mK */ 512 +, /* mKernelTraits */ {} +, /* mLayoutA */ gemm::MatrixLayout(0) +, /* mLayoutB */ gemm::MatrixLayout(0) +, /* mM */ 256 +, /* mMmaK */ 32 +, /* mMmaKind */ trtllm::gen::MmaKind(5) +, /* mMmaM */ 128 +, /* mMmaN */ 8 +, /* mMmaTileK */ 256 +, /* mMockAllReduce */ 0 +, /* mN */ 256 +, /* mNumEpilogueWarps */ 4 +, /* mNumRegsCastAWarps */ 0 +, /* mNumRegsCopySfLdsSttm */ 0 +, /* mNumRegsCopySparsityInfo */ 0 +, /* mNumRegsPerThreadEpilogueWarp */ 0 +, /* mNumRegsPerThreadNonEpilogueWarp */ 0 +, /* mNumSlicesForSplitK */ 1 +, /* mNumStagesA */ 6 +, /* mNumStagesB */ 6 +, /* mNumStagesMma */ 1 +, /* mNumStagesMmaWithinWorkTile */ 1 +, /* mNumStagesMmaAcrossWorkTile */ 1 +, /* mNumStagesSmemSfA */ 6 +, /* mNumStagesSmemSfB */ 6 +, /* mNumStagesTmemSfA */ 6 +, /* mNumStagesTmemSfB */ 6 +, /* mNumStagesWorkId */ 3 +, /* mOutputDebugTensors */ 0 +, /* mPatchF2fp */ 0 +, /* mPerTokenSfDtype */ trtllm::gen::Dtype(1052672) +, /* mSchedHostTask */ gemm::SchedHostTask(0) +, /* mSfBlockSizeA */ 32 +, /* mSfBlockSizeB */ 32 +, /* mSfBlockSizeC */ 32 +, /* mSfLayoutA */ trtllm::gen::SfLayout(3) +, /* mSfLayoutB */ trtllm::gen::SfLayout(0) +, /* mSfLayoutC */ trtllm::gen::SfLayout(1) +, /* mSfReshapeFactor */ 1 +, /* mSparsityA */ trtllm::gen::Sparsity(0) +, /* mSplitK */ gemm::SplitK(0) +, /* mTileK */ 256 +, /* mTileM */ 128 +, /* mTileN */ 8 +, /* mTileScheduler */ gemm::TileScheduler(0) +, /* mTransposeMmaOutput */ 1 +, /* mUseCustomizedMma3xNvFp4 */ 0 +, /* mUseCustomMmaSchedule */ 1 +, /* mUseDeepSeekFp8 */ 0 +, /* mUseFlexibleClusterDims */ 0 +, /* mUseHoistTryWaitForCustomMmaSchedule */ 0 +, /* mUseMaxTmemOverlap */ 0 +, /* mUsePerTokenSfA */ 0 +, /* mUsePerTokenSfB */ 0 +, /* mUseShuffledMatrix */ 1 +, /* mUseTmaStore */ 1 +, /* mUseTwoTmaLoadWarps */ 1 +, /* mUseTwoMmaWarps */ 0 +, /* mUseUnrollLoop2xForMma */ 1 +, /* mValidM */ 256 +, /* mValidN */ 256 +, /* mValidK */ 512 +, /* mWorldSize */ 1 +, /* mActType */ gemmGatedAct::ActType(2) +, /* mClampBeforeAct */ 1 +, /* mBatchedM */ {} +, /* mBatchedN */ {} +, /* mBatchMode */ batchedGemm::BatchedGemmOptions::BatchMode(1) +, /* mBatchStrideInTokens */ -1 +, /* mFusedAct */ 1 +, /* mGridWaitForPrimaryRouting */ 1 +, /* mIsStaticBatch */ 0 +, /* mIsUniformNumTokensPerBatch */ 0 +, /* mNumBatches */ 128 +, /* mNumRegsPerThreadLoadA */ 0 +, /* mNumRegsPerThreadLoadB */ 0 +, /* mNumRegsPerThreadLoadSfA */ 0 +, /* mNumRegsPerThreadLoadSfB */ 0 , /* mNumTokens */ 2 , /* mNumWarpsLoadA */ 0 -, /* mNumWarpsLoadB */ 1 +, /* mNumWarpsLoadB */ 0 , /* mNumWarpsLoadSfA */ 0 -, /* mNumWarpsLoadSfB */ 1 +, /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 222032, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "88e83259cb492acbc07753c63cfc378ea2455531ea8cda69076c60e51f760222", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 221968, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "2b6748125850b92b8fc220bcda73d14c28c53a95c7f5fa2e92c4ac751372a034", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -69991,10 +73916,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17827853) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -70012,7 +73937,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mGridWaitForPrimaryB */ 1 , /* mHoistLoadTaskInit */ 1 , /* mHoistMmaTaskTryWaits */ 0 -, /* mK */ 256 +, /* mK */ 512 , /* mKernelTraits */ {} , /* mLayoutA */ gemm::MatrixLayout(0) , /* mLayoutB */ gemm::MatrixLayout(0) @@ -70031,15 +73956,13 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 -, /* mNumStagesMma */ 2 +, /* mNumStagesMma */ 1 , /* mNumStagesMmaWithinWorkTile */ 1 -, /* mNumStagesMmaAcrossWorkTile */ 2 +, /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -70054,13 +73977,12 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 , /* mTileM */ 128 , /* mTileN */ 8 -, /* mTileScheduler */ gemm::TileScheduler(1) +, /* mTileScheduler */ gemm::TileScheduler(0) , /* mTransposeMmaOutput */ 1 , /* mUseCustomizedMma3xNvFp4 */ 0 , /* mUseCustomMmaSchedule */ 1 @@ -70074,10 +73996,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mUseTmaStore */ 1 , /* mUseTwoTmaLoadWarps */ 1 , /* mUseTwoMmaWarps */ 0 -, /* mUseUnrollLoop2xForMma */ 0 +, /* mUseUnrollLoop2xForMma */ 1 , /* mValidM */ 256 , /* mValidN */ 256 -, /* mValidK */ 256 +, /* mValidK */ 512 , /* mWorldSize */ 1 , /* mActType */ gemmGatedAct::ActType(0) , /* mClampBeforeAct */ 1 @@ -70101,17 +74023,18 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 221792, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "77419fe999b92cfa5849c611acc5e0ad2e43dcd94812f6e87f922e3dc0e98625", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len, 229272, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f", 512, "60a0ca4bb8bd8b77e68a324570b357e92f26d67577990b5e0e662ab1a56d7fcb", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) -, /* mBiasType */ gemm::BiasType(1) +, /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 , /* mClcFastDrain */ 1 , /* mClusterDimX */ 1 , /* mClusterDimY */ 1 -, /* mClusterDimZ */ 1 +, /* mClusterDimZ */ 2 , /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) , /* mDtypeAcc */ trtllm::gen::Dtype(1056776) , /* mDtypeA */ trtllm::gen::Dtype(17826828) @@ -70119,10 +74042,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17827853) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -70140,7 +74063,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mGridWaitForPrimaryB */ 1 , /* mHoistLoadTaskInit */ 1 , /* mHoistMmaTaskTryWaits */ 0 -, /* mK */ 256 +, /* mK */ 1024 , /* mKernelTraits */ {} , /* mLayoutA */ gemm::MatrixLayout(0) , /* mLayoutB */ gemm::MatrixLayout(0) @@ -70158,16 +74081,14 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsCopySparsityInfo */ 0 , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 -, /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 +, /* mNumSlicesForSplitK */ 2 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 -, /* mNumStagesMma */ 1 +, /* mNumStagesMma */ 2 , /* mNumStagesMmaWithinWorkTile */ 1 -, /* mNumStagesMmaAcrossWorkTile */ 1 +, /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -70182,13 +74103,12 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) -, /* mSplitK */ gemm::SplitK(0) +, /* mSplitK */ gemm::SplitK(2) , /* mTileK */ 256 , /* mTileM */ 128 , /* mTileN */ 8 -, /* mTileScheduler */ gemm::TileScheduler(0) +, /* mTileScheduler */ gemm::TileScheduler(1) , /* mTransposeMmaOutput */ 1 , /* mUseCustomizedMma3xNvFp4 */ 0 , /* mUseCustomMmaSchedule */ 1 @@ -70202,12 +74122,12 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mUseTmaStore */ 1 , /* mUseTwoTmaLoadWarps */ 1 , /* mUseTwoMmaWarps */ 0 -, /* mUseUnrollLoop2xForMma */ 0 +, /* mUseUnrollLoop2xForMma */ 1 , /* mValidM */ 256 , /* mValidN */ 256 -, /* mValidK */ 256 +, /* mValidK */ 1024 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(0) +, /* mActType */ gemmGatedAct::ActType(2) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -70229,10 +74149,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 229208, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "ad7959e13fb743833ed9d2853b261d7de6d8226987f07a556b241f971fc1ee63", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 229272, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "42dcc62d512a0e26161cf3aecc8ffab70b437e2990b8e13c3912c8d251f424f4", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -70247,10 +74168,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17827853) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -70268,7 +74189,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mGridWaitForPrimaryB */ 1 , /* mHoistLoadTaskInit */ 1 , /* mHoistMmaTaskTryWaits */ 0 -, /* mK */ 512 +, /* mK */ 1024 , /* mKernelTraits */ {} , /* mLayoutA */ gemm::MatrixLayout(0) , /* mLayoutB */ gemm::MatrixLayout(0) @@ -70287,7 +74208,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 2 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 2 @@ -70295,7 +74215,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -70310,7 +74229,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(2) , /* mTileK */ 256 @@ -70330,10 +74248,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mUseTmaStore */ 1 , /* mUseTwoTmaLoadWarps */ 1 , /* mUseTwoMmaWarps */ 0 -, /* mUseUnrollLoop2xForMma */ 0 +, /* mUseUnrollLoop2xForMma */ 1 , /* mValidM */ 256 , /* mValidN */ 256 -, /* mValidK */ 512 +, /* mValidK */ 1024 , /* mWorldSize */ 1 , /* mActType */ gemmGatedAct::ActType(0) , /* mClampBeforeAct */ 1 @@ -70357,10 +74275,137 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} +, /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 , /* mUseTmaOobOpt */ 1 + }, gemm::SmVersion::Sm100f}, +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len, 220952, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f", 512, "8c425a2ffd03008fadb054b8dc75a1b7a2a8596b3defe81c4edc3a1e4f5867e6", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +, /* mBiasDtype */ trtllm::gen::Dtype(1056776) +, /* mBiasType */ gemm::BiasType(0) +, /* mBlockK */ -1 +, /* mClcFastDrain */ 1 +, /* mClusterDimX */ 1 +, /* mClusterDimY */ 1 +, /* mClusterDimZ */ 2 +, /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) +, /* mDtypeAcc */ trtllm::gen::Dtype(1056776) +, /* mDtypeA */ trtllm::gen::Dtype(17826828) +, /* mDtypeB */ trtllm::gen::Dtype(17827853) +, /* mDtypeC */ trtllm::gen::Dtype(17827853) +, /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) +, /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) +, /* mEltwiseActType */ gemm::EltwiseActType(0) +, /* mEnablesEarlyExit */ 1 +, /* mEnablesDelayedEarlyExit */ 0 +, /* mEpilogueLdtmDps */ 16 +, /* mEpilogueLdtmBits */ 256 +, /* mEpilogueTileM */ 128 +, /* mEpilogueTileN */ 8 +, /* mFallbackClusterDimX */ 1 +, /* mFallbackClusterDimY */ 1 +, /* mFallbackClusterDimZ */ 1 +, /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) +, /* mFuseLoadSfTask */ 0 +, /* mFuseUtccpWithUtcmma */ 0 +, /* mGridTriggerSecondaryA */ 0 +, /* mGridTriggerSecondaryB */ 1 +, /* mGridWaitForPrimaryEarlyExit */ 1 +, /* mGridWaitForPrimaryA */ 0 +, /* mGridWaitForPrimaryB */ 1 +, /* mHoistLoadTaskInit */ 1 +, /* mHoistMmaTaskTryWaits */ 0 +, /* mK */ 1024 +, /* mKernelTraits */ {} +, /* mLayoutA */ gemm::MatrixLayout(0) +, /* mLayoutB */ gemm::MatrixLayout(0) +, /* mM */ 256 +, /* mMmaK */ 32 +, /* mMmaKind */ trtllm::gen::MmaKind(5) +, /* mMmaM */ 128 +, /* mMmaN */ 8 +, /* mMmaTileK */ 256 +, /* mMockAllReduce */ 0 +, /* mN */ 256 +, /* mNumEpilogueWarps */ 4 +, /* mNumRegsCastAWarps */ 0 +, /* mNumRegsCopySfLdsSttm */ 0 +, /* mNumRegsCopySparsityInfo */ 0 +, /* mNumRegsPerThreadEpilogueWarp */ 0 +, /* mNumRegsPerThreadNonEpilogueWarp */ 0 +, /* mNumSlicesForSplitK */ 2 +, /* mNumStagesA */ 6 +, /* mNumStagesB */ 6 +, /* mNumStagesMma */ 1 +, /* mNumStagesMmaWithinWorkTile */ 1 +, /* mNumStagesMmaAcrossWorkTile */ 1 +, /* mNumStagesSmemSfA */ 6 +, /* mNumStagesSmemSfB */ 6 +, /* mNumStagesTmemSfA */ 6 +, /* mNumStagesTmemSfB */ 6 +, /* mNumStagesWorkId */ 3 +, /* mOutputDebugTensors */ 0 +, /* mPatchF2fp */ 0 +, /* mPerTokenSfDtype */ trtllm::gen::Dtype(1052672) +, /* mSchedHostTask */ gemm::SchedHostTask(0) +, /* mSfBlockSizeA */ 32 +, /* mSfBlockSizeB */ 32 +, /* mSfBlockSizeC */ 32 +, /* mSfLayoutA */ trtllm::gen::SfLayout(3) +, /* mSfLayoutB */ trtllm::gen::SfLayout(0) +, /* mSfLayoutC */ trtllm::gen::SfLayout(1) +, /* mSfReshapeFactor */ 1 +, /* mSparsityA */ trtllm::gen::Sparsity(0) +, /* mSplitK */ gemm::SplitK(2) +, /* mTileK */ 256 +, /* mTileM */ 128 +, /* mTileN */ 8 +, /* mTileScheduler */ gemm::TileScheduler(0) +, /* mTransposeMmaOutput */ 1 +, /* mUseCustomizedMma3xNvFp4 */ 0 +, /* mUseCustomMmaSchedule */ 1 +, /* mUseDeepSeekFp8 */ 0 +, /* mUseFlexibleClusterDims */ 0 +, /* mUseHoistTryWaitForCustomMmaSchedule */ 0 +, /* mUseMaxTmemOverlap */ 0 +, /* mUsePerTokenSfA */ 0 +, /* mUsePerTokenSfB */ 0 +, /* mUseShuffledMatrix */ 1 +, /* mUseTmaStore */ 1 +, /* mUseTwoTmaLoadWarps */ 1 +, /* mUseTwoMmaWarps */ 0 +, /* mUseUnrollLoop2xForMma */ 1 +, /* mValidM */ 256 +, /* mValidN */ 256 +, /* mValidK */ 1024 +, /* mWorldSize */ 1 +, /* mActType */ gemmGatedAct::ActType(2) +, /* mClampBeforeAct */ 1 +, /* mBatchedM */ {} +, /* mBatchedN */ {} +, /* mBatchMode */ batchedGemm::BatchedGemmOptions::BatchMode(1) +, /* mBatchStrideInTokens */ -1 +, /* mFusedAct */ 1 +, /* mGridWaitForPrimaryRouting */ 1 +, /* mIsStaticBatch */ 0 +, /* mIsUniformNumTokensPerBatch */ 0 +, /* mNumBatches */ 128 +, /* mNumRegsPerThreadLoadA */ 0 +, /* mNumRegsPerThreadLoadB */ 0 +, /* mNumRegsPerThreadLoadSfA */ 0 +, /* mNumRegsPerThreadLoadSfB */ 0 +, /* mNumTokens */ 2 +, /* mNumWarpsLoadA */ 0 +, /* mNumWarpsLoadB */ 0 +, /* mNumWarpsLoadSfA */ 0 +, /* mNumWarpsLoadSfB */ 0 +, /* mRouteImpl */ batchedGemm::RouteImpl(2) +, /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 220776, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "9c6c48f95d524158c462858ca08f87f2695893646f34b0ddbbaf6c3ec727e463", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 220952, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "1a9392eece3afc1b8dd4da26998c85d9f598a2d098ab96de97eab2ab9a0af035", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -70375,10 +74420,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17827853) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -70396,7 +74441,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mGridWaitForPrimaryB */ 1 , /* mHoistLoadTaskInit */ 1 , /* mHoistMmaTaskTryWaits */ 0 -, /* mK */ 512 +, /* mK */ 1024 , /* mKernelTraits */ {} , /* mLayoutA */ gemm::MatrixLayout(0) , /* mLayoutB */ gemm::MatrixLayout(0) @@ -70415,7 +74460,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 2 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 6 , /* mNumStagesB */ 6 , /* mNumStagesMma */ 1 @@ -70423,7 +74467,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 6 , /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 6 , /* mNumStagesTmemSfB */ 6 , /* mNumStagesWorkId */ 3 @@ -70438,7 +74481,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(2) , /* mTileK */ 256 @@ -70458,6 +74500,258 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mUseTmaStore */ 1 , /* mUseTwoTmaLoadWarps */ 1 , /* mUseTwoMmaWarps */ 0 +, /* mUseUnrollLoop2xForMma */ 1 +, /* mValidM */ 256 +, /* mValidN */ 256 +, /* mValidK */ 1024 +, /* mWorldSize */ 1 +, /* mActType */ gemmGatedAct::ActType(0) +, /* mClampBeforeAct */ 1 +, /* mBatchedM */ {} +, /* mBatchedN */ {} +, /* mBatchMode */ batchedGemm::BatchedGemmOptions::BatchMode(1) +, /* mBatchStrideInTokens */ -1 +, /* mFusedAct */ 1 +, /* mGridWaitForPrimaryRouting */ 1 +, /* mIsStaticBatch */ 0 +, /* mIsUniformNumTokensPerBatch */ 0 +, /* mNumBatches */ 128 +, /* mNumRegsPerThreadLoadA */ 0 +, /* mNumRegsPerThreadLoadB */ 0 +, /* mNumRegsPerThreadLoadSfA */ 0 +, /* mNumRegsPerThreadLoadSfB */ 0 +, /* mNumTokens */ 2 +, /* mNumWarpsLoadA */ 0 +, /* mNumWarpsLoadB */ 0 +, /* mNumWarpsLoadSfA */ 0 +, /* mNumWarpsLoadSfB */ 0 +, /* mRouteImpl */ batchedGemm::RouteImpl(2) +, /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} +, /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 + }, gemm::SmVersion::Sm100f}, +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len, 221840, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f", 512, "d332e5598f37e6c678d1b76290f06fb775d7669aae02855db58b14ea1fbaa1b9", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +, /* mBiasDtype */ trtllm::gen::Dtype(1056776) +, /* mBiasType */ gemm::BiasType(0) +, /* mBlockK */ -1 +, /* mClcFastDrain */ 1 +, /* mClusterDimX */ 1 +, /* mClusterDimY */ 1 +, /* mClusterDimZ */ 1 +, /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) +, /* mDtypeAcc */ trtllm::gen::Dtype(1056776) +, /* mDtypeA */ trtllm::gen::Dtype(17826828) +, /* mDtypeB */ trtllm::gen::Dtype(17827853) +, /* mDtypeC */ trtllm::gen::Dtype(17827853) +, /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) +, /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) +, /* mEltwiseActType */ gemm::EltwiseActType(0) +, /* mEnablesEarlyExit */ 1 +, /* mEnablesDelayedEarlyExit */ 0 +, /* mEpilogueLdtmDps */ 16 +, /* mEpilogueLdtmBits */ 256 +, /* mEpilogueTileM */ 128 +, /* mEpilogueTileN */ 8 +, /* mFallbackClusterDimX */ 1 +, /* mFallbackClusterDimY */ 1 +, /* mFallbackClusterDimZ */ 1 +, /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) +, /* mFuseLoadSfTask */ 0 +, /* mFuseUtccpWithUtcmma */ 0 +, /* mGridTriggerSecondaryA */ 0 +, /* mGridTriggerSecondaryB */ 1 +, /* mGridWaitForPrimaryEarlyExit */ 1 +, /* mGridWaitForPrimaryA */ 0 +, /* mGridWaitForPrimaryB */ 1 +, /* mHoistLoadTaskInit */ 1 +, /* mHoistMmaTaskTryWaits */ 0 +, /* mK */ 512 +, /* mKernelTraits */ {} +, /* mLayoutA */ gemm::MatrixLayout(0) +, /* mLayoutB */ gemm::MatrixLayout(0) +, /* mM */ 256 +, /* mMmaK */ 32 +, /* mMmaKind */ trtllm::gen::MmaKind(5) +, /* mMmaM */ 128 +, /* mMmaN */ 8 +, /* mMmaTileK */ 512 +, /* mMockAllReduce */ 0 +, /* mN */ 256 +, /* mNumEpilogueWarps */ 4 +, /* mNumRegsCastAWarps */ 0 +, /* mNumRegsCopySfLdsSttm */ 0 +, /* mNumRegsCopySparsityInfo */ 0 +, /* mNumRegsPerThreadEpilogueWarp */ 0 +, /* mNumRegsPerThreadNonEpilogueWarp */ 0 +, /* mNumSlicesForSplitK */ 1 +, /* mNumStagesA */ 3 +, /* mNumStagesB */ 3 +, /* mNumStagesMma */ 2 +, /* mNumStagesMmaWithinWorkTile */ 1 +, /* mNumStagesMmaAcrossWorkTile */ 2 +, /* mNumStagesSmemSfA */ 3 +, /* mNumStagesSmemSfB */ 3 +, /* mNumStagesTmemSfA */ 3 +, /* mNumStagesTmemSfB */ 3 +, /* mNumStagesWorkId */ 3 +, /* mOutputDebugTensors */ 0 +, /* mPatchF2fp */ 0 +, /* mPerTokenSfDtype */ trtllm::gen::Dtype(1052672) +, /* mSchedHostTask */ gemm::SchedHostTask(0) +, /* mSfBlockSizeA */ 32 +, /* mSfBlockSizeB */ 32 +, /* mSfBlockSizeC */ 32 +, /* mSfLayoutA */ trtllm::gen::SfLayout(3) +, /* mSfLayoutB */ trtllm::gen::SfLayout(0) +, /* mSfLayoutC */ trtllm::gen::SfLayout(1) +, /* mSfReshapeFactor */ 1 +, /* mSparsityA */ trtllm::gen::Sparsity(0) +, /* mSplitK */ gemm::SplitK(0) +, /* mTileK */ 512 +, /* mTileM */ 128 +, /* mTileN */ 8 +, /* mTileScheduler */ gemm::TileScheduler(1) +, /* mTransposeMmaOutput */ 1 +, /* mUseCustomizedMma3xNvFp4 */ 0 +, /* mUseCustomMmaSchedule */ 1 +, /* mUseDeepSeekFp8 */ 0 +, /* mUseFlexibleClusterDims */ 0 +, /* mUseHoistTryWaitForCustomMmaSchedule */ 0 +, /* mUseMaxTmemOverlap */ 0 +, /* mUsePerTokenSfA */ 0 +, /* mUsePerTokenSfB */ 0 +, /* mUseShuffledMatrix */ 1 +, /* mUseTmaStore */ 1 +, /* mUseTwoTmaLoadWarps */ 1 +, /* mUseTwoMmaWarps */ 0 +, /* mUseUnrollLoop2xForMma */ 0 +, /* mValidM */ 256 +, /* mValidN */ 256 +, /* mValidK */ 512 +, /* mWorldSize */ 1 +, /* mActType */ gemmGatedAct::ActType(2) +, /* mClampBeforeAct */ 1 +, /* mBatchedM */ {} +, /* mBatchedN */ {} +, /* mBatchMode */ batchedGemm::BatchedGemmOptions::BatchMode(1) +, /* mBatchStrideInTokens */ -1 +, /* mFusedAct */ 1 +, /* mGridWaitForPrimaryRouting */ 1 +, /* mIsStaticBatch */ 0 +, /* mIsUniformNumTokensPerBatch */ 0 +, /* mNumBatches */ 128 +, /* mNumRegsPerThreadLoadA */ 0 +, /* mNumRegsPerThreadLoadB */ 0 +, /* mNumRegsPerThreadLoadSfA */ 0 +, /* mNumRegsPerThreadLoadSfB */ 0 +, /* mNumTokens */ 2 +, /* mNumWarpsLoadA */ 0 +, /* mNumWarpsLoadB */ 0 +, /* mNumWarpsLoadSfA */ 0 +, /* mNumWarpsLoadSfB */ 0 +, /* mRouteImpl */ batchedGemm::RouteImpl(2) +, /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} +, /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 + }, gemm::SmVersion::Sm100f}, +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 221840, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "774916dc6f096cb494f14ce9501f962ac35fa1ed3a785256d64afee6bab661da", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +, /* mBiasDtype */ trtllm::gen::Dtype(1056776) +, /* mBiasType */ gemm::BiasType(1) +, /* mBlockK */ -1 +, /* mClcFastDrain */ 1 +, /* mClusterDimX */ 1 +, /* mClusterDimY */ 1 +, /* mClusterDimZ */ 1 +, /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) +, /* mDtypeAcc */ trtllm::gen::Dtype(1056776) +, /* mDtypeA */ trtllm::gen::Dtype(17826828) +, /* mDtypeB */ trtllm::gen::Dtype(17827853) +, /* mDtypeC */ trtllm::gen::Dtype(17827853) +, /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) +, /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) +, /* mEltwiseActType */ gemm::EltwiseActType(0) +, /* mEnablesEarlyExit */ 1 +, /* mEnablesDelayedEarlyExit */ 0 +, /* mEpilogueLdtmDps */ 16 +, /* mEpilogueLdtmBits */ 256 +, /* mEpilogueTileM */ 128 +, /* mEpilogueTileN */ 8 +, /* mFallbackClusterDimX */ 1 +, /* mFallbackClusterDimY */ 1 +, /* mFallbackClusterDimZ */ 1 +, /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) +, /* mFuseLoadSfTask */ 0 +, /* mFuseUtccpWithUtcmma */ 0 +, /* mGridTriggerSecondaryA */ 0 +, /* mGridTriggerSecondaryB */ 1 +, /* mGridWaitForPrimaryEarlyExit */ 1 +, /* mGridWaitForPrimaryA */ 0 +, /* mGridWaitForPrimaryB */ 1 +, /* mHoistLoadTaskInit */ 1 +, /* mHoistMmaTaskTryWaits */ 0 +, /* mK */ 512 +, /* mKernelTraits */ {} +, /* mLayoutA */ gemm::MatrixLayout(0) +, /* mLayoutB */ gemm::MatrixLayout(0) +, /* mM */ 256 +, /* mMmaK */ 32 +, /* mMmaKind */ trtllm::gen::MmaKind(5) +, /* mMmaM */ 128 +, /* mMmaN */ 8 +, /* mMmaTileK */ 512 +, /* mMockAllReduce */ 0 +, /* mN */ 256 +, /* mNumEpilogueWarps */ 4 +, /* mNumRegsCastAWarps */ 0 +, /* mNumRegsCopySfLdsSttm */ 0 +, /* mNumRegsCopySparsityInfo */ 0 +, /* mNumRegsPerThreadEpilogueWarp */ 0 +, /* mNumRegsPerThreadNonEpilogueWarp */ 0 +, /* mNumSlicesForSplitK */ 1 +, /* mNumStagesA */ 3 +, /* mNumStagesB */ 3 +, /* mNumStagesMma */ 2 +, /* mNumStagesMmaWithinWorkTile */ 1 +, /* mNumStagesMmaAcrossWorkTile */ 2 +, /* mNumStagesSmemSfA */ 3 +, /* mNumStagesSmemSfB */ 3 +, /* mNumStagesTmemSfA */ 3 +, /* mNumStagesTmemSfB */ 3 +, /* mNumStagesWorkId */ 3 +, /* mOutputDebugTensors */ 0 +, /* mPatchF2fp */ 0 +, /* mPerTokenSfDtype */ trtllm::gen::Dtype(1052672) +, /* mSchedHostTask */ gemm::SchedHostTask(0) +, /* mSfBlockSizeA */ 32 +, /* mSfBlockSizeB */ 32 +, /* mSfBlockSizeC */ 32 +, /* mSfLayoutA */ trtllm::gen::SfLayout(3) +, /* mSfLayoutB */ trtllm::gen::SfLayout(0) +, /* mSfLayoutC */ trtllm::gen::SfLayout(1) +, /* mSfReshapeFactor */ 1 +, /* mSparsityA */ trtllm::gen::Sparsity(0) +, /* mSplitK */ gemm::SplitK(0) +, /* mTileK */ 512 +, /* mTileM */ 128 +, /* mTileN */ 8 +, /* mTileScheduler */ gemm::TileScheduler(1) +, /* mTransposeMmaOutput */ 1 +, /* mUseCustomizedMma3xNvFp4 */ 0 +, /* mUseCustomMmaSchedule */ 1 +, /* mUseDeepSeekFp8 */ 0 +, /* mUseFlexibleClusterDims */ 0 +, /* mUseHoistTryWaitForCustomMmaSchedule */ 0 +, /* mUseMaxTmemOverlap */ 0 +, /* mUsePerTokenSfA */ 0 +, /* mUsePerTokenSfB */ 0 +, /* mUseShuffledMatrix */ 1 +, /* mUseTmaStore */ 1 +, /* mUseTwoTmaLoadWarps */ 1 +, /* mUseTwoMmaWarps */ 0 , /* mUseUnrollLoop2xForMma */ 0 , /* mValidM */ 256 , /* mValidN */ 256 @@ -70485,12 +74779,13 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 222032, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "b52baa82baaa660c17181885a37c79362317f56a637e7e608a07affb820dc7d3", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len, 221712, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f", 512, "40b596f363fdecd4645e7d748540a4b2899e4922bcf1e084149857c6495d5056", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) -, /* mBiasType */ gemm::BiasType(1) +, /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 , /* mClcFastDrain */ 1 , /* mClusterDimX */ 1 @@ -70503,10 +74798,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17827853) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -70533,7 +74828,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mMmaKind */ trtllm::gen::MmaKind(5) , /* mMmaM */ 128 , /* mMmaN */ 8 -, /* mMmaTileK */ 256 +, /* mMmaTileK */ 512 , /* mMockAllReduce */ 0 , /* mN */ 256 , /* mNumEpilogueWarps */ 4 @@ -70543,17 +74838,15 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 -, /* mNumStagesA */ 6 -, /* mNumStagesB */ 6 -, /* mNumStagesMma */ 2 +, /* mNumStagesA */ 3 +, /* mNumStagesB */ 3 +, /* mNumStagesMma */ 1 , /* mNumStagesMmaWithinWorkTile */ 1 -, /* mNumStagesMmaAcrossWorkTile */ 2 -, /* mNumStagesSmemSfA */ 6 -, /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 -, /* mNumStagesTmemSfA */ 6 -, /* mNumStagesTmemSfB */ 6 +, /* mNumStagesMmaAcrossWorkTile */ 1 +, /* mNumStagesSmemSfA */ 3 +, /* mNumStagesSmemSfB */ 3 +, /* mNumStagesTmemSfA */ 3 +, /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 , /* mOutputDebugTensors */ 0 , /* mPatchF2fp */ 0 @@ -70566,13 +74859,12 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) -, /* mTileK */ 256 +, /* mTileK */ 512 , /* mTileM */ 128 , /* mTileN */ 8 -, /* mTileScheduler */ gemm::TileScheduler(1) +, /* mTileScheduler */ gemm::TileScheduler(0) , /* mTransposeMmaOutput */ 1 , /* mUseCustomizedMma3xNvFp4 */ 0 , /* mUseCustomMmaSchedule */ 1 @@ -70586,12 +74878,12 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mUseTmaStore */ 1 , /* mUseTwoTmaLoadWarps */ 1 , /* mUseTwoMmaWarps */ 0 -, /* mUseUnrollLoop2xForMma */ 1 +, /* mUseUnrollLoop2xForMma */ 0 , /* mValidM */ 256 , /* mValidN */ 256 , /* mValidK */ 512 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(0) +, /* mActType */ gemmGatedAct::ActType(2) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -70613,10 +74905,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 221792, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "a131019c7ea6b47bd64103dd45cfcf6cf991b99416938c7810c4fb20438699ed", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 221712, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "9f3b43f9e4d30b9baf4631ea25b87cdda9cfd262657bd34f325c1fc87ee9462c", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -70631,10 +74924,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17827853) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -70661,7 +74954,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mMmaKind */ trtllm::gen::MmaKind(5) , /* mMmaM */ 128 , /* mMmaN */ 8 -, /* mMmaTileK */ 256 +, /* mMmaTileK */ 512 , /* mMockAllReduce */ 0 , /* mN */ 256 , /* mNumEpilogueWarps */ 4 @@ -70671,17 +74964,15 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 -, /* mNumStagesA */ 6 -, /* mNumStagesB */ 6 +, /* mNumStagesA */ 3 +, /* mNumStagesB */ 3 , /* mNumStagesMma */ 1 , /* mNumStagesMmaWithinWorkTile */ 1 , /* mNumStagesMmaAcrossWorkTile */ 1 -, /* mNumStagesSmemSfA */ 6 -, /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 -, /* mNumStagesTmemSfA */ 6 -, /* mNumStagesTmemSfB */ 6 +, /* mNumStagesSmemSfA */ 3 +, /* mNumStagesSmemSfB */ 3 +, /* mNumStagesTmemSfA */ 3 +, /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 , /* mOutputDebugTensors */ 0 , /* mPatchF2fp */ 0 @@ -70694,10 +74985,9 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) -, /* mTileK */ 256 +, /* mTileK */ 512 , /* mTileM */ 128 , /* mTileN */ 8 , /* mTileScheduler */ gemm::TileScheduler(0) @@ -70714,7 +75004,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mUseTmaStore */ 1 , /* mUseTwoTmaLoadWarps */ 1 , /* mUseTwoMmaWarps */ 0 -, /* mUseUnrollLoop2xForMma */ 1 +, /* mUseUnrollLoop2xForMma */ 0 , /* mValidM */ 256 , /* mValidN */ 256 , /* mValidK */ 512 @@ -70741,10 +75031,137 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} +, /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 , /* mUseTmaOobOpt */ 1 + }, gemm::SmVersion::Sm100f}, +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len, 229016, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f", 512, "d862d8d888775abd117a86faced9b029d3fa8caff245e9b8c2c2e291566eded8", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +, /* mBiasDtype */ trtllm::gen::Dtype(1056776) +, /* mBiasType */ gemm::BiasType(0) +, /* mBlockK */ -1 +, /* mClcFastDrain */ 1 +, /* mClusterDimX */ 1 +, /* mClusterDimY */ 1 +, /* mClusterDimZ */ 2 +, /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) +, /* mDtypeAcc */ trtllm::gen::Dtype(1056776) +, /* mDtypeA */ trtllm::gen::Dtype(17826828) +, /* mDtypeB */ trtllm::gen::Dtype(17827853) +, /* mDtypeC */ trtllm::gen::Dtype(17827853) +, /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) +, /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) +, /* mEltwiseActType */ gemm::EltwiseActType(0) +, /* mEnablesEarlyExit */ 1 +, /* mEnablesDelayedEarlyExit */ 0 +, /* mEpilogueLdtmDps */ 16 +, /* mEpilogueLdtmBits */ 256 +, /* mEpilogueTileM */ 128 +, /* mEpilogueTileN */ 8 +, /* mFallbackClusterDimX */ 1 +, /* mFallbackClusterDimY */ 1 +, /* mFallbackClusterDimZ */ 1 +, /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) +, /* mFuseLoadSfTask */ 0 +, /* mFuseUtccpWithUtcmma */ 0 +, /* mGridTriggerSecondaryA */ 0 +, /* mGridTriggerSecondaryB */ 1 +, /* mGridWaitForPrimaryEarlyExit */ 1 +, /* mGridWaitForPrimaryA */ 0 +, /* mGridWaitForPrimaryB */ 1 +, /* mHoistLoadTaskInit */ 1 +, /* mHoistMmaTaskTryWaits */ 0 +, /* mK */ 1024 +, /* mKernelTraits */ {} +, /* mLayoutA */ gemm::MatrixLayout(0) +, /* mLayoutB */ gemm::MatrixLayout(0) +, /* mM */ 256 +, /* mMmaK */ 32 +, /* mMmaKind */ trtllm::gen::MmaKind(5) +, /* mMmaM */ 128 +, /* mMmaN */ 8 +, /* mMmaTileK */ 512 +, /* mMockAllReduce */ 0 +, /* mN */ 256 +, /* mNumEpilogueWarps */ 4 +, /* mNumRegsCastAWarps */ 0 +, /* mNumRegsCopySfLdsSttm */ 0 +, /* mNumRegsCopySparsityInfo */ 0 +, /* mNumRegsPerThreadEpilogueWarp */ 0 +, /* mNumRegsPerThreadNonEpilogueWarp */ 0 +, /* mNumSlicesForSplitK */ 2 +, /* mNumStagesA */ 3 +, /* mNumStagesB */ 3 +, /* mNumStagesMma */ 2 +, /* mNumStagesMmaWithinWorkTile */ 1 +, /* mNumStagesMmaAcrossWorkTile */ 2 +, /* mNumStagesSmemSfA */ 3 +, /* mNumStagesSmemSfB */ 3 +, /* mNumStagesTmemSfA */ 3 +, /* mNumStagesTmemSfB */ 3 +, /* mNumStagesWorkId */ 3 +, /* mOutputDebugTensors */ 0 +, /* mPatchF2fp */ 0 +, /* mPerTokenSfDtype */ trtllm::gen::Dtype(1052672) +, /* mSchedHostTask */ gemm::SchedHostTask(0) +, /* mSfBlockSizeA */ 32 +, /* mSfBlockSizeB */ 32 +, /* mSfBlockSizeC */ 32 +, /* mSfLayoutA */ trtllm::gen::SfLayout(3) +, /* mSfLayoutB */ trtllm::gen::SfLayout(0) +, /* mSfLayoutC */ trtllm::gen::SfLayout(1) +, /* mSfReshapeFactor */ 1 +, /* mSparsityA */ trtllm::gen::Sparsity(0) +, /* mSplitK */ gemm::SplitK(2) +, /* mTileK */ 512 +, /* mTileM */ 128 +, /* mTileN */ 8 +, /* mTileScheduler */ gemm::TileScheduler(1) +, /* mTransposeMmaOutput */ 1 +, /* mUseCustomizedMma3xNvFp4 */ 0 +, /* mUseCustomMmaSchedule */ 1 +, /* mUseDeepSeekFp8 */ 0 +, /* mUseFlexibleClusterDims */ 0 +, /* mUseHoistTryWaitForCustomMmaSchedule */ 0 +, /* mUseMaxTmemOverlap */ 0 +, /* mUsePerTokenSfA */ 0 +, /* mUsePerTokenSfB */ 0 +, /* mUseShuffledMatrix */ 1 +, /* mUseTmaStore */ 1 +, /* mUseTwoTmaLoadWarps */ 1 +, /* mUseTwoMmaWarps */ 0 +, /* mUseUnrollLoop2xForMma */ 0 +, /* mValidM */ 256 +, /* mValidN */ 256 +, /* mValidK */ 1024 +, /* mWorldSize */ 1 +, /* mActType */ gemmGatedAct::ActType(2) +, /* mClampBeforeAct */ 1 +, /* mBatchedM */ {} +, /* mBatchedN */ {} +, /* mBatchMode */ batchedGemm::BatchedGemmOptions::BatchMode(1) +, /* mBatchStrideInTokens */ -1 +, /* mFusedAct */ 1 +, /* mGridWaitForPrimaryRouting */ 1 +, /* mIsStaticBatch */ 0 +, /* mIsUniformNumTokensPerBatch */ 0 +, /* mNumBatches */ 128 +, /* mNumRegsPerThreadLoadA */ 0 +, /* mNumRegsPerThreadLoadB */ 0 +, /* mNumRegsPerThreadLoadSfA */ 0 +, /* mNumRegsPerThreadLoadSfB */ 0 +, /* mNumTokens */ 2 +, /* mNumWarpsLoadA */ 0 +, /* mNumWarpsLoadB */ 0 +, /* mNumWarpsLoadSfA */ 0 +, /* mNumWarpsLoadSfB */ 0 +, /* mRouteImpl */ batchedGemm::RouteImpl(2) +, /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 229208, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "24606cb5cf12049b486e298b04bd0f306ad7746ac52522f53ee63152a56c006e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 229016, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "3626a50a93aa53dd237c3d7e8ef6f0827ae8bd5deadbb7902bc023f73a405237", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -70759,10 +75176,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17827853) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -70789,7 +75206,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mMmaKind */ trtllm::gen::MmaKind(5) , /* mMmaM */ 128 , /* mMmaN */ 8 -, /* mMmaTileK */ 256 +, /* mMmaTileK */ 512 , /* mMockAllReduce */ 0 , /* mN */ 256 , /* mNumEpilogueWarps */ 4 @@ -70799,17 +75216,15 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 2 -, /* mNumSlicesForSliceK */ 1 -, /* mNumStagesA */ 6 -, /* mNumStagesB */ 6 +, /* mNumStagesA */ 3 +, /* mNumStagesB */ 3 , /* mNumStagesMma */ 2 , /* mNumStagesMmaWithinWorkTile */ 1 , /* mNumStagesMmaAcrossWorkTile */ 2 -, /* mNumStagesSmemSfA */ 6 -, /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 -, /* mNumStagesTmemSfA */ 6 -, /* mNumStagesTmemSfB */ 6 +, /* mNumStagesSmemSfA */ 3 +, /* mNumStagesSmemSfB */ 3 +, /* mNumStagesTmemSfA */ 3 +, /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 , /* mOutputDebugTensors */ 0 , /* mPatchF2fp */ 0 @@ -70822,10 +75237,9 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(2) -, /* mTileK */ 256 +, /* mTileK */ 512 , /* mTileM */ 128 , /* mTileN */ 8 , /* mTileScheduler */ gemm::TileScheduler(1) @@ -70842,7 +75256,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mUseTmaStore */ 1 , /* mUseTwoTmaLoadWarps */ 1 , /* mUseTwoMmaWarps */ 0 -, /* mUseUnrollLoop2xForMma */ 1 +, /* mUseUnrollLoop2xForMma */ 0 , /* mValidM */ 256 , /* mValidN */ 256 , /* mValidK */ 1024 @@ -70869,12 +75283,13 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 220776, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "fee3a9b2d1dc2ec774786f8dcd1a4febb5ba9ff8db9d19c7e19c084f366e375c", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len, 220696, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f", 512, "9dcd8f6398c371c1430a165162aeac426551277ac0e23402a87dd94828e3674e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) -, /* mBiasType */ gemm::BiasType(1) +, /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 , /* mClcFastDrain */ 1 , /* mClusterDimX */ 1 @@ -70887,10 +75302,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17827853) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -70917,7 +75332,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mMmaKind */ trtllm::gen::MmaKind(5) , /* mMmaM */ 128 , /* mMmaN */ 8 -, /* mMmaTileK */ 256 +, /* mMmaTileK */ 512 , /* mMockAllReduce */ 0 , /* mN */ 256 , /* mNumEpilogueWarps */ 4 @@ -70927,17 +75342,15 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 2 -, /* mNumSlicesForSliceK */ 1 -, /* mNumStagesA */ 6 -, /* mNumStagesB */ 6 +, /* mNumStagesA */ 3 +, /* mNumStagesB */ 3 , /* mNumStagesMma */ 1 , /* mNumStagesMmaWithinWorkTile */ 1 , /* mNumStagesMmaAcrossWorkTile */ 1 -, /* mNumStagesSmemSfA */ 6 -, /* mNumStagesSmemSfB */ 6 -, /* mNumStagesTmemA */ 0 -, /* mNumStagesTmemSfA */ 6 -, /* mNumStagesTmemSfB */ 6 +, /* mNumStagesSmemSfA */ 3 +, /* mNumStagesSmemSfB */ 3 +, /* mNumStagesTmemSfA */ 3 +, /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 , /* mOutputDebugTensors */ 0 , /* mPatchF2fp */ 0 @@ -70950,10 +75363,9 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(2) -, /* mTileK */ 256 +, /* mTileK */ 512 , /* mTileM */ 128 , /* mTileN */ 8 , /* mTileScheduler */ gemm::TileScheduler(0) @@ -70970,12 +75382,12 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mUseTmaStore */ 1 , /* mUseTwoTmaLoadWarps */ 1 , /* mUseTwoMmaWarps */ 0 -, /* mUseUnrollLoop2xForMma */ 1 +, /* mUseUnrollLoop2xForMma */ 0 , /* mValidM */ 256 , /* mValidN */ 256 , /* mValidK */ 1024 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(0) +, /* mActType */ gemmGatedAct::ActType(2) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -70997,17 +75409,18 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 221744, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "d18537301f376571aec1c1bbfd6ed7f15e80f19bddad6739261a281b7fc8bf12", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 220696, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "1720f6123c92e49216693c6f04a91c106b1bd9ab55cb2fe7843ab8a722be0127", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 , /* mClcFastDrain */ 1 , /* mClusterDimX */ 1 , /* mClusterDimY */ 1 -, /* mClusterDimZ */ 1 +, /* mClusterDimZ */ 2 , /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) , /* mDtypeAcc */ trtllm::gen::Dtype(1056776) , /* mDtypeA */ trtllm::gen::Dtype(17826828) @@ -71015,10 +75428,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17827853) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -71036,7 +75449,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mGridWaitForPrimaryB */ 1 , /* mHoistLoadTaskInit */ 1 , /* mHoistMmaTaskTryWaits */ 0 -, /* mK */ 512 +, /* mK */ 1024 , /* mKernelTraits */ {} , /* mLayoutA */ gemm::MatrixLayout(0) , /* mLayoutB */ gemm::MatrixLayout(0) @@ -71054,16 +75467,14 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsCopySparsityInfo */ 0 , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 -, /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 +, /* mNumSlicesForSplitK */ 2 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 -, /* mNumStagesMma */ 2 +, /* mNumStagesMma */ 1 , /* mNumStagesMmaWithinWorkTile */ 1 -, /* mNumStagesMmaAcrossWorkTile */ 2 +, /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -71078,13 +75489,12 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) -, /* mSplitK */ gemm::SplitK(0) +, /* mSplitK */ gemm::SplitK(2) , /* mTileK */ 512 , /* mTileM */ 128 , /* mTileN */ 8 -, /* mTileScheduler */ gemm::TileScheduler(1) +, /* mTileScheduler */ gemm::TileScheduler(0) , /* mTransposeMmaOutput */ 1 , /* mUseCustomizedMma3xNvFp4 */ 0 , /* mUseCustomMmaSchedule */ 1 @@ -71101,7 +75511,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mUseUnrollLoop2xForMma */ 0 , /* mValidM */ 256 , /* mValidN */ 256 -, /* mValidK */ 512 +, /* mValidK */ 1024 , /* mWorldSize */ 1 , /* mActType */ gemmGatedAct::ActType(0) , /* mClampBeforeAct */ 1 @@ -71125,12 +75535,13 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 221504, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "98e57993828cee29a1f0257d743a7207ac4d2bfa987cbcf28ff73838e12574bc", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len, 221840, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f", 512, "66d3753fea1dfeafa5fe7f107923fb20380a9cd153979c0bbfa19e58edec1767", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) -, /* mBiasType */ gemm::BiasType(1) +, /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 , /* mClcFastDrain */ 1 , /* mClusterDimX */ 1 @@ -71143,10 +75554,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17827853) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -71164,7 +75575,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mGridWaitForPrimaryB */ 1 , /* mHoistLoadTaskInit */ 1 , /* mHoistMmaTaskTryWaits */ 0 -, /* mK */ 512 +, /* mK */ 1024 , /* mKernelTraits */ {} , /* mLayoutA */ gemm::MatrixLayout(0) , /* mLayoutB */ gemm::MatrixLayout(0) @@ -71183,15 +75594,13 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 -, /* mNumStagesMma */ 1 +, /* mNumStagesMma */ 2 , /* mNumStagesMmaWithinWorkTile */ 1 -, /* mNumStagesMmaAcrossWorkTile */ 1 +, /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -71206,13 +75615,12 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 , /* mTileM */ 128 , /* mTileN */ 8 -, /* mTileScheduler */ gemm::TileScheduler(0) +, /* mTileScheduler */ gemm::TileScheduler(1) , /* mTransposeMmaOutput */ 1 , /* mUseCustomizedMma3xNvFp4 */ 0 , /* mUseCustomMmaSchedule */ 1 @@ -71226,12 +75634,12 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mUseTmaStore */ 1 , /* mUseTwoTmaLoadWarps */ 1 , /* mUseTwoMmaWarps */ 0 -, /* mUseUnrollLoop2xForMma */ 0 +, /* mUseUnrollLoop2xForMma */ 1 , /* mValidM */ 256 , /* mValidN */ 256 -, /* mValidK */ 512 +, /* mValidK */ 1024 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(0) +, /* mActType */ gemmGatedAct::ActType(2) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -71253,17 +75661,18 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 228920, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "d6355671fe4f0d9ff0b51c41866156f4ea96850e9a8a71da6f52f2d0f74a4c0e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 221840, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "75473500f82ba217367c76c935411d7ff28c35d3d614ece97a84f236040f7f30", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 , /* mClcFastDrain */ 1 , /* mClusterDimX */ 1 , /* mClusterDimY */ 1 -, /* mClusterDimZ */ 2 +, /* mClusterDimZ */ 1 , /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) , /* mDtypeAcc */ trtllm::gen::Dtype(1056776) , /* mDtypeA */ trtllm::gen::Dtype(17826828) @@ -71271,10 +75680,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17827853) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -71310,8 +75719,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsCopySparsityInfo */ 0 , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 -, /* mNumSlicesForSplitK */ 2 -, /* mNumSlicesForSliceK */ 1 +, /* mNumSlicesForSplitK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 2 @@ -71319,7 +75727,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -71334,9 +75741,8 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) -, /* mSplitK */ gemm::SplitK(2) +, /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 , /* mTileM */ 128 , /* mTileN */ 8 @@ -71354,7 +75760,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mUseTmaStore */ 1 , /* mUseTwoTmaLoadWarps */ 1 , /* mUseTwoMmaWarps */ 0 -, /* mUseUnrollLoop2xForMma */ 0 +, /* mUseUnrollLoop2xForMma */ 1 , /* mValidM */ 256 , /* mValidN */ 256 , /* mValidK */ 1024 @@ -71381,17 +75787,18 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 220488, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "bf3b717aeedfd94583a4a5fd51f7476ee563dd33c73a1490f7f215ac00dcf9d9", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len, 221712, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f", 512, "f3303c9bae3db25e963f01a79aa9d2e25719dbeb0684bdeeedbf720f19a6d749", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) -, /* mBiasType */ gemm::BiasType(1) +, /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 , /* mClcFastDrain */ 1 , /* mClusterDimX */ 1 , /* mClusterDimY */ 1 -, /* mClusterDimZ */ 2 +, /* mClusterDimZ */ 1 , /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) , /* mDtypeAcc */ trtllm::gen::Dtype(1056776) , /* mDtypeA */ trtllm::gen::Dtype(17826828) @@ -71399,10 +75806,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17827853) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -71438,8 +75845,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsCopySparsityInfo */ 0 , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 -, /* mNumSlicesForSplitK */ 2 -, /* mNumSlicesForSliceK */ 1 +, /* mNumSlicesForSplitK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 1 @@ -71447,7 +75853,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -71462,9 +75867,8 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) -, /* mSplitK */ gemm::SplitK(2) +, /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 , /* mTileM */ 128 , /* mTileN */ 8 @@ -71482,12 +75886,12 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mUseTmaStore */ 1 , /* mUseTwoTmaLoadWarps */ 1 , /* mUseTwoMmaWarps */ 0 -, /* mUseUnrollLoop2xForMma */ 0 +, /* mUseUnrollLoop2xForMma */ 1 , /* mValidM */ 256 , /* mValidN */ 256 , /* mValidK */ 1024 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(0) +, /* mActType */ gemmGatedAct::ActType(2) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -71509,10 +75913,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 221744, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "19c464ae31c83d57f3163cd2a7102f6f4ec7a8db14f4821ad16b521b361a3040", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 221712, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "dfa5825e824237d81c091c669c32defbda27a8a7c7e96d9bb1aede6b3368adde", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -71527,10 +75932,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17827853) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -71567,15 +75972,13 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 -, /* mNumStagesMma */ 2 +, /* mNumStagesMma */ 1 , /* mNumStagesMmaWithinWorkTile */ 1 -, /* mNumStagesMmaAcrossWorkTile */ 2 +, /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -71590,13 +75993,12 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 , /* mTileM */ 128 , /* mTileN */ 8 -, /* mTileScheduler */ gemm::TileScheduler(1) +, /* mTileScheduler */ gemm::TileScheduler(0) , /* mTransposeMmaOutput */ 1 , /* mUseCustomizedMma3xNvFp4 */ 0 , /* mUseCustomMmaSchedule */ 1 @@ -71637,17 +76039,18 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 221504, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "45b919468aa7024f711858d7d466e4ae3ef54fabacc634860191a2685b8e379f", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len, 229016, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f", 512, "e9c21220289b892c23c385261784818548c578a5841e46b87db89718b1082ea0", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) -, /* mBiasType */ gemm::BiasType(1) +, /* mBiasType */ gemm::BiasType(0) , /* mBlockK */ -1 , /* mClcFastDrain */ 1 , /* mClusterDimX */ 1 , /* mClusterDimY */ 1 -, /* mClusterDimZ */ 1 +, /* mClusterDimZ */ 2 , /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) , /* mDtypeAcc */ trtllm::gen::Dtype(1056776) , /* mDtypeA */ trtllm::gen::Dtype(17826828) @@ -71655,10 +76058,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17827853) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -71676,7 +76079,7 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mGridWaitForPrimaryB */ 1 , /* mHoistLoadTaskInit */ 1 , /* mHoistMmaTaskTryWaits */ 0 -, /* mK */ 1024 +, /* mK */ 2048 , /* mKernelTraits */ {} , /* mLayoutA */ gemm::MatrixLayout(0) , /* mLayoutB */ gemm::MatrixLayout(0) @@ -71694,16 +76097,14 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsCopySparsityInfo */ 0 , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 -, /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 +, /* mNumSlicesForSplitK */ 2 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 -, /* mNumStagesMma */ 1 +, /* mNumStagesMma */ 2 , /* mNumStagesMmaWithinWorkTile */ 1 -, /* mNumStagesMmaAcrossWorkTile */ 1 +, /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -71718,13 +76119,12 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) -, /* mSplitK */ gemm::SplitK(0) +, /* mSplitK */ gemm::SplitK(2) , /* mTileK */ 512 , /* mTileM */ 128 , /* mTileN */ 8 -, /* mTileScheduler */ gemm::TileScheduler(0) +, /* mTileScheduler */ gemm::TileScheduler(1) , /* mTransposeMmaOutput */ 1 , /* mUseCustomizedMma3xNvFp4 */ 0 , /* mUseCustomMmaSchedule */ 1 @@ -71741,9 +76141,9 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mUseUnrollLoop2xForMma */ 1 , /* mValidM */ 256 , /* mValidN */ 256 -, /* mValidK */ 1024 +, /* mValidK */ 2048 , /* mWorldSize */ 1 -, /* mActType */ gemmGatedAct::ActType(0) +, /* mActType */ gemmGatedAct::ActType(2) , /* mClampBeforeAct */ 1 , /* mBatchedM */ {} , /* mBatchedN */ {} @@ -71765,10 +76165,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 228920, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "979fd0c1ab7b5db4fe68ca42f9f65de59b64ffbc71aacffed576e1d94e6efe78", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 229016, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "37634a02f09307d6de4db49a9cbee816f8ad510e1db529c7bb9e1aa8ae329a53", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -71783,10 +76184,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17827853) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -71823,7 +76224,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 2 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 2 @@ -71831,7 +76231,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -71846,7 +76245,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(2) , /* mTileK */ 512 @@ -71893,10 +76291,137 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} +, /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 , /* mUseTmaOobOpt */ 1 + }, gemm::SmVersion::Sm100f}, +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f_cubin_len, 220696, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f", 512, "dfe8d7374bcd31e819845aa180bc2a490210929e2bbb11fc5c48a1f6359ccc12", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +, /* mBiasDtype */ trtllm::gen::Dtype(1056776) +, /* mBiasType */ gemm::BiasType(0) +, /* mBlockK */ -1 +, /* mClcFastDrain */ 1 +, /* mClusterDimX */ 1 +, /* mClusterDimY */ 1 +, /* mClusterDimZ */ 2 +, /* mCtaSwizzleType */ gemm::CtaSwizzleType(0) +, /* mDtypeAcc */ trtllm::gen::Dtype(1056776) +, /* mDtypeA */ trtllm::gen::Dtype(17826828) +, /* mDtypeB */ trtllm::gen::Dtype(17827853) +, /* mDtypeC */ trtllm::gen::Dtype(17827853) +, /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) +, /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) +, /* mEltwiseActType */ gemm::EltwiseActType(0) +, /* mEnablesEarlyExit */ 1 +, /* mEnablesDelayedEarlyExit */ 0 +, /* mEpilogueLdtmDps */ 16 +, /* mEpilogueLdtmBits */ 256 +, /* mEpilogueTileM */ 128 +, /* mEpilogueTileN */ 8 +, /* mFallbackClusterDimX */ 1 +, /* mFallbackClusterDimY */ 1 +, /* mFallbackClusterDimZ */ 1 +, /* mFusedBiasShuffleMode */ gemm::FusedBiasShuffleMode(0) +, /* mFuseLoadSfTask */ 0 +, /* mFuseUtccpWithUtcmma */ 0 +, /* mGridTriggerSecondaryA */ 0 +, /* mGridTriggerSecondaryB */ 1 +, /* mGridWaitForPrimaryEarlyExit */ 1 +, /* mGridWaitForPrimaryA */ 0 +, /* mGridWaitForPrimaryB */ 1 +, /* mHoistLoadTaskInit */ 1 +, /* mHoistMmaTaskTryWaits */ 0 +, /* mK */ 2048 +, /* mKernelTraits */ {} +, /* mLayoutA */ gemm::MatrixLayout(0) +, /* mLayoutB */ gemm::MatrixLayout(0) +, /* mM */ 256 +, /* mMmaK */ 32 +, /* mMmaKind */ trtllm::gen::MmaKind(5) +, /* mMmaM */ 128 +, /* mMmaN */ 8 +, /* mMmaTileK */ 512 +, /* mMockAllReduce */ 0 +, /* mN */ 256 +, /* mNumEpilogueWarps */ 4 +, /* mNumRegsCastAWarps */ 0 +, /* mNumRegsCopySfLdsSttm */ 0 +, /* mNumRegsCopySparsityInfo */ 0 +, /* mNumRegsPerThreadEpilogueWarp */ 0 +, /* mNumRegsPerThreadNonEpilogueWarp */ 0 +, /* mNumSlicesForSplitK */ 2 +, /* mNumStagesA */ 3 +, /* mNumStagesB */ 3 +, /* mNumStagesMma */ 1 +, /* mNumStagesMmaWithinWorkTile */ 1 +, /* mNumStagesMmaAcrossWorkTile */ 1 +, /* mNumStagesSmemSfA */ 3 +, /* mNumStagesSmemSfB */ 3 +, /* mNumStagesTmemSfA */ 3 +, /* mNumStagesTmemSfB */ 3 +, /* mNumStagesWorkId */ 3 +, /* mOutputDebugTensors */ 0 +, /* mPatchF2fp */ 0 +, /* mPerTokenSfDtype */ trtllm::gen::Dtype(1052672) +, /* mSchedHostTask */ gemm::SchedHostTask(0) +, /* mSfBlockSizeA */ 32 +, /* mSfBlockSizeB */ 32 +, /* mSfBlockSizeC */ 32 +, /* mSfLayoutA */ trtllm::gen::SfLayout(3) +, /* mSfLayoutB */ trtllm::gen::SfLayout(0) +, /* mSfLayoutC */ trtllm::gen::SfLayout(1) +, /* mSfReshapeFactor */ 1 +, /* mSparsityA */ trtllm::gen::Sparsity(0) +, /* mSplitK */ gemm::SplitK(2) +, /* mTileK */ 512 +, /* mTileM */ 128 +, /* mTileN */ 8 +, /* mTileScheduler */ gemm::TileScheduler(0) +, /* mTransposeMmaOutput */ 1 +, /* mUseCustomizedMma3xNvFp4 */ 0 +, /* mUseCustomMmaSchedule */ 1 +, /* mUseDeepSeekFp8 */ 0 +, /* mUseFlexibleClusterDims */ 0 +, /* mUseHoistTryWaitForCustomMmaSchedule */ 0 +, /* mUseMaxTmemOverlap */ 0 +, /* mUsePerTokenSfA */ 0 +, /* mUsePerTokenSfB */ 0 +, /* mUseShuffledMatrix */ 1 +, /* mUseTmaStore */ 1 +, /* mUseTwoTmaLoadWarps */ 1 +, /* mUseTwoMmaWarps */ 0 +, /* mUseUnrollLoop2xForMma */ 1 +, /* mValidM */ 256 +, /* mValidN */ 256 +, /* mValidK */ 2048 +, /* mWorldSize */ 1 +, /* mActType */ gemmGatedAct::ActType(2) +, /* mClampBeforeAct */ 1 +, /* mBatchedM */ {} +, /* mBatchedN */ {} +, /* mBatchMode */ batchedGemm::BatchedGemmOptions::BatchMode(1) +, /* mBatchStrideInTokens */ -1 +, /* mFusedAct */ 1 +, /* mGridWaitForPrimaryRouting */ 1 +, /* mIsStaticBatch */ 0 +, /* mIsUniformNumTokensPerBatch */ 0 +, /* mNumBatches */ 128 +, /* mNumRegsPerThreadLoadA */ 0 +, /* mNumRegsPerThreadLoadB */ 0 +, /* mNumRegsPerThreadLoadSfA */ 0 +, /* mNumRegsPerThreadLoadSfB */ 0 +, /* mNumTokens */ 2 +, /* mNumWarpsLoadA */ 0 +, /* mNumWarpsLoadB */ 0 +, /* mNumWarpsLoadSfA */ 0 +, /* mNumWarpsLoadSfB */ 0 +, /* mRouteImpl */ batchedGemm::RouteImpl(2) +, /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 220488, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "18e9a7f7aadd02a2d55b4371bfab4bb33e33e61cc4d19fa3b3d79a047e130605", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin, Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f_cubin_len, 220696, "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", 512, "cb7e360a0bf8605e38dfb63e228576c4e425c022ff0e49faf7ada4b0d4bc91c3", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -71911,10 +76436,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17827853) , /* mDtypeMmaA */ trtllm::gen::Dtype(17826828) , /* mDtypeMmaB */ trtllm::gen::Dtype(17827853) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -71951,7 +76476,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 2 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 1 @@ -71959,7 +76483,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -71974,7 +76497,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(1) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(2) , /* mTileK */ 512 @@ -72021,10 +76543,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_MxE4m3_MxE4m3MxE2m1_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7_et128x64_m256x128x32_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_swiGlu_laW8_lsfaW4_dynB_sm100f_cubin, Bmm_MxE4m3_MxE4m3MxE2m1_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7_et128x64_m256x128x32_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_swiGlu_laW8_lsfaW4_dynB_sm100f_cubin_len, 193464, "bmm_MxE4m3_MxE4m3MxE2m1_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7_et128x64_m256x128x32_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_swiGlu_laW8_lsfaW4_dynB_sm100f", 896, "4344d34cfd4f7b888f95feb52c3280028c130c52e3745aa47b0287bec8ae23ad", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_MxE4m3_MxE4m3MxE2m1_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7_et128x64_m256x128x32_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_swiGlu_laW8_lsfaW4_dynB_sm100f_cubin, Bmm_MxE4m3_MxE4m3MxE2m1_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7_et128x64_m256x128x32_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_swiGlu_laW8_lsfaW4_dynB_sm100f_cubin_len, 193688, "bmm_MxE4m3_MxE4m3MxE2m1_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7_et128x64_m256x128x32_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_swiGlu_laW8_lsfaW4_dynB_sm100f", 896, "6c2e767ae62c7da7581d1f9797c45168598de080002b8390645335a9b47502c0", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(2) , /* mBlockK */ -1 @@ -72039,10 +76562,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17827853) , /* mDtypeMmaA */ trtllm::gen::Dtype(17827853) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826828) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 32 , /* mEpilogueLdtmBits */ 32 , /* mEpilogueTileM */ 128 @@ -72079,7 +76602,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 152 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 7 , /* mNumStagesB */ 7 , /* mNumStagesMma */ 2 @@ -72087,7 +76609,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 7 , /* mNumStagesSmemSfB */ 7 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 7 , /* mNumStagesTmemSfB */ 7 , /* mNumStagesWorkId */ 3 @@ -72102,7 +76623,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 128 @@ -72149,10 +76669,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, -{Bmm_MxE4m3_MxE4m3MxE2m1_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4_et128x64_m256x128x32_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_swiGlu_laW8_lsfaW4_dynB_sm100f_cubin, Bmm_MxE4m3_MxE4m3MxE2m1_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4_et128x64_m256x128x32_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_swiGlu_laW8_lsfaW4_dynB_sm100f_cubin_len, 217752, "bmm_MxE4m3_MxE4m3MxE2m1_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4_et128x64_m256x128x32_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_swiGlu_laW8_lsfaW4_dynB_sm100f", 896, "d1808898b3207a654e5a39a0aaa3abce88f38df661c69139f7b6df371e021bfa", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_MxE4m3_MxE4m3MxE2m1_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4_et128x64_m256x128x32_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_swiGlu_laW8_lsfaW4_dynB_sm100f_cubin, Bmm_MxE4m3_MxE4m3MxE2m1_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4_et128x64_m256x128x32_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_swiGlu_laW8_lsfaW4_dynB_sm100f_cubin_len, 217752, "bmm_MxE4m3_MxE4m3MxE2m1_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4_et128x64_m256x128x32_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_swiGlu_laW8_lsfaW4_dynB_sm100f", 896, "149f48cfa67eee8c1615df071852d054eb7f6296dbf3e1bae5c645496ce441e6", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(2) , /* mBlockK */ -1 @@ -72167,10 +76688,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(17827853) , /* mDtypeMmaA */ trtllm::gen::Dtype(17827853) , /* mDtypeMmaB */ trtllm::gen::Dtype(17826828) +, /* mDtypeSfC */ trtllm::gen::Dtype(2063) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 32 , /* mEpilogueLdtmBits */ 32 , /* mEpilogueTileM */ 128 @@ -72207,7 +76728,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 152 , /* mNumRegsPerThreadNonEpilogueWarp */ 48 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 2 @@ -72215,7 +76735,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -72230,7 +76749,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(1) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -72277,12 +76795,13 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(2) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm100f}, #endif // EXCLUDE_SM_100F #ifndef EXCLUDE_SM_103 -{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin_len, 116304, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a", 512, "fc2fb352c667de7485b995363a7afd18d559081a3f6fbd5f8e65307211280861", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin_len, 116368, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a", 512, "07dcafd30f2e349ea818e3d3b816ec40bbdd0a2f261665c3209726508ec8c975", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -72297,10 +76816,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -72337,7 +76856,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -72345,7 +76863,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -72360,7 +76877,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -72407,10 +76923,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm103a}, -{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin_len, 116064, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a", 384, "33aaf553a60bb375af5c2d02715b3425228f28936c887664e133f9b3ae05a07c", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin_len, 116240, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a", 384, "38cd91f274f1cfed2c53dfdd68ca2faae8d11f47d158f24271d4297d8c405b90", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -72425,10 +76942,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -72465,7 +76982,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -72473,7 +76989,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -72488,7 +77003,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -72535,10 +77049,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm103a}, -{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin_len, 116304, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a", 512, "b7deea72f4d04347e989ac02eff6450a670a109055a86f889d8585fec634edbe", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin_len, 116368, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a", 512, "00aa03ce3bcac5ac050aabc7af37a829f073a005f5bdbc33b7daa28f15e09b27", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -72553,10 +77068,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -72593,7 +77108,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -72601,7 +77115,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -72616,7 +77129,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -72663,10 +77175,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm103a}, -{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin_len, 116064, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a", 384, "5a2eb30ab861df78f6f5b745b3ecee7d3f22211f53571328c679077aeec2e55e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin_len, 116240, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a", 384, "fb28f2289161b81d1dc6dc067406ae796bb1d944fef6fd6f55b1f9bd64fdd804", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -72681,10 +77194,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -72721,7 +77234,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -72729,7 +77241,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -72744,7 +77255,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -72791,10 +77301,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm103a}, -{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin_len, 140880, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a", 512, "332effe7f5822a0b9b8fafcd5e71ded6da1308f4f705000b5e761c01cf443fcf", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin_len, 140944, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a", 512, "bc5cda33cbcd55ceae874ce9d799f062a0365d6798895ba0b2033a35a9c3a7ab", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -72809,10 +77320,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -72849,7 +77360,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -72857,7 +77367,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -72872,7 +77381,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -72919,10 +77427,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm103a}, -{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin_len, 140640, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a", 384, "ad8a2566ce95cb0b157d30ac23197461d5cde02feb98654fd4218afd985a25f8", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin_len, 140816, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a", 384, "9a1d894e4393939e9eac9315efade0459a57365cf47f0d558728225fd504947a", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -72937,10 +77446,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -72977,7 +77486,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -72985,7 +77493,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -73000,7 +77507,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -73047,10 +77553,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm103a}, -{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin_len, 140880, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a", 512, "50ba61bf4a4fe4b60a4b01275aee870bb5c3a97ea0ade3ab90fdcf28c0e0bdba", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin_len, 140944, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a", 512, "a5a1c68e10e8872edf445fb486844f7e8e18589c43c0d84b88209e8ec10f27e6", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -73065,10 +77572,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -73105,7 +77612,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -73113,7 +77619,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -73128,7 +77633,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -73175,10 +77679,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm103a}, -{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin_len, 140640, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a", 384, "c296c18a3c4f16cd8c1af077522284c28e8e5613c748c802ce655adc6de77070", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin_len, 140816, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a", 384, "50f45c442887c874f0efc6cebad0e64ea6fea93ad85ce8b16a841c82f8df14c8", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -73193,10 +77698,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -73233,7 +77738,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -73241,7 +77745,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -73256,7 +77759,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -73303,10 +77805,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm103a}, -{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin_len, 157200, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a", 512, "5da51fb3a967c031389a2d83b8dc1c397e02483cb8ca13a64d4c77e079401e50", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin_len, 157328, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a", 512, "e65fb4c0f43c6640ec059f63fb12c0d600916e1fd3086dd200c1ca8ab743a4ff", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -73321,10 +77824,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -73361,7 +77864,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 2 @@ -73369,7 +77871,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -73384,7 +77885,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -73431,10 +77931,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm103a}, -{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin_len, 156960, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a", 384, "bfe4468fb0477b02d603131f9117f02a19a0824ff23bf0861f2f93d4330e0f8a", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin_len, 157200, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a", 384, "2af12ba09fda0fe483397879d84869a26678a144ba99885830fae722a632a30f", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -73449,10 +77950,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -73489,7 +77990,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 1 @@ -73497,7 +77997,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -73512,7 +78011,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -73559,10 +78057,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm103a}, -{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin_len, 157200, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a", 512, "a47917aa2d43eba3b6c2eb6b5345b193a16de9ea53d165a7a05c6435a0d54f97", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin_len, 157328, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a", 512, "468139f4660b3e9c9d69f5daaf7b0db00c1a77a49d456f81f298eb29735a4bbd", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -73577,10 +78076,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -73617,7 +78116,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 2 @@ -73625,7 +78123,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -73640,7 +78137,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -73687,10 +78183,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm103a}, -{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin_len, 156960, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a", 384, "f46fda6f81cb30f3fd3df15a317351b1ffca6ce93a07fa241fb6e819e1aad2e2", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin_len, 157200, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a", 384, "8422c894b48a73264779e2f3deb1cb411e0dbea0f33842a50374214d7be2ddce", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -73705,10 +78202,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -73745,7 +78242,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 1 @@ -73753,7 +78249,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -73768,7 +78263,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -73815,10 +78309,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm103a}, -{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin_len, 104016, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a", 512, "1bd574a0682a7269a2c384bfc9de808ba326842307e540d58932c3ff507a0854", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin_len, 104080, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a", 512, "a718a2fe4af1844acab365ced53da58d9f153d04e72310b3293d147b104b55e6", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -73833,10 +78328,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -73873,7 +78368,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -73881,7 +78375,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -73896,7 +78389,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -73943,10 +78435,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm103a}, -{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin_len, 103776, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a", 384, "1047f5b01b209231ce0e64089f6e1e6d9c20042ce18f1d960c1d715b01b644f0", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin_len, 103952, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a", 384, "d4c46bec43fece4d731780c563dfd45db28edabba6c35af26d9d05f700e78d0a", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -73961,10 +78454,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -74001,7 +78494,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -74009,7 +78501,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -74024,7 +78515,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -74071,10 +78561,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm103a}, -{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin_len, 104016, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a", 512, "fc7b573e80ab5e97b184bd55e1fa29fad0a8bfcb6d74d41e0b0bf17a5b197af2", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin_len, 104080, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a", 512, "8517303b490220192a8a65cebeab77f45295fad7c5a6f4675b34f8d3ff5508d8", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -74089,10 +78580,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -74129,7 +78620,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -74137,7 +78627,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -74152,7 +78641,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -74199,10 +78687,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm103a}, -{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin_len, 103776, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a", 384, "8af3c7bd743e179442e4cc713ce71c9a0f86ad019ae834fc3e353a5317ae430d", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin_len, 103952, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a", 384, "af2f6d683560dac68f537efc9ff98af8fc060027c7c8b64974dee742495fff67", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -74217,10 +78706,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -74257,7 +78746,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -74265,7 +78753,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -74280,7 +78767,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -74327,10 +78813,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm103a}, -{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin_len, 123344, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a", 512, "25696a068c8fc18b9653da7c55e4588bfcb74a91b77b3b27f02eb6184076811f", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin_len, 123536, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a", 512, "f94527d95515605d664db19e06705ad88c6799559780ed938d6548b5b762c8ac", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -74345,10 +78832,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -74385,7 +78872,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 2 @@ -74393,7 +78879,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -74408,7 +78893,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -74455,10 +78939,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm103a}, -{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin_len, 123104, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a", 384, "fb3cc0e56022146970f047078d2a0a21c2161ad65266bb9fd470f211e6b4f36f", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin_len, 123152, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a", 384, "ce12eddee469fcb4732b85aab09e474886ea24c1ee6afeb5a7fbc405aa0d5efa", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -74473,10 +78958,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -74513,7 +78998,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 1 @@ -74521,7 +79005,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -74536,7 +79019,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -74583,10 +79065,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm103a}, -{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin_len, 123344, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a", 512, "9ac50d26585e682639bca101a8d8df0cc9c2ec22fe9042be33247bb1cc917be6", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin_len, 123536, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a", 512, "ead53d4e43bd0757cc0365e1cc406b2db4cc0ff73f9cc2ca5e64d2fe0498509e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -74601,10 +79084,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -74641,7 +79124,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 2 @@ -74649,7 +79131,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -74664,7 +79145,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -74711,10 +79191,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm103a}, -{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin_len, 123104, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a", 384, "703f7647aa14f2f6dbb07416e11c037a646d946e1d40272b40b713e244d4be88", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin, Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a_cubin_len, 123152, "bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a", 384, "82339e09861f14db2d1fad2dc77731439c87664aa9ad0f73c0540549a189118a", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -74729,10 +79210,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1052672) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -74769,7 +79250,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 1 @@ -74777,7 +79257,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -74792,7 +79271,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(3) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -74839,10 +79317,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(0) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(0)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm103a}, -{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin_len, 114256, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a", 512, "e0e784c0c9e55a25880fed1352c0bf0f6b808c2cf563670c06a80b799c51869a", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin_len, 114320, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a", 512, "260495bf013585f26af55fa99ae45a7c40c4aec1cb8bdab53f5240272f947dac", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -74857,10 +79336,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -74897,7 +79376,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -74905,7 +79383,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -74920,7 +79397,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -74967,10 +79443,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm103a}, -{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin_len, 114016, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a", 512, "37276ae08a2719ec48f905ff3d8b31d83fd4d16fc7ff3765b741f0ebaebb6130", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin_len, 114192, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a", 512, "10e2c9b2fe1f4e1df09f67da29a63d8cd17e8deb26c77f8b31ad61ff79cbcee9", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -74985,10 +79462,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -75025,7 +79502,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -75033,7 +79509,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -75048,7 +79523,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -75095,10 +79569,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm103a}, -{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin_len, 114256, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a", 512, "ddb5a0f4e03d62844d4b077d560becb6efba25275166f1ba09ecd63873401f96", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin_len, 114320, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a", 512, "c7e4e00a51fab39a148cbeccb82fe3fb41b0a8a02dc22352aa5d1c9aeee80951", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -75113,10 +79588,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -75153,7 +79628,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -75161,7 +79635,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -75176,7 +79649,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -75223,10 +79695,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm103a}, -{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin_len, 114016, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a", 512, "9171cfa8591d4cbb5d59e24bb1e735d83a5adb350f89d94a13d0ec1136580114", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin_len, 114192, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a", 512, "e5c9d4fb2e0455801021377d9ae75cc134eb8a9f5012ca2cb75474c021822b23", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -75241,10 +79714,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -75281,7 +79754,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -75289,7 +79761,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -75304,7 +79775,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -75351,10 +79821,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm103a}, -{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin_len, 136784, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a", 512, "587273840db12546a1e920c8bcd7b66c86ce68a73772ab1d431a9006b21e643d", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin_len, 136848, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a", 512, "9ea1e535a600e70aadc2431a148ee0808e352600cc6a48f1b88edbea1ba3ad44", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -75369,10 +79840,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -75409,7 +79880,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -75417,7 +79887,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -75432,7 +79901,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -75479,10 +79947,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm103a}, -{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin_len, 136544, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a", 512, "07f9ea2d6964818cfadc08af9e701f55a472492500c99c0aa227200700716f9f", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin_len, 136720, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a", 512, "77ab2ba8abe56046ca5141a0675c8a7ccbe1ffa1d7e9cfa5b1e584b3be52f476", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -75497,10 +79966,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -75537,7 +80006,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -75545,7 +80013,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -75560,7 +80027,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -75607,10 +80073,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm103a}, -{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin_len, 136784, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a", 512, "88b3ff2c0e3d84a95ae5eaef6075d9d655990b2a12fc2ff46f72b8159b4ded71", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin_len, 136848, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a", 512, "a3147ed18dfc064b1bce8cb2f98ecd8733227ba79926e801517c6c62c380d61b", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -75625,10 +80092,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -75665,7 +80132,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -75673,7 +80139,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -75688,7 +80153,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -75735,10 +80199,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm103a}, -{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin_len, 136544, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a", 512, "97860b2b094ab7a72d25efe39b2b99e0e81eb6ea7bb44fa5e75ce72a06f9b4b1", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin_len, 136720, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a", 512, "2ff7c54209005beb4038beb12b4cc8f506784afaa8ab8d3fed8233b5a6f5e646", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -75753,10 +80218,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -75793,7 +80258,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -75801,7 +80265,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -75816,7 +80279,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -75863,10 +80325,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm103a}, -{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin_len, 149008, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a", 512, "b49d6174f42f62f07a31274faf22f222e9c21bde3e429663160ffc24384387d3", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin_len, 149136, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a", 512, "29783f8f9bb85c176340f9cf212f7830e1f406941898cfd40411bed17feeef77", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -75881,10 +80344,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -75921,7 +80384,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 2 @@ -75929,7 +80391,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -75944,7 +80405,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -75991,10 +80451,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm103a}, -{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin_len, 148768, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a", 512, "4e2ea960c7fc5548755a8a6c431d9fbb03b7b33629f30022638c2ca1dac7f102", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin_len, 149008, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a", 512, "ed1dab836fcd8a9a62ce77a3f60e1a8478b7fe62a6e3d5d6b559d291b22c53c3", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -76009,10 +80470,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -76049,7 +80510,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 1 @@ -76057,7 +80517,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -76072,7 +80531,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -76119,10 +80577,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm103a}, -{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin_len, 149008, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a", 512, "a3e7dd289bb6e20114fcdbfc67711cdeb25d75aa5bd3e17e6f81646e7e2fd263", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin_len, 149136, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a", 512, "a1390455e653639d438bb626808e704a7aee4753787d58edd063484f2bdbdbed", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -76137,10 +80596,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -76177,7 +80636,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 2 @@ -76185,7 +80643,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -76200,7 +80657,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -76247,10 +80703,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm103a}, -{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin_len, 148768, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a", 512, "76c814bb7f12d3bee9076a49980ef1c554f537fdc5026acc3da36f080ca65c6d", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin_len, 149008, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a", 512, "37686a0f0b1c832f774f803f6b84fd46163c3dfd945c9d38bd02e3a962cf0baa", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -76265,10 +80722,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -76305,7 +80762,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 4 , /* mNumStagesB */ 4 , /* mNumStagesMma */ 1 @@ -76313,7 +80769,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 4 , /* mNumStagesSmemSfB */ 4 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 4 , /* mNumStagesTmemSfB */ 4 , /* mNumStagesWorkId */ 3 @@ -76328,7 +80783,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -76375,10 +80829,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm103a}, -{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin_len, 102992, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a", 512, "42c54609d64b222c8b601c69f32e6a93a9807a4a3bafa2116544460d6e659890", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin_len, 103056, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a", 512, "8824a03e5abf2e1e80eaa34a6eb405b0003da1756f1b6f5c3785e1ca49ab4ff7", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -76393,10 +80848,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -76433,7 +80888,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -76441,7 +80895,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -76456,7 +80909,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -76503,10 +80955,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm103a}, -{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin_len, 102752, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a", 512, "a8542eac0264e2a5a06e78248024f7a6a5325ede9a55f9a4edc85feb47c027ac", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin_len, 102928, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a", 512, "2d931660bf65d2c3d7b6dcc3f8f8916ab91ccb20f431e3086c98514a5a70e60e", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -76521,10 +80974,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -76561,7 +81014,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -76569,7 +81021,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -76584,7 +81035,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -76631,10 +81081,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm103a}, -{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin_len, 102992, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a", 512, "37b9132cc9720220e5fbca239d6b33f3cf7c7e052b4305311f361e66e3b33aad", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin_len, 103056, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a", 512, "479f512d502398812f6b9fbd4f7c29822fada3be8d5f4a7c3469106035dcae4d", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -76649,10 +81100,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -76689,7 +81140,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 2 @@ -76697,7 +81147,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -76712,7 +81161,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -76759,10 +81207,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm103a}, -{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin_len, 102752, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a", 512, "4aa37c86cd686bb816424d56a4845f1b514b04ffd8c7ab6a6e8ac2c4ae751791", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin_len, 102928, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a", 512, "6f9b00a7256107a70758b8726ad0703ee2ef75d22954773ba0d2f763dbe70034", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -76777,10 +81226,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -76817,7 +81266,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 5 , /* mNumStagesB */ 5 , /* mNumStagesMma */ 1 @@ -76825,7 +81273,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 5 , /* mNumStagesSmemSfB */ 5 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 5 , /* mNumStagesTmemSfB */ 5 , /* mNumStagesWorkId */ 3 @@ -76840,7 +81287,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 256 @@ -76887,10 +81333,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm103a}, -{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin_len, 122320, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a", 512, "b038d2b7c72a90728d03bf541b253610129b674cd9646101fc706cf23be5cb90", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin_len, 122512, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a", 512, "349cb6ad0ea69064545143ffd05def39871f06965835ff303c89b7673a48dc67", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -76905,10 +81352,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -76945,7 +81392,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 2 @@ -76953,7 +81399,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -76968,7 +81413,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -77015,10 +81459,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm103a}, -{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin_len, 122080, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a", 512, "12e301660ac9c0d1b8e6ffafd9f57ea6495a7fd10751e3d13a8063f3d978bcfe", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin_len, 122128, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a", 512, "d311e7dfd8d23393e285fb98d92a47221b75346b6ed1ece3af6c3a786a33dede", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -77033,10 +81478,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -77073,7 +81518,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 1 @@ -77081,7 +81525,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -77096,7 +81539,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -77143,10 +81585,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm103a}, -{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin_len, 122320, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a", 512, "5da63f7be405cf5928a5fe06a6ccaf10e68a1608cfe70f91ef0a8c25e5dd2bb9", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin_len, 122512, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a", 512, "4d4ddb5fd540d75e0ff63a7a13752c07019d87c9b5ceced557c467664aadb76a", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -77161,10 +81604,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -77201,7 +81644,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 2 @@ -77209,7 +81651,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 2 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -77224,7 +81665,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -77271,10 +81711,11 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm103a}, -{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin_len, 122080, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a", 512, "8b56cb73ec2f70e924839a6796725e22d993aabe7e8b4359c79451d5b48f0615", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) +{Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin, Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a_cubin_len, 122128, "bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a", 512, "9344afab994a47ea13ccf248aad2151f2e43de2005a826f18924f3ccacfe87de", "", nullptr, nullptr, nullptr, 0, { /* mAllReduceAlgo */ gemm::AllReduceAlgo(0) , /* mBiasDtype */ trtllm::gen::Dtype(1056776) , /* mBiasType */ gemm::BiasType(1) , /* mBlockK */ -1 @@ -77289,10 +81730,10 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mDtypeC */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaA */ trtllm::gen::Dtype(1050629) , /* mDtypeMmaB */ trtllm::gen::Dtype(1050629) +, /* mDtypeSfC */ trtllm::gen::Dtype(1048597) , /* mEltwiseActType */ gemm::EltwiseActType(0) , /* mEnablesEarlyExit */ 1 , /* mEnablesDelayedEarlyExit */ 0 -, /* mEnablesGlobalPtxKnobs */ 1 , /* mEpilogueLdtmDps */ 16 , /* mEpilogueLdtmBits */ 256 , /* mEpilogueTileM */ 128 @@ -77329,7 +81770,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumRegsPerThreadEpilogueWarp */ 0 , /* mNumRegsPerThreadNonEpilogueWarp */ 0 , /* mNumSlicesForSplitK */ 1 -, /* mNumSlicesForSliceK */ 1 , /* mNumStagesA */ 3 , /* mNumStagesB */ 3 , /* mNumStagesMma */ 1 @@ -77337,7 +81777,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumStagesMmaAcrossWorkTile */ 1 , /* mNumStagesSmemSfA */ 3 , /* mNumStagesSmemSfB */ 3 -, /* mNumStagesTmemA */ 0 , /* mNumStagesTmemSfA */ 3 , /* mNumStagesTmemSfB */ 3 , /* mNumStagesWorkId */ 3 @@ -77352,7 +81791,6 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mSfLayoutB */ trtllm::gen::SfLayout(0) , /* mSfLayoutC */ trtllm::gen::SfLayout(3) , /* mSfReshapeFactor */ 1 -, /* mSliceK */ 0 , /* mSparsityA */ trtllm::gen::Sparsity(0) , /* mSplitK */ gemm::SplitK(0) , /* mTileK */ 512 @@ -77399,13 +81837,13 @@ static const batchedGemm::BatchedGemmConfig tllmGenBatchedGemmList[] = { , /* mNumWarpsLoadSfB */ 0 , /* mRouteImpl */ batchedGemm::RouteImpl(1) , /* mRouteSfsImpl */ {batchedGemm::RouteImpl(1)} -, /* mUseTmaOobOpt */ 1 , /* mPrefetchB */ 0 +, /* mUseCMultiCast */ 0 +, /* mUseTmaOobOpt */ 1 }, gemm::SmVersion::Sm103a}, #endif // EXCLUDE_SM_103 }; // clang-format on } // namespace kernels - TRTLLM_NAMESPACE_END } // namespace batchedGemm diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/KernelParams.h b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/KernelParams.h index d0c0ba256cb7..7650961d2f57 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/KernelParams.h +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/KernelParams.h @@ -89,7 +89,8 @@ bool useTmaOobOptB(BatchedGemmOptions const& options) template bool useTmaOobOptC(BatchedGemmOptions const& options) { - return options.mUseTmaStore && options.mUseTmaOobOpt; + // C multicast uses direct pointers instead of TMA store + return options.mUseTmaStore && !options.mUseCMultiCast && options.mUseTmaOobOpt; } ////////////////////////////////////////////////////////////////////////////////////////////////// @@ -231,7 +232,7 @@ static auto makeTmaShapeStrideAbc(GemmOptions const& options, int sizeM, int siz // Create the TMA shape/stride for A/B block scaling factors. static auto makeTmaShapeStrideSfAb(int mM, int mN, int mK, MatrixType matrixType, int tileM, int tileN, int tileK, - tg::SfLayout layout, int sfReshapeFactor, const int32_t numEltsPerSf) + tg::SfLayout layout, int sfReshapeFactor, int32_t const numEltsPerSf) { // The outer dimension. @@ -247,6 +248,15 @@ static auto makeTmaShapeStrideSfAb(int mM, int mN, int mK, MatrixType matrixType { case tg::SfLayout::R128c4: { + // The scaling factor tensor packs 128x4 tiles into contiguous 512B blocks. + // The 512B block maps to a 32x16B (32x128b) block in TMEM. + // + // Additionally, we have to meet constraints of TMA that the box dimensions are less + // than 256 and boxDim[0] is a multiple of 16B. + // + // The "logical" tensor is: [outer, inner / numEltsPerSf] + // The aforementioned format is: [outer / 128, inner / numEltsPerSf / 4, 512] + // The shape we use for TMA is: [outer / 128, inner / numEltsPerSf / 4, 2, 256] auto shape = std::vector{256, 2, static_cast(ceilDiv(hiddenSize, numEltsPerSf * 4)), static_cast(ceilDiv(numTokens, 128))}; @@ -354,10 +364,11 @@ static KernelParams setKernelParams(GemmOptions_ const& options, bool const batc [[maybe_unused]] void const* ptrSparsityInfoA, void const* ptrBias, int32_t const* ptrPermutedIdxToBiasRowIdx, void* dSfC, float const* ptrScaleC, float const* ptrScaleAct, float const* ptrScaleGate, float const* ptrClampLimit, float const* ptrGatedActAlpha, float const* ptrGatedActBeta, int32_t const* routeMap, float* rowMax, - uint32_t* rowMaxBars, int32_t const* ptrNumNonExitingCtas = nullptr, - int32_t const* ptrTotalNumPaddedTokens = nullptr, int32_t const* ptrCtaIdxXyToBatchIdx = nullptr, - int32_t const* ptrCtaIdxXyToMnLimit = nullptr, int32_t const maxNumCtas = KernelParams::MaxNumCtas, - uint32_t* ptrDynamicTileCounter = nullptr) + uint32_t* rowMaxBars, int rank, int worldSize, uint32_t* ptrMulticastCompletionBarUc, + uint32_t* ptrMulticastCompletionBarMc, int32_t const* permutedIdxToExpandedIdx, void const* expertWeightsPtr, + int32_t const* ptrNumNonExitingCtas = nullptr, int32_t const* ptrTotalNumPaddedTokens = nullptr, + int32_t const* ptrCtaIdxXyToBatchIdx = nullptr, int32_t const* ptrCtaIdxXyToMnLimit = nullptr, + int32_t const maxNumCtas = KernelParams::MaxNumCtas, uint32_t* ptrDynamicTileCounter = nullptr) { static_assert(sizeof(KernelParams) <= 32 * 1024, "sizeof(KernelParams) has to be less or equal than 32KB"); @@ -598,7 +609,8 @@ static KernelParams setKernelParams(GemmOptions_ const& options, bool const batc } // C is the output activation - if (options.mUseTmaStore) + // Skip TMA descriptor setup when using C multicast - direct pointers are used instead + if (options.mUseTmaStore && !options.mUseCMultiCast) { // Shape/stride for gmem tensor C. // NOTE: Output is *always* sanitized across the whole MNK range. This ensures maximum @@ -711,7 +723,8 @@ static KernelParams setKernelParams(GemmOptions_ const& options, bool const batc } // C is the output activation - if (options.mUseTmaStore) + // Skip TMA descriptor setup when using C multicast - direct pointers are used instead + if (options.mUseTmaStore && !options.mUseCMultiCast) { // Shape/stride for gmem tensor C. // NOTE: Output is *always* sanitized across the whole MNK range. This ensures maximum @@ -732,8 +745,10 @@ static KernelParams setKernelParams(GemmOptions_ const& options, bool const batc params.k = options.mK; params.numBatches = options.mNumBatches; - params.rank = 0; - params.tpGrpSize = 1; + params.rank = rank; + params.tpGrpSize = worldSize; + params.ptrMulticastCompletionBarUc = ptrMulticastCompletionBarUc; + params.ptrMulticastCompletionBarMc = ptrMulticastCompletionBarMc; params.ptrPartialRowMax = rowMax; params.ptrRowMaxCompletionBars = rowMaxBars; @@ -745,6 +760,9 @@ static KernelParams setKernelParams(GemmOptions_ const& options, bool const batc params.ptrPerTokenSfB = ptrPerTokenSfB; params.ptrBias = ptrBias; + // Set MoE finalize parameters for direct register-to-gmem store + params.permutedIdxToExpandedIdx = permutedIdxToExpandedIdx; + params.expertWeightsPtr = expertWeightsPtr; // Set the dynamic tile counter pointer for dynamic scheduling. params.ptrDynamicTileCounter = ptrDynamicTileCounter; diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/KernelParamsDecl.h b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/KernelParamsDecl.h index 3833dbff3628..096aa1a8fac9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/KernelParamsDecl.h +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/KernelParamsDecl.h @@ -208,6 +208,26 @@ struct KernelParams // The output matrix C. Check "logical" layout of tmaC to see the shape and strides. void* ptrC{nullptr}; + ////////////////////////////////////////////////////////////////////////////////////////////////// + // + // MoE (Mixture of Experts) finalize parameters for direct register-to-gmem store. + // + ////////////////////////////////////////////////////////////////////////////////////////////////// + + // Mapping from permuted token index to expanded token index. + // Used in MoE finalize to map the batch dimension back to the original token order. + // Shape is [numTokens * topK] where topK is the number of experts per token. + // Value of -1 indicates an invalid/padded entry that should be skipped. + // The dtype is int32_t. + int32_t const* permutedIdxToExpandedIdx{nullptr}; + + // Expert weights (routing probabilities) for each expanded token. + // Used in MoE finalize to scale the output by the expert's routing weight. + // Shape is [numTokens * topK]. + // The dtype matches the output dtype (typically bfloat16 or float16). + // Can be nullptr if no expert weighting is needed. + void const* expertWeightsPtr{nullptr}; + // Inputs and output are MxFp{4,8}, Fp8, NvFp4. // The scaling factors to apply to the output - can be used to incorporate input scaling factors // as described below: C = SEncC * act(SDecA * SDecB * A * Bl) . (SDecA * SDecB * A * Br) @@ -317,8 +337,7 @@ struct KernelParams // // This is used for either: // * Per-token scaling factor quantization schemes, such as MetaFP8. The dtype is Dtype::Float32 - // * When the routing scales are applied to the input activations (only when output is not - // transposed). The dtype is Dtype::Bfloat16 + // * One-sided per-token scaling. The dtype is Dtype::Bfloat16 by default. // // if (batchM (A is activations)): // Logical shape is [sum(divUpMul(M[bi], tileM) for bi in B)] @@ -332,8 +351,7 @@ struct KernelParams // // This is used for either: // * Per-token scaling factor quantization schemes, such as MetaFP8. The dtype is Dtype::Float32 - // * When the routing scales are applied to the input activations (only when output is - // transposed). The dtype is Dtype::Bfloat16 + // * One-sided per-token scaling. The dtype is Dtype::Bfloat16 by default. // // if (batchM (B is weights)): // Logical shape is [B, divUpMul(N, tileN)] @@ -563,6 +581,14 @@ struct KernelParams int rank; // The number of peer devices in tensor-parallel group. int tpGrpSize; + // A barrier that is used to synchronize kernels running on multiple GPUs. + // UC is a unicast pointer in local GPU memory. + // MC is a multicast pointer to all other GPUs' memory. + // + // The value must be initialized to zero. The barrier is self-resetting. It + // does not have to be reset by another kernel after the grid has ended. + uint32_t* ptrMulticastCompletionBarUc; + uint32_t* ptrMulticastCompletionBarMc; ////////////////////////////////////////////////////////////////////////////////////////////////// // diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/KernelTraits.h b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/KernelTraits.h index 54f7f64f2822..462658a518be 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/KernelTraits.h +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/KernelTraits.h @@ -93,6 +93,19 @@ class MemAllocatorHelper throw std::runtime_error("Name not found: " + name); } + // Returns whether a chunk with the given name was allocated. + bool hasChunkByName(std::string const& name) const + { + for (size_t ii = 0; ii < mSmemChunkNames.size(); ++ii) + { + if (mSmemChunkNames[ii] == name) + { + return true; + } + } + return false; + } + // Returns the first chunk reuse flag given chunk name. int getFirstChunkReuseFlagByName(std::string const& name) const { @@ -172,6 +185,8 @@ inline int getNumSmemBitsPerElt(tg::Dtype dtype, tg::MmaKind mmaKind, int mmaK, } if (mmaKind == tg::MmaKind::MxFp8Fp6Fp4) { + (void) mmaK; + (void) isSparseA; return 8; } else @@ -193,11 +208,10 @@ class KernelTraits tg::Dtype dtypeMmaB, tg::MmaKind mmaKind, tg::Sparsity sparsityA, int32_t mmaK, int32_t tileM, int32_t tileN, int32_t tileK, int32_t mmaTileK, int32_t epilogueTileM, int32_t epilogueTileN, int32_t numEltsPerSfA, int32_t numEltsPerSfB, int32_t numStagesA, int32_t numStagesB, int32_t numStagesMma, int32_t numStagesSfA, - int32_t numStagesSfB, int32_t numSlicesForSplitK, int32_t numSlicesForSliceK, SplitK splitK, bool useTmaStore, - bool transposeMmaOutput, AllReduceAlgo allReduceAlgo, bool fuseUtccpWithUtcmma, bool useMaxTmemOverlap, - bool useCustomizedMma3xNvFp4, int32_t numEpilogueWarps, bool usePersistentScheduler, bool useDeepSeekFp8, - bool usePerTokenSfA, bool usePerTokenSfB, bool useTwoCtas, BiasType biasType, - FusedBiasShuffleMode fusedBiasShuffleMode) + int32_t numStagesSfB, int32_t numSlicesForSplitK, SplitK splitK, bool useTmaStore, bool transposeMmaOutput, + AllReduceAlgo allReduceAlgo, bool fuseUtccpWithUtcmma, bool useMaxTmemOverlap, bool useCustomizedMma3xNvFp4, + int32_t numEpilogueWarps, bool usePersistentScheduler, bool useDeepSeekFp8, bool usePerTokenSfA, + bool usePerTokenSfB, bool useTwoCtas, BiasType biasType, FusedBiasShuffleMode fusedBiasShuffleMode) : mMmaKind{mmaKind} , mFuseUtccpWithUtcmma{fuseUtccpWithUtcmma} , mUseMaxTmemOverlap{useMaxTmemOverlap} @@ -215,14 +229,12 @@ class KernelTraits // [gmemC0 ] (1024B aligned) (if needed) // [gmemC1 ] (1024B aligned) (if needed) // [rowMax ] (16B aligned) (if needed) - // [sliceK ] (16B aligned) (if needed) // [per-token SF ] (16B aligned) (if needed) // [bias ] (16B aligned) (if needed) // // SMEM for smemA and smemB might be repurposed and used for gmemC0 and gmemC1: // // [..smemA..][..smemB..][..smemBShuffle..] - // [..gmemC0..][..gmemC1..][..rowMax..][..sliceK..][..per-token SF..][..bias..] // if (mMmaKind == tg::MmaKind::Auto) @@ -277,9 +289,7 @@ class KernelTraits // - Do 4 TMA SW32 loads or several LDGSTS loads. { // Number of bytes in save shuffled B in shared memory. - auto const numSmemBytesLoadB = numSlicesForSliceK > 1 ? numStagesB * tileN * tileK - * getNumSmemBitsPerElt(dtypeB, mMmaKind, mmaK, isSparseA) / 8 /* bits */ - : 0; + auto const numSmemBytesLoadB = 0; // Number of bytes for load B alignment for TMA load. auto const numBytesAlignmentLoadB = 1024; // No need to reuse the first chunk. @@ -306,11 +316,6 @@ class KernelTraits bool usesSmemForGmemC = useTmaStore || doesSplitKUseDsmem(splitK); // SMEM for at leader CTA in DSMEM split-k contains K slices. auto extraGmemCMultiplier = doesSplitKUseDsmem(splitK) ? numSlicesForSplitK : 1; - if (numSlicesForSliceK > 1) - { - // TileN is expanded in N dimension for slice-K. - extraGmemCMultiplier *= numSlicesForSliceK; - } if (resIdx != 0 && !useDeepSeekFp8) { @@ -377,22 +382,6 @@ class KernelTraits firstChunkReuseSmem.emplace_back(false); } - // SliceK - { - // Real tile size before slice-K reduction. - auto const tileSize - = numSlicesForSliceK > 1 ? numSlicesForSliceK * tileM * numSlicesForSliceK * tileN : 0; - // Number of bytes for tile in SMEM. - auto const numBytesSmemTile = tileSize * tg::dtypeGetNumBits(dtypeAcc) / 8 /* bits */; - // Number of bytes alignment for rowMax in SMEM. - auto const numBytesAlignmentTile = 16; - - // Add info. - smemChunkNames.emplace_back("smemSliceK"); - numBytesAndAlignmentPerSmemChunk.emplace_back(std::make_pair(numBytesSmemTile, numBytesAlignmentTile)); - firstChunkReuseSmem.emplace_back(false); - } - // Per-token Scale Factors {{// Number of bytes for per-token scale factors auto const numBytesSmemPerTokenSf = (usePerTokenSfA ? (tileM) * sizeof(float) : 0); @@ -418,26 +407,45 @@ class KernelTraits } // Bias + // + // For DeepSeek FP8 we have two epilogue resources (Epilogue0 and Epilogue1) that share the + // same SMEM stack base pointer. Each handles a different half-tile of the output along the + // (halved) hidden axis. If they both wrote into the same bias SMEM chunk, the later loader + // would clobber the earlier loader's data. So allocate one bias chunk per resource here + // (smemBias0 / smemBias1) and let `getSmemOffsetBias(traits, resIdx)` return the per- + // resource offset. For DeepSeek FP8 + BiasType::Mn, each per-resource bias chunk only + // holds the half-tile (tileM*tileN/2 floats), matching what each Epilogue actually loads. { - int32_t numBytesSmemBias = 0; + int32_t numBytesSmemBiasPerRes = 0; if (isBiasTypeN(biasType)) { - numBytesSmemBias = tileN * sizeof(float); + numBytesSmemBiasPerRes = tileN * sizeof(float); } else if (isBiasTypeM(biasType)) { - numBytesSmemBias = tileM * sizeof(float); + numBytesSmemBiasPerRes = tileM * sizeof(float); } else if (isBiasTypeMn(biasType)) { - numBytesSmemBias = tileM * tileN * sizeof(float); + int32_t const numBiasElts = useDeepSeekFp8 ? (tileM * tileN / 2) : (tileM * tileN); + numBytesSmemBiasPerRes = numBiasElts * sizeof(float); } // Number of bytes alignment for bias auto const numBytesAlignmentBias = 16; - // Add info. - smemChunkNames.emplace_back("smemBias"); - numBytesAndAlignmentPerSmemChunk.emplace_back(std::make_pair(numBytesSmemBias, numBytesAlignmentBias)); - firstChunkReuseSmem.emplace_back(false); + // Only split into per-resource (Epilogue0/Epilogue1) chunks for the DeepSeek FP8 + + // BiasType::Mn path, where each epilogue resource loads its own half-tile of bias + // and they would otherwise clobber each other. For every other configuration — + // including plain DeepSeek FP8 with BiasType::None — keep the original single-chunk + // layout so we don't perturb SMEM offsets of unrelated tests. + int32_t const numBiasResources = (useDeepSeekFp8 && isBiasTypeMn(biasType)) ? 2 : 1; + for (int32_t biasResIdx = 0; biasResIdx < numBiasResources; ++biasResIdx) + { + // Add info. + smemChunkNames.emplace_back("smemBias" + std::to_string(biasResIdx)); + numBytesAndAlignmentPerSmemChunk.emplace_back( + std::make_pair(numBytesSmemBiasPerRes, numBytesAlignmentBias)); + firstChunkReuseSmem.emplace_back(false); + } } // Per-block absolute maximum for multi-warp reduction. @@ -474,19 +482,6 @@ class KernelTraits // Create SMEM helper object. mSmemAllocatorHelper = MemAllocatorHelper(numBytesAndAlignmentPerSmemChunk, firstChunkReuseSmem, smemChunkNames); -#if 0 - // E.g., - // Chunk 0 smemLoadA: 32768 bytes, 1024 alignment, false, offset 0 - // Chunk 1 smemLoadB: 32768 bytes, 1024 alignment, false, offset 32768 - // Chunk 2 smemBShuffle: 0 bytes, 1024 alignment, false, offset 65536 - // Chunk 3 smemGmemC0: 65536 bytes, 1024 alignment, true, offset 0 - // Chunk 4 smemGmemC1: 65536 bytes, 1024 alignment, false, offset 65536 - // Chunk 5 smemRowMax: 512 bytes, 16 alignment, false, offset 131072 - // Chunk 6 smemSliceK: 0 bytes, 16 alignment, false, offset 131584 - // Chunk 7 smemPerTokenSfA: 0 bytes, 16 alignment, false, offset 131584 - // Chunk 8 smemPerTokenSfB: 0 bytes, 16 alignment, false, offset 131584 - mSmemAllocatorHelper.print(); -#endif } // @@ -503,8 +498,8 @@ class KernelTraits // | set0:epiTileN0 | set0:epiTileN1/set1:epiTileN0 | set1:epiTileN1 | auto const numCols = mUseMaxTmemOverlap ? 2 * tileN - epilogueTileN : tileN; // Number of columns for accumulators. - auto const numTmemColsD = numSlicesForSliceK * numCols * numStagesMma * tg::dtypeGetNumBits(dtypeAcc) - / tg::dtypeGetNumBits(tg::Dtype::UInt32); + auto const numTmemColsD + = numCols * numStagesMma * tg::dtypeGetNumBits(dtypeAcc) / tg::dtypeGetNumBits(tg::Dtype::UInt32); // Number of columns for D alignment. auto const numColsAlignmentD = 2; // No need to reuse TMEM. @@ -518,12 +513,10 @@ class KernelTraits // Matrix A { - // We use TMEM for A if we use slice-K or if we need to cast A. - bool const useTmemA = (numSlicesForSliceK > 1) || (dtypeMmaA != dtypeA); - // Number of columns for A. - auto const numTmemColsA = useTmemA ? numStagesA * tileK - / (numSlicesForSliceK * tg::dtypeGetNumBits(tg::Dtype::UInt32) / tg::dtypeGetNumBits(dtypeMmaA)) - : 0; + bool const useTmemA = dtypeMmaA != dtypeA; + auto const numTmemColsA = useTmemA + ? numStagesA * tileK * tg::dtypeGetNumBits(dtypeMmaA) / tg::dtypeGetNumBits(tg::Dtype::UInt32) + : 0; // Number of columns for A alignment. auto const numColsAlignmentA = 4; // No need to reuse TMEM. @@ -692,11 +685,6 @@ inline int32_t getSmemOffsetRowMax(KernelTraits traits) //////////////////////////////////////////////////////////////////////////////////////////////////// -inline int32_t getSmemOffsetSliceK(KernelTraits traits) -{ - return traits.mSmemAllocatorHelper.getChunkOffsetByName("smemSliceK"); -} - //////////////////////////////////////////////////////////////////////////////////////////////////// inline int32_t getSmemOffsetPerTokenSfA(KernelTraits traits) @@ -713,9 +701,16 @@ inline int32_t getSmemOffsetPerTokenSfB(KernelTraits traits) //////////////////////////////////////////////////////////////////////////////////////////////////// -inline int32_t getSmemOffsetBias(KernelTraits traits) +inline int32_t getSmemOffsetBias(KernelTraits traits, int resIdx = 0) { - return traits.mSmemAllocatorHelper.getChunkOffsetByName("smemBias"); + // The allocator decides — based on `useDeepSeekFp8 && isBiasTypeMn(biasType)` — whether to + // create per-resource chunks (smemBias0, smemBias1) or a single shared chunk (smemBias0). + // Rather than have every caller re-derive that predicate, fall back to smemBias0 when the + // requested per-resource chunk does not exist. This keeps the rule in one place — where the + // chunks are added — and lets consumers stay ignorant of the kernel's SMEM layout strategy. + std::string const name = "smemBias" + std::to_string(resIdx); + return traits.mSmemAllocatorHelper.getChunkOffsetByName( + traits.mSmemAllocatorHelper.hasChunkByName(name) ? name : "smemBias0"); } //////////////////////////////////////////////////////////////////////////////////////////////////// diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/config.json b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/config.json index 6349f086368a..a414d177c1ef 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/config.json +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/config.json @@ -10,7 +10,6 @@ "useTwoTmaLoadWarps": true, "clusterDimY": 1, "clusterDimZ": 1, - "sliceK": false, "useTmaStore": true, "useCustomMmaSchedule": true, "gridTriggerSecondaryB": true, @@ -27,11 +26,10 @@ "_template": "BatchedGemmDefault", "tileM": 128, "epilogueTileM": 128, - "numStagesMma": 1, + "numStagesMma": 1, "useShuffledMatrix": true, "useDeepSeekFp8": false }, - "BatchedGemmFp4Base": { "_template": "BatchedGemmNonDsBase", "dtypeA": "e2m1", @@ -74,7 +72,6 @@ "epilogueLdtmDps": 32, "epilogueLdtmBits": 32 }, - "BatchedGemmPerTensorScalingFp8Base": { "_template": "BatchedGemmNonDsBase", "dtypeA": "e4m3", @@ -108,7 +105,6 @@ "epilogueLdtmDps": 32, "epilogueLdtmBits": 32 }, - "BatchedGemmDeepSeekFp8LowLatency": { "_template": "BatchedGemmDefault", "dtypeA": "e4m3", @@ -127,7 +123,6 @@ "batch": "N", "ctaSwizzleType": "rasterizeAlongM" }, - "BatchedGemmMxE2m1E4m3LowLatency": { "_template": "BatchedGemmNonDsBase", "mmaM": 128, @@ -146,7 +141,6 @@ "batch": "N", "ctaSwizzleType": "rasterizeAlongM" }, - "BatchedGemmMxE2m1MxE4m3Base": { "_template": "BatchedGemmNonDsBase", "dtypeC": "mxe4m3", @@ -195,7 +189,6 @@ "epilogueLdtmDps": 32, "epilogueLdtmBits": 32 }, - "BatchedGemmMxE2m1Bf16LowLatency": { "_template": "BatchedGemmNonDsBase", "mmaM": 128, @@ -235,8 +228,15 @@ "_template": "BatchedGemmFp4LowLatency", "routeAct": false, "fusedAct": false, - "useUnrollLoop2xForMma": [true, false], - "dtypeC": ["bf16", "fp16", "e2m1"], + "useUnrollLoop2xForMma": [ + true, + false + ], + "dtypeC": [ + "bf16", + "fp16", + "e2m1" + ], "listN": "8,8", "numExperts": 2, "numStagesMma": 1, @@ -246,9 +246,19 @@ "_template": "BatchedGemmPerTensorScalingFp8LowLatency", "routeAct": false, "fusedAct": false, - "eltwiseActType": ["none", "relu2"], - "useUnrollLoop2xForMma": [true, false], - "dtypeC": ["bf16", "fp16", "e4m3"], + "eltwiseActType": [ + "none", + "relu2" + ], + "useUnrollLoop2xForMma": [ + true, + false + ], + "dtypeC": [ + "bf16", + "fp16", + "e4m3" + ], "tileK": 512, "numStages": 3, "listN": "8,8", @@ -260,8 +270,15 @@ "_template": "BatchedGemmDeepSeekFp8LowLatency", "routeAct": false, "fusedAct": false, - "useUnrollLoop2xForMma": [true, false], - "dtypeC": ["bf16", "fp16", "e4m3"], + "useUnrollLoop2xForMma": [ + true, + false + ], + "dtypeC": [ + "bf16", + "fp16", + "e4m3" + ], "listN": "8,8", "numExperts": 2, "numStagesMma": 2, @@ -274,25 +291,116 @@ "_template": "BatchedGemmFp4LowLatency", "routeAct": "tma", "fusedAct,act,eltwiseActType": [ - [true, "swiglu", "none"], - [true, "geglu", "none"], - [false, "swiglu", "relu2"], - [false, "none", "silu"] + [ + true, + "swiglu", + "none" + ], + [ + true, + "geglu", + "none" + ], + [ + false, + "swiglu", + "relu2" + ], + [ + false, + "none", + "silu" + ] ], "sfLayoutB": "linear", - "useUnrollLoop2xForMma": [true, false], + "useUnrollLoop2xForMma": [ + true, + false + ], "dtypeC": "e2m1", "mmaM,mmaN,tileN,epilogueTileN,tileK,numSlicesForSplitK,clusterDimX,clusterDimZ,numStages,routeSfsAct": [ - [128, 8, 8, 8, 512, 1, 1, 1, 5, "tma"], - [128, 8, 8, 8, 256, 1, 1, 1, 9, "ldgsts"], - [256, 16, 16, 16, 512, 1, 2, 1, 5, "tma"], - [128, 16, 16, 16, 256, 1, 1, 1, 9, "ldgsts"], - [256, 32, 32, 32, 512, 1, 2, 1, 5, "tma"], - [128, 32, 32, 32, 256, 1, 1, 1, 9, "ldgsts"] + [ + 128, + 8, + 8, + 8, + 512, + 1, + 1, + 1, + 5, + "tma" + ], + [ + 128, + 8, + 8, + 8, + 256, + 1, + 1, + 1, + 9, + "ldgsts" + ], + [ + 256, + 16, + 16, + 16, + 512, + 1, + 2, + 1, + 5, + "tma" + ], + [ + 128, + 16, + 16, + 16, + 256, + 1, + 1, + 1, + 9, + "ldgsts" + ], + [ + 256, + 32, + 32, + 32, + 512, + 1, + 2, + 1, + 5, + "tma" + ], + [ + 128, + 32, + 32, + 32, + 256, + 1, + 1, + 1, + 9, + "ldgsts" + ] ], "tileScheduler,numStagesMma": [ - ["static", 1], - ["persistent", 2] + [ + "static", + 1 + ], + [ + "persistent", + 2 + ] ] }, { @@ -301,21 +409,75 @@ "routeAct": "tma", "routeSfsAct": "tma", "fusedAct,act,eltwiseActType": [ - [true, "geglu", "none"], - [false, "none", "relu2"], - [false, "none", "silu"] + [ + true, + "geglu", + "none" + ], + [ + false, + "none", + "relu2" + ], + [ + false, + "none", + "silu" + ] ], "sfLayoutA": "128x4", "sfLayoutB": "linear", "dtypeC": "e2m1", "layoutA": "T", "mmaN,tileN,epilogueTileN,numSlicesForSplitK,clusterDimZ,numStages": [ - [8, 8, 8, 2, 2, 5], - [8, 8, 8, 3, 3, 5], - [8, 8, 8, 4, 4, 5], - [16, 16, 16, 3, 3, 4], - [16, 16, 16, 4, 4, 4], - [32, 32, 32, 2, 2, 3] + [ + 8, + 8, + 8, + 2, + 2, + 5 + ], + [ + 8, + 8, + 8, + 3, + 3, + 5 + ], + [ + 8, + 8, + 8, + 4, + 4, + 5 + ], + [ + 16, + 16, + 16, + 3, + 3, + 4 + ], + [ + 16, + 16, + 16, + 4, + 4, + 4 + ], + [ + 32, + 32, + 32, + 2, + 2, + 3 + ] ], "numStagesMma": 2, "tileScheduler": "persistent" @@ -325,21 +487,64 @@ "_template": "BatchedGemmFp4HighThroughput", "routeAct": "tma", "fusedAct,act,eltwiseActType": [ - [true, "swiglu", "none"], - [true, "geglu", "none"], - [false, "none", "relu2"], - [false, "none", "silu"] + [ + true, + "swiglu", + "none" + ], + [ + true, + "geglu", + "none" + ], + [ + false, + "none", + "relu2" + ], + [ + false, + "none", + "silu" + ] ], "sfLayoutB": "linear", - "useUnrollLoop2xForMma": [true, false], + "useUnrollLoop2xForMma": [ + true, + false + ], "numRegsPerThreadNonEpilogueWarp": 48, "dtypeC": "e2m1", "tileScheduler": "persistent", "numStagesMma": 2, "mmaN,tileN,epilogueTileN,tileK,numStages,sfLayoutC,routeSfsAct": [ - [64, 64, 32, 512, 4, "8x4", "tma"], - [64, 64, 64, 256, 6, "8x4", "ldgsts"], - [128, 128, 32, 256, 6, "128x4", "ldgsts"] + [ + 64, + 64, + 32, + 512, + 4, + "8x4", + "tma" + ], + [ + 64, + 64, + 64, + 256, + 6, + "8x4", + "ldgsts" + ], + [ + 128, + 128, + 32, + 256, + 6, + "128x4", + "ldgsts" + ] ] }, { @@ -348,10 +553,26 @@ "routeAct": "ldgsts", "routeSfsAct": "ldgsts", "fusedAct,act,eltwiseActType": [ - [true, "swiglu", "none"], - [true, "geglu", "none"], - [false, "none", "relu2"], - [false, "none", "silu"] + [ + true, + "swiglu", + "none" + ], + [ + true, + "geglu", + "none" + ], + [ + false, + "none", + "relu2" + ], + [ + false, + "none", + "silu" + ] ], "dtypeC": "e2m1", "sfLayoutB": "linear", @@ -364,7 +585,14 @@ "fuseLoadSfTask": true, "numStagesMma": 2, "mmaN,tileN,epilogueTileN,tileK,numStages,sfLayoutC": [ - [128, 128, 32, 512, 3, "128x4"] + [ + 128, + 128, + 32, + 512, + 3, + "128x4" + ] ] }, { @@ -372,10 +600,26 @@ "_template": "BatchedGemmFp4HighThroughput", "routeAct": "tma", "fusedAct,act,eltwiseActType": [ - [true, "swiglu", "none"], - [true, "geglu", "none"], - [false, "none", "relu2"], - [false, "none", "silu"] + [ + true, + "swiglu", + "none" + ], + [ + true, + "geglu", + "none" + ], + [ + false, + "none", + "relu2" + ], + [ + false, + "none", + "silu" + ] ], "sfLayoutB": "linear", "useUnrollLoop2xForMma": false, @@ -393,7 +637,14 @@ "numRegsPerThreadLoadB": 32, "numRegsPerThreadLoadSfB": 40, "mmaN,tileN,epilogueTileN,tileK,numStages,sfLayoutC": [ - [256, 256, 64, 256, 4, "128x4"] + [ + 256, + 256, + 64, + 256, + 4, + "128x4" + ] ] }, { @@ -402,10 +653,26 @@ "routeAct": "ldgsts", "routeSfsAct": "ldgsts", "fusedAct,act,eltwiseActType": [ - [true, "swiglu", "none"], - [true, "geglu", "none"], - [false, "none", "relu2"], - [false, "none", "silu"] + [ + true, + "swiglu", + "none" + ], + [ + true, + "geglu", + "none" + ], + [ + false, + "none", + "relu2" + ], + [ + false, + "none", + "silu" + ] ], "sfLayoutB": "linear", "useUnrollLoop2xForMma": false, @@ -421,7 +688,14 @@ "numRegsPerThreadEpilogueWarp": 232, "numRegsPerThreadLoadB": 160, "mmaN,tileN,epilogueTileN,tileK,numStages,sfLayoutC": [ - [256, 256, 64, 256, 4, "128x4"] + [ + 256, + 256, + 64, + 256, + 4, + "128x4" + ] ] }, { @@ -429,19 +703,46 @@ "_template": "BatchedGemmFp4HighThroughputNonSwapAb", "routeAct": "tma", "fusedAct,act,eltwiseActType": [ - [true, "swiglu", "none"], - [true, "geglu", "none"], - [false, "none", "relu2"], - [false, "none", "silu"] + [ + true, + "swiglu", + "none" + ], + [ + true, + "geglu", + "none" + ], + [ + false, + "none", + "relu2" + ], + [ + false, + "none", + "silu" + ] ], "sfLayoutA": "linear", - "numWarpsLoadSfA": [2, 4], + "numWarpsLoadSfA": [ + 2, + 4 + ], "useUnrollLoop2xForMma": false, "numRegsPerThreadNonEpilogueWarp": 48, "tileScheduler": "persistent", "numStagesMma": 2, "mmaN,tileN,epilogueTileN,tileK,numStages,sfLayoutB,routeSfsAct": [ - [128, 128, 64, 256, 6, "128x4", "ldgsts"] + [ + 128, + 128, + 64, + 256, + 6, + "128x4", + "ldgsts" + ] ] }, { @@ -449,10 +750,26 @@ "_template": "BatchedGemmFp4HighThroughputNonSwapAb", "routeAct": "tma", "fusedAct,act,eltwiseActType": [ - [true, "swiglu", "none"], - [true, "geglu", "none"], - [false, "none", "relu2"], - [false, "none", "silu"] + [ + true, + "swiglu", + "none" + ], + [ + true, + "geglu", + "none" + ], + [ + false, + "none", + "relu2" + ], + [ + false, + "none", + "silu" + ] ], "sfLayoutA": "linear", "useUnrollLoop2xForMma": false, @@ -465,7 +782,18 @@ "numRegsPerThreadLoadA": 32, "numRegsPerThreadLoadSfA": 40, "mmaN,tileN,epilogueTileN,tileK,numStages,sfLayoutC,numEpilogueWarps,numRegsPerThreadNonEpilogueWarp,numRegsPerThreadEpilogueWarp,ctaSwizzleType": [ - [256, 256, 64, 256, 5, "128x4", 4, 104, 256, "rasterizeAlongN"] + [ + 256, + 256, + 64, + 256, + 5, + "128x4", + 4, + 104, + 256, + "rasterizeAlongN" + ] ] }, { @@ -473,19 +801,88 @@ "_template": "BatchedGemmFp4LowLatency", "routeAct": false, "fusedAct": false, - "useUnrollLoop2xForMma": [true, false], + "useUnrollLoop2xForMma": [ + true, + false + ], "dtypeC": "bf16", "mmaM,mmaN,tileN,epilogueTileN,tileK,numSlicesForSplitK,clusterDimX,clusterDimZ,numStages": [ - [128, 8, 8, 8, 512, 1, 1, 1, 5], - [128, 8, 8, 8, 256, 1, 1, 1, 9], - [256, 16, 16, 16, 512, 1, 2, 1, 5], - [128, 16, 16, 16, 256, 1, 1, 1, 9], - [256, 32, 32, 32, 512, 1, 2, 1, 5], - [128, 32, 32, 32, 256, 1, 1, 1, 9] + [ + 128, + 8, + 8, + 8, + 512, + 1, + 1, + 1, + 5 + ], + [ + 128, + 8, + 8, + 8, + 256, + 1, + 1, + 1, + 9 + ], + [ + 256, + 16, + 16, + 16, + 512, + 1, + 2, + 1, + 5 + ], + [ + 128, + 16, + 16, + 16, + 256, + 1, + 1, + 1, + 9 + ], + [ + 256, + 32, + 32, + 32, + 512, + 1, + 2, + 1, + 5 + ], + [ + 128, + 32, + 32, + 32, + 256, + 1, + 1, + 1, + 9 + ] ], "tileScheduler,numStagesMma": [ - ["static", 1], - ["persistent", 2] + [ + "static", + 1 + ], + [ + "persistent", + 2 + ] ] }, { @@ -497,9 +894,33 @@ "dtypeC": "bf16", "layoutA": "T", "mmaN,tileN,epilogueTileN,numSlicesForSplitK,clusterDimZ,numStagesMma,numStages": [ - [8, 8, 8, 2, 2, 1, 5], - [8, 8, 8, 3, 3, 1, 5], - [8, 8, 8, 4, 4, 1, 5] + [ + 8, + 8, + 8, + 2, + 2, + 1, + 5 + ], + [ + 8, + 8, + 8, + 3, + 3, + 1, + 5 + ], + [ + 8, + 8, + 8, + 4, + 4, + 1, + 5 + ] ], "tileScheduler": "persistent" }, @@ -508,13 +929,30 @@ "_template": "BatchedGemmFp4HighThroughput", "routeAct": false, "fusedAct": false, - "useUnrollLoop2xForMma": [true, false], + "useUnrollLoop2xForMma": [ + true, + false + ], "dtypeC": "bf16", "tileScheduler": "persistent", "numStagesMma": 2, "mmaN,tileN,epilogueTileN,tileK,numStages,sfLayoutB,": [ - [64, 64, 64, 512, 4, "8x4"], - [128, 128, 128, 256, 6, "128x4"] + [ + 64, + 64, + 64, + 512, + 4, + "8x4" + ], + [ + 128, + 128, + 128, + 256, + 6, + "128x4" + ] ] }, { @@ -529,7 +967,14 @@ "fuseUtccpWithUtcmma": true, "useMaxTmemOverlap": true, "mmaN,tileN,epilogueTileN,tileK,numStages,sfLayoutB": [ - [256, 256, 64, 256, 5, "128x4"] + [ + 256, + 256, + 64, + 256, + 5, + "128x4" + ] ] }, { @@ -542,7 +987,14 @@ "tileScheduler": "persistent", "numStagesMma": 2, "mmaN,tileN,epilogueTileN,tileK,numStages,sfLayoutA,": [ - [128, 128, 128, 256, 6, "128x4"] + [ + 128, + 128, + 128, + 256, + 6, + "128x4" + ] ] }, { @@ -556,7 +1008,14 @@ "fuseUtccpWithUtcmma": true, "useMaxTmemOverlap": true, "mmaN,tileN,epilogueTileN,tileK,numStages,sfLayoutA": [ - [256, 256, 64, 256, 5, "128x4"] + [ + 256, + 256, + 64, + 256, + 5, + "128x4" + ] ] }, { @@ -564,19 +1023,68 @@ "_template": "BatchedGemmDeepSeekFp8LowLatency", "routeAct": "tma", "fusedAct": false, - "useUnrollLoop2xForMma": [true, false], + "useUnrollLoop2xForMma": [ + true, + false + ], "numRegsPerThreadNonEpilogueWarp": 80, "dtypeC": "e4m3", "mmaN,tileN,epilogueTileN,numStages,numWarpsLoadB,numRegsPerThreadEpilogueWarp,numRegsPerThreadNonEpilogueWarp": [ - [8, 8, 8, 8, 2, 160, 48], - [16, 16, 16, 6, 2, 160, 48], - [32, 32, 32, 6, 2, 160, 48], - [64, 64, 64, 6, 4, 160, 48], - [128, 128, 128, 5, 4, 152, 80] + [ + 8, + 8, + 8, + 8, + 2, + 160, + 48 + ], + [ + 16, + 16, + 16, + 6, + 2, + 160, + 48 + ], + [ + 32, + 32, + 32, + 6, + 2, + 160, + 48 + ], + [ + 64, + 64, + 64, + 6, + 4, + 160, + 48 + ], + [ + 128, + 128, + 128, + 5, + 4, + 152, + 80 + ] ], "tileScheduler,numStagesMma": [ - ["static", 2], - ["persistent", 4] + [ + "static", + 2 + ], + [ + "persistent", + 4 + ] ], "useShuffledMatrix": false, "layoutA": "T" @@ -586,18 +1094,62 @@ "_template": "BatchedGemmDeepSeekFp8LowLatency", "routeAct": false, "fusedAct": false, - "useUnrollLoop2xForMma": [true, false], + "useUnrollLoop2xForMma": [ + true, + false + ], "dtypeC": "bf16", "mmaN,tileN,epilogueTileN,numStages,numRegsPerThreadEpilogueWarp,numRegsPerThreadNonEpilogueWarp": [ - [8, 8, 8, 8, 160, 48], - [16, 16, 16, 6, 160, 48], - [32, 32, 32, 6, 160, 48], - [64, 64, 64, 6, 160, 48], - [128, 128, 128, 5, 160, 64] + [ + 8, + 8, + 8, + 8, + 160, + 48 + ], + [ + 16, + 16, + 16, + 6, + 160, + 48 + ], + [ + 32, + 32, + 32, + 6, + 160, + 48 + ], + [ + 64, + 64, + 64, + 6, + 160, + 48 + ], + [ + 128, + 128, + 128, + 5, + 160, + 64 + ] ], "tileScheduler,numStagesMma": [ - ["static", 2], - ["persistent", 4] + [ + "static", + 2 + ], + [ + "persistent", + 4 + ] ], "useShuffledMatrix": false, "layoutA": "T" @@ -607,22 +1159,58 @@ "_template": "BatchedGemmPerTensorScalingFp8LowLatency", "routeAct": "tma", "fusedAct,eltwiseActType": [ - [true, "none"], - [false, "relu2"], - [false, "silu"] + [ + true, + "none" + ], + [ + false, + "relu2" + ], + [ + false, + "silu" + ] ], "usePerTokenSfB": true, - "useUnrollLoop2xForMma": [true, false], + "useUnrollLoop2xForMma": [ + true, + false + ], "dtypeC": "e4m3", "layoutA": "T", "mmaN,tileN,epilogueTileN,tileK,numStages": [ - [8, 8, 8, 256, 6], - [16, 16, 16, 256, 6], - [32, 32, 32, 256, 5] + [ + 8, + 8, + 8, + 256, + 6 + ], + [ + 16, + 16, + 16, + 256, + 6 + ], + [ + 32, + 32, + 32, + 256, + 5 + ] ], "tileScheduler,numStagesMma": [ - ["static", 1], - ["persistent", 2] + [ + "static", + 1 + ], + [ + "persistent", + 2 + ] ] }, { @@ -630,22 +1218,62 @@ "_template": "BatchedGemmPerTensorScalingFp8HighThroughput", "routeAct": "tma", "fusedAct,eltwiseActType": [ - [true, "none"], - [false, "relu2"], - [false, "silu"] + [ + true, + "none" + ], + [ + false, + "relu2" + ], + [ + false, + "silu" + ] ], "usePerTokenSfB": true, "numRegsPerThreadNonEpilogueWarp": 56, "numRegsPerThreadEpilogueWarp": 128, "numWarpsLoadB": 8, - "useUnrollLoop2xForMma": [true, false], + "useUnrollLoop2xForMma": [ + true, + false + ], "layoutA": "T", "dtypeC": "e4m3", "mmaN,tileN,epilogueTileN,tileK,numStages,numEpilogueWarps": [ - [64, 64, 64, 256, 5, 4], - [128, 128, 64, 128, 8, 4], - [192, 192, 32, 128, 7, 4], - [256, 256, 64, 128, 6, 8] + [ + 64, + 64, + 64, + 256, + 5, + 4 + ], + [ + 128, + 128, + 64, + 128, + 8, + 4 + ], + [ + 192, + 192, + 32, + 128, + 7, + 4 + ], + [ + 256, + 256, + 64, + 128, + 6, + 8 + ] ], "numStagesMma": 2, "tileScheduler": "persistent" @@ -656,16 +1284,43 @@ "layoutA": "T", "routeAct": false, "fusedAct": false, - "useUnrollLoop2xForMma": [true, false], + "useUnrollLoop2xForMma": [ + true, + false + ], "dtypeC": "bf16", "mmaN,tileN,epilogueTileN,tileK,numStages": [ - [8, 8, 8, 256, 6], - [16, 16, 16, 256, 6], - [32, 32, 32, 256, 5] + [ + 8, + 8, + 8, + 256, + 6 + ], + [ + 16, + 16, + 16, + 256, + 6 + ], + [ + 32, + 32, + 32, + 256, + 5 + ] ], "tileScheduler,numStagesMma": [ - ["static", 1], - ["persistent", 2] + [ + "static", + 1 + ], + [ + "persistent", + 2 + ] ] }, { @@ -676,16 +1331,50 @@ "layoutA": "T", "routeAct": false, "fusedAct": false, - "useUnrollLoop2xForMma": [true, false], + "useUnrollLoop2xForMma": [ + true, + false + ], "dtypeC": "bf16", "mmaN,tileN,epilogueTileN,tileK,numStages,numEpilogueWarps": [ - [64, 64, 64, 128, 8, 4], - [128, 128, 32, 128, 7, 4], - [192, 192, 32, 128, 7, 4], - [256, 256, 32, 128, 6, 8] + [ + 64, + 64, + 64, + 128, + 8, + 4 + ], + [ + 128, + 128, + 32, + 128, + 7, + 4 + ], + [ + 192, + 192, + 32, + 128, + 7, + 4 + ], + [ + 256, + 256, + 32, + 128, + 6, + 8 + ] ], "tileScheduler,numStagesMma": [ - ["persistent", 2] + [ + "persistent", + 2 + ] ] }, { @@ -694,17 +1383,56 @@ "routeAct": "ldgsts", "fusedAct": true, "sfLayoutB": "linear", - "useUnrollLoop2xForMma": [true, false], + "useUnrollLoop2xForMma": [ + true, + false + ], "mmaN,tileN,epilogueTileN,tileK,numStages": [ - [8, 8, 8, 512, 3], - [8, 8, 8, 256, 5], - [16, 16, 16, 256, 5], - [32, 32, 32, 256, 5], - [64, 64, 64, 256, 4] + [ + 8, + 8, + 8, + 512, + 3 + ], + [ + 8, + 8, + 8, + 256, + 5 + ], + [ + 16, + 16, + 16, + 256, + 5 + ], + [ + 32, + 32, + 32, + 256, + 5 + ], + [ + 64, + 64, + 64, + 256, + 4 + ] ], "tileScheduler,numStagesMma": [ - ["static", 1], - ["persistent", 2] + [ + "static", + 1 + ], + [ + "persistent", + 2 + ] ] }, { @@ -712,18 +1440,57 @@ "_template": "BatchedGemmMxE2m1E4m3LowLatency", "routeAct": false, "fusedAct": false, - "useUnrollLoop2xForMma": [true, false], + "useUnrollLoop2xForMma": [ + true, + false + ], "dtypeC": "bf16", "mmaN,tileN,epilogueTileN,tileK,numStages": [ - [8, 8, 8, 512, 3], - [8, 8, 8, 256, 5], - [16, 16, 16, 256, 5], - [32, 32, 32, 256, 5], - [64, 64, 64, 256, 4] + [ + 8, + 8, + 8, + 512, + 3 + ], + [ + 8, + 8, + 8, + 256, + 5 + ], + [ + 16, + 16, + 16, + 256, + 5 + ], + [ + 32, + 32, + 32, + 256, + 5 + ], + [ + 64, + 64, + 64, + 256, + 4 + ] ], "tileScheduler,numStagesMma": [ - ["static", 1], - ["persistent", 2] + [ + "static", + 1 + ], + [ + "persistent", + 2 + ] ] }, { @@ -733,20 +1500,121 @@ "routeSfsAct": "ldgsts", "fusedAct": true, "sfLayoutB": "linear", - "useUnrollLoop2xForMma": [true, false], + "fusedAct,act,biasType": [ + [ + true, + "swiglu", + "m" + ], + [ + true, + "situ", + "none" + ] + ], + "useUnrollLoop2xForMma": [ + true, + false + ], "mmaM,mmaN,tileN,epilogueTileN,tileK,numSlicesForSplitK,clusterDimX,clusterDimZ,numStages": [ - [128, 8, 8, 8, 512, 1, 1, 1, 3], - [128, 8, 8, 8, 512, 2, 1, 2, 3], - [128, 8, 8, 8, 256, 1, 1, 1, 6], - [128, 16, 16, 16, 256, 1, 1, 1, 5], - [256, 16, 16, 16, 256, 1, 2, 1, 6], - [128, 32, 32, 32, 256, 1, 1, 1, 5], - [256, 32, 32, 32, 256, 1, 2, 1, 5], - [128, 8, 8, 8, 256, 2, 1, 2, 6] + [ + 128, + 8, + 8, + 8, + 512, + 1, + 1, + 1, + 3 + ], + [ + 128, + 8, + 8, + 8, + 512, + 2, + 1, + 2, + 3 + ], + [ + 128, + 8, + 8, + 8, + 256, + 1, + 1, + 1, + 6 + ], + [ + 128, + 16, + 16, + 16, + 256, + 1, + 1, + 1, + 5 + ], + [ + 256, + 16, + 16, + 16, + 256, + 1, + 2, + 1, + 6 + ], + [ + 128, + 32, + 32, + 32, + 256, + 1, + 1, + 1, + 5 + ], + [ + 256, + 32, + 32, + 32, + 256, + 1, + 2, + 1, + 5 + ], + [ + 128, + 8, + 8, + 8, + 256, + 2, + 1, + 2, + 6 + ] ], "tileScheduler,numStagesMma": [ - ["static", 1], - ["persistent", 2] + [ + "static", + 1 + ], + [ + "persistent", + 2 + ] ] }, { @@ -756,12 +1624,83 @@ "routeSfsAct": "ldgsts", "fusedAct": true, "sfLayoutB": "linear", - "useUnrollLoop2xForMma": [true, false], + "fusedAct,act,biasType": [ + [ + true, + "swiglu", + "m" + ], + [ + true, + "situ", + "none" + ] + ], + "useUnrollLoop2xForMma": [ + true, + false + ], "mmaN,tileN,epilogueTileN,tileK,numStages,sfLayoutC,numRegsPerThreadEpilogueWarp,numRegsPerThreadNonEpilogueWarp,numStagesMma,numWarpsLoadB,numWarpsLoadSfB,numRegsPerThreadLoadB,numRegsPerThreadLoadSfB": [ - [64, 64, 64, 256, 4, "8x4", 168, 48, 2, 1, 1, 48, 48], - [64, 64, 64, 128, 7, "8x4", 168, 48, 2, 1, 1, 48, 48], - [128, 128, 32, 256, 4, "128x4", 152, 48, 2, 8, 4, 48, 48], - [128, 128, 32, 128, 7, "128x4", 152, 48, 2, 8, 4, 48, 48] + [ + 64, + 64, + 64, + 256, + 4, + "8x4", + 168, + 48, + 2, + 1, + 1, + 48, + 48 + ], + [ + 64, + 64, + 64, + 128, + 7, + "8x4", + 168, + 48, + 2, + 1, + 1, + 48, + 48 + ], + [ + 128, + 128, + 32, + 256, + 4, + "128x4", + 152, + 48, + 2, + 8, + 4, + 48, + 48 + ], + [ + 128, + 128, + 32, + 128, + 7, + "128x4", + 152, + 48, + 2, + 8, + 4, + 48, + 48 + ] ], "tileScheduler": "persistent" }, @@ -780,9 +1719,35 @@ "fuseLoadSfTask": true, "fuseUtccpWithUtcmma": true, "numStagesMma": 2, + "fusedAct,act,biasType": [ + [ + true, + "swiglu", + "m" + ], + [ + true, + "situ", + "none" + ] + ], "mmaN,tileN,epilogueTileN,tileK,numStages,sfLayoutC": [ - [128, 128, 32, 256, 4, "128x4"], - [128, 128, 32, 128, 7, "128x4"] + [ + 128, + 128, + 32, + 256, + 4, + "128x4" + ], + [ + 128, + 128, + 32, + 128, + 7, + "128x4" + ] ] }, { @@ -795,8 +1760,34 @@ "sfLayoutA": "linear", "useUnrollLoop2xForMma": false, "mmaN,tileN,epilogueTileN,tileK,numStages,sfLayoutC,numRegsPerThreadEpilogueWarp,numRegsPerThreadNonEpilogueWarp,numWarpsLoadA,numWarpsLoadSfA,numRegsPerThreadLoadA,numRegsPerThreadLoadSfA": [ - [128, 128, 64, 256, 4, "128x4", 152, 48, 8, 4, 48, 48], - [128, 128, 64, 128, 7, "128x4", 152, 48, 8, 4, 48, 48] + [ + 128, + 128, + 64, + 256, + 4, + "128x4", + 152, + 48, + 8, + 4, + 48, + 48 + ], + [ + 128, + 128, + 64, + 128, + 7, + "128x4", + 152, + 48, + 8, + 4, + 48, + 48 + ] ], "tileScheduler": "persistent", "numStagesMma": 2 @@ -811,9 +1802,47 @@ "fuseUtccpWithUtcmma": true, "useMaxTmemOverlap": true, "useUnrollLoop2xForMma": false, + "fusedAct,act,biasType": [ + [ + true, + "swiglu", + "m" + ], + [ + true, + "situ", + "none" + ] + ], "mmaN,tileN,epilogueTileN,tileK,numStages,sfLayoutC,numRegsPerThreadEpilogueWarp,numRegsPerThreadNonEpilogueWarp,numWarpsLoadB,numWarpsLoadSfB,numRegsPerThreadLoadB,numRegsPerThreadLoadSfB": [ - [256, 256, 64, 128, 6, "128x4", 144, 88, 8, 4, 56, 56], - [256, 256, 64, 256, 3, "128x4", 144, 88, 8, 4, 56, 56] + [ + 256, + 256, + 64, + 128, + 6, + "128x4", + 144, + 88, + 8, + 4, + 56, + 56 + ], + [ + 256, + 256, + 64, + 256, + 3, + "128x4", + 144, + 88, + 8, + 4, + 56, + 56 + ] ], "tileScheduler": "persistent", "numStagesMma": 1 @@ -823,21 +1852,110 @@ "_template": "BatchedGemmMxE2m1MxE4m3LowLatency", "routeAct": false, "fusedAct": false, - "useUnrollLoop2xForMma": [true, false], + "useUnrollLoop2xForMma": [ + true, + false + ], "dtypeC": "bf16", "mmaM,mmaN,tileN,epilogueTileN,tileK,numSlicesForSplitK,clusterDimX,clusterDimZ,numStages": [ - [128, 8, 8, 8, 512, 1, 1, 1, 3], - [128, 8, 8, 8, 512, 2, 1, 2, 3], - [128, 8, 8, 8, 256, 1, 1, 1, 6], - [128, 16, 16, 16, 256, 1, 1, 1, 5], - [256, 16, 16, 16, 256, 1, 2, 1, 6], - [128, 32, 32, 32, 256, 1, 1, 1, 5], - [256, 32, 32, 32, 256, 1, 2, 1, 5], - [128, 8, 8, 8, 256, 2, 1, 2, 6] + [ + 128, + 8, + 8, + 8, + 512, + 1, + 1, + 1, + 3 + ], + [ + 128, + 8, + 8, + 8, + 512, + 2, + 1, + 2, + 3 + ], + [ + 128, + 8, + 8, + 8, + 256, + 1, + 1, + 1, + 6 + ], + [ + 128, + 16, + 16, + 16, + 256, + 1, + 1, + 1, + 5 + ], + [ + 256, + 16, + 16, + 16, + 256, + 1, + 2, + 1, + 6 + ], + [ + 128, + 32, + 32, + 32, + 256, + 1, + 1, + 1, + 5 + ], + [ + 256, + 32, + 32, + 32, + 256, + 1, + 2, + 1, + 5 + ], + [ + 128, + 8, + 8, + 8, + 256, + 2, + 1, + 2, + 6 + ] ], "tileScheduler,numStagesMma": [ - ["static", 1], - ["persistent", 2] + [ + "static", + 1 + ], + [ + "persistent", + 2 + ] ] }, { @@ -845,13 +1963,56 @@ "_template": "BatchedGemmMxE2m1MxE4m3HighThroughput", "routeAct": false, "fusedAct": false, - "useUnrollLoop2xForMma": [true, false], + "useUnrollLoop2xForMma": [ + true, + false + ], "dtypeC": "bf16", "mmaN,tileN,epilogueTileN,tileK,numStages,sfLayoutB,numRegsPerThreadEpilogueWarp,numRegsPerThreadNonEpilogueWarp,numStagesMma": [ - [64, 64, 64, 256, 4, "8x4", 168, 96, 2], - [64, 64, 64, 128, 7, "8x4", 168, 96, 2], - [128, 128, 64, 256, 4, "128x4", 168, 96, 2], - [128, 128, 64, 128, 7, "128x4", 168, 96, 2] + [ + 64, + 64, + 64, + 256, + 4, + "8x4", + 168, + 96, + 2 + ], + [ + 64, + 64, + 64, + 128, + 7, + "8x4", + 168, + 96, + 2 + ], + [ + 128, + 128, + 64, + 256, + 4, + "128x4", + 168, + 96, + 2 + ], + [ + 128, + 128, + 64, + 128, + 7, + "128x4", + 168, + 96, + 2 + ] ], "tileScheduler": "persistent" }, @@ -865,7 +2026,16 @@ "useUnrollLoop2xForMma": false, "dtypeC": "bf16", "mmaN,tileN,epilogueTileN,tileK,numStages,sfLayoutB,numRegsPerThreadEpilogueWarp,numRegsPerThreadNonEpilogueWarp": [ - [256, 256, 64, 128, 5, "128x4", 144, 96] + [ + 256, + 256, + 64, + 128, + 5, + "128x4", + 144, + 96 + ] ], "tileScheduler": "persistent", "numStagesMma": 1 @@ -876,21 +2046,78 @@ "routeAct": "ldgsts", "fusedAct": true, "sfLayoutB": "linear", - "useUnrollLoop2xForMma": [true, false], + "useUnrollLoop2xForMma": [ + true, + false + ], "dtypeC": "bf16", "mmaN,tileN,epilogueTileN,tileK,numStages": [ - [8, 8, 8, 128, 5], - [8, 8, 8, 256, 3], - [16, 16, 16, 128, 5], - [16, 16, 16, 256, 3], - [32, 32, 32, 128, 5], - [32, 32, 32, 256, 3], - [64, 64, 64, 128, 5], - [64, 64, 64, 256, 3] + [ + 8, + 8, + 8, + 128, + 5 + ], + [ + 8, + 8, + 8, + 256, + 3 + ], + [ + 16, + 16, + 16, + 128, + 5 + ], + [ + 16, + 16, + 16, + 256, + 3 + ], + [ + 32, + 32, + 32, + 128, + 5 + ], + [ + 32, + 32, + 32, + 256, + 3 + ], + [ + 64, + 64, + 64, + 128, + 5 + ], + [ + 64, + 64, + 64, + 256, + 3 + ] ], "tileScheduler,numStagesMma": [ - ["static", 1], - ["persistent", 2] + [ + "static", + 1 + ], + [ + "persistent", + 2 + ] ] }, { @@ -898,17 +2125,46 @@ "_template": "BatchedGemmMxE2m1Bf16LowLatency", "routeAct": false, "fusedAct": false, - "useUnrollLoop2xForMma": [true, false], + "useUnrollLoop2xForMma": [ + true, + false + ], "dtypeC": "bf16", "mmaN,tileN,epilogueTileN,numStages": [ - [8, 8, 8, 3], - [16, 16, 16, 3], - [32, 32, 32, 3], - [64, 64, 64, 3] + [ + 8, + 8, + 8, + 3 + ], + [ + 16, + 16, + 16, + 3 + ], + [ + 32, + 32, + 32, + 3 + ], + [ + 64, + 64, + 64, + 3 + ] ], "tileScheduler,numStagesMma": [ - ["static", 1], - ["persistent", 2] + [ + "static", + 1 + ], + [ + "persistent", + 2 + ] ] }, { @@ -918,18 +2174,57 @@ "fusedAct": true, "sfLayoutB": "linear", "mmaK": 32, - "useUnrollLoop2xForMma": [true, false], + "useUnrollLoop2xForMma": [ + true, + false + ], "dtypeC": "e4m3", "mmaN,tileN,epilogueTileN,tileK,numStages": [ - [8, 8, 8, 512, 3], - [8, 8, 8, 256, 5], - [16, 16, 16, 256, 5], - [32, 32, 32, 256, 5], - [64, 64, 64, 256, 4] + [ + 8, + 8, + 8, + 512, + 3 + ], + [ + 8, + 8, + 8, + 256, + 5 + ], + [ + 16, + 16, + 16, + 256, + 5 + ], + [ + 32, + 32, + 32, + 256, + 5 + ], + [ + 64, + 64, + 64, + 256, + 4 + ] ], "tileScheduler,numStagesMma": [ - ["static", 1], - ["persistent", 2] + [ + "static", + 1 + ], + [ + "persistent", + 2 + ] ] }, { @@ -938,18 +2233,57 @@ "routeAct": false, "fusedAct": false, "mmaK": 32, - "useUnrollLoop2xForMma": [true, false], + "useUnrollLoop2xForMma": [ + true, + false + ], "dtypeC": "bf16", "mmaN,tileN,epilogueTileN,tileK,numStages": [ - [8, 8, 8, 512, 3], - [8, 8, 8, 256, 5], - [16, 16, 16, 256, 5], - [32, 32, 32, 256, 5], - [64, 64, 64, 256, 4] + [ + 8, + 8, + 8, + 512, + 3 + ], + [ + 8, + 8, + 8, + 256, + 5 + ], + [ + 16, + 16, + 16, + 256, + 5 + ], + [ + 32, + 32, + 32, + 256, + 5 + ], + [ + 64, + 64, + 64, + 256, + 4 + ] ], "tileScheduler,numStagesMma": [ - ["static", 1], - ["persistent", 2] + [ + "static", + 1 + ], + [ + "persistent", + 2 + ] ] } ] diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x128x256_s6_et128x128_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x128x256_s6_et128x128_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index a381be8089fa..38a41a94e3fc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x128x256_s6_et128x128_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x128x256_s6_et128x128_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c74546b084a88375b4abe1559ba23c868a9588af6fe2081279c7144465ead08f -size 22402 +oid sha256:72865a03c95dc195560b5370cfaa527a7e66fe8f22522db0bf07f3d52b140c76 +size 56714 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x128x256_s6_et128x128_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x128x256_s6_et128x128_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 93e18673caa5..65a1d0fdc5c5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x128x256_s6_et128x128_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x128x256_s6_et128x128_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4c7325983802b10b04caad65ec9643c6ffd92e23e3b7d9c28fd4a95ae6795ae8 -size 23808 +oid sha256:1a28dac8facd8aef6c2d317418842830439b72b641a2a5734a6df3a93c8fd46b +size 59752 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x128x256u2_s6_et128x128_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x128x256u2_s6_et128x128_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 36f41cbfd6c5..95c2816621ff 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x128x256u2_s6_et128x128_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x128x256u2_s6_et128x128_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1a05ebb0385a121c86bb56dc893449adacbeadb25d5667e601a6783dd49717d7 -size 23232 +oid sha256:c8476135c22f23c5da3d32b311882410b13a50e313ea3c149baaa1e5594f96ed +size 59702 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 258f0abf1685..943cdb969146 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ae10bcf2f056afa012b274e3567a79171d175251f7245777e1f35959ddbde930 -size 17445 +oid sha256:12eafca1284eca5dfef2341f6006afc6750a3e22745a64abc277e73da4516b62 +size 45047 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 283f9c30ad3d..0c671369e762 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a1bc9438650f857d81afbb3357a366e904c4a17683f890c0874dc5b5beb3c687 -size 13204 +oid sha256:821d8daa90c238a88573c19bc58587170874d8930dceda6d7801fa9c9983572b +size 31181 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index be9d856d7b09..d98dc14c30b1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1234b8f93b7590954fd8cc1b4e63404522ccc9572ee8935d2169e1848d02aebb -size 18263 +oid sha256:9a92bcbbea1c255499de3a054d5486814109fd0516b13307982c388c528fa821 +size 47539 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 77be81633219..55fd7d890258 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:922a5a129580d369788a4a0a49fd4099ed354240efb37a34407aaecad65b9d59 -size 14106 +oid sha256:4760aa7b80aa9bcee2833a41e978e8e505f33544bd77e78fc766f75e900b0e89 +size 34419 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 76e3be9227d6..18c57fb19ed7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ff0add01df4ac873e0fc7db98cc7eb2b1180543b9562138cf0ce6b031a08feef -size 19493 +oid sha256:74e31f7c293b7767d030ed596d6869d8bd12884e47d798b3c193d26a7e7e0132 +size 52217 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index fd15f9d1a3b8..1f5825b525ed 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fc31bbd1da354e851e331651f40ecc3b873df753a1b1a5f026e0113f14452e4d -size 15076 +oid sha256:5e877a928963b033fdba07cb9fbe3ca0af4a3d6677f15b8e4432ec35b1574d9d +size 38463 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index b6d41caa54c9..ff0b22783338 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d659ead9e774d831df1623132d40d524c1de315b354e030ae94751d1da5b55b5 -size 20945 +oid sha256:55fb411e9104e4010e9c41c524fbb365a2d34cc2060b5742101bae0cbd14af0e +size 56398 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 714eb39fe9f7..7f62072b8ec6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7c81658a06228c636917427192cb8928f24c88e3d8b8ae73318344c9ccc6933d -size 16181 +oid sha256:84ecaef85326a5b15242a46c803c295615b6c8c09ecfa41f504cd88406345803 +size 42367 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x256x256_s5x5x5x5x1x1_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x256x256_s5x5x5x5x1x1_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 443c98ce2f27..f200a2f81162 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x256x256_s5x5x5x5x1x1_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x256x256_s5x5x5x5x1x1_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:877062e8cbcc7b0f74eff8159f621a48ca0354a532ac24c9bd0c822fafd0b09a -size 21350 +oid sha256:5d7f298dcd0b65e0d9660df5e5ea0fcfb4f8543e4b9668abc88ba6e7dc5ada86 +size 59278 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x256x256_s5x5x5x5x1x1_et128x64_m256x256x64_c2x1x1_rN_TN_schPdx3_biasFp32N_fCp_tmOv_bM_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x256x256_s5x5x5x5x1x1_et128x64_m256x256x64_c2x1x1_rN_TN_schPdx3_biasFp32N_fCp_tmOv_bM_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 374d5acb1566..ec7e48bd0b3e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x256x256_s5x5x5x5x1x1_et128x64_m256x256x64_c2x1x1_rN_TN_schPdx3_biasFp32N_fCp_tmOv_bM_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x256x256_s5x5x5x5x1x1_et128x64_m256x256x64_c2x1x1_rN_TN_schPdx3_biasFp32N_fCp_tmOv_bM_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bcdb9692cfb84ce7e1f486fc07a2aaf0db16cbc2baa75414e8be57466cddeb3c -size 26664 +oid sha256:0149eccae89ac422e36c915e980eaaf54eb9a50bd87072ed03e15250e189e67b +size 69349 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 904f3f3f6a9c..f8d41cf9379b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ae26400587d724cc00c57775da5ca1e4db5229566920d3a9c4261234ecec8fdf -size 18260 +oid sha256:c4351aaa4c8f52308c675bfabd7d06b82ef48b66a642d0fe7e5ebdfeaa263714 +size 45957 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 49ffc7e5b5d1..d4d9966b8728 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e8fef4b1c0c730ece00a6190ec266b152b0d39cd95780cc41d98901a1b54f3cb -size 13569 +oid sha256:7e8a4fc4a1efaf58fb321d9616548b10b59e9341d56867d6f344a9cc66b9a2bd +size 32547 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 2786e4f1c3e4..c051daf411a6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:485d51cb96a44add7dc411847bad0255bf41fae7760c2838df575d646b57224e -size 18995 +oid sha256:4a9eb1e0a5dba47adc591bd35ec14bffd5819cc11d66cc96fd6c1146e72ffb84 +size 48605 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 9d2ad0cc987c..9a15dfc88b23 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0bc10ec6ddf19f28f221478b311076f4c4222e7a3e89753814f7929041188350 -size 14493 +oid sha256:fdb40c9e1ca7510051627750905ff5a12dd19e4b5eed584ad8f13ce8d503fa89 +size 35641 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 8d80b1fead2a..ae2031e3ae32 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:48e10f591c2bed591ab38791366e631a28ab0cfed5232ae28c3f126f9ea071b6 -size 19808 +oid sha256:b105f6e14087a91775da03a85e4340a11296ba6f5b52cb0013269cba1f22ce20 +size 53377 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 878f00a462e1..e3e0890f7e83 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ad353e7b0d265af545f50798f83198c731369c1bdfd9bfff3cfec1a1ae868ce9 -size 15449 +oid sha256:0a76fe04064e1e5607c63c4bad0a657a51aa406d62a6b903f11e36a64ae78a33 +size 39556 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 0d2f6f02ecd4..3e912d3fafe0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:dfd678561f26172b6e5c4bfab1f5d6ed94198cc8b7fa476dc9dbb03b625a6f36 -size 21465 +oid sha256:7584045facf596d373b5f0c79b9c3b887eba25c1b0fce0e80b2ffeaf9de4d88a +size 57254 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index b0278cf463ba..4db5bb79b901 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:be18e49f82d7ad1b691f184b838c07c2b702fb12ffd9f1271cf106e7f1602897 -size 16481 +oid sha256:ac2d9dcbfa012af67a197c53d4499e74e492fae16312bd14e11f1347667f959f +size 43528 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x64x512_s4_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x64x512_s4_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index b68cc3a8f649..1a8feb3f53de 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x64x512_s4_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x64x512_s4_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9c5fb984ebeed0a15681440ff9966508ba421e5d4bb6a9bb7b889cef40495896 -size 20430 +oid sha256:a84bb2dfc2f8f86bc1d4dd24a358c2cc406f5bf4eb987c8a0ab1a2142c98146f +size 55439 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x64x512u2_s4_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x64x512u2_s4_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index ab0fd8b45c9f..a87895fab23f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x64x512u2_s4_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x64x512u2_s4_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c5681e416064a9c07c2e0feeec7da4dc98bb7e3257bf9e41c69b0434e17f2df6 -size 22066 +oid sha256:2d81fe64a56da64c5971c2a540b2a67c0cc4e240dddde43ac4d94b01f21715f9 +size 59277 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 1ed9e8de9d55..b6d34d34b672 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:97c68d4becd537e501342309dc780a4f1aaa8ed2aa8ff086f540cd458ba9f694 -size 17498 +oid sha256:c0326c868044643adeb64487431833e655dd1f6166908c3f486f8b63fb1d3dc4 +size 44522 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 593c600fcbcd..6af3439cb12b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d736b49b20d2b75ccec4f14da3f1e9095ec982e5d6a56b9ba7bb203731cdce64 -size 12899 +oid sha256:e74e7d7092d907f27c1bb88a0c124cdd96fc37eaef0f83e5aa82413e677c90f9 +size 30579 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 9fdd895af333..b457a6c55055 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:85176270de633bf5cd7ed92d82c1210a8a41f6cd2eff99e7d32401be450a93e0 -size 18323 +oid sha256:4430512c817c8ffc7076ed62a8702b6ae5b3b315b6e31dd83eef3f64066f89c5 +size 47187 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 08b82994879b..b6e09d40bd4e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:41ee659cecc38385c31eddc12d5709e11864e84d24702e7e4454bd71d3df2750 -size 13780 +oid sha256:7c211df05929d1ed697cc050ff64bb078154a12ff0cc20a5e3b263e20572adc8 +size 33617 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f.cubin.tar.zst index e49424c759cb..b97f07143033 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a484e2f493f4a00fc2d0f4ef23fb1016205e5de8328f19701e7e963ab94aefc2 -size 12297 +oid sha256:4986a4038939d3c0a8e7d463ff4273ed63904ac27983ac213b0218bc1647d882 +size 31733 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 37c7934eb3e9..2cb128a8605c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:dc148dc17540e7fc5ac00c52e8eae6d94341e8bb255347cc2463e67c7b4d0f5a -size 17298 +oid sha256:9b53fbb1350459902cdaab9fe40db07b37c58e96374f40dc0c74f7570a58785a +size 46009 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index df635045e763..ce8530f1b5df 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:94164bc9b7acffa3457cf22c8732d537231e3c131199c704a7ab892ea582293f -size 12529 +oid sha256:b1a49873eade749350d6da1ab0b531bd88754c08a0bdfed998c174eede50e13f +size 31986 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x2_rM_splitK2_TN_transOut_schPdx3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x2_rM_splitK2_TN_transOut_schPdx3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 9e9ef6969e93..244473ca0949 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x2_rM_splitK2_TN_transOut_schPdx3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x2_rM_splitK2_TN_transOut_schPdx3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:af81a1b8b51710f48c04b559ca2026cc2a14e6c4678851fc87ca23b660ba614f -size 22670 +oid sha256:c308b17364776352412f3f2cc59255e5deb8e401e0dc611db27b2077b94a96f4 +size 55848 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x3_rM_splitK3_TN_transOut_schPdx3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x3_rM_splitK3_TN_transOut_schPdx3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index f171709c956f..02362b07e040 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x3_rM_splitK3_TN_transOut_schPdx3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x3_rM_splitK3_TN_transOut_schPdx3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:10c4b463c85d718d73dc63e3db49601e703d490bdb97651d6fcaeee90bd98875 -size 22764 +oid sha256:9931b594d0b7d4c43a5cbec103d6fc18d00ee017c59654d4ed2089bc856ad418 +size 56166 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x4_rM_splitK4_TN_transOut_schPdx3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x4_rM_splitK4_TN_transOut_schPdx3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 0970472d8071..aaa3876f0fca 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x4_rM_splitK4_TN_transOut_schPdx3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x4_rM_splitK4_TN_transOut_schPdx3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1dbb6e8cedbba2606bc7eabef0e357bcc24a55b814875af8bed66688183ee12d -size 22822 +oid sha256:5a363c1918406d891e2122a4bf602f4b5c34da0e51a5e981e3b797070d55f73e +size 56065 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512u2_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512u2_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f.cubin.tar.zst index 2b1e013249c2..65f35cbdcb94 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512u2_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512u2_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:dbd015d3b5ad313bdc0089c52c924ea4d14b357ad145c45077e526e6525c756e -size 13438 +oid sha256:58fbb3626034a9238223cf3a434500fecb9ad70d0725cf2e4c8464eb9baf8177 +size 35574 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index bce84e49b361..8689edcedf01 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c566d57aec81ffa9f2a2c389af8363b1fb5743d670e6bb031901d7c6bee7dffd -size 18258 +oid sha256:0a249ff45e649574a88a04303f97552f3a81a35e9938482e8a2f34d30f8d9027 +size 49499 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 47061f741165..f3eae5c94ce7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0d08673f307b1230878618a486b0dd96728eeab9adef37b4b560f4c34c907ce2 -size 13665 +oid sha256:605b8ab73db7742035df540bbabe0cf33d7d87b52438a6fe7d40c98f41db8dad +size 35784 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst index 00ea404cbb7f..59a87afbba27 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:411ca4da28e5d62043bed6f71ce639b68d59e73810265330cfb2b4d5b08e4c66 -size 17962 +oid sha256:4085a54c25a50b0696825aeccd6f41124e1fb962edb16d13aa0fbf6e8e714c83 +size 46353 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst index cebc2a16a055..f0534460a6d9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4f2334da2f3883aaa87ec567760f9ce84412d4561da828e6f5c113c26ccef863 -size 18048 +oid sha256:74afc2e3cbca778ca7801e183f850fb9e28a4613514aa5c903bb59b0755cee92 +size 46209 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst index 638c02e77c78..16a0975d25d9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e814094544aa10a724bea31300c41f611c03c65d13e3c8d5aebe892e762f6f52 -size 13585 +oid sha256:51e8fee541b65e1d7535aa8aa43babe1c5f0fa9da5c97d05f861a52a13a9c1d6 +size 33869 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst index 813d803f3d01..37eac6947636 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fd6f96413b495f31dd5a13abcaf0151a447efe7337c0fbc8041f740eddffe307 -size 13690 +oid sha256:b47a26d6eca7cc389694755dfbe41c6303adeac2caa2eaa1e4548a9ffdde7fb8 +size 33885 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst index 7911347bb039..3cfeb078f116 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:aeefe93596bf6388eed4cee7a4ce3730170ad2eca7784735b36c56df543ff518 -size 18515 +oid sha256:22c64397b9e044eadc6a64afc055dd3a7646bfa02f19bac064ef34f5aa43c181 +size 48643 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst index 7e3abb706714..c1b1e12c39bd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f6aba4378ef6e1e357861257deb486879eb8c54c4ac2ed1788d3e3bb8f7da265 -size 18607 +oid sha256:bcd430b11921983a1ea9fd0847be7d4b2af58c35e20faf015bd5a520e78624c2 +size 48683 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst index a160e17536ae..cc6f4c8d98f5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1f85a1cda748a67d7837100339810a5829b9b4482b8d2384a7afd27fcf33b49b -size 14214 +oid sha256:0cf5864162fc935c4be88fc654349f2f1cba7ba013d4159571bc374af40860d0 +size 36201 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst index c809c0ad7fd9..e6c6cce2e735 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:08a9ff7cc6974fbd80d0d875957c2fc95d7534fde9df976bf7e764afce9864b1 -size 14317 +oid sha256:2978bf3fc06cd3230fdb7ffd287d79d57e9796b343dc8b3d5e5df947e44cad02 +size 36190 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst index 602ad7e04f50..61780e5631ab 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:97fe1852ce0b006cb3019573eda6ab4746001db646aedbb62a16fbfec512c3e2 -size 18655 +oid sha256:d35d6c7ae9501a1cfb1eb9f8bd17cc60edde66d97e68e3721c4e92d81cb7c76e +size 47478 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst index 7c96c29d1561..d22c7bd067fb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9821f2ac83fec338e9810fbdcf371d15c97cd968b45f1b143f7fb17d97546d95 -size 18737 +oid sha256:a55cd59af31012b085548131c97d2b6f8b0c373d6aff44031cf0d222e60cab97 +size 47450 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst index 9832b94109e3..3aabf957c9a1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8b7628a5b594d3b6834ae89e56d8c5a59c982e484b3f6691eb8d7ef700d7cc54 -size 13949 +oid sha256:0fc97c441c33c8be29206a33ad8a17b834d1fe4ceb044a32b875670809d0d6fe +size 35135 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst index e0fc148772f3..002fd87b0e77 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0b9f9e0e837b0f9d4be420e77179753dca78325b574da7421544864693443204 -size 14040 +oid sha256:2445181254406fe2b465847bb13ed2a9b9e32354146d1f157116576b447ced6a +size 35152 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst index 2616f616f0ae..8a5e74d5492e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6bb71d9f18997c23f0508d92d995fa12932ad49a46c7e13374fdb6816017832d -size 19165 +oid sha256:c84cfc9c0358b4a6b84cdaf91864c3bdbf341e34f0c9aba2b1523575a3f4159f +size 49660 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst index b0fa6c859fcf..3201c700a48b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5a1a3276b5ade4f637e4a7cef60d465de3ec9fe9348cf7c8b326f711b4d398a7 -size 19264 +oid sha256:77b654462e57588a6aedf0472b51a989992c098e2c6c5a1fb759d4c065e3dbf7 +size 49698 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst index 8ae409bb53ad..9193d37bcd8b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bcffa20ee596cf12a66255b6931e6828372d4cdd03f89f3414f7cb86579794d8 -size 14541 +oid sha256:f30d0f6609fdbe199a7fef1012bb65517b4234bb9c889f8c019d1529d336b5b6 +size 37446 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst index b7db646082b5..027843084233 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9418e9e2f2733e7a77ef1635b1f8a122ddd973d6c617b8b173117d7a902e01d1 -size 14687 +oid sha256:9aa9eb30390baa8bddd3a377182009b2c607b2ca23a9a4c35d944e4b4f55aff3 +size 37449 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst index 04f0d4c2d33f..2009e3c46fac 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:af2406bd58464204de548a493607e13498bb25f07d7b5fa607a654fd92c687fb -size 19048 +oid sha256:b2e943b1bb5515e55424645e6ead5566f2ba78480b663e53a22aa336c9eadd4e +size 49020 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst index 181ac72a1ca4..b9780c3592ee 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:01c1faa522b305bae08d76d9bb2df47f634bb13d572c8d2e41b2a14f2be84d64 -size 19155 +oid sha256:954983ec994893f0d3fef11a31e5192304826db34abbcf9464736bfeab54ea9a +size 48956 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst index 6ecdf6fffc1f..51a6a96b12be 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:16353237fcbdcdb1c98f5b9676787131b79b98d4b40cbd4c99d5ebc8e8cd3039 -size 14553 +oid sha256:7e50908288be50474f4ec82df57b1e6b741dcb3e77435325d7977f24b9fc54b8 +size 37267 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst index 81f1eca9391b..22f8bb871d92 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ce0c7ada0f4a7b45599221f72a0b842ba22b83601144de53d11e45f4e286cf09 -size 14634 +oid sha256:9db16d128a153218546fc268b5bb0e35594059dc38ddd8fadb50b032b8be9590 +size 37292 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst index a552db9844c7..2ac9886c6f10 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5ba129a0ae0f126e9a4bb7f987aae6244cb18b8b87f27a40908a9a8cb544ca82 -size 19576 +oid sha256:53c8bcf247ce0723561c58a673ca4db65d68445d57767e8c3b6b5482b78997bb +size 51549 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst index 15a416a793a1..323568770340 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c23fa358ae1d18a075c21a76bb435f0f0310b88544b70194b0b5a5c2ef80d118 -size 19722 +oid sha256:ab7a306e9412a6754402f69d37e583b0101aec506cd0e4c82b19d60dc4b9eb5d +size 51591 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst index 8214aa86ec29..ffd14b91db9b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7937e9a2874c9830576de386b27d1c0a16d0a441a24cf1f4278773ba0945e2e0 -size 15168 +oid sha256:9a25d7170d5e57e4349b3086331275a3c01e4cec0662ae800fa09a365aaf4b9d +size 39489 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst index 036c5ae2b97f..9fcc16deb41c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:69121b470f41b345628daa7afb6d0c14b4df8479ed9e95c3be23f82aa04b798f -size 15268 +oid sha256:d2cfa1fb837cebe8ad782e8d5c85f776422d070a612fb34c2f51f8483536e1c3 +size 39587 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst index 6a0e8577c002..5ced41f42f8c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f4dfcfd7b2b2429a095d6afd0e57950d90c3ca18f18ef13c88aad7625dca607e -size 17871 +oid sha256:ea580334e1b17bf10706f1524a6abe81077f6555e439c207229817d6790edaaf +size 45922 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst index 67577a98ef71..c1ce19a1c173 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e71468ff71a37a72ed9792343f4cc2049150cecf27138bd812b59c7309ac3ef1 -size 17980 +oid sha256:8aef1f9e4d2be7ccf788c8966d664eaa1fc52c369119edbe08588743dd021054 +size 45958 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst index 3e78a24aae6e..bb9ac4de4927 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f24aaf17e51275b346a6de975a4a1e7829695711fc9ee5a8292dc9a166bbc2ce -size 13414 +oid sha256:e21cc69fb82cab5f5411dd6236a43b11c314ab44c67047d77aa8fdbadacc6c74 +size 33221 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst index 0e6800ec4ea4..af450e669356 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:072c26174f3c50a1234a3984f6b0cebccd781b75edcd113229d7369f1d825d95 -size 13528 +oid sha256:d260314f379bc33a7984b87d76fd5fc1c4c50478751e05dcb1fab97f98b49910 +size 33240 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst index aa9673a33b00..a7d37724d632 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5d558a879541036a016d524ebb96bb82adc97e0bc7fd9c5cf454c2613c3372f3 -size 18446 +oid sha256:5e6f86a12f19131edc603fc871c8571528f2b50c027293a425f3b9f5282a215f +size 48099 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst index a3b1a9179ebf..9042dc200c55 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bbe8512b842d73228e98cfb7eaebf53e04bf517376b3e52ffe01670d8189be37 -size 18534 +oid sha256:041aaef5aa30bbc2d91503742baecfd33c7466d8857aa2e501d911b7f8327bd3 +size 48097 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst index 644318da6fe6..afcc7fa32610 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4a7867bd01c8cf1af71f2ad20709a5304b118fafc2e5867d01c8ea910cdd277d -size 13991 +oid sha256:821d6e32a729d709c0ce7b7171ebf616622b2f12a1970200f408011252d3a7fd +size 35645 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst index 7132b8d895cc..032d31be3aef 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d3076b12f09da2ccb4ba50c31da14aab689324c967ba6b48053dbf197b760931 -size 14116 +oid sha256:d0919cb16c2e02ab182c32d74f30d07cffe5c3ade55980f2aece9d7b180beead +size 35669 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst index bda31af38c6d..761e9c7427e1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:83d306398f1024285c3078c838b5cc431fac3335db4d21bd1d9da73025c0d5a6 -size 21844 +oid sha256:de7bba494cce340a4fc6d612660bc46de3477a1a856cf73434c9fee98dc64b4d +size 55563 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst index e1bd407178f1..daedda5e1cb7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a69325f832bb0d70179c30ea063623408e79e3c1b80b05c9966835c217ca1683 -size 21936 +oid sha256:ce0253883dd68d8f59559859830293e76ae32c86a37af740fb21d7a132274973 +size 55722 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst index 9f4ca4fdc947..efb82c1c4f58 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:94a0613d41be14b6b1539d2a9cae4159cbd9db48fd48c13feb0c44d8527c9384 -size 16797 +oid sha256:e36437c143a10c49325073fa5f0b8dab5762fea1563efc11dfd5742641bbcfdc +size 43943 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst index 21c8a4d7720e..539bf0789367 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4a3f12bb1d8b6d9d12eca8e1f1b50fe4f202c2800d07f3f271242ee78d8fdd6b -size 16908 +oid sha256:0377812eb2a7ebafa405da2232d978835c0f35bf99443bd1ee2fac82043233d1 +size 44033 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst index 03bd419d7de0..9041f34bb2aa 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ffded2f87a4a4339abfeb4b81e54e0a2baede939fc327e55e7a4d2ab0ad91105 -size 22494 +oid sha256:4d40939eb06160189b757fc89d33a177d0c8cde322eb19e1f12659dd3043674a +size 58742 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst index cf599598df74..e2fcb9a99a11 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4c8000d9e4ce6f1d90cea519575e1a3b6a48a7d384039651780646a1e846adbc -size 22625 +oid sha256:5a6d84b69a643e74328ac112623bb911fff0d6e06aabda69c7561d18a943ae84 +size 58754 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst index 318d62f04b86..71cf7e733095 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ade208a45717de020aa3fd0cfbc33c4f8d8f001f0f39ca8f15e8f90b3df8a636 -size 17597 +oid sha256:ee9d67a4bfab5b9460d129e1ef7325382094b055150b73e7000d0b1da20015a2 +size 46781 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst index eb50f7e6dfe1..eb7a68c068e2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm103a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6326e541ac243be1ef3a0c59530a56e4bc9d3b814bf45c9e0ef50d6711b05940 -size 17642 +oid sha256:9904b954fb0f3d4b40971412455bed076a3e0515cbc12a9b645637d5d8ad6092 +size 46796 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index bd194c829fc4..317453c88009 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f387cd18e0d1579ab3efea033caa556ec6fbc3269de31452f3c8d9691f9dd142 -size 22932 +oid sha256:41503ae9701d16fa86271618b2c75147d151eda0c5c85426f7bb602ab3fdbdde +size 65762 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 158277cc5f70..679390d485fd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:51f3dbaae1a81e439cde1e25a9a411760199302fb4a89f960aa1a3ec3538dc2a -size 17698 +oid sha256:82e70a5a03ce758c30aaa416bc080ecae5e50c270bf5e38060bd818e37d0ce10 +size 49736 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 3f18eb8caa96..ee9e347e455a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7876b3ff1cc65a4ba7aee940566aa5f5708f40e6366359fc9a6e373ca65122aa -size 14846 +oid sha256:b5754ad584207d68fb2450df5c02a900eab35e16da49a65e901c77073a60f98b +size 39937 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128u2_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128u2_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 5de7a695d01f..165d8eea580a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128u2_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128u2_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:dfd9f9e1edee8e3667ad14dd5dfdc16fcdd017ee5ff3a4b896c3ada85372bdbc -size 24087 +oid sha256:f2b8cccdc87b7d6244d58b9f55eb3b7dda0b347f3e1c9d5402798628e3169c4b +size 68925 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128u2_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128u2_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 9b0906837dbb..ea4cbea7fcdb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128u2_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128u2_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9ee2e5e96eab2f7830f20e93538d9e419155bf938f843a36760e5d8651129ff3 -size 18489 +oid sha256:99efc98e6803c952f8aee9f2def44b3b000b90d5da0f02173d6537dff8232ce5 +size 51826 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128u2_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128u2_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 10341130aeca..192712b504f1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128u2_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x128x128u2_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:aa89b5d33eba549c230c900052ae17096d0694213c5b1c9bbf7e6226fbcc7419 -size 15226 +oid sha256:1df3ce74e579e8a00f639f0bbef9759b2632f293bc7283bcc1a960d728b9a766 +size 42131 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x128_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x128_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index c90c9bb4d79d..e1ca2388ff3b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x128_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x128_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:704e5cc3b964873aecb1616f784a5f3d7bed1cd39af218c14a5e4835c71f0652 -size 16754 +oid sha256:123c660513796308c3fcfd08aeb6cb7197202fbfad4655da9cab46f94438c949 +size 43862 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x128_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x128_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 94719fb7169c..5e5968fc31b9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x128_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x128_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1246588f252e5b98baa08e430b08e387af29dccd30ac8a51de769b5136654999 -size 12101 +oid sha256:7005c2851d7963014d50c57138f18fe837bf1a5aec8c213bcee7c2c408c2945b +size 31937 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x128u2_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x128u2_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 817a88b51f94..5a6fcb263944 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x128u2_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x128u2_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:673fec7224cbb076869b221181fc4b0228c99410223f9202ea4eeed5d1a55341 -size 17988 +oid sha256:dd3a61f7f95b3f0e32fcc77d243d6afb94d084ffbb42b0f8e663049a533f7887 +size 47283 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x128u2_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x128u2_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 18d2102f93cb..420b9e4d300c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x128u2_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x128u2_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0bcc96369440fb42d737c20e1a21d70051b6e3b98fb9248827699809aefe3c5e -size 12811 +oid sha256:ba66ed0ac7ecc929a5fd93cc50f4f881ef2cfc478bbc46bd42b7a446a1501fa4 +size 34292 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index ca8f0b9f9bd0..743e891a9f32 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:41d2ca2d5987b1690ea2420ceb3df481ae39658d45bea01a89d0308913ed7e99 -size 12030 +oid sha256:04ec7ad8671b72cfb042476be9a113d2558bd1615750fa3e2b625cd9225e141f +size 30337 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index ab2952e8bab5..263b5d38c4c2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:43cac02c2cc5dbb72bef53de716f33ad066844624e5fc1b4bbc1687e70b0df4e -size 8270 +oid sha256:d983eb70b9044b556245598686247da47c1122813c605b9d825810ccc7d40895 +size 20475 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 0d3204970848..ee319e642aec 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0839f00e02ecb6e04835bba999a33aeccf5a462f4f35d16573b7bd11aec986f5 -size 12517 +oid sha256:e4afb0445a404e4f8562d9119af6771fc1a2c03e948be2dd16faa80eed74a4df +size 32713 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 2f988e815b49..f02d5955513d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f20536e2a012256b902e1bdcd007c60d8b4fb8dbfbc8b66e4ddec3b248e4a0db -size 8777 +oid sha256:99b344497f7cd0582ce99123c35f48dce598ef24f98086d2d0374d8e85d53439 +size 22996 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x192x128_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x192x128_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 95aeba110ab3..e508e4f13452 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x192x128_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x192x128_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5f33503bc9c1c166259722a2f46d61bac708ca7693bf3e8728b528202e996c53 -size 15552 +oid sha256:b46a67b2dad8a506583179833ee856896fee9402a5b4837aba6c6f4b162de19d +size 45045 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x192x128u2_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x192x128u2_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index ecd13970710c..4bf8169258ce 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x192x128u2_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x192x128u2_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:01ae83e5cfc60162302dc8ecdcdd4b6da4baef136245f4c8f6c1f08d061c5902 -size 16018 +oid sha256:9f64ae745037f52d8cc3e19bbd3a39bb1435556dbe1e30c0585dc1c1f477d1b3 +size 47024 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x256x128_s6_et128x32_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_eW8_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x256x128_s6_et128x32_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_eW8_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 7e3e4cc6617b..8cca9cdc7f92 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x256x128_s6_et128x32_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_eW8_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x256x128_s6_et128x32_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_eW8_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:83ab781395cb5498b34e81ab5bf2949d3904921766e8f6f193a739d8c7982ae1 -size 14868 +oid sha256:d906e232bf910b01dce31a543b19d2e4e900afe27dbf6f4c987a0950e91abc6d +size 40693 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x256x128u2_s6_et128x32_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_eW8_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x256x128u2_s6_et128x32_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_eW8_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index aa51dbef3bd7..f7e5abf4e786 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x256x128u2_s6_et128x32_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_eW8_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x256x128u2_s6_et128x32_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_eW8_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:af87574993d96f5b11165bc580a378720008bbbf1a60dd937a3bdde9adee1a16 -size 15319 +oid sha256:55748f0ab0451bfdc9f5c68f72cdfc2d6ee11a5527511f8a13cc20db7266c94d +size 43007 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x128_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x128_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index c12081a1a524..dd0ccae778a6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x128_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x128_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ce458af974deec49d3397d4ce0db79efb5f474504ad9784b0f47753c089c8a32 -size 17599 +oid sha256:284880cf41ffcd0f93b17ff5489e133011f2c9dc1ee6a8b1296a52947c3395df +size 46785 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x128_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x128_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index aff982b84367..2fe52739b5f2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x128_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x128_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8a34ac7117f839427aa7d447f38a77715c969ed40bfb6db13c639d6f67543b4e -size 12886 +oid sha256:7c60cf771da96ffa94ba2edf39f7791b031f40de4072060f59d68caccd0afc6c +size 34892 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x128u2_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x128u2_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 15f3e47c9e23..6d97214c0c72 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x128u2_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x128u2_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:895cdc7bd88877c8d25f6e6e1efd67f380848bb192efde5126ae38a239cc234e -size 18730 +oid sha256:8b5f0b752dacc0c17488fd0a311183b92fc46a0c52fbc4cca14a8eb531da1de5 +size 49791 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x128u2_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x128u2_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 97105ba6e79c..f6bd84389824 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x128u2_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x128u2_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a9a1121f4ddd0b93d26dd4f6526cff1dea48703423c38b68bbeac7cccd10a215 -size 13596 +oid sha256:8efabbae6d7e1c2780c7eeb078f76b12e3eabe4c640a890d35f3a7d95ea61a92 +size 37349 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index a70b2791fe00..96968153af4e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ac675ff553a0d822ba17a1ce6fee9f558eb23b54068752e51b624d1ea1d65379 -size 12456 +oid sha256:38cb94f27d6704b7b6610e5efdf1260369464fafaa57f1d7f2509508f8ed0464 +size 31144 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 6298c81e7af2..6ac6e2b6a1e8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7361073e524a4951be1b44ae1cd4eacb53086c7fe15a3a9bac9e5a2c5984ba1f -size 8596 +oid sha256:d4304b677bdfd7353184f17fe5dcea216566cd24be689f814752e488797bb2b2 +size 21698 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 8013bdf5bb15..e12a386c803a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:673c97a3ba5c87fb964f5221683a2f62bec7f6f31d9cd0db31fa580f04702be4 -size 12941 +oid sha256:4e5a66f64370cc91ba2c7854e7a8721130d1c3e633c33746615a7714a3edda79 +size 33599 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 056a311c2228..5c38c52aa9c8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:298887074eb8f25794785496b5718871e6ac47cc60aed0b9ed286f2a678cf42f -size 9135 +oid sha256:eb191fe3b932bca060ca66789a9e8cef702daf1e8e1228317a9871d13c7d41a4 +size 24094 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 84cbbea421b4..995f836ad6de 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c1fcc2419df3ef861fd2043e98e948a139f63188485a98bcfaa27c0c4f0a1c67 -size 19624 +oid sha256:d01dc0b71e66069340d7f26222bb4ebb79400a911d0a7db691edc1a99df2c10b +size 53952 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index fbcf67f666c3..48a73ade59c2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7484dae85b074bc0ce0934ebfac00d1b31b3e99b57079cae7cd5472e339fab34 -size 14533 +oid sha256:7562324fbbc588023fda02d25feebea738d6df18d659e08f6ab110603c8863ee +size 40063 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128_s8_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128_s8_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 434f342585db..a6ccaac8a100 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128_s8_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128_s8_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e04ff014f4f593da3a608d1cf17a01fd24726ca732b4d3a6b69bd0e39388c7e9 -size 14383 +oid sha256:6b758c3ac0ac190c3f99dcd99e485b068e95e29e752f65ea614a5600633eec1e +size 36328 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128u2_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128u2_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 89a8b773657b..bb6a2b4b8401 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128u2_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128u2_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f2904d8d80b1d9bff6967bb8f15b0611dcb7f5b4a3208c74e5c73cd36d2aed3e -size 21016 +oid sha256:40b114954a8613d9f6bc513d82a975cc178f1dd0be301a3fde26340bbcef70a8 +size 57196 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128u2_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128u2_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 7fa5242e62cb..68b96783637e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128u2_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128u2_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e0aa24621504e9f684aac8448482b0ab4bee55381143fd4cc2d9ca820ba57ef4 -size 15248 +oid sha256:ecc8f1782df676d8465b1652787d7eb1e0efd8eb521f1648f1982326929eef15 +size 42514 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128u2_s8_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128u2_s8_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index a208cd5e2cb4..daf76c2e2c1f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128u2_s8_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x64x128u2_s8_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:acb337917d29943093d3769ffa066f65201277d730af3f1cc078e335e76fb0d3 -size 14739 +oid sha256:02d656ffcf358474c9b57de55e08cdfc922f54e160ac243c4acfeb4b376589c3 +size 38332 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst index 3cef02b94951..b68bc8a5590e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c23f08a9c57f54de7ee76c2e7cd8304b5443002da3f28dfba7a2037974223dfa -size 11098 +oid sha256:712ec6fa6bd6a461f330e1242ea05e77669cae3ff62e5ea116b9b5b1de43d5dc +size 29787 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 884810f97cb0..f50fc87678eb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4661c3142c8064cd251ab6afaa2fdab0d93fd7cb0eeb6834659a7ab60c95f185 -size 16380 +oid sha256:67af5dccd88d4232e8079bd047f290986eafed8bd06e8c4ecac3bc95b4954037 +size 42534 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 89ed7c66c384..4d43412abf77 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bd61cb9bba9d03d0de8a44d86c0fc2a3dd23fa48b2bc495453fe387a7c2bea9e -size 11741 +oid sha256:7ea33d3c6b911c2b38f35baf91649db11c7b19c65ab49eb1dc9a3003316502c0 +size 30376 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128u2_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128u2_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst index 24ce3c8a96b0..c5b1e66223f6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128u2_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128u2_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d612feabeca34ed75281faaa0446440af0b09894219c9b336f31210d01c5137c -size 11777 +oid sha256:d6c3e37b34cf7f31f1e38148ce8e98c2b7654f7f1bc6b64e1dcfb4560171b515 +size 32422 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128u2_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128u2_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 2b72393fcac6..6a63d2345854 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128u2_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128u2_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8e523d1eee50a77378afb84d0114a8b4f5ee6d2463ad9f7de32ba534f124bd9c -size 17720 +oid sha256:ec1d73d946e24d08d749c278d3cda9621eea1115acfa52386c431180d296fece +size 45876 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128u2_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128u2_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 120b3435553c..8c1a6c638661 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128u2_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x128u2_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:07fa72904e2795db086e5d529783bd7184dbd987237e45f37ab041caa12047ec -size 12505 +oid sha256:aed3f26c8cac91642cc817ae818af041c65e99a431a282df19a422d60ea17550 +size 32523 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 4c2769c61999..80800c0b7fbc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:74ef53d00720742ab263d3f2bfc17a7bdc9106784cfbb2cecea0065ac1bd9d66 -size 11909 +oid sha256:bf84be129fede8d764fb08fa42e91a16fdcbf01e12e37d3015622f6578414494 +size 29679 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 5ce5be1f52da..772ea3f4144d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:97dc9eef9eab5c8a8a69149b942302f99bba5d45a7e4b041272d2e57e2f5f825 -size 8098 +oid sha256:8271da3932118774bfacba7c8554eaf092134f5fb33549615aa80969b138ce6d +size 19949 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index f0ba13ee0609..d2773ec8358f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:37f0be4890b1c13c3a0a7a39c419c9e5659fbf057ea10e08fa35825cce3023d4 -size 12445 +oid sha256:f4a948d603ecdc3fd9fcecfa02d46733a2544c1ba922181e38654dbb33da4067 +size 32248 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 8df683f07099..51607e214c17 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:24a2a8f3eec678f043837786902bdf36d6e6a5069569d9fdef6f206dc8788acc -size 8718 +oid sha256:fe25a2ec024371c671964e2d71dd5124b84679e0d76317027dc3be63bd482c30 +size 22521 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst index fed39d6b022b..0322772b3f8f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d2d2cdc2a5c1e565066d289791531893ded6817ce49a30acacdcd377da8bde16 -size 8272 +oid sha256:5b42cf1c8386008a37fefa89c33b5a0d2adeeac7e25f26f5c7f9e222a382114c +size 21446 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f.cubin.tar.zst index 39e1978adc7c..f31293f5204c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5f5dd38762003607125da75a46382eb5c73b1828d09779e4e60ac1d8d5fb519e -size 8423 +oid sha256:997496af8f942d0b6909591a02e379154e2ca39613e13099306bbb9100b1559f +size 21899 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst index 0fe15831bd08..e89fbdaac9ed 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ac8e2c3c587cfe4e512828f62383cb08287d3881bade6451ed5c0bc0e1cdad1e -size 9019 +oid sha256:d14d3ca8df629bb0514f5ed8ecaa9723acd79a689cf17a52dad36736b683afea +size 24993 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f.cubin.tar.zst index 9885a2111f13..e7e849cbcc4b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:667b1a97580dd3dbae8bb2f7e6378c6b1a473268714981c3a35803a53c7dd379 -size 9207 +oid sha256:026771cc0d451929d2b70b1ba4c13ee8903baa3078ff5614c9b38ca6c2c7cd2f +size 25547 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x128_s5_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x128_s5_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index ccebcb2adc15..2790f24ca810 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x128_s5_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x128_s5_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c6aeace35601050fa6059e4c16e2809fd784065ac1fef8ed1aec5715e1cdd244 -size 21373 +oid sha256:38a109e30500737d30af7e4fd757788e9852fbc8d829a8d7998ab111fdd4a495 +size 53973 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x128_s5_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x128_s5_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 57e6e1274eda..9e45a5b5a4bc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x128_s5_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x128_s5_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:96aeae95dc5354a095b3a6dcc41fd70ec82ff7c3f30407c2f21b162678d3b645 -size 17114 +oid sha256:8ea25f2405c3f8f783d4a87d7fe3f6b0898d241c8d255eb3743228515e7f0d63 +size 42431 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x128u2_s5_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x128u2_s5_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 99dea95b1bf6..a21d683d1dcb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x128u2_s5_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x128u2_s5_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9564eb7e43d4d2cdd4c61a32e1a0cce49d01f970f58c8c79b00c36d1a19ed44c -size 21979 +oid sha256:9d9bb5c832d5cc516256d95b2c4684101e6ab6e9c58d8be296104c0281d551e1 +size 56322 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x128u2_s5_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x128u2_s5_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 9d90c0ae0ac3..a7b3cb5123dd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x128u2_s5_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x128u2_s5_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a8fe47f0bc41ac7514dd78c6d47768d2ff327e4be61081837db9ad815dca2afd -size 17722 +oid sha256:7d0dd9abf026e17e9ce2b1e556984d2072ad96bbd870284449fc9fcc6a29816f +size 44674 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index abc778d89483..537cefe38c94 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c9491819364bf17d8124e0e1e419f72d54c9099a71538a55132d2465669c214b -size 26039 +oid sha256:f5b1a5c5ef3a5f4a70857f5ac9f7e22fe161ae8809b6a02b60297e85e8c74ed1 +size 64709 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 92dfa4293dfc..06b35c0f03d3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:49afd8730d84f7e88944db62db80b10657777ff7486c27de0c217cc87336e704 -size 23925 +oid sha256:8dc52e7951ff15f4c01688b38d68328271c7f6ebedd4ba94fd13d2949973b973 +size 60962 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 65e285bd98a7..f1e705f63b17 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:13e20d676a86451738790bb4963655a98918756c7289f2703d2c131c61ccedbf -size 21407 +oid sha256:916433ce57d132e0f7ef653785ada8e3c7500757ce0ac07e01afaf11d5644466 +size 52245 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 2148ae51089f..8c5503f337cd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e813f43ba42c45143f67dca5fde030da59a0e8e032b7c0a0121f5baaea9fdd67 -size 19164 +oid sha256:c3858b3342133dd75188bf6da1be186b0c6c7bdc41b1aae9d0e67b94d245ec50 +size 49370 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256u2_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256u2_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index c9a1e3b0bbba..e804a9fd8180 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256u2_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256u2_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:56d3cb987a065bae4554bcd839bd0355244c1c4a8f682957ee2c81e66b8bb969 -size 26765 +oid sha256:f40543e28a859f5584c79bf4325f90ddd9cc353e33b1567c956eb182cd6ae5fe +size 67490 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256u2_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256u2_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 4a81b2ab2724..47172fef1d1c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256u2_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256u2_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:aec3e9de77824123d87880d837f3bfeebb8b6d23ce29a036965a4d3cdbf0b097 -size 24668 +oid sha256:ca9aa1db805b2b0462e05469117e635859c9653ecc7492716f2db9d0c7e4fc57 +size 64347 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256u2_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256u2_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 1feddfe1cb8f..97dfda31fadc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256u2_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256u2_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:afd3ef69ebcc5756a25650c63b861242f0536e98813185eb29d61947748c023e -size 22337 +oid sha256:bad086eaa06bedf208d09b0e1a88aebcfe4a66f4239123de7b79a06ed126d4a9 +size 55630 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256u2_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256u2_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 5910ebabf529..9febeab5cbf3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256u2_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x16x256u2_s3_et128x16_m128x16x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:33081cb271b91f5d6baafca71723a11b2b84f71f3dd6fd3f640cbb5a00542601 -size 19940 +oid sha256:6ee598bf3dfcee7e19cad90ee6fd801c89e927b0d54f4ea96646c818853f2963 +size 52618 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x128_s5_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x128_s5_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index b8968691fdf3..fd86f2f1cb4a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x128_s5_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x128_s5_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c0cd49dced7fa038ff60952d2bcdaa965599a60dce0db250538cd77c5b3bcb79 -size 22848 +oid sha256:5cc3f14754eca4eef3a1c0aa14fff0d1427921019279dc1078d7785c9ca58ecb +size 57475 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x128_s5_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x128_s5_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 8fba60a8f65b..c520484fbef9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x128_s5_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x128_s5_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4e369fd82a4e26a242baf7bf269a5f97d719c50fe40d04880a7133365ca1e656 -size 18210 +oid sha256:87a22518852ae0cabb9cb17d2380cf78fcd7ebf120402dc39b20dd56640d82c5 +size 46171 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x128u2_s5_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x128u2_s5_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 48f91416d22b..5ba49acb113a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x128u2_s5_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x128u2_s5_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1334d259af40e13380439fbae4d08ca26e688e578293a02c1761d2a4f3700f7e -size 23388 +oid sha256:841c875191fb197f4ee86b0a852f4468da53ef71a48b3223fcafd7ce608d2298 +size 59624 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x128u2_s5_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x128u2_s5_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 959fd6fd5f55..41438e03ea8a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x128u2_s5_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x128u2_s5_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d0123adcfa104b53138c1f626a73cc60de20321c514d294e4ab0480dc0407b76 -size 18912 +oid sha256:79318e908298355b00b30e935f8a4ae4007ce51bd36d8f530e2eb5db1e6fe896 +size 48056 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 190af822201c..01b90851c83d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d95a9f216ef8f58d95f76882735a33302569ef3ac4714a0fa769082895788360 -size 27210 +oid sha256:7d13cd0b4cd9c66adee966b945199ea368313ac733409602b31ebd1b82084114 +size 68086 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 6001b8c1fe69..059ac2f56527 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b7435dfd662fa07efdd1f01484825cd4867237207a2d61ccd6ab2df866a9ff3f -size 24058 +oid sha256:a73162e76371df11043bdc101ae8aab1a6830fea0bc2b24a8c08375a9fed6cb8 +size 61511 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index bd705861e280..52743489684c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cd4eb432a0af2b2228ad8ef48563607c21e1a57e781763d99ad299f768816d1b -size 22335 +oid sha256:95f1de5cfad240025228e97b2b4f353faec31bbf6b0786c746051a1d24bec233 +size 56258 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index c56b5c1e1c43..d9c2ce7716cf 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ed03965a50592235276b4716444443d8146603b7cdcc62ed5e1e516219ece6af -size 19390 +oid sha256:2e092e73e83172890fa5b270ad265d1e838af3955c861c99e0d101e17fe01740 +size 49847 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256u2_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256u2_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 6256d820d977..26198fa33ba5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256u2_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256u2_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:403a811e812e2c97379c245e5d6800b77ab37be250814f9dbbc55342fb751cb3 -size 28132 +oid sha256:d3671ae4dbf1802f129f4a50b07cb67eb4de2076b1864612b678c4e5c1b9297d +size 71158 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256u2_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256u2_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 5fb088b68e67..18194555438f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256u2_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256u2_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7e1a7681c6589cebbcd20eeb4f842dc2be75b11c36edb1f05dd3722a19a7f83e -size 24776 +oid sha256:24701f8eceb93385456be0ad263d05976aface893ca78c2fb836661d6f9d608e +size 64498 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256u2_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256u2_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index cebadc7fca42..1c215f0923b7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256u2_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256u2_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:eb9270f676f150b57536f523090050a64003405d2ffb1e52674d4cd2365a3c4b -size 23331 +oid sha256:b35b79a7b03e5056a7989c59fccef519a1f361a1bc466fc009483cb705fd3e89 +size 59204 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256u2_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256u2_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 88eec35602cc..cec4e375b55b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256u2_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x32x256u2_s3_et128x32_m128x32x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f47db639a4b4f293f51befe34151b06e0bd3baeb6513fe5b0adf31e91fbda78d -size 20205 +oid sha256:536365af3a34fdb1f6dcac50e7d94710a7b3cd2d4ea82d7d10989cbafde8d48e +size 53002 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x128_s5_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x128_s5_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 3788d9c2b12f..9d443aa0b1f6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x128_s5_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x128_s5_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9f31b78768d95bf890e1fcfadbc7e9737e0bf7aad5e408ef6670e77ef679a361 -size 25124 +oid sha256:366baeff1e3e7eb6225bd080b378798de2abdd0fc899a35a0438fa119972cd46 +size 64932 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x128_s5_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x128_s5_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 5675eb61b002..4014b424564c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x128_s5_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x128_s5_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:09e5d9be9fe10c133219bd58c235645fb7244b683d1258439c2bf763c569515c -size 20268 +oid sha256:1d8f01fe33604fcb615bfa6ff894043f371b2da375a16462be2bf3afdedbb30c +size 52615 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x128u2_s5_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x128u2_s5_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index c24005aeff7f..2596a875d246 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x128u2_s5_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x128u2_s5_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:162c29dfbad4cf05817df63a354bd2c481d2de104a6a320d3ba08aa7d965413a -size 25584 +oid sha256:bb0987acf9b3953899e7b80999de1f961f017caa0c8f6e5c1c55db3e234be3fe +size 67176 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x128u2_s5_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x128u2_s5_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 1a371128d272..02411272b8b4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x128u2_s5_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x128u2_s5_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b022ce0b110428bfeb0d20d60b57792ddb0797074fd2e9218e2cf80d8819ac35 -size 20933 +oid sha256:739c2f700d47fe541405dbdb710354fb2ef05530cc2af43b76fc97d5522d0294 +size 54813 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index ac090785efcf..93911c38be69 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:73e575465d5f67ee088314911858e546fa29129ebce65fdb75480c08ec47f212 -size 29942 +oid sha256:757c5778f5c49f887ef1b7237af873afe50665c742ca1fb18b4aae9380859e4e +size 76291 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 76c1390c3ec2..531a63226e7f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4ee0a330a0f99d14ee8daa08ebdc4de0ad09ef56140355371f539fe877436662 -size 24428 +oid sha256:088ebe0ea42cae11d53fcec46fdd463c908460e18ccb861e917a18b09d337b7f +size 62467 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index ade3554a2a22..a55b74a6186b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0fecbe40f910fb1fe7d1503c0059f52f4bf21383ab73d1f8f9af49610493a506 -size 24380 +oid sha256:1fd8e476f82e53a9232446704c4504abbf21e0612640d8312bb32cf6e43e6a18 +size 62765 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index c5bd851e8e6d..393171853f34 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4f5df2b003e6acff1c0c4d5fa6a4d280c0d2cdc70768f51ca0dbeb6ad75f98a5 -size 19777 +oid sha256:eb41fd7b63244b0f627f18e1dc7aa4a74560e7fd75544d95ba6031929dae4d0b +size 51037 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256u2_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256u2_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 345341fd3caa..84e6a5f29626 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256u2_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256u2_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9d41577d1c94e9b45c4cae2aa2401dbd78771ca9cadb3bb122b89a7abb274190 -size 30779 +oid sha256:7c10b77c674519b1dd630db49165b62b14cadf80598809e222df8fa1b5b409e4 +size 79317 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256u2_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256u2_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index b808b0601147..b97810dfced5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256u2_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256u2_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0ca102324ccd789eff520ba35419e2a6c66d796fda0a8aa1404bf0c531e148d8 -size 25201 +oid sha256:8bb7e4c6bfd8cac525d31a4a70d61d745a5f719985bebfd6061c71ae4728ba4b +size 65691 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256u2_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256u2_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 418a33d46cfe..2ac51ad562f6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256u2_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256u2_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:46a421302a01a5af93a278db3e34492db46db68938ecd72acc304e9075258710 -size 25397 +oid sha256:974c3e05907b4912912efa7ad2f4d95650c8bd82b58f5f88b6002ca2a15bf95e +size 65658 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256u2_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256u2_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 23c4f424ec8e..1f1cb2a7272c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256u2_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x64x256u2_s3_et128x64_m128x64x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:36376c4df143a6b39a41c7e67de455d9ad7fba8647b73fa174a3b4069accb7da -size 20573 +oid sha256:a8c2cf222a19b77fc16fc7d7414a6808b5ee6cd4618d996f8f6bb764a8f1b622 +size 54271 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x128_s5_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x128_s5_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 33726dd61238..5f29144c199a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x128_s5_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x128_s5_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:903eca2bfaa3b4f0751c1eb8fe8bfe55779a4b92607627d179f4f8b7e8b6a831 -size 20749 +oid sha256:fa0bdd1a9d8d280e8359d4842d0d43e233fcf0ede601d636edd4e580d4801f8a +size 51672 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x128_s5_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x128_s5_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 62c4e3e98d10..f9b2cb0e69f0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x128_s5_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x128_s5_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d2054b8f7499a2aa24391d453f053049227e01cd6abd4c265e0dbd3fdab48f8d -size 16455 +oid sha256:a50395a433fe4066e76565dca8c504edaed6b51d1ed66f473d5818a105ebbff6 +size 40410 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x128u2_s5_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x128u2_s5_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 0f86ce5486b8..b5cab297b1ff 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x128u2_s5_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x128u2_s5_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4eede95c89daa255e873f725492cc09c646750c0c8f287ba83b14a412d7c0ddc -size 21328 +oid sha256:edfa6cde89f4aeb0fc14ee8378dabc7b3f2be4b8e84f2ce070081af9b9acd937 +size 53893 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x128u2_s5_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x128u2_s5_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index d272f87bbb7e..b05560a20ebd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x128u2_s5_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x128u2_s5_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:32679403549b921725bb450b3b5145293c162034a490a7cd34a17f8c07bf4dd4 -size 17013 +oid sha256:3c479c49208a0980d1cc462482d09e405f65106149cbea21feefce9e99bb2a55 +size 42692 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index f0d1833b892c..a40ee3a1bda4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e3f4f942dd5a81c7546886c6015af4c29719f0a65d69e5cdbd99fa3f55be90aa -size 25396 +oid sha256:5b382905f1baa207fbfcff500b383dfd6ac3cf811ea900848fb59ada123149d5 +size 62689 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index e910b646c446..2d88618d67e7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8f295a2c7d75c51013b1794d5c48d9f9d5d4173dcac68cb9156793eb1b32bdf5 -size 23796 +oid sha256:fe059870526410c984cf08cfba4bb846c0e56b7478d11f387054c19c606f9e90 +size 60733 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 00cf4770b8ce..020182010547 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e7037dda69f10821ad8ed4404042f3b035a0f875ce4051bd4773ffdedd1e2130 -size 20742 +oid sha256:b722eb732a728574bf21eb09900c36dada893eb68195bc1be88c5d48bc2c132e +size 50170 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 69311d0cb037..b0417b1f7b80 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1d2efee788410566cbe018a0e8dfa85682ed91295dfe7e0d932c4e70c25c064e -size 19030 +oid sha256:38eb34b7b268793cf74502378fbfa0ecc227975fe9761362fe2650ca8d976f6c +size 49024 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256u2_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256u2_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 00e6f57afc84..b8c187d8da1d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256u2_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256u2_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:129b9b113c60e223581b5ffec39d561f2a5760a801dcb07c69f134f58df68055 -size 26139 +oid sha256:567a414db47360bbc223930aaf38c76d75084b62af1fa7ac6e206b6a89375ec2 +size 65843 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256u2_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256u2_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 80ad1cdaaa55..01e39a4cd682 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256u2_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256u2_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4d83dc35d250dd2e17f4e6ee950acdb1530da844c450c6d7646426527cda71a7 -size 24555 +oid sha256:d99970efb80872837c531752d64c0feb8f598407f0089a057a780e8696fc40ee +size 63866 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256u2_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256u2_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 9eaef7a116b2..eca0735a1f6b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256u2_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256u2_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:67adc7d1767ba99abf5d3a7fac000b7ae016c2988c73efb4ee1bae256d3d4f87 -size 21662 +oid sha256:623c25d8aa719905fb6da10fcb14ca2350dd8680c5b070db75aed4a9db4d2e65 +size 53586 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256u2_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256u2_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 701038b83ea9..bb3568656839 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256u2_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1Bfloat16_castBfloat16_Fp32_Ab32_t128x8x256u2_s3_et128x8_m128x8x16_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a5b7405c641b3d5a2fc17874dbc16e4ecbcbb4bc18debdbfc6481de1b8a974c8 -size 19838 +oid sha256:35430cad4f2c6b27609342c4399da8538f90003cb45b410513b8642d98ca8489 +size 52079 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index c58afaf1f6b2..be94db5a7a04 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:edd2364ff1fdf9debc71e51ed10d661287d6ae6ba4fc9dddbee49b4667f8e984 -size 15158 +oid sha256:395060a72aec3cfa1989d722635b7313b6765c3abebf13e56a4577e815ac7a87 +size 40831 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index c6442e76abdb..febe187c4a9b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:64b8e2c2a6716ac5b4aad9f0460ebcb46ee830f0d61131f1c11108cce9b48475 -size 10818 +oid sha256:9e668c626b7e015f2265480377d2d01f318bcd750ced64641c612158d4de0369 +size 28476 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 169155b592f3..bb734f2bbbda 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e1fd3a7acbe75dd049a63483f03fe55afe5f6538e57b85f73ef7fe2ee63c4c12 -size 16156 +oid sha256:39b6bde83584102c7a03df11ab4c96149157b2746e1e0669e2bf62521f355ab2 +size 44700 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index cf3952f3003b..a3988706e522 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d319df628958fa8d614ba0c5f833eb77c38d058f4a0257ca1532b7cb728d718d -size 11747 +oid sha256:7853d2c2b19c5e151dc9b717e9c3d04fb06934b1831480a27f8b6f3d65e2434a +size 31655 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index ea7b9aaaee37..c1b9d2c92e2b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b79e7f965f6da7508fb808d2a270e47bd7bcf2dca5c1d7f948646d2c66ae5139 -size 15946 +oid sha256:891b5b10157e6bfc162e1f48277fc7d0eeca195f4083f692c5692bc018f45fb3 +size 42241 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index afe8c07faf41..6c7e8befc186 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:33aaf3dbef98d9e4910c9220cd0d10a71a7154d1ef04f05b0ee57aae284740bf -size 11273 +oid sha256:408ac96f306065260cf7eab845578063952277d83d2bd91764c02ee6ea2dbc50 +size 29728 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 4a8e737ed312..865e11a381b7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9b52cac8e869db20bd5340a087c6114ed6165603ee852927f98405ee91cb6879 -size 16885 +oid sha256:661a1f27a7a7900abdd7d6d013a8f06306c478f80fa6e0947e1ff58987aa5ff9 +size 46339 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 533ed8aead54..848b53a8fd26 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:10dd4f297cef67fb9af7b140e50f4fae502694b70e375090bfd55d726977b020 -size 12118 +oid sha256:bd8f289fea162fb617f6db9ac6813498167d55f2db146c33d27dc6c6d7821a5c +size 32916 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index a3d1bcf5893d..9ec475d0387c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:aff2318d3af78280e7797be969551c2c1ce0cb7441d417c22d57c4a7a6b9eb0f -size 16352 +oid sha256:3b37cfbfdc4f822a3eeac49d473a7377627e506e838b81c94f74e5a3a6201ce5 +size 43997 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index c6687d33d257..16de5298030e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ff8cdcf0cd3232de073976f730a0e58492379d3616f63af66a8897152bfc4ff4 -size 11893 +oid sha256:59de805ac0cb7a83d61d6251251a494b91fddfe6eb5ce90af5df11d036199de1 +size 31697 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index b706a0bb4c18..ba5e5cf64bac 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9e0331b146e965994fc4ba6f992aba0dbd71cf9d926c001ae182b9d2e4917ec7 -size 17367 +oid sha256:b5fd494674dda44cda6d87db830c7278153df51a91f060972723b9f063530d6a +size 47790 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 9b692db1209b..d9ddf65c2d87 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:29997efe56ed22883861400339104712fb3c45ee24dbd98a0550949014d21c01 -size 12803 +oid sha256:c69cd7c13ec62f41695c4e7d5726b0ec46d05e8dd11d2df3cb9c7749abc4dde2 +size 35150 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index bf0c532ee34b..5bb123c19498 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:305a6a34a313e7bf046350032ad68b3ec29966b83af88a7345766ed1a37d6454 -size 15093 +oid sha256:241b044a5c6e211597898c7092d83ea34e4ec58640ff264531aa612197f8b211 +size 40301 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index c97e7ec2bda0..a1e364661f54 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5a9c18cc18028c9f3d1d6f2b9531485005d57439ebbd4fbab68a980cb1358f2e -size 10657 +oid sha256:fc42034e4cdf3d54353ebcfd76958cb76bb1836a01083b6e0d011161c4fdbcf0 +size 27790 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index a4c03c495db7..da65a735d6e9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:59eb202ffa61ce2f9ec21b6c22f3939f5b33397ce5ddb3ba875a01682f7390e4 -size 16051 +oid sha256:fa4b6877cc529a1f12844ab90d5dbd248d8581dffbdde71aacffe94593acf307 +size 44238 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 4ae0e670fad9..edaac9829dfc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:48a0cd3cd807f99b0a3e1efdbddb3c92dfe595534762d4f93b9850ef8f33d364 -size 11519 +oid sha256:976d8b120b24b4dda63be9c84bc2138e55385445ddc80dc8a9b0384d669fabbf +size 31147 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index e2473dec594d..d5a2904672b3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:56da4f5f9f8d685996e7904e408442555a7febf718cfeb6f7a099fb3f2e68a7c -size 15971 +oid sha256:a5f912eed04128f2bc48d3ac8e8767ad15adf8b92ef129bbb2f25df36d274be1 +size 42761 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 186e3e56c6b6..70b572e91a97 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5fc9d05759ffcc8f72c146bdd01204b39305e057444b067da1230f7354de9642 -size 11239 +oid sha256:390b5bb219a3b588075fcacc43480e0310f50b311f8d1a522c8741431e5ff185 +size 30234 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index b5d4f41b72dc..9b5a48a3b00f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:50474ce916ab796f57de98f7fc5f2592ba14ede89de9d8a6e4fef1bfcc25eb30 -size 16872 +oid sha256:95a6cd5bf8d1be460a5217713a45e0a0505a70f712b78d53d004763c439f8841 +size 47523 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index e6b2f56384a6..14a707a1e02b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4d512d8cee6e87aaa50197ba2bd3d30a2635d934ef1809cb51b1820abf25dfd2 -size 12301 +oid sha256:1c19744d2c611fee02e37d3c73b5a547a5d64818de8cdbab36162be7894b0daa +size 34647 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x128x128_s7_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x128x128_s7_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 184585236fe8..9b3dc42693be 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x128x128_s7_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x128x128_s7_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a66932311dad7582ae7f98cca7867e21e4fb3e9f82f2facfc43063e432222972 -size 20091 +oid sha256:fea813cad69b55c95fc19839311ce92140b3bd69cb85b0ae95f5f6cead927c70 +size 50894 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x128x128u2_s7_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x128x128u2_s7_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 5c99fb20f7c6..bf36fd52466d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x128x128u2_s7_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x128x128u2_s7_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b4f582180f175034220bd26d3b48e196615c163a3ab962e48627f169d51109c2 -size 20803 +oid sha256:b37fe33f49b8cfc743e4e7a198f4605012be7498ba543208dc43d39a12d7c14c +size 53834 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x128x256_s4_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x128x256_s4_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 459837de1120..680d224c636e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x128x256_s4_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x128x256_s4_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:afee30baaaf7fe37c9bbc5247587840c91372a0054ba0e75a5f94681ee11f8df -size 19883 +oid sha256:d61bfd817a9abe6976970a7f092095aa54e85f35b3e0cfe6303fd45f9b686d51 +size 52187 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x128x256u2_s4_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x128x256u2_s4_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index aecf4fb9f5f1..c627988c38ec 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x128x256u2_s4_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x128x256u2_s4_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d0710f93045fd20aaf3c885a0d8f3b599ea8fdf6966a3d97e91763855ab6b7ad -size 21210 +oid sha256:79ffea016500d152e885d8e2f9b993a61f9f561efdeaee48a22e20a99d704bfc +size 55428 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 6181853d43d5..a20ef845c532 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a160edb029f1a65023fdd6fd6eb83af8c390ea4740f2d9286cebee49117aed7a -size 16904 +oid sha256:ba8dd7c8e3344b3390defcf699d5d92f46011cebbd720fc5dc86417d794bb045 +size 44835 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 650aa76f0dc2..a6a0e1eca075 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:14dd256f09765c9cd9071b93a21aa9b02618468c1b1666c03ad6024017bda186 -size 12463 +oid sha256:3d8bba8a49e349ba6b359678a43c8ed08cf925c30c44bce12cca2b9bcc291333 +size 31087 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 27d7cb04718e..e7beaa710445 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7cbd1de2a9781fa6474ba646ff78d5ce5005e677113caf33951e34969011d7d5 -size 19072 +oid sha256:baf08ec7ae6eedeb496899548406cc24cbc53b926de918f03f028e99a8bbc905 +size 50478 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 3bfc33feff37..b8b7a4338009 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:01927ceb7b620040ef493d89a53b76ee9b9c117ed8e7365d308fe4a92b219321 -size 14754 +oid sha256:c0ca2dcda8bea968f0cccf95ef49bc408e7bc408510ef7dd29fff65b21a0dcce +size 36769 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index a2fa7ac91b17..fc56b6f3459f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4de3e0ba43044ff5844356631ad51cb7ee2c4bc44df2f868b380fc14ea290c7d -size 18005 +oid sha256:aed32a6a90a2300d7f32ab532ec5429381c7e35209c51a5a7b42c9a073bee9f8 +size 48403 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 7b1484356543..daf71e9c746d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0d7927d8b2a6f4b59caac47193bd40bea5e456ca9c4fe1ae63cb22f9486e7caf -size 13543 +oid sha256:355a352eddc3d403b5e4c003b0470859a687d18edc98aab8c0d5ddba42e35a62 +size 34829 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index daadb17f7b02..4ee30e5860d4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bd330edc5b942bdede960f079a605da1f4b73bb420e7d4f2f541f9cd70947189 -size 21042 +oid sha256:3c5d096ca7f5a1fe36525abb817aa5c6cee8a2184c74e6a4eb2703731ae48682 +size 53957 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 3b4e49e854a6..fcbfe9b9e33a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7e67d60e5ca2479122cf24cc5642a4ee3a7c57ccb70f0c0c40916c9008f1d8ac -size 15895 +oid sha256:2af92cbb6f875ac2a4363beb0dbd91f9dcad0f6d54808052436abc3080b600d6 +size 40649 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x256x128_s5x5x5x5x1x1_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x256x128_s5x5x5x5x1x1_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 1bcd943ee4fc..513969743676 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x256x128_s5x5x5x5x1x1_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x256x128_s5x5x5x5x1x1_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b0a3be73de08d4253ab7793c0061c88a6b466bede9fa82d35d077ac5df15d48c -size 18949 +oid sha256:2abe18d435f44fb0e3c0fe6a3b519a6193cc79d3662813629546175ef31cf24e +size 51314 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 48625987e74d..35ebd5007149 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6c55109b70c9f853257c4caf9393a7255793676e6849f944ad7e1d6387f38ec0 -size 17475 +oid sha256:0515435c041b7c63600963c470f7cefc3ba4e303b44045ad3cb5577c91354ca0 +size 45410 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index fdf1b2f360c5..5ddc8dae09a4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b8cb6fea60fd52b8107dbe27df128e655a0aad328aebcd189a78567c2c573997 -size 12769 +oid sha256:0a18c69a26ce13b825c8a30e27b92641819b93e7e05d8a1321576a1e0f037dda +size 31758 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 25933e5ffa65..694ae1be166a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2e6425a636621214cf90e4eb22d7b101c194a8c45553261067222db66ce7d01b -size 19102 +oid sha256:87e3fdc154d2d4acb002aa388d605ab5e3ef43605a7d629eb8608a8a4bf86d11 +size 51384 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index f6b47cffb358..32513140f78a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1eca65ce396df5a0bb797c5bfe1f5e2932db9e10684d229e2703e8fdada05ae7 -size 14899 +oid sha256:58f81ce21d8c0718675128e79a93248afda4b05353487430e5ab9717db53d8a9 +size 37277 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 0f9af21efa61..383be14debf6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5f6d2d44b834ea040693ce0de3e45c265c9889a7e2fe3608a699455d86554c0e -size 18540 +oid sha256:e2e330f6f8d858f31aef7889552740d30251454ace735dd4ea882cb7d1e2035d +size 49162 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 6704281c675c..f16c1837c469 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8529797f5f7a91ad26969f39d8225a50778e104e3076a2c6499867c361d91921 -size 13828 +oid sha256:7002dea65df860f0f8240fe1a0b19c23886c499b0f8e9e21432311f2eabaaf87 +size 35473 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index e3d82e224b52..51e83d3c7fde 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5334ffa21a0bbc7728a88d6ae5db9e235a17dc3754a8ddfac32c892d4fe7ee8c -size 21300 +oid sha256:d7d3fc2d9146cae73fda67628892d145bce36e61d3a3f73774cf549975016ea2 +size 54718 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 3edda0aa560c..8b453bb7ea28 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2ce23d423648b01c5a0965c0c5f4884c13241234378cea49e96632c598e51575 -size 15995 +oid sha256:ff0bc5b015ae62faf3ebe39a34e4e34b3e03cd6792af34bc6d9300434eb6623c +size 41252 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x64x128_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x64x128_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 305085e43d85..688ee13428da 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x64x128_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x64x128_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8e74453cbe75e8f69567c1c55d9e18a0a0a4f71534bb82fb7af1a61a85f0463b -size 19798 +oid sha256:5b7a836a90d49127e7c4a3d97c2080670e3fe5c562b9acf0c3df6acc6b16ef14 +size 50576 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x64x128u2_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x64x128u2_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index f344f4b376e4..b55a7ebcde91 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x64x128u2_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x64x128u2_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8176ae7522cb4cba774edd0e4d4abe2f1cedf935002b2b1ca90a4ae7ec8ae5f5 -size 20780 +oid sha256:a6b373d497f37a216d17fe337672505e0c3a9365df751945420039a939a66fd3 +size 54171 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x64x256_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x64x256_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index f167e750f8db..a57e517a80b8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x64x256_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x64x256_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d049cddf53e929cf6b66b292289ebd8006bba6a00f421b6e63640c9ddedb747e -size 19500 +oid sha256:c1b4ac607f89fc86cfdc72777a9d4b51bb5c95cd8119fd013837449fdfb2bb7d +size 51809 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x64x256u2_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x64x256u2_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 027d8e0442fc..b3d4e28fc0b2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x64x256u2_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x64x256u2_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:49103ab863797f972b7b1d8df575c3945ced1819c8ba2bbe3cccfc6a5dbf6ab1 -size 20931 +oid sha256:cc9345f138f9c875a3d88fece86bf2db7a489148d755980324becf993557ecad +size 55976 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 61d411448b57..41fd5d3308d7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e826f300c48ceae8c5194b739a5d65d418ca5615e9c5555e934aafb2631590bf -size 17060 +oid sha256:3e0a7b785855018274f08fc37edc955e49a1eedff60121071f0978a2495691e6 +size 44530 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index cbead6066138..6b9135332ada 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:31074dd75174c5ded717ad944f4b4cdc276149f36fca38799c1c442c5fd90094 -size 12359 +oid sha256:f1e5c8535970f88a20971fb9d88e264d12eca80847d5cbd776f66cdef780cc66 +size 30658 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index a7b49a0b9218..0a96ee6a23bb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5cf80bcaae191215174411b655aa7d6c142f8a10ff08c83d5c8ce3ae9f4d1985 -size 22293 +oid sha256:e8329ec289b75188985b338ba0d585d5f68df54006fc79ca39d54bcf7970a447 +size 55134 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 5067ec6f82db..49841f8969bc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1426ef398eb3ac17448d8ea7d845dd22c75b833f35f07ecf251fa29e2fb9dd96 -size 15192 +oid sha256:de0a15f244e496247ae728ae0d9d9dcffa390c671400c938f8a98b47735c23fb +size 38477 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 65906a931e1e..bc4a211066b0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e024744bf3903a103895f762ebf571e9280c740a4426a64502d546e4b6558fd0 -size 18147 +oid sha256:b443e172491c31964f44af0dd482793d246bf0f5b411d3b8db547ccc752b884e +size 48153 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index b88d0dedef90..ecd78198198a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ba8a560f402db6125b45989ef03f71bb4c8ea55be59db33e67e6ea9a6c505b3c -size 13433 +oid sha256:17d3658e600b8dda123debd3040032777af9adb372a8ae05018b41aba6fd10ce +size 34400 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index ca30cf39ea36..a788af63762f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5a5e852507a8bf8d3c9ac8f1cf08b9b0686141b38fa3703a9e5001178bdab905 -size 23557 +oid sha256:6dcf05081369138c6dcccb226e9353682c5013107801286b312a1ed559f40d5c +size 58882 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 757d3ce2cf4c..3748542c0309 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4e68c486bb895d5dcae669a715b3950825137963028e8fb4aeb5af316d3d44e3 -size 16235 +oid sha256:cf5ea24ed9492b7c575d94d89af5cace1084a0efeea8bba3f9f0782c53e4f72d +size 42432 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index add9bd875e8a..a4186237ab2d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5d1981016f8885afc741a1bbee9d690170f82dfc2641b127310605cbc7f28d79 -size 17484 +oid sha256:f3c975dc4edb6c7228a7806ce57dbcb4347f63636dcacfa09e5583169274e592 +size 46723 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index db578b0b206f..6271692106f8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:567cb9ba59672bb58fbc62395bb12f9d7a3fb09de73b6212d5f9c41ec8bdc047 -size 12534 +oid sha256:8fa10f690ba45b963fee95289bfbfa3c0ddd491566d6b6876dc6f55967ec267d +size 32763 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index fb93db92dcf4..6e8363969d28 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f9cfe87333b1d763fee222d3701e59d2fe6ea980f240e27ed2e19420cf6ecd1a -size 22334 +oid sha256:42b67c6c4176c67c1d7fe5648726cb5ea48fc19bf1416c6e1023ccfdf3fb376c +size 57201 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index f1dbff0fff79..0429028c74e2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7ec13cde422695644bcc227767e2b57206b34f62cf7d6e70ad93e77da27f09a8 -size 15266 +oid sha256:882f2f74ecb5594938808e52b9732a5c2cae11189211b065b5c198ac4e7d9aa8 +size 40489 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 38d010495014..747e0ddc9178 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8f5e34fec7d4cad5c599abe44be27a90dcea211dbd8cdad5f44e9f88a29b4978 -size 18519 +oid sha256:63b5c148a0dd4a0c844054ffe3b7cf79e7433c5aa4d33888fca66488599c4122 +size 51002 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index cd1384abe785..71812e7bb1e7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3c98429d93731bfd11973741cdbc778d6b8e17dfe5c23a1c0009b5f345091159 -size 13932 +oid sha256:96fd011c5f3dd099bb57fa5f20ccc84517ee6670a17e4de1aa8adee30ef3dc1a +size 37919 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index ba5c93e24401..6509c0a93bb7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:810aa5f0bab92aacf406423904402b9836d83b086aea3592d751adc5e6fb3cc0 -size 23568 +oid sha256:5b2286efcd357caf75c6a4bda34578b0e398dcfc04faa10b9d07e5e0d8709bdc +size 61624 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst index b726500c1632..2ef45ed95b34 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Bfloat16_MxE2m1MxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:02eca8f4ef9ba39badcb46ed47c59866c2a95ce56e44a4b96ae64da3124912f3 -size 16766 +oid sha256:35778eae9f6f0483a3c5b1118fa07a874b9add349573cfa086bf25b2a5de4e5c +size 45676 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst index fcc6ffd3bfaa..281e21910005 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5ba882b8770f522b99e71a6d7b2a8c33e668219c487eff884e3996e697c5a03c -size 41559 +oid sha256:9c1e84348c6f9ee2a28963f6d8594ce2b39a10db8c56e05a6a4ab0ac16b12bdc +size 111471 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index b138e6fac99a..bc0276e2744f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:833010cd9f33856a38f738759a23f8c99b37e36b4b1f2783ed551e7ac7e34ace -size 40351 +oid sha256:96f5315fa91d6c08d2b3d6ad090b05152ee2862943c8da02e221dcf61585fbf4 +size 108826 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index db7de2834044..7d394200cb6e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9aa39797c8c0aaa8fca290ee692346624a9362968417ca4830d0b401cdc90d68 -size 37664 +oid sha256:ce2dce278fec28b7886afe7700d7ff1120d2ea07b31ecd4e577fb70ab570d653 +size 95501 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 5b112e1d5412..562111c4bc69 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e4ea9884ae09aa0acfecd0778dd5c060ae08c024a914afcfc352936b3e110002 -size 40348 +oid sha256:8207d2390ecf8c8e7ea4ab88ccf11ad8acc092391905610b95dd2991836529e4 +size 103640 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_geGlu_lsfaW2_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_geGlu_lsfaW2_dynB_sm100f.cubin.tar.zst index ca804cfb3f95..7d0f6b861331 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_geGlu_lsfaW2_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_geGlu_lsfaW2_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c0d9d973d1d553fa3f695b94d7dfadcf13db30f49e2bba33e7c2f8373e8dafa3 -size 36653 +oid sha256:1dd906900f0ad6adad467374259e3eca29b36731357a031aca9c07c10c4a059a +size 96426 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_geGlu_lsfaW4_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_geGlu_lsfaW4_dynB_sm100f.cubin.tar.zst index 068fb42c400b..60272485712d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_geGlu_lsfaW4_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_geGlu_lsfaW4_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:38db4b12df284ccce0d0fdb4c3de91332a96997f3f8e635350785aa0790c7cee -size 34290 +oid sha256:dba02473557c925285ab967a706e41eea95908fd4a9375d1f04e35a07ae3d49c +size 92399 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_swiGlu_lsfaW2_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_swiGlu_lsfaW2_dynB_sm100f.cubin.tar.zst index c055dfc23064..6d15adee28df 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_swiGlu_lsfaW2_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_swiGlu_lsfaW2_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:424de2954d0cf6c609ed2f2fb1a6f7af03c826d362ec0819ad9d3c05393fba79 -size 34894 +oid sha256:363c654c45396f43919c8a24b19e131ebfce0b8016468a67a912dd17d4ffc051 +size 93543 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_swiGlu_lsfaW4_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_swiGlu_lsfaW4_dynB_sm100f.cubin.tar.zst index 1651e8305ab1..40f5d9f5006c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_swiGlu_lsfaW4_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_swiGlu_lsfaW4_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3fc29c50252194b4552f0315ef4d30f612db0887ebf21cd4c8b5e692b4f4eec6 -size 32215 +oid sha256:83afcc51e94d8c0c8ce6a9ea27d75fe91fadc627733b8ee8ba73fa75e3345467 +size 89160 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_relu2_bM_tma_ldgstsSf_rgTma_clmp_lsfaW2_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_relu2_bM_tma_ldgstsSf_rgTma_clmp_lsfaW2_dynB_sm100f.cubin.tar.zst index a39db79f2027..cf4ca0143d84 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_relu2_bM_tma_ldgstsSf_rgTma_clmp_lsfaW2_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_relu2_bM_tma_ldgstsSf_rgTma_clmp_lsfaW2_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3a5f594fc34cc48ec26a5b901bbc906b84382dde2b46b57ddc488a13e3d12657 -size 32708 +oid sha256:b56d1c54ec2a4848f11e759b6607bd78a1e583b68b68f3770e2faf763a143ab6 +size 83333 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_relu2_bM_tma_ldgstsSf_rgTma_clmp_lsfaW4_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_relu2_bM_tma_ldgstsSf_rgTma_clmp_lsfaW4_dynB_sm100f.cubin.tar.zst index 1dc7fd5b4a5f..1a79b89bdbff 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_relu2_bM_tma_ldgstsSf_rgTma_clmp_lsfaW4_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_relu2_bM_tma_ldgstsSf_rgTma_clmp_lsfaW4_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f110b513a95115a8b66db7fb353ea066b21f9f3918377859addd84ab57182d6e -size 30004 +oid sha256:9251c860109d2fd0b49c3fcb3e5f41c8e1f9107b7ed3385355ca19a66d33c198 +size 78118 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_silu_bM_tma_ldgstsSf_rgTma_clmp_lsfaW2_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_silu_bM_tma_ldgstsSf_rgTma_clmp_lsfaW2_dynB_sm100f.cubin.tar.zst index fe459aa2ae6f..95820b5c54c2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_silu_bM_tma_ldgstsSf_rgTma_clmp_lsfaW2_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_silu_bM_tma_ldgstsSf_rgTma_clmp_lsfaW2_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7f52eea7c81383bdef62ef0124544c24d6c8446036d31ff923ae8eb5f9b0d5e5 -size 34305 +oid sha256:7a8c27e3ec3068de70d6eec6eb02e6c8b90e517c2112a89fa066b54527dd880f +size 89182 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_silu_bM_tma_ldgstsSf_rgTma_clmp_lsfaW4_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_silu_bM_tma_ldgstsSf_rgTma_clmp_lsfaW4_dynB_sm100f.cubin.tar.zst index f79ce42a0b0e..6509ad84c676 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_silu_bM_tma_ldgstsSf_rgTma_clmp_lsfaW4_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256_s6_et128x64_m256x128x64_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_silu_bM_tma_ldgstsSf_rgTma_clmp_lsfaW4_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9bf32b4cd58681ebd69fcb807756ba4caf12f35ba01f2cbf8a907a0ecf89e513 -size 32287 +oid sha256:badb6c892aca1df4e71626be60a6679a08ae5eb4755edd214dfd1b9ba27ebd62 +size 84358 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256u2_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256u2_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst index b9ef37fcc767..557ab95c9a16 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256u2_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256u2_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0cc7bd45289b5445f56beb75d55e6178b07d092aa5a2903b780ff846f3bbcde1 -size 42529 +oid sha256:8468b994b07758e2909a9796c8dc158a5466e3420ccc566ba67afe51b15c1b87 +size 114323 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256u2_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256u2_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index d2b5cc7ea403..04dd9f8b95ac 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256u2_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256u2_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:84b985b24936388ede0c92e9663645d24b60b8d19fa5072cc64793559b877905 -size 41339 +oid sha256:f24b7fa19d0cae51518a606768817cff8642a0003224b0e6713c2398e58fac64 +size 111652 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256u2_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256u2_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 947cd6d8599f..abc580313648 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256u2_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256u2_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:60559cfbf0aeeeb4870cf66d93e350ac3ce09ab7c8eb90646a950c267542738f -size 38532 +oid sha256:d5000187b05b14fef8d5058338d80c15b58d8cffc1baf43e5d662a0d576a0f1d +size 98625 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256u2_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256u2_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index abc9de683795..6b4a1a7f146b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256u2_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x256u2_s6_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:82880622f08487bdd39f8e88e2b8b0ce7cd9c46f55a21888bec5a71ff63cb109 -size 41096 +oid sha256:a9242e418cd93970b4ef101cacc7e57151d411eccc7e85a6b8a92afadcfbe6de +size 106325 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x512_s3x3x3x3x1x3_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_geGlu_lbW4_lsfbW4_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x512_s3x3x3x3x1x3_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_geGlu_lbW4_lsfbW4_dynB_sm100f.cubin.tar.zst index 76be2e9338d9..7b81eb96d4e2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x512_s3x3x3x3x1x3_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_geGlu_lbW4_lsfbW4_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x512_s3x3x3x3x1x3_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_geGlu_lbW4_lsfbW4_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9734cc2a20565b8e2af805849877b203bab691e9aba131a815a3293841c98006 -size 32487 +oid sha256:05aed38c5b67ce666317dcbbc5bb3bb88af3fab0611c373e3c8169c86f2f204a +size 93333 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x512_s3x3x3x3x1x3_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_lsfbW4_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x512_s3x3x3x3x1x3_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_lsfbW4_dynB_sm100f.cubin.tar.zst index dce187b05e63..efd1220efef3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x512_s3x3x3x3x1x3_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_lsfbW4_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x512_s3x3x3x3x1x3_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_lsfbW4_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9d97db46430ffa454af8a07a57abccd014168804d8d05e42a69f644b0b68b3b1 -size 30936 +oid sha256:73ab339a2a3d70ec43aedc97a138395347949f08cf2b5f92ca666b00a1ebefc8 +size 89984 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x512_s3x3x3x3x1x3_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_lbW4_lsfbW4_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x512_s3x3x3x3x1x3_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_lbW4_lsfbW4_dynB_sm100f.cubin.tar.zst index 225a4ee357a3..320858fb22ad 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x512_s3x3x3x3x1x3_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_lbW4_lsfbW4_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x512_s3x3x3x3x1x3_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_lbW4_lsfbW4_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1b8d781da1d508613f30ca4cf6bd4ebda893f2b0deb12c51b636153ea14cb031 -size 27634 +oid sha256:cc00cb22c2802dd064b5d9b1e710beb45b57273222ba06e569e67bf95097e896 +size 78040 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x512_s3x3x3x3x1x3_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_lbW4_lsfbW4_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x512_s3x3x3x3x1x3_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_lbW4_lsfbW4_dynB_sm100f.cubin.tar.zst index b934994b85bf..98c5fa444b70 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x512_s3x3x3x3x1x3_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_lbW4_lsfbW4_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x128x512_s3x3x3x3x1x3_et128x32_m256x128x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_lbW4_lsfbW4_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:809ed958c4214a01667b3a688a29c0db59353e6e18b7d97f9a185df45865e252 -size 30388 +oid sha256:af54f4a4394270e8c1a3c7d868d39f2c8e2b40b5474c8cac87f36e284f63018d +size 85191 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst index 79bdaffb5c05..65c54e144719 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:730c537cf6538b7aac1c56e4a8adac2fc8a241fb8b07728acc734c4af77c0cf1 -size 21911 +oid sha256:2eed0e58c0bc5c28f72ae08134d0e5426c9ef5fba53fbc64580dd5d30487e62c +size 54359 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 56ef1a9414b3..60d418a8d71a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:870205013d11ec999276eae553f3032358bbaa9accecc02bdf622069a34085a6 -size 21731 +oid sha256:702d7797ae7034bef6ea9a226acf9aa61cdff8cef2aa5f816c42e15439ffac70 +size 53994 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index a56e1dd5f08d..5157a5e4f351 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:844546f1d29817784fa57015c3bdfcef0566dc3b26c9d6f264b6a3d14031002f -size 20853 +oid sha256:26e661acdccebabd6568386b8aa4de42fc92a2fee512f09a98b1f477ce776378 +size 51187 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index cab43dae8331..7bc9dbc436f7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3092d46b17f77c16eec757ab52e7c5a0392ead6688bcac8875ebc2687711ab18 -size 21242 +oid sha256:77303cae94f2e41a57d6f4a95a9c36f7207d74b60ff37d815c1865bda5c1a09c +size 52215 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst index b290b8f82e99..f73b2c47a9b7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d5126a08bc4d91b1909914576134743ca6a86111bed8cd7126104e662c56373f -size 17093 +oid sha256:91fea010f3770f97399b4bad9a35a3710b0a9bdc807cd4ddd688c88dc961c9d2 +size 41289 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 1dab8a9a7f8a..8c1a9948a41b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3caee6f87a48fc11a769edebce29ef965745c2018325cfdbee5e7a9dd01b5528 -size 16942 +oid sha256:ad9d51412476ac4bd1aad382efb6259cd29467849f24d00949580a0b1d1d980b +size 40880 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 9d48bc0e15d0..37483f8c17d3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c1080a04783d2d8941f7ecb18d87ea314b257e67279700f192af8252ce5a67fc -size 16030 +oid sha256:043837aae51e6b8c942727e214a70547279287c9a1ef5f9f69b826494334947d +size 38197 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index b6616ec5c57a..b6a31f24d890 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d691219179ea44e8869fb61fa0ece22e25a8aaf3e6cc08d63aaec44917862bff -size 16488 +oid sha256:b7f8e5e80ad6eb6641bcc09df0f7c50d7d23765da0e54af916d07b81da604dcb +size 39334 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst index a00cf5c910ca..5bac464d6859 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c7146289e921199d8a8056257a8812d4f52152405cb9f8143ed0121eba0a1b6e -size 22833 +oid sha256:8556a0539ab50c1fa34273ed8922045acb8020fcb804a6b866fda6b030ca86ff +size 57621 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 52ffe27bfbc4..67159547ff61 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8b751780fbdaa11594711aaad5dc2e1a98c4f204ef1dcc2bc16e6b2d43e9bef2 -size 22643 +oid sha256:4a26c28d7f107563ddd65e8ee2bac688a2b9c627c75ac4de17838e00fd683afa +size 57106 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index ecfc057e9709..d96323c31285 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:62f32cf2dcf0173b75d34148a8e3e4665ce05064148371d6e4b999d797dd555d -size 21827 +oid sha256:cb9f2cd7d7f2b15b9affd06e3b4d2adc20c2fafac2899b7e31e9cd94d0c8f320 +size 54431 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 85ef44caf1e9..9c0c7a066493 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4867b2cf040899043ef8f0185fd1f30eeac9ec5fee62b265518f797c4e550ccc -size 22177 +oid sha256:9a69b5ad35655d85d325c54f0c96aa57360b26b6fa55825bfe07d6dc39e840d7 +size 55427 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst index 262927278a0f..f9952e3de879 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b6193527ed09bdfcbc243777644bb65e26204e93dd7d66990c5b7bc005f9be5d -size 18109 +oid sha256:b62a539470bf77f848cfd228203e94e3e67000e05261a50b189d2810c095eb20 +size 44462 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 156289af421f..422f6a74aed8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d387876be1c2a5e663835d1bf761aa76d88003fbbcdc250a1c12ae9f06152903 -size 17889 +oid sha256:77164295046342c608d114faf50384a168cddcd0bf672c6f07d4f7fe30799710 +size 44134 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 4ae82fb6fc44..5284959541b8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8bc1dfb7cf40056d0de74355b4883cac5d09d37e45c4a3e4f3a08a355536585b -size 16974 +oid sha256:e920e69c5fc2ebb62f23b91d6cb1b5c1da0c9c630916ed8bdae5c6da83cdf10a +size 41416 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 54956749a781..005cde433e5d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x256u2_s9_et128x16_m128x16x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cef055a62298d3535d4d21a457f081155f0916906cfc8d1e4a6541a49686a44f -size 17469 +oid sha256:9cea69caacbdc1c7f0477eb7f60fbb1084630e882689ef76c0583aabc2b29fa8 +size 42567 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst index c8cbd7627bbd..7850a4e97e1c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4ed8bdd9302d8546b519d4428ab73a8a1afb1948e956facb60f72d631e46ae2f -size 25605 +oid sha256:1642d9cd2333862bddd517fdbe714b4b12787a4fcee8309d80c4f6261d0b67d7 +size 64037 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 85afa4472102..dad14fc006d9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ff3757f7a8213390878f049a2fbc708dd277a86b10ecfca48f30163ae2b89499 -size 24294 +oid sha256:02b3ffc7c758c0c1c6bbf7aa32242dc44546746e42db319bdb0c6227bd16ef07 +size 62129 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 569ecde4497d..55a286992b14 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:eeb2d780b8334569d06014e8225fd7a49316e8784fcf304a8c8420d7675d2127 -size 24770 +oid sha256:03376e7ef1a15c613c1e5ebb9f58756d2e238016c543ac688984d0b6b845a844 +size 63070 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst index 7557cfd7b408..149aa9ca90cf 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:72011ba14b0ec69eae377269ed023e605f693cc9e2a4413f9d73e1d4ed8a884d -size 25439 +oid sha256:2db44b07e577b21330ca47d085d8c88975a713a91004f60452d971946810c5d9 +size 63700 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 0480fc51f6d4..f376c7062429 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b4a50269ac4f2288810a3b4c058448a6c08e007910559db890759ad46ca70851 -size 24073 +oid sha256:c661109d34f70c7dbb67e697eaf3adc10c1249d7f01af08fbc7f9a6e1a6d8e98 +size 62068 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index b34a3c03b411..5727efa16ac3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s4_et128x16_m128x16x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c99b85a989e4e7644aed50181a50d0a1cfd22865546f697d933c61837421021a -size 24646 +oid sha256:98cdf94604f0f907a9635ccb79f695951bf84230a15a48b19531c25d0303475b +size 63102 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst index 73c89d964a95..d839ad70e583 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4194db3c7e797050e2694dc2670d8d13aadf714352efd281e2f63f267b08d886 -size 22709 +oid sha256:f1cea2e25e1d102d8660326575a7baed0c5edd6fcc5594d9d95d693b7d9711ba +size 59120 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index c42d5aa282c4..c6db01bd05b7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8d5f7c9089e0233b837439a7a74730c769f576593f948ff6c0edd5ddea6d4ef5 -size 22432 +oid sha256:25485d7748e927143246535bb17c4368fcb0bcad504332db95d7065b5602f5e2 +size 58972 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index ddb92239f146..d68e68798c70 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a15b13b31ceb3ecf5b33265bab740c15a1dcfa55ff6c0c1af992b44b7b5090d1 -size 21538 +oid sha256:6e1a76faaafcc4d24de8d9eeb54e6e1efcf7c7a5c809ed054ba9f2555dada366 +size 55917 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index fcf7caf08d21..d83964932d13 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:375f673ff8f93534115516eb0dd39fcd090f0f49ad44fc1833ec12e31c4e65e3 -size 22030 +oid sha256:5e94459b1cde6fefa8bd6fa6ee127f97c574f849217fe97b3f35d884a12e010b +size 57249 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst index 6a91ab12502c..6716b06b7963 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9ac3e959243e8a29e954e9bf08ee0d2791ae18f4de32c1588a1f269f1a617990 -size 17524 +oid sha256:ca2aab24a25765d86eb4babf6ba3bab03541460ce66b057c214001373267367e +size 45343 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 6f7357a690e6..98aae6cf44b4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f1c7dc897cd2761c36648e55d86644baf29ba528edc2cb68ad9f4e56250fc15d -size 17333 +oid sha256:a5c2f6b1d9a60df2dd4c59c8571c1cd0a31bcf7153a6aee05e0c30d1f939c5c0 +size 45021 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index bb98c04669e0..dc341313f40b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a1a0e072d5168786c2b7cc9059bd70a96e34b88bc7bec2947f38a26cc7e4a562 -size 16348 +oid sha256:cd24769260f7b9072e5f6967844cd8312a83ac1cde026cc0c2fc1faad18e5545 +size 42167 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index cfae4ed60df7..3296b1016b46 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2f01a3c6aa3209562e965f0a3848d59665e72bcb60621c7aaf48dfb2f1c1080e -size 16784 +oid sha256:7d1f04bb10de18e37329e8718855c1fa26414d351b2a26050ccde460db4a6ff2 +size 43243 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst index d09bb8fe6667..25ac071c868a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e3dfe4d3ca619487f7342d64063440ec3938123c08c29bf95f25a32210c8065f -size 23596 +oid sha256:fafb14c3df10022ef0d04c15fc49a324dffeaf9e3ec48d44d3ab7a8f4f1f7926 +size 62932 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 62d1ab15abf0..84de040ce45e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cb45790df43b353c09bedb983daa787120d9bff75b52e2be211c91fe3f7478bb -size 23367 +oid sha256:47ebc145318da8749a4118acabfda5b50955b1419de7cbcd5b449ca701a5f4de +size 62637 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 68a9976cfbc3..5a22328ad072 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:42b85e4617e06acf2e2ae2206131cf0f3c02d7545d5eddd7e522e206cd4df519 -size 22767 +oid sha256:148b33d97e8d2e058f6fdb451266bc3eadd52d45621a4a003a962359706b2e11 +size 59715 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 536de266374f..e98d3f40fee3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:11ad2528172600ef5da8cb05cd4f507c6b8d3d1dd8e2f20e737ca2346abf698d -size 23299 +oid sha256:fad489206b96b0293ee21faef5932884419d33848edd1dc8eb547174616310c0 +size 60868 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst index ac63470d7e4f..8ca61904127b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6f9f780da6b223ad49a9bff5d61fa571d1c6df7a934008685ce5f65c62773a51 -size 18631 +oid sha256:bdbe48eb3b2bacacbb24e6b32263fd1fd369261c15192e31b2909d453dfff15e +size 48947 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index e9cf6b5c5152..b80a7a2e04d7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2f95dc88aec5ee444f006003a1a43bc0522dd5481094c85d29ded2ff0da8d92f -size 18425 +oid sha256:f9cdb0dd6171cc01ba47a44d44838ff8501e8850672e082742313ea1fa23939b +size 48691 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index a028f7cd290a..84ebfa6e1c7a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:26184fdb52b442f7f7bc50ae9c66be311f8aa98bb9341a2c1f4a4597ee60d436 -size 17434 +oid sha256:0e40da2bd95845b3e40cb43b460c7f09fe2f9860a27a2c11aecc9d857f1fdbde +size 46374 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index aaea20140d57..a0f7e4696492 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x16x512u2_s5_et128x16_m256x16x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9a916ea7b27b003ce7396cb8b0b5d2850cb69ee862613ba1b59311ca9a3e2f61 -size 17854 +oid sha256:c957af9366fbd2adbb808606875220a19cf63be13c430831f59c6a8251c983c9 +size 47366 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_eW8_fCp_tmOv_bN_tma_ldgSf_rgTma_clmp_geGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_eW8_fCp_tmOv_bN_tma_ldgSf_rgTma_clmp_geGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst index 9e2aa086a85e..0e672879de31 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_eW8_fCp_tmOv_bN_tma_ldgSf_rgTma_clmp_geGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_eW8_fCp_tmOv_bN_tma_ldgSf_rgTma_clmp_geGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d4dfa786da2850248648be054bb12e1d4f79b85b548a8b42f1fde096d4a8f2ba -size 34154 +oid sha256:da65e5e00e4995b05c986a6fedea51dbc3e10c723049900badfb961a024d87aa +size 93665 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_eW8_fCp_tmOv_bN_tma_ldgSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_eW8_fCp_tmOv_bN_tma_ldgSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst index cdbf00fc3421..233fdc9fd5e1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_eW8_fCp_tmOv_bN_tma_ldgSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_eW8_fCp_tmOv_bN_tma_ldgSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e3132deeb192beb97d280de1bda390eda6cddc92c2a3aad0e57c5e7ede885d61 -size 32693 +oid sha256:46365f3d66e10281fe3f9d69515e861cb676477f01731ebb742960de2f26d653 +size 91137 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_ldgsts_ldgstsSf_rgTma_clmp_geGlu_lbW4_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_ldgsts_ldgstsSf_rgTma_clmp_geGlu_lbW4_dynB_sm100f.cubin.tar.zst index d0a42e6552cf..22f39770b433 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_ldgsts_ldgstsSf_rgTma_clmp_geGlu_lbW4_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_ldgsts_ldgstsSf_rgTma_clmp_geGlu_lbW4_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5263dbc09cf83a57e6fd48e46d6f2e931d4957a6e767903877756c0a429e725f -size 42984 +oid sha256:990ba7af323ca426f1e1b1ba0133d4d95ffc23b35702b2abacc9490eea50cbee +size 134471 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_dynB_sm100f.cubin.tar.zst index b9fb55a34ab8..523842e88031 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a1decb01dca6d48bf02415ef32a0be32b6fd9fd4fe8dae7aa52160ff3e315b5c -size 40156 +oid sha256:53db89ff6e8ba3e36fd8c4fe735fea3c597001e05b9f554b927ec67cd93cf095 +size 128778 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_relu2_eW8_fCp_tmOv_bN_tma_ldgSf_rgTma_clmp_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_relu2_eW8_fCp_tmOv_bN_tma_ldgSf_rgTma_clmp_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst index 2f15ef8bb516..2c8051e23cb6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_relu2_eW8_fCp_tmOv_bN_tma_ldgSf_rgTma_clmp_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_relu2_eW8_fCp_tmOv_bN_tma_ldgSf_rgTma_clmp_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ff75170992c9d0bf156cd6f50213eb5653d35e2121933e7d2929744d61540519 -size 29816 +oid sha256:05b0cb7316e62afad3d31815516828a931ac1b32343cb10b3ba77216ec187c27 +size 80216 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_relu2_fCp_tmOv_bN_ldgsts_ldgstsSf_rgTma_clmp_lbW4_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_relu2_fCp_tmOv_bN_ldgsts_ldgstsSf_rgTma_clmp_lbW4_dynB_sm100f.cubin.tar.zst index 1bf81b912fd4..5fd7de763dae 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_relu2_fCp_tmOv_bN_ldgsts_ldgstsSf_rgTma_clmp_lbW4_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_relu2_fCp_tmOv_bN_ldgsts_ldgstsSf_rgTma_clmp_lbW4_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a4c6505a2af94574f8a23593e8793214d77c758e0831004fc4c323a1bcc5bf4c -size 36016 +oid sha256:8011a3e911204efc061707ba6597e36c03093ca98b7eadccf41dd1e38993e170 +size 106027 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_silu_eW8_fCp_tmOv_bN_tma_ldgSf_rgTma_clmp_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_silu_eW8_fCp_tmOv_bN_tma_ldgSf_rgTma_clmp_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst index e51f068d7378..c7efcd70a24d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_silu_eW8_fCp_tmOv_bN_tma_ldgSf_rgTma_clmp_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_silu_eW8_fCp_tmOv_bN_tma_ldgSf_rgTma_clmp_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b846605124d3badbc6772b67d06ca84bd0665394ef96f69bd4b0ed5b3035a303 -size 32726 +oid sha256:2c81e77c303d9cffac6c766a0a451b459a455eb3665cd4256b651e3bb208aabe +size 87659 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_silu_fCp_tmOv_bN_ldgsts_ldgstsSf_rgTma_clmp_lbW4_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_silu_fCp_tmOv_bN_ldgsts_ldgstsSf_rgTma_clmp_lbW4_dynB_sm100f.cubin.tar.zst index b7091e1f226d..66b9c53c2d8b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_silu_fCp_tmOv_bN_ldgsts_ldgstsSf_rgTma_clmp_lbW4_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s4x4x4x4x1x4_et128x64_m256x256x64_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_silu_fCp_tmOv_bN_ldgsts_ldgstsSf_rgTma_clmp_lbW4_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e3e9ba696e50066bc5be152fdbb6e4fe4049d55ee108063452dd0a9a75959e64 -size 43050 +oid sha256:dde9f15e747be92be953bee3df30340c138d754949124f7964e14045e885c3d0 +size 117659 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s5x5x5x5x5x1_et128x64_m256x256x64_c2x1x1_rN_TN_schPdx3_biasFp32N_fCp_tmOv_bM_tma_ldgSf_rgTma_clmp_geGlu_laW8_lsfaW4_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s5x5x5x5x5x1_et128x64_m256x256x64_c2x1x1_rN_TN_schPdx3_biasFp32N_fCp_tmOv_bM_tma_ldgSf_rgTma_clmp_geGlu_laW8_lsfaW4_dynB_sm100f.cubin.tar.zst index b2aeb8ab0efc..16d863d307a7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s5x5x5x5x5x1_et128x64_m256x256x64_c2x1x1_rN_TN_schPdx3_biasFp32N_fCp_tmOv_bM_tma_ldgSf_rgTma_clmp_geGlu_laW8_lsfaW4_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s5x5x5x5x5x1_et128x64_m256x256x64_c2x1x1_rN_TN_schPdx3_biasFp32N_fCp_tmOv_bM_tma_ldgSf_rgTma_clmp_geGlu_laW8_lsfaW4_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3d5fd7aa10d1dfa78631a95fba89ee9a412cb34cb2df7694728ee0fe97f2a8ae -size 38296 +oid sha256:2a6b2f083361835ea52c2c7280fe28008539a8dfcc9dfa3957930f7104b74d6f +size 110712 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s5x5x5x5x5x1_et128x64_m256x256x64_c2x1x1_rN_TN_schPdx3_biasFp32N_fCp_tmOv_bM_tma_ldgSf_rgTma_clmp_swiGlu_laW8_lsfaW4_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s5x5x5x5x5x1_et128x64_m256x256x64_c2x1x1_rN_TN_schPdx3_biasFp32N_fCp_tmOv_bM_tma_ldgSf_rgTma_clmp_swiGlu_laW8_lsfaW4_dynB_sm100f.cubin.tar.zst index 785e09d1e7a3..151ad3dd102e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s5x5x5x5x5x1_et128x64_m256x256x64_c2x1x1_rN_TN_schPdx3_biasFp32N_fCp_tmOv_bM_tma_ldgSf_rgTma_clmp_swiGlu_laW8_lsfaW4_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s5x5x5x5x5x1_et128x64_m256x256x64_c2x1x1_rN_TN_schPdx3_biasFp32N_fCp_tmOv_bM_tma_ldgSf_rgTma_clmp_swiGlu_laW8_lsfaW4_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c71470dea86de8eb6343abc6408131c0e59f96a9efb7ddc412f5dd18dbc55a2c -size 35356 +oid sha256:0e7459b834708ccaf0a93c2547ac3501ed0698173e6f6cfc76bdfc6ea0f3b980 +size 105304 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s5x5x5x5x5x1_et128x64_m256x256x64_c2x1x1_rN_TN_schPdx3_biasFp32N_relu2_fCp_tmOv_bM_tma_ldgSf_rgTma_clmp_laW8_lsfaW4_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s5x5x5x5x5x1_et128x64_m256x256x64_c2x1x1_rN_TN_schPdx3_biasFp32N_relu2_fCp_tmOv_bM_tma_ldgSf_rgTma_clmp_laW8_lsfaW4_dynB_sm100f.cubin.tar.zst index 9508793bd95a..f7584778c1ca 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s5x5x5x5x5x1_et128x64_m256x256x64_c2x1x1_rN_TN_schPdx3_biasFp32N_relu2_fCp_tmOv_bM_tma_ldgSf_rgTma_clmp_laW8_lsfaW4_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s5x5x5x5x5x1_et128x64_m256x256x64_c2x1x1_rN_TN_schPdx3_biasFp32N_relu2_fCp_tmOv_bM_tma_ldgSf_rgTma_clmp_laW8_lsfaW4_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:49eae1ac0fa2733e36f174c9155f86c85e9e7ef2717c58b3e09cda067c0fb50e -size 33461 +oid sha256:8a1a524b52ac4fcc878acabda15a0555ea4a022e0cebee6cb0e866a927084b6a +size 89793 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s5x5x5x5x5x1_et128x64_m256x256x64_c2x1x1_rN_TN_schPdx3_biasFp32N_silu_fCp_tmOv_bM_tma_ldgSf_rgTma_clmp_laW8_lsfaW4_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s5x5x5x5x5x1_et128x64_m256x256x64_c2x1x1_rN_TN_schPdx3_biasFp32N_silu_fCp_tmOv_bM_tma_ldgSf_rgTma_clmp_laW8_lsfaW4_dynB_sm100f.cubin.tar.zst index 50d8844f29c0..87a3c838a4a0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s5x5x5x5x5x1_et128x64_m256x256x64_c2x1x1_rN_TN_schPdx3_biasFp32N_silu_fCp_tmOv_bM_tma_ldgSf_rgTma_clmp_laW8_lsfaW4_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x256x256_s5x5x5x5x5x1_et128x64_m256x256x64_c2x1x1_rN_TN_schPdx3_biasFp32N_silu_fCp_tmOv_bM_tma_ldgSf_rgTma_clmp_laW8_lsfaW4_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f3243036702feac238c402396ae0008e277b17d393058baec342321bfff0aaec -size 37712 +oid sha256:3a9bded9ad53ab7e107049f7a1f117cbdae683df8d4e22f3e986b5b96bd3489c +size 102860 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst index d1df30083dcf..3675b4765aa1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:842be553a93f607bb045bc46dab483409543b0dd68c1030e39a58dbeaf7e60d3 -size 24363 +oid sha256:9e5a4d81efd518227add83b2c2340a92b921bb467713754aeb7b86d8ee4c3a09 +size 61169 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 4fa416c08a38..93b592ae2c7b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4ee7835b7c4bc505a6f4387fce8146547ae16cea9dc7d981f42e4bc681c57d65 -size 23969 +oid sha256:b5ac76ba9aa719a356476121cbc40006ef8d952249463b6cc9b147603571b462 +size 60565 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index d8d0a6cbf7f9..6066d633551c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8658b240e4c2245b00c62bea3d4c5d2ab0a11cf044967eb9ec3f32218bbbff0b -size 22694 +oid sha256:954469d4f61ff4766884b55efbebef6b68ff16b9f041cc7f337da72b27bd3c65 +size 56498 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 61c2700fd725..8b4487252e14 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:dad61cb34deff41888c39af46344556443518b84feffbb04bf9abfd03549b2de -size 23516 +oid sha256:0bda9a30fbe4a9fea2720298e665ef3fb49a51e7542c4803f19c6b4c50cb56c5 +size 58945 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst index 4759636955ce..175ecf06e703 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2d4e0b97bb004fc25e4e2c924b25c7ab4255e5d1cf411dcf3c21e1d88a407848 -size 20191 +oid sha256:48e1560a56064831bf51a282ef90ec687e510f1e6e0d51026a0f434fb4086c5f +size 49478 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 18d31694f490..2c71938ba367 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:135da039a36f5211f15ab38eda95ce2844255689304c7a2d0d03483df734ce0e -size 19888 +oid sha256:36c39ce84a045a39cdc2a246ab36c4995893d6729ef94ad3f6903b342303a51c +size 48915 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 5600f7c4ad57..4a463f463271 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:293a380784c41897a39282019956ca33f63baf1b202ec78ce9c01285129ba5ce -size 18557 +oid sha256:8026417bc38164bdb6a8a7ba327a78a997aa85e07d2551b2d5092d49f093bca8 +size 44736 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 233d4e3f9001..9f9f141eb413 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e5c39cd901309b9f28a84218dc6f986a96d488dc5bbce9c612c6c8e3004fef76 -size 19384 +oid sha256:2729cb70ace2e156f6fb66ac951964a88d0105bdd20bb07cbd1561de690ebcb7 +size 46872 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst index 82ca98a632ff..86663b8404a9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1ec0f89e045cd4486b716339f8128073c5b5154f6638ee6b16d0a6326e0693b9 -size 25255 +oid sha256:9f2d9faf4636631c0003e2e6be934c62abb0558a672ed0d444bf8ae5142a5ce2 +size 64343 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index dd511ff25bd7..a929fe28f430 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d6adbe8a89099a10d642bf2c16e048274c15bd735c400c834e9510fa556cfaa0 -size 24877 +oid sha256:88f261a8d297da016cd0e534f5d05ab9a2d7008739e02d49eff9cbddd7745620 +size 63766 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 5c63d285d939..87a9893a3026 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fbefb1500b75c1929272fbd63154d9711f72ac5c8d5ca4100f1111a027dc8e23 -size 23603 +oid sha256:3d24f8055a1fc6935526aa8512bffb295ea7e5a21a92bac2d20c9ad52baf4724 +size 59608 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 1b547e8b2117..f6dfbb7c4696 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:27b4fc8a51efb26733cfa257ddf23dd3df7ba85f50b0c8c0a8c6906e26653a4d -size 24386 +oid sha256:a1ab4dede285bd57b08576095dab9cd5a0050a7ad90043d06d7a9a566bb11ac6 +size 61869 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst index c98828c15c57..fdb7ca155b4b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7039073f6431af45233f331e70dc84f7be5078cef5dc149ee7a83b73bbd24f8b -size 21109 +oid sha256:85f772257b77b00eb18e4bf1580ecbbc990a138e13df72af9fc4f2094bf58c6d +size 52636 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index bb5588c6949b..74f49192a2ea 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2d4fccfa1fe674d117a6dd59388664d7a2561da4343448eb81a16771fed012f1 -size 20806 +oid sha256:c63f170023b97ede9d2d7bcf51bf9333ad9ccfc2f7aade8100ff3873c1c8ae2d +size 52005 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index d4f89293efd4..28d13449d408 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:260244495930cdd46f2413a526527b2bf8e5c352df4d35aa17bf34e90c2baf1f -size 19528 +oid sha256:9d4c33f2b40a59868f34e912ac7216a683990340b934e7a91dd6103f047bd20b +size 47949 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 577c9f2b228d..055bc040b1d5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x256u2_s9_et128x32_m128x32x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ff9c520aae2e2d8e96d4438b24d9c52dbd0a01caf8a9e37f53f2ddbb3f9de05a -size 20330 +oid sha256:cd56f5d8c168ee39bf3350c8cf15df45c5ba37659267b5e47ddfb6786dde9660 +size 49970 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s3_et128x32_m128x32x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s3_et128x32_m128x32x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst index 8ce7c4e8ee45..736775540422 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s3_et128x32_m128x32x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s3_et128x32_m128x32x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d8ff22225958409f8fd14f7581f4d55a16c1b2142b804b308bda38001f85d8e7 -size 28644 +oid sha256:a1014222bbd6f864b1556d39c343a94152d1d9080ac4b6971b5b1ce058a24443 +size 71685 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s3_et128x32_m128x32x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s3_et128x32_m128x32x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index c015497d2b83..1d936beef160 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s3_et128x32_m128x32x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s3_et128x32_m128x32x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9202936e026eabbdc197d8d1161b077f5d687b8d7495dd7137cb7374ea48bf81 -size 26386 +oid sha256:dffb3e4bfe6a1fb80b176bd612bfb5de5b848fff0048a024ccbbb25cd12c3154 +size 68519 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s3_et128x32_m128x32x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s3_et128x32_m128x32x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 7a73931230d9..802d07f7bba0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s3_et128x32_m128x32x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s3_et128x32_m128x32x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9f4e6d706bd2ff31c8785769636ebd9ad79076439e2b6a62a03618fb3aa1117e -size 27201 +oid sha256:e713266b25ab2292a777b1d85ed847c332816776d73c2a98ace8dbeb3173f03d +size 70726 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst index 11eebf69b585..e332e8e417c9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f32d03d74c186342e7c4c0164678da7029949fd77077a06926bf49da080047c5 -size 24684 +oid sha256:664aba0835d4caae473c394dd9bb7606a3f69b74e93a79ebf1a95ecfeb01a50a +size 64215 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 60810ff1df47..2c6066775f0c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c25b165b7f23c07634853fd95a633cbdf99f4c209607d466c2c3444961b33f71 -size 24319 +oid sha256:c032e52da88794423bb62bf3577cd8aea700e44e3dcdf599f849cd7bc49d767e +size 63598 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index b60d3b25987d..b8b7a69e1e9b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2bcc7f1853b1e78e0b49b2d13d5450f8f70c8e21cd9ae9c0c2c9e127c0b145c4 -size 22846 +oid sha256:1ee2e3ac8d2768abd5528ae1a2c6af98e180725a33930e53ce23db8bf5a82d5e +size 60306 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 35ea50a55dee..9d4be011ffa5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:57945424bc8fcfb9b7b5d32ddc8afaed2ae2781b475bd318b47ed585170a703b -size 23690 +oid sha256:0f1e9ea187e6c74ac6907178d6c62a14e061cd63ce524a629b6299c9c466ac9a +size 62407 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst index 6377fdcc8fd0..04f8ec232c53 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:49d15f9dda804d26c770f64f1f87aee378012022be2e119d126bca79bfa19210 -size 19209 +oid sha256:e3003e9f8294faaa840b364dc409e18d363a0aa3a83cce6069307d8007594355 +size 50607 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 2806c5cc5083..96f307ef4764 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5de586dc82b980914230b4891b4a43c2d45f8239083cca27146aa8fe8b9076a1 -size 18840 +oid sha256:17fa26c666c773d255f57688e758ad8a91585191f35456a324e2d7de10a20820 +size 49767 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index c4130fb8f073..0131bd4df40c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4b0ecea3e2f53236042a44db11bfc2bcaf1cf6873d75a6cc1b8c425d223bbfaf -size 17606 +oid sha256:d20f76890a84164adc3b5738f3697a946c25b75b6cc0494c83b428269bb93de3 +size 45869 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 3cf556228295..ded888ff5a64 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9e9860983e7a2615182e03ed8f6ef2f2522b7f61e490d30e9563e274b965297d -size 18262 +oid sha256:4b1cf4ea1347eb5ee0e9eb6a59f802e75100f602fc93279d635c9c58ae59ed6e +size 47635 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst index 2379aba362a5..de8b4a274988 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cca851f079a33efd7b270ee2c0fd80dedd56d489a3baca0a5574e03a44420e9a -size 25614 +oid sha256:45d2dde7ac84a4c799fa98f91873311cbd01f2e11f20a9d40ae819650aa52aa1 +size 68158 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 52d8350cd896..ec436330332a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e2dcda0cfd795e5719b95bbd15b440fcc4c9639c63ea873fa20ddafd7cbee4b2 -size 25205 +oid sha256:1f24ce4572dbd87b321e847eabfc9c5560c7481995224c7f4fc326ab0041c146 +size 67353 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 081b7ed3ebf6..1c7987fe3aec 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:716376469a2d4a9d909dcd0c2f4f0e92296723ab002a8174725f7066c55a0ee4 -size 24018 +oid sha256:6b3a4844b1fe7c7f3362a5ade0404a9d7d0c820022c25b33fbb6c24b83dfc0ef +size 63592 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index df8d7c001aac..c3785bd5e5b7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:321227ac9b15eda1adf2cc06efb51b43b7520d84c458313137e118f811d4de90 -size 24847 +oid sha256:2568ed48c4904fc7d44a3778f206cf4898a0c01e95dcb3c19158f4fd699a6da4 +size 65728 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst index 7b783eca4aae..188785eac742 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:06d33e2da0aaeb5dbfc52aa4b4091f353332efcabd120e74c8ad07c7f40e9923 -size 20373 +oid sha256:61030bfc78995f0c22270e70a65a13e915a4e0c7ecbcc3b7970a5856f7a4d6c0 +size 53969 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index a6e6b7652eb0..bd34cf6f6a4b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bf0be3ff105aa304f1ce1fb1dacba85c0c44c8b6b844a470f9224d4de34dd3c2 -size 19991 +oid sha256:335dccc8cfc75fafff5f1616962bc1a02f8d457981243a65e81274296689665e +size 53525 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index ab914f480c33..f1d0a2c11c08 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:79a52b6cfa5600368562ac397c38176c9f09dfcba6aceaee10e539e09992a774 -size 18679 +oid sha256:db18120f0658f9c0ab2dd83b90e0549eb3f2229daa2022d5804e19851a3f7e92 +size 49608 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index b4f598d2f4a2..bbc627cea315 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x32x512u2_s5_et128x32_m256x32x64_c2x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1388e7fff8822dbeb43037613e826d10fa92da732b3b2c03c36788304a0e2969 -size 19339 +oid sha256:069de3cb6642ba5add3f7dacadc800aae61f56d81ddd952941de44c2266f584a +size 51672 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst index 650f3b411167..4c783d671c57 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e1dd84adfc30df9661c80c8d03bdd88dcba1f4f7e164d61b7a4f411008811058 -size 34346 +oid sha256:a0f0f935232533bc0055384f67b0fb0e479153b931599a5c9e4e8b9410d94bc2 +size 85935 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index c2afe8c0dc7d..478a8f1fd3ec 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:07b83e6b113f26b15d312a4c2bc37195e8305d0c7157e607cf1d997ce2b249a7 -size 33292 +oid sha256:86c119a9d40a2bdccee471d0e240ea5cd6ec1a55d63e2f00ea743893fb0c7ba6 +size 84083 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 205144c6cc37..52f3f9ea3b80 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:690baf1b14e263642e0b97054800c211e9672451cd4150296001d19b926ff961 -size 30997 +oid sha256:becfff1b7bb6bf27f58ddf06f6f707ef0c91f7aa385dfe49d169f2858df1aa1d +size 76795 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 5111a3bc8b7c..9c0ac12f5dfd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fdc68c368415bd27f79d21d5d3ec6cda3fc58c1cbb1c082d5bf194e745aa1752 -size 32514 +oid sha256:478034e4964e7c38a4665c693eb4d891ad23d8448d5acef63e3ea62ecd469e75 +size 81269 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256u2_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256u2_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst index 8ea555fb28d3..a68262c427d2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256u2_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256u2_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9c0e60f6949eb8f9ada4b96ae46a860e08d038430ff8c282c354f4467a995ab7 -size 35218 +oid sha256:a3ac3235c6aa111301e66e2acb0117cabc376a76aef05e98493a220f9557784a +size 88838 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256u2_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256u2_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index f30a885f87c2..d89342071ca9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256u2_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256u2_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f9df9ff224cb011dc03e4da50403d43a03aa3160ceddb0003e7d262f39bd3cb5 -size 34133 +oid sha256:9bfc7eb21b19b551b408e5cb45302b55e1965c312416add1e14fda2950fce8c5 +size 86837 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256u2_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256u2_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index c8e5c8dc64c5..02a2eb185681 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256u2_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256u2_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:85efbeeaca57c6a97ff3ef0f1635b0d4a4e1f9c1910b36683706d5349d7fba54 -size 32132 +oid sha256:c85bfc1f04793849e787c3ffb16f46e616c3d8c5c31553b1d8792a3e1ae52acb +size 80314 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256u2_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256u2_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index c3b2b7d3d168..a826fdd9aebc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256u2_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x256u2_s6_et128x64_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:38ac01eaa34b75806726a37647b2716aca4fec8612d8b1382ffb115037329fd0 -size 33675 +oid sha256:b7a88ba8ef84ffb35031620ce2e7be6bb3aaadc080af4e70e0fb323b2343b14a +size 84302 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst index 291f85c72f7a..46bcd65a9285 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:dbf4f43d384ab91b3489be6872c0470255ff1d8fd89d1e4fe79a3fdc2e9f885c -size 28772 +oid sha256:a96cee6c22e801bcc5c72ac583a6521593077405d1c3c2721a50ce16c8b1a81b +size 76514 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 3250db424a17..c17ac0c8de51 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:01a64ad403360ea045cada366f308d6c4985114973f5c5f8ac0bd5dcd5562953 -size 27788 +oid sha256:9bd2dd91aace8f48c8d5df19bb4e2dcc4bb7f42e25c64de45229b14f8c4e578a +size 74719 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 2b22086fe243..4cd5c77d34ee 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ca9bba0293c6df956af4952dba05bf8c0245551dca46ea33bdb4784788fd1980 -size 25747 +oid sha256:540d3285d924f4fc9370cff8a54bd6cf6bf9fe819781b8a33237046774cb5727 +size 68539 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 769a7e828aab..fcdda5010722 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:61795fb7fc4e4a84a0884515cdef4cf69a9b1ef6339d6229537e0fafd722a857 -size 27468 +oid sha256:02dedee4d33068106ca7ce53a58496bbc1efca53a4dd5185241ac3935741af8b +size 72309 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512u2_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512u2_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst index bbc1632d5762..862daccba02d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512u2_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512u2_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:03f48c3fb13a4aacbdb057fd3eb3e0c1e0fb3002b86d632fdeb881a2ebf8f716 -size 29560 +oid sha256:9c0bc75dd7ff3a35535b89e5b37c6b90c717e6b9bb39cb19e0858e3a7039dcdc +size 80630 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512u2_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512u2_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 2691e813307f..63c1977fd99c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512u2_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512u2_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a14965304a8f8ccc76c4477a5d587b58322a4e28f3a6f19f09d0ba4c35b021c7 -size 28602 +oid sha256:07c9cdbca6a10d265ca5452d376d28dc9f8e217d4c884868873458c30d4659d7 +size 78889 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512u2_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512u2_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index dcebd1a5d898..c21a72cfaaff 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512u2_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512u2_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3604bbb691a9125b9ba07e9e09ed7ae56482826f69622a9d174f292f8565f755 -size 26918 +oid sha256:4e14d5c6dc0e0aa11dafb4744ef3388a0673b8ab9270bd6746572a5a2d998f60 +size 72381 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512u2_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512u2_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index dbd836fe1047..16a3b92770c5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512u2_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x64x512u2_s4_et128x32_m256x64x64_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:90fde7234e762cab093d4ef2810916cddd1cdee06e01df64911d889343003b7e -size 28449 +oid sha256:4510ffa96738510f8f872b286dbac61b089fb2fa7e5bcebe500ac0c2fe47e58d +size 76842 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst index 4910d9642851..a0acc359da17 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:54f5717cb756b8f9cfa73b4efacb8125a2326b71c54e3774650c97a66294f2ee -size 20134 +oid sha256:af42056e657b12bd9f0ff9e24689eecb5bd6679e4bb8d698fe5d3097b798c2da +size 49191 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 88ea85f35ee4..07161dd8fea0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7cf09c290c6194f0476d90675807d5645881a466bbd45f703393c149902db424 -size 20044 +oid sha256:8f435d7e95d1cff067feb343845e98d5121a1e7438108359bb25b0b483bc5ed0 +size 49011 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index accce7ad632d..ccb6246008ba 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e9ab71c9b2783fa7a933bb5e1a9ef4d346289559e36142f5c9b81da9afaeb2e6 -size 19403 +oid sha256:699482a2083ac68d64195bf98fe6a8ee66138696ab5295f93e55cfd40dc78b1e +size 47332 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index e06c844715e4..5039743811bc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:80332f3982024dc3a377d62565afd2f3b59ba77ebef848580b5cd2ee59d68d91 -size 19534 +oid sha256:687dba635f513a440a9a4819a5ebbe0de1dc2d4f6e7c436f7d995c1d62d17a66 +size 47797 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst index ab2a447865a9..7535872ec06e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c9c1b5750c9cacc8fd2918411c01ad4191ea52bf8465ea42644dc46c1c0a4ee8 -size 15593 +oid sha256:056ce0ffea9d07df839370971ae2d956dbfd1d14ba22026481f15460898f1e3a +size 36964 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 1d742a98fcee..21de025c5aa0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b88d2476e7bf86e03cbc48d0e3e52235e298aa7b12d618c927c004b468799903 -size 15453 +oid sha256:f353a22c058b475307777b81797edb50fa36887b719ec5755c74c4325b07633e +size 36799 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 49d01db618cd..879288300e3a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c4e420c01dfa154c179100bf4a3f6f9d091675d83adfa421498d2ab2fadcb129 -size 14872 +oid sha256:d345711712ef258436b5d56e99ecbfc55b75f42674e0cf20fab6a8b785c8c774 +size 34892 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 792cda952c51..68c1e4c84657 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fa697983ecf8bdf517ca126c65e2b2f83605551a1f443375b5cf8ba2dcc88f8f -size 15024 +oid sha256:4c586b663a446a408d6a5aba653486c4a611215108c07e3582831df6092294e5 +size 35472 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst index 8cf67ca55da2..2a9e0ff7d3bc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0bf4776d47706904c6556f2f3d4cb457237c48ce3abbd28782cd80270d639324 -size 21068 +oid sha256:0d5c3a31753089b4034056444d316711d12c5569a4f34b71ec14a0f3d1a44ff6 +size 52382 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 59b48ed371b8..4aa531223241 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:96d2886cd91732edb0ec6a0fc9b0c617062f3ee1cc32970c2a738621c73a431b -size 20967 +oid sha256:8f0cc0db7a58dd1fea99496f023028886a73dcca569e0ca85c7cde1a1b3d6d28 +size 52246 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 37d55f34fc59..2879dadc1005 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8afd0cd55f8682cc75f3160ab7d6402dc6ff18db5d673fc1f80b21cc6a0e761d -size 20173 +oid sha256:f49229090a7f6213620c6f89683d9c9b590377cbf202be8daeadd9c51cd9f445 +size 50117 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index c3e2c95ff17e..3963fd530879 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5d9df833ceb5138d3d92c64bf074809e518905554ab6764f5f22d1f6b2446d58 -size 20355 +oid sha256:b286ff8d054f96613a6c0fbcbf4a66c46d16444e322636fb56360efec0e664ca +size 50525 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst index 7dda23a442fc..ce3bd56bcecb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ad4322c860cca048bd550a01d2468e3f931c07a7c7dd19a92a94d00d5785a326 -size 16481 +oid sha256:45cecce0fb44a7fdb04af0c2c4ed496938a94b9e8a27ac61883184cd0ba3b06a +size 40205 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 3123f49a5be8..0a3abc84291c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:97b872991b0db95b82f1b4e71f541789b5b50130e725740a7f0293729a77505d -size 16395 +oid sha256:4e3ecf99290c71bb68642833e4cd8253a15ad1830cd7ad687573111f7e353e87 +size 39980 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 7f5ab17c2aad..54e4053001c2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a2990dac0b70d304418bc7eaebee2fec79c7409917a61e290d5f50a67fe01224 -size 15776 +oid sha256:b7c47664cedaca04407cc2485ac1089c16fa30f1002dbd0b0e39b4f6c222ed37 +size 38176 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 699af6363f36..ccbab5b299d7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x256u2_s9_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_ldgstsSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a79d2c70411749c988439e95b3a1884a0f09707fd03f8105d7e19598a9ce60d8 -size 15964 +oid sha256:e08faa28161765af6ff40a18ef5d96fe67789f6ac796927ec2aa96a4992b724f +size 38685 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f.cubin.tar.zst index 56024d0181f7..6b8f63b57ecc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8a5711e37a24140be143c5235b9c3a300ab8e7ff9a607e4345ecb4876f8c2ee6 -size 12944 +oid sha256:cd72cf8811543f216def8464fef4671834f910c2d19107371ef752cb5c9b4e64 +size 33758 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst index f7cebf4dcace..e7d66b01dd4a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:93c85c2e1830b932b5ca939b6a261a341fa6ae2f4348fb70a425bccb10e56d58 -size 19532 +oid sha256:6bc84657ba5c428288261c5bb4a5ea7ec54233429619ec5fd0da3d947ccce50c +size 50318 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index e2a92ffb8937..63a9ea66ef5c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:000ead2764b837ef167a77ba2ff635188a8d93596111fce4b72b60ac0845644c -size 19483 +oid sha256:c898867089dce6ed8f46b74c198c2d7140e8251e7d0e634ab39d212720d53aac +size 50209 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index e2afc527f14f..ec777cf0eced 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d92cd6253a886da4aaa70f8ef2391b03360059a02a2f166f90c6a1627f2b8cc5 -size 18712 +oid sha256:5b3abd5810cebdbfc43c9981b30b346d5fa7346c87fb9f29c31ade9a6c02e965 +size 48341 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 447f73f1f49e..211a51199891 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8523241f04834e3d9cc67ccb8d1c1addc1729d2833af01d2b115cac6c730265e -size 18972 +oid sha256:7b9a23f5d15d25a9c6b06b079861161b58324fe242aba757db7e2986ff75f133 +size 48984 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst index 7aec3e8f85e7..806ff9675b7a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:572a62dff67997c8fa46f086cd2fee2647bb180b2993734ddc55c4095cb9c2e9 -size 14274 +oid sha256:a87569751000e3b48acb9dc3eb841ccc983566397d5df9a8149a99a716219db7 +size 36370 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 5dc795248f19..56a88783756d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0fc39c4b8cc647c9f9ee0f7d2bf55f94190bc7f877d37364986e59124b69029d -size 14200 +oid sha256:3972d351ed5b2902c52808ca03ed2cc04ad8f2297db14e88e797d28c8fc5fe50 +size 36198 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 26600cef01ba..b853a7eb67bf 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:146af4581ce39c1a5e9e6a677ad4af02f03500465e04e7a89809a13fda931b1a -size 13550 +oid sha256:24a5c7266d3de71ee205ed06f64a2b54c1e76b67e03cfe0fe1970ff86c6266a7 +size 34369 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 7b44c5283d5d..8378d44bfd10 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:26f5f611a18b2d66558be4ea3cd112d5d6898e9a8119c020232623624d9e750b -size 13705 +oid sha256:d070402244c6e977c9f4d39ad88022269d64fd4752f1cfa015d20f027dc353a6 +size 34811 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst index 2f685efd8344..a7902514a24d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6142e0a678d88853e17e1547782862cd3271b9d20a082ad9ce213356d32b0f30 -size 24088 +oid sha256:798ee39590cdb47a7284ca2985508465c0ec26cc7d51f48fdda322faccf9c7ce +size 60019 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index c1f190b2b76b..65636d5638d6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2d7cf038627930c7d0e93afb0f011c05fdef82b195bd89e75c7f25f67188ebed -size 23066 +oid sha256:7f94276268a7d8ff38ed4b866e9e15253aa364729f350f41cfc85e4fa2c236b6 +size 58423 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 3b4c99318262..97c60b08dea9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cb81fe4af3fd8dc61d33c604ff85e1b9746ae6473e36d0a9cebee94aa948995d -size 23362 +oid sha256:84546ab399954f12767edbb71a2c5b2e4298dd4888e97951c9ac1cbf9274199d +size 59024 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst index f62c56a9df1d..87ed9bf0a254 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9aa803b99a78fd716daec2c8a7a0531b7ed4b34b6c74750f6f003acfcca85ce4 -size 24381 +oid sha256:deb1d1ee96a3abdcec677f42c1b3908b14073012af0329a0e1b69ae4ee2741cb +size 60476 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index ca87f17fa1fd..b9334909cbce 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1e0fb285264854f2d6860e822a2db368f38845d922cd8bed1d7de8bd271ba46d -size 23412 +oid sha256:209d5df8dec6505d283dbefe1add3d260ccff0d2a6641bf6267d5c011c55ca7b +size 59004 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 17e1fecbbdb7..9d2b3743f683 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x3_rM_splitK3_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:54eab5c1575989f49f37f4b5b7c6241f4b8bb929731dde9abf4083ffcf4ee82d -size 23700 +oid sha256:25e2876590101e075706cdf56eba028eedd3498e88710d94fd6775f9cabb1612 +size 59494 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst index a52f8b99b4da..f73929a67fcc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b9ac99e657dbe139a4f8b65166d56a70bbc02dc7aa572cd9dbad549dd9234d3c -size 24166 +oid sha256:acf4da3443d77d76602122b319bb4b5a2c4f72f9773b8c87b75643b8a7599abf +size 60375 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 6f79cefeb433..44240ef03bf3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0bfa931fa297458228606d1012fb48e9c9f117173f5ce14e56cf6e0d579de668 -size 23263 +oid sha256:7c92691f9ea75e83362e0aa8598a108a515641bbd825f50ed50876b729d1fbdc +size 58925 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index f6ca3bb042b6..023412709e38 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512_s5_et128x8_m128x8x64_c1x1x4_rM_splitK4_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4340bf676e8f1ff211c68b755a26c1d8732ca2af3154554bd1cf5659283b81fd -size 23583 +oid sha256:eeae2640698d9aa285f44cade6dcc8821d601c5b8a8b06524f30e250d9a3db32 +size 59380 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f.cubin.tar.zst index d37532f3dedd..8f03f680065e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2b6127663977ef108b9ed695e4b328f0cbf05bd38166835f81420f31a205321a -size 14029 +oid sha256:ed261d1a734d5874f8d80bbc0aa76a054d60a53671488d001372c92df33c80ca +size 37597 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst index 0eb6e2c45b65..164c762d6df1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c97e49f5cccc1bb3a0dfdc0c0f462474b7952a45199fa5afa9ac2c32cfd04810 -size 20614 +oid sha256:ed4edc9b094aa779cc2190943da517dfdb0c85b39b84357f2c7b694fc723ff5b +size 53927 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index dab1e0b35244..67f6a7889eb8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3a078c292e96f7724bba9e8b14067d80b23ed10758f9d600bfee4e365bd4fd7c -size 20520 +oid sha256:1000f12c9162763964d35914690886f471098dd4ffd86e15694a31c095ea36ea +size 53858 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index f6a8a3463ce6..2a6d22adcadd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0ee5587c67da99afd945b247ab0673231794c50bd7387840e48465690217389d -size 19809 +oid sha256:5edb7a6b42901a09ba2b6d5330eb6a306aa30b1943d9cd85d4a00de90f2386c9 +size 52079 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index c4e40e036f50..f9dcd09db231 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b78c7915141412c8bf6de4adf9e987436810dcf3b3f73a6a0c2243489c590664 -size 20046 +oid sha256:4e7dcbb4d052326bf717e3eb4bffb5983fd7daa06dbe92b0dfbfac8fc6d73a4b +size 52750 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst index 0e9a2f52bd0f..3f0ddf4ff50f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_geGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ba22103aabc77fca89790f27c9a6279ff193199294028cfbd36d9d41179ed3f4 -size 15313 +oid sha256:401c95ada0df004bc26c157876f0e41fc3fa2935721affb0584532bb3098af24 +size 40461 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 702b2b3225cc..06e3457b1ff5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:356a86c7c6bd18e5d8d6bf655ee386e28db0ce8940bfe6fabaf6897614fd4150 -size 15251 +oid sha256:51df065ceeb70da317b9a116938e5b6bb28c38eae13262eccbdf95a214485662 +size 40336 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 2114e3690433..edc8fbf520d4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1227e28a30f186cad6c883b6dc9d18cd8df635f2830bad6cdf67793a7c25ca30 -size 14616 +oid sha256:4c859cfdb53413a1d187c6220eac9551aaf48975c808cf8c69dc7e73eb9d9301 +size 38493 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 6d5f545d7982..19434e275abe 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E2m1_E2m1E2m1_Fp32_Ab16_Bb16_Cb16_t128x8x512u2_s5_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:546a06f174cadfc91a072d8fbca3ce95adf456e972872d250dbd3bbb21daa5ea -size 14785 +oid sha256:d7b4f92e3e317219d7e1817c95b7179892e35a6f9bf288d276399b54b059fabe +size 39070 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst index 4fbb0185c186..c89b2918e39f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7b7075b2545be263a7dceb512f232b511d5db3baa1ea2186550db75c616c5d73 -size 19791 +oid sha256:180e67c9eec1adf492cea3415de51a3cbf16c408e0d31009005c9616fde406d6 +size 49389 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst index ad5afb2e9853..f88494e59613 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c56450f3578726918543d96ecb2a6e69d3946ba7f5c7ab2e62bbb5fdc97b6f84 -size 19879 +oid sha256:d513239c2d6b89e39864fcd022b0989738dfcaf0474cb0af08155c874185dffe +size 49455 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst index c474724620b9..d54d61c12196 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:28c05aee9a23a1eb8d37504bc9918fe323756f242b4d54f4ea6e5b17acddcc48 -size 15338 +oid sha256:16d626b50e20cfe7e8043f1949b8c6264ba9790d68abd7204c20af42852e4211 +size 37319 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst index aec695d7af5a..bb2ad4e2b6e0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:841823a0f2769f61066e8c3dbfadc06e67e038a848671d6fb4751b5303dbbee2 -size 15338 +oid sha256:5ef2ac617ab35fb1ead2aaaee9e844ebe48ec8fab5fc6e4f3428153fff2c8c0e +size 37367 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst index d8fd81958f2b..23e20941a476 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:812b963b820f75f7a9a818907591d7e15975e5a736acdad6c70b84d82b6e64e5 -size 20355 +oid sha256:407737f94847f1a1fdd015b3c91444113f488c939ebe33a38a734e49c790045e +size 51562 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst index cbe2d40cbf14..d14842f0592f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a90c96a2e6961e6859eafaece3637010951210c5680f6946f0ec1b7d35bcee75 -size 20417 +oid sha256:bbed0a81bb6ff83cea2fa830b32bda37014e2859cc1a099b90869117bb714ba2 +size 51557 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst index 6718f056f827..a704891c5af7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:81574399a59c27b3f624d139349e9447cdf394c4bc74ac92a78bbdb17d3566a4 -size 15896 +oid sha256:efa3807d7ec3f896cf8db42981595d25e70033d3d9849eac6838dbd31e00b3cf +size 39651 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst index 1934d3699f81..63733106eae7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:316313109663f6852a70f79fd21ec6cf4f3c5e9b9a1864a1b52e1d2da1d7b1de -size 15906 +oid sha256:f853e35f02169226329d2eaa9b54b98a0365244e1af979314fb3726a65c28678 +size 39742 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst index 0be0f52e2640..73b8caa06b5e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5baacbdd906cc7c3cd783b6b4c37cc71486b25c5173843d1f7d53ca2d01c87e2 -size 20501 +oid sha256:ce0d21704f7cdbf2c503e9cd3fd4b746118dc47c54eba8322da17327eccd0720 +size 52716 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst index a5f212c87776..2fe6494d42fd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:64418cb727f5e43d09a724c8b1f810e8bc6ff90917b2a4d84d2654aa5d6ccfc2 -size 20684 +oid sha256:cbb094534713fcdd677bdebed76854e7a4a19b487f207f4f867144c30380118d +size 52829 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst index 0452ca6ff304..02ef665b99ad 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:95916ceb2a4c1617328c703dcb42bf2baffca5779ba7f06970b7919c9ed63f60 -size 16461 +oid sha256:8945f50dde192b103bc951e31e09f1b1874aa1ec11edb675efc93c63549e34b0 +size 41255 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst index f2c88b2b0e58..86b45104d9fe 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0fa454e6c76607191959004fe0d1996ebb9905a7dd0e0b725c62b1a7a87aa9d9 -size 16579 +oid sha256:cd8ca81bc4d6d99b4b56ef741f54fbc927e4927bf29d1b55cd781a63856ddd93 +size 41347 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst index 9870f16cdb56..f34823dfbb32 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0d8ae73e4c9c30308952275a4ee7778ea7f016aef9bc960a4b93af8be53aa4c3 -size 21031 +oid sha256:242d9445c4e8a82d16864abb9161c564012ac726cddb35d0cac5db7aec806437 +size 54998 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst index f462acca2b00..06344000a915 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:91fde91c4798a8435713b258f290bc7923e610d5126f9f7cf9025a1d0081776a -size 21262 +oid sha256:f2f14dd33ae65dfef9f5be6063f6d5e8223552f7d1d7180ffd18aca2ff7e578f +size 55183 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst index 3d6e3f93bcad..f03a0575809d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9603b2b95c55cd03d6b35c32035c22fde94d8bb92331ab2fd9ab6e71c6977161 -size 17073 +oid sha256:7ebdd3c835ef8520084abacab02bf66933e72dcc30d4cfe19cb5906bfa0662f5 +size 43524 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst index 738d6a07c059..3f12aad1a837 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:460dfe26a7cd5e21858215ee7b2f958a5d9f721fd298b946b85491e8cd7ef55c -size 17200 +oid sha256:b06166a75f10208bb50f224e975007cfc7d7b6d347826293726d67276bbe62bd +size 43511 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst index e4fb2db13b80..b9373b938f5b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:270eeb5dee6200954af94bdeef01743bd6c9999cc08b6f15605341b30497486a -size 22968 +oid sha256:768cfe41273747363e9a93097e3f2ba4f818244f902cd67ea822169e6b831597 +size 59493 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst index e47fe2966396..3f7cb1c5e385 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8fa73f4880b82c787ae1469a3ca18e746d2cac851f5772b79bb140a1311f9a01 -size 23125 +oid sha256:dffeda3cb6d36c1629eedb91778b92d12b8488e6d08dde8d516279a0e1ab48fd +size 59651 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst index b7499bfb49b7..c17350982403 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b37433373a51365142b0baab74a3d8e5ad32cbc9279c967388e2594f9d234273 -size 18376 +oid sha256:ba33885ce1a6169bcbaa4a27b92c108dff23ff5942837d9aa455187420410369 +size 47965 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst index cffe16c3aa8c..4b32f5264e66 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1fd43fdb6557fdf2448b04b988b0a23d84fb98e8286b70da5996723e5900f3e4 -size 18799 +oid sha256:32019f8c70914c0c3315cdc890c0f357900d1d132f5aff4176171253b06b322d +size 48457 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst index 11adad9ac230..4023cf0c1519 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9d28073642169f8729fef2651d72561e6a380ecfc96904180e9aef420fa2d3df -size 23488 +oid sha256:ae49ae07ed87bc37d31a533c5c234a90a65f01b77212e3d31da9f0f1ee52f1a2 +size 61725 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst index dd2d1b31032b..a110fdb26c14 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3c9312df763a610bf529ea3063c15f14de69135f509bd61ab67a68dda89b7c45 -size 23715 +oid sha256:22bdb882c06a32e41ce3a38174b5d201766fb15381dc2780875d14c53fc6f98c +size 61955 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst index 91be455d2c72..ea6500f2e60b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bea40b0033fdd74da20373fdcb26edd2e76ff34f195a5643007fcd8df910b32b -size 19074 +oid sha256:6e3515ef066764a1633c8c538bed605c38807ebfbbb931dcffc5ea7f04fa8e9b +size 50266 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst index 7019c8b7d428..0f1e3f72bdc2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:13b612b9e521bf8513f5355d59c226cb7e3051dd279c540cb3ce1ca356dc6952 -size 19483 +oid sha256:d823f1a4fa3b7fa3eb574600f3220e71cb100f5caa9813e19cdb66e02518308c +size 50551 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst index 7a09031c9979..ddc54806bdcf 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2abbe069794401e20111cb7d1046ecb5c7c5fe87018c6a0500c078252b2ee791 -size 19223 +oid sha256:7f69adf7a9dbef6af75060d19b9f3cea4fcc2fcc1cf3c6b49a640adfda12d8f0 +size 47238 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst index 796ef4e6519f..56566bfd24a1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2a21fa529efcab85795411591e725171dd55b4deccac8fb64b6125f5f6d01511 -size 19298 +oid sha256:6bedf80bf436402fc03ab09ddab704b7ddc98df3b0c7a1bd93eb0281fa547789 +size 47289 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst index 7093ff885b1c..dfbb4c83b212 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1ebd990766a7ccfa1bcdb7dfe8965a7ae60c7bbdaacf4ca6f0f6641599906be8 -size 14615 +oid sha256:9a6017ec9f0894789a69cd5064dc685abb7b9a7e67e54adf516570a853c334fa +size 35116 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst index 48fdffa18aff..78a1f73d260b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e9913e712a00abfb8c6027712aa1590274cd022c4e2da3e4e435f028c4cc4d91 -size 14659 +oid sha256:dc0b6a5af75e235642a9d88db3df0a8e5be8c6a158e1a03859d702632c0a2b7f +size 35115 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst index e2e4e2e39f9d..074617806042 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c2fc4cbe8628a3b517cf13ec8dcfb231c755d6660d45bcce0337d93f57fdf831 -size 19694 +oid sha256:7c904cad72ffe64c6e54367c655d938b40727444e4defdfa2cea0e0249b06f66 +size 49344 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst index 5d0bc69056bd..33878902e3c7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8502fd24e2a0c29a46554c4f092e98ddf29cb837be655704f7ac28cbeabb8575 -size 19775 +oid sha256:676aa59aebd7263f0bc0e607ad2f6e198bb48f3df844a68c8891ecc9aa807cdd +size 49374 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst index a6bf9f01cf86..5d0aebae759c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c649a8a2c3475c702580e4f46e5b72f6809756ab414d39dc21d475f82024b8c0 -size 15177 +oid sha256:83f5b08dede053296530f3eca5ff269adcc4471bfaa71cb14edbb41da57b5c5c +size 37578 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst index dc627f080358..f06b2b521d2d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0ccac09a41a1a6ddc377435c361e40b816e859f7ef414c6e81161d5d2725149f -size 15223 +oid sha256:6e089c001a1b966e56e2017d809dc1840c70c9e064cd52d9aaa522ff42b67716 +size 37520 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst index ab50345e49bd..5158ec9ffeca 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9a5b6283ae4a6f285413a1b328b9f7491cb0dca8555ad112b1ed83370117834a -size 22823 +oid sha256:91501eba5ea59c9bde0547dbc70f77b1a426980f363530a8c301777cdb0893e4 +size 57501 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst index 6ee8bb514197..6e181bc7f8a6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:eee2df4781931c111d7a64d1eef680ea3cfebd7da9a707fc1c7a1f8a34c2bda3 -size 22912 +oid sha256:40cdacb25ad963478a93d27f725c5ce74dd0ee65ad261bbd6c051633acffa7a2 +size 57626 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst index aea78bcb2597..b984f9bc3971 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d07eb31323bf4a8f4ec2d1978e89c15e74859c0a7b2b41c035c93b93785eb98c -size 17905 +oid sha256:06c2f1df5aaabc69df6a1924c6093fafe4f31630546d44fa967a15c7ce68f1c2 +size 44744 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst index 3df91e3865b7..7c551f150c82 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7f34c5f958ca05bcf3ac6351e13ec90beac1d78a9c3c4955b420814fcc1b6b3f -size 17983 +oid sha256:1c5053b8d456839ff28f559ccc9be03da7953b83edcb28913a5b3fdc8ad2dc5b +size 44726 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst index 16592bb3f407..fcb093a8723e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a0ba5e8274e4f7f61b2fd2b05f67fda765f4664964599845b3521d862c6ecbf7 -size 23568 +oid sha256:e4a431cc1674aed1a89e5d98fe218ef8265213c122d567cf2e12bc1f4fe694cc +size 60272 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst index 3172e7645396..d309dba4f58a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:36f14cae63a3ccde17e7d03f353e80b726ecf95331dc0d664d6847db7d1a3944 -size 23669 +oid sha256:140f9d6d524a51e257cddba8f85b71891b2bac244ae775257d0a2b79ef05eb4f +size 60391 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst index 69bb7aa86510..94b45f57d422 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bf24be8be82ccec5f93e131a2fd05ba7129c19976e359ba2586a5cac17de1711 -size 18835 +oid sha256:4417fbcd9dc536932987fa26d887e9680145994a6b7893b8d797805347d66912 +size 47829 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst index 1974967b3d37..85d0e3f39e81 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E2m1E4m3_castE4m3_Fp32_Ab32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm103a.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2730349ca416591ebaa0ecf5a8bf5e5e5dd2fbf1d05cabd8929c3fc01d914b31 -size 18882 +oid sha256:1f8c6b14b60fc5b0b2ed8c374b1a06b4665fe7b0b58da26f898656402ff7223b +size 47752 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f.cubin.tar.zst index 8d5f84af471b..50c9b97c3a7b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:38591b833fe8f3b8fbe5960562ffd02205db98a3d3c92e17b1baf8b8073f0346 -size 21214 +oid sha256:c770cb2fc3b56d26fb645688cb67418a7c89482378cf6e0ef676c31558b428c0 +size 60847 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst index 0359f2a414b5..33b60200f297 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cdfc10e4d9ba2c3623fab8f34fd0cf1da062fb018b3b5fd3ec30563713081ae6 -size 18396 +oid sha256:320093858ecec5d28462d32a1f4893527ba847a783af540dda142007a14732b8 +size 48835 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst index 745ed90adfe5..0321ae7c9f14 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d66d26bca254769d161a8426874720ff1341e89c72f50953838256805ad1f4f7 -size 21427 +oid sha256:231b01bf30b88b4864c3945721b5ba10c958a5ccf0602fdb662253d9553c87e8 +size 56720 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128u2_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128u2_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f.cubin.tar.zst index b3be18d9d83b..f04403da2533 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128u2_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128u2_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f963c769723022f9ea8c83aebfea304a40046c8e3e6962d09b7844e7e92708a4 -size 21634 +oid sha256:72c1b3a269363769155ed7f56293793abd6e98ca278a66707d64603c742cbe63 +size 62975 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128u2_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128u2_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst index 60a477151e8e..e1538fe18941 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128u2_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128u2_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4b5607d41bc9a594c02e6f78d9506508b340d5ec1a19b1388ceaaaf25ec865c1 -size 18769 +oid sha256:ee73d3a65fceb7b9f311cde6559491a025d0526d831b6ae7953df9541eace06b +size 50672 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128u2_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128u2_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst index 35b833b422a6..e60bb6e6dd72 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128u2_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x128x128u2_s8_et128x64_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:90e0fa265221708e88cb1007778eb2d75059f1fd2484829ac1dd74a741efd539 -size 21758 +oid sha256:c863fda20e6d2fa96cfabf8401b93c1ec969f99afb2e82682f4d12db44b29f06 +size 58662 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index bcc9084b983d..004c42dcd632 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:82ea2b00a6c6640f2a74675a5584bcad82a38473e0d39960c0c9ae8b240aea92 -size 13818 +oid sha256:820ed3e29b317d93dfd7e966548c301c59864cb7e76a761cbc8bf7f37fe25379 +size 34896 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 9f40fa0ff660..bed77f9df085 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:35fc1794825338c273d2de98d240ae7df17f40c4591015331aec3b250177f901 -size 12992 +oid sha256:33e1cf3492ffc663c16711a36ca235ed3b30f67eae4841e7f80e710638ef1668 +size 32139 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 0a560ae706bb..e062484c62c9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8270911cd108db3d1f3876624fed967a31e51c72610b4bee0b8ebb507d9588b3 -size 13414 +oid sha256:4d254c32dd877d6ebf67b02c670bb5cfba1dab22d3aa1c76bd09c369e461a9fc +size 33197 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index c9b03fb68183..58d44ad597af 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a9ec62927bbc29fb6323e856881a9df67dbd4f3818b8169458024701371d0e26 -size 9778 +oid sha256:9000ff2e48af5f4e459cf6c364f6d3f9f4a9e68778ba417c1e23d5eb163877d4 +size 24835 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 90d027b1010d..b9629ad24ec8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5d23fa22cd48c76f58445bc2886ddd5206e2a78e91bc8f5390795877bd2989f3 -size 8986 +oid sha256:0d7c99fbe2edde9d185098b820ef94bc57441741392c6be019c60982d5800b56 +size 22487 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index c6fe6e8bd1e6..4f0b30c25f40 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:48b9189ec57039f0737bbc15824e131282643466a6b3590b5497d719683ae06a -size 9480 +oid sha256:913687777e0643044d6d60fa30388f765d1400823209da92fbee6a84a9a03b95 +size 23553 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index e07b5ca195cb..6b9bf78d591a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3437bc7b2be86fc9107e092427a722d81b35d23f7824bad027331d15d15a4f0e -size 14265 +oid sha256:f2b6fd56eacb023d06283f4547ed8194375afacb8f17fb0d7564ccdbcdcdaf50 +size 37294 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index cd2a24471c41..91948572a09d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6e83adf1a340d2a6502f32bdaed6cae3de16316c19de1f0ed8aa9604488eae78 -size 13561 +oid sha256:0ffc602d74c26bfbe784fb20c359957507e5dcdd0d301a12001ba401eda781ac +size 34678 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index a1c7f4319287..23e58ad1c49f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f8c9ec9f0d04f7c0f20ad8d8f5b7cdd0960ea3d419f5bd8a7940728468250d62 -size 14027 +oid sha256:8764858e8791f6c79d3fff5dd929c8d5b29d02ba9030a4a8f3c5ef3f74857a45 +size 35886 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 8225dd332a38..63e7a7d88faf 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ab496a6b48ec056deb4e018b06cf8cf814138bbca38db984466010457648af00 -size 10314 +oid sha256:c2ef6ab896481cec1696bd4c70b3a740029cdcb465feb5d08019d366fa2a8a0d +size 27335 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 51cd86f2dda8..a2367553e4b0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7a295e2f9eb1ae8278daa51bcf99bc821ca2aadb24244341a2c3511b8a8514d5 -size 9539 +oid sha256:9c8b278b4fe8622de149eb79ebd444c4ffd10bde39ea5f005683c5e693c44553 +size 24869 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 7e42970476f8..c18be0abeb0b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x16x256u2_s6_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9a53768db6cd9c46022c31c242504f6a591a7a072e4bb9668c7326ec20a963f6 -size 10049 +oid sha256:1a74585c7f24c40ba5dfd8d2348aa6b57ca06d261d558c1328a455075434062d +size 26072 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f.cubin.tar.zst index 02de22cc441c..4cd6f7fe295f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:58d6bd88b0c74b73d8a43eb193af63c39be98bd39cee12a5671b5546ab93999e -size 19684 +oid sha256:a78c710a449525cc425c2ee9027fd014f31ccdc29bbccaacf63bb85f4de23e67 +size 68766 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst index 5993d96ea1c9..9786175d673f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:412f9ded846277b33e4277e91a148062efcab780f8710d1349f4b891eebe05a2 -size 17266 +oid sha256:21a15b8804498600077eb99f1171fe22baa03e830e78b0ed3b5580a77d37bf1d +size 53223 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst index f170a94d0af3..5fe9c58627e1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:9662475779d9ff4d81eb736e6aa8efcdbc865d7205d714e080dd6499f83dae87 -size 18474 +oid sha256:baecadab3f6925ca9e316fa2e798adfaef402d6dcc8929fd6419504ec933f1b2 +size 60820 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128u2_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128u2_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f.cubin.tar.zst index 5be0186a1d16..b9b3f1c8d48e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128u2_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128u2_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:464d0ae4affb7b8b1639b160c9ef893e0de478f110514ea7dc64ea223259b64c -size 20107 +oid sha256:54512b47f6a1c476a3f90447bc2a5f60065c3b1dedca605c8a5db9125ba7decb +size 70593 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128u2_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128u2_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst index 6ee59c1c4d54..ebb452820aae 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128u2_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128u2_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2ccbbcff7cda9aafc96871d3ecb9ae5cfaf04fe263e39d9e70459d24f18e4b5b -size 17672 +oid sha256:e646973edd99605c03815bf232565be0e1dd3ebe28151fd18ead3ef5a888dca8 +size 55128 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128u2_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128u2_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst index 2f39ed29e82f..f7404e3efcb7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128u2_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x192x128u2_s7_et128x32_m256x192x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:39933c42538a05b5f001d9d706632a828b0b2e190f213c230843f3f3d1040c28 -size 18889 +oid sha256:3e428c480307e29cfae09c171749cebf86814b6ee0d6b5d5eff1355faf2d6e81 +size 63003 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_eW8_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_eW8_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f.cubin.tar.zst index 7252e3712657..0dfe13cca20d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_eW8_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_eW8_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b601508ebd72e265e3dc082696578fc7b3d4e7df63f824966f5badf2e999f486 -size 20630 +oid sha256:a8a1ee6aae038f3b8564fd657499adc8dcd2228595daec772e4337c3ccc57d90 +size 60301 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_eW8_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_eW8_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst index d874422f4d77..55e61cd06896 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_eW8_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_eW8_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:37d532cf994cdd8e954e96d0c6be4b3db66c47bc2b7e4db1a66ff7fdca87b4a9 -size 17979 +oid sha256:50bd7e45a6e30913b1648fb3962d6dea8b2c1525f3b16a557ffc4daf7fbc1dea +size 49252 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_eW8_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_eW8_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst index 701b01a98fc7..89735a135fd9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_eW8_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_eW8_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8ad799ac6ed15e6c169f94e3f27444dabb4979e4a2b7b5ac236cb87a93a7760a -size 21130 +oid sha256:e1ebf34ffd6cb9d042d828cbb725f8d7d07c54e675feac805b7ee002237bd937 +size 58528 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128u2_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_eW8_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128u2_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_eW8_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f.cubin.tar.zst index b8e6b1b442da..6f1cfa925260 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128u2_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_eW8_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128u2_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_eW8_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:82763e46663dfd4817b7e1f41b2d59fcc224f776307b453a178f82839e5df11e -size 21103 +oid sha256:fc974d06f541c475f3512b65bed9f7de4fe3b7b60c66d8db514dd30462bb7059 +size 62052 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128u2_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_eW8_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128u2_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_eW8_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst index 7fdd0d325715..0e7c3bbc7050 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128u2_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_eW8_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128u2_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_eW8_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:81a9671daef35bd6ede21293f7722ac97f93602d85013dfe70240693866b28e7 -size 18492 +oid sha256:11f109b6eb58124cc1838b4d8609718c247aa5b4995bba8f1c48147f98656968 +size 50971 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128u2_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_eW8_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128u2_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_eW8_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst index 20b8295af564..0494a6842201 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128u2_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_eW8_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x256x128u2_s6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_eW8_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c115de57a5679fb385f506d14dee8f0b272fa986bd1c08714131b60fe9492e50 -size 21523 +oid sha256:722d9966822f2168d212a56ecb1437bc1cfb7bc0239a7d4d6edaddce37802fa9 +size 60346 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 7aa79925f280..0303f63d7609 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:158610dfa83f9f7466636ecd8b18928aafde464a5b543f231fbf42d7d4a7ae49 -size 15098 +oid sha256:6ed4adbab5d81fccd8969d029f220528947ec26274dc24715649372e15b78be6 +size 38829 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 92ee6c95cea8..ec72fb2ac90f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:aad170393e2a8779c40a90810cae9e37ddc7880172b96f852963b65356b1a53e -size 14000 +oid sha256:abba265e5facdf4a0d90ae478b628abc1f8ad3550d6ed68cf9fb27496b2415ae +size 34759 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 12c3c90743b6..8cfbac20263c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3ec7f0d31349b7456fe3b2578c3b2664c1b6f3a688ad4aa97f485b50a1d49b8a -size 14887 +oid sha256:467d14b3c8751bbadec5d3b684dd6cc1da30bf851ddcd0e4a0a17a7901ce9bd8 +size 36922 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 1dd66f57e5d0..e4c5be1b04fa 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b4687210bd0162f3e73565faa1275a3bc32b79942a3141299f66a4d0da89f8a0 -size 11001 +oid sha256:d8de28892fadb41d62db25280d851cdafbcfd6bb5487aa2ced25f3a28a88a9bf +size 28917 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index bea04dc44807..f5e69c4c0e6e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f64e2b006bf395d68ecc5849b556c9dd5085395b51f4aa79e2e3dda13a4bbfbd -size 9993 +oid sha256:e5c39972758b8bd549f5a486592930ad9b49b82087dcd8eaf7192b1ac2059905 +size 25230 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index f6019f4f957f..69d0dceacf97 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3f35f46eb2925117eaa2ec1c634318417dd4baadf42f04ba9f9409cc956a0d49 -size 10802 +oid sha256:d7ea5b89db1b6ba4733d794d10a7e8f1f2b9bd2c1315bdbfea7ae4fc8bc9b25b +size 27330 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 93f24f7c072a..611a72708d0a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1e0a8127b941a516c49326ccecb2e9b5ec4d03c80bc433eaba6ac2f098d4eb56 -size 15581 +oid sha256:e610edbb21361049d76ee4aa6e9b592574d3e0abbc0399ba6232968bfe40fca3 +size 41154 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index dc767db0aef2..6873edd1a222 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0ee46b82b66674201e3671e6597a22229c74c9825e296966c3e0727f421953d7 -size 14489 +oid sha256:c1bcfd03a025b2350ca5a8b1eb816e0a9848db010dca40b2a2e2a5d1878ad15f +size 37392 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 0327f8aa52c1..eaa527374c9b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:815c7bc639f57ff064047a1cc8c773efd9b815e6088173b316263fe04bd4a0bd -size 15416 +oid sha256:50ff4dd8808247667d5b6c16bb165971a2a55bf0fc0b3a2177609b1ae2d733a5 +size 39563 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index f4806937c0d7..f415e032d02e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:857270901aa74ae438c9700b4a4b48c2e08c0b55361fd82cd726af1dffbfab5b -size 11555 +oid sha256:8efaf5c80fc10561d7aeb93ae169da0c1a38cbdd383ba563035889922e64cb30 +size 31303 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 37ea9476de3b..0c0413944602 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7449c8e11a220d2ae423d19a198f9df68f1025681953fda646b5e2e0f7c4dc5d -size 10592 +oid sha256:7742a27af5e2f0afe2133c32dd55f0ddd1b1c27e19fd5eb8e641bbe392845544 +size 27698 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index df2deec71a90..702f482cd2eb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6390411604c89476e6a798c0cbba24c110649e007b18177bdc615d9563066ed9 -size 11433 +oid sha256:0f71558135906f8fad77c90822cbd97d79a872943a7f790357864a4e0c71e6e5 +size 29880 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f.cubin.tar.zst index fbcb1414897e..950b8d35debd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:36a13cb9d9af28eb0a4aec8cfe200c94b6482baaebfdaf78df964590cb61f128 -size 18606 +oid sha256:1abe9a243da7f747000487cfc51eeedbc98826317a5d1deeb1f442601c689bbc +size 48910 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst index 512471a2ed9f..28998846f129 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:55c69ce1482cac672f4c41b292722d3515734663d41d80d1f41a4815b56cded9 -size 16407 +oid sha256:6795c4e733cc21d057d0437544952df8e4128f758653e2edb2b5d22831bfdfe4 +size 41904 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst index ed595663513b..9d85f1586741 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7646d1535455d2d1387676bcef1dba4b6d06966c48460cc1216333ef970cb99b -size 18065 +oid sha256:7c33b604ffd199977e54fadff6c3807b335415bd3101762594c709555f64a409 +size 46291 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256u2_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256u2_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f.cubin.tar.zst index 98aee376a0ca..c3af2cd186ea 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256u2_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256u2_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_lbW8_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:49ebeb00c7d8033181bf6f22c5e1ab56dad737361841fc73832890e98152e9cc -size 19200 +oid sha256:639110f343124ae493ca797c73074bd7bcc2f6ea4a69c816ff2b180681967caf +size 51163 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256u2_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256u2_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst index d42fc980526f..b2dc4f30c1b5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256u2_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256u2_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a83363b31251d02ba3944606733adefa4fccfed67c8659c766d4eba59ed90ad7 -size 16910 +oid sha256:9b4400fe2db9d6aa0185734077737c8e42af4c40ae5b1de8a3c3f095b1f4311e +size 44321 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256u2_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256u2_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst index 81ab4fc6316b..d3c5cab77146 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256u2_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x64x256u2_s5_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_lbW8_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bb562e7e4f29a52b8f770bc987b25539243cda2eb7f9c532ecab2ebab733d037 -size 18541 +oid sha256:443cf674001766bbf9ba5a3c4533716364fd8fdd255cac5c2faf37a343241bb7 +size 48787 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index c623a932e97f..ef7f4bee4f1b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:7cf6be7643df71674a8f2b28ff1d583c08911c53f5754cc8b96c3481f331a2cc -size 13337 +oid sha256:bfdd10653e425347f62044917f1ec6d49811c0b5a4f24ee592477c959c08e746 +size 33086 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index a2833726d41b..774ab7fe0c89 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6284a5eda974271bd8ddbc5e1e44efccf65d2bf786331ea99f0b72d9da51cd3e -size 12762 +oid sha256:cf321ee3b4570ad41944daaf83f8716f808f8f90c93a5eb6e195587090d79713 +size 31309 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 6177bbd5de5a..a415e25d67cc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f8e61c22e87c5cb5d2bbb2d173028c08d2bbc47a908abba53ec5cb8c3524f295 -size 12931 +oid sha256:913ad2f59866aa17bf0684c3d30553fa4f1bd963e1d6ebed4e4120816ee2c609 +size 31796 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 6ee7ae3199da..bf0c2a65ecac 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3edad0b06cfb03ff974e4a5e3872bf267ed6801b4e90445501309bddfe09ddc2 -size 9109 +oid sha256:1990ee95c9fcf3d2af852e3a0cdd066f5f7f01368479db9fb837f6cb7517f714 +size 22396 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index fa2c3d93c25b..1fa581acf2f7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:974fd7e2a8f55f016a06b5d18ca03c7b63b1ac631521dec67a019070cffe00e0 -size 8567 +oid sha256:4156f6e9bcf3b54e658267cdcefe0818757dc65fee915f0e8984567f8a301d4f +size 20579 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 95f38b63e53b..c167aa312844 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:164a6c57f9a60ed3419b4363686d1e5b0ebf7228ef9d79fcdb97a65bbb18ed3a -size 8791 +oid sha256:25e51f5fc4b9b76fb93edc9616868c1c5caa6af1f083fe4792070eba9ab6b1dc +size 21294 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index f951d8ef4a12..900dabe30803 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6794a17241f5d76980ef04bb315926646d4fa1bb278df6d0a7fcbbd47ca2053b -size 13866 +oid sha256:536b0511225e6a83c957b78ad83a6735258327e91e6056a41ed4eb8e1ce762d6 +size 35460 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 29c604703943..89df66772fdf 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:713aaca1385cda0539da76e6d736048c137ec0b2c8e2b362c5c03f2175a47380 -size 13256 +oid sha256:528fe92d65e0cde11e109bd0441f0880025856c485d229df5c1ef97277878171 +size 33563 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 94adf14f8348..e15a66478fea 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8bedb03a4f2dbeae26a8c680f39ee8e8990bc9a4f5e2ed19cc37b3b1cef8c452 -size 13472 +oid sha256:96f197988eb8206661302186fe61c09b7ee69b3ae6a6216d9fa5c9be6b4ba1dd +size 34097 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index e20bf9c00ba0..c4910370d72d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_tmaSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:627068a7bdba07adb7ea2c4424ef46ec2799e6674d41c26b654a0e3377816695 -size 9678 +oid sha256:a7ec882681476603569d329f28850fac201b376e75364b19bd6ad88cbd10b8de +size 24633 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index 32e33e2d5fc9..68d8e72dbcf0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e812e0d4474e68fa05e9763f2c60a045884961b2ac81a697b712979b4670edab -size 9135 +oid sha256:5a2649b06be114b33ed6a9ee820ede50f1d4abde5456ce0fb8aa2072aa4f3174 +size 22835 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst index fb5dbffcff2e..42c1e14594ae 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_BtokBfloat16_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_silu_bN_tma_tmaSf_rgTma_clmp_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8164376d23ae7f8537abc132005a2fad2c6b011b523ffbcb58067ac3b0971037 -size 9379 +oid sha256:ec3623c39477c7b4e6461c4578d2c088468dd677fc9179caf2cd85d3b7e5f80d +size 23530 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x128x128_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x128x128_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f.cubin.tar.zst index 4963f2920b0f..8b68f1fd5ff1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x128x128_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x128x128_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cd974b62244ad6d6bd0a0ba44d89fd5bbe6da82142b722b45d63f4d515e46d7d -size 32721 +oid sha256:9182d47ea47a4e75089cfccd7261b8d0e5741d04d618b63e3781f236b1d6e866 +size 92493 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x128x128_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x128x128_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f.cubin.tar.zst index 6ddee28fa1b4..35b8b6dfef33 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x128x128_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x128x128_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d27834057b20c3d8c6929cabef2f034df25ef8e26c16ca835a2f0ffc0019d106 -size 27066 +oid sha256:d739bc56b2e0a03d63f5f4c48b5f34528b3eba8501c21b32051a8cc403fe6367 +size 78865 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x128x128u2_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x128x128u2_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f.cubin.tar.zst index 347459ac741a..b7aa7ff23298 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x128x128u2_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x128x128u2_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:daea56cf141ef311a2adf40a9cfdcdd219d100c9ce5ebcf772706788ab22f8c1 -size 33852 +oid sha256:593a961224774d9d511b8df883ff238ad2a01a1aa66afaed52de21baa87a21f3 +size 95786 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x128x128u2_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x128x128u2_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f.cubin.tar.zst index 0853d480b75b..531d86e6a711 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x128x128u2_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x128x128u2_s5_et64x128_m64x128x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6dcc67e15cee644eb8551c40ba48b503183ed1b1d2713b9ebd13ad858cc28423 -size 27848 +oid sha256:1b413edd55642e60d116d498c3dd7aef1ae6d6544e1ff65b1e2351f1ebf41688 +size 81561 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x16x128_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x16x128_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst index 058070af5256..456d7f8ccb6a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x16x128_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x16x128_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5b24c5f7491a5b668a3dbd07bbc43cf8154ceb891d602d836f599ac333642852 -size 18535 +oid sha256:df53d101bb379263b8a163dfa44ffde272323cf54521e70bbf530f0c2ec8234f +size 50202 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x16x128_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x16x128_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst index ced532cc06dc..1f77a385aefb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x16x128_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x16x128_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8094e2a7f07b9a303e2a49e5bf7fd0c061f8cf3364fd6c84a5c48b53fd62f3ce -size 14369 +oid sha256:da0c248419b7f827bcd5cc17f6af72952bd60cf671cb348ba0bb032af3b417e8 +size 38797 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x16x128u2_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x16x128u2_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst index 69d3acabb0d8..fbfd480ed7bc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x16x128u2_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x16x128u2_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ef56a38ba02072e940eb44f52b25e9c339f8095c6a5686056903e5398fb8ba23 -size 19968 +oid sha256:f8466b71a395969a139f0e357676b203c77e6ce924add551c61c55b6ad6f149c +size 53250 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x16x128u2_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x16x128u2_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst index c1c419c54814..25cc6b269450 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x16x128u2_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x16x128u2_s6_et64x16_m64x16x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d67fc0c10b0ec529887ef088fd6f6782a8dd36c6c48e45404f46ce8d99acafd3 -size 14943 +oid sha256:c8816b432404375ece2e6e482f3af51d269f19d8139c5adf701ee3fc0fd72029 +size 41714 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x32x128_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x32x128_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst index bd393cf77488..46540d2a443a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x32x128_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x32x128_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e86366d61fe0a0cebb2187d2861e3f5bfaf7ca3f2314bd140a3e9d01f1581a8d -size 21062 +oid sha256:4344c61cdb81793401e4b3f539d65e41170b99fed748f51b4654d73707fa7609 +size 57176 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x32x128_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x32x128_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst index 5dc9b10cf278..b2b0fb66ff4a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x32x128_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x32x128_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b3634f59f47b038335438c82f61df9fdbb554b3f0a9dc0bb4fa3a168c1422ba9 -size 16571 +oid sha256:231665dee8b6b917dfceac8a0eae6442bb310757cd1a0f47723ddc1fce09fa28 +size 46130 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x32x128u2_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x32x128u2_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst index b38c40c3dd57..b5bbe09fcf62 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x32x128u2_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x32x128u2_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6d9b4d551c111d8c7ae3c244f1064eb4c31fe5c4dc0d2a58322a65157e91bfc5 -size 22376 +oid sha256:e12e43c19822f74955641ccd0cb0d399f70d37b82a491a2c277a8d0db009e02a +size 60419 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x32x128u2_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x32x128u2_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst index 1e2ec0a2720b..d4a982a40ef8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x32x128u2_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x32x128u2_s6_et64x32_m64x32x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:325435ecb42640055942c3f60acd6f5df8257b88f4a921229d66502e17682e86 -size 17208 +oid sha256:e5e0e66e2e7eee86d2cca8353f30679d121aed4d8a0faad68447dd1e16a141ad +size 48723 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x64x128_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x64x128_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f.cubin.tar.zst index 9d26612f7252..f394c589ad49 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x64x128_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x64x128_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:146223351b0eb6b8d144eb2307ab8f7c6638b78a8e8f482ad970f0bb0be9f700 -size 24306 +oid sha256:ed7d142dd7128627339ae1f44602ea51b34bd62a6aaa9cccb7fe08ebe1da786a +size 68433 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x64x128_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x64x128_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f.cubin.tar.zst index 89f35105b799..b3bb231683aa 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x64x128_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x64x128_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3cdefd8f0a9ae697b4b87055128d482a68e664e9d4624bc8a0bd3fcb52e6cd77 -size 19479 +oid sha256:522584de50b8a23440a45883d156eef0ce111adc9871561520c4b32358ad66ef +size 56473 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x64x128u2_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x64x128u2_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f.cubin.tar.zst index 55d353e0a7e9..7a50b8f31893 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x64x128u2_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x64x128u2_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8b5dcca4aff2771c32ede697d3fcad40719792d5824360f5db09c350534fa102 -size 25428 +oid sha256:de6d5244a8a64fc6108ac315ad997e22dacf191a83e29009d0ec0483e8745492 +size 71669 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x64x128u2_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x64x128u2_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f.cubin.tar.zst index fe5a7c7361b0..8351c7448466 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x64x128u2_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x64x128u2_s6_et64x64_m64x64x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW4_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f2ead783be273f6c8ef93afa179e7d49569899bd0bff61372348b4cde991078e -size 20136 +oid sha256:5ec3c7ac5b97d9d0ae1129179e37378fd30822dba230621f57d3f79cb4da6a60 +size 59195 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x8x128_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x8x128_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst index 10cc2045c219..43e7fed65f2f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x8x128_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x8x128_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a693ef9177e94d1333d2ff2ccb31eff4010fc66dd27523b6906209cc11416b39 -size 12251 +oid sha256:714b744e624ffdc102ffc1afdf61d25c7a1143ad15a97184f823d0b59575d1d5 +size 33080 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x8x128_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x8x128_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst index 1a6538a21ddc..a5650ce3c140 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x8x128_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x8x128_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:42b481635ec6ef10b4575233b54e7a82ed1f7da78a10a55a3fb914a6542ac228 -size 17291 +oid sha256:aa03fb09c366dfa0ecf5d492a032b69a10e3e8b99411809f5d99fdf37a67c492 +size 46440 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x8x128_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x8x128_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst index 78d60f545f09..f94ed165f2f1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x8x128_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x8x128_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fbde5b4ae5f37f4ea18f5d34c7c46a5fa80d498f3c498db1494417b9c71a873e -size 13020 +oid sha256:f5e5122576af1624acb3e91440d035635beb84aa21fe12401c640b79ad6d3e82 +size 34375 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x8x128u2_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x8x128u2_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst index 01760ad2f838..57930adab7a5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x8x128u2_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x8x128u2_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8d1ca85513293a91528013d86d2683f98a1edf9f65e13e8710bd66d4655700e5 -size 12887 +oid sha256:67c3c6991f7a1269d38a10a8b5769366f5eee5d47903f3c95ff0a5e0af557d38 +size 35635 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x8x128u2_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x8x128u2_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst index ac7a58350d74..5d54970b04eb 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x8x128u2_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x8x128u2_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schPd4x2x2x3_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5e4d21a1396a1b77863994f65c18fce48e0dd3af5837e2811ff41dcf77852c2e -size 18512 +oid sha256:71e742707b6ea8999a720e84166e581a8dd2281d392820f32a392832efe75fe6 +size 49786 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x8x128u2_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x8x128u2_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst index 8a7e4c93d8cf..bcb46b0d6606 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x8x128u2_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x8x128u2_s8_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_tma_tmaSf_rgTma_clmp_lbW2_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:adde2eecc316bb79e58808a2b36fae48e8552795ffa0fedf6674274e0a4e1938 -size 13598 +oid sha256:d28ed6a96fa9c34365c0a9b6b08b5b21281e52cb035ba31267e85ed139f8b47e +size 36973 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst index 2c9a636a813e..e5223bce3bd5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:93a766f96643dd5c004c895b1bf24a3e90531f70f6f960296911fee449cba8dd -size 8301 +oid sha256:024ce0efdbe7a7a50ba4129dd50a9289a8d8b6108ee950212d5f3a224418cac2 +size 21331 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f.cubin.tar.zst index 58c0a4f7d598..3398cd3d32d8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a1147dfcd88b68b2b6b45e78d4a6ebffaa58148b336b673328b3e81bc95ff90d -size 8372 +oid sha256:d68b97071957dcd913d52d3abf1d598ecf729a05ac5f0ae5255b23023c71af46 +size 21809 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst index 86a2caee7411..ff7fbe00455f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:55d605d6cb4791e5c78e43c032008118d230e4ae029c84e0b37eb8bc15151b64 -size 9026 +oid sha256:5d45f66d7c24d0735aea327c4774cff5349123e3154a459597c640191fe5565f +size 24866 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f.cubin.tar.zst index bffad70342f4..9bb47739cf7e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:624309721937c007abf52f4c3538530e9b5b292531abe5e3113f483fb836dd85 -size 9126 +oid sha256:e23b5bebf5ff620a8e626ba70fe59c95200933412974c3da4a68f9ae5f881ca5 +size 25418 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index c7f223329b6b..15e02c876f9b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3e201847ac9b8b5e9a2f0c4a3fbd4fe6ad7758e29dc542f1e3b97273bb9c8559 -size 17313 +oid sha256:5202253a3114b3ceae488e7d9707a02091e51d03935b14860ccac09d42a3cb99 +size 44654 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index dfa38514ee63..ed5f012c3283 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d23ee2e2bdf2691735684dfb340a47de3d1b048f0422f9f6cbbc16caac00fa5b -size 13181 +oid sha256:d51bc424121245c2d9ed34a4471274aad8a813c0ab8dfbe50f41f3391ad1bbbf +size 32326 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 8ff0e4082cba..cb2f1f6ff983 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1f7c8630d6cfd73c560516579eb555273f1159a88d4484440eee266a6d46c5e7 -size 18142 +oid sha256:3df4d045ec1fd91da6e291e6d8f5d3ffe20f1bccde1ff07d82ea14628df3d5e5 +size 48133 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 908491f622c0..311262cfac62 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f7956363d4d7ec3129f0ec01ace853f6eab4c9133aa0e25c68807d4f182e9bbc -size 14049 +oid sha256:ae7124206f9ccb4415ae195fceddfda791e4e5ad850b676d93acda528a18c7ff +size 36017 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 8f4b2af05b40..7d2b38271b39 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bd47fd305d6e8a6bfb6bec6734f19ac475d21c0525d9988081976e35c5a0d655 -size 18987 +oid sha256:b18159abaee52bdd20c3cb81b8a9d7683273744f4798c9a637408e1c89bc4346 +size 48503 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 4ab6754260b3..2ea2ada53072 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:97949ecac41fc2adfb5d38e8a9cdeb1f0e079f96b8c7ca43162af61b9066fa11 -size 14281 +oid sha256:ff065ff7e31b8c02faaaa1a0146f9d08f36c22ffab14a5894f6f14fa258a774d +size 36097 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index fec55a9636ed..ce516e83a1c3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3a0879c267dfe4646a521851daf41648d918143f10b9e8d275b9f89e613b80ab -size 19738 +oid sha256:c64ceffa1a051d6b5a5802825e1cff1fbe60862ce24afbf94af87921d81762ce +size 51824 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index d1f62f522789..dfcac141ce5d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1e07236c0680d7a3c9b876e8c6f431c7b2bd932bbedac9124ceab93e1f58cbc8 -size 15146 +oid sha256:8d6351d78d7f634d53290946d780159d43db1d34fbcfed36a8436c82485f3e3c +size 39917 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 555ff16eb74f..d0fb7cfca89e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6b5115346c1c715f64ea481b2c611976e889fa62682e5d382c8627fbdfaab8b9 -size 20570 +oid sha256:d5db29f0eec69753ee7d360eb72207290bfcb25cadf05e558956185bedecfd6f +size 54746 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index fe42ce273e25..d10dc4b9b576 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:5b67133564e8b1bd8e11ae6aa248207b4e0fbe8c932314d36c0a05e506e22415 -size 16267 +oid sha256:c852ea2538802cf5fa7b60230494bd9d58ef64a1483f4074f24905d58f525272 +size 43128 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 51408f01b4ee..f68a67609cb0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:598cf7285ec2081ff59381b8a17c7d39e3a7c2a35e16b84bf4171c1edad500f6 -size 21738 +oid sha256:88fff557e069e451967a2f544a4d17ae6d0058ec8fd6b4e508f085dab72bbc6e +size 58428 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 1a6d29e9c9c3..39f0e653e20b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x64x256u2_s4_et128x64_m128x64x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f8c2813f3e41d5f6823ce05a4d84b6fc867a4c7e01a79703d3cb4afd12170bdb -size 17176 +oid sha256:cd1f3bb4d74110d17ea815171976f1511987d5dcce2595e5367b663fbc9855bd +size 46752 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 78a0b6bcb26a..a6bf24983aae 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4e6f3a04b5e6dc54b2dc7bda5e8cbede6a1b348e4a02bcca8719a15b705d1b33 -size 16579 +oid sha256:032387376bebf2b09c4fa196167e67f7bc3ca9719c7aa24adfcf8858bad19a05 +size 42434 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 92d31329cc1b..d7d414c36be3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3712c70a8ee611dba8296aec9774d0e71efc20709b2ac50af0faf0c6ec625640 -size 12358 +oid sha256:9b0f6f8fd5e26be131fb97c0323651a915e7866f892f4f81a2a32ec33f0879ad +size 30367 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 7189d481bcae..158cc7fc8e39 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b3d82a02c6b8d6a071a4a1c97c05bb4bd689a69d852c2bca5ac249d85c6b260a -size 17493 +oid sha256:f06ad33869ab7c1fa9ec046c5b4ac867274b2fdfafc9b307655da1895088396a +size 46352 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 363efac67451..9ab2e0b341a2 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x256u2_s5_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:b09553becf5e16965866b9e341f85977bfe766e8a15b1dfbf939088d99815bd2 -size 13251 +oid sha256:d7b4b362c14044dec145b5f4913bf5a51c4fbfaf64033e70c26fbf5f066ddd0c +size 33755 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index a94406292cef..405281081357 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:a5b79656d817c9259ea7b6e98fb22c7e713c7aeb6f8742cb7b438a61f5eb7a4a -size 17060 +oid sha256:19a04d7565ee21c41a5f0ac24a43648504319021320902aa1aaeaed0ebf9fe11 +size 44505 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 49ab8e1dba07..dc36a8223311 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:bc743a04482cf8e861aadce86383b9c15e8674bba556cc4777fc03141dcd1146 -size 12927 +oid sha256:eeb1da383d722e2fb476a7c5177c70b3fa99a41fbfa9ed78091ecabba3d51071 +size 32332 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 019a662a1a53..6e6a8137790e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4c566a0ca8df99f03c2e1dc660d2cc00c4cd4cd55dffe316333288ebc2f37407 -size 18145 +oid sha256:6fb50f40ba94b353c75877a1f12a0603ee05737f0e34f6936a619ced991ed860 +size 48913 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index f9643d925c4d..8ba598687f84 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_E4m3_MxE2m1E4m3_castMxE4m3_Fp32_Ab32_Bb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:388359abcb1da972135005ad3e2b2db4e9cd4cce3618464493108f2f249eff25 -size 14067 +oid sha256:e0ed6c6f23a76597ce5c698e0704c41c94598cbeff6cd5fc6d1b959ac22fd051 +size 37445 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Fp16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Fp16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f.cubin.tar.zst index 527c9adfa491..feac1ad373ee 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Fp16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Fp16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:30d866e52ebb54d89bbdb47868b70148a1ddd75a7d6266b39b0b2cac93f93ca9 -size 12301 +oid sha256:1292d91f20b0d4f8409af2ae9364efe3a7dde09210908c24c9f54d24ad72855e +size 31828 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Fp16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512u2_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Fp16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512u2_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f.cubin.tar.zst index 93b0d32283c1..5c824412068d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Fp16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512u2_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Fp16_E2m1E2m1_Fp32_Ab16_Bb16_t128x8x512u2_s4_et128x8_m128x8x64_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_rgTma_clmp_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cd300aba8dc68ac0851a661f5009ba41c432241d7deaafca0dd4d7b3a636dd88 -size 13440 +oid sha256:a4a851763eec47cc6ce9496e6eb0b83e58a3bbe914bb943ed908b653e68cb0f9 +size 35682 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Fp16_E4m3E4m3_Fp32_t128x8x128_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Fp16_E4m3E4m3_Fp32_t128x8x128_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst index aae3f5459b0a..86b8349e67a7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Fp16_E4m3E4m3_Fp32_t128x8x128_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Fp16_E4m3E4m3_Fp32_t128x8x128_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1462f755c1c4e288aa26af4acb78f597bb21d903e97212cc1bd185aecb2507b7 -size 11104 +oid sha256:a4dc00383510c496d2a6b7402062556d81e48415252f4bf0871bcefb2e6057c6 +size 30053 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Fp16_E4m3E4m3_Fp32_t128x8x128u2_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Fp16_E4m3E4m3_Fp32_t128x8x128u2_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst index 594635122805..df4c805e92b5 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Fp16_E4m3E4m3_Fp32_t128x8x128u2_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Fp16_E4m3E4m3_Fp32_t128x8x128u2_s4_et64x8_m64x8x32_c1x1x1_rM_TN_transOut_noShfl_dsFp8_schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:01bf5ad59867084a19e2a06fb01042bafd12fde0032ba44d3b57057f1f907e40 -size 11786 +oid sha256:6e7756ea6466848062163ee07a9fe49ec5d56e1c301f7c15da860e08c624c690 +size 32546 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Fp16_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Fp16_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst index b88826f10676..d4a80bd67238 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Fp16_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Fp16_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c2d959879308e21675cb7013eb315779204c59e9d7d84334cb9e7fc580795526 -size 8277 +oid sha256:ded4f02645f73e78ae8213f15be2c9f66d9a8f5f83737e49985ac38f59c51f0a +size 21512 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Fp16_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Fp16_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f.cubin.tar.zst index dc9b18b61918..dd8479bee8b9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Fp16_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Fp16_E4m3E4m3_Fp32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:df770cf3e98e3199f149983179902387bbca38144be0b7944e5b268c5e31013a -size 8399 +oid sha256:dd6675b226060968f69b188c1f9c2bf4b99b74a4a3774dfe57d9fe7bb8876b1a +size 22036 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Fp16_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Fp16_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst index 08f7b603f074..311001b22837 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Fp16_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Fp16_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_rgTma_clmp_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:dd67552f759f0591feea74dafb9b329bc90c50561b2b843a37748db347c328ff -size 9014 +oid sha256:52909062183c21f72f9fc21f519b4968bc849093d42518c682b71fa5ac9be1dc +size 25073 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Fp16_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Fp16_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f.cubin.tar.zst index 64eef71ec4f9..dd0eed555764 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Fp16_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_Fp16_E4m3E4m3_Fp32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_relu2_bN_rgTma_clmp_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d1d30f28540b1335783c4f7a90017ab65a2313307b7a4cab457f807302d829f8 -size 9199 +oid sha256:8edb13ac7730d80a0cd8e480d65262bacca4b6c63a42f0c206a0a9d9f9bd7c69 +size 25672 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst new file mode 100644 index 000000000000..bdbe3a3391ca --- /dev/null +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:e931e93b4ac79d1d338bef933f6db2bf37c21feca456b8ca0649bf95ec7bbd7d +size 85176 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst index b1eed8de45b9..059bfd7bfa91 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f3e8fb0c6ddf08c5cfdeac476c2cb40192e3edfac447bd914246bc09bc0806e0 -size 28628 +oid sha256:d4d99d08259e9e43a59500d7ed3c5e6cb3229d54a3379c23903db4e941bb274e +size 85660 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7x7x7x7x1x7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_lsfbW4_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7x7x7x7x1x7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_lsfbW4_dynB_sm100f.cubin.tar.zst index 4b6d1a8c0857..54dc2431cc8e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7x7x7x7x1x7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_lsfbW4_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7x7x7x7x1x7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_lsfbW4_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:0927d2c610e1c6ade6fe92fa467abf0207ad8ac7e30d7785dbe2a436134a3b18 -size 26703 +oid sha256:6da9fa7f2d8c47868a1deef35bd0f2a6023ed7ed53963da8926336c28d8b1487 +size 80034 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7x7x7x7x1x7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_siTuGlu_lbW4_lsfbW4_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7x7x7x7x1x7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_siTuGlu_lbW4_lsfbW4_dynB_sm100f.cubin.tar.zst new file mode 100644 index 000000000000..2bc532bce969 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7x7x7x7x1x7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_siTuGlu_lbW4_lsfbW4_dynB_sm100f.cubin.tar.zst @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:d8d965f61e04f9c5d7af9b7e371e07f6536607cacbea9668bc470fe345d797f9 +size 79283 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128u2_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128u2_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst new file mode 100644 index 000000000000..e3f57fae580f --- /dev/null +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128u2_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:beb87dfeceb5981bccdf75380d61484bdcdd7afcc44f7389177c3a5c8899d441 +size 90088 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128u2_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128u2_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst index 764f324316a1..be9764bc7c90 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128u2_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x128u2_s7_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4c747d4e29ab172dab0dcaf0e6ebc3cd94556e0d9a207f3fed8225c5aec7f593 -size 29098 +oid sha256:89f837edbfe00b401c43ca170a06572e84cf42ad5e9e53547c0aa4e7674d4c1b +size 88375 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst new file mode 100644 index 000000000000..094fad66a4ba --- /dev/null +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:242726ebd5597dde8d6cbefaf6a1a8ca32ff43b8719a3119fa925c778477de47 +size 90536 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst index 68568e1e6f71..680e59d6a963 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cea3f961ff1c6837ccc4ebb46c45dd52d3b2b5edaf290e9c20c3b3694b7245ec -size 29275 +oid sha256:cd9c35ead5d44dcebe0f578e27ee224373d0c017c747e62fb7ab70116fc6200c +size 88875 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4x4x4x4x1x4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_lsfbW4_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4x4x4x4x1x4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_lsfbW4_dynB_sm100f.cubin.tar.zst index e9fe0ae0e380..4494a75920e8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4x4x4x4x1x4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_lsfbW4_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4x4x4x4x1x4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_swiGlu_lbW4_lsfbW4_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f6daba19a397efaa38b44cd37aff0084bf40afd3a764515dd1bc28f4d25acc22 -size 27384 +oid sha256:b5340ef04bf91adfcb7fc1c4d6c4457b09de4497dd248d67d08efec5a9376451 +size 83701 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4x4x4x4x1x4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_siTuGlu_lbW4_lsfbW4_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4x4x4x4x1x4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_siTuGlu_lbW4_lsfbW4_dynB_sm100f.cubin.tar.zst new file mode 100644 index 000000000000..f230e7c3f31d --- /dev/null +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4x4x4x4x1x4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_fCp_bN_ldgsts_ldgstsSf_rgTma_clmp_siTuGlu_lbW4_lsfbW4_dynB_sm100f.cubin.tar.zst @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:782ffef9bf6a287a4d03990d1cda9ee9dfaad79795fd0a0a5dad2326d3602f48 +size 84919 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256u2_s4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256u2_s4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst new file mode 100644 index 000000000000..ded57de5cd11 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256u2_s4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:c815a55cc2f6a496897f4a8a17dd336ef3cbeb47f8080ce11576f226f7599f5e +size 93778 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256u2_s4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256u2_s4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst index dad5bf5f431c..f416ae110df3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256u2_s4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x128x256u2_s4_et128x32_m256x128x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:59870c6fb20cc09791059ca780abe1050a9e7df9b00cced29d5859c033cbb7a7 -size 29357 +oid sha256:68817a65ab6334c7a68a320a967d7c46abcfa65e37524479818d4dd93a666d74 +size 92912 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst new file mode 100644 index 000000000000..a27fd5c634b7 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b41856c1585cc9b8b663e0d63460e22c672600c4be60d3cf6e89b7abab7e4c49 +size 52454 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index c8bcbca1c9d0..29a1f4e69923 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:116efaa2f3a11fadc2fc6bcc1a4c955a141c9115dd55467da90e710f16693896 -size 20790 +oid sha256:b3cbd74b6e68eff8e06cc43703b8d40276051e456b71f8c93193cee9d0e57eae +size 53156 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst new file mode 100644 index 000000000000..586fb527fd47 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:50aaaa2d7c5781bc3d9afa6219d5b02c6d663c8756946669c3df623736421459 +size 39585 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 0909f31e9d21..5be1ac984a5e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:cafd4d7cb33e94de631e66fdf971804abea4d41a9f18f4c65294bf1d189bb433 -size 15647 +oid sha256:86cdaa4528f071f7e3d3dac6950ea03f00cc16419c2557e9773f4207ba954cb1 +size 39818 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst new file mode 100644 index 000000000000..99b947c7a636 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:54ac7183d0da1d2fbe08b85c35c130845ec9dfb7848b90457973b54f85c54e28 +size 59871 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index a9df4c590d1e..1b7e5f050962 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:907bbaf9b156f85611743273b9d2f10fc5f1ed076c2578175fc5384104cf2127 -size 23672 +oid sha256:616cdb6b5e76e1a8355ea5fe513ab77bd06f391a0245bf4b7a4ad5966fa0a4a9 +size 60296 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst new file mode 100644 index 000000000000..ab76b75ee3b5 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:cb114a651ed49d79dda25fe46c441cb7614dc7a28b676f52b887b332a1911dfe +size 47253 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 00000f6f23d0..9bef8565625c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f6631d09d8f85dbb207e1c731aa3a903e0eea9a6a74a070cdb266d9d3ce4d253 -size 18815 +oid sha256:4fee5aa3f1b15a81bf543a96d5b984d7edc3a57449d9c88adc5a7ade8f931f81 +size 47608 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst new file mode 100644 index 000000000000..0a9b1a67aabb --- /dev/null +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:c7d787f1c3daea801e8d16bcd54c9dd31a160a41412f9260dd61390d012a5274 +size 56302 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 0064f2caaf6c..b307b75846cd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1a43756982be9230ef09fe61dc7d0a68b4f71b9ec0a39e3b73dab399861d0c81 -size 21559 +oid sha256:57636600d4328413e82ea4bc0dd15200eb5979305864ebbd3a6d3a02ff691275 +size 56404 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst new file mode 100644 index 000000000000..cec6f4b7834a --- /dev/null +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:c66d5ee51620bc36134e123f94e226afc5a732483883a9dec958d3a4de704c9f +size 43072 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index cba8bf3b8bc8..d77005edf9b4 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s5_et128x16_m128x16x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:41985f7404f00a247a65929608246aee6dee8d6497d1af5f9bcb34f1dcda3dd9 -size 16693 +oid sha256:41067543f7f95cccc8f103ea9f89d07d6720f9d8f03e678c317cd01ebab48e3a +size 43456 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst new file mode 100644 index 000000000000..75121cbfc256 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:0b02198db0613bf83ccd6b16f06374921c4a499c798b517b2c6cac97d2368af8 +size 63345 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 763f104f6d7d..c25b897a6112 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:fb77d86e57ca0f805321c7a449a01f4764e2f99b3f0fa9e49eaf69ff4400cc8e -size 24788 +oid sha256:4dfba0f0bf1906e4d1ae22ed29cda105ce5838f65b47f776046e9e28fd195441 +size 63684 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst new file mode 100644 index 000000000000..c15027dc421f --- /dev/null +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:38475bdae7c610540f2390f275ebf98a1ff2704ac2e503c23deb5f730073a7f9 +size 50813 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 9abec6e9e031..7cb239ad5539 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x16x256u2_s6_et128x16_m256x16x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2f173516fe5e41d62dd16c5f0c5f05c323e9bb4beee8036217cff65f6484e6a9 -size 19849 +oid sha256:894acd360252ab0da1f49ea82c71888d5b57bb712d24dcf3473d717876a7c5b6 +size 51189 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x256x128_s6x6x6x6x1x6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x256x128_s6x6x6x6x1x6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst index 8b481b6b2dd1..e4c7b58a4a19 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x256x128_s6x6x6x6x1x6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x256x128_s6x6x6x6x1x6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:e649d016837f27888d4a65fb8bf301fb5440a3f6123cec4138451e9356beb438 -size 38941 +oid sha256:eb190fd8b4b12acada0c9eec4c83492302f7c6dfaa45953a98121258eafeb2b4 +size 123701 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x256x128_s6x6x6x6x1x6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_fCp_tmOv_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x256x128_s6x6x6x6x1x6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_fCp_tmOv_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst new file mode 100644 index 000000000000..9df0705d1ee4 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x256x128_s6x6x6x6x1x6_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_fCp_tmOv_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:1aa5dc980c575e4d0779310e2a0e173c93371d69a09be88f218816a826abe595 +size 131787 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x256x256_s3x3x3x3x1x3_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x256x256_s3x3x3x3x1x3_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst index 8d511a5c0b69..ebbe83aeda4b 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x256x256_s3x3x3x3x1x3_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x256x256_s3x3x3x3x1x3_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_biasFp32M_fCp_tmOv_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:372501f37d057248ad26c285a7ad6d5e6bd119eff0dab49aa5ef548072226963 -size 41619 +oid sha256:09c22638a72e8b39a290c8f71be7d3620ebcfc8ed8e995b9a688cdd7ee15dc6f +size 125015 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x256x256_s3x3x3x3x1x3_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_fCp_tmOv_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x256x256_s3x3x3x3x1x3_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_fCp_tmOv_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst new file mode 100644 index 000000000000..dd82cde1699c --- /dev/null +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x256x256_s3x3x3x3x1x3_et128x64_m256x256x32_c2x1x1_rM_TN_transOut_schPdx3_fCp_tmOv_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW8_lsfbW4_dynB_sm100f.cubin.tar.zst @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:fee4c4c8e612def8ee2d910153e686c418b1ede9fb440a767ae9bee0ec1aa435 +size 136031 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst new file mode 100644 index 000000000000..db45b13bb49f --- /dev/null +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:c5fd6ae8460fde86646fec8c922ea54384db4ec978c05c075dcb2c306f9ed6c2 +size 59459 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index aec16a0dc959..ee9cf3853e89 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8638f7b687dd2e326140db0232902ab2cf771e545b6888e0134bbd7e4d02f77f -size 23216 +oid sha256:25e6e7f16c5c87d394f31eda2be216baa40d5214ca4629093d78b3dfcdb79a65 +size 59153 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst new file mode 100644 index 000000000000..80ab0accd65b --- /dev/null +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:9f739b7cebd82174a09e1d720b445089ef13dec3c9bc4e4089bd602d0c68728e +size 46314 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 03ca94d44e62..816d4b62a38d 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3e41b907dc5668ae1b943e03ba3a76a2a93e6a9b4bcf62973cd98bbb40ae45ff -size 18136 +oid sha256:295881343f19511e9ae87575b4faaef640be2a2051df57916131d10b09742bbb +size 46415 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst new file mode 100644 index 000000000000..575405c0f6e3 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:abe1d46847218db224154e25baf92689a669f603582a992b060fa51007d0395b +size 66711 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 1de47842452b..36545d0e12b3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:68285c6777e0cbae290f91cd7d2cfe0fc04a4c13df659761589dc130f9e24c2f -size 25766 +oid sha256:c1e1a7f3a7b1ba29f4bcd779ef50e972fafc4e79dde31c0ab71192793fa7d838 +size 66774 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst new file mode 100644 index 000000000000..2d6d6058b0be --- /dev/null +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:8794f3685691aa71597f4d77ea994146f459e034ad9f768fec5fb9d00610460f +size 53310 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 773d0a1c369d..892206e5650e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:91151befc1f8eaefa91515cdc23aec9f8e12e8214c06df4b38ab5c57fd71a25d -size 20678 +oid sha256:bbfca9bc509492d5b6450463cd6270fe1913669176ee9eb3e29593822b70f2d8 +size 53391 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst new file mode 100644 index 000000000000..a91f169105cf --- /dev/null +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:6abc3ffbba51418fa0f1e05e1d69dd406da39b98194db602306e758a602559f6 +size 63242 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index fb851bcd99e9..bc2b9337382c 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d15f3bf5a5c1e2a38a6f4c8575fb58d66fa28002930529337ec58d077ebe64f9 -size 23891 +oid sha256:dedf4f8b9688cc57c247e746952439e6cd5bc9039135a971e09f303e3fcefa5c +size 63032 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst new file mode 100644 index 000000000000..f90ced06d929 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:03a2d1679def061efa0939ca78440127e76e7440a14ec128d07c0ee88cb0b598 +size 49831 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index c3e828dec748..27f24378bf14 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f7fdca2450698347ffbf6f1d6df13f6952360e70dbc8bb1861db0c85b523e5b6 -size 19197 +oid sha256:c8da916f1f9b60a36778819ebe88e96d9325c3ea56468caa44990a79357690f9 +size 49956 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst new file mode 100644 index 000000000000..b5b77d1eb446 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:623daae40fdf3e4d8cba2d3c7ae4923149a8fc081694b7aeb17a59c4e8ade801 +size 70236 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index b3c885962c4d..32074e799798 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:83df8fabbba6b9eb7d5d413fb06f22f8866d3a365419ec7a4706d366a67279aa -size 26650 +oid sha256:f9bf61a065d821de47a58b149959f2a6acec10e9641c851b8e55ff852753c943 +size 70436 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst new file mode 100644 index 000000000000..c0c6286599fa --- /dev/null +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:95842461625f600a6c5b02248715b42b0ba76f4a921c2458bac63c80aee301b5 +size 56941 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index ca700a2961b9..311be2095b25 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x32x256u2_s5_et128x32_m256x32x32_c2x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d1221740a943e65f7ee2ecb3f912b4fe1d9711d3342da77667b0587e55dcdfb5 -size 21800 +oid sha256:321ca0107654af2877d790f8fcebfa65f13c5a7c05e9d54d22d07c9e5c2e921b +size 56854 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x128_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW1_lsfbW1_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x128_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW1_lsfbW1_dynB_sm100f.cubin.tar.zst new file mode 100644 index 000000000000..8c94c5fbccac --- /dev/null +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x128_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW1_lsfbW1_dynB_sm100f.cubin.tar.zst @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:6b507d39c5a9a63b5fa5740433334a001db5f5911dc153e8c3357004446cc0fd +size 77132 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x128_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW1_lsfbW1_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x128_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW1_lsfbW1_dynB_sm100f.cubin.tar.zst index 33de56ea92b4..4cdbbdeb5b97 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x128_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW1_lsfbW1_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x128_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW1_lsfbW1_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3abec70154b190c0ce5c5bcdbf06aec5470a1713f51fc9febe98dd4857e0f1bf -size 28001 +oid sha256:7cdcf8a46100ccfe8a9048eaa0bbdb91136c7688df15fbfff137e6ed303402dc +size 76284 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x128u2_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW1_lsfbW1_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x128u2_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW1_lsfbW1_dynB_sm100f.cubin.tar.zst new file mode 100644 index 000000000000..42539f6bb949 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x128u2_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW1_lsfbW1_dynB_sm100f.cubin.tar.zst @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:30cc3638b3779520d5cdd37ec711e17e5c43c5d2401389dfa1b389f9db0bfcf9 +size 80054 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x128u2_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW1_lsfbW1_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x128u2_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW1_lsfbW1_dynB_sm100f.cubin.tar.zst index c3f5a7bd071f..ac63ca665c1a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x128u2_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW1_lsfbW1_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x128u2_s7_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW1_lsfbW1_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8ae8b0870f0e32da19d415686342bb80442f552e07722620a19ad33746fe700a -size 29225 +oid sha256:f0c730eeb3ee582bd9da74f5808736bc25c0a79a532763d6c2ffef6dd714a52e +size 79199 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x256_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW1_lsfbW1_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x256_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW1_lsfbW1_dynB_sm100f.cubin.tar.zst new file mode 100644 index 000000000000..483195f4ddf5 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x256_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW1_lsfbW1_dynB_sm100f.cubin.tar.zst @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:6058a8acbcd22714d5d3fb9897ee2df54b1ff6ee26157c7c297aa6ac9ec0cf15 +size 83225 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x256_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW1_lsfbW1_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x256_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW1_lsfbW1_dynB_sm100f.cubin.tar.zst index a2f1049e7304..c03b7960d0a6 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x256_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW1_lsfbW1_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x256_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW1_lsfbW1_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:223bf650b818664b64a4e8d43e0c64e7419c557d559ee1d894c091e9026ae319 -size 30472 +oid sha256:3ba3adcb6a979cf34df4555bcf225a631b1054a030bab429fa3865fdeb0fbcf4 +size 82071 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x256u2_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW1_lsfbW1_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x256u2_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW1_lsfbW1_dynB_sm100f.cubin.tar.zst new file mode 100644 index 000000000000..5c31f04cd51f --- /dev/null +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x256u2_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_lbW1_lsfbW1_dynB_sm100f.cubin.tar.zst @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:5b0ef7f22bbc80f091816af10459b544b296171a4a32d16233c88f46146335d2 +size 85866 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x256u2_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW1_lsfbW1_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x256u2_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW1_lsfbW1_dynB_sm100f.cubin.tar.zst index 8469c8efafe8..33f0babba0b0 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x256u2_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW1_lsfbW1_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x64x256u2_s4_et128x64_m256x64x32_c2x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_lbW1_lsfbW1_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:ee959fefef818a681bbd5971232bff4c1dce15322303e7494e51e61447a8693a -size 31698 +oid sha256:c3a65c8ef4e38eda1b836039ca4c9d422f7ee33c8bea4d9328a81c332dc324e9 +size 85646 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst new file mode 100644 index 000000000000..ad3a52f33617 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:0d5469eba92f88827efcfd343d1e49ab04f4908bc9188afc0f88cf5d7f371e65 +size 48744 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 31c95db64a75..dcb54f02e61e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:93c9caa3918084c3b059084958300c84b776b7c3f22bd4c80de9266490138adf -size 19816 +oid sha256:5d71b1cdf799fe0a118a47dadc4c9bdc96ea6cb5c81a098b73a38f2848b32ebd +size 49373 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst new file mode 100644 index 000000000000..cf2ccf6c2f8f --- /dev/null +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:900651f9751e93e5fcd5e7a605bdac3cb6379990bcf67be91574cb733a7402dd +size 36707 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index fd2894aacf90..eecc28ff7acf 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2a18c17b6593b5fa48e9083cc5fccc546438f46d51b65ebc03516dba1b6fb95a -size 15080 +oid sha256:c62bbc704a088838720063e9b8f70df9dc508f89a5a255d8077f9e5f161f3c50 +size 37198 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst new file mode 100644 index 000000000000..ce8265479e2c --- /dev/null +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:8b856c952314567c4cf6bacdc7558937133dbb1cb16ac729c2487d47cd83a206 +size 58806 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index f272a48cffbe..de86458cb7c8 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:078b59b4b3380a623a7a70978e408b3c8191adec59253c4553ac85339b268da3 -size 24377 +oid sha256:bd00b1466fb2ae39cda25f61c6f60d3039e962b53ecaf733c1c1014d6c101e75 +size 58980 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst new file mode 100644 index 000000000000..824695f13b31 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b4377465a8e5cd1166d3c461965dbe8e1e8c7b1b24e03730abafe4335370f8ab +size 44439 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index a51a0e93bc7a..aa5648e9ad94 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:f880f52923d2d16e4b4b690e76bca25f2c57b7f140034a628d985f06f1df04f3 -size 18100 +oid sha256:106137331cfa1d581443075a6cf2d9b18defac478bd13894f34786c86072658d +size 44469 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst new file mode 100644 index 000000000000..0e4a4c6951a2 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:3c09bbf6c85ab45a3d76dbe49c6fad4a2ec286b84c34289f7c246bd34c634f68 +size 52693 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 0dcc1d7f789f..803b3d5d498a 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:c440d7d44dd0db6de3698043f921fce6595a7187c2788d2b49ffb62970643f1d -size 20744 +oid sha256:b053f7bf38854af13b56c5cd90431a48e7f229e7f62cc4d1930e730d3021ebb8 +size 52920 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst new file mode 100644 index 000000000000..93c904cbe793 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:c08e34e0be92cc4b1e8c182fed71b422b4e2876e045066c3fd078457aba1e23d +size 40389 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 5659fc2cbd1f..5b32a9d8d110 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:4d93c90b854304756516d9c08e9c443269c4631a0cef70339e33b2601b0d6854 -size 16117 +oid sha256:a8de8cc7648968fe5dadd4c72c7d991a5c07c36d027ff65ded9f8a90c89b636e +size 40861 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst new file mode 100644 index 000000000000..a4963f062570 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:f8961d9acf9fc0e41be4e4862e9bb63fbf5d38c4b9abf203b8a54ff534f665d6 +size 62417 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 0c12c21bde76..c6ecbf7cd8ed 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:d87b55456b88e898001cade6acb50d0b91328bb68e9983139d6debd0dea3b45b -size 25239 +oid sha256:d97945b71da1454db440fee03ecd9b343049c0e6146c9855f09aa5974a42561a +size 62667 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst new file mode 100644 index 000000000000..dcc154a2e753 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:69f9598562f8fc57d456573cfd697ad341d41debf383bf3e01186ec1c92af5e0 +size 48193 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 87ec3c3a21dc..e13370daab0e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x256u2_s6_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:2fc7bff4741fd80e9e4fc4ad238b7425ab244c159edadaa21ec524d1455ed1dd -size 19147 +oid sha256:4c9593eaa8a62e418237d089b57df053439e2e4a964331a7209a08a5e7c478ec +size 48486 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst new file mode 100644 index 000000000000..c75dc505c3d8 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:a55f28e435cd9ac7e86cdcd09fee4dd41a9696856ba9cf631640843422f03ecb +size 50501 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index d45e1ade41da..da8f7f66c4dd 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:be7d696864226d4e7041a65ca8f65d43f2f51178cc1e619bc56f54c85164827d -size 19849 +oid sha256:852f895e411bed93c9e75488b2795e7fb11c6f763dfef167b2f523b5a5117934 +size 51156 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst new file mode 100644 index 000000000000..6d22a6b7310d --- /dev/null +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:edcd72ea9e4d96295880a3e4280dc1f785c634bd49d29867159855189dd8f904 +size 38773 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index d11a254fc89b..96889e57f3b3 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:262f85b2d0799dd9ba044e76f04060f0703e7488037f72649ae8f3ef8624ade3 -size 15105 +oid sha256:1a62ba6aeb55ad77db14c00593ae727bb898bfc73b8e7b3074ae7484aaff0d17 +size 39203 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst new file mode 100644 index 000000000000..a32439b31304 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:828c3c9d719a2da3bac10258b82db08decdde5ab2b38305239c8205c3abef7eb +size 60724 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 9837482b12ba..1099b4ca30ea 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:05b4054229642e34b3afba8504d954098ce26dfc464626db275218787a67c297 -size 24316 +oid sha256:f35268a0286868dee2af4f5575d65a856252b005197768a2174b8f4615f95388 +size 60498 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst new file mode 100644 index 000000000000..32e7cd7708e7 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:3baa0f9c234044722f2a5d2839e45b40d5f9766a56a46cea8d4fa9ffefe6c1f6 +size 46399 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 0ba710cb8ea5..7078b8c8838e 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:80d93dea336aa00c438a32cb1adf3c8d977df18045c28c197c71839c87a76c06 -size 18365 +oid sha256:9eea75f3d939a2308064038a871262e5fb730b81e273027b19ad661de62a175e +size 46467 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst new file mode 100644 index 000000000000..c4b4f3d42d68 --- /dev/null +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:90f67b1fb2d8010227a14de435a3174b0e9d1fd90697696af432145b49dc5ae0 +size 55331 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 6f03c3a6200a..263b169d2667 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:61e2a930a03b7468f40f612400b27a611dbb7455c167aad5040daf667edbc618 -size 20974 +oid sha256:4f6b4d0f32f9f54582d6265f801c538b9c01ae8502c0b3ba89e93823212e1982 +size 55391 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst new file mode 100644 index 000000000000..47403bfe7aee --- /dev/null +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b83ac839fa9abed200650decb265bf38578720d8b4a23836769b9d4ad18957a3 +size 43876 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index c9620d06b17e..0b8450ca1559 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x1_rM_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8f46ae2bcf96668d252482d22702931140b826a7551cc02754edeb780c6eca1b -size 16645 +oid sha256:7310cd6b59d78c21a5e76509f4b4bb3b800fbfa17993dbf981605a5722bba14b +size 44321 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst new file mode 100644 index 000000000000..318ffdd1f14d --- /dev/null +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:fb10d7dcc5e310c2abdd49c7855f47241b35d88472a3371b71b7ac4fd4786ae7 +size 65064 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index d97d8d3b13b8..45881ee5e925 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schPd2x1x2x3_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:64cf91502fd8eef2dbc66ef42a5662b67e6b2d0b429e42d57fddedce05bc7fa7 -size 25513 +oid sha256:40fb4db421639030afa488e8e46a0ffa9eda7ac76413b34037a47150c6689bb2 +size 65195 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst new file mode 100644 index 000000000000..84a1491da22d --- /dev/null +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_bN_tma_ldgstsSf_rgTma_clmp_siTuGlu_dynB_sm100f.cubin.tar.zst @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:fa1d9a7ab64332d275392ad2cccf403b9d214822ea6de6a2585bdfb372495842 +size 51199 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst index 809a884fd269..2773abd5ecf7 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE2m1MxE4m3_Fp32_Ab32_Bb32_Cb32_t128x8x512u2_s3_et128x8_m128x8x32_c1x1x2_rM_splitK2_TN_transOut_schedS_biasFp32M_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:1a1c239400bc4dfc8d37b1097731d34a74922b9320ee6abb0c60fcde27852076 -size 19834 +oid sha256:d9a364248261c47a50aa692f4c446932db317d9f1d94ef0030a2f1b0bb8bdd11 +size 51283 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE4m3MxE2m1_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7_et128x64_m256x128x32_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_swiGlu_laW8_lsfaW4_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE4m3MxE2m1_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7_et128x64_m256x128x32_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_swiGlu_laW8_lsfaW4_dynB_sm100f.cubin.tar.zst index 37f04a36eb64..954b5d8bfbb9 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE4m3MxE2m1_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7_et128x64_m256x128x32_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_swiGlu_laW8_lsfaW4_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE4m3MxE2m1_Fp32_Ab32_Bb32_Cb32_t128x128x128_s7_et128x64_m256x128x32_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_swiGlu_laW8_lsfaW4_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:805fbfe96c746aad57d23e4448a272a765a8cbe764c20e4224670077414cce8e -size 30698 +oid sha256:776324bf080073f0059a2c6d7b45508db0e812471f56454ef59874a5b3aec773 +size 84260 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE4m3MxE2m1_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4_et128x64_m256x128x32_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_swiGlu_laW8_lsfaW4_dynB_sm100f.cubin.tar.zst b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE4m3MxE2m1_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4_et128x64_m256x128x32_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_swiGlu_laW8_lsfaW4_dynB_sm100f.cubin.tar.zst index eeab9e1e0bdc..b9b443633c1f 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE4m3MxE2m1_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4_et128x64_m256x128x32_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_swiGlu_laW8_lsfaW4_dynB_sm100f.cubin.tar.zst +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/batchedGemm/trtllmGen_bmm_export/cubins/Bmm_MxE4m3_MxE4m3MxE2m1_Fp32_Ab32_Bb32_Cb32_t128x128x256_s4_et128x64_m256x128x32_c2x1x1_rN_TN_schPd2x1x2x3_biasFp32N_bM_tma_ldgstsSf_rgTma_clmp_swiGlu_laW8_lsfaW4_dynB_sm100f.cubin.tar.zst @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:83c7a9348d014ba1adec826276a6826a95d21484781bc8cb96c0cc66f0142757 -size 31936 +oid sha256:6b990868fd33bbf1c8756215e4ad7136aca0363eda19a56d0123523ed8fd8f6a +size 87396 diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/blockScaleMoe/DevKernel.cu b/cpp/tensorrt_llm/kernels/trtllmGenKernels/blockScaleMoe/DevKernel.cu index 11450f738934..43a1a6846cdc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/blockScaleMoe/DevKernel.cu +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/blockScaleMoe/DevKernel.cu @@ -93,7 +93,11 @@ __global__ void activationKernel(KernelParams params) for (int hiddenIdx = threadIdx.x + blockDim.x * blockIdx.x; hiddenIdx < params.innerDim / 2; hiddenIdx += blockDim.x * gridDim.x) { - int const baseIdx = permutedIdx * params.innerDim + hiddenIdx; + // Compute global-memory offsets in int64: under Attention DP + AllGather the + // permuted token count (permutedIdx up to totalNumPaddedTokens) times a hidden/inner + // dimension can exceed INT_MAX, so 32-bit index math would overflow. Applies to all + // permutedIdx/tokenIdx * dim offsets in this file. + int64_t const baseIdx = static_cast(permutedIdx) * params.innerDim + hiddenIdx; float x1 = (float) params.inPtr[baseIdx]; // up (linear) float x2 = (float) params.inPtr[baseIdx + params.innerDim / 2]; // gate (silu input) @@ -107,7 +111,7 @@ __global__ void activationKernel(KernelParams params) float act = silu(x2); Type out = (Type) (act * x1); - int const outIdx = permutedIdx * (params.innerDim / 2) + hiddenIdx; + int64_t const outIdx = static_cast(permutedIdx) * (params.innerDim / 2) + hiddenIdx; params.outPtr[outIdx] = out; } } @@ -453,7 +457,7 @@ __global__ void activationDeepSeekKernel(KernelParams params) } // Process blocks for this CTA - int const baseIdx = permutedIdx * params.innerDim + hiddenIdx; + int64_t const baseIdx = static_cast(permutedIdx) * params.innerDim + hiddenIdx; int const scale1Idx = permutedIdx + totalNumPaddedTokens * (hiddenIdx / 128); int const scale2Idx @@ -522,7 +526,7 @@ __global__ void activationDeepSeekKernel(KernelParams params) continue; } float const scaleOut = s_scaleOutArr[tokenInCtaIdx]; - int const outIdx = permutedIdx * (params.innerDim / 2) + hiddenIdx; + int64_t const outIdx = static_cast(permutedIdx) * (params.innerDim / 2) + hiddenIdx; params.outPtr[outIdx] = static_cast(outArr[tokenInCtaIdx] / scaleOut); } } @@ -848,14 +852,14 @@ __global__ void permuteKernel(KernelParams params) { // Load chunk of token into registers - const Type data = params.inPtr[tokenIdx * params.hiddenDim + hiddenIdx]; + const Type data = params.inPtr[static_cast(tokenIdx) * params.hiddenDim + hiddenIdx]; // Write to topK places for (int k = 0; k < params.topK; k++) { int const expandedIdx = tokenIdx * params.topK + k; int const permutedIdx = params.expandedIdxToPermutedIdx[expandedIdx]; - params.outPtr[permutedIdx * params.hiddenDim + hiddenIdx] = data; + params.outPtr[static_cast(permutedIdx) * params.hiddenDim + hiddenIdx] = data; } } if (params.useDeepSeekFp8) @@ -943,15 +947,16 @@ __global__ void finalizeKernel(KernelParams params) if (params.expertWeightsPtr != nullptr) { TypeExpW const scale = params.expertWeightsPtr[expandedIdx]; - data += float{scale} * float{params.inPtr[permutedIdx * params.hiddenDimPadded + hiddenIdx]}; + data += float{scale} + * float{params.inPtr[static_cast(permutedIdx) * params.hiddenDimPadded + hiddenIdx]}; } else { - data += float{params.inPtr[permutedIdx * params.hiddenDimPadded + hiddenIdx]}; + data += float{params.inPtr[static_cast(permutedIdx) * params.hiddenDimPadded + hiddenIdx]}; } } - params.outPtr[tokenIdx * params.hiddenDim + hiddenIdx] = static_cast(data); + params.outPtr[static_cast(tokenIdx) * params.hiddenDim + hiddenIdx] = static_cast(data); } } } @@ -1081,7 +1086,9 @@ __global__ void finalizeDeepSeekKernel(KernelParams params) float const expertProb = (float) params.expertWeightsPtr[tokenIdx * params.topK + k]; float const scale = expertProb * blockScale; - acc += scale * static_cast(params.inPtr[permutedIdx * params.hiddenDimPadded + hiddenIdx]); + acc += scale + * static_cast( + params.inPtr[static_cast(permutedIdx) * params.hiddenDimPadded + hiddenIdx]); } // The largest (finite) value that can be represented using E4m3. @@ -1106,7 +1113,7 @@ __global__ void finalizeDeepSeekKernel(KernelParams params) __syncthreads(); float const scaleOut = s_scaleOut; __syncthreads(); - params.outPtr[tokenIdx * params.hiddenDim + hiddenIdx] = (Type) (acc / scaleOut); + params.outPtr[static_cast(tokenIdx) * params.hiddenDim + hiddenIdx] = (Type) (acc / scaleOut); } } } diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/blockScaleMoe/routing/RoutingCustom.cu b/cpp/tensorrt_llm/kernels/trtllmGenKernels/blockScaleMoe/routing/RoutingCustom.cu index db0bf48e79ad..6a0ba120d4cc 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/blockScaleMoe/routing/RoutingCustom.cu +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/blockScaleMoe/routing/RoutingCustom.cu @@ -28,6 +28,8 @@ #include "RoutingCustomPolicy.cuh" +#include + namespace moe::dev::routing { namespace routingCustom @@ -387,6 +389,405 @@ void launchBlockKernel(Data const& data, uint32_t numThreadsHist, void* stream) stream); } +//////////////////////////////////////////////////////////////////////////////////////////////////// +// +// 1a. Cooperative block kernel — single-block fused kernel for ≤4 tokens and ≤1024 experts, +// raw-scores input, ELEMENTWISE preprocess policies only (None / Sigmoid / SigmoidBias). +// +// Motivation: routingIndicesBlockKernel runs TopK with one warp per token over +// VecSize = MaxNumExperts/32 register-resident candidates. For the large-expert tiers +// (e.g. 1024 experts / MaxNumTopExperts 32) that needs ~200 registers per thread +// (score[V] + idx[V] + the 64-bit sort network + out arrays) while +// __launch_bounds__(1024) caps the kernel at 64 — everything spills to local memory +// and a single decode token costs ~33 us on GB300 (896 experts, topK 16). +// +// This kernel instead assigns ONE THREAD PER EXPERT (ExpertsPerThread for >1024-thread +// tiers) so per-thread state is a couple of registers: +// Phase A: every thread computes its expert's selection score (elementwise preprocess), +// packs it with TopKRedType (identical encoding/tie-break as the classic path), +// and each warp extracts its own top-`topK` by iterated redux.sync max into a +// shared-memory candidate list (no register arrays, no sort network). +// Phase B: one warp per token merges the NumWarps sorted candidate lists with a +// lane-per-list cursor loop — the k-th extracted winner is the k-th largest +// packed value overall, i.e. bit-identical selection (packed values are unique +// since the expert index is embedded in the low bits). +// Lane k keeps the k-th (score, expertIdx) — the same lane layout the classic +// kernels use — then applies the postprocess policy and writes the weights. +// Phase C: histogram / prefix-scan / CTA configs / permutation, identical math to +// routingIndicesBlockKernel but with the fused dual warp-scan (2 barriers) +// instead of two cub::BlockScans. +// +// Bit-exactness vs routingIndicesBlockKernel (required — expert selection must not +// change under the small-batch fast path): +// * per-element preprocess math is the same ops in the same per-element order; +// * selection order is the same total order on the same packed 64/32-bit values; +// * postprocess warp reductions see the same values in the same lanes. +// +//////////////////////////////////////////////////////////////////////////////////////////////////// + +template +__global__ void __launch_bounds__(KernelParams::MaxNumExperts <= 1024 ? KernelParams::MaxNumExperts : 1024) + routingIndicesCoopBlockKernel(KernelParams params) +{ + using OutputT = typename KernelParams::OutputT; + using InputT = typename KernelParams::InputT; + using ExpertSelect = typename KernelParams::ExpertSelectPolicy; + using PreProc = typename ExpertSelect::PreprocessPolicy; + using PostProc = typename ExpertSelect::PostprocessPolicy; + using BaseType = typename ExpertSelect::template BaseType; + using RedType = topk::TopKRedType; + using TypeCmp = typename RedType::TypeCmp; + + static constexpr int MaxNumExperts = KernelParams::MaxNumExperts; + static constexpr int MaxTopK = KernelParams::MaxNumTopExperts; + static constexpr int NumThreadsBlock = MaxNumExperts <= 1024 ? MaxNumExperts : 1024; + static constexpr int ExpertsPerThread = MaxNumExperts / NumThreadsBlock; + static constexpr int NumWarpsBlock = NumThreadsBlock / WarpSize; + static_assert(MaxNumExperts % NumThreadsBlock == 0, "MaxNumExperts must be a multiple of NumThreadsBlock"); + static_assert(NumWarpsBlock <= WarpSize, "the merge phase holds one candidate list per lane"); + + // The host dispatch (run()) only selects this kernel for tiers within these limits and for + // elementwise preprocess policies; other instantiations exist only to satisfy the generic + // policy dispatch macros and must stay compilable (and small). + static constexpr bool Supported = MaxNumExperts <= CoopBlockKernelMaxNumExperts && PreProc::IsElementwise; + + if constexpr (Supported) + { + static constexpr int MaxNumTokens = BlockKernelMaxNumTokens; + + int32_t const warpIdx = __shfl_sync(0xffffffff, threadIdx.x / WarpSize, 0); + int32_t const laneIdx = cutlass::arch::LaneId(); + + __shared__ int8_t __attribute((aligned(128))) smemKIdx[MaxNumTokens * MaxNumExperts]; + __shared__ int8_t __attribute((aligned(128))) smemOffset[MaxNumTokens * MaxNumExperts]; + // Per-warp sorted candidate lists (descending packed values), per token. + __shared__ TypeCmp __attribute((aligned(128))) smemCand[MaxNumTokens][NumWarpsBlock * MaxTopK]; + __shared__ int32_t __attribute((aligned(128))) warpTotals1[NumWarpsBlock]; + __shared__ int32_t __attribute((aligned(128))) warpTotals2[NumWarpsBlock]; + + auto block = cg::this_thread_block(); + auto warp = cg::tiled_partition(block); + + int const numSlots = params.mNumTokens * MaxNumExperts; + for (int i = threadIdx.x; i < numSlots; i += blockDim.x) + { + smemKIdx[i] = int8_t{-1}; + } + __syncthreads(); + +#if (defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900)) + // Wait on the primary grid: the scores are produced by the preceding kernel. + if (params.mUsePdl) + { + cudaGridDependencySynchronize(); + } +#endif + + BaseType const minScore = BaseType{-INFINITY}; + bool const expertThread = threadIdx.x < NumThreadsBlock; + + // ----- Phase A: per-warp topK extraction, one token at a time (≤4) ----- + for (int tokenIdx = 0; tokenIdx < params.mNumTokens; ++tokenIdx) + { + InputT const* scorePtr = params.mPtrScores + tokenIdx * params.mNumExperts; + + BaseType score[ExpertsPerThread]; + int32_t idx[ExpertsPerThread]; +#pragma unroll + for (int e = 0; e < ExpertsPerThread; ++e) + { + // Contiguous per-thread expert assignment — same as the histogram phase below. + int32_t const expertIdx = threadIdx.x * ExpertsPerThread + e; + score[e] = expertThread && expertIdx < params.mNumExperts ? static_cast(scorePtr[expertIdx]) + : minScore; + idx[e] = expertIdx; + } + // Elementwise preprocess: identical per-element math to the classic kernels. + PreProc::apply(warp, score, idx, params.mNumExperts, params.mExpertSelectParams.mPreprocessParams); + + RedType cand[ExpertsPerThread]; +#pragma unroll + for (int e = 0; e < ExpertsPerThread; ++e) + { + cand[e] = RedType{score[e], idx[e]}; + } + + // Iteratively extract this warp's top `topK` packed candidates (descending). + for (int kk = 0; kk < params.mTopK; ++kk) + { + RedType localMax = cand[0]; +#pragma unroll + for (int e = 1; e < ExpertsPerThread; ++e) + { + localMax.compVal = max(localMax.compVal, cand[e].compVal); + } + TypeCmp const warpMax = localMax.reduce(warp); +#pragma unroll + for (int e = 0; e < ExpertsPerThread; ++e) + { + // Invalidate the winner (packed values are unique — idx is embedded). + cand[e] = warpMax == cand[e].compVal ? RedType{minScore, idx[e]} : cand[e]; + } + if (laneIdx == 0 && warpIdx < NumWarpsBlock) + { + smemCand[tokenIdx][warpIdx * MaxTopK + kk] = warpMax; + } + } + } + __syncthreads(); + + // ----- Phase B: merge the per-warp lists — warp t handles token t ----- + if (warpIdx < params.mNumTokens) + { + int32_t const tokenIdx = warpIdx; + // Lane l walks warp-list l. Empty-lane sentinel 0 is strictly below any candidate: + // every candidate has (65535 - idx) > 0 in its low bits. + int cursor = 0; + TypeCmp head = laneIdx < NumWarpsBlock ? smemCand[tokenIdx][laneIdx * MaxTopK] : TypeCmp{0}; + + BaseType myScore = BaseType{0}; + int32_t myIdx = int32_t{-1}; + for (int kk = 0; kk < params.mTopK; ++kk) + { + RedType redCand; + redCand.compVal = head; + TypeCmp const winner = redCand.reduce(warp); + if (head == winner && laneIdx < NumWarpsBlock) + { + ++cursor; + head = cursor < params.mTopK ? smemCand[tokenIdx][laneIdx * MaxTopK + cursor] : TypeCmp{0}; + } + if (laneIdx == kk) + { + RedType::unpack(myScore, myIdx, winner); + } + } + + // Postprocess with the canonical lane layout (lane k holds the k-th top entry). + // The policies only access element [laneIdx] of these arrays. + BaseType warpTopKScore[MaxTopK]; + int32_t warpTopKExpertIdx[MaxTopK]; + if (laneIdx < params.mTopK) + { + warpTopKScore[laneIdx] = myScore; + warpTopKExpertIdx[laneIdx] = myIdx; + } + PostProc::apply(warp, warpTopKScore, warpTopKExpertIdx, laneIdx, params.mTopK, + params.mExpertSelectParams.mPostprocessParams); + + if (laneIdx < params.mTopK) + { + smemKIdx[tokenIdx * MaxNumExperts + myIdx] = static_cast(laneIdx); + if (params.mPtrTopKWeights != nullptr) + { + params.mPtrTopKWeights[tokenIdx * params.mTopK + laneIdx] = OutputT{warpTopKScore[laneIdx]}; + } + } + } + __syncthreads(); + + // ----- Phase C: histogram / scan / CTA configs / permutation ----- + // Identical math to routingIndicesBlockKernel; fused dual warp-scan like the dyn-block kernel. + int accExpertCount[ExpertsPerThread]; +#pragma unroll + for (int e = 0; e < ExpertsPerThread; ++e) + { + accExpertCount[e] = 0; + } + if (expertThread) + { +#pragma unroll + for (int e = 0; e < ExpertsPerThread; ++e) + { + int const expert = threadIdx.x * ExpertsPerThread + e; + auto const localExpIdx = expert - params.mLocalExpertsStartIdx; + auto const isLocal = localExpIdx >= 0 && localExpIdx < params.mNumLocalExperts + && (localExpIdx & ((1 << params.mLocalExpertsStrideLog2) - 1)) == 0; + if (isLocal) + { + int offset = expert; + for (int j = 0; j < params.mNumTokens; ++j) + { + if (smemKIdx[offset] >= 0) + { + smemOffset[offset] = static_cast(accExpertCount[e]); + accExpertCount[e]++; + } + offset += MaxNumExperts; + } + } + } + } + + int32_t numCtaPerExpert[ExpertsPerThread]; + int32_t tmpCountPerExpert[ExpertsPerThread]; + int32_t ctaOffsetPerExpert[ExpertsPerThread]; + int32_t expertScanCountsPerExpert[ExpertsPerThread]; + int32_t numNonExitingCtas; + { +#pragma unroll + for (int e = 0; e < ExpertsPerThread; ++e) + { + if (expertThread) + { + if (params.mIsPow2) + { + numCtaPerExpert[e] = divUpLog2(accExpertCount[e], params.mPaddingLog2); + tmpCountPerExpert[e] = divUpMulLog2(accExpertCount[e], params.mPaddingLog2); + } + else + { + numCtaPerExpert[e] = divUpTileN(accExpertCount[e], params.mTileTokensDim); + tmpCountPerExpert[e] = divUpMulTileN(accExpertCount[e], params.mTileTokensDim); + } + } + else + { + numCtaPerExpert[e] = 0; + tmpCountPerExpert[e] = 0; + } + } + + int32_t localPrefix1[ExpertsPerThread], localPrefix2[ExpertsPerThread]; + int32_t threadTotal1 = 0, threadTotal2 = 0; +#pragma unroll + for (int e = 0; e < ExpertsPerThread; ++e) + { + localPrefix1[e] = threadTotal1; + localPrefix2[e] = threadTotal2; + threadTotal1 += numCtaPerExpert[e]; + threadTotal2 += tmpCountPerExpert[e]; + } + + int32_t threadPrefix1, threadPrefix2; + warpExclusiveScan(threadTotal1, threadTotal2, laneIdx, warpIdx, warpTotals1, warpTotals2, + threadPrefix1, threadPrefix2, numNonExitingCtas); + +#pragma unroll + for (int e = 0; e < ExpertsPerThread; ++e) + { + ctaOffsetPerExpert[e] = threadPrefix1 + localPrefix1[e]; + expertScanCountsPerExpert[e] = threadPrefix2 + localPrefix2[e]; + } + } + + if (expertThread) + { +#pragma unroll + for (int e = 0; e < ExpertsPerThread; ++e) + { + int const expert = threadIdx.x * ExpertsPerThread + e; + auto const localExpIdx = expert - params.mLocalExpertsStartIdx; + auto const isLocal = localExpIdx >= 0 && localExpIdx < params.mNumLocalExperts + && (localExpIdx & ((1 << params.mLocalExpertsStrideLog2) - 1)) == 0; + if (isLocal) + { + for (int cta = 0; cta < numCtaPerExpert[e]; ++cta) + { + int32_t const mappedLocalIdx + = (expert - params.mLocalExpertsStartIdx) >> params.mLocalExpertsStrideLog2; + params.mPtrCtaIdxXyToBatchIdx[ctaOffsetPerExpert[e] + cta] = mappedLocalIdx; + int32_t mnLimit1; + int32_t mnLimit2; + if (params.mIsPow2) + { + mnLimit1 = mulLog2(ctaOffsetPerExpert[e] + cta + 1, params.mPaddingLog2); + mnLimit2 = mulLog2(ctaOffsetPerExpert[e], params.mPaddingLog2) + accExpertCount[e]; + } + else + { + mnLimit1 = mulTileN(ctaOffsetPerExpert[e] + cta + 1, params.mTileTokensDim); + mnLimit2 + = mulTileN(ctaOffsetPerExpert[e], params.mTileTokensDim) + accExpertCount[e]; + } + params.mPtrCtaIdxXyToMnLimit[ctaOffsetPerExpert[e] + cta] = min(mnLimit1, mnLimit2); + } + } + } + } + + if (threadIdx.x == 0) + { + int32_t permutedIdxSize; + if (params.mIsPow2) + { + permutedIdxSize = mulLog2(numNonExitingCtas, params.mPaddingLog2); + } + else + { + permutedIdxSize = mulTileN(numNonExitingCtas, params.mTileTokensDim); + } + params.mPtrPermutedIdxSize[0] = permutedIdxSize; + params.mPtrNumNonExitingCtas[0] = numNonExitingCtas; + } + + if (expertThread) + { + for (int tokenIdx = 0; tokenIdx < params.mNumTokens; ++tokenIdx) + { +#pragma unroll + for (int e = 0; e < ExpertsPerThread; ++e) + { + int const expert = threadIdx.x * ExpertsPerThread + e; + int const offset = tokenIdx * MaxNumExperts + expert; + if (smemKIdx[offset] >= 0) + { + auto const localExpIdx = expert - params.mLocalExpertsStartIdx; + auto const isLocal = localExpIdx >= 0 && localExpIdx < params.mNumLocalExperts + && (localExpIdx & ((1 << params.mLocalExpertsStrideLog2) - 1)) == 0; + + int const expandedIdx = tokenIdx * params.mTopK + smemKIdx[offset]; + int const offsetWithinExpert = static_cast(smemOffset[offset]); + int const offsetForExpert = expertScanCountsPerExpert[e]; + int const permutedIdx = isLocal ? offsetForExpert + offsetWithinExpert : int32_t{-1}; + + if (params.mPtrExpandedIdxToPermutedIdx != nullptr) + { + params.mPtrExpandedIdxToPermutedIdx[expandedIdx] = permutedIdx; + } + if (params.mPtrPermutedIdxToExpandedIdx != nullptr && isLocal) + { + params.mPtrPermutedIdxToExpandedIdx[permutedIdx] = expandedIdx; + } + if (params.mPtrPermutedIdxToTokenIdx != nullptr && isLocal) + { + params.mPtrPermutedIdxToTokenIdx[permutedIdx] = tokenIdx; + } + } + } + } + } + +#if (defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900)) + // Trigger the secondary kernel AFTER all global memory writes (including permutation + // indices) — downstream kernels depend on all routing outputs being visible. + if (params.mUsePdl) + { + cudaTriggerProgrammaticLaunchCompletion(); + } +#endif + } + else + { + // Unsupported instantiation — never launched (see run()). + (void) params; +#if (defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900)) + if (params.mUsePdl) + { + cudaGridDependencySynchronize(); + cudaTriggerProgrammaticLaunchCompletion(); + } +#endif + } +} + +void launchCoopBlockKernel(Data const& data, uint32_t numThreadsHist, void* stream) +{ + LAUNCH_ROUTING_CUSTOM(data, false, routingIndicesCoopBlockKernel, 1, numThreadsHist, + /*smemSize=*/0, // No dynamic smem + stream); +} + //////////////////////////////////////////////////////////////////////////////////////////////////// // // 1b. Dynamic-block kernel — single-block with dynamic thread count and dynamic shared memory. @@ -1228,6 +1629,21 @@ void run(Data const& data, void* stream) bool const useStaticBlock = data.mNumTokens <= BlockKernelMaxNumTokens; int32_t const dispatchedMaxExperts = queryDispatchedMaxExperts(data); + // Cooperative block kernel: fastest path for tiny batches. Requires an elementwise + // preprocess (any but softmax-over-experts) and one CUDA block's worth of experts. + // Critical for large expert counts, where the classic one-warp-per-token TopK spills + // registers under the 1024-thread launch bounds (e.g. 896 experts / topK 16 at decode). + bool const preprocessIsElementwise = data.mPreprocessType == RoutingPreprocessType::None + || data.mPreprocessType == RoutingPreprocessType::Sigmoid + || data.mPreprocessType == RoutingPreprocessType::SigmoidBias; + // Escape hatch for A/B validation and emergency fallback to the classic block kernel. + static bool const disableCoopBlock = [] + { + char const* env = std::getenv("TLLM_ROUTING_DISABLE_COOP_BLOCK"); + return env != nullptr && env[0] == '1'; + }(); + bool const useCoopBlock = !disableCoopBlock && useStaticBlock && preprocessIsElementwise + && dispatchedMaxExperts <= CoopBlockKernelMaxNumExperts; bool const useDynBlock = !useStaticBlock && data.mNumTokens <= DynBlockKernelMaxNumTokens && dispatchedMaxExperts <= DynBlockKernelMaxNumExperts; bool const useSingleBlock = useStaticBlock || useDynBlock; @@ -1245,7 +1661,11 @@ void run(Data const& data, void* stream) Data lastKernelData = data; - if (useDynBlock) + if (useCoopBlock) + { + launchCoopBlockKernel(lastKernelData, numThreadsHist, stream); + } + else if (useDynBlock) { launchDynBlockKernel(lastKernelData, numThreadsHist, stream); } diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/blockScaleMoe/routing/RoutingCustomPolicy.cuh b/cpp/tensorrt_llm/kernels/trtllmGenKernels/blockScaleMoe/routing/RoutingCustomPolicy.cuh index 8da6a827ffa7..ad0163cf9294 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/blockScaleMoe/routing/RoutingCustomPolicy.cuh +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/blockScaleMoe/routing/RoutingCustomPolicy.cuh @@ -37,6 +37,12 @@ namespace moe::dev::routing /// No-op: scores are passed through unchanged. struct NoOpPreprocess { + /// Elementwise: each score is transformed independently of the others. + /// Elementwise preprocess policies can run with any expert-to-thread partitioning + /// (see routingIndicesCoopBlockKernel); non-elementwise ones (softmax) require the + /// one-warp-per-token layout of the classic kernels. + static constexpr bool IsElementwise = true; + /// BaseType: when no preprocess is applied, use the input type directly. template using BaseType = InputT; @@ -58,6 +64,9 @@ struct NoOpPreprocess /// Softmax: applies softmax over all expert scores before topK selection. struct SoftmaxPreprocess { + /// Softmax couples all scores of a token — NOT elementwise (see NoOpPreprocess). + static constexpr bool IsElementwise = false; + /// BaseType: softmax is always computed in float for numerical stability. template using BaseType = float; @@ -80,6 +89,9 @@ struct SoftmaxPreprocess /// Sigmoid: applies sigmoid(score) for topK selection (no bias). struct SigmoidPreprocess { + /// Elementwise — see NoOpPreprocess. + static constexpr bool IsElementwise = true; + /// BaseType: sigmoid is computed in float for numerical stability. template using BaseType = float; @@ -107,6 +119,9 @@ struct SigmoidPreprocess /// Used by DeepSeek-style routing where expert selection is based on biased sigmoid scores. struct SigmoidBiasPreprocess { + /// Elementwise — see NoOpPreprocess. + static constexpr bool IsElementwise = true; + /// BaseType: sigmoid is computed in float for numerical stability. template using BaseType = float; @@ -290,6 +305,11 @@ struct ScaledSumNormalizePostprocess template struct TopKExpertSelect { + /// Expose the wrapped policies so kernels with a non-standard work partitioning + /// (e.g. routingIndicesCoopBlockKernel) can apply them piecewise. + using PreprocessPolicy = PreprocessPolicy_; + using PostprocessPolicy = PostprocessPolicy_; + /// BaseType: delegated to the preprocess policy. template using BaseType = typename PreprocessPolicy_::template BaseType; @@ -371,6 +391,8 @@ static constexpr int MaxNumTokensSingleClusterScores = NumBlocksPerCluster * Num static constexpr int BlockKernelMaxNumTokens = 4; static constexpr int DynBlockKernelMaxNumTokens = 16; static constexpr int DynBlockKernelMaxNumExperts = 256; +// Cooperative block kernel: one thread per expert, so at most 1024 experts (1 CUDA block). +static constexpr int CoopBlockKernelMaxNumExperts = 1024; //////////////////////////////////////////////////////////////////////////////////////////////////// diff --git a/cpp/tensorrt_llm/kernels/trtllmGenKernels/blockScaleMoe/routing/RoutingKernel.h b/cpp/tensorrt_llm/kernels/trtllmGenKernels/blockScaleMoe/routing/RoutingKernel.h index 3ef3dd05cb43..f4569238fad1 100644 --- a/cpp/tensorrt_llm/kernels/trtllmGenKernels/blockScaleMoe/routing/RoutingKernel.h +++ b/cpp/tensorrt_llm/kernels/trtllmGenKernels/blockScaleMoe/routing/RoutingKernel.h @@ -377,6 +377,12 @@ struct KernelParams : public KernelParamsBase Runner::getWorkspaceSizeInBytes(MoERunnerArgs const } std::vector Runner::getValidConfigIndices(int32_t topK, int32_t hiddenSize, int32_t intermediateSize, - int32_t numLocalExperts, int32_t numTokens, int32_t validIntermediateSize, int32_t validHiddenSize) const + int32_t numLocalExperts, int32_t numTokens, int32_t validHiddenSize, int32_t validIntermediateSize) const { std::vector validIndices; diff --git a/cpp/tensorrt_llm/nanobind/batch_manager/bindings.cpp b/cpp/tensorrt_llm/nanobind/batch_manager/bindings.cpp index f1c57ec064ac..6da2739ed681 100644 --- a/cpp/tensorrt_llm/nanobind/batch_manager/bindings.cpp +++ b/cpp/tensorrt_llm/nanobind/batch_manager/bindings.cpp @@ -32,7 +32,9 @@ #include "tensorrt_llm/runtime/torchView.h" #include +#include #include +#include #include #include #include @@ -40,6 +42,7 @@ #include #include #include +#include #include #include @@ -129,6 +132,20 @@ void initBindings(nb::module_& m) return GenLlmReq::VecTokens(tokens.begin() + begin, tokens.begin() + end); }, nb::arg("beam"), nb::arg("begin"), nb::arg("end")) + // Zero-copy read-only int32 view of a beam's tokens (aliases the internal buffer; no + // per-token PyLong allocation, unlike get_tokens). reference_internal keeps this request + // alive for the view's lifetime. Consume synchronously — the view is invalidated if the + // token buffer is mutated/reallocated (e.g. by add_new_token). Used on the KV-cache-v2 + // block-reuse hot path where it feeds an int32 fast path with no Python round-trip. + .def( + "get_tokens_view", + [](GenLlmReq const& self, GenLlmReq::SizeType32 beam) + { + auto const& tokens = self.getTokens(beam); + return nb::ndarray, nb::c_contig>( + tokens.data(), {tokens.size()}); + }, + nb::arg("beam"), nb::rv_policy::reference_internal) .def("get_last_tokens", nb::overload_cast(&GenLlmReq::getLastTokens), nb::arg("beam")) .def("get_last_tokens", nb::overload_cast<>(&GenLlmReq::getLastTokens)) .def("get_beam_width_by_iter", &GenLlmReq::getBeamWidthByIter, nb::arg("for_next_iteration") = false) @@ -577,6 +594,153 @@ void initBindings(nb::module_& m) "Add new tokens to multiple LLM requests. The tokens vector should contain tokens for beam beam_idx of all " "requests in order."); + m.def( + "prepare_encoder_decoder_inputs", + [](std::vector> const& contextRequests, + std::vector> const& generationRequests, at::Tensor const& inputIds, + at::Tensor const& positionIds, at::Tensor const& sequenceLengths, at::Tensor const& promptLengths, + at::Tensor const& cachedTokenLengths, at::Tensor const& kvLengths, at::Tensor const& encoderKvLengths, + at::Tensor const& previousBatchIndices, SizeType32 positionIdOffset) + { + auto checkIntBuffer = [](at::Tensor const& tensor, char const* name) + { + TLLM_CHECK_WITH_INFO(tensor.device().is_cpu(), "%s must be a CPU tensor", name); + TLLM_CHECK_WITH_INFO(tensor.scalar_type() == at::kInt, "%s must have torch.int32 dtype", name); + TLLM_CHECK_WITH_INFO(tensor.is_contiguous(), "%s must be contiguous", name); + }; + checkIntBuffer(inputIds, "input_ids"); + checkIntBuffer(positionIds, "position_ids"); + checkIntBuffer(sequenceLengths, "sequence_lengths"); + checkIntBuffer(promptLengths, "prompt_lengths"); + checkIntBuffer(cachedTokenLengths, "cached_token_lengths"); + checkIntBuffer(kvLengths, "kv_lengths"); + checkIntBuffer(encoderKvLengths, "encoder_kv_lengths"); + checkIntBuffer(previousBatchIndices, "previous_batch_indices"); + + auto const numSequences = contextRequests.size() + generationRequests.size(); + TLLM_CHECK_WITH_INFO(sequenceLengths.numel() >= static_cast(numSequences), + "sequence_lengths capacity is smaller than the batch"); + TLLM_CHECK_WITH_INFO(promptLengths.numel() >= static_cast(numSequences), + "prompt_lengths capacity is smaller than the batch"); + TLLM_CHECK_WITH_INFO(cachedTokenLengths.numel() >= static_cast(numSequences), + "cached_token_lengths capacity is smaller than the batch"); + TLLM_CHECK_WITH_INFO(kvLengths.numel() >= static_cast(numSequences), + "kv_lengths capacity is smaller than the batch"); + TLLM_CHECK_WITH_INFO(encoderKvLengths.numel() >= static_cast(numSequences), + "encoder_kv_lengths capacity is smaller than the batch"); + TLLM_CHECK_WITH_INFO(previousBatchIndices.numel() >= static_cast(generationRequests.size()), + "previous_batch_indices capacity is smaller than the generation batch"); + + auto* inputIdsPtr = inputIds.data_ptr(); + auto* positionIdsPtr = positionIds.data_ptr(); + auto* sequenceLengthsPtr = sequenceLengths.data_ptr(); + auto* promptLengthsPtr = promptLengths.data_ptr(); + auto* cachedTokenLengthsPtr = cachedTokenLengths.data_ptr(); + auto* kvLengthsPtr = kvLengths.data_ptr(); + auto* encoderKvLengthsPtr = encoderKvLengths.data_ptr(); + auto* previousBatchIndicesPtr = previousBatchIndices.data_ptr(); + + std::vector requestIds; + std::vector encoderSequenceLengths; + std::vector encoderCachedTokenLengths; + requestIds.reserve(numSequences); + encoderSequenceLengths.reserve(numSequences); + encoderCachedTokenLengths.reserve(numSequences); + + SizeType32 numTokens{0}; + SizeType32 numContextTokens{0}; + SizeType32 numPreviousBatchRequests{0}; + SizeType32 cachedKvTokens{0}; + SizeType32 contextKvTokens{0}; + SizeType32 generationKvTokens{0}; + SizeType32 maxKvLength{0}; + SizeType32 contextEncoderKvTokens{0}; + SizeType32 generationEncoderKvTokens{0}; + SizeType32 maxEncoderKvLength{0}; + for (auto const& request : contextRequests) + { + auto const sequenceIdx = requestIds.size(); + auto const begin = request->getContextCurrentPosition(); + auto const chunkSize = request->getContextChunkSize(); + auto const& tokens = request->getTokens(0); + TLLM_CHECK_WITH_INFO(begin + chunkSize <= static_cast(tokens.size()), + "Context chunk exceeds the request token count"); + TLLM_CHECK_WITH_INFO(inputIds.numel() >= static_cast(numTokens + chunkSize), + "input_ids capacity is smaller than the packed context"); + TLLM_CHECK_WITH_INFO(positionIds.numel() >= static_cast(numTokens + chunkSize), + "position_ids capacity is smaller than the packed context"); + + std::copy_n(tokens.data() + begin, chunkSize, inputIdsPtr + numTokens); + std::iota(positionIdsPtr + numTokens, positionIdsPtr + numTokens + chunkSize, begin + positionIdOffset); + sequenceLengthsPtr[sequenceIdx] = chunkSize; + promptLengthsPtr[sequenceIdx] = chunkSize; + cachedTokenLengthsPtr[sequenceIdx] = begin; + auto const kvLength = begin + chunkSize; + kvLengthsPtr[sequenceIdx] = kvLength; + auto const encoderKvLength = request->getEncoderOutputLen(); + encoderKvLengthsPtr[sequenceIdx] = encoderKvLength; + cachedKvTokens += begin; + contextKvTokens += kvLength; + maxKvLength = std::max(maxKvLength, kvLength); + contextEncoderKvTokens += encoderKvLength; + maxEncoderKvLength = std::max(maxEncoderKvLength, encoderKvLength); + numTokens += chunkSize; + numContextTokens += chunkSize; + + requestIds.push_back(request->mRequestId); + encoderSequenceLengths.push_back(encoderKvLength); + encoderCachedTokenLengths.push_back(0); + } + + bool sawDummyRequest{false}; + for (auto const& request : generationRequests) + { + auto const sequenceIdx = requestIds.size(); + auto const isDummy = request->isDummyRequest(); + sawDummyRequest = sawDummyRequest || isDummy; + TLLM_CHECK_WITH_INFO( + isDummy || !sawDummyRequest, "CUDA graph dummy requests must follow real generation requests"); + TLLM_CHECK_WITH_INFO( + positionIds.numel() > numTokens, "position_ids capacity is smaller than the packed batch"); + + auto const pastSeenTokens = request->getMaxBeamNumTokens() - (isDummy ? 1 : 0); + positionIdsPtr[numTokens] = pastSeenTokens + positionIdOffset; + sequenceLengthsPtr[sequenceIdx] = 1; + promptLengthsPtr[sequenceIdx] = request->mPromptLen; + cachedTokenLengthsPtr[sequenceIdx] = pastSeenTokens; + auto const kvLength = pastSeenTokens + 1; + kvLengthsPtr[sequenceIdx] = kvLength; + auto const encoderKvLength = request->getEncoderOutputLen(); + encoderKvLengthsPtr[sequenceIdx] = encoderKvLength; + cachedKvTokens += pastSeenTokens; + generationKvTokens += kvLength; + maxKvLength = std::max(maxKvLength, kvLength); + generationEncoderKvTokens += encoderKvLength; + maxEncoderKvLength = std::max(maxEncoderKvLength, encoderKvLength); + ++numTokens; + + if (!isDummy) + { + TLLM_CHECK_WITH_INFO( + request->mSeqSlot.has_value(), "A real generation request must have a sequence slot"); + previousBatchIndicesPtr[numPreviousBatchRequests++] = request->mSeqSlot.value(); + } + + requestIds.push_back(request->mRequestId); + encoderSequenceLengths.push_back(0); + encoderCachedTokenLengths.push_back(encoderKvLength); + } + + return std::make_tuple(requestIds, encoderSequenceLengths, encoderCachedTokenLengths, numTokens, + numContextTokens, numPreviousBatchRequests, cachedKvTokens, contextKvTokens, generationKvTokens, + maxKvLength, contextEncoderKvTokens, generationEncoderKvTokens, maxEncoderKvLength); + }, + nb::arg("context_requests"), nb::arg("generation_requests"), nb::arg("input_ids"), nb::arg("position_ids"), + nb::arg("sequence_lengths"), nb::arg("prompt_lengths"), nb::arg("cached_token_lengths"), nb::arg("kv_lengths"), + nb::arg("encoder_kv_lengths"), nb::arg("previous_batch_indices"), nb::arg("position_id_offset") = 0, + nb::call_guard(), + "Prepare the persistent CPU input buffers for a simple encoder-decoder batch."); + m.def( "make_decoding_batch_input", [](tb::DecoderInputBuffers& decoderInputBuffers, runtime::decoder::DecoderState& decoderState, diff --git a/cpp/tensorrt_llm/nanobind/batch_manager/kvCacheManager.cpp b/cpp/tensorrt_llm/nanobind/batch_manager/kvCacheManager.cpp index 559934e1ef72..222f2e528cc3 100644 --- a/cpp/tensorrt_llm/nanobind/batch_manager/kvCacheManager.cpp +++ b/cpp/tensorrt_llm/nanobind/batch_manager/kvCacheManager.cpp @@ -748,6 +748,9 @@ void tb::BasePeftCacheManagerBindings::initBindings(nb::module_& m) nb::call_guard()) .def("is_task_cached_device", &tb::PeftCacheManager::isTaskCachedDevice, nb::arg("taskId"), nb::call_guard()) // ; + .def("configure_data_type", &tb::PeftCacheManager::configureDataType, nb::arg("data_type"), + nb::call_guard()) + .def_prop_ro("data_type", &tb::PeftCacheManager::getDataType) .def("ensure_batch_map_task_id", &tb::PeftCacheManager::ensureBatchMapTaskId, nb::arg("context_requests"), nb::arg("generation_requests"), nb::arg("reset_gpu_cache") = false, nb::call_guard()); diff --git a/cpp/tensorrt_llm/nanobind/batch_manager/kvCacheManagerV2.cpp b/cpp/tensorrt_llm/nanobind/batch_manager/kvCacheManagerV2.cpp index a5fcbdd22c2e..dcebcc4e6446 100644 --- a/cpp/tensorrt_llm/nanobind/batch_manager/kvCacheManagerV2.cpp +++ b/cpp/tensorrt_llm/nanobind/batch_manager/kvCacheManagerV2.cpp @@ -51,6 +51,7 @@ #include #include #include +#include #include #include #include @@ -61,11 +62,14 @@ namespace kv = tensorrt_llm::batch_manager::kv_cache_manager_v2; namespace tensorrt_llm::nanobind::batch_manager { -// Helper: convert a Python iterable of int|bytes to vector. -// nanobind's variant caster can't auto-convert bytes → DigestToken. -static std::vector castTokenIterable(nb::handle tokens) +// Helper: convert a Python iterable of int|bytes to a token vector, and report +// whether it is digest-free (knownNoDigest). A normal int becomes a 31-bit token id +// (range-checked); 32 bytes become a digest. The digest-free flag is computed for free +// here while iterating, feeding the hashing fast path. Python contract: `int | bytes(32)`. +static std::pair, bool> castTokenIterable(nb::handle tokens) { std::vector vec; + bool knownNoDigest = true; for (auto item : nb::cast(tokens)) { if (nb::isinstance(item)) @@ -75,16 +79,54 @@ static std::vector castTokenIterable(nb::handle tokens) { throw std::invalid_argument("Token bytes must have length kDIGEST_LEN"); } - kv::Digest d; - std::memcpy(d.data(), b.c_str(), kv::kDIGEST_LEN); - vec.emplace_back(kv::DigestToken(d)); + kv::Digest digest; + std::memcpy(digest.data(), b.c_str(), kv::kDIGEST_LEN); + vec.emplace_back(digest); + knownNoDigest = false; } else { - vec.emplace_back(nb::cast(item)); + auto const tokenId = nb::cast(item); + if (tokenId < 0 || tokenId > static_cast(kv::TokenIdExt::kMaxValue)) + { + throw std::invalid_argument("Token id out of range [0, 2^31)"); + } + vec.emplace_back(static_cast(tokenId)); } } - return vec; + return {std::move(vec), knownNoDigest}; +} + +// Zero-copy-friendly token ingestion. Invokes fn(TokenSpan, knownNoDigest) with a non-owning +// view of the tokens; fn must consume it synchronously (it may release the GIL — the backing +// buffer must outlive the call). +// +// Fast path: a contiguous 1-D int32 CPU buffer (numpy view, memoryview, array.array, torch CPU +// tensor) is reinterpret_cast to TokenIdExt const* — no copy, no per-token boxing — since a normal +// token id is bit-identical to a 4-byte TokenIdExt (see tokenIdExt.h). convert=false keeps it +// strictly zero-copy; a non-int32 / non-contiguous input falls through. +// +// Fallback: any int|bytes(32) iterable via castTokenIterable — the multimodal/digest path. +template +static auto withTokens(nb::handle tokens, Fn&& fn) +{ + nb::ndarray, nb::c_contig, nb::device::cpu> arr; + if (nb::try_cast(tokens, arr, /*convert=*/false)) + { + auto const count = static_cast(arr.shape(0)); + auto const* raw = arr.data(); + // Every element must be a normal id (high/digest bit clear); a negative int32 would look + // like a pooled digest (see tokenIdExt.h) and corrupt the hash. Digests never reach here — + // they take the fallback — so knownNoDigest=true holds; this asserts it in debug builds. + TLLM_CHECK_DEBUG_WITH_INFO(std::all_of(raw, raw + count, [](int32_t id) { return id >= 0; }), + "token id must be in [0, 2^31) for the zero-copy path"); + // Sound reinterpret: TokenIdExt is bit-identical to its int32 id (tokenIdExt.h) and this + // buffer is only ever read const (never written through the alias). + auto const* data = reinterpret_cast(raw); + return fn(kv::TokenSpan{data, count}, /*knownNoDigest=*/true); + } + auto [vec, knownNoDigest] = castTokenIterable(tokens); + return fn(kv::toSpan(vec), knownNoDigest); } static kv::TypedVec typedPoolSizeList(std::vector const& slotSizeList) @@ -165,14 +207,14 @@ static nb::list tokenList(std::vector const& tokens) nb::list result; for (auto const& tok : tokens) { - if (auto* id = std::get_if(&tok)) + if (!tok.isDigest()) { - result.append(*id); + result.append(tok.tokenId()); } else { - auto const& d = std::get(tok); - result.append(nb::bytes(reinterpret_cast(d.data()), d.size())); + auto const& digest = tok.digest(); + result.append(nb::bytes(reinterpret_cast(digest.data()), digest.size())); } } return result; @@ -499,6 +541,85 @@ static kv::ReuseScope castReuseScope(nb::object reuseScope) "reuse_scope must be None, ReuseScope, an int lora_task_id, or an object with lora_id and salt"); } +class EventManagerTestBlock +{ +public: + EventManagerTestBlock(kv::SharedPtr block_, std::vector> pages_) + : block(std::move(block_)) + , pages(std::move(pages_)) + { + } + + ~EventManagerTestBlock() + { + try + { + close(); + } + catch (...) + { + } + } + + void close() + { + for (auto const& page : pages) + { + block->unlinkPage(page->lifeCycle, page.get()); + } + pages.clear(); + } + + kv::SharedPtr block; + std::vector> pages; +}; + +// Lazy Python iterator over a token sequence's blockchain keys. Each __next__ pulls +// one key from the C++ generator (one SHA-256 hash), so a caller that stops early +// (e.g. on the first cache-key mismatch) skips the remaining hashing. Yields +// (token_block, key) pairs matching Python's sequence_to_blockchain_keys: root +// ([], reuseScope digest) first, then one per tokensPerBlock chunk. +class BlockchainKeyIterator +{ +public: + BlockchainKeyIterator( + int tokensPerBlock, kv::ReuseScope const& reuseScope, std::vector tokens, bool knownNoDigest) + : mTokens(std::move(tokens)) + , mGen(kv::sequenceToBlockchainKeys(tokensPerBlock, reuseScope, mTokens.data(), mTokens.size(), knownNoDigest)) + { + } + + nb::object next() + { + std::optional const step = mGen(); + if (!step) + { + throw nb::stop_iteration(); + } + nb::bytes keyBytes(reinterpret_cast(step->key.data()), step->key.size()); + // The step carries its token range, so the root naturally yields the empty [0,0) block. + std::vector block(mTokens.begin() + static_cast(step->tokens.beg), + mTokens.begin() + static_cast(step->tokens.end)); + return nb::make_tuple(tokenList(block), std::move(keyBytes)); + } + +private: + // Concrete generator (closure) type — sequenceToBlockchainKeys's return type is + // fixed, so decltype deduces it directly, avoiding std::function's type erasure and + // per-iterator heap allocation (the closure captures a 32-byte key, over the SBO limit). + using KeyGen = decltype(kv::sequenceToBlockchainKeys(std::declval(), std::declval(), + std::declval(), std::declval(), std::declval())); + + std::vector mTokens; // owned; mGen holds a stable pointer into it + KeyGen mGen; // declared after mTokens (init order) +}; + +// nanobind move-constructs the iterator from the factory return; that must stay valid, +// since mGen captures mTokens.data() and vector-move preserves the buffer address. Adding +// a non-movable member would silently force a copy and dangle that pointer — forbid it. +static_assert(std::is_move_constructible_v, + "BlockchainKeyIterator must be move-constructible (mGen captures mTokens.data())"); + void KvCacheManagerV2Bindings::initBindings(nb::module_& m) { // Export the C++ debug mode as an immutable Python bool snapshot. @@ -512,6 +633,43 @@ void KvCacheManagerV2Bindings::initBindings(nb::module_& m) static nb::object sLogicError = nb::exception(m, "LogicError"); static nb::object sResourceBusyError = nb::exception(m, "ResourceBusyError"); static nb::object sOutOfPagesError = nb::exception(m, "OutOfPagesError"); + static nb::object sCuError = nb::exception(m, "CuError"); + // Default attribute so the class mirrors the pure-Python CuError surface. + sCuError.attr("error_code") = nb::none(); + + // Translate kv::CuError so the Python instance carries the numeric CUDA + // error code (mirrors the pure-Python CuError.error_code). Registered after + // the nb::exception auto-translator so it is tried first. + nb::register_exception_translator( + [](std::exception_ptr const& p, void*) + { + try + { + if (p) + { + std::rethrow_exception(p); + } + } + catch (kv::CuError const& e) + { + nb::object inst = sCuError(nb::str(e.what())); + // Match Python's error_code type (cuda.bindings.driver.CUresult) + // when available; fall back to a plain int otherwise. + nb::object code; + try + { + nb::object cuResult = nb::module_::import_("cuda.bindings.driver").attr("CUresult"); + code = cuResult(static_cast(e.errorCode)); + } + catch (nb::python_error const&) + { + PyErr_Clear(); + code = nb::cast(static_cast(e.errorCode)); + } + inst.attr("error_code") = code; + PyErr_SetObject(sCuError.ptr(), inst.ptr()); + } + }); // Map kv::AssertionError to Python's builtin AssertionError so shared tests see // the same exception type as the pure-Python backend (which uses `assert`). @@ -522,7 +680,9 @@ void KvCacheManagerV2Bindings::initBindings(nb::module_& m) try { if (p) + { std::rethrow_exception(p); + } } catch (kv::AssertionError const& e) { @@ -1298,7 +1458,8 @@ void KvCacheManagerV2Bindings::initBindings(nb::module_& m) nb::list layers, float maxUtilForResume, bool enablePartialReuse, std::optional typicalStep, std::vector constraints, std::optional> initialPoolRatio, - std::optional swaScratchReuse, bool commitMinSnapshot, bool enableStats) + std::optional swaScratchReuse, bool commitMinSnapshot, bool enableStats, + bool textOnly) { new (cfg) kv::KVCacheManagerConfig(); cfg->tokensPerBlock = tokensPerBlock; @@ -1319,6 +1480,7 @@ void KvCacheManagerV2Bindings::initBindings(nb::module_& m) cfg->swaScratchReuse = std::move(swaScratchReuse); cfg->commitMinSnapshot = commitMinSnapshot; cfg->enableStats = enableStats; + cfg->textOnly = textOnly; // Mirror Python's __post_init__: validate at construction. Config-integrity // failures raise AssertionError (translated below). cfg->validate(); @@ -1327,7 +1489,7 @@ void KvCacheManagerV2Bindings::initBindings(nb::module_& m) nb::arg("max_util_for_resume") = 0.97f, nb::arg("enable_partial_reuse") = true, nb::arg("typical_step") = std::nullopt, nb::arg("constraints") = std::vector{}, nb::arg("initial_pool_ratio").none() = std::nullopt, nb::arg("swa_scratch_reuse").none() = std::nullopt, - nb::arg("commit_min_snapshot") = false, nb::arg("enable_stats") = true) + nb::arg("commit_min_snapshot") = false, nb::arg("enable_stats") = true, nb::arg("text_only") = false) .def_rw("tokens_per_block", &kv::KVCacheManagerConfig::tokensPerBlock) .def_rw("cache_tiers", &kv::KVCacheManagerConfig::cacheTiers) .def_rw("layers", &kv::KVCacheManagerConfig::layers) @@ -1339,6 +1501,7 @@ void KvCacheManagerV2Bindings::initBindings(nb::module_& m) .def_rw("swa_scratch_reuse", &kv::KVCacheManagerConfig::swaScratchReuse) .def_rw("commit_min_snapshot", &kv::KVCacheManagerConfig::commitMinSnapshot) .def_rw("enable_stats", &kv::KVCacheManagerConfig::enableStats) + .def_rw("text_only", &kv::KVCacheManagerConfig::textOnly) .def_prop_ro("enable_swa_scratch_reuse", &kv::KVCacheManagerConfig::enableSwaScratchReuse) .def("validate", &kv::KVCacheManagerConfig::validate) DEF_COPY(kv::KVCacheManagerConfig); @@ -1378,7 +1541,6 @@ void KvCacheManagerV2Bindings::initBindings(nb::module_& m) "commit", [](kv::KvCache& self, nb::object acceptedInputTokens, nb::object beamSearchIndices, bool isEnd) { - auto vec = castTokenIterable(acceptedInputTokens); if (!beamSearchIndices.is_none()) { PyErr_SetString(PyExc_AssertionError, "beam_search_indices must be None"); @@ -1386,8 +1548,17 @@ void KvCacheManagerV2Bindings::initBindings(nb::module_& m) } // Note: an empty token list with is_end=True must still stop committing, // so we do not early-return on empty; commit() handles it. - nb::gil_scoped_release release; - self.commit(vec, isEnd); + withTokens(acceptedInputTokens, + [&](kv::TokenSpan view, bool knownNoDigest) + { + // commit() sources knownNoDigest from the KvCache's text_only flag. Guard + // that claim against the actual tokens: a text_only sequence committing a + // digest would silently corrupt the block-key hash. + TLLM_CHECK_WITH_INFO(!(self.textOnly() && !knownNoDigest), + "commit() received a digest token on a text_only sequence — hashing would be corrupted"); + nb::gil_scoped_release release; + self.commit(view, isEnd); + }); }, nb::arg("accepted_input_tokens"), nb::arg("beam_search_indices").none() = nb::none(), nb::arg("is_end") = false) @@ -1425,6 +1596,8 @@ void KvCacheManagerV2Bindings::initBindings(nb::module_& m) .def_prop_ro("has_scratch_slots", &kv::KvCache::hasScratchSlots) .def_prop_rw("enable_swa_scratch_reuse", &kv::KvCache::isSwaScratchReuseEnabled, [](kv::KvCache& self, bool enable) { self.setEnableSwaScratchReuse(enable); }) + .def_prop_rw( + "text_only", &kv::KvCache::textOnly, [](kv::KvCache& self, bool textOnly) { self.setTextOnly(textOnly); }) .def("supports_index_mode", &kv::KvCache::supportsIndexMode, nb::arg("mode")) .def_prop_ro("status", [](kv::KvCache const& kvc) { return kvc.status(); }) .def_prop_ro("is_active", &kv::KvCache::isActive) @@ -1506,6 +1679,79 @@ void KvCacheManagerV2Bindings::initBindings(nb::module_& m) // ---- Introspection ------------------------------------------------------- auto mIntrospection = m.def_submodule("_introspection", "KV cache manager v2 introspection helpers"); + + nb::class_(mIntrospection, "TestBlock").def("close", &EventManagerTestBlock::close); + mIntrospection.def( + "make_test_block", + [](kv::KvCacheManager& manager, nb::object tokenObject, std::vector coveragePerLc, nb::object parentObject, + nb::object reuseScopeObject) + { + auto [tokens, knownNoDigest] = castTokenIterable(tokenObject); + if (tokens.empty()) + { + throw std::invalid_argument("make_test_block requires at least one token"); + } + if (coveragePerLc.size() != static_cast(manager.lifeCycles().size().value())) + { + throw std::invalid_argument("coverage_per_lc length must match the manager lifecycle count"); + } + for (int coverage : coveragePerLc) + { + if (coverage < 0 || coverage > static_cast(tokens.size())) + { + throw std::invalid_argument("lifecycle coverage must be between zero and the block token count"); + } + } + + kv::NodeBase* parent = nullptr; + if (parentObject.is_none()) + { + parent = &manager.radixTree().addOrGetExisting(castReuseScope(std::move(reuseScopeObject))); + } + else + { + parent = nb::cast(parentObject).block.get(); + } + auto block = kv::addOrGetExistingBlock(parent, std::move(tokens), knownNoDigest); + + kv::TypedVec counts(manager.lifeCycles().size(), 0); + for (kv::LifeCycleId lifeCycle{0}; lifeCycle < manager.lifeCycles().size(); ++lifeCycle) + { + counts[lifeCycle] = coveragePerLc.at(lifeCycle.value()) > 0 ? 1 : 0; + } + auto slots = manager.storage().newGpuSlots(counts); + std::vector> pages; + pages.reserve(coveragePerLc.size()); + for (kv::LifeCycleId lifeCycle{0}; lifeCycle < manager.lifeCycles().size(); ++lifeCycle) + { + int const coverage = coveragePerLc.at(lifeCycle.value()); + if (coverage == 0) + { + continue; + } + auto page = kv::makeShared( + &manager.storage(), block, lifeCycle, kv::kGpuLevel, coverage, kv::kPriorityDefault); + page->setSlot(slots[lifeCycle].front()); + block->storage[lifeCycle] = page.get(); + pages.push_back(std::move(page)); + } + return std::make_unique(std::move(block), std::move(pages)); + }, + nb::arg("manager"), nb::arg("tokens"), nb::arg("coverage_per_lc"), nb::arg("parent").none() = nb::none(), + nb::arg("reuse_scope").none() = nb::none(), nb::keep_alive<0, 1>(), nb::keep_alive<0, 4>()); + mIntrospection.def( + "test_block_key", [](EventManagerTestBlock const& block) { return digestBytes(block.block->key); }, + nb::arg("block")); + mIntrospection.def( + "event_manager_add_stored_block", + [](kv::EventManager& eventManager, EventManagerTestBlock const& block) + { eventManager.addStoredBlock(*block.block); }, + nb::arg("event_manager"), nb::arg("block"), nb::call_guard()); + mIntrospection.def( + "event_manager_add_stored_life_cycle", + [](kv::EventManager& eventManager, EventManagerTestBlock const& block, int lifeCycleId) + { eventManager.addStoredLifeCycle(*block.block, kv::LifeCycleId{lifeCycleId}); }, + nb::arg("event_manager"), nb::arg("block"), nb::arg("life_cycle_id"), nb::call_guard()); nb::class_(mIntrospection, "StorageStatistics") .def_prop_ro("slot_sizes", [](kv::StorageStatistics const& self) { return self.slotSizes.raw(); }) .def_ro("total", &kv::StorageStatistics::total) @@ -1617,26 +1863,28 @@ void KvCacheManagerV2Bindings::initBindings(nb::module_& m) [](kv::KvCacheManager& manager, nb::object reuseScope, nb::object tokens, int lcId, bool enablePartial) { auto rs = castReuseScope(reuseScope); - auto vec = castTokenIterable(tokens); int numTokens = 0; std::vector>> pages; - { - nb::gil_scoped_release release; - auto matchResult = manager.radixTree().match(rs, vec, enablePartial); - numTokens = matchResult.numTokens; - kv::LifeCycleId lc{lcId}; - pages.reserve(matchResult.blocks.stdSize()); - for (auto* block : matchResult.blocks) + withTokens(tokens, + [&](kv::TokenSpan view, bool knownNoDigest) { - auto* page = block->getPage(lc); - if (page == nullptr) + nb::gil_scoped_release release; + auto matchResult = manager.radixTree().match(rs, view, knownNoDigest, enablePartial); + numTokens = matchResult.numTokens; + kv::LifeCycleId lc{lcId}; + pages.reserve(matchResult.blocks.stdSize()); + for (auto* block : matchResult.blocks) { - pages.emplace_back(std::nullopt); - continue; + auto* page = block->storage.at(lc); + if (page == nullptr) + { + pages.emplace_back(std::nullopt); + continue; + } + int const slotId = page->slotId().value(); + pages.emplace_back(std::make_pair(slotId, page->numTokensInBlock)); } - pages.emplace_back(std::make_pair(page->slotId().value(), page->numTokensInBlock)); - } - } + }); return std::make_tuple(numTokens, std::move(pages)); }, nb::arg("manager"), nb::arg("reuse_scope"), nb::arg("tokens"), nb::arg("lc_id"), @@ -1648,24 +1896,25 @@ void KvCacheManagerV2Bindings::initBindings(nb::module_& m) [](kv::KvCacheManager& manager, nb::object reuseScope, nb::object tokens, int lcId, bool enablePartial) { auto rs = castReuseScope(reuseScope); - auto vec = castTokenIterable(tokens); int numTokens = 0; std::vector> counts; - { - nb::gil_scoped_release release; - auto matchResult = manager.radixTree().match(rs, vec, enablePartial); - numTokens = matchResult.numTokens; - kv::LifeCycleId lc{lcId}; - counts.reserve(matchResult.blocks.stdSize()); - for (auto* block : matchResult.blocks) + withTokens(tokens, + [&](kv::TokenSpan view, bool knownNoDigest) { - auto* page = block->storage.at(lc); - if (page == nullptr) - counts.emplace_back(std::nullopt); - else - counts.emplace_back(page->plannedDropCount); - } - } + nb::gil_scoped_release release; + auto matchResult = manager.radixTree().match(rs, view, knownNoDigest, enablePartial); + numTokens = matchResult.numTokens; + kv::LifeCycleId lc{lcId}; + counts.reserve(matchResult.blocks.stdSize()); + for (auto* block : matchResult.blocks) + { + auto* page = block->storage.at(lc); + if (page == nullptr) + counts.emplace_back(std::nullopt); + else + counts.emplace_back(page->plannedDropCount); + } + }); return std::make_tuple(numTokens, std::move(counts)); }, nb::arg("manager"), nb::arg("reuse_scope"), nb::arg("tokens"), nb::arg("lc_id"), @@ -1753,38 +2002,53 @@ void KvCacheManagerV2Bindings::initBindings(nb::module_& m) "create_kv_cache", [](std::shared_ptr self, nb::object reuseScopeObj, nb::object inputTokens, std::optional id, nb::object customPriorityCallback, - std::optional expectedPromptLength) + std::optional expectedPromptLength, std::optional textOnly) { kv::ReuseScope reuseScope = castReuseScope(std::move(reuseScopeObj)); - std::vector tokens; - bool const hasInputTokens = !inputTokens.is_none(); - if (!inputTokens.is_none()) - { - tokens = castTokenIterable(inputTokens); - } - if (!expectedPromptLength.has_value() && hasInputTokens) + kv::KvCache::PriorityCb priorityCb = castPriorityCallback(*self, std::move(customPriorityCallback)); + if (inputTokens.is_none()) { - expectedPromptLength = static_cast(tokens.size()); + nb::gil_scoped_release release; + return self->createKvCache(std::move(reuseScope), kv::TokenSpan{}, id, std::move(priorityCb), + expectedPromptLength, textOnly); } - kv::KvCache::PriorityCb priorityCb = castPriorityCallback(*self, std::move(customPriorityCallback)); - nb::gil_scoped_release release; - return self->createKvCache( - std::move(reuseScope), tokens, id, std::move(priorityCb), expectedPromptLength); + return withTokens(inputTokens, + [&](kv::TokenSpan view, bool knownNoDigest) + { + // Guard the text_only claim against the actual input tokens (see commit()). + bool const resolvedTextOnly = textOnly.value_or(self->textOnly()); + TLLM_CHECK_WITH_INFO(!(resolvedTextOnly && !knownNoDigest), + "create_kv_cache received digest input_tokens on a text_only sequence — hashing would be " + "corrupted"); + std::optional promptLen = expectedPromptLength; + if (!promptLen.has_value() && view.size() != 0) + { + promptLen = static_cast(view.size()); + } + nb::gil_scoped_release release; + return self->createKvCache( + std::move(reuseScope), view, id, std::move(priorityCb), promptLen, textOnly); + }); }, nb::arg("reuse_scope") = nb::none(), nb::arg("input_tokens") = nb::none(), nb::arg("id") = std::nullopt, - nb::arg("custom_priority_callback") = nb::none(), nb::arg("expected_prompt_length") = std::nullopt) + nb::arg("custom_priority_callback") = nb::none(), nb::arg("expected_prompt_length") = std::nullopt, + nb::arg("text_only") = std::nullopt) .def( "probe_reuse", [](std::shared_ptr self, nb::object reuseScopeObj, nb::object inputTokens) { kv::ReuseScope reuseScope = castReuseScope(std::move(reuseScopeObj)); - std::vector tokens; - if (!inputTokens.is_none()) + if (inputTokens.is_none()) { - tokens = castTokenIterable(inputTokens); + nb::gil_scoped_release release; + return self->probeReuse(std::move(reuseScope), kv::TokenSpan{}, /*knownNoDigest=*/true); } - nb::gil_scoped_release release; - return self->probeReuse(std::move(reuseScope), tokens); + return withTokens(inputTokens, + [&](kv::TokenSpan view, bool knownNoDigest) + { + nb::gil_scoped_release release; + return self->probeReuse(std::move(reuseScope), view, knownNoDigest); + }); }, nb::arg("reuse_scope") = nb::none(), nb::arg("input_tokens") = nb::none()) .def("get_mem_pool_base_address", &kv::KvCacheManager::getMemPoolBaseAddress, nb::arg("layer_id"), @@ -1895,6 +2159,34 @@ void KvCacheManagerV2Bindings::initBindings(nb::module_& m) return self(baseIndices, indexMode, scratch); }, nb::arg("base_indices"), nb::arg("index_mode") = nb::none(), nb::arg("scratch") = nb::none()); + + m.def( + "gen_multimodal_cache_key_tokens", + [](int idOffset, nb::bytes multiModalDataDigest, int numTokens, int tokenOffset) + { + auto const digestSize = nb::len(multiModalDataDigest); + if (digestSize != kv::kDIGEST_LEN) + { + throw std::invalid_argument("multi_modal_data_digest must have length kDIGEST_LEN"); + } + auto const* first = reinterpret_cast(multiModalDataDigest.c_str()); + std::vector digest(first, first + digestSize); + return tokenList(kv::genMultimodalCacheKeyTokens(idOffset, digest, numTokens, tokenOffset)); + }, + nb::arg("id_offset"), nb::arg("multi_modal_data_digest"), nb::arg("num_tokens"), nb::arg("token_offset") = 0); + // Lazy iterator yielding (token_block, key) pairs; hashes one block per __next__. + nb::class_(m, "_BlockchainKeyIterator") + .def("__iter__", [](nb::handle self) { return self; }) + .def("__next__", &BlockchainKeyIterator::next); + m.def( + "sequence_to_blockchain_keys", + [](int tokensPerBlock, nb::object reuseScopeObj, nb::object tokensObj) + { + auto const rs = castReuseScope(std::move(reuseScopeObj)); + auto [vec, knownNoDigest] = castTokenIterable(tokensObj); + return BlockchainKeyIterator(tokensPerBlock, rs, std::move(vec), knownNoDigest); + }, + nb::arg("tokens_per_block"), nb::arg("reuse_scope"), nb::arg("tokens")); } } // namespace tensorrt_llm::nanobind::batch_manager diff --git a/cpp/tensorrt_llm/nanobind/thop/bindings.cpp b/cpp/tensorrt_llm/nanobind/thop/bindings.cpp index 2ac9cc671277..429110a40496 100644 --- a/cpp/tensorrt_llm/nanobind/thop/bindings.cpp +++ b/cpp/tensorrt_llm/nanobind/thop/bindings.cpp @@ -165,7 +165,7 @@ void initBindings(nb::module_& m) nb::arg("spec_bl_tree_first_sparse_mask_offset_kv").none(), nb::arg("sparse_kv_indices").none(), nb::arg("sparse_kv_offsets").none(), nb::arg("sparse_attn_indices").none(), nb::arg("sparse_attn_offsets").none(), nb::arg("sparse_attn_indices_block_size"), - nb::arg("num_sparse_topk") = std::nullopt, nb::arg("sparse_mla_topk_lens") = std::nullopt, + nb::arg("num_sparse_topk") = std::nullopt, nb::arg("sparse_attn_kv_lens") = std::nullopt, nb::arg("skip_softmax_threshold_scale_factor_prefill") = std::nullopt, nb::arg("skip_softmax_threshold_scale_factor_decode") = std::nullopt, nb::arg("skip_softmax_stat") = std::nullopt, nb::arg("cu_q_seqlens") = std::nullopt, @@ -175,13 +175,13 @@ void initBindings(nb::module_& m) nb::arg("flash_mla_num_splits") = std::nullopt, nb::arg("sage_attn_num_elts_per_blk_q") = 0, nb::arg("sage_attn_num_elts_per_blk_k") = 0, nb::arg("sage_attn_num_elts_per_blk_v") = 0, nb::arg("sage_attn_qk_int8") = false, nb::arg("num_contexts") = 0, nb::arg("num_ctx_tokens") = 0, - nb::arg("trtllm_gen_jit_warmup") = false, nb::arg("compressed_kv_cache_pool_ptr") = std::nullopt, + nb::arg("trtllm_gen_jit_warmup") = false, nb::arg("aux_kv_cache_pool_ptr") = std::nullopt, nb::arg("is_cross") = false, nb::arg("cross_kv") = std::nullopt, nb::arg("relative_attention_bias") = std::nullopt, nb::arg("relative_attention_max_distance") = 0, nb::arg("spec_decoding_target_max_draft_tokens") = std::nullopt, nb::arg("quant_scale_qkv") = std::nullopt, nb::arg("dsv4_inv_rope_cos_sin_cache") = std::nullopt, nb::arg("enable_dsv4_epilogue_fusion") = false, - nb::arg("force_prepare_spec_dec_tree_mask") = false, "Multi-head attention operation", - nb::call_guard()); + nb::arg("force_prepare_spec_dec_tree_mask") = false, nb::arg("max_num_sequences") = std::nullopt, + "Multi-head attention operation", nb::call_guard()); m.def( "get_helix_workspace_size_per_rank", diff --git a/cpp/tensorrt_llm/runtime/loraCache.cpp b/cpp/tensorrt_llm/runtime/loraCache.cpp index 36fb0363816f..6749377445b2 100644 --- a/cpp/tensorrt_llm/runtime/loraCache.cpp +++ b/cpp/tensorrt_llm/runtime/loraCache.cpp @@ -1,5 +1,5 @@ /* - * Copyright (c) 2022-2023, NVIDIA CORPORATION. All rights reserved. + * Copyright (c) 2022-2026, NVIDIA CORPORATION. All rights reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -511,6 +511,28 @@ LoraCache::LoraCache(LoraCachePageManagerConfig const& pageManagerConfig, ModelC } } +void LoraCache::setDataType(tensorrt_llm::DataType dataType) +{ + std::scoped_lock lock(mPagesMutex, mCacheMutex); + TLLM_CHECK_WITH_INFO(mCacheMap.empty(), "Cannot change LoRA cache dtype after a task has been inserted"); + if (mPageManagerConfig.getDataType() == dataType) + { + return; + } + + // Release the old allocation before creating the replacement to avoid a + // temporary peak equal to both cache allocations. + mCachePageManager.reset(); + mPageManagerConfig.setDataType(dataType); + mCachePageManager = std::make_unique(mPageManagerConfig, *mBufferManager); +} + +tensorrt_llm::DataType LoraCache::getDataType() const +{ + std::lock_guard lock(mPagesMutex); + return mPageManagerConfig.getDataType(); +} + template void LoraCache::splitTransposeCpuInner(ITensor& output, ITensor const& input, SizeType32 tpSize, SizeType32 tpRank) { @@ -562,6 +584,9 @@ std::vector LoraCache::copyToPages(TensorPtr s TLLM_LOG_DEBUG("%s start", __PRETTY_FUNCTION__); TLLM_CHECK_WITH_INFO(!pages.empty(), "empty pages"); + TLLM_CHECK_WITH_INFO(sourceWeights->getDataType() == pages.front()->getDataType(), + "Expected LoRA weights dtype %s to match cache dtype %s", sourceWeights->getDataTypeName(), + pages.front()->getDataTypeName()); TensorPtr weights = sourceWeights->getShape().nbDims == 2 ? sourceWeights @@ -775,7 +800,6 @@ void LoraCache::copyTask(TaskIdType taskId, LoraCache& deviceCache, bool markDon TLLM_CHECK_WITH_INFO(deviceCache.mPageManagerConfig.getMemoryType() == runtime::MemoryType::kGPU && !deviceCache.mDeviceBufferManagers.empty(), "The deviceCache must hold GPU memory and have at least one bufferManager / copy stream"); - // First get the taskValue from this cache // TaskValue& taskValue = copyTaskGetThisTaskValue(taskId); TaskValuePtr taskValue = [&]() -> TaskValuePtr @@ -805,6 +829,10 @@ void LoraCache::copyTask(TaskIdType taskId, LoraCache& deviceCache, bool markDon std::optional optOtherTaskValuePtr = [&]() -> std::optional { std::lock_guard deviceCacheLock(deviceCache.mCacheMutex); + // setDataType also holds mCacheMutex, so the check and target task + // insertion are atomic with respect to cache reconfiguration. + TLLM_CHECK_WITH_INFO(mPageManagerConfig.getDataType() == deviceCache.mPageManagerConfig.getDataType(), + "LoRA host and device cache dtypes must match"); auto otherStatus = deviceCache.getStatus(taskId); if (kVALUE_STATUS_MISSING != otherStatus) { diff --git a/cpp/tensorrt_llm/runtime/loraUtils.cpp b/cpp/tensorrt_llm/runtime/loraUtils.cpp index da7f1475ddec..7b78a34d5ef1 100644 --- a/cpp/tensorrt_llm/runtime/loraUtils.cpp +++ b/cpp/tensorrt_llm/runtime/loraUtils.cpp @@ -1,5 +1,5 @@ /* - * Copyright (c) 2023 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * Copyright (c) 2023-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. * SPDX-License-Identifier: Apache-2.0 * * Licensed under the Apache License, Version 2.0 (the "License"); @@ -72,7 +72,7 @@ void loraValidateRequestTensorDims(std::optional const& optR void loraValidateRequestTensors(std::optional const& optTaskId, std::optional const& optReqLoraWeights, std::optional const& optReqLoraConfig, runtime::ModelConfig const& modelConfig, - runtime::WorldConfig const& worldConfig) + runtime::WorldConfig const& worldConfig, std::optional loraDataType) { TLLM_CHECK_WITH_INFO(optTaskId.has_value(), "lora_task_id must be set for LoRA inference"); if (optReqLoraWeights.has_value() || optReqLoraConfig.has_value()) @@ -86,8 +86,10 @@ void loraValidateRequestTensors(std::optional const& optTaskId, : ITensor::view(config, ITensor::makeShape({config->getShape().d[1], config->getShape().d[2]})); SizeType32 nbModelLayers = modelConfig.getNbLoraLayers(); - TLLM_CHECK_WITH_INFO(weights->getDataType() == modelConfig.getDataType(), - "Expected lora weights to be the same data type as base model"); + auto const expectedDataType = loraDataType.value_or(modelConfig.getDataType()); + TLLM_CHECK_WITH_INFO(weights->getDataType() == expectedDataType, + "Expected LoRA weights dtype %s to match homogeneous cache dtype %s", weights->getDataTypeName(), + IBuffer::getDataTypeName(expectedDataType)); auto loraModules = modelConfig.getLoraModules(); auto maxAdapterSize = modelConfig.getMaxLoraRank(); diff --git a/cpp/tensorrt_llm/runtime/loraUtils.h b/cpp/tensorrt_llm/runtime/loraUtils.h index a3e2edf303aa..21401e332926 100644 --- a/cpp/tensorrt_llm/runtime/loraUtils.h +++ b/cpp/tensorrt_llm/runtime/loraUtils.h @@ -1,5 +1,5 @@ /* - * Copyright (c) 2023 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * Copyright (c) 2023-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. * SPDX-License-Identifier: Apache-2.0 * * Licensed under the Apache License, Version 2.0 (the "License"); @@ -22,6 +22,8 @@ #include "tensorrt_llm/runtime/modelConfig.h" #include "tensorrt_llm/runtime/worldConfig.h" +#include + namespace tensorrt_llm::runtime::lora { @@ -57,5 +59,5 @@ void loraValidateRequestTensorDims(std::optional const& optR void loraValidateRequestTensors(std::optional const& optTaskId, std::optional const& optReqLoraWeights, std::optional const& optReqLoraConfig, runtime::ModelConfig const& modelConfig, - runtime::WorldConfig const& worldConfig); + runtime::WorldConfig const& worldConfig, std::optional loraDataType = std::nullopt); } // namespace tensorrt_llm::runtime::lora diff --git a/cpp/tensorrt_llm/thop/CMakeLists.txt b/cpp/tensorrt_llm/thop/CMakeLists.txt index 4ba8cd88bd9c..7af68a90f1c9 100644 --- a/cpp/tensorrt_llm/thop/CMakeLists.txt +++ b/cpp/tensorrt_llm/thop/CMakeLists.txt @@ -75,6 +75,8 @@ add_library( deepseekV4QNormOp.cpp deepseekV4BlockTableOp.cpp inverseRopeFp8QuantOp.cpp + kdaDecodeOp.cpp + attnResOp.cpp fusedQKNormRopeOp.cpp fusedAdaptiveLayerNormOp.cpp fusedDiTQKNormRopeOp.cpp diff --git a/cpp/tensorrt_llm/thop/attentionOp.cpp b/cpp/tensorrt_llm/thop/attentionOp.cpp index 24d869f704dc..e472894a22ea 100644 --- a/cpp/tensorrt_llm/thop/attentionOp.cpp +++ b/cpp/tensorrt_llm/thop/attentionOp.cpp @@ -335,11 +335,12 @@ class RunnerBase public: int32_t beam_width; int32_t max_num_requests; + int32_t max_num_sequences; int32_t attention_window_size; auto data() const { - return std::make_tuple(beam_width, max_num_requests, attention_window_size); + return std::make_tuple(beam_width, max_num_requests, max_num_sequences, attention_window_size); }; virtual ~RunnerBase() = default; @@ -373,13 +374,13 @@ class RunnerBase torch::optional attention_sinks, torch::optional sparse_kv_indices, torch::optional sparse_kv_offsets, torch::optional sparse_attn_indices, torch::optional sparse_attn_offsets, int64_t const sparse_attn_indices_block_size, - int32_t const num_sparse_topk, std::optional sparse_mla_topk_lens, + int32_t const num_sparse_topk, std::optional sparse_attn_kv_lens, std::optional cu_q_seqlens, std::optional cu_kv_seqlens, std::optional fmha_scheduler_counter, std::optional mla_bmm1_scale, std::optional mla_bmm2_scale, std::optional quant_q_buffer, std::optional flash_mla_tile_scheduler_metadata, std::optional flash_mla_num_splits, bool trtllm_gen_jit_warmup, - std::optional compressed_kv_cache_pool_ptr, bool const is_cross, std::optional cross_kv, + std::optional aux_kv_cache_pool_ptr, bool const is_cross, std::optional cross_kv, std::optional relative_attention_bias, std::optional quant_scale_qkv = std::nullopt, std::optional dsv4_inv_rope_cos_sin_cache = std::nullopt, @@ -406,7 +407,7 @@ class Runner : public RunnerBase // is not enough. // The attention kernel might split the heads into multiple blocks, so we might need to reserve more semaphores. // Use mMultiProcessorCount as the lower-bound to make sure we reserve enough semaphores. - op.reserveSemaphoreArray(std::max(op.mNumHeads * max_num_requests, op.getMultiProcessorCount())); + op.reserveSemaphoreArray(std::max(op.mNumHeads * max_num_sequences, op.getMultiProcessorCount())); } int64_t getWorkspaceSize(AttentionOp const& op, int const num_tokens, int const max_attention_window_size, @@ -415,7 +416,7 @@ class Runner : public RunnerBase size_t const context_workspace_size = op.getWorkspaceSizeForContext( op.mType, max_num_requests, op.mMaxContextLength, 0, num_tokens, ctx_total_kv_len); size_t const generation_workspace_size = op.getWorkspaceSizeForGeneration( - op.mType, max_num_requests, max_attention_window_size, num_gen_tokens, max_blocks_per_sequence); + op.mType, max_num_sequences, max_attention_window_size, num_gen_tokens, max_blocks_per_sequence); return std::max(context_workspace_size, generation_workspace_size); } @@ -445,13 +446,13 @@ class Runner : public RunnerBase torch::optional attention_sinks, torch::optional sparse_kv_indices, torch::optional sparse_kv_offsets, torch::optional sparse_attn_indices, torch::optional sparse_attn_offsets, int64_t const sparse_attn_indices_block_size, - int32_t const num_sparse_topk, std::optional sparse_mla_topk_lens, + int32_t const num_sparse_topk, std::optional sparse_attn_kv_lens, std::optional cu_q_seqlens, std::optional cu_kv_seqlens, std::optional fmha_scheduler_counter, std::optional mla_bmm1_scale, std::optional mla_bmm2_scale, std::optional quant_q_buffer, std::optional flash_mla_tile_scheduler_metadata, std::optional flash_mla_num_splits, bool trtllm_gen_jit_warmup, - std::optional compressed_kv_cache_pool_ptr, bool const is_cross, std::optional cross_kv, + std::optional aux_kv_cache_pool_ptr, bool const is_cross, std::optional cross_kv, std::optional relative_attention_bias, std::optional quant_scale_qkv, std::optional dsv4_inv_rope_cos_sin_cache, bool enable_dsv4_epilogue_fusion) const override { @@ -765,8 +766,8 @@ class Runner : public RunnerBase op.mRuntimeSparseAttentionParams.sparse_attn_indices_stride = sparse_attn_indices.has_value() ? sparse_attn_indices.value().size(-1) : 0; op.mRuntimeSparseAttentionParams.num_sparse_topk = num_sparse_topk; - op.mRuntimeSparseAttentionParams.sparse_mla_topk_lens - = sparse_mla_topk_lens.has_value() ? sparse_mla_topk_lens.value().data_ptr() : nullptr; + op.mRuntimeSparseAttentionParams.sparse_attn_kv_lens + = sparse_attn_kv_lens.has_value() ? sparse_attn_kv_lens.value().data_ptr() : nullptr; op.mRuntimeSparseAttentionParams.sparse_kv_cache_pool = nullptr; op.mRuntimeSparseAttentionParams.sliding_window_kv_cache_pool = nullptr; if (op.mUseSparseAttention && use_kv_cache) @@ -775,14 +776,14 @@ class Runner : public RunnerBase { auto* kvCachePool = reinterpret_cast( host_kv_cache_pool_pointers.value().index({pool_index, 0}).item()); - if (sparse_mla_topk_lens.has_value()) + if (sparse_attn_kv_lens.has_value()) { // Deepseek V4 dynamic sparse MLA always uses the SWA pool for now. op.mRuntimeSparseAttentionParams.sliding_window_kv_cache_pool = kvCachePool; - if (compressed_kv_cache_pool_ptr.has_value()) + if (aux_kv_cache_pool_ptr.has_value()) { op.mRuntimeSparseAttentionParams.sparse_kv_cache_pool - = reinterpret_cast(compressed_kv_cache_pool_ptr.value()); + = reinterpret_cast(aux_kv_cache_pool_ptr.value()); } } else @@ -1101,8 +1102,7 @@ void attention(torch::Tensor q, std::optional k, std::optional sparse_kv_indices, std::optional sparse_kv_offsets, std::optional sparse_attn_indices, std::optional sparse_attn_offsets, int64_t const sparse_attn_indices_block_size, std::optional num_sparse_topk, - std::optional sparse_mla_topk_lens, - std::optional skip_softmax_threshold_scale_factor_prefill, + std::optional sparse_attn_kv_lens, std::optional skip_softmax_threshold_scale_factor_prefill, std::optional skip_softmax_threshold_scale_factor_decode, std::optional skip_softmax_stat, std::optional cu_q_seqlens, std::optional cu_kv_seqlens, std::optional fmha_scheduler_counter, std::optional mla_bmm1_scale, @@ -1110,11 +1110,11 @@ void attention(torch::Tensor q, std::optional k, std::optional flash_mla_tile_scheduler_metadata, std::optional flash_mla_num_splits, int64_t sage_attn_num_elts_per_blk_q, int64_t sage_attn_num_elts_per_blk_k, int64_t sage_attn_num_elts_per_blk_v, bool sage_attn_qk_int8, int64_t num_contexts, int64_t num_ctx_tokens, bool trtllm_gen_jit_warmup, - std::optional compressed_kv_cache_pool_ptr, bool const is_cross, std::optional cross_kv, + std::optional aux_kv_cache_pool_ptr, bool const is_cross, std::optional cross_kv, std::optional relative_attention_bias, int64_t relative_attention_max_distance, std::optional spec_decoding_target_max_draft_tokens, std::optional quant_scale_qkv, std::optional dsv4_inv_rope_cos_sin_cache, bool enable_dsv4_epilogue_fusion, - bool const force_prepare_spec_dec_tree_mask) + bool const force_prepare_spec_dec_tree_mask, std::optional const max_num_sequences) { TLLM_LOG_TRACE("Attention op starts at layer %d", local_layer_idx); // Use these tensors to infer if the attention is using KV cache @@ -1194,6 +1194,7 @@ void attention(torch::Tensor q, std::optional k, std::optionalbeam_width = beam_width; runner->max_num_requests = max_num_requests; + runner->max_num_sequences = max_num_sequences.value_or(max_num_requests); runner->attention_window_size = attention_window_size; auto op = std::make_shared(); @@ -1397,10 +1398,10 @@ void attention(torch::Tensor q, std::optional k, std::optional 0) && (attn_input_type != AttentionInputType::ContextOnly)) @@ -1420,10 +1421,10 @@ void attention(torch::Tensor q, std::optional k, std::optional k, std::optional sparse_kv_indices, std::optional sparse_kv_offsets, std::optional sparse_attn_indices, std::optional sparse_attn_offsets, int64_t const sparse_attn_indices_block_size, std::optional num_sparse_topk, - std::optional sparse_mla_topk_lens, - std::optional skip_softmax_threshold_scale_factor_prefill, + std::optional sparse_attn_kv_lens, std::optional skip_softmax_threshold_scale_factor_prefill, std::optional skip_softmax_threshold_scale_factor_decode, std::optional skip_softmax_stat, std::optional cu_q_seqlens, std::optional cu_kv_seqlens, std::optional fmha_scheduler_counter, std::optional mla_bmm1_scale, @@ -90,13 +89,13 @@ void attention(torch::Tensor q, std::optional k, std::optional flash_mla_num_splits = std::nullopt, int64_t sage_attn_num_elts_per_blk_q = 0, int64_t sage_attn_num_elts_per_blk_k = 0, int64_t sage_attn_num_elts_per_blk_v = 0, bool sage_attn_qk_int8 = false, int64_t num_contexts = 0, int64_t num_ctx_tokens = 0, bool trtllm_gen_jit_warmup = false, - std::optional compressed_kv_cache_pool_ptr = std::nullopt, bool const is_cross = false, + std::optional aux_kv_cache_pool_ptr = std::nullopt, bool const is_cross = false, std::optional cross_kv = std::nullopt, std::optional relative_attention_bias = std::nullopt, int64_t relative_attention_max_distance = 0, std::optional spec_decoding_target_max_draft_tokens = std::nullopt, std::optional quant_scale_qkv = std::nullopt, std::optional dsv4_inv_rope_cos_sin_cache = std::nullopt, bool enable_dsv4_epilogue_fusion = false, - bool const force_prepare_spec_dec_tree_mask = false); + bool const force_prepare_spec_dec_tree_mask = false, std::optional const max_num_sequences = std::nullopt); struct KvCachePoolPointers { diff --git a/cpp/tensorrt_llm/thop/attnResOp.cpp b/cpp/tensorrt_llm/thop/attnResOp.cpp new file mode 100644 index 000000000000..6a7d16cadaf2 --- /dev/null +++ b/cpp/tensorrt_llm/thop/attnResOp.cpp @@ -0,0 +1,142 @@ +/* + * Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/kimiK3AttnRes/attnResFwd.h" + +#include +#include +#include +#include +#include +#include + +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +namespace +{ + +bool is_sm100_family() +{ + int dev = 0; + cudaGetDevice(&dev); + static int cached_state[64] = {}; // 0 unknown, 1 false, 2 true + if (dev >= 0 && dev < 64 && cached_state[dev] != 0) + { + return cached_state[dev] == 2; + } + cudaDeviceProp prop; + cudaGetDeviceProperties(&prop, dev); + // The kernel binary is compiled for the sm_100 family only (it relies on + // tcgen05/TMEM, which later architectures such as sm_120 do not support), + // so require compute capability major == 10 rather than >= 10. + bool const ok = prop.major == 10; + if (dev >= 0 && dev < 64) + { + cached_state[dev] = ok ? 2 : 1; + } + return ok; +} + +void check_attn_res_contract(int N, int T, int B, int H) +{ + TORCH_CHECK(is_sm100_family(), "attn_res_fwd requires an sm_100-family (datacenter Blackwell) GPU"); + TORCH_CHECK(B == 1, "attn_res_fwd: unsupported B=", B, " (only B=1 is supported)"); + TORCH_CHECK(N >= 1 && N <= 12, "attn_res_fwd: unsupported N=", N, " (must be in [1, 12])"); + TORCH_CHECK(T >= 1 && T <= 16384, "attn_res_fwd: unsupported T=", T, " (must be in [1, 16384])"); + TORCH_CHECK(H >= 4096 && H <= 8192 && H % 1024 == 0, "attn_res_fwd: unsupported H=", H, + " (must be a multiple of 1024 in [4096, 8192])"); +} + +std::tuple attn_res_fwd( + at::Tensor layer_residual, at::Tensor block_residual, at::Tensor res_weight, at::Tensor rms_weight, double rms_eps) +{ + TORCH_CHECK(layer_residual.dim() == 3, "attn_res_fwd: layer_residual must be [T, B, H]"); + TORCH_CHECK(block_residual.dim() == 4, "attn_res_fwd: block_residual must be [K, T, B, H]"); + + int const T = static_cast(layer_residual.size(0)); + int const B = static_cast(layer_residual.size(1)); + int const H = static_cast(layer_residual.size(2)); + int const N = static_cast(block_residual.size(0)) + 1; + TORCH_CHECK(layer_residual.is_cuda() && block_residual.is_cuda() && res_weight.is_cuda() && rms_weight.is_cuda(), + "attn_res_fwd: all input tensors must be CUDA tensors"); + // Set the device before check_attn_res_contract: is_sm100_family() reads + // the current device, which must match the tensors' device. + c10::cuda::CUDAGuard device_guard(layer_residual.device()); + check_attn_res_contract(N, T, B, H); + + TORCH_CHECK(layer_residual.scalar_type() == at::kBFloat16, "attn_res_fwd: layer_residual must be bf16"); + TORCH_CHECK(block_residual.scalar_type() == at::kBFloat16, "attn_res_fwd: block_residual must be bf16"); + TORCH_CHECK(res_weight.scalar_type() == at::kBFloat16, "attn_res_fwd: res_weight must be bf16"); + TORCH_CHECK(rms_weight.scalar_type() == at::kBFloat16, "attn_res_fwd: rms_weight must be bf16"); + TORCH_CHECK(layer_residual.is_contiguous() && block_residual.is_contiguous() && res_weight.is_contiguous() + && rms_weight.is_contiguous(), + "attn_res_fwd: inputs must be contiguous"); + TORCH_CHECK(block_residual.sizes() == at::IntArrayRef({N - 1, T, B, H}), + "attn_res_fwd: block_residual shape must match layer_residual"); + TORCH_CHECK(res_weight.numel() == H, "attn_res_fwd: res_weight must have H elements"); + TORCH_CHECK(rms_weight.numel() == H, "attn_res_fwd: rms_weight must have H elements"); + + auto output = at::empty_like(layer_residual); + auto float_options = layer_residual.options().dtype(at::kFloat); + auto rsigma = at::empty({N, T, B}, float_options); + auto probs = at::empty({N, T, B}, float_options); + auto logits = at::empty({N, T, B}, float_options); + + kernels::kimiK3AttnRes::AttnResFwdParams params{}; + params.blockResidual = N > 1 ? reinterpret_cast<__nv_bfloat16 const*>(block_residual.const_data_ptr()) : nullptr; + params.layerResidual = reinterpret_cast<__nv_bfloat16 const*>(layer_residual.const_data_ptr()); + params.resWeight = reinterpret_cast<__nv_bfloat16 const*>(res_weight.const_data_ptr()); + params.rmsWeight = reinterpret_cast<__nv_bfloat16 const*>(rms_weight.const_data_ptr()); + params.output = reinterpret_cast<__nv_bfloat16*>(output.data_ptr()); + params.rsigma = rsigma.data_ptr(); + params.probs = probs.data_ptr(); + params.logits = logits.data_ptr(); + params.numCandidates = N; + params.seqLen = T; + params.batchSize = B; + params.hiddenSize = H; + params.rmsEps = static_cast(rms_eps); + + cudaStream_t stream = at::cuda::getCurrentCUDAStream(); + kernels::kimiK3AttnRes::invokeAttnResFwd(params, stream); + C10_CUDA_KERNEL_LAUNCH_CHECK(); + + return {output, rsigma, probs, logits}; +} + +} // namespace + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "attn_res_fwd(Tensor layer_residual, Tensor block_residual, " + "Tensor res_weight, Tensor rms_weight, float rms_eps) " + "-> (Tensor, Tensor, Tensor, Tensor)"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("attn_res_fwd", &tensorrt_llm::torch_ext::attn_res_fwd); +} diff --git a/cpp/tensorrt_llm/thop/fusedQKNormRopeOp.cpp b/cpp/tensorrt_llm/thop/fusedQKNormRopeOp.cpp index 4ff4cff6d3ba..12c124b85aaf 100644 --- a/cpp/tensorrt_llm/thop/fusedQKNormRopeOp.cpp +++ b/cpp/tensorrt_llm/thop/fusedQKNormRopeOp.cpp @@ -18,6 +18,7 @@ #include "tensorrt_llm/thop/thUtils.h" #include +#include #include TRTLLM_NAMESPACE_BEGIN @@ -25,6 +26,47 @@ TRTLLM_NAMESPACE_BEGIN namespace torch_ext { +namespace +{ + +// Shared input validation for the in-place and out-of-place operators. Returns num_tokens. +int64_t validateFusedQKNormRopeInputs(torch::Tensor const& qkv, torch::Tensor const& position_ids, + torch::Tensor const& q_weight, torch::Tensor const& k_weight, int64_t num_heads_q, int64_t num_heads_k, + int64_t num_heads_v, int64_t head_dim, bool use_mrope) +{ + TORCH_CHECK(qkv.dim() == 2, "QKV tensor must be 2D: [num_tokens, (num_heads_q+num_heads_k+num_heads_v)*head_dim]"); + // Plain RoPE: position_ids is 1D [num_tokens]. Interleaved mRoPE: 2D [3, num_tokens]. + TORCH_CHECK(position_ids.dim() == 1 || (position_ids.dim() == 2 && position_ids.size(0) == 3), + "Position IDs must be 1D [num_tokens] (plain RoPE) or 2D [3, num_tokens] (mRoPE)"); + TORCH_CHECK(!use_mrope || position_ids.dim() == 2, "use_mrope requires 2D [3, num_tokens] position_ids"); + TORCH_CHECK(q_weight.dim() == 1, "Query weights must be 1D: [head_dim]"); + TORCH_CHECK(k_weight.dim() == 1, "Key weights must be 1D: [head_dim]"); + TORCH_CHECK(q_weight.size(0) == head_dim, "Query weights size must match head dimension"); + TORCH_CHECK(k_weight.size(0) == head_dim, "Key weights size must match head dimension"); + + CHECK_INPUT(qkv, torch::kBFloat16); + CHECK_INPUT(position_ids, torch::kInt32); + CHECK_INPUT(q_weight, torch::kBFloat16); + CHECK_INPUT(k_weight, torch::kBFloat16); + + int64_t num_tokens = qkv.size(0); + TORCH_CHECK(position_ids.size(-1) == num_tokens, "Number of tokens in position_ids must match QKV"); + + // The kernel narrows these to int, so reject anything that would not survive it. + TORCH_CHECK(num_heads_q >= 0 && num_heads_k >= 0 && num_heads_v >= 0 && head_dim > 0, + "Head counts must be non-negative and head_dim must be positive"); + int64_t total_heads = num_heads_q + num_heads_k + num_heads_v; + TORCH_CHECK( + num_tokens <= std::numeric_limits::max() && total_heads * head_dim <= std::numeric_limits::max(), + "QKV dimensions exceed the kernel's supported range"); + TORCH_CHECK( + qkv.size(1) == total_heads * head_dim, "QKV tensor size must match total number of heads and head dimension"); + + return num_tokens; +} + +} // namespace + // Function for fused QK Norm and RoPE // This operator applies RMS normalization and RoPE to Q and K tensors in a single CUDA kernel. // The OP performs operations in-place on the input qkv tensor. @@ -53,28 +95,8 @@ void fused_qk_norm_rope( int64_t mrope_section2 // mrope_section[2] (width) ) { - // Input validation - TORCH_CHECK(qkv.dim() == 2, "QKV tensor must be 2D: [num_tokens, (num_heads_q+num_heads_k+num_heads_v)*head_dim]"); - // Plain RoPE: position_ids is 1D [num_tokens]. Interleaved mRoPE: 2D [3, num_tokens]. - TORCH_CHECK(position_ids.dim() == 1 || (position_ids.dim() == 2 && position_ids.size(0) == 3), - "Position IDs must be 1D [num_tokens] (plain RoPE) or 2D [3, num_tokens] (mRoPE)"); - TORCH_CHECK(!use_mrope || position_ids.dim() == 2, "use_mrope requires 2D [3, num_tokens] position_ids"); - TORCH_CHECK(q_weight.dim() == 1, "Query weights must be 1D: [head_dim]"); - TORCH_CHECK(k_weight.dim() == 1, "Key weights must be 1D: [head_dim]"); - TORCH_CHECK(q_weight.size(0) == head_dim, "Query weights size must match head dimension"); - TORCH_CHECK(k_weight.size(0) == head_dim, "Key weights size must match head dimension"); - - CHECK_INPUT(qkv, torch::kBFloat16); - CHECK_INPUT(position_ids, torch::kInt32); - CHECK_INPUT(q_weight, torch::kBFloat16); - CHECK_INPUT(k_weight, torch::kBFloat16); - - int64_t num_tokens = qkv.size(0); - TORCH_CHECK(position_ids.size(-1) == num_tokens, "Number of tokens in position_ids must match QKV"); - - int64_t total_heads = num_heads_q + num_heads_k + num_heads_v; - TORCH_CHECK( - qkv.size(1) == total_heads * head_dim, "QKV tensor size must match total number of heads and head dimension"); + int64_t num_tokens = validateFusedQKNormRopeInputs( + qkv, position_ids, q_weight, k_weight, num_heads_q, num_heads_k, num_heads_v, head_dim, use_mrope); auto stream = at::cuda::getCurrentCUDAStream(qkv.get_device()); @@ -89,6 +111,45 @@ void fused_qk_norm_rope( static_cast(mrope_section1), static_cast(mrope_section2)); } +// Out-of-place FP8 variant of fused_qk_norm_rope: applies RMSNorm + RoPE to Q/K, +// copy-casts V, and returns a new FP8 (E4M3) tensor of the same shape. +torch::Tensor fused_qk_norm_rope_to_fp8(torch::Tensor const& qkv, // [num_tokens, (num_q+num_k+num_v)*head_dim] BF16 + int64_t num_heads_q, int64_t num_heads_k, int64_t num_heads_v, int64_t head_dim, int64_t rotary_dim, double eps, + torch::Tensor const& q_weight, torch::Tensor const& k_weight, double base, bool is_neox, + torch::Tensor const& position_ids, double factor, double low, double high, double attention_factor, bool is_qk_norm, + bool use_gemma, bool use_mrope, int64_t mrope_section1, int64_t mrope_section2) +{ + int64_t num_tokens = validateFusedQKNormRopeInputs( + qkv, position_ids, q_weight, k_weight, num_heads_q, num_heads_k, num_heads_v, head_dim, use_mrope); + + int64_t total_heads = num_heads_q + num_heads_k + num_heads_v; + auto out = torch::empty({num_tokens, total_heads * head_dim}, qkv.options().dtype(torch::kFloat8_e4m3fn)); + + auto stream = at::cuda::getCurrentCUDAStream(qkv.get_device()); + + tensorrt_llm::kernels::launchFusedQKNormRopeToFp8(qkv.data_ptr(), out.data_ptr(), static_cast(num_tokens), + static_cast(num_heads_q), static_cast(num_heads_k), static_cast(num_heads_v), + static_cast(head_dim), static_cast(rotary_dim), static_cast(eps), q_weight.data_ptr(), + k_weight.data_ptr(), static_cast(base), !is_neox, reinterpret_cast(position_ids.data_ptr()), + static_cast(factor), static_cast(low), static_cast(high), + static_cast(attention_factor), stream, is_qk_norm, use_gemma, use_mrope, + static_cast(mrope_section1), static_cast(mrope_section2)); + + return out; +} + +// Meta (fake) implementation for torch.compile / tracing: only shape+dtype. +torch::Tensor fused_qk_norm_rope_to_fp8_meta(torch::Tensor const& qkv, int64_t num_heads_q, int64_t num_heads_k, + int64_t num_heads_v, int64_t head_dim, int64_t /*rotary_dim*/, double /*eps*/, torch::Tensor const& /*q_weight*/, + torch::Tensor const& /*k_weight*/, double /*base*/, bool /*is_neox*/, torch::Tensor const& /*position_ids*/, + double /*factor*/, double /*low*/, double /*high*/, double /*attention_factor*/, bool /*is_qk_norm*/, + bool /*use_gemma*/, bool /*use_mrope*/, int64_t /*mrope_section1*/, int64_t /*mrope_section2*/) +{ + int64_t num_tokens = qkv.size(0); + int64_t total_heads = num_heads_q + num_heads_k + num_heads_v; + return torch::empty({num_tokens, total_heads * head_dim}, qkv.options().dtype(torch::kFloat8_e4m3fn)); +} + // Register the PyTorch operators TORCH_LIBRARY_FRAGMENT(trtllm, m) { @@ -98,12 +159,24 @@ TORCH_LIBRARY_FRAGMENT(trtllm, m) "eps, Tensor q_weight, Tensor k_weight, float base, bool is_neox, Tensor position_ids, float factor, float " "low, float high, float attention_factor, bool is_qk_norm, bool use_gemma, bool use_mrope, int " "mrope_section1, int mrope_section2) -> ()"); + m.def( + "fused_qk_norm_rope_to_fp8(Tensor qkv, int num_heads_q, int num_heads_k, int num_heads_v, int head_dim, int " + "rotary_dim, float eps, Tensor q_weight, Tensor k_weight, float base, bool is_neox, Tensor position_ids, float " + "factor, float low, float high, float attention_factor, bool is_qk_norm, bool use_gemma, bool use_mrope, int " + "mrope_section1, int mrope_section2) -> Tensor"); } // Register the CUDA implementation TORCH_LIBRARY_IMPL(trtllm, CUDA, m) { m.impl("fused_qk_norm_rope", &fused_qk_norm_rope); + m.impl("fused_qk_norm_rope_to_fp8", &fused_qk_norm_rope_to_fp8); +} + +// Register the Meta implementation (shape/dtype inference for torch.compile). +TORCH_LIBRARY_IMPL(trtllm, Meta, m) +{ + m.impl("fused_qk_norm_rope_to_fp8", &fused_qk_norm_rope_to_fp8_meta); } } // namespace torch_ext diff --git a/cpp/tensorrt_llm/thop/kdaDecodeOp.cpp b/cpp/tensorrt_llm/thop/kdaDecodeOp.cpp new file mode 100644 index 000000000000..9a2b4271ab00 --- /dev/null +++ b/cpp/tensorrt_llm/thop/kdaDecodeOp.cpp @@ -0,0 +1,252 @@ +/* + * Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/kernels/kdaDecode/kdaDecode.h" + +#include +#include +#include +#include +#include +#include +#include + +TRTLLM_NAMESPACE_BEGIN + +namespace torch_ext +{ + +namespace +{ + +constexpr int kDimK = 128; +constexpr int kDimV = 128; +constexpr int kKernelWidth = 4; + +void validate_kda_decode_fusion_inputs(at::Tensor x_q, at::Tensor x_k, at::Tensor x_v, at::Tensor w_q_t, + at::Tensor w_k_t, at::Tensor w_v_t, at::Tensor bias_q, at::Tensor bias_k, at::Tensor bias_v, at::Tensor cs_q, + at::Tensor cs_k, at::Tensor cs_v, at::Tensor a_log, at::Tensor g, at::Tensor dt_bias, at::Tensor beta, + at::Tensor onorm_g, at::Tensor onorm_weight, std::optional const& ssm_state_indices, + at::Tensor cu_seqlens, at::Tensor state, bool apply_onorm, bool update_conv_cache) +{ + TORCH_CHECK(x_q.is_cuda() && x_q.scalar_type() == at::kBFloat16, "x_q must be a CUDA bfloat16 tensor"); + TORCH_CHECK(x_k.is_cuda() && x_k.scalar_type() == at::kBFloat16, "x_k must be a CUDA bfloat16 tensor"); + TORCH_CHECK(x_v.is_cuda() && x_v.scalar_type() == at::kBFloat16, "x_v must be a CUDA bfloat16 tensor"); + TORCH_CHECK(w_q_t.is_cuda() && w_q_t.scalar_type() == at::kBFloat16, "w_q_t must be a CUDA bfloat16 tensor"); + TORCH_CHECK(w_k_t.is_cuda() && w_k_t.scalar_type() == at::kBFloat16, "w_k_t must be a CUDA bfloat16 tensor"); + TORCH_CHECK(w_v_t.is_cuda() && w_v_t.scalar_type() == at::kBFloat16, "w_v_t must be a CUDA bfloat16 tensor"); + TORCH_CHECK(bias_q.is_cuda() && bias_q.scalar_type() == at::kBFloat16, "bias_q must be a CUDA bfloat16 tensor"); + TORCH_CHECK(bias_k.is_cuda() && bias_k.scalar_type() == at::kBFloat16, "bias_k must be a CUDA bfloat16 tensor"); + TORCH_CHECK(bias_v.is_cuda() && bias_v.scalar_type() == at::kBFloat16, "bias_v must be a CUDA bfloat16 tensor"); + TORCH_CHECK(cs_q.is_cuda() && cs_q.scalar_type() == at::kBFloat16, "cs_q must be a CUDA bfloat16 tensor"); + TORCH_CHECK(cs_k.is_cuda() && cs_k.scalar_type() == at::kBFloat16, "cs_k must be a CUDA bfloat16 tensor"); + TORCH_CHECK(cs_v.is_cuda() && cs_v.scalar_type() == at::kBFloat16, "cs_v must be a CUDA bfloat16 tensor"); + TORCH_CHECK(g.is_cuda() && g.scalar_type() == at::kBFloat16, "g must be a CUDA bfloat16 tensor"); + TORCH_CHECK(beta.is_cuda() && beta.scalar_type() == at::kBFloat16, "beta must be a CUDA bfloat16 tensor"); + TORCH_CHECK(onorm_g.is_cuda() && onorm_g.scalar_type() == at::kBFloat16, "onorm_g must be a CUDA bfloat16 tensor"); + TORCH_CHECK(a_log.is_cuda() && a_log.scalar_type() == at::kFloat, "a_log must be a CUDA float32 tensor"); + TORCH_CHECK(dt_bias.is_cuda() && dt_bias.scalar_type() == at::kFloat, "dt_bias must be a CUDA float32 tensor"); + TORCH_CHECK(onorm_weight.is_cuda() && onorm_weight.scalar_type() == at::kFloat, + "onorm_weight must be a CUDA float32 tensor"); + TORCH_CHECK(state.is_cuda() && state.scalar_type() == at::kFloat, "state must be a CUDA float32 tensor"); + + TORCH_CHECK(x_q.dim() == 4 && x_k.dim() == 4 && x_v.dim() == 4, "x_q, x_k, and x_v must be rank-4 tensors"); + TORCH_CHECK(x_q.size(0) == 1 && x_k.size(0) == 1 && x_v.size(0) == 1, "only T=1 decode inputs are supported"); + TORCH_CHECK(x_q.size(3) == kDimK && x_k.size(3) == kDimK, "only K=128 is supported"); + TORCH_CHECK(x_v.size(3) == kDimV, "only V=128 is supported"); + TORCH_CHECK( + x_q.is_contiguous() && x_k.is_contiguous() && x_v.is_contiguous(), "x_q, x_k, and x_v must be contiguous"); + TORCH_CHECK(w_q_t.dim() == 2 && w_k_t.dim() == 2 && w_v_t.dim() == 2, "w_q_t, w_k_t, and w_v_t must be rank-2"); + TORCH_CHECK(w_q_t.size(0) == kKernelWidth && w_k_t.size(0) == kKernelWidth && w_v_t.size(0) == kKernelWidth, + "only convolution width 4 is supported"); + TORCH_CHECK(w_q_t.is_contiguous() && w_k_t.is_contiguous() && w_v_t.is_contiguous(), + "w_q_t, w_k_t, and w_v_t must be contiguous [4, dim] tensors"); + + int const B = static_cast(x_q.size(1)); + int const H = static_cast(x_q.size(2)); + int const HV = static_cast(x_v.size(2)); + TORCH_CHECK(B > 0, "KDA decode requires a non-empty batch"); + bool const supportedHeads = H == 1 || H == 2 || H == 3 || H == 4 || H == 6 || H == 8 || H == 12 || H == 16 + || H == 24 || H == 32 || H == 48 || H == 96; + TORCH_CHECK( + H == HV && supportedHeads, "KDA decode fusion CUDA supports H == HV in {1,2,3,4,6,8,12,16,24,32,48,96}"); + TORCH_CHECK(x_k.size(1) == B && x_k.size(2) == H && x_v.size(1) == B, + "x_q, x_k, and x_v batch/head dimensions are inconsistent"); + TORCH_CHECK(HV % H == 0, "HV must be divisible by H"); + + int64_t const qk_dim = static_cast(H) * kDimK; + int64_t const v_dim = static_cast(HV) * kDimV; + TORCH_CHECK(w_q_t.size(1) == qk_dim && w_k_t.size(1) == qk_dim && w_v_t.size(1) == v_dim, + "w_q_t and w_k_t must be [4, H*128], w_v_t must be [4, HV*128]"); + TORCH_CHECK(bias_q.is_contiguous() && bias_k.is_contiguous() && bias_v.is_contiguous(), + "bias_q, bias_k, and bias_v must be contiguous"); + TORCH_CHECK(bias_q.numel() == qk_dim && bias_k.numel() == qk_dim && bias_v.numel() == v_dim, + "bias_q and bias_k must hold H*128 elements, bias_v must hold HV*128 elements"); + TORCH_CHECK(a_log.is_contiguous() && a_log.numel() == H, "a_log must be contiguous with H elements"); + TORCH_CHECK(dt_bias.is_contiguous() && dt_bias.numel() == qk_dim, "dt_bias must be contiguous with H*128 elements"); + TORCH_CHECK( + g.is_contiguous() && g.dim() == 4 && g.size(0) == 1 && g.size(1) == B && g.size(2) == HV && g.size(3) == kDimK, + "g must be a contiguous [1, B, HV, 128] tensor"); + TORCH_CHECK(beta.is_contiguous() && beta.dim() == 3 && beta.size(0) == 1 && beta.size(1) == B && beta.size(2) == HV, + "beta must be a contiguous [1, B, HV] tensor"); + if (apply_onorm) + { + TORCH_CHECK(onorm_g.is_contiguous() && onorm_g.dim() == 4 && onorm_g.size(0) == 1 && onorm_g.size(1) == B + && onorm_g.size(2) == HV && onorm_g.size(3) == kDimV, + "onorm_g must be a contiguous [1, B, HV, 128] tensor when apply_onorm is set"); + TORCH_CHECK(onorm_weight.is_contiguous() && onorm_weight.numel() == kDimV, + "onorm_weight must be contiguous with 128 elements when apply_onorm is set"); + } + + TORCH_CHECK(state.dim() == 4 && state.size(0) >= B && state.size(1) == HV && state.size(2) == kDimV + && state.size(3) == kDimK, + "state must have shape [slots, HV, 128, 128] with slots >= B"); + TORCH_CHECK(state.stride(3) == 1 && state.stride(2) == kDimK && state.stride(1) == kDimV * kDimK + && state.stride(0) >= HV * kDimV * kDimK, + "state must be contiguous within each [HV, 128, 128] slot and have a non-overlapping slot stride"); + // The kernel moves recurrent state with 16B cp.async loads and float4 stores at element + // offsets of `slot * stride(0) + `, so both the slot stride and the base + // pointer have to keep those accesses 16B aligned. + TORCH_CHECK(state.stride(0) % 4 == 0, + "state slot stride must be a multiple of 4 floats so that per-slot float4 accesses stay 16B aligned, got ", + state.stride(0)); + TORCH_CHECK(reinterpret_cast(state.data_ptr()) % 16 == 0, + "state must start at a 16B-aligned address (check the storage offset of the view passed in)"); + if (ssm_state_indices.has_value()) + { + TORCH_CHECK(ssm_state_indices->is_cuda() && ssm_state_indices->scalar_type() == at::kInt, + "ssm_state_indices must be a CUDA int32 tensor"); + TORCH_CHECK( + ssm_state_indices->is_contiguous() && ssm_state_indices->dim() == 1 && ssm_state_indices->size(0) == B, + "ssm_state_indices must be contiguous with shape [B]"); + } + TORCH_CHECK(cu_seqlens.is_cuda() && cu_seqlens.scalar_type() == at::kInt, "cu_seqlens must be a CUDA int32 tensor"); + TORCH_CHECK(cu_seqlens.is_contiguous() && cu_seqlens.dim() == 1 && cu_seqlens.size(0) == B + 1, + "cu_seqlens must be contiguous with shape [B + 1]"); + + if (update_conv_cache) + { + TORCH_CHECK(H == HV, "conv state update currently assumes H == HV"); + TORCH_CHECK( + cs_q.dim() == 3 && cs_k.dim() == 3 && cs_v.dim() == 3, "update_conv_cache expects rank-3 conv-state pools"); + TORCH_CHECK(cs_q.size(0) >= state.size(0) && cs_k.size(0) >= state.size(0) && cs_v.size(0) >= state.size(0), + "conv-state pools must cover every recurrent-state slot"); + TORCH_CHECK(cs_q.size(1) == H * kDimK && cs_k.size(1) == H * kDimK && cs_v.size(1) == HV * kDimV + && cs_q.size(2) == kKernelWidth - 1 && cs_k.size(2) == kKernelWidth - 1 + && cs_v.size(2) == kKernelWidth - 1, + "update_conv_cache expects [slots, dim, 3] conv-state pools"); + TORCH_CHECK( + cs_q.stride(0) == 3 * H * kDimK && cs_k.stride(0) == 3 * H * kDimK && cs_v.stride(0) == 3 * HV * kDimV, + "update_conv_cache expects densely packed conv-state slots"); + TORCH_CHECK(cs_q.stride(1) == 1 && cs_k.stride(1) == 1, + "update_conv_cache expects cs_q/cs_k transposed layout with " + "contiguous dim axis"); + TORCH_CHECK(cs_q.stride(2) == H * kDimK && cs_k.stride(2) == H * kDimK, + "update_conv_cache expects cs_q/cs_k token stride H*K"); + TORCH_CHECK(cs_v.stride(1) == 1, + "update_conv_cache expects cs_v transposed layout with " + "contiguous dim axis"); + TORCH_CHECK(cs_v.stride(2) == HV * kDimV, "update_conv_cache expects cs_v token stride HV*V"); + } + else + { + TORCH_CHECK(cs_q.dim() == 3 && cs_k.dim() == 3 && cs_v.dim() == 3, "batch-local conv states must be rank-3"); + TORCH_CHECK(cs_q.size(0) == B && cs_k.size(0) == B && cs_v.size(0) == B && cs_q.size(1) == H * kDimK + && cs_k.size(1) == H * kDimK && cs_v.size(1) == HV * kDimV && cs_q.size(2) == kKernelWidth - 1 + && cs_k.size(2) == kKernelWidth - 1 && cs_v.size(2) == kKernelWidth - 1, + "batch-local conv states must have shapes [B, H*128, 3], " + "[B, H*128, 3], and [B, HV*128, 3]"); + TORCH_CHECK(cs_q.is_contiguous(), "cs_q must be contiguous [B, H*K, 3]"); + TORCH_CHECK(cs_k.is_contiguous(), "cs_k must be contiguous [B, H*K, 3]"); + TORCH_CHECK(cs_v.is_contiguous(), "cs_v must be contiguous [B, HV*V, 3]"); + } +} + +void launch_selected_kernel(at::Tensor x_q, at::Tensor x_k, at::Tensor x_v, at::Tensor w_q_t, at::Tensor w_k_t, + at::Tensor w_v_t, at::Tensor bias_q, at::Tensor bias_k, at::Tensor bias_v, at::Tensor cs_q, at::Tensor cs_k, + at::Tensor cs_v, at::Tensor a_log, at::Tensor g, at::Tensor dt_bias, at::Tensor beta, at::Tensor onorm_g, + at::Tensor onorm_weight, std::optional const& ssm_state_indices, at::Tensor cu_seqlens, + at::Tensor state, at::Tensor out, bool apply_onorm, bool update_conv_cache, bool use_lower_bound, + bool apply_beta_sigmoid, double lower_bound, double scale, double onorm_eps) +{ + int const B = static_cast(x_q.size(1)); + int const H = static_cast(x_q.size(2)); + int const HV = static_cast(x_v.size(2)); + + tensorrt_llm::kernels::kdaDecode::KdaDecodeParams const params{x_q.data_ptr(), x_k.data_ptr(), x_v.data_ptr(), + w_q_t.data_ptr(), w_k_t.data_ptr(), w_v_t.data_ptr(), bias_q.data_ptr(), bias_k.data_ptr(), bias_v.data_ptr(), + cs_q.data_ptr(), cs_k.data_ptr(), cs_v.data_ptr(), a_log.data_ptr(), g.data_ptr(), + dt_bias.data_ptr(), beta.data_ptr(), onorm_g.data_ptr(), onorm_weight.data_ptr(), + ssm_state_indices.has_value() ? ssm_state_indices->data_ptr() : nullptr, cu_seqlens.data_ptr(), + state.data_ptr(), state.stride(0), out.data_ptr(), B, H, HV, apply_onorm, update_conv_cache, + use_lower_bound, apply_beta_sigmoid, static_cast(lower_bound), static_cast(scale), + static_cast(onorm_eps)}; + + cudaStream_t stream = at::cuda::getCurrentCUDAStream(); + tensorrt_llm::kernels::kdaDecode::invokeKdaDecode(params, stream); + C10_CUDA_KERNEL_LAUNCH_CHECK(); +} + +at::Tensor kda_decode_fusion_forward(at::Tensor x_q, at::Tensor x_k, at::Tensor x_v, at::Tensor w_q_t, at::Tensor w_k_t, + at::Tensor w_v_t, at::Tensor bias_q, at::Tensor bias_k, at::Tensor bias_v, at::Tensor cs_q, at::Tensor cs_k, + at::Tensor cs_v, at::Tensor a_log, at::Tensor g, at::Tensor dt_bias, at::Tensor beta, at::Tensor onorm_g, + at::Tensor onorm_weight, std::optional ssm_state_indices, at::Tensor cu_seqlens, at::Tensor state, + bool apply_onorm, bool update_conv_cache, bool use_lower_bound, bool apply_beta_sigmoid, double lower_bound, + double scale, double onorm_eps, std::optional output) +{ + validate_kda_decode_fusion_inputs(x_q, x_k, x_v, w_q_t, w_k_t, w_v_t, bias_q, bias_k, bias_v, cs_q, cs_k, cs_v, + a_log, g, dt_bias, beta, onorm_g, onorm_weight, ssm_state_indices, cu_seqlens, state, apply_onorm, + update_conv_cache); + int const B = static_cast(x_q.size(1)); + int const HV = static_cast(x_v.size(2)); + auto out = output.has_value() ? *output : at::empty({B, 1, HV, kDimV}, x_q.options()); + if (output.has_value()) + { + TORCH_CHECK(out.is_cuda() && out.scalar_type() == at::kBFloat16, "out must be a CUDA bfloat16 tensor"); + TORCH_CHECK(out.is_contiguous(), "out must be contiguous"); + TORCH_CHECK(out.dim() == 4 && out.size(0) == B && out.size(1) == 1 && out.size(2) == HV && out.size(3) == kDimV, + "out must have shape [B, 1, HV, 128]"); + } + launch_selected_kernel(x_q, x_k, x_v, w_q_t, w_k_t, w_v_t, bias_q, bias_k, bias_v, cs_q, cs_k, cs_v, a_log, g, + dt_bias, beta, onorm_g, onorm_weight, ssm_state_indices, cu_seqlens, state, out, apply_onorm, update_conv_cache, + use_lower_bound, apply_beta_sigmoid, lower_bound, scale, onorm_eps); + return out; +} + +} // namespace + +} // namespace torch_ext + +TRTLLM_NAMESPACE_END + +TORCH_LIBRARY_FRAGMENT(trtllm, m) +{ + m.def( + "kda_decode(Tensor x_q, Tensor x_k, Tensor x_v, Tensor w_q_t, " + "Tensor w_k_t, Tensor w_v_t, Tensor bias_q, Tensor bias_k, " + "Tensor bias_v, Tensor(a!) conv_state_q, Tensor(b!) conv_state_k, " + "Tensor(c!) conv_state_v, Tensor a_log, Tensor g, Tensor dt_bias, " + "Tensor beta, Tensor onorm_g, Tensor onorm_weight, " + "Tensor? ssm_state_indices, Tensor cu_seqlens, Tensor(d!) state, " + "bool apply_onorm, bool update_conv_cache, bool use_lower_bound, " + "bool apply_beta_sigmoid, float lower_bound, float scale, " + "float onorm_eps, Tensor(e!)? output=None) -> Tensor(e!)"); +} + +TORCH_LIBRARY_IMPL(trtllm, CUDA, m) +{ + m.impl("kda_decode", &tensorrt_llm::torch_ext::kda_decode_fusion_forward); +} diff --git a/cpp/tensorrt_llm/thop/loraOp.cpp b/cpp/tensorrt_llm/thop/loraOp.cpp index 6957987be6b8..51575a5873ed 100644 --- a/cpp/tensorrt_llm/thop/loraOp.cpp +++ b/cpp/tensorrt_llm/thop/loraOp.cpp @@ -1,6 +1,6 @@ /* - * Copyright (c) 2022-2025, NVIDIA CORPORATION. All rights reserved. + * Copyright (c) 2022-2026, NVIDIA CORPORATION. All rights reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -157,7 +157,10 @@ std::vector lora_grouped_gemm(th::Tensor const& input, th::Tensor co { case torch::kFloat16: loraRuntimeDataType = tensorrt_llm::DataType::kHALF; break; case torch::kBFloat16: loraRuntimeDataType = tensorrt_llm::DataType::kBF16; break; - default: throw std::invalid_argument("Invalid dtype, only supports float16, bfloat16"); +#ifdef ENABLE_FP8 + case torch::kFloat8_e4m3fn: loraRuntimeDataType = tensorrt_llm::DataType::kFP8; break; +#endif + default: throw std::invalid_argument("Invalid dtype, only supports float16, bfloat16, float8_e4m3fn"); } auto mLoraImpl = std::make_shared( @@ -227,7 +230,11 @@ void lora_grouped_gemm_cuda_graph(th::Tensor const& lora_in_sizes, // [layer_mod { case torch::kFloat16: loraRuntimeDataType = tensorrt_llm::DataType::kHALF; break; case torch::kBFloat16: loraRuntimeDataType = tensorrt_llm::DataType::kBF16; break; - default: TORCH_CHECK(false, "Invalid dtype, only supports float16, bfloat16, got %s", c10::toString(dtype)); +#ifdef ENABLE_FP8 + case torch::kFloat8_e4m3fn: loraRuntimeDataType = tensorrt_llm::DataType::kFP8; break; +#endif + default: + TORCH_CHECK(false, "Invalid dtype, only supports float16, bfloat16, float8_e4m3fn, got ", c10::toString(dtype)); } int const minKnInt = std::max(1, static_cast(minKN)); @@ -307,7 +314,11 @@ void lora_group_gemm_param_fill_row_reorder_fusion(th::Tensor const& in_sizes, / { case torch::kFloat16: loraRuntimeDataType = tensorrt_llm::DataType::kHALF; break; case torch::kBFloat16: loraRuntimeDataType = tensorrt_llm::DataType::kBF16; break; - default: TORCH_CHECK(false, "Invalid dtype, only supports float16, bfloat16, got %s", c10::toString(dtype)); +#ifdef ENABLE_FP8 + case torch::kFloat8_e4m3fn: loraRuntimeDataType = tensorrt_llm::DataType::kFP8; break; +#endif + default: + TORCH_CHECK(false, "Invalid dtype, only supports float16, bfloat16, float8_e4m3fn, got ", c10::toString(dtype)); } int64_t const dtype_element_size = input.element_size(); diff --git a/cpp/tensorrt_llm/thop/moeAlltoAllOp.cpp b/cpp/tensorrt_llm/thop/moeAlltoAllOp.cpp index e985eb943ee1..81b3dc33fe28 100644 --- a/cpp/tensorrt_llm/thop/moeAlltoAllOp.cpp +++ b/cpp/tensorrt_llm/thop/moeAlltoAllOp.cpp @@ -21,6 +21,7 @@ #include "tensorrt_llm/thop/moeAlltoAllMeta.h" #include "tensorrt_llm/thop/thUtils.h" +#include #include #include #include @@ -34,6 +35,15 @@ namespace torch_ext namespace moe_comm { +// Whether the engine is in its startup warmup phase, which uses a larger +// completion-flag budget. See moeA2AGetTimeoutCycles(). +static std::atomic gInWarmup{false}; + +void moeA2ASetWarmupOp(bool in_warmup) +{ + gInWarmup.store(in_warmup, std::memory_order_relaxed); +} + static constexpr size_t CACHELINE_ALIGNMENT = 128; // TODO: Is Alignment necessary? @@ -409,6 +419,8 @@ std::tuple, int64_t, torch::Tensor> moeA2ADispatchOp( } params.stream = at::cuda::getCurrentCUDAStream(); + params.timeout_cycles + = tensorrt_llm::kernels::moe_comm::moeA2AGetTimeoutCycles(gInWarmup.load(std::memory_order_relaxed)); // Prepare for dispatch (zero counters/indices and increment flag_val) moe_a2a_prepare_dispatch_launch(params); @@ -582,6 +594,8 @@ torch::Tensor moeA2ACombineOp(torch::Tensor const& payload, int64_t localNumToke } params.stream = at::cuda::getCurrentCUDAStream(); + params.timeout_cycles + = tensorrt_llm::kernels::moe_comm::moeA2AGetTimeoutCycles(gInWarmup.load(std::memory_order_relaxed)); moe_a2a_prepare_combine_launch(params); @@ -692,6 +706,7 @@ TORCH_LIBRARY_FRAGMENT(trtllm, module) "moe_a2a_get_combine_payload_tensor(Tensor(a) workspace, int ep_rank, int ep_size, int " "runtime_max_tokens_per_rank, " "int combine_payload_offset, ScalarType out_dtype, int hidden_size) -> Tensor(a)"); + module.def("moe_a2a_set_warmup(bool in_warmup) -> ()", &tensorrt_llm::torch_ext::moe_comm::moeA2ASetWarmupOp); module.def("moe_a2a_get_aux_data_size(int ep_size, int max_num_tokens, int? eplb_stats_num_experts=None) -> int", &tensorrt_llm::torch_ext::moe_comm::moeA2AGetAuxDataSizeOp); } diff --git a/cpp/tensorrt_llm/thop/moeOp.cpp b/cpp/tensorrt_llm/thop/moeOp.cpp index 52ef700495bf..9d5fe70e7696 100644 --- a/cpp/tensorrt_llm/thop/moeOp.cpp +++ b/cpp/tensorrt_llm/thop/moeOp.cpp @@ -475,8 +475,19 @@ class FusedMoeRunner : public torch::CustomClassHolder { // Note: The weight shape for INT8 weight only quantization is different, e.g., fc2_expert_weights: // [num_experts, inter_size, hidden_size] - TORCH_CHECK(fc1_expert_weights.sizes()[2] == fc2_expert_weights.sizes()[1] * mInnerDimMultiplier * 2, - "fc1_expert_weights inter size must be 2 times fc2_expert_weights inter size."); + // Mirror the non-woq else-branch below: gated activations (Swiglu/Geglu) require fc1's + // intermediate dim to be 2x fc2's (one half each for gate and up), while non-gated + // activations (Relu2/Identity/ReLU/SiLU/Gelu, e.g. Nemotron-H) require them to be equal. + if (isGatedActivation(base_activation_type)) + { + TORCH_CHECK(fc1_expert_weights.sizes()[2] == fc2_expert_weights.sizes()[1] * mInnerDimMultiplier * 2, + "fc1_expert_weights inter size must be 2 times fc2_expert_weights inter size."); + } + else + { + TORCH_CHECK(fc1_expert_weights.sizes()[2] == fc2_expert_weights.sizes()[1] * mInnerDimMultiplier, + "fc1_expert_weights inter size must be equal to fc2_expert_weights inter size."); + } } else { @@ -757,8 +768,6 @@ class FusedMoeRunner : public torch::CustomClassHolder } TORCH_CHECK(fc1_expert_weights.sizes()[0] == fc2_expert_weights.sizes()[0], "fc1_expert_weights and fc2_expert_weights must have the same number of experts."); - TORCH_CHECK(fc1_expert_weights.sizes()[1] == fc2_expert_weights.sizes()[2] * mInnerDimMultiplier * 2, - "fc1_expert_weights inter size must be 2 times fc2_expert_weights inter size."); TORCH_CHECK(!input_sf.has_value() || isWMxfp4AMxfp8Quant() || isNvfp4Quant(), "Block-scaling factors provided for non block-scaling quantization"); @@ -769,6 +778,13 @@ class FusedMoeRunner : public torch::CustomClassHolder int64_t unpadded_hidden_size_val = unpadded_hidden_size.has_value() ? unpadded_hidden_size.value() : hidden_size; int64_t inter_size = fc2_expert_weights.sizes()[2] * mInnerDimMultiplier; + if (mUseINT8WoqPerChannel) + { + // Note: The weight shape for INT8 weight only quantization is different, e.g., fc2_expert_weights: + // [num_experts, inter_size, hidden_size] + hidden_size = fc2_expert_weights.sizes()[2] * mInnerDimMultiplier; + inter_size = fc2_expert_weights.sizes()[1]; + } int const num_experts_on_rank = fc2_expert_weights.sizes()[0]; auto const num_experts_total = static_cast(num_experts_on_rank * ep_size); auto parallelism_config @@ -802,6 +818,38 @@ class FusedMoeRunner : public torch::CustomClassHolder reinterpret_cast(swiglu_beta.has_value() ? swiglu_beta.value().const_data_ptr() : nullptr), reinterpret_cast(swiglu_limit.has_value() ? swiglu_limit.value().const_data_ptr() : nullptr)); + // Validate the fc1/fc2 inter-size relationship now that the activation type (gated vs + // non-gated) is finalized. INT8-woq uses a transposed weight layout, so its fc1/fc2 dim + // ordering differs from the non-woq path; both mirror the gated/non-gated split used in + // runMoe(). Gated activations (Swiglu/Geglu) require fc1's intermediate dim to be 2x fc2's; + // non-gated (Relu2/Identity/ReLU/SiLU/Gelu, e.g. Nemotron-H) require them to be equal. + if (mUseINT8WoqPerChannel) + { + if (isGatedActivation(base_activation_type)) + { + TORCH_CHECK(fc1_expert_weights.sizes()[2] == fc2_expert_weights.sizes()[1] * mInnerDimMultiplier * 2, + "fc1_expert_weights inter size must be 2 times fc2_expert_weights inter size."); + } + else + { + TORCH_CHECK(fc1_expert_weights.sizes()[2] == fc2_expert_weights.sizes()[1] * mInnerDimMultiplier, + "fc1_expert_weights inter size must be equal to fc2_expert_weights inter size."); + } + } + else + { + if (isGatedActivation(base_activation_type)) + { + TORCH_CHECK(fc1_expert_weights.sizes()[1] == fc2_expert_weights.sizes()[2] * mInnerDimMultiplier * 2, + "fc1_expert_weights inter size must be 2 times fc2_expert_weights inter size."); + } + else + { + TORCH_CHECK(fc1_expert_weights.sizes()[1] == fc2_expert_weights.sizes()[2] * mInnerDimMultiplier, + "fc1_expert_weights inter size must be equal to fc2_expert_weights inter size."); + } + } + setRunnerProfiles(profile_ids); auto stream = at::cuda::getCurrentCUDAStream(input.get_device()); diff --git a/cpp/tests/unit_tests/batch_manager/CMakeLists.txt b/cpp/tests/unit_tests/batch_manager/CMakeLists.txt index 02f214986712..420d07dd543e 100644 --- a/cpp/tests/unit_tests/batch_manager/CMakeLists.txt +++ b/cpp/tests/unit_tests/batch_manager/CMakeLists.txt @@ -37,6 +37,11 @@ target_include_directories( kvCacheManagerV2StatsTest PRIVATE ${PROJECT_SOURCE_DIR}/tensorrt_llm/batch_manager ${PROJECT_SOURCE_DIR}/tensorrt_llm/common/sha256) +add_gtest(kvCacheManagerV2DigestPoolTest kvCacheManagerV2DigestPoolTest.cpp) +target_include_directories( + kvCacheManagerV2DigestPoolTest + PRIVATE ${PROJECT_SOURCE_DIR}/tensorrt_llm/batch_manager + ${PROJECT_SOURCE_DIR}/tensorrt_llm/common/sha256) add_gtest(kvCacheUtilsTest kvCacheUtilsTest.cpp) add_gtest(llmRequestTest llmRequestTest.cpp) add_gtest(microBatchSchedulerTest microBatchSchedulerTest.cpp) diff --git a/cpp/tests/unit_tests/batch_manager/kvCacheManagerV2DigestPoolTest.cpp b/cpp/tests/unit_tests/batch_manager/kvCacheManagerV2DigestPoolTest.cpp new file mode 100644 index 000000000000..77f03b74e490 --- /dev/null +++ b/cpp/tests/unit_tests/batch_manager/kvCacheManagerV2DigestPoolTest.cpp @@ -0,0 +1,251 @@ +/* + * SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "tensorrt_llm/batch_manager/kv_cache_manager_v2/blockRadixTree.h" +#include "tensorrt_llm/batch_manager/kv_cache_manager_v2/common.h" +#include "tensorrt_llm/batch_manager/kv_cache_manager_v2/tokenIdExt.h" + +#include + +#include +#include +#include +#include +#include +#include +#include +#include + +namespace +{ +using namespace tensorrt_llm::batch_manager::kv_cache_manager_v2; + +// Build a deterministic 32-byte digest from a seed byte. +Digest makeDigest(std::byte seed) +{ + Digest digest; + for (size_t i = 0; i < kDIGEST_LEN; ++i) + { + digest[i] = static_cast(static_cast(seed) + static_cast(i)); + } + return digest; +} + +// Core cache-key-stability guarantee: hashing an all-normal token block in one +// bulk Write must be bit-identical to the per-element path. +TEST(DigestPoolTest, BulkHashEqualsPerElementForNormalTokens) +{ + std::vector tokens; + for (int i = 0; i < 37; ++i) // odd, non-power-of-two count + { + tokens.emplace_back(TokenId{i * 7 + 1}); + } + ASSERT_FALSE(std::any_of(tokens.begin(), tokens.end(), [](TokenIdExt const& t) { return t.isDigest(); })); + + // Per-element (slow) path. + Hasher slow; + for (auto const& tok : tokens) + { + slow.update(tok); + } + + // Bulk (fast) path — knownNoDigest=true does one bulk Write, no per-element scan. + Hasher fast; + fast.update(tokens.data(), tokens.size(), /*knownNoDigest=*/true); + + EXPECT_EQ(slow.digest(), fast.digest()); +} + +// Two digests with identical bytes in DISTINCT pool slots must compare equal +// (equality dereferences the pool) and hash identically. Slots are released when +// the owning tokens go out of scope. +TEST(DigestPoolTest, DigestValueEqualityAcrossDistinctSlots) +{ + size_t const baseline = detail::digestPoolLiveCount(); + Digest const bytes = makeDigest(std::byte{0x42}); + + TokenIdExt const tokA(bytes); + TokenIdExt const tokB(bytes); + EXPECT_EQ(detail::digestPoolLiveCount(), baseline + 2); // two distinct slots + ASSERT_TRUE(tokA.isDigest()); + EXPECT_EQ(tokA, tokB); // by-value (pooled) equality + EXPECT_NE(tokA, TokenIdExt(TokenId{5})); // digest != normal + + // Hashing the two distinct-slot digests yields the same contribution. + Hasher hashA; + hashA.update(tokA); + Hasher hashB; + hashB.update(tokB); + EXPECT_EQ(hashA.digest(), hashB.digest()); +} + +// A copied digest token clones its slot; both are freed on destruction. +TEST(DigestPoolTest, CopyDigestTokenClonesSlot) +{ + size_t const baseline = detail::digestPoolLiveCount(); + Digest const bytes = makeDigest(std::byte{0x5A}); + { + TokenIdExt const original(bytes); + EXPECT_EQ(detail::digestPoolLiveCount(), baseline + 1); + TokenIdExt const copy = original; // clone → second slot + EXPECT_EQ(detail::digestPoolLiveCount(), baseline + 2); + EXPECT_EQ(original, copy); + EXPECT_EQ(copy.digest(), bytes); + } + EXPECT_EQ(detail::digestPoolLiveCount(), baseline); // both slots freed +} + +// Moving transfers ownership without cloning or double-freeing a digest slot. +TEST(DigestPoolTest, MoveTransfersSlotWithoutCloning) +{ + size_t const baseline = detail::digestPoolLiveCount(); + Digest const bytes = makeDigest(std::byte{0x33}); + Digest const other = makeDigest(std::byte{0x77}); + { + TokenIdExt source(bytes); + EXPECT_EQ(detail::digestPoolLiveCount(), baseline + 1); + + TokenIdExt moved(std::move(source)); + EXPECT_EQ(detail::digestPoolLiveCount(), baseline + 1); + EXPECT_EQ(source.raw(), TokenIdExt::kBadToken); + EXPECT_EQ(moved.digest(), bytes); + EXPECT_THROW((void) (source == moved), std::out_of_range); + + TokenIdExt target(other); + EXPECT_EQ(detail::digestPoolLiveCount(), baseline + 2); + target = std::move(moved); + EXPECT_EQ(detail::digestPoolLiveCount(), baseline + 1); + EXPECT_EQ(moved.raw(), TokenIdExt::kBadToken); + EXPECT_EQ(target.digest(), bytes); + } + EXPECT_EQ(detail::digestPoolLiveCount(), baseline); +} + +// clone-on-copy: the copy owns an independent slot; destroying the original +// leaves the copy valid, and slots return to the free-list (liveCount delta 0). +TEST(DigestPoolTest, CloneIndependenceAndFreeReuse) +{ + size_t const baseline = detail::digestPoolLiveCount(); + Digest const bytes = makeDigest(std::byte{0x11}); + + { + std::vector original; + original.emplace_back(TokenId{1}); + original.emplace_back(bytes); + original.emplace_back(TokenId{2}); + ASSERT_TRUE(std::any_of(original.begin(), original.end(), [](TokenIdExt const& t) { return t.isDigest(); })); + EXPECT_EQ(detail::digestPoolLiveCount(), baseline + 1); + + std::vector copy(original); // deep clone → a second slot + EXPECT_EQ(detail::digestPoolLiveCount(), baseline + 2); + + // Destroy the original; the copy's digest bytes must remain valid. + { + std::vector dying(std::move(original)); + } // dying (holding original's slot) destructs here + EXPECT_EQ(detail::digestPoolLiveCount(), baseline + 1); + + ASSERT_EQ(copy.size(), 3U); + ASSERT_TRUE(copy[1].isDigest()); + EXPECT_EQ(copy[1].digest(), bytes); + } // copy destructs + + EXPECT_EQ(detail::digestPoolLiveCount(), baseline); // all slots freed / reused +} + +// Exercise front-packed allocation + tail-shrink reclamation: allocate well past +// the shrink threshold, release, and confirm content integrity survives the +// intervening deque shrinks and that every slot is reclaimed. +TEST(DigestPoolTest, FrontPackAndTailShrinkChurn) +{ + size_t const baseline = detail::digestPoolLiveCount(); + + auto distinctDigest = [](int seed) + { + Digest digest; + for (size_t i = 0; i < kDIGEST_LEN; ++i) + { + digest[i] = static_cast((seed + static_cast(i)) & 0xFF); + } + // Embed the full seed so digests are distinct beyond 256 entries. + std::memcpy(digest.data(), &seed, sizeof(seed)); + return digest; + }; + + int const count = 600; // > kSlackHigh (256), forces growth then shrink + std::vector tokens; + tokens.reserve(count); + for (int i = 0; i < count; ++i) + { + tokens.emplace_back(distinctDigest(i)); + } + EXPECT_EQ(detail::digestPoolLiveCount(), baseline + static_cast(count)); + for (int i = 0; i < count; ++i) + { + EXPECT_EQ(tokens[i].digest(), distinctDigest(i)) << "content at " << i; + } + + uint32_t const reclaimedSlot = tokens.front().raw(); + + // Release the first half (front slots), then confirm the survivors are intact + // after the shrink churn triggered by freeing. + tokens.erase(tokens.begin(), tokens.begin() + count / 2); + EXPECT_EQ(detail::digestPoolLiveCount(), baseline + static_cast(count - count / 2)); + for (size_t i = 0; i < tokens.size(); ++i) + { + EXPECT_EQ(tokens[i].digest(), distinctDigest(static_cast(i) + count / 2)) << "survivor at " << i; + } + + // A fresh allocation must front-pack into a reclaimed low slot. + { + TokenIdExt const refill(distinctDigest(9999)); + EXPECT_EQ(refill.raw(), reclaimedSlot); + EXPECT_EQ(refill.digest(), distinctDigest(9999)); + + tokens.clear(); + // `refill` still alive here. + EXPECT_EQ(detail::digestPoolLiveCount(), baseline + 1); + } + // `refill` destroyed: its slot must return to the pool, or a leak in the refill + // allocation path would go unnoticed and silently raise the baseline for later tests. + EXPECT_EQ(detail::digestPoolLiveCount(), baseline); +} + +// A block containing a sparse digest still hashes deterministically; the +// per-slice fast path applies to the all-text portion. +TEST(DigestPoolTest, MixedBlockHashesDeterministically) +{ + auto build = [](Digest const& mm) + { + std::vector tokens; + tokens.emplace_back(TokenId{10}); + tokens.emplace_back(mm); + tokens.emplace_back(TokenId{20}); + tokens.emplace_back(TokenId{30}); + return tokens; + }; + Digest const mm = makeDigest(std::byte{0x7E}); + std::vector const a = build(mm); + std::vector const b = build(mm); // distinct slots, identical content + + Hasher ha; + ha.update(a.data(), a.size()); + Hasher hb; + hb.update(b.data(), b.size()); + EXPECT_EQ(ha.digest(), hb.digest()); +} +} // namespace diff --git a/cpp/tests/unit_tests/batch_manager/kvCacheManagerV2StatsTest.cpp b/cpp/tests/unit_tests/batch_manager/kvCacheManagerV2StatsTest.cpp index 2f7169638d29..116697230bc3 100644 --- a/cpp/tests/unit_tests/batch_manager/kvCacheManagerV2StatsTest.cpp +++ b/cpp/tests/unit_tests/batch_manager/kvCacheManagerV2StatsTest.cpp @@ -219,7 +219,7 @@ TEST(KvCacheManagerV2StatsTest, PeakBlockStatsResetStartsNextIntervalFromCurrent { tokens.emplace_back(TokenId{token++}); } - auto block = addOrGetExistingBlock(previous, LifeCycleId{1}, std::move(tokens)); + auto block = addOrGetExistingBlock(previous, std::move(tokens), /*knownNoDigest=*/true); auto page = makeShared( &storage, block, lifeCycle, kGpuLevel, static_cast(block->tokens.size()), kPriorityDefault); page->setSlot(slot); @@ -311,7 +311,7 @@ TEST(KvCacheManagerV2StatsTest, MigrationAndLastTierDropRecordersReceiveExactPag { tokens.emplace_back(TokenId{tokenBase++}); } - auto block = addOrGetExistingBlock(previous, LifeCycleId{1}, std::move(tokens)); + auto block = addOrGetExistingBlock(previous, std::move(tokens), /*knownNoDigest=*/true); auto page = makeShared( &storage, block, lifeCycle, kGpuLevel, static_cast(block->tokens.size()), kPriorityDefault); page->setSlot(slot); diff --git a/cpp/tests/unit_tests/batch_manager/peftCacheManagerTest.cpp b/cpp/tests/unit_tests/batch_manager/peftCacheManagerTest.cpp index ef513894bfb6..6e4ded72ae26 100644 --- a/cpp/tests/unit_tests/batch_manager/peftCacheManagerTest.cpp +++ b/cpp/tests/unit_tests/batch_manager/peftCacheManagerTest.cpp @@ -1,5 +1,5 @@ /* - * SPDX-FileCopyrightText: Copyright (c) 2023-2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + * SPDX-FileCopyrightText: Copyright (c) 2023-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. * SPDX-License-Identifier: Apache-2.0 * * Licensed under the Apache License, Version 2.0 (the "License"); @@ -143,6 +143,73 @@ TEST_F(PeftCacheManagerTest, addRequestPeftMissingTask) mPeftManager->addRequestPeft(llmRequest); } +TEST_F(PeftCacheManagerTest, unsupportedAdapterDataTypeDoesNotConfigureCache) +{ + SamplingConfig samplingConfig; + auto tokens = std::make_shared>(std::initializer_list{1, 2, 3, 4}); + TensorPtr int8Weights = mManager->cpu(loraWeightsTp2->getShape(), tensorrt_llm::DataType::kINT8); + auto int8Request = std::make_shared(0, 4, tokens, samplingConfig, false); + int8Request->setLoraTaskId(1234); + int8Request->setLoraWeights(int8Weights); + int8Request->setLoraConfig(loraConfigTp2); + + EXPECT_THAT([&]() { mPeftManager->addRequestPeft(int8Request); }, + testing::Throws(testing::Property( + &std::runtime_error::what, testing::HasSubstr("Unsupported LoRA weights dtype int8 for PEFT cache")))); + + auto floatRequest = std::make_shared(1, 4, tokens, samplingConfig, false); + floatRequest->setLoraTaskId(5678); + floatRequest->setLoraWeights(loraWeightsTp2); + floatRequest->setLoraConfig(loraConfigTp2); + EXPECT_NO_THROW(mPeftManager->addRequestPeft(floatRequest)); + EXPECT_EQ(mPeftManager->getDataType(), tensorrt_llm::DataType::kFLOAT); +} + +#if defined(ENABLE_FP8) +TEST_F(PeftCacheManagerTest, invalidAdapterDoesNotConfigureCacheDataType) +{ + SamplingConfig samplingConfig; + auto tokens = std::make_shared>(std::initializer_list{1, 2, 3, 4}); + TensorPtr fp8Weights = mManager->cpu(loraWeightsTp2->getShape(), tensorrt_llm::DataType::kFP8); + auto invalidRequest = std::make_shared(0, 4, tokens, samplingConfig, false); + invalidRequest->setLoraTaskId(1234); + invalidRequest->setLoraWeights(fp8Weights); + + EXPECT_THAT([&]() { mPeftManager->addRequestPeft(invalidRequest); }, + testing::Throws(testing::Property( + &std::runtime_error::what, testing::HasSubstr("must have both lora_weights and lora_keys")))); + + auto floatRequest = std::make_shared(1, 4, tokens, samplingConfig, false); + floatRequest->setLoraTaskId(5678); + floatRequest->setLoraWeights(loraWeightsTp2); + floatRequest->setLoraConfig(loraConfigTp2); + EXPECT_NO_THROW(mPeftManager->addRequestPeft(floatRequest)); + EXPECT_EQ(mPeftManager->getDataType(), tensorrt_llm::DataType::kFLOAT); +} + +TEST_F(PeftCacheManagerTest, adapterSelectsHomogeneousCacheDataType) +{ + SamplingConfig samplingConfig; + auto tokens = std::make_shared>(std::initializer_list{1, 2, 3, 4}); + TensorPtr fp8Weights = mManager->cpu(loraWeightsTp2->getShape(), tensorrt_llm::DataType::kFP8); + auto fp8Request = std::make_shared(0, 4, tokens, samplingConfig, false); + fp8Request->setLoraTaskId(1234); + fp8Request->setLoraWeights(fp8Weights); + fp8Request->setLoraConfig(loraConfigTp2); + + mPeftManager->addRequestPeft(fp8Request); + EXPECT_EQ(mPeftManager->getDataType(), tensorrt_llm::DataType::kFP8); + + auto floatRequest = std::make_shared(1, 4, tokens, samplingConfig, false); + floatRequest->setLoraTaskId(5678); + floatRequest->setLoraWeights(loraWeightsTp2); + floatRequest->setLoraConfig(loraConfigTp2); + EXPECT_THAT([&]() { mPeftManager->addRequestPeft(floatRequest); }, + testing::Throws(testing::Property( + &std::runtime_error::what, testing::HasSubstr("PEFT cache supports one homogeneous LoRA dtype")))); +} +#endif + TEST_F(PeftCacheManagerTest, putGet) { SamplingConfig sampleConfig; diff --git a/docker/Dockerfile.multi b/docker/Dockerfile.multi index 828859395043..ec9d789dda5a 100644 --- a/docker/Dockerfile.multi +++ b/docker/Dockerfile.multi @@ -118,12 +118,19 @@ ARG GITHUB_MIRROR="" ARG BUILD_WHEEL_ARGS="--clean" ARG BUILD_WHEEL_SCRIPT="scripts/build_wheel.py" RUN --mount=type=cache,target=/root/.cache/pip --mount=type=cache,target=${CCACHE_DIR} \ + if [ -n "$GITHUB_MIRROR" ]; then \ + export PIP_INDEX_URL="https://urm.nvidia.com/artifactory/api/pypi/pypi-remote/simple"; \ + fi && \ GITHUB_MIRROR=$GITHUB_MIRROR python3 ${BUILD_WHEEL_SCRIPT} ${BUILD_WHEEL_ARGS} FROM ${DEVEL_IMAGE} AS release WORKDIR /app/tensorrt_llm +ARG GITHUB_MIRROR="" RUN --mount=type=cache,target=/root/.cache/pip --mount=type=bind,from=wheel,source=/src/tensorrt_llm/build,target=/tmp/wheel \ + if [ -n "$GITHUB_MIRROR" ]; then \ + export PIP_INDEX_URL="https://urm.nvidia.com/artifactory/api/pypi/pypi-remote/simple"; \ + fi && \ TRTLLM_WHEEL=$(find /tmp/wheel -maxdepth 1 -name 'tensorrt_llm*.whl' -print -quit) && \ test -n "${TRTLLM_WHEEL}" && \ pip install "${TRTLLM_WHEEL}[mx]" diff --git a/docker/README.md b/docker/README.md index ca1fd2196b11..d355e92e3dd2 100644 --- a/docker/README.md +++ b/docker/README.md @@ -35,7 +35,7 @@ make -C docker release_run The full image name and tag can be controlled by supplying `IMAGE_WITH_TAG` to `make`: ```bash -make -C docker devel_push IMAGE_WITH_TAG="urm.nvidia.com/sw-tensorrt-docker/tensorrt-llm:dev" +make -C docker devel_push IMAGE_WITH_TAG="artifactory.nvidia.com/sw-tensorrt-llm-docker-local/tensorrt-llm:dev" ``` Containers can be started with the local user instead of `root` by appending `LOCAL_USER=1` to the run target: @@ -105,7 +105,7 @@ Start a new container using the same image as Jenkins using your local user acco make -C docker jenkins_run LOCAL_USER=1 ``` -If you do not have access to the [internal artifact repository](https://urm.nvidia.com/artifactory/sw-tensorrt-docker/tensorrt-llm/), you can instead either use the [NGC Develop +If you do not have access to the [internal artifact repository](https://artifactory.nvidia.com/artifactory/sw-tensorrt-llm-docker-local/tensorrt-llm/), you can instead either use the [NGC Develop image](#ngc-integration) or [build an image locally](#building-docker-images-with-gnu-make). #### Release images based on Jenkins image @@ -117,10 +117,10 @@ make -C docker trtllm_build CUDA_ARCHS="80-real;90-real" ``` Note that the above requires access to the Jenkins development image from the -[internal artifact repository](https://urm.nvidia.com/artifactory/sw-tensorrt-docker/tensorrt-llm/). +[internal artifact repository](https://artifactory.nvidia.com/artifactory/sw-tensorrt-llm-docker-local/tensorrt-llm/). The resulting images can be pushed to -the [internal artifact repository](https://urm.nvidia.com/artifactory/sw-tensorrt-docker/tensorrt-llm-staging/release/): +the [internal artifact repository](https://artifactory.nvidia.com/artifactory/sw-tensorrt-llm-docker-local/tensorrt-llm-staging/release/): ```bash make -C docker trtllm_push diff --git a/docs/source/blogs/tech_blog/blog17_Sparse_Attention_in_TensorRT-LLM.md b/docs/source/blogs/tech_blog/blog17_Sparse_Attention_in_TensorRT-LLM.md index 91274ffc5459..e964e1149bc7 100644 --- a/docs/source/blogs/tech_blog/blog17_Sparse_Attention_in_TensorRT-LLM.md +++ b/docs/source/blogs/tech_blog/blog17_Sparse_Attention_in_TensorRT-LLM.md @@ -220,7 +220,7 @@ Figure 4 illustrates the prediction implementation within TensorRT LLM. To suppo **Auxiliary memory management.** Managing the paged KT cache presented another challenge. `RocketKVCacheManager` inherits from `KVCacheManager` and extends it with a dedicated `BlockManager` for the auxiliary KT cache at the Python level. The main KV cache and the KT cache share block IDs for each request, so that the lifecycle of KT cache blocks is automatically tied to the corresponding KV cache blocks. The `BlockManager` handles slot allocation and deallocation for the KT cache independently, while `RocketKVCacheManager` overrides methods such as `get_cache_bytes_per_token` and `prepare_resources` to ensure that memory sizing accounts for the extra KT cache footprint and that the correct KT cache pointers are passed to prediction kernels at each step. This design keeps the integration lightweight and easy to iterate on, though it inherits the limitations of Python-level management—namely, no automatic support for KV cache reuse or disaggregated serving. -The concrete implementation can be found in `tensorrt_llm/_torch/attention_backend/sparse/rocket.py`. +The concrete implementation can be found in `tensorrt_llm/_torch/attention_backend/sparse/rocket/`. ### DeepSeek Sparse Attention (DSA) @@ -261,7 +261,7 @@ As with RocketKV, a dedicated metadata class `DSATrtllmAttentionMetadata` is def **Auxiliary memory management.** DSA requires an auxiliary **indexer K cache** to store the low-rank K projections for reuse across decoding steps. `DSAKVCacheManager` inherits from `KVCacheManager`, but unlike RocketKV's Python-level KT cache management, DSA's indexer K cache is integrated directly into the C++ `KVCacheManager`. This design enables compatibility with advanced features such as KV cache reuse, chunked prefill, and disaggregated serving—features that would be difficult to support with a Python-level manager. -The concrete implementation can be found in `tensorrt_llm/_torch/attention_backend/sparse/dsa.py`. +The concrete implementation can be found in `tensorrt_llm/_torch/attention_backend/sparse/dsa/`. For a comprehensive description of DSA kernel optimizations, precision strategies, feature support (MTP, disaggregated serving, Wide-EP), and benchmark results, please refer to the dedicated blog post: [Optimizing DeepSeek-V3.2 on NVIDIA Blackwell GPUs](blog15_Optimizing_DeepSeek_V32_on_NVIDIA_Blackwell_GPUs.md). diff --git a/docs/source/blogs/tech_blog/blog25_Scaling_Video_Generation_Across_NVL72_Rack_with_TensorRT-LLM.md b/docs/source/blogs/tech_blog/blog25_Scaling_Video_Generation_Across_NVL72_Rack_with_TensorRT-LLM.md index 21e929137a73..398dea4054a1 100644 --- a/docs/source/blogs/tech_blog/blog25_Scaling_Video_Generation_Across_NVL72_Rack_with_TensorRT-LLM.md +++ b/docs/source/blogs/tech_blog/blog25_Scaling_Video_Generation_Across_NVL72_Rack_with_TensorRT-LLM.md @@ -13,8 +13,8 @@ On Wan 2.2 T2V-A14B (a 5-second 1280×720 video, 40 steps), the `CFG=2 × Ulysse The same runtime carries over unchanged to Cosmos3-Super (a 64B Mixture-of-Transformers generating a 189-frame 1280×720 clip), where the identical `CFG=2 × Ulysses=4 × Attention2D 3×3` recipe delivers **~55× on the denoise loop (~33× end-to-end)** on the full GB200 NVL72 rack.

- Wan 2.2 T2V-A14B scaling from one B200 GPU to the 72-GPU GB200 NVL72 recipe: ~53x denoise, ~41x end-to-end - Cosmos3-Super scaling from one B200 GPU to the full 72-GPU GB200 NVL72 recipe: ~55x denoise, ~33x end-to-end + Wan 2.2 T2V-A14B scaling from one B200 GPU to the 72-GPU GB200 NVL72 recipe: ~53x denoise, ~41x end-to-end + Cosmos3-Super scaling from one B200 GPU to the full 72-GPU GB200 NVL72 recipe: ~55x denoise, ~33x end-to-end

Figure 1. The same VisualGen recipe scaling two models from a single GPU to a full GB200 NVL72 rack. Left: Wan 2.2 T2V-A14B (5-second 1280×720, 40 steps). Right: Cosmos3-Super (189-frame 1280×720, 35 steps) under the identical `CFG=2 × Ulysses=4 × Attention2D 3×3` recipe.

@@ -67,11 +67,11 @@ The rest of this post walks through each of these axes first, then shows how the ## CFG Parallelism -Classifier-Free Guidance evaluates the diffusion transformer twice per denoising step - once with the positive prompt embedding to produce $`\text{noise}_\text{cond}`$, once with the negative prompt embedding to produce $`\text{noise}_\text{uncond}`$ - and combines the two predictions as +Classifier-Free Guidance evaluates the diffusion transformer twice per denoising step - once with the positive prompt embedding to produce the conditional noise prediction, once with the negative prompt embedding to produce the unconditional one - and combines the two predictions as -```math -\text{noise} = \text{noise}_\text{uncond} + \text{guidance\_scale} \cdot (\text{noise}_\text{cond} - \text{noise}_\text{uncond}) -``` +$$ +\text{noise} = \text{noise}_\text{uncond} + \text{guidance scale} \cdot (\text{noise}_\text{cond} - \text{noise}_\text{uncond}) +$$ The two evaluations share weights, timestep, and shape, so they are perfectly parallelisable across two disjoint GPU groups. CFG parallelism exploits this by setting `cfg_size = 2`: the conditional stream runs on one half of the mesh and the unconditional stream on the other half. Each half runs its full DiT forward end to end with no cross-stream communication; only the two scalar noise tensors meet at the combine step. @@ -91,7 +91,7 @@ That near-linear scaling makes CFG the cheapest scaling axis available, and the The key insight behind Ulysses ([arXiv:2309.14509](https://arxiv.org/abs/2309.14509)) is that outside the attention operation, activations are already sharded across GPUs along the sequence axis — each rank holds $[B, S/P, H, D]$ (`P` is the sequence-parallel degree), and FFN, norms, and modulation all run locally on that shard with no communication. The problem is that standard attention is inherently global: a query at position $i$ needs to see all keys and values across the full sequence length. Ulysses solves this by swapping the sharding axis just for the attention operation. A pair of all-to-alls converts $[B, S/P, H, D] \rightarrow [B, S, H/P, D]$ before attention and back again after, so each rank sees the full sequence but only a $1/P$ slice of heads. The FFN and everything else never touch a collective (unless composed with an FFN-sharding scheme like TP, which adds its own).

- Ulysses sequence parallelism data layout: a pair of all-to-alls transposes a frame-sharded layout into a head-sharded layout for attention, then back + Ulysses sequence parallelism data layout: a pair of all-to-alls transposes a frame-sharded layout into a head-sharded layout for attention, then back

Figure 2. Ulysses keeps every stage except attention sequence-sharded - each rank owns a slice of the video frames (all heads), so QKV projection, norms, and the MLP run locally with no collective. A pair of all-to-alls transposes the layout to head-sharded (each rank owns all frames but a 1/P slice of heads) just for the full-sequence attention, then transposes back for the MLP.

@@ -107,12 +107,12 @@ The stacked QKV buffer produced by the all-to-all is a single contiguous allocat **Communication cost.** Ulysses parallelism involves **only two all-to-all communications per layer, regardless of sequence length**. For a sequence length $N$ and $P$ GPUs, each GPU sends a data chunk of size $O(N/P^2)$ to each of the other $P-1$ GPUs. This leads to a communication cost of $O(N/P)$. -**Limitation.** Ulysses sequence parallelism is constrained by the model’s attention head count: the Ulysses degree $P$ must not exceed the head count `num_heads` and must evenly divide `num_heads`; to use all GPUs in an NVL72 system uniformly, $P$ must also divide 72. For Wan2.2‑T2V‑A14B with $`\text{num\_heads} = 40`$, this constraint makes $P = 8$ the largest feasible Ulysses degree on NVL72. +**Limitation.** Ulysses sequence parallelism is constrained by the model’s attention head count: the Ulysses degree $P$ must not exceed the head count `num_heads` and must evenly divide `num_heads`; to use all GPUs in an NVL72 system uniformly, $P$ must also divide 72. For Wan2.2‑T2V‑A14B with $\text{num heads} = 40$, this constraint makes $P = 8$ the largest feasible Ulysses degree on NVL72. ### Toward Async Ulysses: Overlapping Communication and Computation

- Async Ulysses V-to-Q-to-K pipeline: each tensor's all-to-all runs on the Copy Engines and overlaps the next tensor's projection compute + Async Ulysses V-to-Q-to-K pipeline: each tensor's all-to-all runs on the Copy Engines and overlaps the next tensor's projection compute

Figure 3. Async Ulysses moves the Q/K/V all-to-all onto the GPU's Copy Engines on a separate stream and pipelines it V→Q→K, so each tensor's cross-rank transfer overlaps the next tensor's projection compute (V's transfer hides under Q's compute, Q's under K's), hiding the collective latency behind useful work.

@@ -170,28 +170,28 @@ Rank r holds Q_r (fixed) and K/V block for step i **Overlapping communication and compute.** The K/V exchange for the next block is posted with non-blocking point-to-point (`batch_isend_irecv`) *before* the current block's FA4 kernel runs, so the neighbor transfer overlaps with attention compute. Even ranks send-then-recv and odd ranks recv-then-send to avoid deadlock; the final ring step skips the exchange entirely. -**Communication cost.** Ring pays $P-1$ neighbor exchanges per attention layer, and communication volume scales as $O(N)$ in sequence length $N$ — more sequential steps than Ulysses. Each ring step also runs a partial FA4 pass plus an online-softmax merge, so attention work grows with ring degree $P$, not just communication. The upside is flexibility: **no head-count ceiling**, a simple 1D topology, and straightforward composition with Ulysses on the same mesh ($`\text{cp\_size} \times \text{ulysses\_size}`$). P2P can overlap with compute, which keeps Ring competitive at small $P$ and short sequences. For production video lengths, though, we **generally prefer Attention2D** (detailed in the next section) for the context-parallel axis, owing to its lower communication cost – $O(N/\sqrt{P})$ versus $O(N)$ for Ring attention. +**Communication cost.** Ring pays $P-1$ neighbor exchanges per attention layer, and communication volume scales as $O(N)$ in sequence length $N$ — more sequential steps than Ulysses. Each ring step also runs a partial FA4 pass plus an online-softmax merge, so attention work grows with ring degree $P$, not just communication. The upside is flexibility: **no head-count ceiling**, a simple 1D topology, and straightforward composition with Ulysses on the same mesh ($\text{cp size} \times \text{ulysses size}$). P2P can overlap with compute, which keeps Ring competitive at small $P$ and short sequences. For production video lengths, though, we **generally prefer Attention2D** (detailed in the next section) for the context-parallel axis, owing to its lower communication cost – $O(N/\sqrt{P})$ versus $O(N)$ for Ring attention. ### 2. Attention2D Parallelism -Attention2D parallelism ([arXiv:2503.15758](https://arxiv.org/abs/2503.15758)) treats the $P$ context-parallel GPUs as a logical 2D grid of $`\text{row\_size} \times \text{col\_size}`$ workers. It is our preferred **context-parallel strategy** once Ulysses has consumed the head-divisible fraction of the mesh and more GPUs remain. Like Ring, it shards the spatiotemporal sequence with **no head-count ceiling** — any $`\text{row\_size} \times \text{col\_size}`$ grid is valid regardless of how many attention heads the DiT has. It also **generalizes 1D context parallelism** - setting $`\text{col\_size}=1`$ reduces to the no-merge scheme (all-gather K/V), while setting $`\text{row\_size}=1`$ reduces to the merge scheme (all-gather Q). - and **scales the CP degree better**, gathering only $S/\sqrt{P}$ per axis ($O(N/\sqrt{P})$ communication) on a symmetric grid, which suits long-sequence DiT inference. The runtime adapts the original Attention2D design from causal LLM training to **full bidirectional** DiT inference. +Attention2D parallelism ([arXiv:2503.15758](https://arxiv.org/abs/2503.15758)) treats the $P$ context-parallel GPUs as a logical 2D grid of $\text{row size} \times \text{col size}$ workers. It is our preferred **context-parallel strategy** once Ulysses has consumed the head-divisible fraction of the mesh and more GPUs remain. Like Ring, it shards the spatiotemporal sequence with **no head-count ceiling** — any $\text{row size} \times \text{col size}$ grid is valid regardless of how many attention heads the DiT has. It also **generalizes 1D context parallelism** - setting $\text{col size}=1$ reduces to the no-merge scheme (all-gather K/V), while setting $\text{row size}=1$ reduces to the merge scheme (all-gather Q). - and **scales the CP degree better**, gathering only $S/\sqrt{P}$ per axis ($O(N/\sqrt{P})$ communication) on a symmetric grid, which suits long-sequence DiT inference. The runtime adapts the original Attention2D design from causal LLM training to **full bidirectional** DiT inference. -**Gather phase: Q along rows and K/V along columns.** Ranks form a $`\text{row\_size} \times \text{col\_size}`$ grid ($`P = \text{row\_size} \times \text{col\_size}`$). Each rank starts with a $1/P$ shard of all three tensors — $Q_i, K_i, V_i$, each $[B, S/P, H, D]$. Attention2D then runs two all-gathers on the two independent grid axes: +**Gather phase: Q along rows and K/V along columns.** Ranks form a $\text{row size} \times \text{col size}$ grid ($P = \text{row size} \times \text{col size}$). Each rank starts with a $1/P$ shard of all three tensors — $Q_i, K_i, V_i$, each $[B, S/P, H, D]$. Attention2D then runs two all-gathers on the two independent grid axes: -- **Q gather** over the **row group** (`row_process_group`): the `col_size` ranks that share a row. Concatenates their $Q_i$ → every member holds $`[B, S/\text{row\_size}, H, D]`$. K/V are untouched. -- **K/V gather** over the **column group** (`col_process_group`): the `row_size` ranks that share a column, fused into a single collective. Concatenates their $`K_i / V_i`$ → every member holds $`[B, S/\text{col\_size}, H, D]`$. Q is untouched. +- **Q gather** over the **row group** (`row_process_group`): the `col_size` ranks that share a row. Concatenates their $Q_i$ → every member holds $[B, S/\text{row size}, H, D]$. K/V are untouched. +- **K/V gather** over the **column group** (`col_process_group`): the `row_size` ranks that share a column, fused into a single collective. Concatenates their $K_i / V_i$ → every member holds $[B, S/\text{col size}, H, D]$. Q is untouched. -So **Q expands by `col_size`** (a row holds `col_size` ranks → $`S/\text{row\_size}`$) and **K/V expand by `row_size`** (a column holds `row_size` ranks → $`S/\text{col\_size}`$); on a symmetric $\sqrt{P} \times \sqrt{P}$ mesh both land at $S/\sqrt{P}$. The picture below uses a $2 \times 3$ grid (`row_size=2, col_size=3`, $P=6$), with $S$ split into shards $0..5$: +So **Q expands by `col_size`** (a row holds `col_size` ranks → $S/\text{row size}$) and **K/V expand by `row_size`** (a column holds `row_size` ranks → $S/\text{col size}$); on a symmetric $\sqrt{P} \times \sqrt{P}$ mesh both land at $S/\sqrt{P}$. The picture below uses a $2 \times 3$ grid (`row_size=2, col_size=3`, $P=6$), with $S$ split into shards $0..5$:

- Attention2D per-rank data flow on a 2x3 grid: gather Q over the row group, K/V over the column group, one attention pass per rank produces a partial output, then an all-to-all over the row group exchanges partials and a local flash_attn_combine reduction merges them + Attention2D per-rank data flow on a 2x3 grid: gather Q over the row group, K/V over the column group, one attention pass per rank produces a partial output, then an all-to-all over the row group exchanges partials and a local flash_attn_combine reduction merges them

Figure 4. Attention2D data flow traced for rank0 on a 2x3 grid (`row_size=2, col_size=3`, P=6). Each rank starts with a 1/P shard; ① an all-gather over the `row_process_group` collects Q, ② an all-gather over the `col_process_group` collects K/V, giving each rank the [B, S/row, H, D] queries x [B, S/col, H, D] keys it attends in one pass to produce a single partial output (with its LSE). The row group's ranks share the same Q but attend over disjoint K/V slices, so ③ an `all_to_all` over the `row_process_group` exchanges those partials, then a local `flash_attn_combine` reduction merges them into the final [B, S/P, H, D] shard.

Members of a **row (Q) group** end up with identical Q but **disjoint** K/V slices; members of a **column (K/V) group** share identical K/V but hold different Q. -**Local Attention.** The inner backend ([`Attention2DAttention`](https://github.com/NVIDIA/TensorRT-LLM/blob/main/tensorrt_llm/_torch/visual_gen/attention_backend/parallel.py)) runs attention with LSE returning a partial output $`[B, S/\text{row\_size}, H, D]`$ and a log-sum-exp (LSE) statistic $`[B, H, S/\text{row\_size}]`$. The LSE records the softmax normalizer for that partial pass. +**Local Attention.** The inner backend ([`Attention2DAttention`](https://github.com/NVIDIA/TensorRT-LLM/blob/main/tensorrt_llm/_torch/visual_gen/attention_backend/parallel.py)) runs attention with LSE returning a partial output $[B, S/\text{row size}, H, D]$ and a log-sum-exp (LSE) statistic $[B, H, S/\text{row size}]$. The LSE records the softmax normalizer for that partial pass. **Merge phase.** The `col_size` ranks that share the same Q each attended over a **disjoint slice of K/V** (above: ranks `0`, `1`, `2` all hold queries `{0,1,2}` but keys `{0,3}`, `{1,4}`, `{2,5}` respectively). Their FA4 outputs are partial results for the **same query positions** over **complementary key spans**, so they must be combined **within that row group** to recover the full softmax. For `col_size = 3`, the partial results are spread across 3 GPUs. These are combined into the final output as follows: @@ -212,10 +212,10 @@ Members of a **row (Q) group** end up with identical Q but **disjoint** K/V slic | **Communication cost** | $O(N)$ | $O(N / \sqrt{P})$ | | **Scaling** | No scaling: communication cost remains constant as $P$ increases | Efficient scaling: communication cost decreases as $O(1 / \sqrt{P})$ as $P$ increases | -**Communication cost.** Across the full layer, Attention2D runs three collective phases: Q all-gather (row), K/V all-gather (column), and output+LSE all-to-all (row). On a symmetric mesh ($`\text{row\_size} \approx \text{col\_size} \approx \sqrt{P}`$), moved volume scales as $O(N / \sqrt{P})$ in sequence length $N$, compared to $O(N)$ for Ring — the same head-count freedom, but better communication scaling and one attention pass instead of $P$. +**Communication cost.** Across the full layer, Attention2D runs three collective phases: Q all-gather (row), K/V all-gather (column), and output+LSE all-to-all (row). On a symmetric mesh ($\text{row size} \approx \text{col size} \approx \sqrt{P}$), moved volume scales as $O(N / \sqrt{P})$ in sequence length $N$, compared to $O(N)$ for Ring — the same head-count freedom, but better communication scaling and one attention pass instead of $P$.

- Bar chart of denoise-loop latency for Attention2D vs Ring at 32 GPU (UL=8, CP=2), 64 GPU (UL=8, CP=4), and 72 GPU (UL=4, CP=9); Ring is +3%, +13%, and +83% slower respectively + Bar chart of denoise-loop latency for Attention2D vs Ring at 32 GPU (UL=8, CP=2), 64 GPU (UL=8, CP=4), and 72 GPU (UL=4, CP=9); Ring is +3%, +13%, and +83% slower respectively

Figure 5. Denoise-loop latency for Attention2D vs Ring on GB200 NVL72 (Wan 2.2 T2V-A14B, 5-second 1280×720 video, 40 steps). `cfg=2` is fixed. Attention2D scales favorably with increasing context-parallel degree — its communication cost reduces as O(1 / sqrt{P}) while Ring Attention's remains flat at O(1), for a fixed sequence length N — yielding progressively larger latency gaps of +3% at CP=2, +13% at CP=4, and +83% at CP=9. Ring Attention suffers a sharp latency regression at 72 GPUs: to evenly divide the 36 available GPUs per cfg replica, the Ulysses parallelism degree (UL) must be reduced from 8 to 4, which increases communication volume along the CP axis. This amplifies Ring Attention's latency, as its communication volume scales as O(N). Attention2D is far less sensitive to the increased communication volume because it scales as O(N / sqrt{P}), allowing the higher parallelism degree to offset the additional volume.

@@ -227,7 +227,7 @@ After the DiT denoising loop finishes, a single latent tensor still has to be tu **Spatial sharding.** Unlike the DiT - which shards the *token sequence* - the VAE is a convolutional UNet-style stack that operates on a `(B, C, T, H, W)` video tensor, so the natural axis to split is **image space**. `parallel_vae_split_dim` selects height or width, and the latent (decode) or video (encode) tensor is chunked along that dimension across the `parallel_vae_size` ranks. Each rank owns a $1/P$ spatial slice and decodes it locally; a single all-gather at the end reconstructs the full frame:

- Parallel VAE: route the final latent to the parallel_vae_size VAE-group ranks while the rest idle, split image space along W into 1/P slices, decode each slice locally, all-gather into the full video, and use halo exchange so boundary convolutions stay correct + Parallel VAE: route the final latent to the parallel_vae_size VAE-group ranks while the rest idle, split image space along W into 1/P slices, decode each slice locally, all-gather into the full video, and use halo exchange so boundary convolutions stay correct

Figure 6. Parallel VAE decode. The full latent is split along an image-space axis (`W` here) into 1/P slices across the `parallel_vae_size` VAE-group ranks (the rest idle); each rank decodes its slice locally — boundary convolutions exchange only a thin k-1 "halo" with their neighbors (edge ranks zero-pad), and attention blocks gather the full split dimension — then a final all-gather reconstructs the full video.

@@ -322,7 +322,7 @@ The headline result is summarized in [Figure 1](#scaling-video-generation-across Sweeping the best recipe at every width on GB200 NVL72 shows the DiT denoise loop scaling almost linearly across the full rack:

- WAN 2.2 T2V-A14B DiT denoise-loop latency vs ideal linear scaling on GB200 NVL72, from 1 to 72 GPUs, with per-point parallel efficiency + WAN 2.2 T2V-A14B DiT denoise-loop latency vs ideal linear scaling on GB200 NVL72, from 1 to 72 GPUs, with per-point parallel efficiency

Figure 7. DiT denoise-loop scaling on GB200 NVL72 (best config at each width, GB200 1-GPU baseline). The loop tracks ideal linear scaling closely - ~49× at 72 GPUs - because CFG and sequence parallelism shard the transformer almost perfectly.

@@ -371,7 +371,7 @@ The headline result is summarized on the right of [Figure 1](#scaling-video-gene Sweeping the best recipe at every width gives the same near-linear denoise-loop scaling as Wan:

- Cosmos3-Super DiT denoise-loop latency vs ideal linear scaling on GB200 NVL72, with per-point parallel efficiency + Cosmos3-Super DiT denoise-loop latency vs ideal linear scaling on GB200 NVL72, with per-point parallel efficiency

Figure 8. Cosmos3-Super DiT denoise-loop scaling on GB200 NVL72 (best config at each width, GB200 1-GPU baseline). The loop tracks ideal linear scaling closely - ~49× at 72 GPUs - because CFG and sequence parallelism shard the transformer almost perfectly.

diff --git a/docs/source/blogs/tech_blog/blog26_DeepSeek_V4_on_NVIDIA_Blackwell_Model_Specific_and_Agentic_Workload_Optimizations_in_TensorRT-LLM.md b/docs/source/blogs/tech_blog/blog26_DeepSeek_V4_on_NVIDIA_Blackwell_Model_Specific_and_Agentic_Workload_Optimizations_in_TensorRT-LLM.md index 260b94448f42..7686ecc10075 100644 --- a/docs/source/blogs/tech_blog/blog26_DeepSeek_V4_on_NVIDIA_Blackwell_Model_Specific_and_Agentic_Workload_Optimizations_in_TensorRT-LLM.md +++ b/docs/source/blogs/tech_blog/blog26_DeepSeek_V4_on_NVIDIA_Blackwell_Model_Specific_and_Agentic_Workload_Optimizations_in_TensorRT-LLM.md @@ -93,7 +93,7 @@ DeepSeek-V4's largest architectural change is its attention. DeepSeek-V3.2 intro
- DeepSeek-V4 hybrid attention architecture. Panel a shows SWA attending over the latest 128 raw tokens. Panel b shows CSA combining the sliding window with 4× compressed KV entries selected by an Indexer and Top-K stage. Panel c shows HCA combining the sliding window with all 128× compressed KV entries without an Indexer. + DeepSeek-V4 hybrid attention architecture. Panel a shows SWA attending over the latest 128 raw tokens. Panel b shows CSA combining the sliding window with 4× compressed KV entries selected by an Indexer and Top-K stage. Panel c shows HCA combining the sliding window with all 128× compressed KV entries without an Indexer.

Figure 1. DeepSeek-V4 hybrid attention. (a) SWA attends densely within the 128-token sliding window. (b) CSA augments that window with Top-K entries selected from the 4× compressed history. (c) HCA attends to the window and the complete 128× compressed history without an Indexer.

@@ -116,7 +116,7 @@ TensorRT LLM implements the Compressor as a fused `wkv_gate` projection plus ded
- DeepSeek-V4 mHC architecture and MoE routing. Panel a shows the repeated transformer stack with mHC pre- and post-mappings around the attention and MoE blocks, followed by the HC head and language-model head. Panel b shows hash routing through a token-ID embedding table and learned routing through Sqrt-Softplus scores, correction bias, and Top-K expert selection. + DeepSeek-V4 mHC architecture and MoE routing. Panel a shows the repeated transformer stack with mHC pre- and post-mappings around the attention and MoE blocks, followed by the HC head and language-model head. Panel b shows hash routing through a token-ID embedding table and learned routing through Sqrt-Softplus scores, correction bias, and Top-K expert selection.

Figure 2. DeepSeek-V4 components beyond attention. (a) mHC wraps the attention and MoE sublayers with pre- and post-mappings, then uses an HC head to collapse the expanded residual stream before the language-model head. (b) The MoE router supports checkpoint-defined hash routing in the first three layers (left) and learned Sqrt-Softplus scoring with correction bias and Top-6 selection in later layers (right).

@@ -262,7 +262,7 @@ After the major kernels are optimized, runtime overhead shifts to the boundaries
- SWA scratch reuse during long-context prefill. Scratch-eligible KV blocks produced by the current prefill chunk and falling outside the final retained window reuse the same physical storage across SWA layers, while blocks inside the sliding window remain in the normal KV cache. + SWA scratch reuse during long-context prefill. Scratch-eligible KV blocks produced by the current prefill chunk and falling outside the final retained window reuse the same physical storage across SWA layers, while blocks inside the sliding window remain in the normal KV cache.

Figure 3. SWA scratch reuse during long-context prefill. Scratch-eligible blocks from the current prefill chunk reuse the same physical storage across SWA layers, while the final sliding-window state remains in the normal KV cache.

@@ -337,7 +337,7 @@ The following figure summarizes the closed-loop request path for aggregated and
- AA-AgentPerf closed-loop workflow. A simulated agent sends the next recorded turn with its accumulated conversation. The inference deployment routes the request, reuses matching KV blocks, prefills the uncached suffix, and streams the generated response. In disaggregated serving, the generation side retrieves the KV cache using context metadata. The client drains the complete response, simulates tool time when needed, advances the trajectory, and sends the next turn only after the current cycle completes. + AA-AgentPerf closed-loop workflow. A simulated agent sends the next recorded turn with its accumulated conversation. The inference deployment routes the request, reuses matching KV blocks, prefills the uncached suffix, and streams the generated response. In disaggregated serving, the generation side retrieves the KV cache using context metadata. The client drains the complete response, simulates tool time when needed, advances the trajectory, and sends the next turn only after the current cycle completes.

Figure 5. AA-AgentPerf closed-loop request workflow. Repeated conversation prefixes can be reused through the KV cache. Aggregated serving performs prefill and generation on the same worker. In disaggregated serving, context metadata allows the generation worker to retrieve the KV cache and continue decoding. Tool time is simulated by the client and consumes no LLM compute.

@@ -369,7 +369,7 @@ The DeepSeek-V3.2 study showed the central routing tradeoff: affinity without lo
- Two-level routing for context locality in disaggregated serving. The front-end orchestrator first selects a context server, then the selected server's attention data-parallel router selects a rank. At each level, the router can use either conversation-aware affinity or KV-cache-aware scoring. Full conversation-prefix reuse requires both decisions to return to the context server and rank that own the corresponding KV blocks. The selected context rank transfers the KV cache to a generation server for decoding. + Two-level routing for context locality in disaggregated serving. The front-end orchestrator first selects a context server, then the selected server's attention data-parallel router selects a rank. At each level, the router can use either conversation-aware affinity or KV-cache-aware scoring. Full conversation-prefix reuse requires both decisions to return to the context server and rank that own the corresponding KV blocks. The selected context rank transfers the KV cache to a generation server for decoding.

Figure 6. Two-level routing for context locality in disaggregated serving. The front-end orchestrator first selects a CTX server, and that server's ADP router then selects a rank. Conversation-aware routing uses explicit conversation affinity, while KV-cache-aware routing infers placement from reusable prefix blocks and load. Full conversation-prefix reuse requires both decisions to return the request to the CTX server and ADP rank that hold the corresponding KV blocks.

diff --git a/docs/source/commands/trtllm-bench.rst b/docs/source/commands/trtllm-bench.rst index 309422df3fba..54867d5f2868 100644 --- a/docs/source/commands/trtllm-bench.rst +++ b/docs/source/commands/trtllm-bench.rst @@ -1,7 +1,7 @@ trtllm-bench =========================== -trtllm-bench is a comprehensive benchmarking tool for TensorRT LLM engines. It provides three main subcommands for different benchmarking scenarios: +trtllm-bench is a comprehensive benchmarking tool for TensorRT LLM engines. It provides subcommands for different benchmarking scenarios. .. include:: ../_includes/note_sections.rst :start-after: .. start-note-config-flag-alias @@ -13,17 +13,12 @@ Syntax .. click:: tensorrt_llm.commands.bench:main :prog: trtllm-bench :nested: full - :commands: throughput, latency, build + :commands: throughput, latency, prepare-dataset Dataset preparation ------------------- - -prepare-dataset -^^^^^^^^^^^^^^^ - -trtllm-bench ships a ``prepare-dataset`` subcommand which generates benchmark datasets in the required format. It supports: +------------------- **Dataset Types:** @@ -33,136 +28,7 @@ trtllm-bench ships a ``prepare-dataset`` subcommand which generates benchmark da **Key Features:** -- Tokenizer integration for proper text preprocessing +- Tokenizer integration for proper text preprocessing — the tokenizer is resolved from the model passed to ``trtllm-bench --model`` - Configurable random seeds for reproducible results - Support for LoRA adapters and task IDs - Output in JSON format compatible with trtllm-bench - -.. note:: - The tokenizer is taken from the model passed to ``trtllm-bench --model``. Use ``--output`` to write the dataset to a file, or ``--stdout`` to stream it with a JSON dataset entry on each line. - -**Usage:** - -prepare-dataset -""""""""""""""" - -.. code-block:: bash - - trtllm-bench --model prepare-dataset [OPTIONS] - -**Options** - ----- - -.. list-table:: - :widths: 20 80 - :header-rows: 1 - - * - Option - - Description - * - ``--output`` - - Output JSON filename (default: preprocessed_dataset.json) - * - ``--stdout`` - - Print output to stdout with a JSON dataset entry on each line instead of writing a file - * - ``--random-seed`` - - Random seed for token generation (default: 420) - * - ``--task-id`` - - LoRA task ID (default: -1) - * - ``--rand-task-id`` - - Random LoRA task range (two integers) - * - ``--lora-dir`` - - Directory containing LoRA adapters - * - ``--log-level`` - - Logging level: info or debug (default: info) - -real-dataset -"""""""""""" - -Process real datasets from various sources. - -.. code-block:: bash - - trtllm-bench --model prepare-dataset real-dataset [OPTIONS] - -**Options** - ----- - -.. list-table:: - :widths: 20 80 - :header-rows: 1 - - * - Option - - Description - * - ``--input`` - - Input dataset file or directory (required) - * - ``--max-input-length`` - - Maximum input sequence length (default: 2048) - * - ``--max-output-length`` - - Maximum output sequence length (default: 512) - * - ``--num-samples`` - - Number of samples to process (default: all) - * - ``--format`` - - Input format: json, jsonl, csv, or txt (default: auto-detect) - - -token-norm-dist -""""""""""""""" - -Generate synthetic datasets with normal token distribution. - -.. code-block:: bash - - trtllm-bench --model prepare-dataset token-norm-dist [OPTIONS] - -**Options** - ----- - -.. list-table:: - :widths: 20 80 - :header-rows: 1 - - * - Option - - Description - * - ``--num-requests`` - - Number of requests to be generated (required) - * - ``--input-mean`` - - Normal distribution mean for input tokens (required) - * - ``--input-stdev`` - - Normal distribution standard deviation for input tokens (required) - * - ``--output-mean`` - - Normal distribution mean for output tokens (required) - * - ``--output-stdev`` - - Normal distribution standard deviation for output tokens (required) - - -token-unif-dist -""""""""""""""" - -Generate synthetic datasets with uniform token distribution - -.. code-block:: bash - - trtllm-bench --model prepare-dataset token-unif-dist [OPTIONS] - -**Options** - ----- - -.. list-table:: - :widths: 20 80 - :header-rows: 1 - - * - Option - - Description - * - ``--num-requests`` - - Number of requests to be generated (required) - * - ``--input-min`` - - Uniform distribution minimum for input tokens (required) - * - ``--input-max`` - - Uniform distribution maximum for input tokens (required) - * - ``--output-min`` - - Uniform distribution minimum for output tokens (required) - * - ``--output-max`` - - Uniform distribution maximum for output tokens (required) diff --git a/docs/source/deployment-guide/deployment-guide-for-deepseek-r1-on-trtllm.md b/docs/source/deployment-guide/deployment-guide-for-deepseek-r1-on-trtllm.md index 1a948213246e..8a993f0b08c3 100644 --- a/docs/source/deployment-guide/deployment-guide-for-deepseek-r1-on-trtllm.md +++ b/docs/source/deployment-guide/deployment-guide-for-deepseek-r1-on-trtllm.md @@ -29,8 +29,8 @@ There are multiple MOE backends inside TensorRT LLM, not all of them supporting | H100/H200 | FP8 | CUTLASS | | B200/GB200 EP<=8 | NVFP4 | CUTLASS, TRTLLM | | B200/GB200 EP<=8 | FP8 | DEEPGEMM | -| GB200 NVL72 EP>8 | NVFP4 | WIDEEP | -| GB200 NVL72 EP>8 | FP8 | WIDEEP without EPLB | +| GB200 NVL72 EP>8 | NVFP4 | CUTEDSL | +| GB200 NVL72 EP>8 | FP8 | DEEPGEMM | The default moe backend is `CUTLASS`, so for the combination which is not supported by `CUTLASS`, one must set the `moe_config.backend` explicitly to run the model. @@ -207,7 +207,7 @@ See the [`TorchLlmArgs` class](https://nvidia.github.io/TensorRT-LLM/llm-api/ref Add the following fields to the YAML configuration file `/tmp/config.yml` to enable wide EP: ```yaml moe_config: - backend: WIDEEP + backend: CUTEDSL max_num_tokens: 9216 load_balancer: # configure online EP balancer num_slots: 288 diff --git a/docs/source/deployment-guide/deployment-guide-for-kimi-k3-on-trtllm.md b/docs/source/deployment-guide/deployment-guide-for-kimi-k3-on-trtllm.md new file mode 100644 index 000000000000..af3d60c9e0e2 --- /dev/null +++ b/docs/source/deployment-guide/deployment-guide-for-kimi-k3-on-trtllm.md @@ -0,0 +1,359 @@ +# Deployment Guide for Kimi K3 on TensorRT LLM - Blackwell + +## Introduction + +This deployment guide provides step-by-step instructions for running the Kimi K3 model using TensorRT LLM on NVIDIA Blackwell GPUs. The deployment configurations and results in this guide were validated on NVIDIA GB300 NVL GPUs. It covers the complete setup required; from preparing the model weights and building the software environment to configuring TensorRT LLM parameters, launching the multi-node server, and validating inference output and accuracy. + +Kimi K3 is a large hybrid Mixture-of-Experts (MoE) model. Its 93 decoder layers interleave two attention families: most layers use Kimi Delta Attention (KDA), a linear-attention mixer that keeps a constant-size recurrent state per sequence, while every fourth layer uses full Multi-head Latent Attention (MLA). The MoE block routes each token to 16 of 896 experts; the routed expert weights are stored in MXFP4 (group size 32) while the attention, shared-expert, dense-MLP, and LM-head weights stay in BF16. In TensorRT LLM the model is served through the `KimiK3ForConditionalGeneration` / `KimiLinearForCausalLM` architectures (the text decoder path). + +This guide uses Slurm and the `trtllm-llmapi-launch` multi-node launcher. The configuration walkthrough focuses on the **DEP16** high-throughput deployment: attention runs data-parallel across 16 ranks and the 896 experts shard across the expert-parallel group (`enable_attention_dp: true`, `moe_expert_parallel_size: 16`). The repository also provides the **TEP16** low-latency deployment and an 8-GPU deployment, **TEP8**. + +## Prerequisites + +* GPU: NVIDIA Blackwell GPUs. DEP16 and TEP16 use 16 GPUs; the TEP8 recipe uses 8 GPUs. These deployment recipes were validated on GB300 NVL GPUs. The repository's Slurm examples assume 4 GPUs per node. Other GPU architectures are not currently supported. +* Multi-node launcher: Slurm with the pyxis/enroot container plugin (or an equivalent MPI launcher) to start one rank per GPU across the nodes. +* High-speed inter-node interconnect (e.g., NVLink/InfiniBand) for the expert-parallel traffic. +* Shared filesystem visible to all nodes for the repository, the model weights, and the configuration file. +* OS: Linux +* Drivers: CUDA Driver 580 or later +* Container runtime with NVIDIA GPU support on each node + +## Models + +* A complete Hugging Face-format Kimi K3 checkpoint and tokenizer, e.g. [moonshotai/Kimi-K3](https://huggingface.co/moonshotai/Kimi-K3) downloaded from the Hugging Face Hub (`model_type: kimi_k3`, architecture `KimiK3ForConditionalGeneration`). The checkpoint ships the routed-expert weights pre-quantized to MXFP4; no additional quantization step is needed. + +The checkpoint and the configuration file must live on a shared filesystem visible to all nodes. The examples below use `/path/to/kimi-k3-checkpoint` — replace it with your local path. + +## Feature Support Notes + +* **Blackwell only.** NVIDIA Blackwell GPUs are supported. The configurations and results in this guide were validated on NVIDIA GB300 NVL GPUs. Support for other GPU architectures may be added in a future release. +* **High-throughput and low-latency deployments are provided.** DEP16 (`enable_attention_dp: true`, `moe_expert_parallel_size: 16`) is the high-throughput deployment. TEP16 (`enable_attention_dp: false`, `moe_expert_parallel_size: 16`) is the low-latency deployment. An 8-GPU deployment, TEP8 (`enable_attention_dp: false`, `moe_expert_parallel_size: 8`), is also provided. Select the deployment and concurrency appropriate for your workload. +* **CUDA graphs and the overlap scheduler are enabled.** The performance-sweep recipes set `disable_overlap_scheduler: false` and enable CUDA graphs. DEP16 additionally sets `cuda_graph_config.enable_padding: true`. +* **Chunked prefill is supported and enabled** (`enable_chunked_prefill: true`), so prompts longer than `max_num_tokens` are scheduled across multiple steps. +* **`kv_cache_config.tokens_per_block` must be `64`** — required by the MLA (576, 512) trtllm-gen generation kernel. +* **Speculative decoding and disaggregated serving are not yet available** for Kimi K3; support is under development. See the "Current limitations" section of `examples/kimi_k3/README.md`. + +## Deployment Steps + +### Build TensorRT LLM from Source + +Kimi K3 support currently requires TensorRT LLM built from source and installed in place. Inside the TensorRT LLM container, from the repository root: + +```bash +python3 scripts/build_wheel.py --cuda_architectures 103-real --skip_building_wheel --yes +.venv-3.12/bin/python -m pip install -e . +``` + +`build_wheel.py` creates the virtual environment at the repository root, named after the container's Python version: `.venv-3.12` for the current containers (Python 3.12). If your container ships a different Python, substitute the matching `.venv-.` path in the commands on this page. Adjust `--cuda_architectures` to the target GPUs (`103-real` for GB300). The multi-node jobs below run TensorRT LLM from this in-place environment, so build and install with the repository at the same path the jobs use. + +Kimi K3 additionally depends on `fla` and `einops`, installed into the same in-place environment (these dependencies might be removed in future releases, replaced by other kernels): + +```bash +.venv-3.12/bin/python -m pip install fla-core einops +``` + +To use the optimized CuTeDSL MLA kernel, install the FlashInfer revision used by the Kimi K3 example into the same in-place environment: + +```bash +.venv-3.12/bin/python -u -m pip install --force-reinstall --no-deps \ + --no-build-isolation \ + "flashinfer-python[cu13] @ git+https://github.com/PerkzZheng/flashinfer-k3.git@b6cc594918baf76c40c3a6236fd53f0f8fb9d2dc" +``` + +The `packaging>=24.2` requirement of this source build is already satisfied by `requirements.txt`. The TensorRT LLM environment already provides FlashInfer's runtime dependencies. The `--no-deps` option prevents `pip` from replacing the pinned PyTorch, Triton, CUDA, and CuTeDSL packages. Install FlashInfer after TensorRT LLM because a later dependency-resolving TensorRT LLM installation can replace this source revision with the currently pinned `flashinfer-python==0.6.16`. + +For general build-from-source instructions see [https://nvidia.github.io/TensorRT-LLM/latest/installation/build-from-source.html](https://nvidia.github.io/TensorRT-LLM/latest/installation/build-from-source.html). + +### Recommended Performance Settings + +**Treat these as a starting point and tune the parameters for your workload.** + +Create a YAML configuration file on the shared filesystem. The settings below are the tested DEP16 configuration, matching `examples/kimi_k3/eval_extra_llm_options.yaml`: + +```shell +EXTRA_LLM_API_FILE=/path/to/kimi-k3-config.yml + +cat << EOF > ${EXTRA_LLM_API_FILE} +# Kimi K3 DEP16 deployment: attention runs data-parallel and the 896 +# experts shard across the expert-parallel group. +tensor_parallel_size: 16 +enable_attention_dp: true +moe_expert_parallel_size: 16 +max_batch_size: 32 +max_num_tokens: 8192 +max_seq_len: 8192 +trust_remote_code: true +disable_overlap_scheduler: false +enable_chunked_prefill: true +cuda_graph_config: + enable_padding: true + max_batch_size: 32 +moe_config: + max_num_tokens: 33024 + use_low_precision_moe_combine: true +kv_cache_config: + enable_block_reuse: false + free_gpu_memory_fraction: 0.25 + tokens_per_block: 64 +EOF +``` + +Notes: + +* The configuration uses `free_gpu_memory_fraction: 0.25` to leave runtime headroom. Each KDA layer also keeps a per-request recurrent state outside the paged KV pool, so reduce this value if the deployment runs out of memory. +* This YAML is specifically the DEP16 high-throughput configuration. For the TEP16 low-latency and TEP8 8-GPU configurations, use `examples/kimi_k3/perf_sweep/perf_sweep.sbatch`; changing only `enable_attention_dp` does not reproduce those recipes. + +### Launch the TensorRT LLM Server + +Kimi K3 is launched through the `trtllm-llmapi-launch` wrapper, which sets up the multi-rank (MPI/Slurm) environment that the parallel server requires. The wrapper is run once per rank by Slurm (`srun`), with one task (rank) per GPU. The example below launches the server across 4 nodes (`-N 4`), 4 GPUs per node (`--ntasks-per-node 4`, 16 ranks total), using the YAML file to drive parallelism, batching, and the KV-cache constraints: + +```bash +MODEL=/path/to/kimi-k3-checkpoint +REPO=/path/to/TensorRT-LLM # in-place build from the previous step + +srun -N 4 \ + --ntasks 16 --ntasks-per-node 4 \ + --mpi=pmix --gpus-per-node=4 \ + --container-image=/path/to/tensorrt-llm-container.sqsh \ + --container-mount-home \ + --container-mounts=${REPO}:${REPO},${MODEL}:${MODEL},${EXTRA_LLM_API_FILE}:${EXTRA_LLM_API_FILE}:ro \ + bash -c " + ulimit -n 65536 + + # Node-local Triton cache: ~/.triton on shared NFS races across + # ranks during autotune JIT (stale file handles). + export TRITON_CACHE_DIR=/tmp/triton-cache-rank\${SLURM_PROCID:-0} + mkdir -p \"\$TRITON_CACHE_DIR\" + + # Node-local flashinfer cache: the default (\$HOME) is shared NFS and + # races across ranks during cubin download (stale file handles). + export FLASHINFER_WORKSPACE_BASE=/tmp/flashinfer-rank\${SLURM_PROCID:-0} + + # Run from the in-place installation created by build_wheel.py. + export PATH=\"${REPO}/.venv-3.12/bin:\$PATH\" + + exec ${REPO}/tensorrt_llm/llmapi/trtllm-llmapi-launch \ + trtllm-serve ${MODEL} \ + --host 0.0.0.0 --port 8000 \ + --config ${EXTRA_LLM_API_FILE} + " +``` + +> [!NOTE] +> Adjust `-N`, `--ntasks`, `--ntasks-per-node`, and `--gpus-per-node` to match your cluster's GPUs-per-node; the total number of tasks (ranks) must equal `tensor_parallel_size` (`16`). Add the partition / account / time flags (`-p`, `-A`, `-t`) required by your Slurm setup, and ensure the repository, checkpoint, and YAML paths resolve identically on all nodes (shared filesystem). + +TensorRT LLM will load weights and select the best kernels during startup. The server is successfully launched when the following log is shown: + +```log +INFO: Started server process [xxxxx] +INFO: Waiting for application startup. +INFO: Application startup complete. +INFO: Uvicorn running on http://localhost:8000 (Press CTRL+C to quit) +``` + +### Quick Start Without a Server (LLM API) + +For a first functional check without standing up a server, the repository ships a ready-made multi-node quick-start job that loads the model once, runs four prompts, and reports whether each response contains the expected text (a successful run reports `True` for all four checks): + +```bash +sbatch examples/kimi_k3/quick_start_kimi_k3.sbatch \ + --model /path/to/kimi-k3-checkpoint \ + --image /path/to/tensorrt-llm-container.sqsh +``` + +See [`examples/kimi_k3/README.md`](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/kimi_k3) for details, including how to override the Slurm partition, account, and time limit. + +### Key YAML Options + +These options are set within the YAML file passed to `trtllm-serve` via the `--config` argument. Only the Kimi-K3-relevant knobs are described here; see the [`TorchLlmArgs` class](https://nvidia.github.io/TensorRT-LLM/llm-api/reference.html#tensorrt_llm.llmapi.TorchLlmArgs) for the full reference. + +#### `tensor_parallel_size` + +* **Description:** The width of the parallel group. Set it to `16` for DEP16 and TEP16, or `8` for TEP8. + +#### `enable_attention_dp` + +* **Description:** Set it to `true` for the DEP16 high-throughput deployment. The TEP16 low-latency and TEP8 8-GPU deployments set it to `false`. + +#### `moe_expert_parallel_size` + +* **Description:** The expert-parallel width for the MoE layers. Set it to `16` for DEP16 and TEP16, or `8` for TEP8. + +#### `cuda_graph_config` + +* **Description:** The standard DEP16, TEP16, and TEP8 performance recipes enable CUDA graphs. DEP16 also sets `enable_padding: true`, which pads generation batches to a captured size. Set the CUDA-graph `max_batch_size` to the same value as the top-level `max_batch_size`. + +#### `disable_overlap_scheduler` + +* **Description:** Set it to `false` to enable the overlap scheduler, as used by the standard DEP16, TEP16, and TEP8 performance recipes. + +#### `moe_config` + +* **Options:** + + * `max_num_tokens`: The DEP16 performance recipe uses `33024` to reduce serialized all-to-all chunk rounds during prefill. + * `use_low_precision_moe_combine`: Set to `true` in the DEP16 performance recipe to use a low-precision expert-parallel combine payload. + +#### `enable_chunked_prefill` + +* **Description:** The recommended DEP16 configuration sets this option to `true`. Prompts longer than `max_num_tokens` are then scheduled across multiple prefill steps; this is verified on 16-GPU DEP16 at GSM8K parity with the unchunked baseline. + +#### `kv_cache_config` + +* **Options:** + * `enable_block_reuse`: Off by default; set to `true` to enable prefix-cache reuse across requests. + * `mamba_state_config.periodic_snapshot_interval`: With block reuse on, the KDA recurrent state is snapshotted every this many tokens so prefix hits can restore it (default `0` = snapshots off; hybrid models only expose reusable prefixes at snapshot boundaries, so set e.g. `256` for block reuse to engage; see `examples/kimi_k3/eval_extra_llm_options_reuse.yaml`). + * `tokens_per_block`: Must be `64`, required by the MLA (576, 512) trtllm-gen generation kernel. + * `free_gpu_memory_fraction`: Fraction of free GPU memory reserved for the paged KV cache after model load. The configuration above uses `0.25` to leave runtime headroom. Lower it if you hit out-of-memory errors. + +#### `trust_remote_code` + +* **Description:** Required to load the Kimi K3 configuration and tokenizer code shipped with the checkpoint. + +## Testing API Endpoint + +The server (the OpenAI-compatible REST endpoint) runs on the rank-0 node, listening on port `8000`. Send requests to that node's hostname or IP; `localhost` only works from the rank-0 node itself. + +### Health Check + +```shell +curl -s -o /dev/null -w "Status: %{http_code}\n" "http://localhost:8000/health" +``` + +When the `Status: 200` code is returned, the server is ready for queries. Note that the very first query may take longer due to initialization and compilation. + +### Basic Test + +After the TensorRT LLM server is set up and shows `Application startup complete`, you can send requests to the server: + +```shell +curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ + "model": "/path/to/kimi-k3-checkpoint", + "messages": [ + { + "role": "user", + "content": "Where is New York?" + } + ], + "max_tokens": 64, + "temperature": 0 +}' -w "\n" +``` + +The response should contain a `choices[0].message.content` field completing the request, plus a `usage` section with the token counts. + +## Running Evaluations to Verify Accuracy (Optional) + +The repository ships a ready-made multi-node GSM8K evaluation job built on `trtllm-eval` with the tested DEP16 settings: + +```bash +sbatch examples/kimi_k3/run_gsm8k_kimi_k3.sbatch \ + --model /path/to/kimi-k3-checkpoint \ + --image /path/to/tensorrt-llm-container.sqsh +``` + +The job writes progress and results to `kimi-k3-gsm8k-.log` in the submission directory. If no local dataset path is configured, `trtllm-eval` downloads GSM8K from the Hugging Face Hub. The completed log contains a results table with the normalized GSM8K exact-match scores. With the tested checkpoint and the settings in this example, users should expect approximately: + +| Filter | Exact match | +| :-- | --: | +| Flexible extract | 96.51 | +| Strict match | 96.44 | + +The expected average accuracy is approximately 96.47. Small differences (roughly ±0.5 points) are possible with different checkpoint or dependency revisions. + +## Benchmarking Performance + +### Run the End-to-End Performance Sweep + +Use `examples/kimi_k3/perf_sweep/perf_sweep.sbatch` to reproduce the performance measurements. The script generates the tuned serving configuration for the selected deployment mode, launches the server, waits for it to become healthy, runs a warmup, and measures each requested concurrency: + +```bash +sbatch examples/kimi_k3/perf_sweep/perf_sweep.sbatch \ + --mode dep16 \ + --model /path/to/kimi-k3-checkpoint \ + --image /path/to/tensorrt-llm-container.sqsh \ + --isl 8192 \ + --osl 1024 \ + --concurrencies "64 128 256" +``` + +Set `--mode` to `dep16`, `tep16`, or `tep8`. The script derives `max_seq_len` from the requested input and output lengths as `ISL + OSL + 128`, and writes the server log, client log, generated YAML, and per-concurrency JSON results to the job's output directory. + +### Benchmark an Existing Server + +Launch the server with the deployment YAML that you want to measure, as described in [Launch the TensorRT LLM Server](#launch-the-tensorrt-llm-server). From a machine that can reach the rank-0 server, choose the input length, output length, and concurrency for your workload: + +```bash +MODEL=/path/to/kimi-k3-checkpoint +HOST=localhost +PORT=8000 +ISL=8192 +OSL=1024 +CONCURRENCIES="64 128 256" +NUM_ROUNDS=5 +RESULT_DIR=/path/to/kimi-k3-results + +mkdir -p "${RESULT_DIR}" + +for concurrency in ${CONCURRENCIES}; do + python3 -m tensorrt_llm.serve.scripts.benchmark_serving \ + --model "${MODEL}" \ + --host "${HOST}" \ + --port "${PORT}" \ + --dataset-name random \ + --random-ids \ + --tokenize-on-client \ + --random-input-len "${ISL}" \ + --random-output-len "${OSL}" \ + --num-prompts "$((concurrency * NUM_ROUNDS))" \ + --max-concurrency "${concurrency}" \ + --ignore-eos \ + --trust-remote-code \ + --save-result \ + --result-dir "${RESULT_DIR}" \ + --result-filename "concurrency_${concurrency}.json" +done +``` + +Before running the benchmark, adjust `ISL` and `OSL` for the server configuration being measured. Their sum must not exceed the server's `max_seq_len`. To reproduce the 8K input / 1K output results below, use the performance-sweep script, which sets `max_seq_len: 9344` (`ISL + OSL + 128`). + +### Sample Measured Results + +The following results were measured on GB300 NVL GPUs with the 8K/1K sweep. All requests completed. `TPS/user` is `1000 / median TPOT`; it represents the median decode rate seen by one active user. Your results may vary with checkpoint, software, and cluster I/O revisions. + +For the exact server settings and environment variables used to collect the results, refer to `examples/kimi_k3/perf_sweep/perf_sweep.sbatch`. + +| Mode | GPUs | Concurrency | Output tok/s | Output tok/s/GPU | Median TTFT (ms) | Median TPOT (ms) | TPS/user | +| :-- | --: | --: | --: | --: | --: | --: | --: | +| TEP16 | 16 | 4 | 201.2 | 12.6 | 1769.2 | 18.12 | 55.18 | +| DEP16 | 16 | 64 | 1669.1 | 104.3 | 4512.3 | 34.06 | 29.36 | +| DEP16 | 16 | 256 | 3823.9 | 239.0 | 4586.7 | 62.24 | 16.07 | + +The TEP16 point illustrates the low-latency deployment at small concurrency. The DEP16 points illustrate aggregate-throughput scaling at higher concurrencies. + +### Key Metrics + +#### Time to First Token (TTFT) + * The typical time elapsed from when a request is sent until the first output token is generated. + +#### Time Per Output Token (TPOT) and Inter-Token Latency (ITL) + * TPOT is the typical time required to generate each token *after* the first one. + * ITL is the typical time delay between the completion of one token and the completion of the next. + * Both TPOT and ITL ignore TTFT. + +#### End-to-End (E2E) Latency + * The typical total time from when a request is submitted until the final token of the response is received. + +#### Total Token Throughput + * The combined rate at which the system processes both input (prompt) tokens and output (generated) tokens. + +## Troubleshooting Tips + +* **Multi-node startup hangs or ranks can't find each other:** Verify that the repository, the checkpoint, and the YAML resolve identically on all nodes (shared filesystem), that the total Slurm task count equals `tensor_parallel_size` (`16`), and that the inter-node interconnect is healthy. +* **Stale file handle / cache errors during startup:** Point `TRITON_CACHE_DIR` and `FLASHINFER_WORKSPACE_BASE` at node-local storage (e.g., `/tmp`) as shown in the launch command; the shared-NFS defaults race across ranks during autotune JIT and cubin download. +* **CUDA out-of-memory errors:** Reduce `kv_cache_config.free_gpu_memory_fraction`, `max_batch_size`, or `max_seq_len`. Remember that the KDA layers keep per-request FP32 recurrent state outside the paged KV pool. +* **Job reaches its Slurm time limit before the model loads:** Weight loading for a checkpoint of this size is dominated by filesystem throughput; request a longer allocation (`--time=...`), particularly with cold caches or a busy filesystem. +* **Model fails to load:** Make sure `trust_remote_code: true` is set and that the checkpoint path is a complete Hugging Face-format Kimi K3 checkpoint with its tokenizer files. +* **`fla` / `einops` import errors:** Install the extra dependencies into the in-place environment: `.venv-3.12/bin/python -m pip install fla-core einops`. +* **Accuracy looks off:** Confirm `tokens_per_block: 64` (a hard constraint for Kimi K3), and that the remaining settings match the tested configuration above. +* **GPU utilization:** For performance issues, check GPU utilization with `nvidia-smi` on the compute nodes while the server is running. diff --git a/docs/source/deployment-guide/index.rst b/docs/source/deployment-guide/index.rst index eaa43fde0d97..006f63fe87a2 100644 --- a/docs/source/deployment-guide/index.rst +++ b/docs/source/deployment-guide/index.rst @@ -36,5 +36,6 @@ The deployment guides below provide more detailed instructions for serving speci deployment-guide-for-qwen3-on-trtllm.md deployment-guide-for-qwen3.5-on-trtllm.md deployment-guide-for-kimi-k2-thinking-on-trtllm.md + deployment-guide-for-kimi-k3-on-trtllm.md deployment-guide-for-glm-5-on-trtllm.md deployment-guide-for-minimax-m3-on-trtllm.md diff --git a/docs/source/developer-guide/overview.md b/docs/source/developer-guide/overview.md index d8fe31631612..104a6957e330 100644 --- a/docs/source/developer-guide/overview.md +++ b/docs/source/developer-guide/overview.md @@ -122,3 +122,32 @@ export TLLM_LOG_LEVEL_BY_MODULE="debug:_torch,runtime;info:serve" ``` This example sets the global level to `warning` but enables `debug` output for `_torch` and `runtime` modules, and `info` for `serve`. Valid levels: `trace`, `debug`, `verbose`, `info`, `warning`, `error`, `internal_error`. + +## Multi-Rank Crash Handling + +When one rank's executor loop dies in a multi-rank job, the peer ranks have +nothing to fail on: they sit in a collective waiting for a rank that will never +arrive. Left alone they hold every GPU in the job until an external wall-clock +kill. + +To avoid that, a rank whose executor loop crashes tears the whole job down +(`MPI_Abort`, falling back to self-`SIGKILL`) after a short grace period. The +grace exists so cleaner paths can win the race: if the crash is reported to a +client first — the caller gets the real traceback — the kill stands down, so a +symmetric crash still ends in N tracebacks rather than a bare exit 137. + +**This is on by default for multi-rank runs.** Single-rank runs are unaffected; +there are no peers to strand. + +| Variable | Default | Meaning | +|---|---|---| +| `TLLM_RANK_CRASH_HARD_KILL_GRACE` | `10` | Seconds to wait before killing the job. `0` kills immediately. Any negative value (e.g. `-1`) **disables** the kill entirely — peers then fall back to the hang detector. Unparsable and non-finite values (`nan`, `inf`) fall back to the default. | + +```bash +# Opt out: let the hang detector handle it instead (slower, but no world kill). +export TLLM_RANK_CRASH_HARD_KILL_GRACE=-1 +``` + +Use the escape hatch when you would rather have peer ranks reach their own +timeouts — for example when attaching a debugger to a surviving rank, or when a +harness collects per-rank state that a job-wide abort would destroy. diff --git a/docs/source/developer-guide/sparse-attention-development-guide.md b/docs/source/developer-guide/sparse-attention-development-guide.md index c645d4ec4514..14a74899a153 100644 --- a/docs/source/developer-guide/sparse-attention-development-guide.md +++ b/docs/source/developer-guide/sparse-attention-development-guide.md @@ -137,19 +137,30 @@ KV eviction are tracked as future work. subclasses override: ```python -sparse_kv_indices, sparse_kv_offsets = self.sparse_kv_predict(q, k, metadata, **kwargs) -sparse_attn_indices, sparse_attn_offsets = self.sparse_attn_predict(q, k, metadata, **kwargs) +sparse_kv_indices, sparse_kv_offsets = self.sparse_kv_predict(q, k, metadata, forward_args) +sparse_attn_indices, sparse_attn_offsets = self.sparse_attn_predict(q, k, metadata, forward_args) ``` +`hooks.py` writes these results to `SparseRuntimeParams`. SkipSoftmax writes +its thresholds to the same runtime interface consumed by `AttentionOp`. +`AttentionForwardArgs.sparse_backend_args` carries algorithm inputs from the +module to the backend, while `sparse_runtime_params` carries lowered inputs +from the backend to `AttentionOp`. + Different KV heads are allowed to emit different sparse index sets; Q heads that map to the same KV head share the KV head's sparse pattern. Algorithm implementations live under `tensorrt_llm/_torch/attention_backend/sparse/`: -- `rocket.py`, `dsa.py` — concrete algorithms. -- `kernel.py` — custom Triton kernels (importance scoring, Top-K). -- `utils.py` — dispatch helpers. +- `rocket/` — RocketKV backend, metadata, cache manager, parameters, and kernels. +- `dsa/` — DSA backend, indexer, metadata, cache manager, parameters, custom ops, and kernels. +- `deepseek_v4/` — DeepSeek-V4 backend, indexer, metadata, cache manager, + parameters, module hooks, and index conversion kernels. +- `skip_softmax/` — SkipSoftmax parameter parsing and runtime scheduler. +- `hooks.py` — typed MLA/Attention module adapters and common backend + prediction orchestration. +- `registry.py` — backend, metadata, and cache-manager dispatch helpers. ### AttentionOp behavior @@ -231,7 +242,7 @@ Create a new backend class inheriting from `TrtllmAttention` (or `tensorrt_llm/_torch/attention_backend/sparse/`. Override one or both prediction methods. -**`sparse_kv_predict(self, q, k, metadata, **kwargs)`** +**`sparse_kv_predict(self, q, k, metadata, forward_args)`** - **Behavior**: return the indices of tokens to retain in the KV cache. - **Outputs**: @@ -245,7 +256,7 @@ prediction methods. in-place gather (`updateSparseKvCacheAfterFmha`) is safe. The sort cost buys compatibility with chunked prefill and similar features. -**`sparse_attn_predict(self, q, k, metadata, **kwargs)`** +**`sparse_attn_predict(self, q, k, metadata, forward_args)`** - **Behavior**: return the sparse indices used by the generation-phase attention computation. @@ -285,9 +296,12 @@ If the algorithm needs extra tensors beyond the main KV cache: ### 4. Registration and dispatch - Register the new config + backend in - `tensorrt_llm/_torch/attention_backend/sparse/utils.py` and + `tensorrt_llm/_torch/attention_backend/sparse/registry.py` and `tensorrt_llm/_torch/pyexecutor/_util.py` so the runtime routes requests to your backend when the config is present. +- If the algorithm customizes module-layer behavior, implement and register a + concrete `MLASparseHooks` or `AttentionSparseHooks` adapter from the + algorithm's `module.py`. - If your algorithm exposes new C++ parameters, plumb them through `cpp/tensorrt_llm/thop/attentionOp.cpp` and `cpp/tensorrt_llm/kernels/sparseAttentionKernels.h`. @@ -318,5 +332,3 @@ for the kernel-side specifics. eviction as a compromise that keeps KV cache flexibility manageable. - **Unified auxiliary memory management** — let custom auxiliary pools inherit KV-cache features (reuse, offloading) by default. -- **Code refactoring** — as more algorithms land, unify the - framework-level scaffolding for maintainability. diff --git a/docs/source/developer-guide/telemetry.md b/docs/source/developer-guide/telemetry.md index b6b26dcd4ec1..c95b0674be6a 100644 --- a/docs/source/developer-guide/telemetry.md +++ b/docs/source/developer-guide/telemetry.md @@ -106,7 +106,7 @@ unset or when the safety sanitizer rejects the runtime value. | `iter_stats_max_iterations` | `Optional[int]` | `value` | | | | `kv_cache_config.attention_dp_events_gather_period_ms` | `` | `value` | | | | `kv_cache_config.avg_seq_len` | `Optional[Annotated[int, Gt(gt=0)]]` | `value` | | | -| `kv_cache_config.block_reuse_config.block_reuse_policy` | `Literal['all_reusable', 'per_request', 'per_conversation']` | `categorical` | | `all_reusable`, `per_request`, `per_conversation` | +| `kv_cache_config.block_reuse_config.policy` | `Literal['all_reusable', 'per_request', 'per_conversation']` | `categorical` | | `all_reusable`, `per_request`, `per_conversation` | | `kv_cache_config.copy_on_partial_reuse` | `` | `value` | | | | `kv_cache_config.cross_kv_cache_fraction` | `Optional[float]` | `value` | | | | `kv_cache_config.disk_cache_size` | `Optional[Annotated[int, Ge(ge=0)]]` | `value` | | | diff --git a/docs/source/features/kvcache.md b/docs/source/features/kvcache.md index b8b395abdc4d..576927c59475 100644 --- a/docs/source/features/kvcache.md +++ b/docs/source/features/kvcache.md @@ -74,6 +74,34 @@ scheduler_config: enable_prefix_aware_scheduling: false ``` +### Selecting the KV Cache Manager + +TensorRT LLM ships two KV cache manager implementations. `use_kv_cache_manager_v2` +selects between them and defaults to `auto`, which adopts the model's own +preference and falls back to the V1 C++ manager for models that do not declare +one. Set it to `true` or `false` to override the model default. + +Models that select the V2 manager by default: + +| Model | Reason | +| --- | --- | +| Hybrid Mamba (NemotronH, Qwen3-Next) | Attention KV and Mamba state pools must be sized together | +| DeepSeek-V4 | Sparse attention attaches auxiliary per-layer buffers | +| GPT-OSS | Sliding window on every other layer (VSWA), so the sliding-window and full-attention pools are sized independently | + +Separately, Gemma4 hybrid attention and sparse-attention models are routed to +V2 unconditionally: their per-layer buffer layouts cannot be represented by V1's +unified pool, so `use_kv_cache_manager_v2` does not apply to them. + +Two-model speculative decoding (for example Eagle3 with +`eagle3_one_model=False`) is not supported by V2: the draft model runs in a +separate engine with its own KV cache manager, and V2 sizes both managers from +the full `max_gpu_total_bytes` budget instead of partitioning it between them. +Under `auto`, a model default of V2 falls back to V1 for that combination; +setting `use_kv_cache_manager_v2: true` explicitly raises an error. This +applies to every model that selects its manager through +`use_kv_cache_manager_v2`, not just GPT-OSS. + ### Mamba Snapshot Boundaries Hybrid Mamba models must retain the recurrent Mamba state together with the @@ -97,6 +125,9 @@ kv_cache_config: enable_block_reuse: true use_kv_cache_manager_v2: true avg_seq_len: 2048 + block_reuse_config: + policy: per_conversation + max_num_turns: 2 mamba_state_config: periodic_snapshot_interval: 0 additional_snapshot_offsets_from_start: [128] @@ -111,8 +142,9 @@ If neither `avg_seq_len` nor an explicit `pool_ratio` is configured, hybrid Mamba models warn and fall back to half of `max_seq_len`, which can produce a suboptimal pool split. Exact explicit boundaries currently require `MambaHybridCacheManagerV2`, `max_beam_width=1`, and no KV connector. Hybrid -Mamba models select V2 by default when -`use_kv_cache_manager_v2: auto`; set it to `false` to select the V1 C++ +Mamba models select V2 by default (see +[Selecting the KV Cache Manager](#selecting-the-kv-cache-manager)); set +`use_kv_cache_manager_v2` to `false` to select the V1 C++ compatibility manager. In disaggregated serving, V2 Mamba requires the Python NIXL transceiver (`transceiver_runtime: PYTHON`); V1 routes support periodic snapshots only. diff --git a/docs/source/features/sampling.md b/docs/source/features/sampling.md index 3bee98bb87e7..4f8eb101f450 100644 --- a/docs/source/features/sampling.md +++ b/docs/source/features/sampling.md @@ -72,12 +72,61 @@ llm.generate(["Hello, my name is", sampling_params_1]) ``` +### Model generation config defaults + +The PyTorch backend can use compatible sampling defaults explicitly specified +in a model's `generation_config.json`. This behavior is opt-in: + +```python +from tensorrt_llm import LLM + +llm = LLM(model='nvidia/Llama-3.1-8B-Instruct-FP8', + generation_config='auto') +``` + +For `trtllm-serve`, enable it on the command line: + +```bash +trtllm-serve nvidia/Llama-3.1-8B-Instruct-FP8 --generation-config auto +``` + +or in the server YAML configuration: + +```yaml +generation_config: auto +``` + +The `generation_config` option has two modes: + +* `trtllm` (default) keeps the TRT-LLM sampling behavior and defaults. +* `auto` loads supported sampling values from the model's + `generation_config.json`. + +In `auto` mode, values are resolved in this order: + +1. A value explicitly specified by the request. +2. A value explicitly present in `generation_config.json`. +3. The existing default for the LLM API or serving protocol. + +The supported fields are `temperature`, `top_p`, `top_k`, `min_p`, +`repetition_penalty`, `no_repeat_ngram_size`, `length_penalty`, and +`early_stopping` when its value is a boolean or integer. Defaults synthesized +by Hugging Face Transformers for fields absent from the JSON file are not +applied. + +TRT-LLM's existing model-specific handling of `eos_token_id`, BART +`forced_bos_token_id`, and Whisper suppression tokens remains active in both +modes. + ### LLM API sampling behavior when using Torch Sampler * The sampling is controlled via `SamplingParams`. * By default (`temperature = top_p = top_k = None`), greedy sampling is used - (unless min-p or top-p decay is active, see below). + (unless min-p or top-p decay is active, see below). With + `generation_config='auto'`, values explicitly specified in the model's + `generation_config.json` take the place of these defaults; see + [Model generation config defaults](#model-generation-config-defaults). * If either `temperature = 0`, `top_p = 0`, `top_k = 1`, and/or `min_p = 1`, is specified, sampling is greedy, irrespective of the values of the remaining parameters. @@ -120,7 +169,7 @@ llm.generate(["Hello, my name is", * Top-P decay is not supported in combination with beam search or with speculative decoding modes that route draft tokens through the Torch Sampler; such requests are rejected. -* Positive Min-P is not supported in combination with one-model speculative decoding. Such +* Positive Min-P is not supported in combination with one-model speculative decoding. Such requests are rejected at admission. * Occurrence penalties are supported: `repetition_penalty`, `presence_penalty` and diff --git a/docs/source/legacy/performance/perf-benchmarking.md b/docs/source/legacy/performance/perf-benchmarking.md index 8d1567aee020..8c7a54f8cfdb 100644 --- a/docs/source/legacy/performance/perf-benchmarking.md +++ b/docs/source/legacy/performance/perf-benchmarking.md @@ -110,7 +110,7 @@ of 128:128. To run the benchmark from start to finish, run the following commands: ```shell -trtllm-bench --tokenizer meta-llama/Llama-3.1-8B prepare-dataset --output /tmp/synthetic_128_128.txt token-norm-dist --input-mean 128 --output-mean 128 --input-stdev 0 --output-stdev 0 --num-requests 3000 +trtllm-bench --model meta-llama/Llama-3.1-8B prepare-dataset --output /tmp/synthetic_128_128.txt token-norm-dist --input-mean 128 --output-mean 128 --input-stdev 0 --output-stdev 0 --num-requests 3000 trtllm-bench --model meta-llama/Llama-3.1-8B build --dataset /tmp/synthetic_128_128.txt --quantization FP8 trtllm-bench --model meta-llama/Llama-3.1-8B throughput --dataset /tmp/synthetic_128_128.txt --engine_dir /tmp/meta-llama/Llama-3.1-8B/tp_1_pp_1 ``` @@ -207,7 +207,7 @@ directory. For example, to generate a synthetic dataset of 1000 requests with a 128/128 for [meta-llama/Llama-3.1-8B](https://huggingface.co/meta-llama/Llama-3.1-8B), run: ```shell -trtllm-bench --tokenizer meta-llama/Llama-3.1-8B prepare-dataset --output /tmp/synthetic_128_128.txt token-norm-dist --input-mean 128 --output-mean 128 --input-stdev 0 --output-stdev 0 --num-requests 3000 +trtllm-bench --model meta-llama/Llama-3.1-8B prepare-dataset --output /tmp/synthetic_128_128.txt token-norm-dist --input-mean 128 --output-mean 128 --input-stdev 0 --output-stdev 0 --num-requests 3000 ``` ### Building a Benchmark Engine diff --git a/docs/source/models/encoder-decoder.md b/docs/source/models/encoder-decoder.md index 5675eed4d742..8d09f6816282 100644 --- a/docs/source/models/encoder-decoder.md +++ b/docs/source/models/encoder-decoder.md @@ -45,7 +45,7 @@ The following table describes the supported and recommended configurations. | Beam search | Yes with V1 | Configure `max_beam_width` when constructing `LLM`, then set `use_beam_search=True` in `SamplingParams`. | | Attention backend | `TRTLLM` | Use this backend for encoder-decoder models. It is required when `tensor_parallel_size > 1`. | | Decoder CUDA graphs | Yes, except in FP32 | `CudaGraphConfig` captures decoder work. V1 supports greedy and beam search; V2 supports its single-beam path. FP32 encoder-decoder models decline capture at engine init and log a warning instead of failing. | -| Encoder CUDA graphs | No | `EncodeCudaGraphConfig` is disabled for encoder-decoder models. The encoder runs eagerly. | +| Encoder CUDA graphs | Yes | Set `encoder_cuda_graph_config=EncodeCudaGraphConfig(...)` and `encoder_max_batch_size`. Usually set `encoder_max_batch_size` lower than `max_batch_size`. The `TRTLLM` attention backend is required. | | Overlap scheduler | Yes | Enabled by default. V1 supports greedy decoding and beam search; V2 remains limited to `max_beam_width=1`. | | Tensor parallelism | Yes | Use `tensor_parallel_size > 1` with `attn_backend="TRTLLM"`. Attention head counts must be divisible by the TP size. | | Pipeline parallelism | No | Keep `pipeline_parallel_size=1`. | @@ -397,12 +397,12 @@ to return only the best hypothesis. Beam search expands decoder-side cache and compute requirements. Include this expansion when sizing the self-attention KV pool and CUDA graph batch sizes. -## Enable decoder CUDA graphs +## Enable encoder and decoder CUDA graphs -Pass `CudaGraphConfig` to capture and replay decoder iterations: +Configure the decoder and encoder graph grids separately: ```python -from tensorrt_llm.llmapi import CudaGraphConfig +from tensorrt_llm.llmapi import CudaGraphConfig, EncodeCudaGraphConfig llm = LLM( @@ -410,10 +410,19 @@ llm = LLM( backend="pytorch", attn_backend="TRTLLM", max_batch_size=8, + encoder_max_batch_size=2, + encoder_max_num_tokens=2048, cuda_graph_config=CudaGraphConfig( max_batch_size=8, enable_padding=True, ), + encoder_cuda_graph_config=EncodeCudaGraphConfig( + batch_sizes=[1, 2], + num_tokens=[128, 256, 512, 1024, 2048], + seq_lens=[128, 256, 512, 1024], + enable_padding=True, + ), + enable_encoder_decoder_mixed_cuda_graph=True, kv_cache_config=KvCacheConfig( free_gpu_memory_fraction=0.8, cross_kv_cache_fraction=0.5, @@ -421,14 +430,31 @@ llm = LLM( ) ``` -This configuration captures decoder work only; the encoder continues to run -eagerly. With beam search, graph batch sizes must cover the active decoder -sequences after beam expansion. Padding lets nearby runtime batch sizes reuse a -captured graph. - -Do not use `EncodeCudaGraphConfig` for an encoder-decoder model. The runtime -warns and disables it. Piecewise CUDA graphs through `TorchCompileConfig` are -also unsupported for this model type. +`cuda_graph_config` controls decoder and mixed decoder graphs. +`encoder_cuda_graph_config` controls encoder-forward graph buckets for batch +size, total packed tokens, and maximum sequence length. The +`encoder_max_batch_size` value is the hard encoder capacity and admission +limit. With beam search, decoder graph batch sizes must cover the active +decoder sequences after beam expansion. + +`max_batch_size` controls the total decoder concurrency, while +`encoder_max_batch_size` controls encoder microbatch admission. For better +performance, tune `encoder_max_batch_size`, `encoder_max_num_tokens`, and the +encoder CUDA graph buckets together for the production workload. Start with +`encoder_max_batch_size` smaller than `max_batch_size`, such as 2 versus 8, +then adjust the limits and capture buckets based on benchmark results. + +`enable_encoder_decoder_mixed_cuda_graph` is primarily a performance option. It +reduces CPU launch overhead for decoder iterations that mix newly admitted +context requests with ongoing generation requests. The option defaults to +`True`, but becomes effective only when the encoder and decoder graph +configurations produce usable capture shapes. Set it to `False` to disable +mixed graphs while retaining the separate encoder and decoder CUDA graphs. + +Passing `EncodeCudaGraphConfig` through `cuda_graph_config` remains unsupported +for encoder-decoder models; pass it through `encoder_cuda_graph_config` +instead. Piecewise CUDA graphs through `TorchCompileConfig` are also +unsupported for this model type. ## Control the overlap scheduler @@ -491,10 +517,22 @@ attn_backend: TRTLLM dtype: bfloat16 disable_overlap_scheduler: false enable_chunked_prefill: false +max_batch_size: 8 +encoder_max_batch_size: 2 +encoder_max_num_tokens: 1024 max_beam_width: 1 max_input_len: 512 max_num_tokens: 2048 max_seq_len: 512 +cuda_graph_config: + max_batch_size: 8 + enable_padding: true +encoder_cuda_graph_config: + batch_sizes: [1, 2] + num_tokens: [128, 256, 512, 1024] + seq_lens: [128, 256, 512] + enable_padding: true +enable_encoder_decoder_mixed_cuda_graph: true kv_cache_config: enable_block_reuse: false free_gpu_memory_fraction: 0.8 @@ -509,7 +547,6 @@ Start the server: ```bash trtllm-serve google/flan-t5-small \ --backend pytorch \ - --max_batch_size 4 \ --config enc-dec-config.yaml ``` @@ -549,68 +586,36 @@ Use these guidelines as a starting point: - Set `max_seq_len` to at least the larger of the maximum encoder input length and maximum decoded sequence length. The current encoder-decoder runtime uses this value while sizing both phases. -- Set `max_num_tokens` high enough for all encoder tokens admitted together and - for the active decoder tokens. This is especially important for mixed-length - batches. -- Increase `max_batch_size` for more concurrent requests. Beam width multiplies - the number of active decoder sequences but not the number of source requests. +- Set `max_num_tokens` high enough for the active decoder tokens. +- Set `encoder_max_num_tokens` high enough for all encoder tokens in one + encoder microbatch. This is especially important for mixed-length batches. +- Increase `max_batch_size` for more concurrent requests. Start with a smaller + `encoder_max_batch_size`, such as 2 when `max_batch_size=8`, to bound encoder + memory and admission cost without reducing decoder concurrency. Beam width + multiplies the number of active decoder sequences but not the number of + source requests. - Tune `free_gpu_memory_fraction` first, then tune `cross_kv_cache_fraction` based on whether the cross-attention or self-attention pool is exhausted. ## Performance -The following benchmarks compare the PyTorch backend with the legacy TensorRT -encoder-decoder path for large-batch inference. The measurements use BF16 on -one H100 80 GB GPU with greedy decoding, an output limit of 128 tokens, and -mixed encoder input lengths from 260 to 440 tokens. The Flan-T5-XL results are -the average of ten timed runs after three warmup runs. The BART results are the -average of 20 timed runs after five warmup runs. Executed-token throughput -includes the terminal EOS token when a sequence emits it. - -The PyTorch configuration uses the `TRTLLM` attention backend, the overlap -scheduler, the Python scheduler, decoder CUDA graphs with padding, KV cache -manager V1, `max_input_len=512`, `max_seq_len=1024`, and -`max_num_tokens=65536`. Block reuse and chunked prefill are disabled. The KV -cache uses `free_gpu_memory_fraction=0.3` and -`cross_kv_cache_fraction=0.5`. - -The legacy TensorRT configuration uses separate BF16 encoder and decoder -engines built for batch size 128 and beam width 1. The encoder supports 512 -input tokens and 65,536 tokens per iteration; the decoder supports a sequence -length of 129. The benchmark runs these engines through `ModelRunnerCpp` with -greedy `top_k=1` decoding and the same KV cache fractions. For BART, the legacy -TensorRT benchmark starts the decoder with token IDs `[2, 0]` and generates at -most 127 more tokens. The PyTorch LLM API applies the same decoder prefix -internally and counts token ID 0 as the first output token; customers do not -need to provide the decoder prefix. Both paths use token ID 2 as EOS and stop -when the model generates it naturally. If a sequence reaches the output limit, -it retains the model-selected final token and reports a length stop instead of -forcing EOS. This setup also lets beam search begin after the shared decoder -prefix without a per-step Python logits processor. - -### Flan-T5-XL - -For Flan-T5-XL, the PyTorch backend performs on par with the legacy TensorRT -path, with slightly lower latency and higher executed-token throughput across -the tested batch sizes. - -| Batch size | Legacy TensorRT latency | PyTorch latency | PyTorch latency improvement over legacy TensorRT | Legacy TensorRT executed tokens/s | PyTorch executed tokens/s | -| ---: | ---: | ---: | ---: | ---: | ---: | -| 32 | 727.6 ms | 706.1 ms | 3.0% | 3,153 | 3,312 | -| 64 | 1,225.0 ms | 1,136.7 ms | 7.2% | 3,863 | 4,184 | -| 128 | 2,056.8 ms | 1,999.3 ms | 2.8% | 4,601 | 4,768 | - -### BART-large-CNN - -For BART-large-CNN, the PyTorch backend has 21.9% to 36.0% higher latency than -the legacy TensorRT path across the tested batch sizes. - -| Batch size | Legacy TensorRT latency | PyTorch latency | PyTorch latency difference | Legacy TensorRT executed tokens/s | PyTorch executed tokens/s | -| ---: | ---: | ---: | ---: | ---: | ---: | -| 32 | 229.9 ms | 280.2 ms | 21.9% slower | 12,611 | 10,662 | -| 64 | 252.2 ms | 343.0 ms | 36.0% slower | 22,007 | 16,209 | -| 128 | 352.3 ms | 472.7 ms | 34.2% slower | 31,544 | 23,518 | +Configure encoder, decoder, and mixed decoder CUDA graphs for the expected +serving workload. With representative capture buckets, the PyTorch backend can +outperform the legacy TensorRT encoder-decoder path while avoiding the engine +build and checkpoint conversion steps. + +For example, a BF16 FLAN-T5 Large serving benchmark on one H100 80 GB GPU used +encoder CUDA graphs, padded decoder CUDA graphs, and mixed encoder-decoder CUDA +graphs. Compared with the legacy TensorRT path, the PyTorch backend delivered +65.8%, 12.6%, and 11.8% higher request throughput at concurrencies 8, 32, and +64, respectively. P99 latency was 51.6%, 31.0%, and 33.5% lower. + +Follow [Enable encoder and decoder CUDA graphs](#enable-encoder-and-decoder-cuda-graphs) +and choose capture buckets that cover the batch sizes, packed encoder token +counts, and sequence lengths expected in production. Capture grids that omit +common runtime shapes fall back to eager execution and can lose these +performance benefits. Performance depends on the model, request distribution, decoding settings, and GPU configuration. Benchmark with a representative workload before deployment. @@ -644,11 +649,12 @@ Check all of the following: - `pipeline_parallel_size=1` and `context_parallel_size=1`. - `enable_attention_dp=False`. -### CUDA graphs do not capture the encoder +### Encoder CUDA graphs fall back to eager execution -This is expected. `CudaGraphConfig` accelerates decoder iterations only. The -encoder path runs eagerly, and `EncodeCudaGraphConfig` is disabled for -encoder-decoder models. +Check that `encoder_cuda_graph_config` and `encoder_max_batch_size` are set, +that the encoder graph buckets cover the request shape, and that +`attn_backend="TRTLLM"`. Unsupported shapes and attention backends fall back to +eager encoder execution. ### Output quality differs from the Hugging Face example diff --git a/docs/source/models/supported-models.md b/docs/source/models/supported-models.md index 21ee5daf3433..9c09a53d1c9f 100644 --- a/docs/source/models/supported-models.md +++ b/docs/source/models/supported-models.md @@ -6,6 +6,7 @@ The following is a table of supported models for the PyTorch backend: | Architecture | Model | HuggingFace Example | | ------------------------------------ | ---------------------------------- | -------------------------------------------- | | `AfmoeForCausalLM` | Arcee Foundation MoE (Trinity) | `arcee-ai/Trinity-Mini` | +| `BartForConditionalGeneration` | BART | `facebook/bart-large-cnn` | | `BertForSequenceClassification` | BERT-based | `textattack/bert-base-uncased-yelp-polarity` | | `Cohere2ForCausalLM` | Command A | `CohereLabs/c4ai-command-a-03-2025` | | `DeciLMForCausalLM` | Nemotron | `nvidia/Llama-3_1-Nemotron-51B-Instruct` | @@ -34,6 +35,7 @@ The following is a table of supported models for the PyTorch backend: | `LagunaForCausalLM` | Laguna-XS | `poolside/laguna-XS.2` | | `LlamaForCausalLM` | Llama 3.1, Llama 3, Llama 2, LLaMA | `meta-llama/Meta-Llama-3.1-70B` | | `Llama4ForConditionalGeneration` | Llama 4 | `meta-llama/Llama-4-Scout-17B-16E-Instruct` | +| `MBartForConditionalGeneration` | mBART | `facebook/mbart-large-50-many-to-one-mmt` | | `MiniCPMV4_6ForConditionalGeneration` [^14]| MiniCPM-V 4.6 | `openbmb/MiniCPM-V-4.6` | | `MiniMaxM2ForCausalLM` [^5] | MiniMax M2/M2.1/M2.7 | `MiniMaxAI/MiniMax-M2.7` | | `MiniMaxM3SparseForConditionalGeneration` [^12]| MiniMax-M3 | `MiniMaxAI/MiniMax-M3` | @@ -58,6 +60,8 @@ The following is a table of supported models for the PyTorch backend: | `SkyworkR1V2ForConditionalGeneration` [^5] | Skywork R1V2, Skywork SWE | `Skywork/Skywork-R1V2-38B` | | `SmolLM3ForCausalLM` [^5] | SmolLM3 | `HuggingFaceTB/SmolLM3-3B` | | `Step3p7ForConditionalGeneration` [^8]| Step-3.7-Flash | `stepfun-ai/Step-3.7-Flash` | +| `T5ForConditionalGeneration` | T5, Flan-T5, ByT5 | `google/flan-t5-small` | +| `WhisperForConditionalGeneration` | Whisper | `openai/whisper-large-v3` | ## Model-Feature Support Matrix (Key Models) @@ -96,6 +100,24 @@ Note: Support for other models may vary. Features marked "N/A" are not applicabl [^13]: The Cosmos 3 family also supports visual generation through the VisualGen API. See [Visual Generation Models](#visual-generation-models). [^14]: Requires `transformers>=5.7.0`: MiniCPM-V 4.6 was upstreamed into transformers as a native model type (`minicpmv4_6`) and the checkpoint ships no remote code (`auto_map`) to fall back on. The Qwen3.5-hybrid text tower runs in BF16. Image, video, and text inputs are supported in this release (video reuses the same NaViT-packed vision path as image via `MiniCPMV4_6InputProcessor`). +# Encoder-Decoder Feature Support Matrix (PyTorch Backend) + +The following capabilities apply to the supported encoder-decoder architectures. For configuration guidance and +limitations, see [Use encoder-decoder models with the PyTorch backend](./encoder-decoder.md). + +| Model Architecture/Feature | Overlap Scheduler | Decoder CUDA Graph | Encoder CUDA Graph | KV Cache Manager V1 | KV Cache Manager V2 | Beam Search | Tensor Parallelism | Pipeline Parallelism | Chunked Prefill | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| `BartForConditionalGeneration` | Yes | Yes (except FP32) | Yes | Yes | Yes (single beam) | Yes (V1 only) | Yes | No | No (encoder phase) | +| `MBartForConditionalGeneration` | Yes | Yes (except FP32) | Yes | Yes | Yes (single beam) | Yes (V1 only) | Yes | No | No (encoder phase) | +| `T5ForConditionalGeneration` | Yes | Yes (except FP32) | Yes | Yes | Yes (single beam) | Yes (V1 only) | Yes | No | No (encoder phase) | +| `WhisperForConditionalGeneration` | Yes | Yes (except FP32) | No (feature inputs) | Yes | Yes (single beam) | Yes (V1 only) | Yes | No | No (encoder phase) | + +Decoder CUDA graphs support greedy and beam-search decoding with KV cache manager V1 and single-beam decoding with +V2. Encoder CUDA graphs support the token-input BART, mBART, and T5 families; Whisper's feature-driven audio encoder +runs eagerly. Use the `TRTLLM` attention backend for encoder-decoder models; tensor parallelism also requires attention +head counts divisible by the tensor parallel size. Chunked prefill is not supported for the encoder phase, so the +complete encoder input must fit in the iteration token budget. + # Multimodal Feature Support Matrix (PyTorch Backend) | Model Architecture/Feature | Overlap Scheduler | CUDA Graph | Chunked Prefill | Torch Sampler | TLLM C++ Sampler | KV Cache Reuse | Logits Post Processor | EPD Disaggregated Serving | Modality | diff --git a/docs/source/models/visual-generation.md b/docs/source/models/visual-generation.md index ab0c952f40bc..138e41d11c87 100644 --- a/docs/source/models/visual-generation.md +++ b/docs/source/models/visual-generation.md @@ -35,6 +35,7 @@ TensorRT-LLM **VisualGen** provides a unified inference stack for diffusion mode | `Wan-AI/Wan2.2-T2V-A14B-Diffusers` | Text-to-Video | | `Wan-AI/Wan2.2-I2V-A14B-Diffusers` | Image-to-Video | | `Wan-AI/Wan2.2-TI2V-5B-Diffusers` | Text-to-Video, Image-to-Video | +| `FastVideo/FastWan2.2-TI2V-5B-FullAttn-Diffusers` | Text-to-Video | | `Lightricks/LTX-2` | Text-to-Video (with Audio), Image-to-Video (with Audio) | | `Qwen/Qwen-Image` | Text-to-Image | | `Qwen/Qwen-Image-2512` | Text-to-Image | @@ -45,6 +46,7 @@ TensorRT-LLM **VisualGen** provides a unified inference stack for diffusion mode | `nvidia/Cosmos3-Super-Text2Image-4Step` | Text-to-Image (DMD2-distilled, fixed 4-step schedule) | | `nvidia/Cosmos3-Super-Image2Video-4Step` | Image-to-Video (DMD2-distilled, fixed 4-step schedule) | + Models are auto-detected from the checkpoint directory. Diffusers-format models are detected via `model_index.json`; LTX-2 monolithic safetensors checkpoints are detected via embedded metadata. The `AutoPipeline` registry selects the appropriate pipeline class automatically. ### Feature Matrix @@ -53,14 +55,15 @@ Models are auto-detected from the checkpoint directory. Diffusers-format models |---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| | **FLUX.1** | Yes | Yes | Yes | Yes | No | No [^1] | Yes | No | Yes | Yes | Yes | Yes | Yes | Yes | No | | **FLUX.2** | Yes | Yes | Yes | Yes | No | No [^1] | Yes | No | Yes | Yes | Yes | Yes | Yes | Yes | No | -| **Wan 2.1** | Yes | Yes | Yes | Yes | Yes [^7] | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | -| **Wan 2.1 VSA** [^2] | Yes | Yes | Yes | Yes | Yes [^7] | Yes | Yes | Yes | Yes | Yes | Yes | No | No | Yes | Yes | -| **Wan 2.2** | Yes | Yes | Yes [^3] | Yes | Yes [^7] | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | +| **Wan 2.1** | Yes | Yes | Yes | Yes | Yes [^8] | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | +| **Wan 2.1 VSA** [^2] | Yes | Yes | Yes | Yes | Yes [^8] | Yes | Yes | Yes | Yes | Yes | Yes | No | No | Yes | Yes | +| **Wan 2.2** | Yes | Yes | Yes [^3] | Yes | Yes [^8] | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | +| **FastWan 2.2** | Yes | Yes | No | No | No | No [^7] | No | No | Yes | Yes | Yes | No | No | No | No | | **LTX-2** | Yes | Yes | Yes [^4] | Yes | No | Yes | Yes | No | No | Yes | Yes | Yes | Yes | No | No | | **Qwen-Image** | Yes | Yes | Yes | Yes | No | Yes | Yes | No | Yes | Yes | Yes | Yes | Yes | No | No | | **Qwen-Image-Layered** [^6] | No | No | No | No | No | No | No | No | Yes | Yes | No | No | No | No | No | | **Qwen-Image-Edit-2511** | Yes | Yes | No | No | No | Yes | No | No | Yes | Yes | No | No | No | No | No | -| **Cosmos3** | Yes | Yes | No | No | Yes [^7] | Yes | Yes | Yes | Yes | Yes | Yes | No | No | Yes | No | +| **Cosmos3** | Yes | Yes | No | No | Yes [^8] | Yes | Yes | Yes | Yes | Yes | Yes | No | No | Yes | No | [^1]: FLUX models use embedded guidance and do not have a separate negative prompt path, so CFG parallelism is not applicable. @@ -72,7 +75,9 @@ Models are auto-detected from the checkpoint directory. Diffusers-format models [^6]: Qwen-Image-Layered supports baseline BF16 image-conditioned layer decomposition and returns the generated RGBA layer stack as a saveable image grid. FP8 blockwise, NVFP4, cache acceleration, attention-parallel/Sage/VSA backends, Tensor Parallelism, and `trtllm-serve` image-edit routing are not enabled for this pipeline yet. -[^7]: CPU offloading of Cosmos3 is currently validated for the text-to-video pipelines. +[^7]: `FastVideo/FastWan2.2-TI2V-5B-FullAttn-Diffusers` — a distilled version of Wan2.2-TI2V-5B with 3 denoising steps. CFG parallelism, TeaCache, and Cache-DiT are not applicable. + +[^8]: CPU offloading is currently validated for Wan and Cosmos3 text-to-video pipelines. ## Quick Start diff --git a/docs/source/torch/adding_custom_kernels.md b/docs/source/torch/adding_custom_kernels.md index 7a3bd12104ec..568ee683cf94 100644 --- a/docs/source/torch/adding_custom_kernels.md +++ b/docs/source/torch/adding_custom_kernels.md @@ -290,13 +290,13 @@ The adapter and env helpers live in [`tensorrt_llm/_torch/custom_ops/cutedsl_mat - **The kernel.** [`cpp/tensorrt_llm/kernels/IndexerKCacheScatter.h`](https://github.com/NVIDIA/TensorRT-LLM/blob/main/cpp/tensorrt_llm/kernels/IndexerKCacheScatter.h) declares `invokeIndexerKCacheScatter(...)`; [`cpp/tensorrt_llm/kernels/indexerKCacheScatter.cu`](https://github.com/NVIDIA/TensorRT-LLM/blob/main/cpp/tensorrt_llm/kernels/indexerKCacheScatter.cu) defines the `__global__` kernel and the host launcher. Picked up automatically by the `*.cu` glob in `cpp/tensorrt_llm/kernels/CMakeLists.txt`. - **The binding.** [`cpp/tensorrt_llm/thop/IndexerKCacheScatterOp.cpp`](https://github.com/NVIDIA/TensorRT-LLM/blob/main/cpp/tensorrt_llm/thop/IndexerKCacheScatterOp.cpp) validates inputs, fetches the stream, and calls the launcher; registers the op via `TORCH_LIBRARY_FRAGMENT(trtllm, m)` and `TORCH_LIBRARY_IMPL(trtllm, CUDA, m)`. A one-line addition to [`cpp/tensorrt_llm/thop/CMakeLists.txt`](https://github.com/NVIDIA/TensorRT-LLM/blob/main/cpp/tensorrt_llm/thop/CMakeLists.txt) wires it into `th_common` (this directory is not globbed). -- **The integration.** In [`tensorrt_llm/_torch/attention_backend/sparse/dsa.py`](https://github.com/NVIDIA/TensorRT-LLM/blob/main/tensorrt_llm/_torch/attention_backend/sparse/dsa.py), `_update_k_cache` calls `torch.ops.trtllm.indexer_k_cache_scatter_op(...)` in place of the prior Python scatter loop. +- **The integration.** In [`tensorrt_llm/_torch/attention_backend/sparse/dsa/indexer.py`](https://github.com/NVIDIA/TensorRT-LLM/blob/main/tensorrt_llm/_torch/attention_backend/sparse/dsa/indexer.py), `_update_k_cache` calls `torch.ops.trtllm.indexer_k_cache_scatter_op(...)` in place of the prior Python scatter loop. - **The tests.** [`tests/unittest/_torch/attention/sparse/test_dsa_indexer.py`](https://github.com/NVIDIA/TensorRT-LLM/blob/main/tests/unittest/_torch/attention/sparse/test_dsa_indexer.py) — `test_indexer_k_cache_scatter_custom_op` runs the CUDA op against a PyTorch reference on two layers of the same cache pool and asserts the outputs match. The full call chain is three lines: ```py -# tensorrt_llm/_torch/attention_backend/sparse/dsa.py +# tensorrt_llm/_torch/attention_backend/sparse/dsa/indexer.py torch.ops.trtllm.indexer_k_cache_scatter_op( k_fp8, k_scale, k_cache, metadata.slot_mapping_fp8, metadata.slot_mapping_scale, diff --git a/examples/constraints.txt b/examples/constraints.txt index 1aef1aa12fab..304b837372fc 100644 --- a/examples/constraints.txt +++ b/examples/constraints.txt @@ -1,3 +1,3 @@ -tensorrt_llm==1.3.0rc24 +tensorrt_llm==1.3.0rc25 evaluate~=0.4.1 rouge_score~=0.1.2 diff --git a/examples/disaggregated/slurm/benchmark/run_benchmark.sh b/examples/disaggregated/slurm/benchmark/run_benchmark.sh index 8fa89815704c..154bf7840d7c 100644 --- a/examples/disaggregated/slurm/benchmark/run_benchmark.sh +++ b/examples/disaggregated/slurm/benchmark/run_benchmark.sh @@ -5,6 +5,22 @@ set -e set -u trap 'echo "Error occurred at line $LINENO"; exit 1' ERR +# Container runtimes (pyxis/enroot) reset image-defined variables like PATH +# at container start, so values passed via srun --export are lost for them. +# Allow the launcher config to prepend entries from inside the container. +# The client_cmds srun passes these via --export with single-quoted values +# (submit.py convert_envs_to_str), and srun keeps the quotes literal — strip them. +TRTLLM_PATH_PREPEND="${TRTLLM_PATH_PREPEND:-}" +TRTLLM_PATH_PREPEND="${TRTLLM_PATH_PREPEND#\'}"; TRTLLM_PATH_PREPEND="${TRTLLM_PATH_PREPEND%\'}" +TRTLLM_PYTHONPATH_PREPEND="${TRTLLM_PYTHONPATH_PREPEND:-}" +TRTLLM_PYTHONPATH_PREPEND="${TRTLLM_PYTHONPATH_PREPEND#\'}"; TRTLLM_PYTHONPATH_PREPEND="${TRTLLM_PYTHONPATH_PREPEND%\'}" +if [ -n "${TRTLLM_PATH_PREPEND:-}" ]; then + export PATH="${TRTLLM_PATH_PREPEND}:${PATH}" +fi +if [ -n "${TRTLLM_PYTHONPATH_PREPEND:-}" ]; then + export PYTHONPATH="${TRTLLM_PYTHONPATH_PREPEND}${PYTHONPATH:+:${PYTHONPATH}}" +fi + # Add parameter validation if [ "$#" -lt 10 ]; then echo "Error: Missing required arguments, got $# arguments, args: $@" diff --git a/examples/disaggregated/slurm/benchmark/start_server.sh b/examples/disaggregated/slurm/benchmark/start_server.sh index ff8e5aa90277..690748a7d816 100644 --- a/examples/disaggregated/slurm/benchmark/start_server.sh +++ b/examples/disaggregated/slurm/benchmark/start_server.sh @@ -5,4 +5,26 @@ set -x config_file=$1 -trtllm-serve disaggregated -c ${config_file} -t 7200 -r 7200 +# Container runtimes (pyxis/enroot) reset image-defined variables like PATH +# at container start, so values passed via srun --export are lost for them. +# Allow the launcher config to prepend entries from inside the container. +# srun --export keeps any quotes in the exported values literal; strip them. +TRTLLM_PATH_PREPEND="${TRTLLM_PATH_PREPEND:-}" +TRTLLM_PATH_PREPEND="${TRTLLM_PATH_PREPEND#\'}"; TRTLLM_PATH_PREPEND="${TRTLLM_PATH_PREPEND%\'}" +TRTLLM_PYTHONPATH_PREPEND="${TRTLLM_PYTHONPATH_PREPEND:-}" +TRTLLM_PYTHONPATH_PREPEND="${TRTLLM_PYTHONPATH_PREPEND#\'}"; TRTLLM_PYTHONPATH_PREPEND="${TRTLLM_PYTHONPATH_PREPEND%\'}" +if [ -n "${TRTLLM_PATH_PREPEND:-}" ]; then + export PATH="${TRTLLM_PATH_PREPEND}:${PATH}" +fi +if [ -n "${TRTLLM_PYTHONPATH_PREPEND:-}" ]; then + export PYTHONPATH="${TRTLLM_PYTHONPATH_PREPEND}${PYTHONPATH:+:${PYTHONPATH}}" +fi + +# In-place (.pth-style) TRT-LLM installs may lack the trtllm-serve console +# script; fall back to the module entry point in that case. +trtllm_serve_cmd="trtllm-serve" +if ! command -v trtllm-serve >/dev/null 2>&1; then + trtllm_serve_cmd="python3 -m tensorrt_llm.commands.serve" +fi + +${trtllm_serve_cmd} disaggregated -c ${config_file} -t 7200 -r 7200 diff --git a/examples/disaggregated/slurm/benchmark/start_worker.sh b/examples/disaggregated/slurm/benchmark/start_worker.sh index 0a5b5897b773..3f695bd48a32 100644 --- a/examples/disaggregated/slurm/benchmark/start_worker.sh +++ b/examples/disaggregated/slurm/benchmark/start_worker.sh @@ -38,8 +38,30 @@ else export CUDA_VISIBLE_DEVICES=${cuda_devices} fi -# Clear UCX_TLS for specific clusters -unset UCX_TLS +# Container runtimes (pyxis/enroot) reset image-defined variables like PATH +# at container start, so values passed via srun --export are lost for them. +# Allow the launcher config to prepend entries from inside the container. +# srun --export keeps any quotes in the exported values literal; strip them. +TRTLLM_PATH_PREPEND="${TRTLLM_PATH_PREPEND:-}" +TRTLLM_PATH_PREPEND="${TRTLLM_PATH_PREPEND#\'}"; TRTLLM_PATH_PREPEND="${TRTLLM_PATH_PREPEND%\'}" +TRTLLM_PYTHONPATH_PREPEND="${TRTLLM_PYTHONPATH_PREPEND:-}" +TRTLLM_PYTHONPATH_PREPEND="${TRTLLM_PYTHONPATH_PREPEND#\'}"; TRTLLM_PYTHONPATH_PREPEND="${TRTLLM_PYTHONPATH_PREPEND%\'}" +if [ -n "${TRTLLM_PATH_PREPEND:-}" ]; then + export PATH="${TRTLLM_PATH_PREPEND}:${PATH}" +fi +if [ -n "${TRTLLM_PYTHONPATH_PREPEND:-}" ]; then + export PYTHONPATH="${TRTLLM_PYTHONPATH_PREPEND}${PYTHONPATH:+:${PYTHONPATH}}" +fi + +# Clear UCX_TLS for specific clusters. Some clusters instead need an +# explicit transport list (e.g. NVL72 nodes whose verbs transports cannot +# initialize): set TRTLLM_WORKER_UCX_TLS in worker_env_var to re-pin +# UCX_TLS here, after the container-provided value is cleared. +if [ -n "${TRTLLM_WORKER_UCX_TLS:-}" ]; then + export UCX_TLS="${TRTLLM_WORKER_UCX_TLS}" +else + unset UCX_TLS +fi echo "SLURM_PROCID: ${SLURM_PROCID}, hostname: $(hostname), instance_id: ${instance_id}" echo "CUDA_VISIBLE_DEVICES: ${CUDA_VISIBLE_DEVICES}" @@ -63,7 +85,14 @@ else nsys_prefix="nsys profile -o ${nsys_file} -f true -t cuda,nvtx,python-gil -c cudaProfilerApi --cuda-graph-trace node --capture-range-end=stop --gpu-metrics-devices=none" fi +# In-place (.pth-style) TRT-LLM installs may lack the trtllm-serve console +# script; fall back to the module entry point in that case. +trtllm_serve_cmd="trtllm-serve" +if ! command -v trtllm-serve >/dev/null 2>&1; then + trtllm_serve_cmd="python3 -m tensorrt_llm.commands.serve" +fi + ${nsys_prefix} trtllm-llmapi-launch ${numa_bind_cmd} \ - trtllm-serve ${model_path} \ + ${trtllm_serve_cmd} ${model_path} \ --host $(hostname) --port ${port} \ --config ${config_file} diff --git a/examples/disaggregated/slurm/cache_transceiver_test/configs/kda_payload_kimi_k3.yaml b/examples/disaggregated/slurm/cache_transceiver_test/configs/kda_payload_kimi_k3.yaml new file mode 100644 index 000000000000..674c275890a8 --- /dev/null +++ b/examples/disaggregated/slurm/cache_transceiver_test/configs/kda_payload_kimi_k3.yaml @@ -0,0 +1,83 @@ +# Kimi K3 KDA-payload cache-transceiver micro-benchmark. +# Run from the repository root: +# python3 examples/disaggregated/slurm/cache_transceiver_test/submit.py \ +# -c examples/disaggregated/slurm/cache_transceiver_test/configs/kda_payload_kimi_k3.yaml +# +# Synthetic KV geometry sized so ONE request's per-rank transfer equals the +# exact per-request KDA state payload of Kimi K3: 454,459,392 bytes +# (69 layers x (conv [3*96*128, 4] bf16 + delta [96,128,128] fp32)). +# Per-rank bytes/token = 69 layers * kvFactor(2) * (24/4 kv heads) * 128 * 2B +# = 211,968 B -> 2144 tokens == 454,459,392 B exactly. +# request_lengths 512 / 1024 also match MLA-latent payloads at ~4k / ~8k ISL +# (bf16) for the KDA-vs-MLA comparison. +# +# Fill in the <...> placeholders for your cluster before submitting. + +slurm: + partition: "" + account: "" + job_time: "00:30:00" + job_name: "ctt_kda_payload" + extra_args: "--gpus-per-node=4" + +hardware: + gpus_per_node: 4 + +environment: + container_image: "" + # Mount the TRT-LLM checkout into the container. + container_mount: "" # Format: path1:path1,path2:path2 + # TRT-LLM comes from the checkout's venv (PATH below) + checkout code + # (PYTHONPATH below); skip pip install. + trtllm_repo: "" + # Output directory (results.json, logs/, csv/); point at any writable + # scratch location. + work_dir: "" + trtllm_wheel_path: "" + build_wheel: false + cuda_architectures: "" + +test_matrix: + combinations: + # NIXL/PYTHON dropped for now: V2 _exchange_rank_info mpi_allgather can + # hang under this harness (srun --mpi=pmix). + - {backend: "NIXL", runtime: "CPP"} + - {backend: "UCX", runtime: "CPP"} + cache_manager_versions: ["V1"] + # 512 tok = 108.5 MB/rank (~ MLA bf16 @4k ISL), 1024 = 217.1 MB (~ MLA @8k), + # 2144 = 454.46 MB = exact K3 per-request KDA payload. + request_lengths: [512, 1024, 2144] + num_requests_per_length: 8 + warmup_requests: 2 + +kv_cache: + num_layers: 69 + num_kv_heads: 24 # /tp4 -> 6 heads per rank + head_dim: 128 + tokens_per_block: 32 + dtype: "HALF" + max_tokens_in_buffer: 4096 + +parallel: + ctx_tp: 4 + ctx_pp: 1 + gen_tp: 4 + gen_pp: 1 + +ucx_env_sweep: + # Single sweep. PATH/PYTHONPATH are exported inside the container prelude so + # python3 resolves to the checkout's venv (system-site-packages=true, + # so container torch/mpi4py are visible) and the checkout on PYTHONPATH + # wins. Keep your cluster's SLURM bin dir on PATH so host-side srun works. + # The UCX_TLS pin (no verbs) is for clusters where verbs transports cannot + # initialize on the compute nodes; drop the pin where verbs works. + - name: "venv_no_verbs" + env: + UCX_TLS: "tcp,self,sm,cuda_copy,cuda_ipc" + PATH: ":/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:" + PYTHONPATH: "" + +run: + timeout_per_cell_s: 120 + max_sweep_s: 900 + capture_proto_info: true diff --git a/examples/disaggregated/slurm/cache_transceiver_test/run_cache_transceiver_test.py b/examples/disaggregated/slurm/cache_transceiver_test/run_cache_transceiver_test.py index c4ebd8c6460c..3bfb64a8dcbc 100644 --- a/examples/disaggregated/slurm/cache_transceiver_test/run_cache_transceiver_test.py +++ b/examples/disaggregated/slurm/cache_transceiver_test/run_cache_transceiver_test.py @@ -287,6 +287,35 @@ class _TransferError(Exception): pass +def _wait_ctx_complete(xcvr, rid, runtime): + """Block until this ctx request's send finishes (or errors). + + * C++ transceiver: check_context_transfer_status(None) is a true block-all; + a single call suffices (see the comment at the call site). + * PYTHON (V2) transceiver: even under block_all, each TxSession wait is + bounded by kv_transfer_sender_future_timeout_ms (default 1000 ms). A + slower peer handshake makes the call log "TxSession ... timed out" and + return with the request still DISAGG_CONTEXT_TRANS_IN_PROGRESS -- in the + real executor that is benign (the session stays open and is re-polled + every iteration), but returning here would free and refill the KV blocks + mid-flight, so the receiver reads the NEXT request's pattern (verify FAIL + on every request except the last). So poll until + this rid lands in the completed/failed lists. Collectively safe: every + ctx rank loops on the same rid and the per-call consensus makes all + ranks observe completion on the same iteration. The per-cell + signal.alarm and the hang detector bound the loop. + """ + if runtime != "PYTHON": + xcvr.check_context_transfer_status(None) + return + while True: + completed, failed = xcvr.check_context_transfer_status(None) + if rid in failed: + raise _TransferError(f"ctx transfer failed for rid={rid}") + if rid in completed: + return + + def _wait_gen_complete(xcvr, req, runtime): """Block until this gen request's receive finishes (or errors). @@ -372,7 +401,9 @@ def run_one_request( # transfer is still in progress. NIXL/UCX cold-start connection setup can # exceed that, so the harness would free the request mid-transfer, leaving # the gen side hung and the ctx sender thread asserting on a freed session. - xcvr.check_context_transfer_status(None) + # The PYTHON runtime additionally needs a poll loop on top of block_all; + # see _wait_ctx_complete. + _wait_ctx_complete(xcvr, rid, runtime) state = req.state tensorrt_llm.logger.info(f"[ctx r{rank}] rid={rid}: transfer DONE (send), state={state}") free_sequence(kvm, req, kv_handle, use_v2) @@ -648,6 +679,13 @@ def arm_watchdog(combination_idx, reqlen_idx, what): backend=backend, transceiver_runtime=(None if runtime == "CPP" else "PYTHON"), max_tokens_in_buffer=cfg["kv_cache"]["max_tokens_in_buffer"], + # PYTHON (V2) only; 0 keeps bounce off. With bounce on, the KV data + # rides a fabric-VMM staging buffer (CU_MEM_HANDLE_TYPE_FABRIC), which + # is what lets UCX pick cuda_ipc across NVL72 nodes -- direct + # pool-to-pool transfers from non-fabric allocations fall back to + # much slower host-staged tcp, so enable bounce for cross-node + # transfers inside an NVLink domain. + kv_cache_bounce_size_mb=int(cfg["kv_cache"].get("bounce_size_mb", 0)), ) # Build the cache manager + transceiver ONCE per case (the manager is diff --git a/examples/kimi_k3/README.md b/examples/kimi_k3/README.md new file mode 100644 index 000000000000..8a84e9ead7fd --- /dev/null +++ b/examples/kimi_k3/README.md @@ -0,0 +1,211 @@ +# Kimi K3 + +This example runs Kimi K3 with TensorRT-LLM. It includes an LLM API quick +start and configuration for GSM8K evaluation. + +## Hardware support + +Only NVIDIA Blackwell GPUs are currently supported and tested. Support for +other GPU architectures may be added in a future release. + +## Prerequisites + +- TensorRT-LLM built from this repository and installed. Inside + the TensorRT-LLM container, from the repository root: + + ```bash + python3 scripts/build_wheel.py --cuda_architectures 103-real --skip_building_wheel --yes + .venv-3.12/bin/python -m pip install --no-deps -e . + ``` + Using editable mode is recommended for development and testing; see + [build from source](../../docs/source/installation/build-from-source.md) + for details. + + `build_wheel.py` creates the virtual environment at the repository + root, named after the container's Python version: `.venv-3.12` for the + current containers (Python 3.12). If your container ships a different + Python, substitute the matching `.venv-.` path in the + commands on this page. Adjust `--cuda_architectures` to the target + GPUs (`103-real` for GB300). +- A complete Hugging Face-format Kimi K3 checkpoint and tokenizer, e.g. + [moonshotai/Kimi-K3](https://huggingface.co/moonshotai/Kimi-K3) downloaded + from the Hugging Face Hub (the example scripts take a local filesystem + path). +- A Slurm cluster with 16 NVIDIA Blackwell GPUs and a TensorRT-LLM container + image. The image passed as `--image` below must already provide + TensorRT-LLM's runtime dependencies, that is, a release-style TensorRT-LLM + container; a build or devel image without them does not work. The Slurm + scripts start a fresh container from that image. They mount the + repository, including `.venv-3.12`, and the checkpoint, so packages installed in + that virtual environment are available. Packages installed only in the preparation + container's base environment are not: + repository-root `.venv-3.12` is the build environment created by `build_wheel.py` + (it contains Conan and pip, not PyTorch or Transformers), and `pip install + --no-deps -e .` installs `tensorrt_llm` alone. With an image that lacks the dependencies, every rank + fails with a `ModuleNotFoundError` that does not name the image, such as + `No module named 'transformers'`. + + The scripts also export `PYTHONPATH="$REPO"` before launching the example. + Keep that export when adapting them: `python3