diff --git a/flashinfer/experimental/dense_projection_det_gemm/README.md b/flashinfer/experimental/dense_projection_det_gemm/README.md index c9295a8d7cd..c5cc91e82b2 100644 --- a/flashinfer/experimental/dense_projection_det_gemm/README.md +++ b/flashinfer/experimental/dense_projection_det_gemm/README.md @@ -158,7 +158,7 @@ stream-K split, raster group, wave working set) and the L2 size (hint gate); | L2 promotion, prefetch | `default_promo` = `none`, `default_pf` = 0 (measured, off); the symbol carries `_` / `_pf` when set | | CTA form | `form`: `""` (the ten-warp pipeline of every other program) or one of `W8_FORMS = ("w8r", "w8rs", "w8")`, the eight-warp CTA form of the same GEMM measured for six `sm_103a` input-gradient rows (`w8r` / `w8rs`) and five `sm_100a` input-gradient rows (`w8`, the knob-free program): four epilogue warps, a loader, a scheduler (`w8r`; `w8` = the same without the early accumulator release) or the loader's elected lane issuing the cluster-launch-control queries in its place (`w8rs`) and the MMA warp on CTA-pair 256 x 256 tiles over six 32 KiB stages, whole tiles only, the BF16 output through the TMA store, the raster quotients resolved by four fast divisors the host passes (`fd_tiles_l` / `fd_tiles_g` / `fd_group` / `fd_last`: launch arguments of these programs alone). Symbols `_tma1_w8r` / `_tma1_w8rs` / `_tma1_w8` (the form token is the last knob suffix; `instance_key` field 37). A rule's form yields where the plan is not whole-tile or a caller forces a pipeline, tile, output, epilogue, tail or raster knob (the row's ten-warp program runs the plan asked for; the rule's `pd` / `pro` / `lw` / `mg` / `sw` are its program there, and the `sm_103a` 4096 -> 48 rule carries none, so the architecture default one-slot program runs there; forcing `epi` `tma`, two slots, `htail`, `lw` and `pro` 15 plans its previous half-tail program); where the form stands, those knobs are cleared from the key. A caller-forced form is refused there (`instance_key`; the planner for stream-K units) | | template | `instance_symbol(instance_key(...))`, e.g. `dense_proj_det_gemm_kk_n256`, `dense_proj_det_gemm_kk_n128_hen`, `dense_proj_det_gemm_kn_n256_f32_tma1`, `dense_proj_det_gemm_nn_n256_f32_tma1`, `dense_proj_det_gemm_nn_n128_hen_t`, `dense_proj_det_gemm_kk_n256_m64`, `dense_proj_det_gemm_kk_n256_m256_ov_ht`, `dense_proj_det_gemm_nn_n160_m256_bz64`, `dense_proj_det_gemm_nn_n128_hen_t_skx`, `dense_proj_det_gemm_kn_n256_tma1_bg8`, `dense_proj_det_gemm_kn_n256_f32_v8_ef_s9`, `dense_proj_det_gemm_nn_n256_m256_sks_sb2`, `dense_proj_det_gemm_kk_n128_hen_c1`, `dense_proj_det_gemm_kn_n256_tma1_w8r`, `dense_proj_det_gemm_kn_n256_tma1_w8rs`, `dense_proj_det_gemm_kn_n256_tma1_w8` (round 13: `_pk` parked / `_ov` overlapped single-TMEM-buffer / `_ht` half-height-tail-wave tall epilogues; `htail` replaces the stream-K tail plan by 2 x tail whole-K half items when they fit the CTA pairs, so `ws_f32_elems = 0` and `sk_iters = sk_units * k_blocks`; `_bz64` = 32-column MN-major B panels (`b_swz` 64, smaller stages, deeper default pipeline), `_skx` = exact p-way K split of the tail tiles (`sk_exact`, pre-empts `sk_parts` / `htail`, p travels in `sk_iters`); `_sks` = synchronised stream-K tail (round 18: `sk_sync`, the tail tiles' main units over K steps [0, s) plus collector units over the leftovers, one fp32 slab per tail tile and a deterministic two-addend fixup; `sk_iters` = the main-unit count, `iters_per_unit` = s = ceil((q K + `sk_sync_m`) / (q + 1)); yields to `sk_exact`, pre-empts `sk_parts` / `htail`); `_bg8` = eight adjacent batch entries (heads) interleaved across the CTA pairs (`batch_group`, batched MLA rows only, 0 for a batch of one); `_ef` = fp32 v8 register stores with `L1::no_allocate.L2::evict_first` (`store_ef`), `_s9` = nine pipeline stages (oversized dynamic shared memory on sm_107a); round 19: `_sb` = the slab path of the synchronised fixup (`sk_slab`: 1 early arrival, 2 + the bulk slab read through the dead mainloop stages, 3 + the fp32 output as the slab - the same `_sks` plan, forced to 0 without `sk_sync`; a rule's 3 yields to 2 when a caller forces the row-major fp32 register epilogue, a rule's 2 to 0 when the eight warp slabs do not fit a caller-forced pipeline), `_c1` = the single-CTA form (`cta1`: one `CTA_ROWS` x `BLOCK_N` tile per CTA with cluster dims (1, 1, 1), so `m_tiles` is not rounded to pairs, `pair_tiles` counts CTA tiles, `sm_pairs` is the SM count (`sm_units`), the grid is `num_full` CTAs and the form carries no tail policy; 128-row family only); `_skf` directly after `_skx` = the tail-first order of the exact split (`sk_first`: pair work items `[0, sk_units)` are the K parts of the tail tiles, then the whole tiles; the same plan, counters and slabs as `_skx`), `_lw` between `_or` and `_pro` = the load warp leaving the pair prelude before the post-allocation CTA rendezvous (`lw`; pair programs whose MMA warp allocates TMEM, `pro` bit 4), `_mg` directly after `_lw` = the whole-tile tile resolve (`mg`: the load warp and the MMA warp walk whole tiles, one segment per item, with the raster quotients resolved through corrected fp32 reciprocals; whole tiles only - no batch raster, column-grouped raster, half-height tail or stream-K form, and the plan carries no stream-K unit); `_sw` directly after `_mg` = the paced output stores (`sw`: each epilogue warp of the bf16 TMA-store pd-2 drain delays its bulk tensor store issue, after the staging write and its fences, by a per-warp number of steps of `n` dependent u32 multiply-adds seeded by a clock sample, so each per-item eight-store burst becomes a seven-step ramp; the stores and bytes are unchanged; needs `slots` > 0, `pd` 2 and a bf16 output; 0 = off); `_w8r` / `_w8rs` / `_w8` directly after `_pro` = the eight-warp CTA form (`form`, the CTA form row above)) | -| rules | `ROW_RULES[(arch, A MN-major, B MN-major, fp32 out, transposed out, batched, N, K, M)]` (a ragged rule leaves the token count out: `M = None` for forward / input gradients, `K = None` for weight gradients; an exact key names one token count, as the `sm_107a` T-exact weight-gradient rules do at T 16231 and 16384, each in the orientation the row plans without it, and the GB T-exact forward / input-gradient rules of `sm_100a` and `sm_103a` do at `M` = T: 16 keys, each with the per-tile K partitions of the row's plan without it, so every output is bitwise equal to that plan's - the 4096 -> 48 input gradient on `sm_100a` (`_tma1_w8r`) and the 4096 -> 512 forward on both architectures and the 5120 -> 512 forward on `sm_103a` (`_ht_cg_pro15`) at T 16231 / 16384, the 6144 x 2048 input gradient (q_a / shared_gate_up) and the 2048 -> 6144 shared_down forward (`_ht_cg_pro15`) at T 16172 / 16231 on both architectures; and the GB T-exact rules of round 2: 10 keys with the same K partition property - the q_b 2048 x 16384 input gradient (`_ov_ht_cg`, the ragged rule's program with the column-grouped raster: 8-column-tile groups at T 16172 and 16-column-tile groups at T 16231 on `sm_100a`, 16-column-tile groups at both T on `sm_103a`), the 17408 -> 5120 weight gradient at T 16231 / 16384 on both architectures and the 5120 -> 17408 weight gradient at T 16231 / 16384 on `sm_103a` (`_ht_tma1`, 12-row raster groups, keyed at `K` = T in the direct orientation the row plans without them); and the GB T-exact rules of round 3: 4 keys with the same K partition property - the indexer_q 2048 -> 4096 weight gradient at T 16172 / 16231 on both architectures (`_tma1_sks_sb2`, the ragged rule's synchronised two-partial stream-K plan with its bf16 tiles through the one-slot TMA store, keyed at `K` = T in the direct orientation); and the GB T-exact rules of round 4: 4 keys with the same K partition property - the 5120 -> 17408 input gradient at T 16231 / 16384 on both architectures (`_ov_ht_tma1_cg`, the overlapped half-tail program with the column-grouped raster in 12-column-tile groups, in place of the ragged rule's row-half program), each with its bf16 tiles through the one-slot TMA store; and the GB T-exact rules of round 5: 2 keys with the same K partition property - the 12288 -> 4096 forward at T 16231 / 16384 on `sm_100a` (`_ov_ht_tma1_cg`, the ragged rule's overlapped half-tail program with the column-grouped raster in 8-column-tile groups), with its bf16 tiles through the one-slot TMA store; and the GB T-exact rules of round 6: 4 keys with the same K partition property - the 17408 -> 5120 forward at T 16231 / 16384 on both architectures (`_ov_ht_cg`, the ragged rule's own overlapped half-tail program with the column-grouped raster, in 12-column-tile groups in place of the ragged rule's 8); and the GB T-exact rules of round 7: 2 keys with the same K partition property - the 5120 -> 17408 weight gradient at T 16231 / 16384 on `sm_100a` (`_ht_tma1`, 12-row raster groups, the rule of the `sm_103a` keys of the row, keyed at `K` = T in the direct orientation the row plans without them); the ragged rules keep every other T) lays per-row knobs over the defaults above, merged over `ARCH_RULE_DEFAULTS` (`pro` 7 on `sm_100a`, `sm_103a` and `sm_107a`). Rounds 2-33 mirror the Cake table byte for byte; the shapes tranche (`sm_100a` and `sm_103a`: forward / input gradient / weight gradient rows at K 4096-17408, N 32-17408, measured at T 16172-16384 on GB200 / GB300 against cuBLAS) and its kernel-level round 2 (the half-tail column-grouped 4096 squares, 12-column raster groups on the N 5120 rows, the load-warp thin input gradients with the `l2_256b` promotion of the N 512 rows on `sm_103a`, the tail-first 16-way exact split of the 4096 x 12288 weight gradients) extend it per architecture. A rule's `sk_first` / `lw` yield where the planned program cannot carry them (an exact split the arithmetic does not admit, the single-CTA form, a `pro` without bit 4, a caller-forced tile family). Three `sm_103a` input-gradient rules (4096 -> 512, 5120 -> 512, the kv_a 6144 -> 576) carry `mg` with `sk: False` (the whole-tile tile resolve measured faster than the row's program above the pair spread at both T on GB300); a rule's `mg` yields where the plan carries stream-K units or exceeds the kernel's cluster-tile bound (a caller `sk` / `sk_max_units` / `sk_parts` / exact or synchronised split), where a caller forces a raster or tail form the `_mg` program excludes (`batch_group` / `cgrp` / `htail` / `sk_first` / `stt`), the single-CTA form or another tile family, and a caller-forced `mg` is refused on such a plan (`sk_units` is 0 on every `_mg` program). Two `sm_103a` input-gradient rules (5120 -> 512, 5120 -> 1024) carry `sw` 25 (the paced output stores measured faster than the row's program above the pair spread at both T on GB300); a rule's `sw` yields where the planned program is not the bf16 TMA-store pd-2 drain (a caller-forced register epilogue or `slots` 0, another `pd`, the fp32 output) or the single-CTA form, and a caller-forced `sw` is refused there. Six `sm_103a` input-gradient rules (4096 -> 512, 4096 -> 1024 and the kv_a 6144 -> 576 with `w8rs`; 4096 -> 48, 5120 -> 512 and 5120 -> 1024 with `w8r`) carry `form`: the eight-warp CTA form measured faster than the row's program above the pair spread at both T on GB300 (the `sm_103a` rules govern B300 as well, where no measurement exists), with raster groups of 8 row tiles on 4096 -> 1024, 12 on 5120 -> 512, 5120 -> 1024 and kv_a, and 16 on the other two; the knobs after it stay the row's program where the form yields (4096 -> 48: the architecture default program). Five `sm_100a` input-gradient rules (4096 -> 512, 4096 -> 1024, 5120 -> 512, 5120 -> 1024 and the kv_a 6144 -> 576) carry `form` `w8`: the knob-free eight-warp program measured faster than the row's program above the pair spread at both T on GB200 (the `sm_100a` rules govern B200 as well, where no measurement exists), with raster groups of 12 row tiles on 4096 -> 512 and 12 on kv_a and 8 on the other three rows; the knobs after it stay the row's program where the form yields. A rule's `htail` names the program (`_ht`); the half-tail fit (`0 < 2 * tail <= pairs`) selects only the plan (`plan.htail_plan`): where the half items do not fit the device's pairs (the K 5120 thin input gradients: 1280 tiles on 76 pairs, tail 64; the 4096 x 12288 projections: tail 56) the `_ht` program runs the row's plain whole-tile plan (never linearised stream-K units, whatever `sk` the row or the caller asked for), as measured. `sm_103a` registers every `sm_100a` program and its lanes measured the shapes tranche alone: a row without an `sm_103a` rule plans with the `sm_100a` rule (`RULE_PARENT`; the same knobs, so the same program, kept where the `sm_103a` registry has it), a rule `sm_103a` measured itself always wins, and the weight-gradient orientation follows the same lookup; `sm_100a` and `sm_107a` inherit nothing | +| rules | `ROW_RULES[(arch, A MN-major, B MN-major, fp32 out, transposed out, batched, N, K, M)]` (a ragged rule leaves the token count out: `M = None` for forward / input gradients, `K = None` for weight gradients; an exact key names one token count, as the `sm_107a` T-exact weight-gradient rules do at T 16231 and 16384, each in the orientation the row plans without it, and the GB T-exact forward / input-gradient rules of `sm_100a` and `sm_103a` do at `M` = T: 16 keys, each with the per-tile K partitions of the row's plan without it, so every output is bitwise equal to that plan's - the 4096 -> 48 input gradient on `sm_100a` (`_tma1_w8r`) and the 4096 -> 512 forward on both architectures and the 5120 -> 512 forward on `sm_103a` (`_ht_cg_pro15`) at T 16231 / 16384, the 6144 x 2048 input gradient (q_a / shared_gate_up) and the 2048 -> 6144 shared_down forward (`_ht_cg_pro15`) at T 16172 / 16231 on both architectures; and the GB T-exact rules of round 2: 10 keys with the same K partition property - the q_b 2048 x 16384 input gradient (`_ov_ht_cg`, the ragged rule's program with the column-grouped raster: 8-column-tile groups at T 16172 and 16-column-tile groups at T 16231 on `sm_100a`, 16-column-tile groups at both T on `sm_103a`), the 17408 -> 5120 weight gradient at T 16231 / 16384 on both architectures and the 5120 -> 17408 weight gradient at T 16231 / 16384 on `sm_103a` (`_ht_tma1`, 12-row raster groups, keyed at `K` = T in the direct orientation the row plans without them); and the GB T-exact rules of round 3: 4 keys with the same K partition property - the indexer_q 2048 -> 4096 weight gradient at T 16172 / 16231 on both architectures (`_tma1_sks_sb2`, the ragged rule's synchronised two-partial stream-K plan with its bf16 tiles through the one-slot TMA store, keyed at `K` = T in the direct orientation); and the GB T-exact rules of round 4: 4 keys with the same K partition property - the 5120 -> 17408 input gradient at T 16231 / 16384 on both architectures (`_ov_ht_tma1_cg`, the overlapped half-tail program with the column-grouped raster in 12-column-tile groups, in place of the ragged rule's row-half program), each with its bf16 tiles through the one-slot TMA store; and the GB T-exact rules of round 5: 2 keys with the same K partition property - the 12288 -> 4096 forward at T 16231 / 16384 on `sm_100a` (`_ov_ht_tma1_cg`, the ragged rule's overlapped half-tail program with the column-grouped raster in 8-column-tile groups), with its bf16 tiles through the one-slot TMA store; and the GB T-exact rules of round 6: 4 keys with the same K partition property - the 17408 -> 5120 forward at T 16231 / 16384 on both architectures (`_ov_ht_cg`, the ragged rule's own overlapped half-tail program with the column-grouped raster, in 12-column-tile groups in place of the ragged rule's 8); and the GB T-exact rules of round 7: 2 keys with the same K partition property - the 5120 -> 17408 weight gradient at T 16231 / 16384 on `sm_100a` (`_ht_tma1`, 12-row raster groups, the rule of the `sm_103a` keys of the row, keyed at `K` = T in the direct orientation the row plans without them); and the GB T-exact rules of round 8: 2 keys with the same K partition property - the 16384 -> 6144 forward at T 16231 on both architectures (`_ht_cg_or`, the half-tail program with the tall epilogue released by row half and the column-grouped raster in 8-column-tile groups, in place of the ragged rule's overlapped half-tail program `_ov_ht`), and the round-6 keys at T 16231 / 16384 on `sm_100a` re-valued to `_ht_cg_or` (the same 12-column-tile raster groups, the tall epilogue released by row half in place of the column-chunk one); the ragged rules keep every other T) lays per-row knobs over the defaults above, merged over `ARCH_RULE_DEFAULTS` (`pro` 7 on `sm_100a`, `sm_103a` and `sm_107a`). Rounds 2-33 mirror the Cake table byte for byte; the shapes tranche (`sm_100a` and `sm_103a`: forward / input gradient / weight gradient rows at K 4096-17408, N 32-17408, measured at T 16172-16384 on GB200 / GB300 against cuBLAS) and its kernel-level round 2 (the half-tail column-grouped 4096 squares, 12-column raster groups on the N 5120 rows, the load-warp thin input gradients with the `l2_256b` promotion of the N 512 rows on `sm_103a`, the tail-first 16-way exact split of the 4096 x 12288 weight gradients) extend it per architecture. A rule's `sk_first` / `lw` yield where the planned program cannot carry them (an exact split the arithmetic does not admit, the single-CTA form, a `pro` without bit 4, a caller-forced tile family). Three `sm_103a` input-gradient rules (4096 -> 512, 5120 -> 512, the kv_a 6144 -> 576) carry `mg` with `sk: False` (the whole-tile tile resolve measured faster than the row's program above the pair spread at both T on GB300); a rule's `mg` yields where the plan carries stream-K units or exceeds the kernel's cluster-tile bound (a caller `sk` / `sk_max_units` / `sk_parts` / exact or synchronised split), where a caller forces a raster or tail form the `_mg` program excludes (`batch_group` / `cgrp` / `htail` / `sk_first` / `stt`), the single-CTA form or another tile family, and a caller-forced `mg` is refused on such a plan (`sk_units` is 0 on every `_mg` program). Two `sm_103a` input-gradient rules (5120 -> 512, 5120 -> 1024) carry `sw` 25 (the paced output stores measured faster than the row's program above the pair spread at both T on GB300); a rule's `sw` yields where the planned program is not the bf16 TMA-store pd-2 drain (a caller-forced register epilogue or `slots` 0, another `pd`, the fp32 output) or the single-CTA form, and a caller-forced `sw` is refused there. Six `sm_103a` input-gradient rules (4096 -> 512, 4096 -> 1024 and the kv_a 6144 -> 576 with `w8rs`; 4096 -> 48, 5120 -> 512 and 5120 -> 1024 with `w8r`) carry `form`: the eight-warp CTA form measured faster than the row's program above the pair spread at both T on GB300 (the `sm_103a` rules govern B300 as well, where no measurement exists), with raster groups of 8 row tiles on 4096 -> 1024, 12 on 5120 -> 512, 5120 -> 1024 and kv_a, and 16 on the other two; the knobs after it stay the row's program where the form yields (4096 -> 48: the architecture default program). Five `sm_100a` input-gradient rules (4096 -> 512, 4096 -> 1024, 5120 -> 512, 5120 -> 1024 and the kv_a 6144 -> 576) carry `form` `w8`: the knob-free eight-warp program measured faster than the row's program above the pair spread at both T on GB200 (the `sm_100a` rules govern B200 as well, where no measurement exists), with raster groups of 12 row tiles on 4096 -> 512 and 12 on kv_a and 8 on the other three rows; the knobs after it stay the row's program where the form yields. A rule's `htail` names the program (`_ht`); the half-tail fit (`0 < 2 * tail <= pairs`) selects only the plan (`plan.htail_plan`): where the half items do not fit the device's pairs (the K 5120 thin input gradients: 1280 tiles on 76 pairs, tail 64; the 4096 x 12288 projections: tail 56) the `_ht` program runs the row's plain whole-tile plan (never linearised stream-K units, whatever `sk` the row or the caller asked for), as measured. `sm_103a` registers every `sm_100a` program and its lanes measured the shapes tranche alone: a row without an `sm_103a` rule plans with the `sm_100a` rule (`RULE_PARENT`; the same knobs, so the same program, kept where the `sm_103a` registry has it), a rule `sm_103a` measured itself always wins, and the weight-gradient orientation follows the same lookup; `sm_100a` and `sm_107a` inherit nothing | | exact-wave gate | a rule's `BLOCK_N` 224 (`EXACT_WAVE_BLOCK_N`) was measured for the whole-wave fit of its tiles on the 76-pair devices (`exact_waves`: 152 tall tiles of the 4096 x 4096 weight gradient = two waves, 76 tiles of 4096 -> 1024 = one); on a device whose pairs do not fit them in whole waves and where the default tile needs fewer waves (74 pairs of a 148-SM part) the rule's `BLOCK_N` yields to the default tile (`plan.wave_fallback`; the rule's other knobs stay); a caller-forced `block_n` is never touched, and the plan is the launcher's where the fit is exact or both tiles need the same waves (152 and 160 SMs). The two `sm_107a` T-exact rules with `BLOCK_N` 224 (the 5120 x 512 weight gradient at T 16231 and 16384) were measured on the 216-SM part instead; the same gate applies to them. | | registry fallback | a plan whose template this architecture's registry lacks is redone without the batched small-M swap (`plan.swap_fallback`), then without the row's measured rule (`plan.rule_fallback`: the architecture default `pro` form), then lands on the nearest generated program of the row's layout / output class (`plan.knob_fallback`): the hand-ordered knob variants first (plain TMA L2 policy, the single-CTA form, the 64-row pair family, the tall tile, the register epilogue, BLOCK_N 128, each in the three `pro` forms), then every generated program of the class, nearest symbol first, with the knobs its symbol decodes to; a caller-forced knob always wins. Only a class without any generated program on the architecture raises (`NotImplementedError` at binding) | | synchronised TMA-store gate | the synchronised stream-K programs that store their bf16 tiles through the TMA store (`_tma_sks`; the fp32-output `_f32_tma_sks` programs, which take the TMA store by default at every K, are served as before) were generated for measured rows whose rule names `epi="tma"` (`_serves`); a ragged synchronised row whose short K (`K <= K_TMA_BF16`) turns the epilogue default to the TMA store plans the same symbol unmeasured (two dense weight gradients on the 160-SM `sm_103a` part at T 961-1024), so the registry does not serve it there and the row keeps the registry fallback it ran before those programs were generated; a caller-forced `epi` is served | diff --git a/flashinfer/experimental/dense_projection_det_gemm/build.json b/flashinfer/experimental/dense_projection_det_gemm/build.json index c43c4758cda..72328949051 100644 --- a/flashinfer/experimental/dense_projection_det_gemm/build.json +++ b/flashinfer/experimental/dense_projection_det_gemm/build.json @@ -2061,6 +2061,70 @@ "csrc/cake_dense_projection_det_gemm/cake_dense_projection_det_gemm_0618ae0aec1af120afd0_binding.cu" ] }, + { + "arch": "sm_100a", + "closure_sha256": "7dd41f8aa73af03f18200991ed06d3621312fb773b7e1824b7d7e3fc24fc407e", + "entry": "run", + "gencode": "-gencode=arch=compute_100a,code=sm_100a", + "min_cuda": "12.8", + "name": "cake_dense_projection_det_gemm_53a63652be2155219d76_sm_100a", + "nvcc_flags": [ + "-DCAKE_DPG_KK_N256_M256_CG_OR_PRO7__HT_SELECT=1", + "-DCAKE_DPG_DENSE_PROJ_DET_GEMM_KK_N256_M256_HT_CG_OR_PRO7=1", + "-DCAKE_PROGRAM_TEMPLATE=kk_n256_m256_cg_or_pro7__ht", + "-DCAKE_PROGRAM_KNOB=KNOB_HT", + "-DCAKE_PROGRAM_KNOB_TYPE=bool", + "-DCAKE_PROGRAM_KNOB_VALUE=true", + "-DCAKE_PROGRAM_SHIM=cake_host_shim_73e395348a674ca0", + "-DCAKE_SMEM_TOTAL=197760", + "-DCAKE_ENCODE_A=EncodeTma_A_20a9f0b4904f", + "-DCAKE_ENCODE_B=EncodeTma_B_0bd184b3b05e", + "-DCAKE_ENCODE_OUT32=EncodeTma_OUT32_dea947d1de88", + "-DCAKE_ENCODE_OUT16=EncodeTma_OUT16_5c144af57a50" + ], + "nvcc_flags_if": [], + "path": "aot/cake_dense_projection_det_gemm_53a63652be2155219d76_sm_100a.so", + "sha256": { + "csrc/cake_dense_projection_det_gemm/cake_dense_projection_det_gemm_0c72e864a4afb5347da6_tpl_binding.cu": "c3e47e8b5d2b127807bf31dc2b53b45f5440e576a3aa48b6fa1cbbf5702a3d8a", + "csrc/cake_dense_projection_det_gemm/cake_dense_projection_det_gemm_tpl_kk_n256_m256_cg_or_pro7__ht_kernel.cu": "632f79a24e4e792ce36a8e61bf039dc55fc56aa58af3abec07886f9b55ef8613" + }, + "sources": [ + "csrc/cake_dense_projection_det_gemm/cake_dense_projection_det_gemm_tpl_kk_n256_m256_cg_or_pro7__ht_kernel.cu", + "csrc/cake_dense_projection_det_gemm/cake_dense_projection_det_gemm_0c72e864a4afb5347da6_tpl_binding.cu" + ] + }, + { + "arch": "sm_103a", + "closure_sha256": "c00bf606aafc1a5a15e75b280aba2ab8847affbba345832005b166fb53b24488", + "entry": "run", + "gencode": "-gencode=arch=compute_103a,code=sm_103a", + "min_cuda": "12.9", + "name": "cake_dense_projection_det_gemm_53a63652be2155219d76_sm_103a", + "nvcc_flags": [ + "-DCAKE_DPG_KK_N256_M256_CG_OR_PRO7__HT_SELECT=1", + "-DCAKE_DPG_DENSE_PROJ_DET_GEMM_KK_N256_M256_HT_CG_OR_PRO7=1", + "-DCAKE_PROGRAM_TEMPLATE=kk_n256_m256_cg_or_pro7__ht", + "-DCAKE_PROGRAM_KNOB=KNOB_HT", + "-DCAKE_PROGRAM_KNOB_TYPE=bool", + "-DCAKE_PROGRAM_KNOB_VALUE=true", + "-DCAKE_PROGRAM_SHIM=cake_host_shim_73e395348a674ca0", + "-DCAKE_SMEM_TOTAL=197760", + "-DCAKE_ENCODE_A=EncodeTma_A_20a9f0b4904f", + "-DCAKE_ENCODE_B=EncodeTma_B_0bd184b3b05e", + "-DCAKE_ENCODE_OUT32=EncodeTma_OUT32_dea947d1de88", + "-DCAKE_ENCODE_OUT16=EncodeTma_OUT16_5c144af57a50" + ], + "nvcc_flags_if": [], + "path": "aot/cake_dense_projection_det_gemm_53a63652be2155219d76_sm_103a.so", + "sha256": { + "csrc/cake_dense_projection_det_gemm/cake_dense_projection_det_gemm_0c72e864a4afb5347da6_tpl_binding.cu": "c3e47e8b5d2b127807bf31dc2b53b45f5440e576a3aa48b6fa1cbbf5702a3d8a", + "csrc/cake_dense_projection_det_gemm/cake_dense_projection_det_gemm_tpl_kk_n256_m256_cg_or_pro7__ht_kernel.cu": "632f79a24e4e792ce36a8e61bf039dc55fc56aa58af3abec07886f9b55ef8613" + }, + "sources": [ + "csrc/cake_dense_projection_det_gemm/cake_dense_projection_det_gemm_tpl_kk_n256_m256_cg_or_pro7__ht_kernel.cu", + "csrc/cake_dense_projection_det_gemm/cake_dense_projection_det_gemm_0c72e864a4afb5347da6_tpl_binding.cu" + ] + }, { "arch": "sm_107a", "closure_sha256": "0af8c94c5bded573403bcfb3566924dc7fecf360552aeaaa4f0f873be8125a27", diff --git a/flashinfer/experimental/dense_projection_det_gemm/cake_backend.py b/flashinfer/experimental/dense_projection_det_gemm/cake_backend.py index eb781ea4edc..4c7308095cc 100644 --- a/flashinfer/experimental/dense_projection_det_gemm/cake_backend.py +++ b/flashinfer/experimental/dense_projection_det_gemm/cake_backend.py @@ -1580,16 +1580,22 @@ def swap_small_m(L: int, M: int, N: int, transposed_out: bool) -> bool: ('sm_100a', False, False, False, False, False, 4096, 12288, 16384): {"cta_rows": 256, "ovl": True, "htail": True, "cgrp": True, "group_m": 8, "epi": 'tma', "slots": 1}, # ---- sm_100a: the GB T-exact rules of round 6 - the 17408 -> 5120 forward at T 16231 / 16384 (the ragged rule's # overlapped half-tail program with the column-grouped raster in 12-column-tile groups, in place of the ragged - # rule's 8), with the per-tile K partitions of the row's plan without them, measured on GB200 against - # that plan ---- - ('sm_100a', False, False, False, False, False, 5120, 17408, 16231): {"cta_rows": 256, "ovl": True, "htail": True, "cgrp": True, "group_m": 12}, - ('sm_100a', False, False, False, False, False, 5120, 17408, 16384): {"cta_rows": 256, "ovl": True, "htail": True, "cgrp": True, "group_m": 12}, + # rule's 8; round 8 at T 16231 / 16384: the tall epilogue released by row half in place of the column-chunk + # one, measured against the round-6 program), with the per-tile K partitions of the row's plan without them, + # measured on GB200 against that plan ---- + ('sm_100a', False, False, False, False, False, 5120, 17408, 16231): {"cta_rows": 256, "ovr": True, "htail": True, "cgrp": True, "group_m": 12}, + ('sm_100a', False, False, False, False, False, 5120, 17408, 16384): {"cta_rows": 256, "ovr": True, "htail": True, "cgrp": True, "group_m": 12}, # ---- sm_100a: the GB T-exact rules of round 7 - the 5120 -> 17408 weight gradient at T 16231 / 16384 in its direct # orientation (the half-tail program through the one-slot TMA store, 12-row raster groups: the sm_103a # keys' rule of the row), with the per-tile K partitions of the row's plan without them, measured on # GB200 against that plan ---- ('sm_100a', True, True, False, False, False, 5120, 16231, 17408): {"cta_rows": 256, "htail": True, "epi": 'tma', "slots": 1, "group_m": 12}, ('sm_100a', True, True, False, False, False, 5120, 16384, 17408): {"cta_rows": 256, "htail": True, "epi": 'tma', "slots": 1, "group_m": 12}, + # ---- sm_100a: the GB T-exact rules of round 8 - the 16384 -> 6144 forward at T 16231 (the half-tail program with + # the tall epilogue released by row half and the column-grouped raster in 8-column-tile groups, in place of the + # ragged rule's overlapped half-tail program), with the per-tile K partitions of the row's plan without it, + # measured on GB200 against that plan ---- + ('sm_100a', False, False, False, False, False, 6144, 16384, 16231): {"cta_rows": 256, "ovr": True, "htail": True, "cgrp": True, "group_m": 8}, # ---- sm_103a: the shapes tranche - forward / input gradient / weight gradient rows at K 4096 - 17408 and N 32 - 17408, # measured with this package's planner against cuBLAS on GB300 at T 16172 - 16384 (see the test table # MEASURED_SHAPE_ROWS); a row whose measured choice equals the planner's default carries no rule ---- @@ -1729,6 +1735,11 @@ def swap_small_m(L: int, M: int, N: int, transposed_out: bool) -> bool: # that plan ---- ('sm_103a', False, False, False, False, False, 5120, 17408, 16231): {"cta_rows": 256, "ovl": True, "htail": True, "cgrp": True, "group_m": 12}, ('sm_103a', False, False, False, False, False, 5120, 17408, 16384): {"cta_rows": 256, "ovl": True, "htail": True, "cgrp": True, "group_m": 12}, + # ---- sm_103a: the GB T-exact rules of round 8 - the 16384 -> 6144 forward at T 16231 (the sm_100a key's program: + # the half-tail program with the tall epilogue released by row half and the column-grouped raster in + # 8-column-tile groups, in place of the inherited sm_100a ragged rule's overlapped half-tail program), with the + # per-tile K partitions of the row's plan without it, measured on GB300 against that plan ---- + ('sm_103a', False, False, False, False, False, 6144, 16384, 16231): {"cta_rows": 256, "ovr": True, "htail": True, "cgrp": True, "group_m": 8}, } # fmt: on diff --git a/flashinfer/experimental/dense_projection_det_gemm/cake_jit.py b/flashinfer/experimental/dense_projection_det_gemm/cake_jit.py index 25ff34e8ecd..4ba1f9cc671 100644 --- a/flashinfer/experimental/dense_projection_det_gemm/cake_jit.py +++ b/flashinfer/experimental/dense_projection_det_gemm/cake_jit.py @@ -106,7 +106,7 @@ "cake_dense_projection_det_gemm_505fbe05ce8b62f0a589": {"template": "dense_proj_det_gemm_nn_n160_m256_bz64_pro15", "kernel": "kernel_cake_dense_projection_det_gemm_505fbe05ce8b62f0a589", "library": "cake_dense_projection_det_gemm_505fbe05ce8b62f0a589", "ffi_entry": "run", "arg_plan": "86cc2ed61d02", "launch": {"block": [320, 1, 1], "cluster": [2, 1, 1]}, "arches": ["sm_107a"]}, "cake_dense_projection_det_gemm_51c81f11e3b490b5aebb": {"template": "dense_proj_det_gemm_nn_n128_hen_t_pro15", "kernel": "kernel_cake_dense_projection_det_gemm_51c81f11e3b490b5aebb", "library": "cake_dense_projection_det_gemm_51c81f11e3b490b5aebb", "ffi_entry": "run", "arg_plan": "86cc2ed61d02", "launch": {"block": [320, 1, 1], "cluster": [2, 1, 1]}, "arches": ["sm_100a", "sm_103a", "sm_107a"], "build": {"module": "tart.bmm_bf16.sm100_sm103_sm107.dense_projection_det_gemm", "kwargs": {"a_hint": "evict_first", "a_mn": True, "b_mn": True, "block_n": 128, "out_t": True, "pro": 12}}}, "cake_dense_projection_det_gemm_5226b6c854d7db395d7e": {"template": "dense_proj_det_gemm_kn_n256_m256_tma1_bg4_pro7", "kernel": "kernel_cake_dense_projection_det_gemm_5226b6c854d7db395d7e", "library": "cake_dense_projection_det_gemm_5226b6c854d7db395d7e", "ffi_entry": "run", "arg_plan": "86cc2ed61d02", "launch": {"block": [320, 1, 1], "cluster": [2, 1, 1]}, "arches": ["sm_100a", "sm_103a"]}, - "cake_dense_projection_det_gemm_53a63652be2155219d76": {"template": "dense_proj_det_gemm_kk_n256_m256_ht_cg_or_pro7", "kernel": "cake::dense_projection_det_gemm::kk_n256_m256_cg_or_pro7__ht", "library": "cake_dense_projection_det_gemm_53a63652be2155219d76", "ffi_entry": "run", "arg_plan": "86cc2ed61d02", "launch": {"block": [320, 1, 1], "cluster": [2, 1, 1]}, "arches": ["sm_107a"]}, + "cake_dense_projection_det_gemm_53a63652be2155219d76": {"template": "dense_proj_det_gemm_kk_n256_m256_ht_cg_or_pro7", "kernel": "cake::dense_projection_det_gemm::kk_n256_m256_cg_or_pro7__ht", "library": "cake_dense_projection_det_gemm_53a63652be2155219d76", "ffi_entry": "run", "arg_plan": "86cc2ed61d02", "launch": {"block": [320, 1, 1], "cluster": [2, 1, 1]}, "arches": ["sm_100a", "sm_103a", "sm_107a"]}, "cake_dense_projection_det_gemm_53e98138d567774579ac": {"template": "swap_ab_w128_t64_s8", "kernel": "kernel_cake_dense_projection_det_gemm_53e98138d567774579ac", "library": "cake_dense_projection_det_gemm_53e98138d567774579ac", "ffi_entry": "run", "arg_plan": "48d32744aae2", "launch": {"block": [256, 1, 1], "cluster": [1, 1, 1]}, "arches": ["sm_100a", "sm_103a", "sm_107a"], "build": {"module": "tart.bmm_bf16.sm100_sm103_sm107.swap_ab_bmm", "kwargs": {"tile_w": 128, "tile_t": 64, "stages": 8, "w_evict_first": True, "strided": True, "weights_early": False}}}, "cake_dense_projection_det_gemm_5445e0f16ad84868fa01": {"template": "dense_proj_det_gemm_nn_n256_m256_f32_v8_ef_ht_pro7", "kernel": "kernel_cake_dense_projection_det_gemm_5445e0f16ad84868fa01", "library": "cake_dense_projection_det_gemm_5445e0f16ad84868fa01", "ffi_entry": "run", "arg_plan": "86cc2ed61d02", "launch": {"block": [320, 1, 1], "cluster": [2, 1, 1]}, "arches": ["sm_107a"]}, "cake_dense_projection_det_gemm_552070c21b97faa67a71": {"template": "dense_proj_det_gemm_kn_n256_tma1_pd2_sw25_pro15", "kernel": "kernel_cake_dense_projection_det_gemm_552070c21b97faa67a71", "library": "cake_dense_projection_det_gemm_552070c21b97faa67a71", "ffi_entry": "run", "arg_plan": "86cc2ed61d02", "launch": {"block": [320, 1, 1], "cluster": [2, 1, 1]}, "arches": ["sm_103a"], "build": {"module": "tart.bmm_bf16.sm100_sm103_sm107.dense_projection_det_gemm", "kwargs": {"b_mn": True, "pd": 2, "pro": 12, "slots": 1, "sw": 25}}}, @@ -201,8 +201,8 @@ # fmt: off KERNELS: dict[str, dict[str, str]] = { - "sm_100a": {"dense_proj_det_gemm_kk_n128_c1": "cake_dense_projection_det_gemm_cc2c62bd9132f8273125", "dense_proj_det_gemm_kk_n128_hen_c1": "cake_dense_projection_det_gemm_c195846d649ef9968762", "dense_proj_det_gemm_kk_n128_hen_pro15": "cake_dense_projection_det_gemm_20a2e5c04427e93572da", "dense_proj_det_gemm_kk_n192_hee_s8_pro15": "cake_dense_projection_det_gemm_56be8af43d5d0822da87", "dense_proj_det_gemm_kk_n192_pro7": "cake_dense_projection_det_gemm_f2e6d4e79cf21105fa1e", "dense_proj_det_gemm_kk_n256_cg_pro15": "cake_dense_projection_det_gemm_e33ac44bfff66daaee64", "dense_proj_det_gemm_kk_n256_cg_pro7": "cake_dense_projection_det_gemm_73a471b3db6b5aaf0c32", "dense_proj_det_gemm_kk_n256_ht_cg_pro15": "cake_dense_projection_det_gemm_ec7277a904a7211d67ce", "dense_proj_det_gemm_kk_n256_m256_ov_ht_cg_pro7": "cake_dense_projection_det_gemm_3d7b66f61c710f893b29", "dense_proj_det_gemm_kk_n256_m256_ov_ht_pro7": "cake_dense_projection_det_gemm_295aef2f2e0332de2372", "dense_proj_det_gemm_kk_n256_m256_ov_ht_tma1_cg_pro7": "cake_dense_projection_det_gemm_37a131cea24ecbc0ce7e", "dense_proj_det_gemm_kk_n256_pro15": "cake_dense_projection_det_gemm_e51cf65dcafa642e2c51", "dense_proj_det_gemm_kk_n256_pro7": "cake_dense_projection_det_gemm_a6ff52c845eac9296b34", "dense_proj_det_gemm_kk_n256_q_pro7": "cake_dense_projection_det_gemm_0db65f3ddd780ed42d4d", "dense_proj_det_gemm_kn_n128_f32_tma1_pro7": "cake_dense_projection_det_gemm_5a06c5e2f33772a33174", "dense_proj_det_gemm_kn_n128_f32_tma2_s4_c1": "cake_dense_projection_det_gemm_29d013dfb72e932eaee5", "dense_proj_det_gemm_kn_n256": "cake_dense_projection_det_gemm_e98b3ee7b774c63d7544", "dense_proj_det_gemm_kn_n256_cg_pro15": "cake_dense_projection_det_gemm_3a87cc189b4a7c1a8d7d", "dense_proj_det_gemm_kn_n256_f32_tma1_pro7": "cake_dense_projection_det_gemm_1350861ef426bd310e0f", "dense_proj_det_gemm_kn_n256_f32_tma2_pd2_pro15": "cake_dense_projection_det_gemm_624eb3d7232071ddaf91", "dense_proj_det_gemm_kn_n256_f32_v8_ef_pro7": "cake_dense_projection_det_gemm_895f9ba7159bea07aedd", "dense_proj_det_gemm_kn_n256_f32_v8_pro7": "cake_dense_projection_det_gemm_7d403fa7b167188ccc24", "dense_proj_det_gemm_kn_n256_ht_cg_pro15": "cake_dense_projection_det_gemm_eec6f87f8b7f25eb6aa2", "dense_proj_det_gemm_kn_n256_ht_tma2_lw_pro15": "cake_dense_projection_det_gemm_7ef28d420de555e7a28d", "dense_proj_det_gemm_kn_n256_m256_f32_v8_ef_ov_ht_pro7": "cake_dense_projection_det_gemm_765ece06955b70a95b29", "dense_proj_det_gemm_kn_n256_m256_ht_or_pro7": "cake_dense_projection_det_gemm_d2ea37e4d394bf40922a", "dense_proj_det_gemm_kn_n256_m256_ov_ht_cg_pro7": "cake_dense_projection_det_gemm_9ffc37c198f2d2f42c1f", "dense_proj_det_gemm_kn_n256_m256_ov_ht_pro7": "cake_dense_projection_det_gemm_9cc6363566760a38b4fe", "dense_proj_det_gemm_kn_n256_m256_ov_ht_tma1_cg_pro7": "cake_dense_projection_det_gemm_3040ba9398166d979a72", "dense_proj_det_gemm_kn_n256_m256_tma1_bg4_pro7": "cake_dense_projection_det_gemm_5226b6c854d7db395d7e", "dense_proj_det_gemm_kn_n256_pro15": "cake_dense_projection_det_gemm_1c1874c560375d29237d", "dense_proj_det_gemm_kn_n256_pro7": "cake_dense_projection_det_gemm_c15429b8d7d808a760ea", "dense_proj_det_gemm_kn_n256_tma1_pd2_lw_pro15": "cake_dense_projection_det_gemm_0d487a8f71dac54321ef", "dense_proj_det_gemm_kn_n256_tma1_pd2_pro15": "cake_dense_projection_det_gemm_0618ae0aec1af120afd0", "dense_proj_det_gemm_kn_n256_tma1_pro7": "cake_dense_projection_det_gemm_8053ebc84a22444c52af", "dense_proj_det_gemm_kn_n256_tma1_w8": "cake_dense_projection_det_gemm_0b91f29d88de00658fc5", "dense_proj_det_gemm_kn_n256_tma1_w8r": "cake_dense_projection_det_gemm_a24a3d3fd3ceac3e1cdf", "dense_proj_det_gemm_kn_n256_tma1_w8rs": "cake_dense_projection_det_gemm_bab00fb2b95c93758095", "dense_proj_det_gemm_kn_n256_tma2_pd2_lw_pro15": "cake_dense_projection_det_gemm_4de304a13db1f2a0e850", "dense_proj_det_gemm_kn_n256_tma2_pd2_pro15": "cake_dense_projection_det_gemm_3088b8056e0b7f9ca32c", "dense_proj_det_gemm_kn_n256_tma2_pd2_pro7": "cake_dense_projection_det_gemm_39200edabebcd9eb2edd", "dense_proj_det_gemm_kn_n256_tma2_pd3_c1": "cake_dense_projection_det_gemm_eb2bb538645fe303bcca", "dense_proj_det_gemm_nk_n256_t_pro7": "cake_dense_projection_det_gemm_d2a5e70bd49083888179", "dense_proj_det_gemm_nn_n128_hen_f32_t_pro15": "cake_dense_projection_det_gemm_59fb66ccddd67565e592", "dense_proj_det_gemm_nn_n128_hen_f32_t_sks_pro7": "cake_dense_projection_det_gemm_f993d8de6f29f9e8a1ac", "dense_proj_det_gemm_nn_n128_hen_t_pro15": "cake_dense_projection_det_gemm_51c81f11e3b490b5aebb", "dense_proj_det_gemm_nn_n128_hen_t_sks_pro7": "cake_dense_projection_det_gemm_f589a6aff5e88318c36c", "dense_proj_det_gemm_nn_n128_m64_f32_t_pro7": "cake_dense_projection_det_gemm_f258d45b7e98001e0c09", "dense_proj_det_gemm_nn_n128_m64_hen_f32_t_pro15": "cake_dense_projection_det_gemm_b2a9665ee12fd6afea2b", "dense_proj_det_gemm_nn_n128_m64_hen_f32_t_pro7": "cake_dense_projection_det_gemm_3507ceddbf4e88a13231", "dense_proj_det_gemm_nn_n128_m64_hen_t_pro15": "cake_dense_projection_det_gemm_e2555d533680e08880ae", "dense_proj_det_gemm_nn_n128_m64_hen_t_pro7": "cake_dense_projection_det_gemm_f861bf1312b9c7fe78c1", "dense_proj_det_gemm_nn_n128_m64_t_pro7": "cake_dense_projection_det_gemm_0c72e864a4afb5347da6", "dense_proj_det_gemm_nn_n160_m256_bz64_hne_cg_pro15": "cake_dense_projection_det_gemm_29de51d5e9cdd6d22491", "dense_proj_det_gemm_nn_n160_m256_bz64_hne_f32_v8_ef_cg_pro15": "cake_dense_projection_det_gemm_0dd5420b570cabc144ab", "dense_proj_det_gemm_nn_n160_m256_bz64_hne_f32_v8_ef_pro15": "cake_dense_projection_det_gemm_0bbd5d9f1d72a90ef3ea", "dense_proj_det_gemm_nn_n160_m256_bz64_hne_pro15": "cake_dense_projection_det_gemm_23305bdb787b7c6d7378", "dense_proj_det_gemm_nn_n192_bz64_hne_f32_v8_ef_sks_sb2_pro15": "cake_dense_projection_det_gemm_f6c9b40184336f49390d", "dense_proj_det_gemm_nn_n192_bz64_hne_sks_sb2_pro15": "cake_dense_projection_det_gemm_6579637169bec3c2f1d5", "dense_proj_det_gemm_nn_n192_hee_f32_t_tma1_pro7": "cake_dense_projection_det_gemm_36e10fde6497f82171d4", "dense_proj_det_gemm_nn_n192_hee_t_tma1_pro7": "cake_dense_projection_det_gemm_df649506b6ba6f6e47ef", "dense_proj_det_gemm_nn_n192_m256_t_tma1_pro7": "cake_dense_projection_det_gemm_b26ac3f3ed56a78cb052", "dense_proj_det_gemm_nn_n224_f32_v8_ef_pro15": "cake_dense_projection_det_gemm_13790f025541d7d162a7", "dense_proj_det_gemm_nn_n224_hne_cg_pro15": "cake_dense_projection_det_gemm_aa4f7c2fbd0b9a99e535", "dense_proj_det_gemm_nn_n224_m256_pro7": "cake_dense_projection_det_gemm_c63d1400e64208d7865c", "dense_proj_det_gemm_nn_n256_f32_tma1_cg_pro15": "cake_dense_projection_det_gemm_dfd65a75564d04d3826f", "dense_proj_det_gemm_nn_n256_f32_v8_ef_sks_sb2_pro7": "cake_dense_projection_det_gemm_9475731f174aaefb2e44", "dense_proj_det_gemm_nn_n256_f32_v8_pro7": "cake_dense_projection_det_gemm_b43a23d219b0196769dc", "dense_proj_det_gemm_nn_n256_m256_f32_ht_tma1_pro7": "cake_dense_projection_det_gemm_6235599f21b10d4ab9f2", "dense_proj_det_gemm_nn_n256_m256_f32_tma1_pro7": "cake_dense_projection_det_gemm_6e17162acb2711a87d39", "dense_proj_det_gemm_nn_n256_m256_f32_tma1_sks_sb3_pro7": "cake_dense_projection_det_gemm_f31e1dcb1df00973dc0c", "dense_proj_det_gemm_nn_n256_m256_ht_pro7": "cake_dense_projection_det_gemm_f052eb094121b03461ab", "dense_proj_det_gemm_nn_n256_m256_ht_tma1_pro7": "cake_dense_projection_det_gemm_806927d101fe72fc1317", "dense_proj_det_gemm_nn_n256_m256_ov_ht_pro7": "cake_dense_projection_det_gemm_5568657a526e2040f0bc", "dense_proj_det_gemm_nn_n256_m256_pro7": "cake_dense_projection_det_gemm_367fc93d3342f82a1c9f", "dense_proj_det_gemm_nn_n256_m256_sks_sb2_pro7": "cake_dense_projection_det_gemm_736ad30e83d88796883f", "dense_proj_det_gemm_nn_n256_m256_skx_skf_pro7": "cake_dense_projection_det_gemm_b773cc0e3894fd0ef2e0", "dense_proj_det_gemm_nn_n256_m256_t_tma1_pro7": "cake_dense_projection_det_gemm_ca9a846738b1443f7318", "dense_proj_det_gemm_nn_n256_m256_tma1_sks_sb2_pro7": "cake_dense_projection_det_gemm_a555af4859495921a208", "dense_proj_det_gemm_nn_n256_m64_hen_f32_t_pro15": "cake_dense_projection_det_gemm_fb766b1be68412e6fda7", "dense_proj_det_gemm_nn_n256_m64_hen_t_pro15": "cake_dense_projection_det_gemm_bab9e9dfd7c6ca5e17c0", "dense_proj_det_gemm_nn_n256_sks_sb2_pro7": "cake_dense_projection_det_gemm_c363601373e2dfb8e9af", "router_fp32_det_gemm_kk": "cake_dense_projection_det_gemm_ebbb505a3077e8da0a50", "router_fp32_det_gemm_kn": "cake_dense_projection_det_gemm_7a0ce55fd7221e354322", "router_fp32_det_gemm_nn_t": "cake_dense_projection_det_gemm_9269f6841d7aeb58471c", "split_fp32_det_bf16x3": "cake_dense_projection_det_gemm_5b4e87b523f799e09384", "swap_ab_w128_t64_s8": "cake_dense_projection_det_gemm_53e98138d567774579ac", "swap_ab_w128_t64_s8_we": "cake_dense_projection_det_gemm_edf7fa4c4f889f0c4981", "swap_ab_w64_t64_s12": "cake_dense_projection_det_gemm_9d269d61e56ad3d54fae", "swap_ab_w64_t64_s12_we": "cake_dense_projection_det_gemm_02803e1d2393113c0f6c"}, - "sm_103a": {"dense_proj_det_gemm_kk_n128_c1": "cake_dense_projection_det_gemm_cc2c62bd9132f8273125", "dense_proj_det_gemm_kk_n128_hen_c1": "cake_dense_projection_det_gemm_c195846d649ef9968762", "dense_proj_det_gemm_kk_n128_hen_pro15": "cake_dense_projection_det_gemm_20a2e5c04427e93572da", "dense_proj_det_gemm_kk_n192_hee_s8_pro15": "cake_dense_projection_det_gemm_56be8af43d5d0822da87", "dense_proj_det_gemm_kk_n192_pro7": "cake_dense_projection_det_gemm_f2e6d4e79cf21105fa1e", "dense_proj_det_gemm_kk_n256_cg_pro15": "cake_dense_projection_det_gemm_e33ac44bfff66daaee64", "dense_proj_det_gemm_kk_n256_cg_pro7": "cake_dense_projection_det_gemm_73a471b3db6b5aaf0c32", "dense_proj_det_gemm_kk_n256_ht_cg_pro15": "cake_dense_projection_det_gemm_ec7277a904a7211d67ce", "dense_proj_det_gemm_kk_n256_m256_ov_ht_cg_pro7": "cake_dense_projection_det_gemm_3d7b66f61c710f893b29", "dense_proj_det_gemm_kk_n256_m256_ov_ht_pro7": "cake_dense_projection_det_gemm_295aef2f2e0332de2372", "dense_proj_det_gemm_kk_n256_pro15": "cake_dense_projection_det_gemm_e51cf65dcafa642e2c51", "dense_proj_det_gemm_kk_n256_pro7": "cake_dense_projection_det_gemm_a6ff52c845eac9296b34", "dense_proj_det_gemm_kk_n256_q_pro7": "cake_dense_projection_det_gemm_0db65f3ddd780ed42d4d", "dense_proj_det_gemm_kn_n128_f32_tma1_pro7": "cake_dense_projection_det_gemm_5a06c5e2f33772a33174", "dense_proj_det_gemm_kn_n128_f32_tma2_s4_c1": "cake_dense_projection_det_gemm_29d013dfb72e932eaee5", "dense_proj_det_gemm_kn_n256": "cake_dense_projection_det_gemm_e98b3ee7b774c63d7544", "dense_proj_det_gemm_kn_n256_cg_pro15": "cake_dense_projection_det_gemm_3a87cc189b4a7c1a8d7d", "dense_proj_det_gemm_kn_n256_f32_tma1_pro7": "cake_dense_projection_det_gemm_1350861ef426bd310e0f", "dense_proj_det_gemm_kn_n256_f32_tma2_pd2_pro15": "cake_dense_projection_det_gemm_624eb3d7232071ddaf91", "dense_proj_det_gemm_kn_n256_f32_v8_ef_pro7": "cake_dense_projection_det_gemm_895f9ba7159bea07aedd", "dense_proj_det_gemm_kn_n256_f32_v8_pro7": "cake_dense_projection_det_gemm_7d403fa7b167188ccc24", "dense_proj_det_gemm_kn_n256_ht_cg_pro15": "cake_dense_projection_det_gemm_eec6f87f8b7f25eb6aa2", "dense_proj_det_gemm_kn_n256_ht_tma2_lw_pro15": "cake_dense_projection_det_gemm_7ef28d420de555e7a28d", "dense_proj_det_gemm_kn_n256_m256_f32_v8_ef_ov_ht_pro7": "cake_dense_projection_det_gemm_765ece06955b70a95b29", "dense_proj_det_gemm_kn_n256_m256_ht_or_pro7": "cake_dense_projection_det_gemm_d2ea37e4d394bf40922a", "dense_proj_det_gemm_kn_n256_m256_ov_ht_cg_pro7": "cake_dense_projection_det_gemm_9ffc37c198f2d2f42c1f", "dense_proj_det_gemm_kn_n256_m256_ov_ht_pro7": "cake_dense_projection_det_gemm_9cc6363566760a38b4fe", "dense_proj_det_gemm_kn_n256_m256_ov_ht_tma1_cg_pro7": "cake_dense_projection_det_gemm_3040ba9398166d979a72", "dense_proj_det_gemm_kn_n256_m256_tma1_bg4_pro7": "cake_dense_projection_det_gemm_5226b6c854d7db395d7e", "dense_proj_det_gemm_kn_n256_pro15": "cake_dense_projection_det_gemm_1c1874c560375d29237d", "dense_proj_det_gemm_kn_n256_pro7": "cake_dense_projection_det_gemm_c15429b8d7d808a760ea", "dense_proj_det_gemm_kn_n256_tma1_pd2_lw_mg_pro15": "cake_dense_projection_det_gemm_bb0e7457db664d6f0f6a", "dense_proj_det_gemm_kn_n256_tma1_pd2_lw_pro15": "cake_dense_projection_det_gemm_0d487a8f71dac54321ef", "dense_proj_det_gemm_kn_n256_tma1_pd2_mg_pro15": "cake_dense_projection_det_gemm_4eb96fb2eb1aaa14c7f7", "dense_proj_det_gemm_kn_n256_tma1_pd2_mg_sw25_pro15": "cake_dense_projection_det_gemm_75b1dcdaef42319f5ed4", "dense_proj_det_gemm_kn_n256_tma1_pd2_pro15": "cake_dense_projection_det_gemm_0618ae0aec1af120afd0", "dense_proj_det_gemm_kn_n256_tma1_pd2_sw25_pro15": "cake_dense_projection_det_gemm_552070c21b97faa67a71", "dense_proj_det_gemm_kn_n256_tma1_pro7": "cake_dense_projection_det_gemm_8053ebc84a22444c52af", "dense_proj_det_gemm_kn_n256_tma1_w8r": "cake_dense_projection_det_gemm_a24a3d3fd3ceac3e1cdf", "dense_proj_det_gemm_kn_n256_tma1_w8rs": "cake_dense_projection_det_gemm_bab00fb2b95c93758095", "dense_proj_det_gemm_kn_n256_tma2_pd2_lw_pro15": "cake_dense_projection_det_gemm_4de304a13db1f2a0e850", "dense_proj_det_gemm_kn_n256_tma2_pd2_pro15": "cake_dense_projection_det_gemm_3088b8056e0b7f9ca32c", "dense_proj_det_gemm_kn_n256_tma2_pd2_pro7": "cake_dense_projection_det_gemm_39200edabebcd9eb2edd", "dense_proj_det_gemm_kn_n256_tma2_pd3_c1": "cake_dense_projection_det_gemm_eb2bb538645fe303bcca", "dense_proj_det_gemm_nk_n256_t_pro7": "cake_dense_projection_det_gemm_d2a5e70bd49083888179", "dense_proj_det_gemm_nn_n128_hen_f32_t_pro15": "cake_dense_projection_det_gemm_59fb66ccddd67565e592", "dense_proj_det_gemm_nn_n128_hen_f32_t_sks_pro7": "cake_dense_projection_det_gemm_f993d8de6f29f9e8a1ac", "dense_proj_det_gemm_nn_n128_hen_t_pro15": "cake_dense_projection_det_gemm_51c81f11e3b490b5aebb", "dense_proj_det_gemm_nn_n128_hen_t_sks_pro7": "cake_dense_projection_det_gemm_f589a6aff5e88318c36c", "dense_proj_det_gemm_nn_n128_m64_f32_t_pro7": "cake_dense_projection_det_gemm_f258d45b7e98001e0c09", "dense_proj_det_gemm_nn_n128_m64_hen_f32_t_pro15": "cake_dense_projection_det_gemm_b2a9665ee12fd6afea2b", "dense_proj_det_gemm_nn_n128_m64_hen_f32_t_pro7": "cake_dense_projection_det_gemm_3507ceddbf4e88a13231", "dense_proj_det_gemm_nn_n128_m64_hen_t_pro15": "cake_dense_projection_det_gemm_e2555d533680e08880ae", "dense_proj_det_gemm_nn_n128_m64_hen_t_pro7": "cake_dense_projection_det_gemm_f861bf1312b9c7fe78c1", "dense_proj_det_gemm_nn_n128_m64_t_pro7": "cake_dense_projection_det_gemm_0c72e864a4afb5347da6", "dense_proj_det_gemm_nn_n160_m256_bz64_hne_cg_pro15": "cake_dense_projection_det_gemm_29de51d5e9cdd6d22491", "dense_proj_det_gemm_nn_n160_m256_bz64_hne_f32_v8_ef_cg_pro15": "cake_dense_projection_det_gemm_0dd5420b570cabc144ab", "dense_proj_det_gemm_nn_n160_m256_bz64_hne_f32_v8_ef_pro15": "cake_dense_projection_det_gemm_0bbd5d9f1d72a90ef3ea", "dense_proj_det_gemm_nn_n160_m256_bz64_hne_pro15": "cake_dense_projection_det_gemm_23305bdb787b7c6d7378", "dense_proj_det_gemm_nn_n192_bz64_hne_f32_v8_ef_sks_sb2_pro15": "cake_dense_projection_det_gemm_f6c9b40184336f49390d", "dense_proj_det_gemm_nn_n192_bz64_hne_sks_sb2_pro15": "cake_dense_projection_det_gemm_6579637169bec3c2f1d5", "dense_proj_det_gemm_nn_n192_hee_f32_t_tma1_pro7": "cake_dense_projection_det_gemm_36e10fde6497f82171d4", "dense_proj_det_gemm_nn_n192_hee_t_tma1_pro7": "cake_dense_projection_det_gemm_df649506b6ba6f6e47ef", "dense_proj_det_gemm_nn_n192_m256_t_tma1_pro7": "cake_dense_projection_det_gemm_b26ac3f3ed56a78cb052", "dense_proj_det_gemm_nn_n224_f32_v8_ef_pro15": "cake_dense_projection_det_gemm_13790f025541d7d162a7", "dense_proj_det_gemm_nn_n224_hne_cg_pro15": "cake_dense_projection_det_gemm_aa4f7c2fbd0b9a99e535", "dense_proj_det_gemm_nn_n224_m256_pro7": "cake_dense_projection_det_gemm_c63d1400e64208d7865c", "dense_proj_det_gemm_nn_n256_f32_tma1_cg_pro15": "cake_dense_projection_det_gemm_dfd65a75564d04d3826f", "dense_proj_det_gemm_nn_n256_f32_v8_ef_sks_sb2_pro7": "cake_dense_projection_det_gemm_9475731f174aaefb2e44", "dense_proj_det_gemm_nn_n256_f32_v8_pro7": "cake_dense_projection_det_gemm_b43a23d219b0196769dc", "dense_proj_det_gemm_nn_n256_m256_f32_ht_tma1_pro7": "cake_dense_projection_det_gemm_6235599f21b10d4ab9f2", "dense_proj_det_gemm_nn_n256_m256_f32_tma1_pro7": "cake_dense_projection_det_gemm_6e17162acb2711a87d39", "dense_proj_det_gemm_nn_n256_m256_f32_tma1_sks_sb3_pro7": "cake_dense_projection_det_gemm_f31e1dcb1df00973dc0c", "dense_proj_det_gemm_nn_n256_m256_ht_pro7": "cake_dense_projection_det_gemm_f052eb094121b03461ab", "dense_proj_det_gemm_nn_n256_m256_ht_tma1_pro7": "cake_dense_projection_det_gemm_806927d101fe72fc1317", "dense_proj_det_gemm_nn_n256_m256_ov_ht_pro7": "cake_dense_projection_det_gemm_5568657a526e2040f0bc", "dense_proj_det_gemm_nn_n256_m256_pro7": "cake_dense_projection_det_gemm_367fc93d3342f82a1c9f", "dense_proj_det_gemm_nn_n256_m256_sks_sb2_pro7": "cake_dense_projection_det_gemm_736ad30e83d88796883f", "dense_proj_det_gemm_nn_n256_m256_skx_skf_pro7": "cake_dense_projection_det_gemm_b773cc0e3894fd0ef2e0", "dense_proj_det_gemm_nn_n256_m256_t_tma1_pro7": "cake_dense_projection_det_gemm_ca9a846738b1443f7318", "dense_proj_det_gemm_nn_n256_m256_tma1_sks_sb2_pro7": "cake_dense_projection_det_gemm_a555af4859495921a208", "dense_proj_det_gemm_nn_n256_m64_hen_f32_t_pro15": "cake_dense_projection_det_gemm_fb766b1be68412e6fda7", "dense_proj_det_gemm_nn_n256_m64_hen_t_pro15": "cake_dense_projection_det_gemm_bab9e9dfd7c6ca5e17c0", "dense_proj_det_gemm_nn_n256_sks_sb2_pro7": "cake_dense_projection_det_gemm_c363601373e2dfb8e9af", "router_fp32_det_gemm_kk": "cake_dense_projection_det_gemm_ebbb505a3077e8da0a50", "router_fp32_det_gemm_kn": "cake_dense_projection_det_gemm_7a0ce55fd7221e354322", "router_fp32_det_gemm_nn_t": "cake_dense_projection_det_gemm_9269f6841d7aeb58471c", "split_fp32_det_bf16x3": "cake_dense_projection_det_gemm_5b4e87b523f799e09384", "swap_ab_w128_t64_s8": "cake_dense_projection_det_gemm_53e98138d567774579ac", "swap_ab_w128_t64_s8_we": "cake_dense_projection_det_gemm_edf7fa4c4f889f0c4981", "swap_ab_w64_t64_s12": "cake_dense_projection_det_gemm_9d269d61e56ad3d54fae", "swap_ab_w64_t64_s12_we": "cake_dense_projection_det_gemm_02803e1d2393113c0f6c"}, + "sm_100a": {"dense_proj_det_gemm_kk_n128_c1": "cake_dense_projection_det_gemm_cc2c62bd9132f8273125", "dense_proj_det_gemm_kk_n128_hen_c1": "cake_dense_projection_det_gemm_c195846d649ef9968762", "dense_proj_det_gemm_kk_n128_hen_pro15": "cake_dense_projection_det_gemm_20a2e5c04427e93572da", "dense_proj_det_gemm_kk_n192_hee_s8_pro15": "cake_dense_projection_det_gemm_56be8af43d5d0822da87", "dense_proj_det_gemm_kk_n192_pro7": "cake_dense_projection_det_gemm_f2e6d4e79cf21105fa1e", "dense_proj_det_gemm_kk_n256_cg_pro15": "cake_dense_projection_det_gemm_e33ac44bfff66daaee64", "dense_proj_det_gemm_kk_n256_cg_pro7": "cake_dense_projection_det_gemm_73a471b3db6b5aaf0c32", "dense_proj_det_gemm_kk_n256_ht_cg_pro15": "cake_dense_projection_det_gemm_ec7277a904a7211d67ce", "dense_proj_det_gemm_kk_n256_m256_ht_cg_or_pro7": "cake_dense_projection_det_gemm_53a63652be2155219d76", "dense_proj_det_gemm_kk_n256_m256_ov_ht_cg_pro7": "cake_dense_projection_det_gemm_3d7b66f61c710f893b29", "dense_proj_det_gemm_kk_n256_m256_ov_ht_pro7": "cake_dense_projection_det_gemm_295aef2f2e0332de2372", "dense_proj_det_gemm_kk_n256_m256_ov_ht_tma1_cg_pro7": "cake_dense_projection_det_gemm_37a131cea24ecbc0ce7e", "dense_proj_det_gemm_kk_n256_pro15": "cake_dense_projection_det_gemm_e51cf65dcafa642e2c51", "dense_proj_det_gemm_kk_n256_pro7": "cake_dense_projection_det_gemm_a6ff52c845eac9296b34", "dense_proj_det_gemm_kk_n256_q_pro7": "cake_dense_projection_det_gemm_0db65f3ddd780ed42d4d", "dense_proj_det_gemm_kn_n128_f32_tma1_pro7": "cake_dense_projection_det_gemm_5a06c5e2f33772a33174", "dense_proj_det_gemm_kn_n128_f32_tma2_s4_c1": "cake_dense_projection_det_gemm_29d013dfb72e932eaee5", "dense_proj_det_gemm_kn_n256": "cake_dense_projection_det_gemm_e98b3ee7b774c63d7544", "dense_proj_det_gemm_kn_n256_cg_pro15": "cake_dense_projection_det_gemm_3a87cc189b4a7c1a8d7d", "dense_proj_det_gemm_kn_n256_f32_tma1_pro7": "cake_dense_projection_det_gemm_1350861ef426bd310e0f", "dense_proj_det_gemm_kn_n256_f32_tma2_pd2_pro15": "cake_dense_projection_det_gemm_624eb3d7232071ddaf91", "dense_proj_det_gemm_kn_n256_f32_v8_ef_pro7": "cake_dense_projection_det_gemm_895f9ba7159bea07aedd", "dense_proj_det_gemm_kn_n256_f32_v8_pro7": "cake_dense_projection_det_gemm_7d403fa7b167188ccc24", "dense_proj_det_gemm_kn_n256_ht_cg_pro15": "cake_dense_projection_det_gemm_eec6f87f8b7f25eb6aa2", "dense_proj_det_gemm_kn_n256_ht_tma2_lw_pro15": "cake_dense_projection_det_gemm_7ef28d420de555e7a28d", "dense_proj_det_gemm_kn_n256_m256_f32_v8_ef_ov_ht_pro7": "cake_dense_projection_det_gemm_765ece06955b70a95b29", "dense_proj_det_gemm_kn_n256_m256_ht_or_pro7": "cake_dense_projection_det_gemm_d2ea37e4d394bf40922a", "dense_proj_det_gemm_kn_n256_m256_ov_ht_cg_pro7": "cake_dense_projection_det_gemm_9ffc37c198f2d2f42c1f", "dense_proj_det_gemm_kn_n256_m256_ov_ht_pro7": "cake_dense_projection_det_gemm_9cc6363566760a38b4fe", "dense_proj_det_gemm_kn_n256_m256_ov_ht_tma1_cg_pro7": "cake_dense_projection_det_gemm_3040ba9398166d979a72", "dense_proj_det_gemm_kn_n256_m256_tma1_bg4_pro7": "cake_dense_projection_det_gemm_5226b6c854d7db395d7e", "dense_proj_det_gemm_kn_n256_pro15": "cake_dense_projection_det_gemm_1c1874c560375d29237d", "dense_proj_det_gemm_kn_n256_pro7": "cake_dense_projection_det_gemm_c15429b8d7d808a760ea", "dense_proj_det_gemm_kn_n256_tma1_pd2_lw_pro15": "cake_dense_projection_det_gemm_0d487a8f71dac54321ef", "dense_proj_det_gemm_kn_n256_tma1_pd2_pro15": "cake_dense_projection_det_gemm_0618ae0aec1af120afd0", "dense_proj_det_gemm_kn_n256_tma1_pro7": "cake_dense_projection_det_gemm_8053ebc84a22444c52af", "dense_proj_det_gemm_kn_n256_tma1_w8": "cake_dense_projection_det_gemm_0b91f29d88de00658fc5", "dense_proj_det_gemm_kn_n256_tma1_w8r": "cake_dense_projection_det_gemm_a24a3d3fd3ceac3e1cdf", "dense_proj_det_gemm_kn_n256_tma1_w8rs": "cake_dense_projection_det_gemm_bab00fb2b95c93758095", "dense_proj_det_gemm_kn_n256_tma2_pd2_lw_pro15": "cake_dense_projection_det_gemm_4de304a13db1f2a0e850", "dense_proj_det_gemm_kn_n256_tma2_pd2_pro15": "cake_dense_projection_det_gemm_3088b8056e0b7f9ca32c", "dense_proj_det_gemm_kn_n256_tma2_pd2_pro7": "cake_dense_projection_det_gemm_39200edabebcd9eb2edd", "dense_proj_det_gemm_kn_n256_tma2_pd3_c1": "cake_dense_projection_det_gemm_eb2bb538645fe303bcca", "dense_proj_det_gemm_nk_n256_t_pro7": "cake_dense_projection_det_gemm_d2a5e70bd49083888179", "dense_proj_det_gemm_nn_n128_hen_f32_t_pro15": "cake_dense_projection_det_gemm_59fb66ccddd67565e592", "dense_proj_det_gemm_nn_n128_hen_f32_t_sks_pro7": "cake_dense_projection_det_gemm_f993d8de6f29f9e8a1ac", "dense_proj_det_gemm_nn_n128_hen_t_pro15": "cake_dense_projection_det_gemm_51c81f11e3b490b5aebb", "dense_proj_det_gemm_nn_n128_hen_t_sks_pro7": "cake_dense_projection_det_gemm_f589a6aff5e88318c36c", "dense_proj_det_gemm_nn_n128_m64_f32_t_pro7": "cake_dense_projection_det_gemm_f258d45b7e98001e0c09", "dense_proj_det_gemm_nn_n128_m64_hen_f32_t_pro15": "cake_dense_projection_det_gemm_b2a9665ee12fd6afea2b", "dense_proj_det_gemm_nn_n128_m64_hen_f32_t_pro7": "cake_dense_projection_det_gemm_3507ceddbf4e88a13231", "dense_proj_det_gemm_nn_n128_m64_hen_t_pro15": "cake_dense_projection_det_gemm_e2555d533680e08880ae", "dense_proj_det_gemm_nn_n128_m64_hen_t_pro7": "cake_dense_projection_det_gemm_f861bf1312b9c7fe78c1", "dense_proj_det_gemm_nn_n128_m64_t_pro7": "cake_dense_projection_det_gemm_0c72e864a4afb5347da6", "dense_proj_det_gemm_nn_n160_m256_bz64_hne_cg_pro15": "cake_dense_projection_det_gemm_29de51d5e9cdd6d22491", "dense_proj_det_gemm_nn_n160_m256_bz64_hne_f32_v8_ef_cg_pro15": "cake_dense_projection_det_gemm_0dd5420b570cabc144ab", "dense_proj_det_gemm_nn_n160_m256_bz64_hne_f32_v8_ef_pro15": "cake_dense_projection_det_gemm_0bbd5d9f1d72a90ef3ea", "dense_proj_det_gemm_nn_n160_m256_bz64_hne_pro15": "cake_dense_projection_det_gemm_23305bdb787b7c6d7378", "dense_proj_det_gemm_nn_n192_bz64_hne_f32_v8_ef_sks_sb2_pro15": "cake_dense_projection_det_gemm_f6c9b40184336f49390d", "dense_proj_det_gemm_nn_n192_bz64_hne_sks_sb2_pro15": "cake_dense_projection_det_gemm_6579637169bec3c2f1d5", "dense_proj_det_gemm_nn_n192_hee_f32_t_tma1_pro7": "cake_dense_projection_det_gemm_36e10fde6497f82171d4", "dense_proj_det_gemm_nn_n192_hee_t_tma1_pro7": "cake_dense_projection_det_gemm_df649506b6ba6f6e47ef", "dense_proj_det_gemm_nn_n192_m256_t_tma1_pro7": "cake_dense_projection_det_gemm_b26ac3f3ed56a78cb052", "dense_proj_det_gemm_nn_n224_f32_v8_ef_pro15": "cake_dense_projection_det_gemm_13790f025541d7d162a7", "dense_proj_det_gemm_nn_n224_hne_cg_pro15": "cake_dense_projection_det_gemm_aa4f7c2fbd0b9a99e535", "dense_proj_det_gemm_nn_n224_m256_pro7": "cake_dense_projection_det_gemm_c63d1400e64208d7865c", "dense_proj_det_gemm_nn_n256_f32_tma1_cg_pro15": "cake_dense_projection_det_gemm_dfd65a75564d04d3826f", "dense_proj_det_gemm_nn_n256_f32_v8_ef_sks_sb2_pro7": "cake_dense_projection_det_gemm_9475731f174aaefb2e44", "dense_proj_det_gemm_nn_n256_f32_v8_pro7": "cake_dense_projection_det_gemm_b43a23d219b0196769dc", "dense_proj_det_gemm_nn_n256_m256_f32_ht_tma1_pro7": "cake_dense_projection_det_gemm_6235599f21b10d4ab9f2", "dense_proj_det_gemm_nn_n256_m256_f32_tma1_pro7": "cake_dense_projection_det_gemm_6e17162acb2711a87d39", "dense_proj_det_gemm_nn_n256_m256_f32_tma1_sks_sb3_pro7": "cake_dense_projection_det_gemm_f31e1dcb1df00973dc0c", "dense_proj_det_gemm_nn_n256_m256_ht_pro7": "cake_dense_projection_det_gemm_f052eb094121b03461ab", "dense_proj_det_gemm_nn_n256_m256_ht_tma1_pro7": "cake_dense_projection_det_gemm_806927d101fe72fc1317", "dense_proj_det_gemm_nn_n256_m256_ov_ht_pro7": "cake_dense_projection_det_gemm_5568657a526e2040f0bc", "dense_proj_det_gemm_nn_n256_m256_pro7": "cake_dense_projection_det_gemm_367fc93d3342f82a1c9f", "dense_proj_det_gemm_nn_n256_m256_sks_sb2_pro7": "cake_dense_projection_det_gemm_736ad30e83d88796883f", "dense_proj_det_gemm_nn_n256_m256_skx_skf_pro7": "cake_dense_projection_det_gemm_b773cc0e3894fd0ef2e0", "dense_proj_det_gemm_nn_n256_m256_t_tma1_pro7": "cake_dense_projection_det_gemm_ca9a846738b1443f7318", "dense_proj_det_gemm_nn_n256_m256_tma1_sks_sb2_pro7": "cake_dense_projection_det_gemm_a555af4859495921a208", "dense_proj_det_gemm_nn_n256_m64_hen_f32_t_pro15": "cake_dense_projection_det_gemm_fb766b1be68412e6fda7", "dense_proj_det_gemm_nn_n256_m64_hen_t_pro15": "cake_dense_projection_det_gemm_bab9e9dfd7c6ca5e17c0", "dense_proj_det_gemm_nn_n256_sks_sb2_pro7": "cake_dense_projection_det_gemm_c363601373e2dfb8e9af", "router_fp32_det_gemm_kk": "cake_dense_projection_det_gemm_ebbb505a3077e8da0a50", "router_fp32_det_gemm_kn": "cake_dense_projection_det_gemm_7a0ce55fd7221e354322", "router_fp32_det_gemm_nn_t": "cake_dense_projection_det_gemm_9269f6841d7aeb58471c", "split_fp32_det_bf16x3": "cake_dense_projection_det_gemm_5b4e87b523f799e09384", "swap_ab_w128_t64_s8": "cake_dense_projection_det_gemm_53e98138d567774579ac", "swap_ab_w128_t64_s8_we": "cake_dense_projection_det_gemm_edf7fa4c4f889f0c4981", "swap_ab_w64_t64_s12": "cake_dense_projection_det_gemm_9d269d61e56ad3d54fae", "swap_ab_w64_t64_s12_we": "cake_dense_projection_det_gemm_02803e1d2393113c0f6c"}, + "sm_103a": {"dense_proj_det_gemm_kk_n128_c1": "cake_dense_projection_det_gemm_cc2c62bd9132f8273125", "dense_proj_det_gemm_kk_n128_hen_c1": "cake_dense_projection_det_gemm_c195846d649ef9968762", "dense_proj_det_gemm_kk_n128_hen_pro15": "cake_dense_projection_det_gemm_20a2e5c04427e93572da", "dense_proj_det_gemm_kk_n192_hee_s8_pro15": "cake_dense_projection_det_gemm_56be8af43d5d0822da87", "dense_proj_det_gemm_kk_n192_pro7": "cake_dense_projection_det_gemm_f2e6d4e79cf21105fa1e", "dense_proj_det_gemm_kk_n256_cg_pro15": "cake_dense_projection_det_gemm_e33ac44bfff66daaee64", "dense_proj_det_gemm_kk_n256_cg_pro7": "cake_dense_projection_det_gemm_73a471b3db6b5aaf0c32", "dense_proj_det_gemm_kk_n256_ht_cg_pro15": "cake_dense_projection_det_gemm_ec7277a904a7211d67ce", "dense_proj_det_gemm_kk_n256_m256_ht_cg_or_pro7": "cake_dense_projection_det_gemm_53a63652be2155219d76", "dense_proj_det_gemm_kk_n256_m256_ov_ht_cg_pro7": "cake_dense_projection_det_gemm_3d7b66f61c710f893b29", "dense_proj_det_gemm_kk_n256_m256_ov_ht_pro7": "cake_dense_projection_det_gemm_295aef2f2e0332de2372", "dense_proj_det_gemm_kk_n256_pro15": "cake_dense_projection_det_gemm_e51cf65dcafa642e2c51", "dense_proj_det_gemm_kk_n256_pro7": "cake_dense_projection_det_gemm_a6ff52c845eac9296b34", "dense_proj_det_gemm_kk_n256_q_pro7": "cake_dense_projection_det_gemm_0db65f3ddd780ed42d4d", "dense_proj_det_gemm_kn_n128_f32_tma1_pro7": "cake_dense_projection_det_gemm_5a06c5e2f33772a33174", "dense_proj_det_gemm_kn_n128_f32_tma2_s4_c1": "cake_dense_projection_det_gemm_29d013dfb72e932eaee5", "dense_proj_det_gemm_kn_n256": "cake_dense_projection_det_gemm_e98b3ee7b774c63d7544", "dense_proj_det_gemm_kn_n256_cg_pro15": "cake_dense_projection_det_gemm_3a87cc189b4a7c1a8d7d", "dense_proj_det_gemm_kn_n256_f32_tma1_pro7": "cake_dense_projection_det_gemm_1350861ef426bd310e0f", "dense_proj_det_gemm_kn_n256_f32_tma2_pd2_pro15": "cake_dense_projection_det_gemm_624eb3d7232071ddaf91", "dense_proj_det_gemm_kn_n256_f32_v8_ef_pro7": "cake_dense_projection_det_gemm_895f9ba7159bea07aedd", "dense_proj_det_gemm_kn_n256_f32_v8_pro7": "cake_dense_projection_det_gemm_7d403fa7b167188ccc24", "dense_proj_det_gemm_kn_n256_ht_cg_pro15": "cake_dense_projection_det_gemm_eec6f87f8b7f25eb6aa2", "dense_proj_det_gemm_kn_n256_ht_tma2_lw_pro15": "cake_dense_projection_det_gemm_7ef28d420de555e7a28d", "dense_proj_det_gemm_kn_n256_m256_f32_v8_ef_ov_ht_pro7": "cake_dense_projection_det_gemm_765ece06955b70a95b29", "dense_proj_det_gemm_kn_n256_m256_ht_or_pro7": "cake_dense_projection_det_gemm_d2ea37e4d394bf40922a", "dense_proj_det_gemm_kn_n256_m256_ov_ht_cg_pro7": "cake_dense_projection_det_gemm_9ffc37c198f2d2f42c1f", "dense_proj_det_gemm_kn_n256_m256_ov_ht_pro7": "cake_dense_projection_det_gemm_9cc6363566760a38b4fe", "dense_proj_det_gemm_kn_n256_m256_ov_ht_tma1_cg_pro7": "cake_dense_projection_det_gemm_3040ba9398166d979a72", "dense_proj_det_gemm_kn_n256_m256_tma1_bg4_pro7": "cake_dense_projection_det_gemm_5226b6c854d7db395d7e", "dense_proj_det_gemm_kn_n256_pro15": "cake_dense_projection_det_gemm_1c1874c560375d29237d", "dense_proj_det_gemm_kn_n256_pro7": "cake_dense_projection_det_gemm_c15429b8d7d808a760ea", "dense_proj_det_gemm_kn_n256_tma1_pd2_lw_mg_pro15": "cake_dense_projection_det_gemm_bb0e7457db664d6f0f6a", "dense_proj_det_gemm_kn_n256_tma1_pd2_lw_pro15": "cake_dense_projection_det_gemm_0d487a8f71dac54321ef", "dense_proj_det_gemm_kn_n256_tma1_pd2_mg_pro15": "cake_dense_projection_det_gemm_4eb96fb2eb1aaa14c7f7", "dense_proj_det_gemm_kn_n256_tma1_pd2_mg_sw25_pro15": "cake_dense_projection_det_gemm_75b1dcdaef42319f5ed4", "dense_proj_det_gemm_kn_n256_tma1_pd2_pro15": "cake_dense_projection_det_gemm_0618ae0aec1af120afd0", "dense_proj_det_gemm_kn_n256_tma1_pd2_sw25_pro15": "cake_dense_projection_det_gemm_552070c21b97faa67a71", "dense_proj_det_gemm_kn_n256_tma1_pro7": "cake_dense_projection_det_gemm_8053ebc84a22444c52af", "dense_proj_det_gemm_kn_n256_tma1_w8r": "cake_dense_projection_det_gemm_a24a3d3fd3ceac3e1cdf", "dense_proj_det_gemm_kn_n256_tma1_w8rs": "cake_dense_projection_det_gemm_bab00fb2b95c93758095", "dense_proj_det_gemm_kn_n256_tma2_pd2_lw_pro15": "cake_dense_projection_det_gemm_4de304a13db1f2a0e850", "dense_proj_det_gemm_kn_n256_tma2_pd2_pro15": "cake_dense_projection_det_gemm_3088b8056e0b7f9ca32c", "dense_proj_det_gemm_kn_n256_tma2_pd2_pro7": "cake_dense_projection_det_gemm_39200edabebcd9eb2edd", "dense_proj_det_gemm_kn_n256_tma2_pd3_c1": "cake_dense_projection_det_gemm_eb2bb538645fe303bcca", "dense_proj_det_gemm_nk_n256_t_pro7": "cake_dense_projection_det_gemm_d2a5e70bd49083888179", "dense_proj_det_gemm_nn_n128_hen_f32_t_pro15": "cake_dense_projection_det_gemm_59fb66ccddd67565e592", "dense_proj_det_gemm_nn_n128_hen_f32_t_sks_pro7": "cake_dense_projection_det_gemm_f993d8de6f29f9e8a1ac", "dense_proj_det_gemm_nn_n128_hen_t_pro15": "cake_dense_projection_det_gemm_51c81f11e3b490b5aebb", "dense_proj_det_gemm_nn_n128_hen_t_sks_pro7": "cake_dense_projection_det_gemm_f589a6aff5e88318c36c", "dense_proj_det_gemm_nn_n128_m64_f32_t_pro7": "cake_dense_projection_det_gemm_f258d45b7e98001e0c09", "dense_proj_det_gemm_nn_n128_m64_hen_f32_t_pro15": "cake_dense_projection_det_gemm_b2a9665ee12fd6afea2b", "dense_proj_det_gemm_nn_n128_m64_hen_f32_t_pro7": "cake_dense_projection_det_gemm_3507ceddbf4e88a13231", "dense_proj_det_gemm_nn_n128_m64_hen_t_pro15": "cake_dense_projection_det_gemm_e2555d533680e08880ae", "dense_proj_det_gemm_nn_n128_m64_hen_t_pro7": "cake_dense_projection_det_gemm_f861bf1312b9c7fe78c1", "dense_proj_det_gemm_nn_n128_m64_t_pro7": "cake_dense_projection_det_gemm_0c72e864a4afb5347da6", "dense_proj_det_gemm_nn_n160_m256_bz64_hne_cg_pro15": "cake_dense_projection_det_gemm_29de51d5e9cdd6d22491", "dense_proj_det_gemm_nn_n160_m256_bz64_hne_f32_v8_ef_cg_pro15": "cake_dense_projection_det_gemm_0dd5420b570cabc144ab", "dense_proj_det_gemm_nn_n160_m256_bz64_hne_f32_v8_ef_pro15": "cake_dense_projection_det_gemm_0bbd5d9f1d72a90ef3ea", "dense_proj_det_gemm_nn_n160_m256_bz64_hne_pro15": "cake_dense_projection_det_gemm_23305bdb787b7c6d7378", "dense_proj_det_gemm_nn_n192_bz64_hne_f32_v8_ef_sks_sb2_pro15": "cake_dense_projection_det_gemm_f6c9b40184336f49390d", "dense_proj_det_gemm_nn_n192_bz64_hne_sks_sb2_pro15": "cake_dense_projection_det_gemm_6579637169bec3c2f1d5", "dense_proj_det_gemm_nn_n192_hee_f32_t_tma1_pro7": "cake_dense_projection_det_gemm_36e10fde6497f82171d4", "dense_proj_det_gemm_nn_n192_hee_t_tma1_pro7": "cake_dense_projection_det_gemm_df649506b6ba6f6e47ef", "dense_proj_det_gemm_nn_n192_m256_t_tma1_pro7": "cake_dense_projection_det_gemm_b26ac3f3ed56a78cb052", "dense_proj_det_gemm_nn_n224_f32_v8_ef_pro15": "cake_dense_projection_det_gemm_13790f025541d7d162a7", "dense_proj_det_gemm_nn_n224_hne_cg_pro15": "cake_dense_projection_det_gemm_aa4f7c2fbd0b9a99e535", "dense_proj_det_gemm_nn_n224_m256_pro7": "cake_dense_projection_det_gemm_c63d1400e64208d7865c", "dense_proj_det_gemm_nn_n256_f32_tma1_cg_pro15": "cake_dense_projection_det_gemm_dfd65a75564d04d3826f", "dense_proj_det_gemm_nn_n256_f32_v8_ef_sks_sb2_pro7": "cake_dense_projection_det_gemm_9475731f174aaefb2e44", "dense_proj_det_gemm_nn_n256_f32_v8_pro7": "cake_dense_projection_det_gemm_b43a23d219b0196769dc", "dense_proj_det_gemm_nn_n256_m256_f32_ht_tma1_pro7": "cake_dense_projection_det_gemm_6235599f21b10d4ab9f2", "dense_proj_det_gemm_nn_n256_m256_f32_tma1_pro7": "cake_dense_projection_det_gemm_6e17162acb2711a87d39", "dense_proj_det_gemm_nn_n256_m256_f32_tma1_sks_sb3_pro7": "cake_dense_projection_det_gemm_f31e1dcb1df00973dc0c", "dense_proj_det_gemm_nn_n256_m256_ht_pro7": "cake_dense_projection_det_gemm_f052eb094121b03461ab", "dense_proj_det_gemm_nn_n256_m256_ht_tma1_pro7": "cake_dense_projection_det_gemm_806927d101fe72fc1317", "dense_proj_det_gemm_nn_n256_m256_ov_ht_pro7": "cake_dense_projection_det_gemm_5568657a526e2040f0bc", "dense_proj_det_gemm_nn_n256_m256_pro7": "cake_dense_projection_det_gemm_367fc93d3342f82a1c9f", "dense_proj_det_gemm_nn_n256_m256_sks_sb2_pro7": "cake_dense_projection_det_gemm_736ad30e83d88796883f", "dense_proj_det_gemm_nn_n256_m256_skx_skf_pro7": "cake_dense_projection_det_gemm_b773cc0e3894fd0ef2e0", "dense_proj_det_gemm_nn_n256_m256_t_tma1_pro7": "cake_dense_projection_det_gemm_ca9a846738b1443f7318", "dense_proj_det_gemm_nn_n256_m256_tma1_sks_sb2_pro7": "cake_dense_projection_det_gemm_a555af4859495921a208", "dense_proj_det_gemm_nn_n256_m64_hen_f32_t_pro15": "cake_dense_projection_det_gemm_fb766b1be68412e6fda7", "dense_proj_det_gemm_nn_n256_m64_hen_t_pro15": "cake_dense_projection_det_gemm_bab9e9dfd7c6ca5e17c0", "dense_proj_det_gemm_nn_n256_sks_sb2_pro7": "cake_dense_projection_det_gemm_c363601373e2dfb8e9af", "router_fp32_det_gemm_kk": "cake_dense_projection_det_gemm_ebbb505a3077e8da0a50", "router_fp32_det_gemm_kn": "cake_dense_projection_det_gemm_7a0ce55fd7221e354322", "router_fp32_det_gemm_nn_t": "cake_dense_projection_det_gemm_9269f6841d7aeb58471c", "split_fp32_det_bf16x3": "cake_dense_projection_det_gemm_5b4e87b523f799e09384", "swap_ab_w128_t64_s8": "cake_dense_projection_det_gemm_53e98138d567774579ac", "swap_ab_w128_t64_s8_we": "cake_dense_projection_det_gemm_edf7fa4c4f889f0c4981", "swap_ab_w64_t64_s12": "cake_dense_projection_det_gemm_9d269d61e56ad3d54fae", "swap_ab_w64_t64_s12_we": "cake_dense_projection_det_gemm_02803e1d2393113c0f6c"}, "sm_107a": {"dense_proj_det_gemm_kk_n128_hen_pro15": "cake_dense_projection_det_gemm_0a71a49da71c7d34ab37", "dense_proj_det_gemm_kk_n128_hen_pro7": "cake_dense_projection_det_gemm_7ad01af3a13775199254", "dense_proj_det_gemm_kk_n128_pro7": "cake_dense_projection_det_gemm_451a0f92e10205e345eb", "dense_proj_det_gemm_kk_n192_m256_hen_s5_pro15": "cake_dense_projection_det_gemm_ed1f3ab86ddc55345e69", "dense_proj_det_gemm_kk_n192_pro7": "cake_dense_projection_det_gemm_f2e6d4e79cf21105fa1e", "dense_proj_det_gemm_kk_n256_cg_pro15": "cake_dense_projection_det_gemm_438841f42a0743b63fb5", "dense_proj_det_gemm_kk_n256_ht_cg_pro15": "cake_dense_projection_det_gemm_ec7277a904a7211d67ce", "dense_proj_det_gemm_kk_n256_ht_pro15": "cake_dense_projection_det_gemm_087ba4f157914949c26a", "dense_proj_det_gemm_kk_n256_m256_cg_or_pro7": "cake_dense_projection_det_gemm_4af009e00a1d5c0ad794", "dense_proj_det_gemm_kk_n256_m256_ht_cg_or_pro7": "cake_dense_projection_det_gemm_53a63652be2155219d76", "dense_proj_det_gemm_kk_n256_m256_ov_ht_cg_pro7": "cake_dense_projection_det_gemm_3d7b66f61c710f893b29", "dense_proj_det_gemm_kk_n256_pro15": "cake_dense_projection_det_gemm_e51cf65dcafa642e2c51", "dense_proj_det_gemm_kk_n256_q_pro7": "cake_dense_projection_det_gemm_0db65f3ddd780ed42d4d", "dense_proj_det_gemm_kn_n128_f32_tma1_c1": "cake_dense_projection_det_gemm_011c85b820ec0cf6863d", "dense_proj_det_gemm_kn_n128_f32_tma1_pd2_c1": "cake_dense_projection_det_gemm_55c1b40f71e3fe4a892c", "dense_proj_det_gemm_kn_n256_cg_pro15": "cake_dense_projection_det_gemm_3a87cc189b4a7c1a8d7d", "dense_proj_det_gemm_kn_n256_cg_pro7": "cake_dense_projection_det_gemm_14192244441c1003db74", "dense_proj_det_gemm_kn_n256_f32_tma1_pro15": "cake_dense_projection_det_gemm_b58b50619ef5ed46e109", "dense_proj_det_gemm_kn_n256_f32_tma2_pd2_pro15": "cake_dense_projection_det_gemm_624eb3d7232071ddaf91", "dense_proj_det_gemm_kn_n256_f32_v8_ef_s9_pro15": "cake_dense_projection_det_gemm_31be7faa868eabcedada", "dense_proj_det_gemm_kn_n256_f32_v8_ht_pro7": "cake_dense_projection_det_gemm_b9c3f4eb125b638d7c87", "dense_proj_det_gemm_kn_n256_f32_v8_pro7": "cake_dense_projection_det_gemm_7d403fa7b167188ccc24", "dense_proj_det_gemm_kn_n256_hne_pro7": "cake_dense_projection_det_gemm_66df6446dd2cca3106d3", "dense_proj_det_gemm_kn_n256_ht_cg_pro15": "cake_dense_projection_det_gemm_eec6f87f8b7f25eb6aa2", "dense_proj_det_gemm_kn_n256_ht_pro15": "cake_dense_projection_det_gemm_b58324007fd26df8cf2c", "dense_proj_det_gemm_kn_n256_ht_tma2_lw_pro15": "cake_dense_projection_det_gemm_7ef28d420de555e7a28d", "dense_proj_det_gemm_kn_n256_m256_cg_or_pro7": "cake_dense_projection_det_gemm_13d9310f37868a18eace", "dense_proj_det_gemm_kn_n256_m256_f32_tma1_pro7": "cake_dense_projection_det_gemm_de7492eeb6e24b31687a", "dense_proj_det_gemm_kn_n256_m256_ht_or_pro7": "cake_dense_projection_det_gemm_d2ea37e4d394bf40922a", "dense_proj_det_gemm_kn_n256_m256_ov_ht_cg_pro7": "cake_dense_projection_det_gemm_9ffc37c198f2d2f42c1f", "dense_proj_det_gemm_kn_n256_m256_ov_ht_pro15": "cake_dense_projection_det_gemm_ed6ef54b238c5da2f5a7", "dense_proj_det_gemm_kn_n256_pro15": "cake_dense_projection_det_gemm_1c1874c560375d29237d", "dense_proj_det_gemm_kn_n256_pro7": "cake_dense_projection_det_gemm_c15429b8d7d808a760ea", "dense_proj_det_gemm_kn_n256_q_pd1_pro7": "cake_dense_projection_det_gemm_fb4bcdf13915dfb96c59", "dense_proj_det_gemm_kn_n256_tma1_bg8_pro7": "cake_dense_projection_det_gemm_cb45dbd62c7edd4410f1", "dense_proj_det_gemm_kn_n256_tma1_bg8_sol_pro7": "cake_dense_projection_det_gemm_332b601bf82c22b05ef2", "dense_proj_det_gemm_kn_n256_tma1_pd2_lw_pro15": "cake_dense_projection_det_gemm_0d487a8f71dac54321ef", "dense_proj_det_gemm_kn_n256_tma2_pd2_lw_pro15": "cake_dense_projection_det_gemm_4de304a13db1f2a0e850", "dense_proj_det_gemm_kn_n256_tma2_pd2_pro15": "cake_dense_projection_det_gemm_3088b8056e0b7f9ca32c", "dense_proj_det_gemm_kn_n256_tma2_pd3_c1": "cake_dense_projection_det_gemm_49acb91191bbcced9051", "dense_proj_det_gemm_nk_n256_t_pro7": "cake_dense_projection_det_gemm_d2a5e70bd49083888179", "dense_proj_det_gemm_nn_n128_f32_t_pro7": "cake_dense_projection_det_gemm_563dd919de0e74c71bde", "dense_proj_det_gemm_nn_n128_hen_f32_t_pro15": "cake_dense_projection_det_gemm_59fb66ccddd67565e592", "dense_proj_det_gemm_nn_n128_hen_f32_t_sks_pro7": "cake_dense_projection_det_gemm_f993d8de6f29f9e8a1ac", "dense_proj_det_gemm_nn_n128_hen_f32_t_skx_pro7": "cake_dense_projection_det_gemm_b09222ffcfa7a7e4ead7", "dense_proj_det_gemm_nn_n128_hen_t_pro15": "cake_dense_projection_det_gemm_51c81f11e3b490b5aebb", "dense_proj_det_gemm_nn_n128_hen_t_sks_pro7": "cake_dense_projection_det_gemm_f589a6aff5e88318c36c", "dense_proj_det_gemm_nn_n128_hen_t_skx_pro7": "cake_dense_projection_det_gemm_6f17e515b2e1b7e2fde9", "dense_proj_det_gemm_nn_n128_m64_hen_f32_t_pro15": "cake_dense_projection_det_gemm_b2a9665ee12fd6afea2b", "dense_proj_det_gemm_nn_n128_m64_hen_f32_t_skx_pro15": "cake_dense_projection_det_gemm_56c1924c362457a1c3fe", "dense_proj_det_gemm_nn_n128_m64_hen_t_pro15": "cake_dense_projection_det_gemm_e2555d533680e08880ae", "dense_proj_det_gemm_nn_n128_m64_hen_t_skx_pro15": "cake_dense_projection_det_gemm_4f31f3828842f20176d6", "dense_proj_det_gemm_nn_n128_t_pro7": "cake_dense_projection_det_gemm_36d0401698df21c9425f", "dense_proj_det_gemm_nn_n160_m256_bz64_f32_v8_ef_pro15": "cake_dense_projection_det_gemm_5e77ef909e88a99c4357", "dense_proj_det_gemm_nn_n160_m256_bz64_hne_cg_pro15": "cake_dense_projection_det_gemm_29de51d5e9cdd6d22491", "dense_proj_det_gemm_nn_n160_m256_bz64_hne_f32_v8_ef_cg_pro15": "cake_dense_projection_det_gemm_0dd5420b570cabc144ab", "dense_proj_det_gemm_nn_n160_m256_bz64_hne_f32_v8_ef_pro15": "cake_dense_projection_det_gemm_0bbd5d9f1d72a90ef3ea", "dense_proj_det_gemm_nn_n160_m256_bz64_hne_pro15": "cake_dense_projection_det_gemm_23305bdb787b7c6d7378", "dense_proj_det_gemm_nn_n160_m256_bz64_pro15": "cake_dense_projection_det_gemm_505fbe05ce8b62f0a589", "dense_proj_det_gemm_nn_n192_bz64_hee_f32_t_pro7": "cake_dense_projection_det_gemm_d6ebaab2415ea7b00de9", "dense_proj_det_gemm_nn_n192_bz64_hee_f32_t_sks_sb2_pro7": "cake_dense_projection_det_gemm_fdb106e6e73897521cbb", "dense_proj_det_gemm_nn_n192_bz64_hee_t_sks_stt_pro15": "cake_dense_projection_det_gemm_ddff2b911137449c1dfc", "dense_proj_det_gemm_nn_n192_bz64_hne_f32_v8_ef_sks_sb2_pro15": "cake_dense_projection_det_gemm_f6c9b40184336f49390d", "dense_proj_det_gemm_nn_n192_bz64_hne_sks_sb2_pro15": "cake_dense_projection_det_gemm_6579637169bec3c2f1d5", "dense_proj_det_gemm_nn_n192_m256_bz64_hee_t_tma1_s6_pro7": "cake_dense_projection_det_gemm_9aa9aa60b0c544bfab06", "dense_proj_det_gemm_nn_n224_f32_v8_ef_pro15": "cake_dense_projection_det_gemm_13790f025541d7d162a7", "dense_proj_det_gemm_nn_n224_hne_cg_pro15": "cake_dense_projection_det_gemm_aa4f7c2fbd0b9a99e535", "dense_proj_det_gemm_nn_n224_m256_pro7": "cake_dense_projection_det_gemm_c63d1400e64208d7865c", "dense_proj_det_gemm_nn_n256_f32_tma1_cg_pro15": "cake_dense_projection_det_gemm_dfd65a75564d04d3826f", "dense_proj_det_gemm_nn_n256_f32_v8_ef_sks_sb2_pro7": "cake_dense_projection_det_gemm_9475731f174aaefb2e44", "dense_proj_det_gemm_nn_n256_f32_v8_pro15": "cake_dense_projection_det_gemm_8e1f6805342f620cca8c", "dense_proj_det_gemm_nn_n256_m256_f32_ht_tma1_pro7": "cake_dense_projection_det_gemm_6235599f21b10d4ab9f2", "dense_proj_det_gemm_nn_n256_m256_f32_tma1_pro7": "cake_dense_projection_det_gemm_6e17162acb2711a87d39", "dense_proj_det_gemm_nn_n256_m256_f32_v8_ef_ht_pro7": "cake_dense_projection_det_gemm_5445e0f16ad84868fa01", "dense_proj_det_gemm_nn_n256_m256_hee_t_tma1_s6_pro7": "cake_dense_projection_det_gemm_446497e750658dfb057b", "dense_proj_det_gemm_nn_n256_m256_ht_pro7": "cake_dense_projection_det_gemm_f052eb094121b03461ab", "dense_proj_det_gemm_nn_n256_m256_pro7": "cake_dense_projection_det_gemm_367fc93d3342f82a1c9f", "dense_proj_det_gemm_nn_n256_m256_skx_skf_pro7": "cake_dense_projection_det_gemm_b773cc0e3894fd0ef2e0", "dense_proj_det_gemm_nn_n256_m256_tma1_pro7": "cake_dense_projection_det_gemm_e2692c71bc236502ba68", "dense_proj_det_gemm_nn_n256_m64_hen_f32_t_pro15": "cake_dense_projection_det_gemm_fb766b1be68412e6fda7", "dense_proj_det_gemm_nn_n256_m64_hen_t_pro15": "cake_dense_projection_det_gemm_bab9e9dfd7c6ca5e17c0", "dense_proj_det_gemm_nn_n256_pro15": "cake_dense_projection_det_gemm_82f698f67a1cca6e4291", "dense_proj_det_gemm_nn_n256_sks_sb2_pro7": "cake_dense_projection_det_gemm_c363601373e2dfb8e9af", "router_fp32_det_gemm_kk": "cake_dense_projection_det_gemm_ebbb505a3077e8da0a50", "router_fp32_det_gemm_kn": "cake_dense_projection_det_gemm_7a0ce55fd7221e354322", "router_fp32_det_gemm_nn_t": "cake_dense_projection_det_gemm_9269f6841d7aeb58471c", "split_fp32_det_bf16x3": "cake_dense_projection_det_gemm_5b4e87b523f799e09384", "swap_ab_w128_t64_s8": "cake_dense_projection_det_gemm_53e98138d567774579ac", "swap_ab_w128_t64_s8_we": "cake_dense_projection_det_gemm_edf7fa4c4f889f0c4981", "swap_ab_w64_t64_s12": "cake_dense_projection_det_gemm_9d269d61e56ad3d54fae", "swap_ab_w64_t64_s12_we": "cake_dense_projection_det_gemm_02803e1d2393113c0f6c"}, } # fmt: on diff --git a/tests/experimental/test_cake_dense_projection_det_gemm.py b/tests/experimental/test_cake_dense_projection_det_gemm.py index 67ee1f635f9..8ed1f9a4c17 100644 --- a/tests/experimental/test_cake_dense_projection_det_gemm.py +++ b/tests/experimental/test_cake_dense_projection_det_gemm.py @@ -2474,7 +2474,8 @@ def test_round24_rules_plan_like_the_cake_launcher(T): and old_o.ovl and not old_o.ovr ) - # the sm_100a dense_down / o_proj forwards carry no ovr rule and keep their round-21 programs + # the sm_100a dense_down / o_proj forwards carry no ovr rule and keep their round-21 programs (o_proj at T 16231: + # the round-8 T-exact key's row-half-release program, test_gb_t_exact_rules_plan_their_programs) pd100, *_ = plan_dense_projection_det_gemm(d["A"], d["B"], d["out"], **kw) assert ( _nopro(pd100.template) == "dense_proj_det_gemm_kk_n256_m256_ov_ht" @@ -2482,12 +2483,20 @@ def test_round24_rules_plan_like_the_cake_launcher(T): and not pd100.cgrp ) po100, *_ = plan_dense_projection_det_gemm(o["A"], o["B"], o["out"], **kw) - assert ( - _nopro(po100.template) == "dense_proj_det_gemm_kk_n256_m256_ov_ht" - and not po100.ovr - and not po100.cgrp - and po100.group_m == 8 - ) + if T == 16231: + assert ( + _nopro(po100.template) == "dense_proj_det_gemm_kk_n256_m256_ht_cg_or" + and po100.ovr + and po100.cgrp + and po100.group_m == 8 + ) + else: + assert ( + _nopro(po100.template) == "dense_proj_det_gemm_kk_n256_m256_ov_ht" + and not po100.ovr + and not po100.cgrp + and po100.group_m == 8 + ) @pytest.mark.parametrize("T", [16172, 16231]) @@ -3183,18 +3192,21 @@ def test_round33_rules_plan_like_the_cake_launcher(T): # test_gb_t_exact_rules_plan_their_programs); their round 5 adds 2 sm_100a keys (the 12288 -> 4096 # forward: test_gb_t_exact_rules_plan_their_programs); their round 6 adds 2 sm_100a and 2 sm_103a keys # (the 17408 -> 5120 forward: test_gb_t_exact_rules_plan_their_programs); their round 7 adds - # 2 sm_100a keys (the 5120 -> 17408 weight gradient: test_gb_t_exact_wgrad_rules_plan_their_programs); + # 2 sm_100a keys (the 5120 -> 17408 weight gradient: test_gb_t_exact_wgrad_rules_plan_their_programs); their round 8 + # adds one sm_100a key and one sm_103a key (the 16384 -> 6144 forward: test_gb_t_exact_rules_plan_their_programs) + # and re-values two round-6 keys; # the 4096 -> 1024 fp32 weight gradient adds one sm_107a T-exact key at T 16384 (the T 16231 key's rule: # test_sm107a_t16384_wgrad_key_plans_the_t16231_rule) assert ( len(ROW_RULES) - == 109 + 79 + 86 + 1 + 6 + 26 + 34 + 16 + 10 + 4 + 4 + 2 + 4 + 2 + 1 + == 109 + 79 + 86 + 1 + 6 + 26 + 34 + 16 + 10 + 4 + 4 + 2 + 4 + 2 + 2 + 1 ) assert ( - sum(k[0] == "sm_100a" for k in ROW_RULES) == 51 + 79 + 8 + 4 + 2 + 2 + 2 + 2 + 2 + sum(k[0] == "sm_100a" for k in ROW_RULES) + == 51 + 79 + 8 + 4 + 2 + 2 + 2 + 2 + 2 + 1 ) assert sum(k[0] == "sm_107a" for k in ROW_RULES) == 58 + 6 + 26 + 34 + 1 - assert sum(k[0] == "sm_103a" for k in ROW_RULES) == 86 + 1 + 8 + 6 + 2 + 2 + 2 + assert sum(k[0] == "sm_103a" for k in ROW_RULES) == 86 + 1 + 8 + 6 + 2 + 2 + 2 + 1 assert ARCH_RULE_DEFAULTS == { "sm_107a": {"pro": 7}, "sm_100a": {"pro": 7}, @@ -3772,7 +3784,7 @@ def test_round21_cgrp_knob(T): @pytest.mark.parametrize("T", [16172, 16231]) -def test_round24_ovr_knob(T): +def test_round24_ovr_knob(T, monkeypatch): # field 30 of the key - the LAST field through round 26, 31 fields since round 27 (round 24; the round-20 W3 knob ported in round 23 by W4, Cake 8d837b1ac8): the overlapped tall # epilogue released by ROW HALF (symbol ``_or``, after ``_cg``): the eight epilogue warps drain TMEM buffer 0 and release it to the next # tile's half-0 MMAs while buffer 1 drains; the B stage keeps the plain tall layout (one N = 256 MMA per row half per K step). Tall @@ -3825,44 +3837,54 @@ def test_round24_ovr_knob(T): # launcher's round-24 precedence rule, asserted below on the sm_107a o_proj / dense_down forwards) kw = dict(sm_count=148, l2_bytes=L2_BYTES, arch="sm_100a", _fallback=False) v = _views("proj", "o_proj", "fwd", "bf16", T) - base, *_ = plan_dense_projection_det_gemm(v["A"], v["B"], v["out"], **kw) - assert ( - base.ovl - and base.htail - and not base.ovr - and _nopro(base.template) == "dense_proj_det_gemm_kk_n256_m256_ov_ht" - ) - forced, *_ = plan_dense_projection_det_gemm( - v["A"], v["B"], v["out"], ovr=True, **kw - ) - assert ( - forced.ovr - and not forced.ovl - and forced.htail - and _nopro(forced.template) == "dense_proj_det_gemm_kk_n256_m256_ht_or" - ) - for f in ( - "m_tiles", - "n_tiles", - "pair_tiles", - "sm_pairs", - "num_full", - "tail_tiles", - "sk_units", - "iters_per_unit", - "stages", - "epi", - "slots", - "hints", - "group_m", - "grid", - "ws_f32_elems", - ): - assert getattr(forced, f) == getattr(base, f), f - with pytest.raises(ValueError): - plan_dense_projection_det_gemm( - v["A"], v["B"], v["out"], ovr=True, ovl=True, **kw + # the round-8 T-exact key of this row at T 16231 plans the row-half-release program; (1) reads the ragged rule, so + # it plans without that key + with monkeypatch.context() as m: + m.delitem( + ROW_RULES, + ("sm_100a", False, False, False, False, False, 6144, 16384, 16231), + raising=False, + ) + cake_backend.clear_convenience_caches() + base, *_ = plan_dense_projection_det_gemm(v["A"], v["B"], v["out"], **kw) + assert ( + base.ovl + and base.htail + and not base.ovr + and _nopro(base.template) == "dense_proj_det_gemm_kk_n256_m256_ov_ht" ) + forced, *_ = plan_dense_projection_det_gemm( + v["A"], v["B"], v["out"], ovr=True, **kw + ) + assert ( + forced.ovr + and not forced.ovl + and forced.htail + and _nopro(forced.template) == "dense_proj_det_gemm_kk_n256_m256_ht_or" + ) + for f in ( + "m_tiles", + "n_tiles", + "pair_tiles", + "sm_pairs", + "num_full", + "tail_tiles", + "sk_units", + "iters_per_unit", + "stages", + "epi", + "slots", + "hints", + "group_m", + "grid", + "ws_f32_elems", + ): + assert getattr(forced, f) == getattr(base, f), f + with pytest.raises(ValueError): + plan_dense_projection_det_gemm( + v["A"], v["B"], v["out"], ovr=True, ovl=True, **kw + ) + cake_backend.clear_convenience_caches() # (2) the reverse direction (Cake 95da40d6e8 L2745-L2746, `if ovr and ovl and not rule_ovl: ovr = False` inside the rule read of ovr): a # caller-forced ovl=True on the sm_107a o_proj fwd row (rule {cta_rows 256, htail, ovr, cgrp, group_m 8}) drops the rule's ovr - the # rule's cgrp / htail / group_m stay and the plan is the `_ov_ht_cg` program of test_round21_cgrp_knob, not a ValueError (ovl + ovr); @@ -6649,8 +6671,8 @@ def test_registry_fallback_decodes_the_nearest_generated_program(monkeypatch): # whose choice is the planner's default, the default) must plan exactly this program. The (key, T) pairs the GB T-exact # rules re-measured (the sm_100a 4096 -> 48 input gradient and 4096 -> 512 forward, the sm_103a 4096 -> 512 and # 5120 -> 512 forwards, at T 16231 / 16384; round 4: the 5120 -> 17408 input gradient; round 5: the sm_100a -# 12288 -> 4096 forward; round 6: the 17408 -> 5120 forward, the ragged rule's program) name their T-exact key and -# its program. +# 12288 -> 4096 forward; round 6: the 17408 -> 5120 forward, the ragged rule's program; round 8: its form with the tall +# epilogue released by row half) name their T-exact key and its program. # --------------------------------------------------------------------------- # fmt: off MEASURED_SHAPE_ROWS = ( @@ -6688,8 +6710,8 @@ def test_registry_fallback_decodes_the_nearest_generated_program(monkeypatch): (('sm_100a', False, False, False, False, False, 4096, 12288, 16384), 16384, 152, 'dense_proj_det_gemm_kk_n256_m256_ov_ht_tma1_cg_pro7'), (('sm_100a', False, False, False, False, False, 5120, 5120, None), 16231, 152, 'dense_proj_det_gemm_kk_n256_cg_pro15'), (('sm_100a', False, False, False, False, False, 5120, 5120, None), 16384, 152, 'dense_proj_det_gemm_kk_n256_cg_pro15'), - (('sm_100a', False, False, False, False, False, 5120, 17408, 16231), 16231, 152, 'dense_proj_det_gemm_kk_n256_m256_ov_ht_cg_pro7'), - (('sm_100a', False, False, False, False, False, 5120, 17408, 16384), 16384, 152, 'dense_proj_det_gemm_kk_n256_m256_ov_ht_cg_pro7'), + (('sm_100a', False, False, False, False, False, 5120, 17408, 16231), 16231, 152, 'dense_proj_det_gemm_kk_n256_m256_ht_cg_or_pro7'), + (('sm_100a', False, False, False, False, False, 5120, 17408, 16384), 16384, 152, 'dense_proj_det_gemm_kk_n256_m256_ht_cg_or_pro7'), (('sm_100a', False, False, False, False, False, 12288, 4096, None), 16231, 152, 'dense_proj_det_gemm_kk_n256_cg_pro15'), (('sm_100a', False, False, False, False, False, 12288, 4096, None), 16384, 152, 'dense_proj_det_gemm_kk_n256_cg_pro15'), (('sm_100a', False, False, False, False, False, 16384, 2048, None), 16172, 152, 'dense_proj_det_gemm_kk_n256_cg_pro15'), @@ -7142,7 +7164,8 @@ def test_sm103a_inherits_the_sm100a_rules_where_it_has_none(): a_mn, b_mn, out_f32, out_t, batched, N, K, M = key dims = (N, 16231 if K is None else K, 16231 if M is None else M) # an own sm_103a rule of the row at these dims wins over the parent's: the GB T-exact keys of the 6144 x 2048 - # rows at T 16231 over the sm_100a ragged rules, the ragged 4096 -> 48 rule over the sm_100a T-exact keys; + # rows and of the 16384 -> 6144 forward at T 16231 over the sm_100a ragged rules, the ragged 4096 -> 48 rule + # over the sm_100a T-exact keys; # an own T-exact key wins over the row's own ragged rule (the 5120 -> 17408 input gradient and the # 17408 -> 5120 forward at T 16231) src = (arch_of,) + key @@ -8230,7 +8253,10 @@ def test_eight_warp_form_rules_plan_their_programs(arch, T, monkeypatch): # gradient (the overlapped half-tail program with the column-grouped raster in place of the ragged rule's row-half # program), with the bf16 tiles through the one-slot TMA store; round 5 adds the sm_100a 12288 -> 4096 forward (the # ragged rule's overlapped half-tail program with the column-grouped raster), the same way; round 6 adds the -# 17408 -> 5120 forward (the ragged rule's own program, 12-column-tile raster groups in place of its 8). +# 17408 -> 5120 forward (the ragged rule's own program, 12-column-tile raster groups in place of its 8); round 8 moves +# it to the tall epilogue released by row half on sm_100a and adds the 16384 -> 6144 forward at T 16231 (the half-tail +# program with the tall epilogue released by row half and the column-grouped raster in 8-column-tile groups, in place of +# the ragged rule's overlapped half-tail program). GB_T_EXACT_ROWS = { ("sm_100a", True, 2048, 16384): ( (16172, 16231), @@ -8286,19 +8312,25 @@ def test_eight_warp_form_rules_plan_their_programs(arch, T, monkeypatch): ), ("sm_100a", False, 5120, 17408): ( (16231, 16384), - "dense_proj_det_gemm_kk_n256_m256_ov_ht_cg_pro7", + "dense_proj_det_gemm_kk_n256_m256_ht_cg_or_pro7", ), ("sm_103a", False, 5120, 17408): ( (16231, 16384), "dense_proj_det_gemm_kk_n256_m256_ov_ht_cg_pro7", ), + ("sm_100a", False, 6144, 16384): ( + (16231,), + "dense_proj_det_gemm_kk_n256_m256_ht_cg_or_pro7", + ), + ("sm_103a", False, 6144, 16384): ( + (16231,), + "dense_proj_det_gemm_kk_n256_m256_ht_cg_or_pro7", + ), } # The GB T-exact rows whose keys keep the ragged rule's program and change only its raster group (round 6: the -# 17408 -> 5120 forward in 12-column-tile groups, the ragged rule's 8): the row's plan without the key runs the same -# program at the ragged rule's raster group. -GB_T_EXACT_RASTER_ROWS = frozenset( - {("sm_100a", False, 5120, 17408), ("sm_103a", False, 5120, 17408)} -) +# 17408 -> 5120 forward in 12-column-tile groups, the ragged rule's 8; sm_103a only since round 8): +# the row's plan without the key runs the same program at the ragged rule's raster group. +GB_T_EXACT_RASTER_ROWS = frozenset({("sm_103a", False, 5120, 17408)}) GB_T_EXACT_KEYS = sorted( (arch, False, b_mn, False, False, False, N, K, T) for (arch, b_mn, N, K), (Ts, _template) in GB_T_EXACT_ROWS.items() @@ -8326,7 +8358,8 @@ def test_gb_t_exact_rules_plan_their_programs(arch, monkeypatch): round 2 on both: the q_b 2048 x 16384 input gradient with the column-grouped raster at the rule's raster group; round 4: the 5120 -> 17408 input gradient with the overlapped half-tail program through the one-slot TMA store; round 5 on sm_100a: the 12288 -> 4096 forward, the same way; round 6: the 17408 -> 5120 forward - with the ragged rule's program in 12-column-tile raster groups) + with the ragged rule's program in 12-column-tile raster groups; round 8: on sm_100a the tall + epilogue released by row half, and the 16384 -> 6144 forward at T 16231 with that form in 8-column-tile groups) plan their program at their T on every SM count of the architecture's parts, with whole K per output tile like the row's plan without the key (another program, or for round 6 the same program at another raster group); one token more or less plans that plan. The sm_103a 4096 -> 48 input gradient