Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions aiter/aot/flydsl/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ the JIT path hits the cache instead of compiling again.
| `grouped_moe.py` | `GROUPED_MOE` | gfx1250 grouped MoE GEMM kernels |
| `chunk_gdn_h.py` | `CHUNK_GDN_H` | chunk-gdn-h opt (K5) kernels |
| `mega_moe.py` | `MEGA_MOE` | MegaMoE A8W4 profile bundles for MTPR 8192/16384/32768 |
| `fmha_fp8.py` | `FMHA_FP8` | gfx950 FP8 flash-attention forward, per head shape listed in `configs/fmha_fp8_aot.csv` + `configs/model_configs/*_fmha_fp8_aot.csv` |
| `common.py` | — | Shared job collection, the deadlock-free fork pool, and cache-hit checking logic |

---
Expand Down Expand Up @@ -56,6 +57,22 @@ python -m aiter.aot.flydsl.mega_moe
python -m aiter.aot.flydsl.mega_moe --experts-per-rank 48
```

The FP8 flash-attention kernel picks its tile, rescale threshold and split-K
factor per call, so a serving process JIT-compiles each new combination on the
request path. `fmha_fp8` enumerates every combination the wrapper's heuristics
can reach for each head shape listed in `AITER_CONFIG_FMHA_FP8_AOT_FILE`: the
header-only `configs/fmha_fp8_aot.csv` merged with
`configs/model_configs/*_fmha_fp8_aot.csv`. Add a model by adding a row
(`cu_num,num_heads,num_kv_heads,head_dim,head_dim_v,layout,model`, heads per rank,
layout `varlen_cross` for the chunked-prefill serving path).

```bash
# every configured shape (Kimi-K3 at TP8: 12:12:192:128 -> 92 kernels)
python -m aiter.aot.flydsl.fmha_fp8
# ad hoc shape, without touching the CSVs
python -m aiter.aot.flydsl.fmha_fp8 --shape 12:12:192:128
```

MegaMoE defaults to all three DeepSeek-V4-Pro deployment profiles: r0/r32/r64
(`experts_per_rank=48/52/56`). The expert count is part of the compiled ABI, so
one profile cannot safely reuse another profile's bundle.
Expand Down
5 changes: 5 additions & 0 deletions aiter/aot/flydsl/common.py
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ class OpKind(enum.Enum):
GROUPED_MOE = "grouped_moe"
CHUNK_GDN_H = "chunk_gdn_h"
MEGA_MOE = "mega_moe"
FMHA_FP8 = "fmha_fp8"


@dataclass(frozen=True)
Expand Down Expand Up @@ -151,6 +152,8 @@ def _collect_aot_jobs_for(kind: OpKind) -> list[dict[str, Any]]:
from .grouped_moe import DEFAULT_CSVS, parse_csv
elif kind is OpKind.CHUNK_GDN_H:
from .chunk_gdn_h import DEFAULT_CSVS, parse_csv
elif kind is OpKind.FMHA_FP8:
from .fmha_fp8 import DEFAULT_CSVS, parse_csv
else:
raise ValueError(f"unknown FlyDSL AOT kind: {kind!r}")
return collect_aot_jobs(DEFAULT_CSVS, parse_csv)
Expand All @@ -159,6 +162,8 @@ def _collect_aot_jobs_for(kind: OpKind) -> list[dict[str, Any]]:
def _compile_one_config_for(kind: OpKind) -> Callable[..., dict[str, Any]]:
if kind is OpKind.MEGA_MOE:
from .mega_moe import compile_one_config
elif kind is OpKind.FMHA_FP8:
from .fmha_fp8 import compile_one_config
elif kind is OpKind.MOE:
from .moe import compile_one_config
elif kind is OpKind.MXFP4_MOE:
Expand Down
Loading
Loading