Skip to content
Merged
Show file tree
Hide file tree
Changes from 14 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -537,6 +537,9 @@ match what the code uses today; values are strings unless noted.
| `FLASHINFER_EXTRA_CFLAGS` | unset | `flashinfer/jit/cpp_ext.py` | Extra compiler flags passed to the host C++ compiler. |
| `FLASHINFER_EXTRA_CUDAFLAGS` | unset | `flashinfer/jit/cpp_ext.py` | Extra compiler flags passed to `nvcc`. |
| `FLASHINFER_EXTRA_LDFLAGS` | unset | `flashinfer/jit/cpp_ext.py` | Extra linker flags passed to the linker. |
| `FLASHINFER_JIT_CACHE_WHEEL_KIND` | `legacy` | `flashinfer-jit-cache/build_backend.py` | Select `legacy` for the monolithic binary wheel or experimental `shim` mode for a pure-Python wheel whose dependencies install binary providers. |
| `FLASHINFER_JIT_CACHE_PROVIDER_ARCHS` | falls back to `FLASHINFER_CUDA_ARCH_LIST` | `flashinfer-jit-cache/build_backend.py` | Space-separated provider architectures added to a shim wheel's exact `Requires-Dist` metadata. Required for a shim build when `FLASHINFER_CUDA_ARCH_LIST` is unset. |
| `FLASHINFER_JIT_CACHE_PROVIDER_ARCH` | unset | `flashinfer-jit-cache-provider/package_config.py` | Select exactly one architecture, such as `9.0a` or `sm120f`, when building an experimental binary provider wheel. |

##### Cubin / Artifact Loader

Expand Down
326 changes: 326 additions & 0 deletions docs/design_docs/jit_cache_provider_wheels.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,326 @@
# JIT Cache Provider Wheels

## Status

This document describes an experimental replacement for the monolithic
`flashinfer-jit-cache` wheel. The runtime discovery contract and a single-arch
provider build prototype are implemented, but release workflows still build the
legacy wheel by default. The provider inventory must be validated on the fork
before this becomes the release format.

## Problem

The current wheel puts every generated AOT shared library in one distribution.
Adding a CUDA architecture adds code to many of those libraries, so the wheel
continues to grow and is already close to release-asset limits. A local-version
split such as `flashinfer-jit-cache+cu130.sm90` does not solve installation:
those artifacts are versions of the same distribution, and pip can install only
one of them at a time.

The published jit-cache wheels contain host code plus SASS cubins. They do not
contain PTX. An unsuffixed sm80 cubin has CUDA's same-major binary compatibility,
but cannot provide a fallback for Hopper or Blackwell. Architecture-specific
targets such as sm90a and sm121a are exact-target binaries and are not forward
compatible.

The current monolithic build also uses size-oriented fatbin compression, omits
single-request prefill/decode modules, and excludes selected architectures from
size-constrained matrix entries. Those are useful tactical reductions, but they
do not replace provider splitting: removing an architecture from an aggregate
wheel removes its warm-cache coverage entirely. Provider coverage must therefore
be a separate policy from the architecture list used to keep a monolithic wheel
under its publication limit. In particular, standalone sm75 and sm121a providers
can retain coverage even when those targets are absent from the corresponding
aggregate wheel.

The new layout must support both of these workflows:

- A normal installation gets every provider published for its CUDA and CPU
platform.
- A size-sensitive installation gets only the provider for the target GPU, or
an explicitly selected set of GPU architectures.

## Package Model

`flashinfer-jit-cache` becomes a small shim. For a CUDA 13.0 build, its version
remains aligned with FlashInfer, for example `0.6.16+cu130`, and its dependency
metadata names all providers built in that release.

Each binary wheel has a distinct distribution name and can therefore coexist:

| Distribution | Python package | Declared target |
| --- | --- | --- |
| `flashinfer-jit-cache-sm80` | `flashinfer_jit_cache.providers.sm80` | `sm80` |
| `flashinfer-jit-cache-sm90a` | `flashinfer_jit_cache.providers.sm90a` | `sm90a` |
| `flashinfer-jit-cache-sm120f` | `flashinfer_jit_cache.providers.sm120f` | `sm120f` |

CUDA compatibility remains in the version, and CPU compatibility remains in
the wheel platform tag. For example, an x86 CUDA 13 provider is
`flashinfer_jit_cache_sm120f-0.6.16+cu130-...-manylinux_2_28_x86_64.whl`.

```mermaid
flowchart LR
shim["flashinfer-jit-cache shim"] --> sm80["provider sm80"]
shim --> sm90["provider sm90a"]
shim --> sm120["provider sm120f"]
runtime["FlashInfer AOT resolver"] --> shim
runtime -->|"module + target SM"| selected["one compatible provider root"]
```

Provider wheels register the `flashinfer.jit_cache.providers` entry-point group.
The entry point returns a generated manifest with this schema:

```json
{
"schema_version": 1,
"provider_id": "sm120f",
"distribution": "flashinfer-jit-cache-sm120f",
"version": "0.6.16+cu130",
"cuda_architectures": ["sm120f"],
"modules": ["module_name_1", "module_name_2"]
}
```

The module list is generated from the packaged files, rather than maintained by
hand. Runtime accepts a provider only when its FlashInfer version matches, its
manifest contains the requested module, and its architecture set covers every
target in the active compilation context. If no provider matches, normal JIT
compilation remains the fallback. The legacy monolithic directory is checked
first while both formats are supported.

### Target Compatibility Policy

CUDA distinguishes three target kinds:

- An architecture-specific `smXXa` or `compute_XXa` target runs only on that
exact compute capability. It is neither forward nor backward compatible.
- A family-specific `smXXf` or `compute_XXf` target runs only on devices in the
CUDA-defined family for that target.
- An unsuffixed cubin uses CUDA's baseline binary-compatibility rules, including
forward compatibility to a higher minor compute capability in the same major
family. Unsuffixed cubins are not compatible across major families.

The initial provider implementation deliberately requires an exact manifest
target match for all three forms. Minimal auto-detection on an SM121 device
therefore installs the `sm121a` provider, not `sm120f`; similarly, an `sm100a`
provider cannot satisfy an `sm103a` target. This conservative rule treats
binary executability and complete AOT module coverage as separate questions.
Broader baseline or family coverage can be introduced later only through
explicit manifest coverage that has been validated for every packaged module;
the resolver must never infer a closest lower provider from the target name.

## Installation Modes

The normal mode installs the shim with dependencies enabled:

```bash
flashinfer install-jit-cache-wheel
```

This leaves provider selection to the shim's static `Requires-Dist` metadata.
Pip does not detect GPUs and should not be asked to make a hardware-dependent
resolution decision.

Minimal mode installs the shim and selected providers in one no-dependencies
transaction:

```bash
flashinfer install-jit-cache-wheel --mode minimal
flashinfer install-jit-cache-wheel --mode minimal --sm sm120f
```

Without `--sm`, the CLI uses the visible CUDA devices. `--sm` may be repeated
when preparing an image on a different machine. The current prototype provider
is self-contained for one target. Minimal mode installs exactly the detected or
requested providers and never adds an implicit sm80 baseline.

## SM80 Is Not a Baseline Provider

FlashInfer's NVCC flags emit SASS targets such as `code=sm_80`; they do not emit
a PTX fallback such as `code=compute_80`. NVIDIA documents unsuffixed cubin
compatibility within the same major compute capability. Unsuffixed PTX can
provide broader forward compatibility, but architecture-specific `compute_XXa`
PTX remains exact-target only. An sm80 cubin therefore cannot run on Hopper or
Blackwell.

Artifact inspection confirms that this is also true of the published wheels,
not just the current source flags. Every CUDA-bearing `.so` in the complete
v0.4.0 cu128, cu129, and cu130 matrix for both x86_64 and AArch64 contained SASS
and no PTX. A complete scan of the v0.6.16.post1 cu130 AArch64 wheel also found
no PTX. The standalone `flashinfer-jit-cache` package has therefore never
provided the proposed compute_80 PTX fallback in the audited release range.

The sm80 provider remains useful for systems that actually contain Ampere GPUs.
It is an independent provider, not a dependency of sm90, sm100, sm120, or sm121
providers. Shim dependency lists are literal: sm80 is installed only when the
published all-provider set or an explicit user selection includes it.

The existing `sm80` capability in `flashinfer/aot.py` is a source and module
enumeration condition: it selects kernels whose implementation requires the
Ampere `cp.async` baseline. It does not establish binary portability. Adding
`8.0` to a Blackwell build can change which module names are generated, but it
does not make those modules usable on Blackwell, and it can conceal missing AOT
registration conditions.

For this design, use these terms precisely:

- **Common module**: a module name used by several runtime architectures.
- **Architecture module**: a module name selected only by architecture-specific
dispatch.
- **Provider coverage**: the SASS or PTX targets actually present in every
`.so` in a provider.

The initial provider build compiles the common and architecture modules together
for one target. This duplicates common modules across the complete provider set,
but it gives minimal installations correct native binaries and makes each wheel
small enough to publish independently. A later common-module split would be a
size optimization only. It is not needed for compatibility and should be added
only if inventory data shows that it materially reduces total size.

The first SM121 experiment found 652 SM121a-only modules, one host-only module,
and four SM90a-only FA3 attention-sink modules in a 657-module provider. The
foreign modules were 6.1 MB compressed out of a 250.2 MB compressed shared-object
payload. They came from unconditional FA3 registration, not from a reusable
binary base, and are now omitted unless the AOT target includes SM90. This is an
example of an AOT inventory bug that a provider-wide architecture label alone
would conceal.

Relevant CUDA compatibility references:

- [CUDA C Programming Guide: Binary Compatibility](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#binary-compatibility)
- [Blackwell Compatibility Guide](https://docs.nvidia.com/cuda/blackwell-compatibility-guide/index.html)

## Prototype Builds

The existing project remains a legacy monolithic build unless explicitly put in
shim mode. This keeps release jobs unchanged during fork testing.

Build one provider from `flashinfer-jit-cache-provider`:

```bash
FLASHINFER_JIT_CACHE_PROVIDER_ARCH=12.0f \
FLASHINFER_LOCAL_VERSION=cu130 \
python -m build --wheel flashinfer-jit-cache-provider
```

A direct build uses the native `linux_<arch>` platform tag. The wheelhouse
wrapper opts into `manylinux_2_28` only after verifying the selected container
uses glibc 2.28 or older; custom `DOCKER_IMAGE` values remain native-tagged
unless the caller explicitly requests and satisfies that check.

Build a shim whose default dependencies include the tested provider set:

```bash
# sm80 is an independent provider in this explicit all-provider set.
FLASHINFER_JIT_CACHE_WHEEL_KIND=shim \
FLASHINFER_JIT_CACHE_PROVIDER_ARCHS="8.0 9.0a 12.0f" \
FLASHINFER_LOCAL_VERSION=cu130 \
python -m build --wheel flashinfer-jit-cache
```

The provider backend forces `FLASHINFER_CUDA_ARCH_LIST` to exactly one declared
target, installs the centrally configured provider-build dependencies in its
isolated build environment, compiles the AOT inventory, and generates the
manifest from the copied libraries. Shim requirements exactly pin every provider
to the shim version.

### One-off SM121 wheelhouse

On an AArch64 Docker host, the experiment can be built with:

```bash
scripts/build_jit_cache_provider_wheelhouse.sh
```

The wrapper defaults to target `12.1a`, local version `cu130`, one NVCC thread,
and at most four concurrent jobs. It resolves the CUDA version, PyTorch index,
and manylinux builder from `ci/cuda-versions.json`, then constrains isolated PEP
517 builds to the selected PyTorch distribution. It builds these aligned-version
artifacts under `dist/`:

- `flashinfer-python`
- `flashinfer-jit-cache-sm121a`
- `flashinfer-jit-cache`, with an exact dependency on the SM121 provider only

Set `OUTPUT_DIR`, `FLASHINFER_DEV_RELEASE_SUFFIX`, or
`FLASHINFER_JIT_CACHE_PROVIDER_ARCH` to override the one-off defaults. Existing
output is retained unless `CLEAN_OUTPUT=1` is explicit.

Use the same wrapper for a CUDA 13.4 provider by selecting the central matrix
label. This chooses the `nightly/cu134` PyTorch index and CUDA 13.4 builder:

```bash
FLASHINFER_LOCAL_VERSION=cu134 \
FLASHINFER_JIT_CACHE_PROVIDER_ARCH=12.1a \
scripts/build_jit_cache_provider_wheelhouse.sh
```

`--print-config` resolves and prints this configuration without starting Docker.
The selected provider may intentionally be absent from the monolithic matrix
architecture list; sm121a is one such size-driven exception.

The build does not require a GPU. Its final validation checks distribution and
version metadata, the shim's exact provider pin, provider entry-point discovery,
manifest/module parity, and `cuobjdump --list-elf` output for every shared
library. It writes `wheelhouse.json` with sizes, hashes, the module inventory,
and observed per-module CUDA targets, plus a conventional `SHA256SUMS` file.
Strict CUDA target validation is the default; set
`CUDA_ARCHITECTURE_POLICY=report` only when inventorying a known mixed-target
prototype. After installing the resulting wheels on the target system, run the
JIT-disabled GPU gate with:

```bash
FLASHINFER_DISABLE_JIT=1 \
python scripts/smoke_test_jit_cache_provider.py --provider sm121a
```

The initial DGX Spark build produced a 250,767,917-byte provider wheel containing
657 modules (487,366,496 bytes uncompressed), a 16,105,950-byte
`flashinfer-python` wheel, and a 4,349-byte shim. The build generated 3,404 Ninja
steps with seven concurrent jobs. The installed provider resolved and launched
`silu_and_mul` on an SM121 GB10 with JIT disabled. These are historical prototype
measurements; current module pruning and compression flags require a fresh
wheelhouse measurement before release integration.

## Validation Gates

Before changing release workflows or making shim mode the default:

1. Build each CUDA and CPU matrix entry on the fork and record compressed size,
uncompressed size, module count, and build time per provider.
2. Use `cuobjdump --list-elf` and `cuobjdump --list-ptx` on every packaged `.so`.
Compare actual cubin targets with the provider manifest and require native
providers to contain no PTX.
3. Compare module inventories from one-target builds with the current multi-arch
build. Any module that appears only when `8.0` is added needs its AOT
registration condition corrected or an explicit support decision.
4. Install only the target provider and run with `FLASHINFER_DISABLE_JIT=1` on
representative sm80, sm90a, sm100a, sm120f, and sm121a systems.
5. Install the default shim with every provider and repeat the tests to verify
deterministic architecture selection.
6. Test a process with heterogeneous visible GPUs. Until a provider contains
all required targets, it should miss AOT cleanly and fall back to JIT.
7. Add negative selection tests proving that architecture-specific providers do
not match another compute capability, including sm100a versus sm103a and
sm121a versus future SM12x targets. Keep baseline and family providers exact
until broader coverage is represented explicitly and validated module by
module.
8. Update release and nightly matrices, wheel-index parsing, documentation, and
stale-provider uninstall behavior only after the inventories pass. Keep the
CUDA version and PyTorch index in `ci/cuda-versions.json`, and add explicit
provider coverage fields rather than inferring them from size-pruned
monolithic architecture lists.

## Open Decisions

- Whether later provider manifests should add broader baseline or family
coverage after every `.so` and module inventory has been validated for the
additional devices. Architecture-specific `a` targets always remain exact.
- Whether a measured common module set warrants separate common-module
providers as a size optimization.
- Whether provider manifests should include hashes and per-module code targets,
rather than the provider-wide target list used by the prototype.
- Which architectures removed from aggregate wheels for size, including sm75
and sm121a, should remain in each platform's default provider set.
- How long to retain legacy monolithic wheel production after the shim becomes
the default.
6 changes: 6 additions & 0 deletions flashinfer-jit-cache-provider/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
build/
dist/
*.egg-info/
flashinfer_jit_cache_provider/_build_meta.py
flashinfer_jit_cache_provider/manifest.json
flashinfer_jit_cache_provider/jit_cache/
Loading
Loading