Skip to content

[Diffusion] Cache-DiT 1.5.1: DMD Calibrator, SVDQuant DQ, etc. #37774

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
DefTruth wants to merge 24 commits into
base: main
Choose a base branch
from
Open

[Diffusion] Cache-DiT 1.5.1: DMD Calibrator, SVDQuant DQ, etc. #37774

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
b10d333
upgrade cache-dit -> 1.5.0
DefTruth Jun 23, 2026
c1884b7
upgrade cache-dit -> 1.5.0
DefTruth Jun 23, 2026
2c3ac64
update
DefTruth Jun 25, 2026
1890bac
upgrade cache-dit -> 1.5.1
DefTruth Sep 3, 2026
c0e38b7
test: cover DMD calibrator knobs and TaylorSeer exclusivity
DefTruth Sep 3, 2026
de728a7
chore: update .gitignore
DefTruth Sep 3, 2026
8225cd7
diffusion: add sglang-diffusion-cache-dit skill
DefTruth Sep 3, 2026
51c3b41
Merge branch 'main' into cache-dit-1.5.1
DefTruth Sep 3, 2026
5f01700
Fix Cache-DiT skill markdown lint
BBuf Sep 4, 2026
0b1164b
Merge pull request #1 from BBuf/fix-pr-37774-lint
DefTruth Sep 4, 2026
5b2628d
Merge branch 'main' into cache-dit-1.5.1
DefTruth Sep 4, 2026
29d2984
diffusion: update dmd calibrator and svdq-dq docs
DefTruth Sep 4, 2026
7b04a12
skills: update skills
DefTruth Sep 7, 2026
af77d72
Merge branch 'main' into cache-dit-1.5.1
DefTruth Sep 8, 2026
14211a0
[Diffusion] Keep cache-dit integration importable with cache-dit < 1.5.0
DefTruth Sep 8, 2026
7095951
Merge branch 'main' into cache-dit-1.5.1
DefTruth Sep 8, 2026
c057794
Merge branch 'main' into cache-dit-1.5.1
DefTruth Sep 9, 2026
c9b22eb
Merge branch 'main' into cache-dit-1.5.1
DefTruth Sep 11, 2026
9054938
Merge branch 'main' into cache-dit-1.5.1
DefTruth Sep 11, 2026
9fefa7f
Merge branch 'main' into cache-dit-1.5.1
DefTruth Sep 11, 2026
294fdb3
Merge branch 'main' into cache-dit-1.5.1
DefTruth Sep 13, 2026
5151b23
Merge branch 'main' into cache-dit-1.5.1
DefTruth Sep 14, 2026
323aa02
Merge branch 'main' into cache-dit-1.5.1
DefTruth Sep 14, 2026
5fb7afd
Merge branch 'main' into cache-dit-1.5.1
DefTruth Sep 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -174,6 +174,7 @@ benchmark/llava_bench/mme_pack
!tools/sglang-simulator/examples/replay/trace.jsonl
tmp*.txt
/tmp/
.tmp/

# Torch Compile logs
tl_out/
Expand Down
224 changes: 219 additions & 5 deletions docs/docs/sglang-diffusion/cache_dit.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,12 @@ SGLang integrates [Cache-DiT](https://github.com/vipshop/cache-dit), a caching a

- **DBCache (Dual Block Cache)**: Dynamically decides when to cache transformer blocks based on residual differences
- **TaylorSeer**: Uses Taylor expansion for calibration to optimize caching decisions
- **DMD Calibrator**: An **exponential-basis** forecasting calibrator (Dynamic Mode Decomposition, not Distribution Matching Distillation) that serves as a drop-in alternative to TaylorSeer's polynomial basis; strongest on flow-matching models
- **SCM (Step Computation Masking)**: Step-level caching control for additional speedup

Cache-DiT also ships **SVDQuant** W4A4 (int4 / NVFP4) dynamic quantization, which can be combined
with DBCache caching (see [Quantization](#quantization)).

## Basic Usage

Cache-DiT is a **per-request** switch: each request decides whether to run
Expand Down Expand Up @@ -47,8 +51,11 @@ client.images.generate(
`SGLANG_CACHE_DIT_ENABLED` server default). `cache_dit_params` accepts the
DBCache knobs (`Fn_compute_blocks`, `Bn_compute_blocks`, `max_warmup_steps`,
`residual_diff_threshold`, `max_continuous_cached_steps`, `enable_taylorseer`,
`taylorseer_order`), the SCM knobs (`scm_preset`, `scm_compute_bins`,
`scm_cache_bins`, `scm_policy`), and a nested `secondary` dict with the DBCache
`taylorseer_order`), the DMD knobs (`enable_dmd`, `dmd_history`, `dmd_rank`,
`dmd_ridge`, `dmd_svd_precision`; DMD and TaylorSeer are mutually exclusive
calibrators and cannot be enabled together), the SCM knobs (`scm_preset`,
`scm_compute_bins`, `scm_cache_bins`, `scm_policy`), and a nested `secondary`
dict with the DBCache
knobs for the second transformer of dual-DiT models (unset secondary keys
inherit the request's primary values, then the
`SGLANG_CACHE_DIT_SECONDARY_*` defaults).
Expand Down Expand Up @@ -134,6 +141,49 @@ cache_config:
enable_sperate_cfg: true # e.g, Qwen-Image, Wan, Chroma, Ovis-Image, etc.
```

- DBCache + DMD Calibrator

Instead of TaylorSeer, you can use the DMD calibrator: an **exponential-basis** forecasting
calibrator that serves as a drop-in alternative to TaylorSeer's polynomial basis. DMD models
the cached feature stream as a linear dynamical system (`Y_{t+1} ~= A @ Y_t`), forecasts
cached features from the fitted eigen-modes, and stays accurate over longer cache skips where
polynomial extrapolation diverges. DMD here refers to Dynamic Mode Decomposition (Schmid
2010), **not** Distribution Matching Distillation. DMD works best on flow-matching models
(e.g., FLUX), while TaylorSeer is often better on DDPM-style models — try both. DMD and
TaylorSeer are mutually exclusive — enable only one of `enable_dmd` / `enable_taylorseer`:

```yaml Config
cache_config:
max_warmup_steps: 8
warmup_interval: 2
max_cached_steps: -1
max_continuous_cached_steps: 2
Fn_compute_blocks: 1
Bn_compute_blocks: 0 # Bn=0 since the DMD calibrator replaces the Bn calibrator
residual_diff_threshold: 0.12
enable_dmd: true
dmd_history: 6 # snapshot window length, 5-6 typical
dmd_svd_precision: "medium" # "low", "medium" or "high"
```

A `dmd_history` window of 5–6 snapshots is typically the sweet spot — longer histories do not
always help, because the feature dynamics drift across timesteps. With fewer than 4 uniformly
spaced snapshots available, DMD transparently falls back to the Taylor expansion it maintains
internally. See the
[Cache-DiT DMD documentation](https://cache-dit.readthedocs.io/en/latest/user_guide/CACHE_API/#dmd-calibrator-dynamic-mode-decomposition)
for the mathematical principle and quantitative comparisons. A ready-made config is available
at
[examples/configs/cache_dmd.yaml](https://github.com/vipshop/cache-dit/blob/main/examples/configs/cache_dmd.yaml)
in the Cache-DiT repository. Apply it with the same `--cache-dit-config` flag:

```bash
sglang generate \
--backend diffusers \
--model-path Qwen/Qwen-Image \
--cache-dit-config cache_dmd.yaml \
--prompt "A beautiful sunset over the mountains"
```

### Distributed inference

- 1D Parallelism
Expand Down Expand Up @@ -300,6 +350,81 @@ sglang generate \
--prompt "A beautiful sunset over the mountains"
```

#### SVDQuant (W4A4 int4 / NVFP4)

SVDQuant is Cache-DiT's built-in W4A4 PTQ quantization (weights and activations in int4 or
NVFP4, with smoothed low-rank branches). It can be freely combined with DBCache caching and
the DMD calibrator for the largest speedups.

::::note
SVDQuant requires a cache-dit build **with CUDA extension support** — a plain
`pip install cache-dit` does NOT include it. Install one of:

```bash Command
# Option 1: prebuilt CUDA 13 wheel
pip install cache-dit-cu13==<version> --no-deps

# Option 2: build from source with SVDQuant enabled
git clone https://github.com/vipshop/cache-dit
cd cache-dit
export CUDA_HOME=/usr/local/cuda
CACHE_DIT_BUILD_SVDQUANT=1 pip install ".[quantization]" --no-build-isolation
```
::::

Valid `quant_type` values are `svdq_int4_r{32,64,128,256}_dq` (int4 W4A4) and
`svdq_nvfp4_r{32,64,128,256}_dq` (NVFP4 W4A4; requires a Blackwell GPU). Example config
combining SVDQuant NVFP4 with DBCache + DMD (see
[examples/configs/blackwell/cache_dmd_svdq.yaml](https://github.com/vipshop/cache-dit/blob/main/examples/configs/blackwell/cache_dmd_svdq.yaml)):

```yaml Config
cache_config:
max_warmup_steps: 8
warmup_interval: 2
max_cached_steps: -1
max_continuous_cached_steps: 2
Fn_compute_blocks: 1
Bn_compute_blocks: 0
residual_diff_threshold: 0.12
enable_dmd: true
dmd_history: 6
dmd_svd_precision: "medium"
quantize_config:
quant_type: "svdq_nvfp4_r128_dq" # nvfp4 for Blackwell; use svdq_int4_r128_dq for int4
svdq_kwargs:
quantize_device: "cuda"
fused_mlp: true
exclude_layers:
- "embedder"
- "embed"
verbose: false
```

For int4 W4A4 (pre-Blackwell GPUs), the same config with
`quant_type: "svdq_int4_r128_dq"` is available at
[examples/configs/cache_dmd_svdq.yaml](https://github.com/vipshop/cache-dit/blob/main/examples/configs/cache_dmd_svdq.yaml)
(add `runtime_kernel: "v2"` to `svdq_kwargs`).

Enable `torch.compile` for the best SVDQuant performance, and make sure `--warmup-steps`
covers the compile warmup (use the same value as `--num-inference-steps`):

```bash Command
sglang generate \
--backend diffusers \
--model-path black-forest-labs/FLUX.1-dev \
--num-inference-steps=28 \
--warmup-mode request \
--warmup-steps 28 \
--cache-dit-config cache_dmd_svdq.yaml \
--enable-torch-compile \
--dit-cpu-offload false \
--text-encoder-cpu-offload false \
--prompt "A beautiful sunset over the mountains"
```

You can verify from the log that the quantization is active:
`[Cache-DiT] SVDQuant Type: svdq_nvfp4_r128_dq, Rank: 128`.

### Combined Configs: Cache + Parallelism + Quantization

You can also combine all the above configs together in a single yaml file `combined.yaml` that contains:
Expand Down Expand Up @@ -418,12 +543,89 @@ TaylorSeer improves caching accuracy using Taylor expansion:
</tbody>
</table>

### DMD Calibrator Configuration

DMD (Dynamic Mode Decomposition, Schmid 2010 — **not** Distribution Matching Distillation) is
an **exponential-basis** forecasting calibrator and a drop-in alternative to TaylorSeer's
polynomial basis. At each full-compute step it records a snapshot of the computed features; at
a cached step it identifies a linear propagator from the recent snapshot window (one economy
SVD with rank truncation, then eigendecomposition) and forecasts the current features via
eigenvalue powers — cheap to advance, and stable over longer cache skips where polynomial
extrapolation diverges. It typically improves both speed and quality over pure DBCache.
**DMD and TaylorSeer are mutually exclusive** (enabling both raises a `ValueError`); DMD is
best for flow-matching models, TaylorSeer for DDPM-style ones. See the
[Cache-DiT DMD documentation](https://cache-dit.readthedocs.io/en/latest/user_guide/CACHE_API/#dmd-calibrator-dynamic-mode-decomposition)
for details:

<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
<col style={{width: "14%"}} />
<col style={{width: "38%"}} />
<col style={{width: "14%"}} />
<col style={{width: "34%"}} />
</colgroup>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Parameter</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Env Variable</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Default</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Enable</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_DMD`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>false</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Enable the DMD calibrator</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>History</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_DMD_HISTORY`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>6</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Snapshot window length; 5-6 typical. Needs >= 4 uniformly spaced snapshots, otherwise DMD falls back to TaylorSeer</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Rank</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_DMD_RANK`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>0</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>SVD truncation rank; 0 = automatic (drop modes below 1e-4 of the leading singular value)</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Ridge</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_DMD_RIDGE`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>1e-8</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Tikhonov regularization added to the inverted singular values</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>SVD Precision</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`SGLANG_CACHE_DIT_DMD_SVD_PRECISION`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>medium</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>SVD precision: "low", "medium" or "high"</td>
</tr>
</tbody>
</table>

Usage (SGLD backend, env-driven):

```bash Command
SGLANG_CACHE_DIT_ENABLED=true \
SGLANG_CACHE_DIT_DMD=true \
sglang generate --model-path black-forest-labs/FLUX.1-dev \
--prompt "A curious raccoon in a forest"
```

On the diffusers backend, enable DMD from the yaml config instead
(`enable_dmd: true` in `cache_config`, see
[Diffusers Backend](#diffusers-backend)); DMD can also be set per request via
`cache_dit_params: {"enable_dmd": true}`.

### Combined Configuration Example

DBCache and TaylorSeer are complementary strategies that work together, you can configure both sets of parameters
simultaneously:

```bash
```bash Command
SGLANG_CACHE_DIT_ENABLED=true \
SGLANG_CACHE_DIT_FN=2 \
SGLANG_CACHE_DIT_BN=1 \
Expand Down Expand Up @@ -496,7 +698,7 @@ SCM is configured with presets:

**Usage**

```bash
```bash Command
SGLANG_CACHE_DIT_ENABLED=true \
SGLANG_CACHE_DIT_SCM_PRESET=medium \
sglang generate --model-path Qwen/Qwen-Image \
Expand All @@ -507,7 +709,7 @@ sglang generate --model-path Qwen/Qwen-Image \

For fine-grained control over which steps to compute vs cache:

```bash
```bash Command
SGLANG_CACHE_DIT_ENABLED=true \
SGLANG_CACHE_DIT_SCM_COMPUTE_BINS="8,3,3,2,2" \
SGLANG_CACHE_DIT_SCM_CACHE_BINS="1,2,2,2,3" \
Expand Down Expand Up @@ -617,6 +819,18 @@ SGLang Diffusion x Cache-DiT supports almost all models originally supported in
For models with < 8 inference steps (e.g., DMD distilled models), SCM will be automatically disabled. DBCache
acceleration still works.

### SVDQuant unavailable or load failure

SVDQuant cases raise `svdq_is_available() = False` or
`undefined symbol: ... materialize_cow_storage ...` when the installed cache-dit has no CUDA
extension, or the prebuilt wheel was compiled against an incompatible torch. Fix: reinstall
from the `cache-dit-cu13` wheel matching your torch version, or build cache-dit from source
with `CACHE_DIT_BUILD_SVDQUANT=1` (see [Quantization](#quantization)). Quick self-check:

```bash Command
python -c "from cache_dit.quantization.svdquant import svdq_is_available, svdq_get_load_error as e; print(svdq_is_available(), e())"
```

## References

- [Cache-DiT](https://github.com/vipshop/cache-dit)
Expand Down
50 changes: 50 additions & 0 deletions docs/docs/sglang-diffusion/environment_variables.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -253,6 +253,31 @@ See [cache-dit documentation](./cache_dit) for detailed configuration.
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>TaylorSeer order (1 or 2)</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_CACHE_DIT_DMD`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>false</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Enable the DMD (Dynamic Mode Decomposition) calibrator (mutually exclusive with TaylorSeer)</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_CACHE_DIT_DMD_HISTORY`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>6</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>DMD snapshot window length (5-6 typical; needs >= 4 uniformly spaced snapshots, otherwise DMD falls back to TaylorSeer)</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_CACHE_DIT_DMD_RANK`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>0</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>DMD SVD truncation rank (0 = automatic)</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_CACHE_DIT_DMD_RIDGE`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1e-8</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>DMD Tikhonov regularization added to the inverted singular values</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_CACHE_DIT_DMD_SVD_PRECISION`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>medium</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>DMD SVD precision (low/medium/high)</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`SGLANG_CACHE_DIT_SCM_PRESET`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>none</td>
Expand Down Expand Up @@ -329,6 +354,31 @@ For dual-transformer models (e.g., Wan2.2 with high/low-noise experts), these va
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>(from primary)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>TaylorSeer order (1 or 2)</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>SGLANG_CACHE_DIT_SECONDARY_DMD</code></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>(from primary)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Enable the DMD calibrator (mutually exclusive with TaylorSeer)</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>SGLANG_CACHE_DIT_SECONDARY_DMD_HISTORY</code></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>(from primary)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>DMD snapshot window length</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>SGLANG_CACHE_DIT_SECONDARY_DMD_RANK</code></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>(from primary)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>DMD SVD truncation rank (0 = automatic)</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>SGLANG_CACHE_DIT_SECONDARY_DMD_RIDGE</code></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>(from primary)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>DMD Tikhonov regularization term</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>SGLANG_CACHE_DIT_SECONDARY_DMD_SVD_PRECISION</code></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>(from primary)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>DMD SVD precision (low/medium/high)</td>
</tr>
</tbody>
</table>

Expand Down
2 changes: 1 addition & 1 deletion python/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@ runai = ["runai-model-streamer[s3,gcs,azure]>=0.15.7"]
diffusion = [
"addict==2.4.0",
"av==16.1.0",
"cache-dit==1.3.0",
"cache-dit==1.5.1",

@mickqian mickqian Sep 8, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Apply the cache-dit upgrade to Base jobs that import diffusion modules. This pin is under the diffusion extra, but base-b-test-1-gpu-small (1) invokes ci_install_dependency.sh without that extra and installs python[dev,runai,tracing]. On this exact head, its package list still reports cache-dit 1.3.0; importing test/registered/unit/models/test_qwen_image_fp8_norm_quant.py then reaches the new top-level imports in cache_dit_integration.py and fails with ImportError: cannot import name 'BlockAdapterRegister' from 'cache_dit' (failed job). Please make the Base installation path install the required Cache-DiT version before running these tests, and cover the real package/import boundary with a smoke test. The Cache-DiT unit tests add these exports to a stub, so they cannot detect the installed-package mismatch. Rerunning alone does not ensure that this dependency is upgraded.

@DefTruth DefTruth Sep 8, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pointing out this error. This really needs to be fixed. We do need to consider compatibility with other cases where cache-dit hasn't been upgraded to 1.5.1. I'm working on this issue.

@DefTruth DefTruth Sep 8, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you prefer to upgrade all scenarios to cache-dit 1.5.1, or only upgrade python/pyproject.toml first?

@DefTruth DefTruth Sep 8, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mickqian fixed.

python -m  pytest test/registered/unit/models/test_qwen_image_fp8_norm_quant.py python/sglang/multimodal_gen/test/unit/test_cache_dit_integration.py -v
======================================================= test session starts =======================================================
platform linux -- Python 3.12.13, pytest-9.1.1, pluggy-1.6.0 -- /workspace/dev/miniconda3/envs/sgl/bin/python
cachedir: .pytest_cache
rootdir: /workspace/dev/vipshop/sglang/test
configfile: pytest.ini
plugins: anyio-4.13.0
collected 19 items

test/registered/unit/models/test_qwen_image_fp8_norm_quant.py::TestQwenImageFp8NormQuantGate::test_merged_qkv_uses_its_materialized_input_scale PASSED [  5%]
test/registered/unit/models/test_qwen_image_fp8_norm_quant.py::TestQwenImageFp8NormQuantGate::test_nonpositive_scale_keeps_fusion_disabled PASSED [ 10%]
test/registered/unit/models/test_qwen_image_fp8_norm_quant.py::TestQwenImageFp8NormQuantGate::test_separate_qkv_requires_identical_input_scales PASSED [ 15%]
test::TestCacheDitRefreshContext::test_dual_refresh_without_scm_preset_skips_steps_mask PASSED                              [ 21%]
test::TestCacheDitRefreshContext::test_refresh_context_with_scm_preset_uses_steps_mask PASSED                               [ 26%]
test::TestCacheDitRefreshContext::test_refresh_context_without_scm_preset_skips_steps_mask PASSED                           [ 31%]
test::TestBuildCustomBlockAdapter::test_builds_adapter_for_registered_class PASSED                                          [ 36%]
test::TestBuildCustomBlockAdapter::test_custom_adapter_is_retained_until_disable PASSED                                     [ 42%]
test::TestBuildCustomBlockAdapter::test_has_separate_cfg_follows_runtime PASSED                                             [ 47%]
test::TestBuildCustomBlockAdapter::test_minimax_h3_uses_main_blocks_with_hidden_state_pattern PASSED                        [ 52%]
test::TestBuildCustomBlockAdapter::test_raises_when_blocks_attr_missing PASSED                                              [ 57%]
test::TestBuildCustomBlockAdapter::test_returns_none_for_unknown_class PASSED                                               [ 63%]
test::TestCalibratorSelection::test_both_calibrators_raise_on_dual_transformer PASSED                                       [ 68%]
test::TestCalibratorSelection::test_both_calibrators_raise_on_transformer PASSED                                            [ 73%]
test::TestCalibratorSelection::test_dmd_takes_calibrator_slot PASSED                                                        [ 78%]
test::TestCacheDitLegacyFallback::test_enable_dmd_raises_clear_error PASSED                                                 [ 84%]
test::TestCacheDitLegacyFallback::test_fallback_import_binds_registry_and_nulls_dmd PASSED                                  [ 89%]
test::TestCacheDitLegacyFallback::test_taylorseer_path_still_enables_cache PASSED                                           [ 94%]
test::TestCacheDitRealPackageBoundary::test_import_chain_matches_installed_package PASSED                                   [100%]

======================================================== warnings summary =========================================================
<frozen importlib._bootstrap>:488
  <frozen importlib._bootstrap>:488: DeprecationWarning: builtin type SwigPyPacked has no __module__ attribute

<frozen importlib._bootstrap>:488
  <frozen importlib._bootstrap>:488: DeprecationWarning: builtin type SwigPyObject has no __module__ attribute

../../miniconda3/envs/sgl/lib/python3.12/site-packages/torch/jit/_script.py:365: 14 warnings
  /workspace/dev/miniconda3/envs/sgl/lib/python3.12/site-packages/torch/jit/_script.py:365: DeprecationWarning: `torch.jit.script_method` is deprecated. Please switch to `torch.compile` or `torch.export`.
    warnings.warn(

../../miniconda3/envs/sgl/lib/python3.12/site-packages/_pytest/config/__init__.py:1464
  /workspace/dev/miniconda3/envs/sgl/lib/python3.12/site-packages/_pytest/config/__init__.py:1464: PytestConfigWarning: Unknown config option: asyncio_mode

    self._warn_or_fail_if_strict(f"Unknown config option: {key}\n")

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================================ 19 passed, 17 warnings in 10.24s =================================================

@nvpohanh nvpohanh Sep 9, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seeing these CI failures caused by this:

https://github.com/sgl-project/sglang/actions/runs/34321708659/job/102392704016?pr=38426
https://github.com/sgl-project/sglang/actions/runs/34334438529/job/102430646325?pr=35599

Root cause:

PR #37774’s multimodal CI ran on 5090-e-runner-3, upgraded the shared system environment from cache-dit 1.3.0 to 1.5.1, and skipped virtualenv cleanup. PR #38426 later reused that runner without diffusion extras, inheriting the incompatible package.

"cloudpickle==3.1.2",
"diffusers==0.37.0",
"imageio==2.36.0",
Expand Down
Loading
Loading