diff --git a/.claude/rules/cookbook-first.md b/.claude/rules/cookbook-first.md new file mode 100644 index 000000000000..92f2e94a055f --- /dev/null +++ b/.claude/rules/cookbook-first.md @@ -0,0 +1,21 @@ +# Serving a model? Read its cookbook page first + +Before running, deploying, benchmarking, or reproducing a specific model, read +its page under `docs/cookbook/`: + +- `docs/cookbook/autoregressive/` — LLM and VLM serving +- `docs/cookbook/diffusion/` — image, video, 3D, robotics +- `docs/cookbook/omni/`, `docs/cookbook/vla/`, `docs/cookbook/specbundle/`, + `docs/cookbook/base/` + +That page carries the deployment we recommend to users — GPU count, parallelism +degrees, quantization, and the flags that matter for that model — so it is the +answer to "how should this be served", and the baseline any tuning starts from. + +Configs under `test/` are a different thing: pinned, reproducible CI setups for +correctness and regression checks, not deployment advice. Reach for them when +you need an exactly reproducible run, and say which cookbook flags you diverged +from — a measurement taken on a configuration nobody deploys describes nothing. + +Keep it true in both directions: when a change alters a model's recommended +deployment, update that model's cookbook page in the same PR. diff --git a/.claude/rules/unit-test-admission.md b/.claude/rules/unit-test-admission.md index fc573d5f1133..4731a596e4a6 100644 --- a/.claude/rules/unit-test-admission.md +++ b/.claude/rules/unit-test-admission.md @@ -1,6 +1,7 @@ --- paths: - "test/**/*.py" + - "python/sglang/multimodal_gen/test/**/*.py" --- # Unit Test Admission Criteria @@ -68,5 +69,13 @@ One strong case beats several weak ones: each additional case must guard a distinct failure mode. Ask "which bug escapes if I delete this case?" -- no answer means delete it. +New cases join an existing file in the same subsystem by default. Create a new +file only when it needs a different fixture, dependency, owner, or CI contract; +every file pays a separate interpreter-import cost in the CPU gate. + +Suite cadence is part of admission: if a failing run cannot be attributed to a +single PR's diff, the test belongs in a nightly or weekly suite rather than a +per-commit lane. + Test mechanics (placement, CI registration, fixtures) live in [`write-sglang-test`](../skills/write-sglang-test/SKILL.md). diff --git a/.claude/skills/add-jit-kernel/SKILL.md b/.claude/skills/add-jit-kernel/SKILL.md index 9789ce7fd36a..11fa1d02e6ac 100644 --- a/.claude/skills/add-jit-kernel/SKILL.md +++ b/.claude/skills/add-jit-kernel/SKILL.md @@ -28,12 +28,15 @@ Add a new operation that scales each element of a tensor by a scalar factor: These hold for every step below. - **`namespace sglang` is where JIT code lives.** Open it after the include block and close it at the end of the file, with the device kernels, traits and host wrapper inside. The shared `host::` / `device::` helpers are nested in it too, so they resolve unqualified. `load_jit` emits the `TVM_FFI_DLL_EXPORT_TYPED_FUNC` wrapper inside `namespace sglang` as well, so the `kernel_name` you pass from Python needs no `sglang::` prefix. +- **Tuning policy belongs in Python; C++ gets the answer, not the decision.** For a hyperparameter — split factor, threads per item, block size, vector width, an algorithm variant — prefer (not required) making it a **template parameter** and letting the `@cache_once` module factory choose the value, over a runtime `if`/`switch` in the launcher that picks among pre-instantiated kernels. The kernel then compiles exactly one specialisation and asserts its own preconditions with `static_assert`, while the heuristic that produced the value sits in Python where it is readable, adjustable, and inspectable without recompiling CUDA. `store_cache`'s `num_threads` is the worked example: a `get_kernel(num_split)` ladder guarded by byte-alignment `if constexpr`s became one template argument plus a heuristic in `_jit_kvcache_module`. The exception is a knob that genuinely varies per call with the runtime shape — that has to stay a kernel argument. - **Check where the check is cheapest: `static_assert` > C++ host check > cached Python > per-call Python.** Anything fixed at compile time is a `static_assert`. Anything about the tensors is a `TensorMatcher` / `CHECK_HOST` in the C++ launcher, free next to a kernel launch. A check Python cannot delegate goes inside the `@cache_once` module factory, where it runs once per specialisation. What remains in the per-call entry point costs interpreter time on *every* forward, so it should be nothing but picking the module and allocating `out`. - **Fixed-width integer types.** Prefer `int32_t` / `int64_t` / `uint32_t` / `size_t` over `int`, `long`, or `long long`, so an index has the same width on both sides of the FFI boundary. Bare `int` is fine only where the width plainly cannot matter — an unrolled loop counter over a `constexpr` bound, a template `int` parameter. Shapes arrive as `int64_t` (`SymbolicSize::unwrap()`); narrowing to `uint32_t` for in-kernel indexing is a deliberate act, so write the `static_cast` explicitly and only where the range is known. - **Doxygen comments in C++.** Document exported entities with `///` or `/** ... */` blocks using `\brief`, `\param`, `\tparam`, `\return`, the way `include/sgl_kernel/` does. `python -m sglang.kernels.jit` writes `CommentFormat: Doxygen` into `.clangd` when clangd is 21 or newer, so these render on hover in the editor. Plain `//` remains fine for implementation notes inside a function body. - **ASCII only in C++ and CUDA sources.** Write `--`, `->`, `<=` instead of `—`, `→`, `≤`, including in comments. `grep -nP '[^\x00-\x7F]' ` before committing. +- **`namespace details` is private.** Anything inside one is an implementation detail of its own header, free to change without notice. Do not name `details::` from another module, and never `using namespace details`. If you find yourself wanting something in there, that is the signal to promote it to a documented name instead. - **`const T* __restrict__` for read-only pointers.** This is what `csrc/` does throughout, and it lets the compiler emit non-coherent (`LDG`) loads. -- **Watch the register budget.** For memory-bound kernels, keep to roughly 64 registers per thread so occupancy does not become the limit. Build once with `extra_cuda_cflags=["-Xptxas", "-v"]` to see the actual count, and prefer recomputing a value over letting it spill. +- **Watch the register budget.** For memory-bound kernels, keep to roughly 64 registers per thread so occupancy does not become the limit; prefer recomputing a value over letting it spill. Run `SGLANG_JIT_LOG_RESOURCE_USAGE=1 SGLANG_JIT_FORCE_RECOMPILE=1` to log per-kernel registers, spills and shared memory at INFO. Both halves are load-bearing: a cache hit has no compiler output to report, and passing `-Xptxas -v` by hand does nothing on its own because the build captures compiler output and replays it only on failure. +- **Alignment is a contract you enforce, not a property you hope for.** A vectorized copy derives its access width from the row *size* (`load_bytes` picks 16B for a 1024B row), but the address it lands on is `base + index * row_stride` — and a stride is not constrained by the size. `TensorMatcher` admits any stride unless you say otherwise, so a padded cache row silently produces `misaligned address`. Every kernel that vectorizes is highly recommended to end its validation with `.ensure_alignment(w)`, where `w` is the width the kernel actually uses. --- @@ -62,7 +65,9 @@ These hold for every step below. - **Type aliases**: `fp16_t`, `bf16_t`, `fp32_t`, `fp8_e4m3_t`, `fp8_e5m2_t` and their packed variants `fp16x2_t`, `bf16x2_t`, `fp32x2_t`, etc. - **`SGL_DEVICE`** — Expands to `__forceinline__ __device__`. Use on all device functions. -- **`device::kWarpThreads`** — Constant `32`. +- **`SGL_DEVICE_HOST`** — `__forceinline__ __device__ __host__`. Use it when a `constexpr` helper must be callable from both the kernel and its launcher, so the two cannot drift. +- **`device::kWarpThreads`** — Constant `32` (the codebase's *logical* warp width). `kWarpSize` is an alias. Note this is a group size, not the hardware wave: on gfx950 `warpSize` is 64, and `warp::kFullWidth` reflects that. +- **`device::get_lane_id()`** — this thread's index within its logical group. **Prefer it over `threadIdx.x % kNumThreads` whenever the value feeds an address.** On CUDA the lane register is one read that folds into the address computation, whereas the modulo is a derived value the compiler re-materialises at *every* address scale — 8 instructions and 2 registers on a two-tile warp copy, measured on sm_100a. It picks the cheaper form per platform, so callers never need to care. Only equals the true in-warp lane when `blockDim.x` is a multiple of `kNumThreads`. - **`device::load_as(ptr, offset)`** / **`device::store_as(ptr, val, offset)`** — Type-safe loads/stores from `void*`. - **`device::pointer::offset(ptr, offsets...)`** — Pointer arithmetic on device. - **`host::LaunchKernel(grid, block, device_or_stream [, smem])`** — RAII kernel launcher that: @@ -93,6 +98,7 @@ This is the **primary validation API** for all kernel launchers. Use it to valid - `.with_dtype()` — allow a set of types - `.with_device(device_sym)` — require CUDA and bind the checked device to a `SymbolicDevice` - `.with_strides({strides...})` — validate strides (omit to require contiguous) + - `.ensure_alignment(bytes)` — require the data pointer **and every stride except the innermost** to be a multiple of `bytes` (a power of two). Size-1 dimensions are skipped, since their strides are arbitrary. This is how a vectorized kernel states its precondition; see the alignment convention above. - `.verify(tensor_view)` — execute the check; throws `PanicError` with full context on failure; **chainable** (`verify(a).verify(b)` to check multiple tensors with the same shape) - **`host::is_type(dtype)`** — whether a `DLDataType` denotes the C++ type `T` (e.g. `fp16_t`). @@ -104,6 +110,7 @@ device.set_options(); TensorMatcher({N}) // .with_dtype() .with_device(device) + .ensure_alignment(16) // e.g. for 128-bit vectorized load/store .verify(dst) .verify(src); // same shape, dtype, device as dst const int64_t n = N.unwrap(); @@ -155,6 +162,7 @@ The counterpart to `tensor.h`: that one validates what came in, this one produce - `.store(ptr, offset)` — vectorized store to `ptr[offset]` - `.fill(value)` — fill all N elements with `value` - `operator[](i)` — element access +- **`device::LoadStoreBytes`** — named widths to pass around instead of bare integers: `MAX_GMEM` (arch-dependent, 32 on Blackwell), `MAX_SMEM` (16), `MAX_PORTABLE` (16, safe on CUDA and HIP), `MIN_COALALESCED` (4), and `RAW_1B` .. `RAW_32B`. ### `tile.cuh` — `tile::Memory` (strided memory access pattern) @@ -164,12 +172,22 @@ The counterpart to `tensor.h`: that one validates what came in, this one produce - `tile::Memory` is fundamentally a **1D cooperative accessor** over a contiguous region. - **`device::tile::Memory::cta(blockDim.x)`** — Creates a tile accessor where each thread handles `tid = threadIdx.x` with stride `tsize` (for `cta(blockDim.x)`, this is `blockDim.x`). Common for loops over a 1D array. +- **`device::tile::Memory::warp()`** — the 32-lane flavour, and what `warp::load_bytes` is built on. It takes its lane index from `get_lane_id()` rather than `threadIdx.x % 32`, for the addressing reason described under `utils.cuh`. Use `warp(int n)` for a narrower sub-group; that overload keeps the modulo, since a hardware lane index is the wrong grouping below 32. +- **`device::tile::Memory::thread()`** — single-thread accessor, no cooperation. - **`.load(ptr, offset)`** — loads `ptr[tid + offset * tsize]` - **`.store(ptr, val, offset)`** — stores to `ptr[tid + offset * tsize]` - **`.in_bound(n, offset)`** — boundary check For a **2D tile**, either flatten `(row, col)` into a linear tile index first, or compute the address manually with `ptr[row * stride + col]` using your thread/block coordinates. +### `bits.h` — Compile-time bit helpers (`host::`) + +```cpp +#include +``` + +`constexpr` wrappers over ``, usable in `static_assert` and in template arguments: `host::is_pow2(x)`, `log2_floor(x)`, `log2_ceil(x)` (both `-1` for `x == 0`), `round_up_pow2(x)`, `round_down_pow2(x)`. They live in `host::` but are constant-expression-only, so device-side `static_assert` can call them. + ### `math.cuh` — Device math (`device::math::`) ```cpp @@ -185,8 +203,19 @@ For a **2D tile**, either flatten `(row, col)` into a linear tile index first, o #include ``` -- `device::warp::reduce(value, active_mask)` — generic warp reduction via `__shfl_xor_sync`. `Op` is a `device::ReductionOp` (`SUM`/`MAX`/`MIN`); `kNumThreads` is a power-of-two group size (default 32 = full warp); `kInner=true` (default) reduces within each `kNumThreads`-sized group, `kInner=false` reduces across groups (lanes at the same offset in different groups). -- `device::warp::reduce_sum/reduce_max/reduce_min(value)` — convenience wrappers over `reduce`. Work for any type with a `ReductionTrait`: floats, integers, and packed x2 types. +**Vectorized row copy — reach for this before hand-rolling.** A warp cooperatively moving one contiguous row is the single most common shape in this tree, and it used to be re-implemented per kernel. + +- **`warp::load_bytes(src)`** / **`warp::store_bytes(dst, val)`** — the whole warp moves `kBytes` from/to one contiguous row. `kBytes` is arbitrary, down to 1; the helper picks the widest vector that divides it and handles the ragged tail. The returned value is opaque and encodes the width, so a load and its matching store **must use the same ``**. +- **`warp::LoadStorePattern`** — the `kPattern` values, and the sign carries the meaning. A **negative** one (`WARP_UNIFORM_16B` and friends: `_GMEM`, `_SMEM`, `_4B`/`_8B`/`_32B`) says *the whole warp splits this row*, so the width is derived from the per-lane share (`gcd(kBytes / 32, |kPattern|)`), falling back to `gcd(kBytes, 4)` when the row is too narrow to give every lane 4 bytes. A **positive** one (a bare `LoadStoreBytes` value) makes no warp-splitting assumption and just caps the per-thread vector at `gcd(kBytes, kPattern)`. `WARP_UNIFORM_16B` is the usual choice for a row copy. +- **`LoadStorePattern::get_vec_bytes()`** — the width actually chosen. `SGL_DEVICE_HOST`, so **the launcher must call this to feed `.ensure_alignment(...)`**. Pass it the same value the kernel passes `load_bytes` — for a split row that is the *per-warp share*, not the whole row; the full row can resolve to a different width and would under-constrain the strides. + +**Reductions.** + +- `warp::reduce(value, active_mask)` — `Op` is a `device::ReductionOp` (`SUM`/`MAX`/`MIN`). `kStart` and `kFinish` bound a range of lane-index bits: `` reduces within contiguous groups of `N` lanes, `` reduces *across* groups at the same offset. The operation is **symmetric** — `` and `` reduce the same lane set. +- `warp::reduce_sum/reduce_max/reduce_min(value)` — wrappers. Work for any type with a `ReductionTrait`: floats, integers, packed x2. This is more widely used in practice than generic `reduce`. +- `warp::inclusive_reduce(val, lane_id, mask)` and `inclusive_sum` / `inclusive_max` / `inclusive_min` — segmented inclusive scan: every lane keeps its own running total, unlike `reduce`. Forward when `kStart < kFinish`, backward when `kStart > kFinish`. `lane_id` defaults to `get_lane_id()`, which is correct by construction — only pass it if you already have the **segment-relative** index (`threadIdx.x % kWidth`); a warp-relative one silently corrupts every segment past the first when `kWidth < 32`. +- `warp::broadcast(value, src_lane)` — `src_lane` is **segment-relative**: each `kWidth` segment reads its own lane, not one warp-wide source. +- `warp::get_lane_id`, `warp::elect_one_lane()` (one elected lane, for gating a single-thread TMA issue; CUDA-only), `warp::kFullMask` and `warp::kFullWidth` (32 on CUDA, 64 on HIP — note this differs from `kWarpThreads`). ### `cta.cuh` — CTA-level primitives @@ -203,6 +232,11 @@ For a **2D tile**, either flatten `(row, col)` into a linear tile index first, o ``` - `device::atomic::max(float* addr, float value)` — float atomic max (handles negative values correctly via bit tricks). +- **`device::atomic::Event`** — a cross-CTA arrive/wait counter in one 32-bit word. Producers call `arrive()`; consumers call `wait(num_producers)` (exactly one consumer) or `wait_multi(num_producers, num_consumers, n)` (several). The last released consumer resets the word, so one `Event` is reusable across launches with no host re-zero. Preconditions, all of them load-bearing: + - the word must be **zeroed by the host** before first use, and must live in **global** memory — a shared or local `Event` is an illegal address; + - `arrive()` is a release and the waits are acquires, so a producer's writes before `arrive()` are visible after the wait; + - **generations must be explicitly ordered** — the word has no phase bit, so overlapping generation N+1 with a consumer still in generation N is undefined behaviour (a lagging consumer never exits); + - `wait()` and `wait_multi()` lay the word out incompatibly; mixing them on one `Event` is undefined behaviour. ### `runtime.cuh` — Occupancy and device info @@ -211,8 +245,10 @@ For a **2D tile**, either flatten `(row, col)` into a linear tile index first, o ``` - `host::runtime::get_blocks_per_sm(kernel, block_dim)` — max active blocks per SM (occupancy) -- `host::runtime::get_sm_count(device_id)` — number of SMs on the device -- `host::runtime::get_cc_major(device_id)` — compute capability major version +- `host::runtime::get_sm_count(device_id [, use_cache])` — number of SMs on the device. Memoized per device ordinal, so it is cheap enough to call on a launch path; pass `use_cache=false` to force a driver query. +- `host::runtime::get_cc_major` / `get_cc_minor` / `get_sm_version` — compute capability, same caching. + +**Do not query the architecture at runtime.** The JIT compiles for the exact local GPU, so the arch is a *compile-time* fact: `SGL_CUDA_ARCH` is injected by `load_jit`, and `SGL_ARCH_HOPPER_OR_GREATER` / `SGL_ARCH_BLACKWELL_OR_GREATER` / `device::kMaxVecBytes` are derived from it. Reach for `get_cc_*` only for something the arch genuinely does not determine. SM *count* is the opposite case — it varies within an arch (B200 vs B300 vs a MIG slice), so it has to stay a runtime query. **Persistent kernel pattern** (cap blocks to SM count × occupancy): ```cpp @@ -586,7 +622,7 @@ Benchmarks use the project's own `marker` framework (in `python/sglang/kernels/j - `graph_clone_args` / `graph_clone_kwargs`: which inputs to clone per CUDA-graph iteration to defeat L2 cache reuse. Defaults to `"all"` — pass an iterable of indices/keys to limit to the *read* args (writes don't need cloning). - `use_cuda_graph=False` for kernels that can't be captured. - `metrics=(0.5, "avg")` controls reported quantiles (the first metric becomes the table latency column). - - `disable_log_bandwidth` (defaults from `SGLANG_KERNEL_DISABLE_LOG_BANDWIDTH=1`) skips the bandwidth column entirely. + - `disable_log_bandwidth` (defaults from `SGLANG_JIT_BENCHMARK_DISABLE_LOG_BANDWIDTH=1`) skips the bandwidth column entirely. - **`utils.create_random(*shape)` / `utils.create_empty(*shape)`** — shorthand for `torch.randn` / `torch.empty` with `DEFAULT_DTYPE` (`bfloat16`) and `DEFAULT_DEVICE` (`"cuda"`). Override via the `dtype=` / `device=` kwargs. - **`utils.get_benchmark_range(full_range, ci_range)`** — returns the smaller `ci_range` under CI (`is_in_ci()`), the `full_range` locally. Still available for the `benchmark(...)` column axis (which has no `ci_vals`); for `parametrize` row axes prefer the built-in `ci_vals` argument. @@ -640,7 +676,7 @@ if __name__ == "__main__": - The `line_arg` name passed to `benchmark` (`"impl"` here) must match a parameter on `benchmark(...)`; same for every `parametrize` name (`"size"`). - Stack `@parametrize` once per swept axis. The required `@marker.benchmark` is the **innermost** decorator (bottom of the stack, directly above the function) — `@parametrize` rows go above it. - Prefer `create_random` / `create_empty` from `utils.py` over open-coding `torch.randn(..., dtype=..., device=...)`. -- The GB/s column appears by default (`memory_args="all"` + `memory_output="out"`). For memory-bound kernels it's the most informative number; scope `memory_args` / `memory_output` to the tensors actually touched if the defaults over- or under-count. For compute-bound kernels where bandwidth is misleading, set `SGLANG_KERNEL_DISABLE_LOG_BANDWIDTH=1` (or `disable_log_bandwidth=True`). +- The GB/s column appears by default (`memory_args="all"` + `memory_output="out"`). For memory-bound kernels it's the most informative number; scope `memory_args` / `memory_output` to the tensors actually touched if the defaults over- or under-count. For compute-bound kernels where bandwidth is misleading, set `SGLANG_JIT_BENCHMARK_DISABLE_LOG_BANDWIDTH=1` (or `disable_log_bandwidth=True`). - For in-place kernels (which return `None`), pass the written tensors via `memory_output=(...)` since the `"out"` default would capture nothing. - Tune `graph_clone_args` / `graph_clone_kwargs` to all the arguments that might be read by the kernel. We can only skip cloning for write-only args. For in-place modified args, we still need to clone them to get accurate timing (reusing the same buffer keeps it L2-hot and skews results). - Call `benchmark.run()` (no `print_data=` kwarg — the marker framework prints directly). @@ -665,7 +701,7 @@ cd test && python3 run_suite.py --hw cuda --suite base-b-kernel-benchmark-test-1 - **JIT compilation fails**: ensure the `.cuh` file is under `python/sglang/kernels/jit/csrc/`; reduce template argument combinations - **CUDA crash / illegal memory access**: `CUDA_LAUNCH_BLOCKING=1`; `compute-sanitizer --tool memcheck python ...` - **Unstable benchmark results**: `marker.do_bench` uses CUDA-graph-based timing by default; set `use_cuda_graph=False` only if the kernel can't be captured. `graph_clone_args` defaults to `"all"`; if you narrow it, it must still cover every *read* tensor — reusing a single buffer keeps it L2-hot and skews results. Keep *write* tensors in it too: they are what sets the rotation count, and a shared output buffer stays L2-hot the same way. -- **Missing GB/s column**: the column is on by default; check that `SGLANG_KERNEL_DISABLE_LOG_BANDWIDTH` is not `1` and `disable_log_bandwidth` is not `True`. For in-place kernels (return `None`) the `memory_output="out"` default counts nothing — pass the written tensors via `memory_output=(...)` +- **Missing GB/s column**: the column is on by default; check that `SGLANG_JIT_BENCHMARK_DISABLE_LOG_BANDWIDTH` is not `1` and `disable_log_bandwidth` is not `True`. For in-place kernels (return `None`) the `memory_output="out"` default counts nothing — pass the written tensors via `memory_output=(...)` --- @@ -681,6 +717,7 @@ cd test && python3 run_suite.py --hw cuda --suite base-b-kernel-benchmark-test-1 - `python/sglang/kernels/jit/include/sgl_kernel/utils.cuh` — type aliases, `LaunchKernel`, `SGL_DEVICE` - `python/sglang/kernels/jit/include/sgl_kernel/vec.cuh` — `AlignedVector` - `python/sglang/kernels/jit/include/sgl_kernel/tile.cuh` — `tile::Memory` +- `python/sglang/kernels/jit/include/sgl_kernel/bits.h` — compile-time bit helpers - `python/sglang/kernels/jit/include/sgl_kernel/type.cuh` — `DTypeTrait`, `packed_t`, `device::cast`, `device::unpack`, `ReductionTrait` - `python/sglang/kernels/jit/include/sgl_kernel/math.cuh` — `device::math::` - `python/sglang/kernels/jit/include/sgl_kernel/warp.cuh` — `warp::reduce` and `reduce_sum/max/min` wrappers diff --git a/.claude/skills/babysit-pr-to-pass-ci/SKILL.md b/.claude/skills/babysit-pr-to-pass-ci/SKILL.md index 9afaa11a2243..350b4ac86eb4 100644 --- a/.claude/skills/babysit-pr-to-pass-ci/SKILL.md +++ b/.claude/skills/babysit-pr-to-pass-ci/SKILL.md @@ -30,7 +30,7 @@ Examples: $babysit-pr-to-pass-ci $babysit-pr-to-pass-ci 12345 $babysit-pr-to-pass-ci https://github.com/sgl-project/sglang/pull/12345 pr-test-extra.yml -$babysit-pr-to-pass-ci 12345 --only pr-test-amd-rocm720.yml +$babysit-pr-to-pass-ci 12345 --only pr-test-amd.yml ``` ## Start or continue the durable goal diff --git a/.claude/skills/ci-test-audit/SKILL.md b/.claude/skills/ci-test-audit/SKILL.md new file mode 100644 index 000000000000..aff5247436bb --- /dev/null +++ b/.claude/skills/ci-test-audit/SKILL.md @@ -0,0 +1,40 @@ +--- +name: ci-test-audit +description: Audit the existing test tree and CI configuration for improvements, using a catalog of patterns previously applied in this repo. Use when asked to audit tests or CI, shrink CI time or cost, find redundant or misplaced tests, clean up a test group, or review whether a CI change follows established practice. +--- + +# CI / Test Audit + +Audits what is already in `test/`, `.github/` and `scripts/ci/`. For writing a new test see +[write-sglang-test](../write-sglang-test/SKILL.md); for how the pipeline dispatches and +gates work see [ci-workflow-guide](../ci-workflow-guide/SKILL.md). + +## How to use it + +Read [action-items.md](action-items.md) first. It is a catalog of patterns that have +been applied to this repo, each with a way to spot it and example PRs. The catalog is the substance of this skill; everything below is only scaffolding. + +The user says what to audit -- a test group, a stage, a workflow file, or a complaint +like "this suite is slow" -- and how thoroughly. Read that scope and decide which +patterns apply. + +An audit is only useful if it is specific. "This suite could be trimmed" is not a +finding; "`TestFooLargePage` is a strict subset of `TestFooRetractLargePage`, drop it" +is. Every finding needs the file, the pattern it matches, and what to do. + +## Reporting + +Group findings by confidence in the claim, not by pattern id. Lead with the ones where +the evidence is in the file you just read; keep the speculative ones separate and say +what you would need to check. Skip anything you are not reasonably sure about -- a long +list of maybes costs more to triage than it saves. + +Propose, do not apply. Deleting a test, moving a registration to another stage, and +changing a threshold are all decisions for the user. Land them as separate changes, +since a trim and a threshold change fail for different reasons. + +## Keeping the catalog current + +Example PRs age. When a pattern shows up in newer work, or a new pattern recurs, update +`action-items.md`: the pattern text should stay general, the examples are +replaceable evidence. diff --git a/.claude/skills/ci-test-audit/action-items.md b/.claude/skills/ci-test-audit/action-items.md new file mode 100644 index 000000000000..451d0d78ff29 --- /dev/null +++ b/.claude/skills/ci-test-audit/action-items.md @@ -0,0 +1,261 @@ +# CI / Test Action Items + +Patterns that have been applied to this repo's tests and CI. Each entry is a general +rule plus a way to spot it in this repo; the PR numbers are evidence, not definitions, +and are expected to be replaced as newer examples appear. + +Sections follow the order of questions an audit asks about a test: should it exist, is +it in the right place, does it test the right thing, is it stable. The remaining +sections are about the pipeline around the tests. + +--- + +## A. Should this test exist in this form? + +**A1. Every case answers "what future diff would turn this red?"; if the only answer is editing the test, delete it.** +The admission criteria in `.claude/rules/unit-test-admission.md` apply to existing cases as +much as new ones. Spot: a test for a code path or kernel that no longer exists; +assertions that only check a mock was called; a stress loop that cannot reproduce the +failure it claims to guard; a case whose every assertion is also made by another case in +the group. Not this pattern: the rule's own carve-outs -- an external-source literal, or +a completeness / negative-branch contract -- are bookkeeping, even when the code is a +one-liner; and a path that was renamed is not a path that no longer exists, so check +before declaring a test orphaned. Examples: #34464, #34667, #39013, #38881, #38259. + +**A2. A test earns its cost by what it protects; a guard on a standalone, rarely touched, rarely used surface can be deleted outright.** +Tests exist to stop other people's changes from breaking code. That value scales with +how often the code changes, how many users hit it, and how entangled it is with other +paths. Spot: the module under test has had no non-test commits in months; it has few +importers; the feature sits behind a flag almost nobody enables or serves one model few +people run. The heavier the test (server launch, multi-GPU), the higher this bar. +Not this pattern: low churn alone. A stable, load-bearing path (cache eviction, PD +transfer) is where a regression hurts most; all three signals -- little churn, few +users, little entanglement -- must hold before deleting on this ground. +Examples: none applied on this ground yet; #37990 (cases for models nobody runs) and +#38011 (low-signal model tests demoted from the PR gate) are partial. + +**A3. A variant whose configuration is a strict subset of another variant's is redundant.** +Spot: several classes in one file whose launch configs differ by a single flag or env +var. Holds when three things are true: the configs differ by one dimension; that +dimension is additive (it adds load or a knob position rather than gating a path or skipping +assertions); and the heavier variant's assertions cover the lighter one's. Watch for +variants that override a test method, and for looser expectations in the heavier variant +(a longer timeout or relaxed bound), which make the lighter one still worth keeping. +Not this pattern (and this one has the highest false-positive rate of the catalog): +sharing a helper or fixture, which does not make one variant a subset of another; a +variant whose flipped flag is itself the feature under test (e.g. an overlap-loading +toggle), which fails the additive condition; variants that drive different integration +layers of the same model (engine input format vs. server endpoint), so neither's +assertions cover the other's; or a lighter class that is skip-decorated and launches +nothing, so it costs nothing to keep and its skip is an A6 question instead. +Examples: #38093, #39544, #33745, #33586, #34070, #33763, #34464, #33752, #39013, #34882. + +**A4. Classes that can run against one server configuration share the launch.** +Spot: several classes in a group each launching a server in `setUpClass` with the same +or request-equivalent arguments. Launch cost usually dominates the cases themselves, so +this saves more than the case count suggests. Not this pattern: a flag that changes +server state (attention backend, page size, speculative decoding on or off) needs its +own launch; and sharing couples the classes, so one crash takes the others down with it. +Examples: #33641, #33756, #33944, #36736, #38014, #37252. + +**A5. An end-to-end matrix whose only varying dimension is one layer belongs in a layer-level unit test.** +Spot: a full server launch matrix that exists to select a kernel or backend. Not this +pattern: the one end-to-end smoke per backend that stays -- the unit test covers the +numerics, but backend selection can still change integration (graph capture, memory pool +layout). Examples: #33596, #33611. + +**A6. Skipped, disabled, or unreachable registrations are dead coverage; each carries a reason and an expiry, or is deleted.** +Spot: `disabled=` registrations or `skipTest` calls with no reference to what would +un-skip them; skips older than the issue they were waiting on; registrations naming a +retired `runner_config`; registered files whose TestCase classes never execute. Not +this pattern: a hardware-conditional skip (`skipUnless` on compute capability), which is +a gate; or a `disabled=` whose linked issue is still open, which is doing its job. +Examples: #32324, #39368, #38585, #34377, #33772, #34779, #33654, #34070. + +--- + +## B. Is it in the right place, at the right cost? + +**B1. A test runs on the stage and runner its resources actually require.** +Spot: CPU-only tests registered to a GPU `runner_config`; multi-GPU registrations for a +test that launches one server. Not this pattern: a test that does not use a GPU but +imports a CUDA-only module or `sgl_kernel` at import time; check before moving it. +Examples: #34074, #33654, #33605, #34913, #35220. + +**B2. Trigger frequency matches signal per unit of cost: per-commit for fast, high-signal checks; nightly for model-level accuracy and perf; weekly or a shadow cadence for slow-moving coverage.** +Spot: a multi-GPU accuracy suite registered per-commit whose failures have never been +PR-specific; a nightly job that has been green for weeks, occupies several GPUs per run, +and whose coverage a weekly run would hold just as well; a retired platform still +running daily. Demoting is not +deleting -- the nightly and weekly grids exist to hold this coverage at the right cost. +Examples: #38011, #37990, #33809, #36814, #37532, #34204, #37586, #31749. + +**B3. Declared cost matches measured cost; shard counts and timeouts derive from it.** +Spot: an `est_time` off from the measured run by an order of magnitude (the partitioner +packs shards with it when live stats are missing); a shard count or job timeout that has +not moved while the suite's contents have. Not this pattern: small drift on a file that +already has live stats, which override `est_time`. Examples: #35407, #36242, +#38238, #32408, #37435, #37452, #39194, #37532. + +--- + +## C. Does it test the right thing? + +**C1. Fixtures go through the real production path; the oracle is written independently.** +The logic that builds a fixture should stay inside the tested surface, and the reference +value must not come from the implementation under test, or a shared bug makes both sides +agree. Spot: weights copied into parameters by hand instead of through the weight +loader; a config assembled without the override entry points; literals that duplicate +what the model or config already declares; a mock missing a field the code under test +reads; a hand-built fake of an internal structure that has been patched in several +fix-up PRs, which is the signal to construct it through the real constructor or a +shared helper. Not this pattern: a deliberately minimal fake that isolates one unit -- +the signal is repeated patching, not the existence of a fake. The oracle is the one thing +that must stay independent; harnesses, fixtures and eval entry points are shared (C6). +Examples: #33615, #37339, #34100, #33509, #33179, #34746, #36424, #37148, #37182, +#38314, #38315, #38418, #39019, #39100, #39101. + +**C2. The test, not a default or the environment, decides which path it exercises.** +Spot: a test that relies on defaults to select an execution mode, capture range or memory +fraction, so a default change silently moves what is covered; a job-level env var that +changes which strategy a class exercises, so the class tests the wrong thing on every run +of that lane; a filter or setting leaking from one suite into the next; cached process +state that an env pin cannot override. Not this pattern: a test that exists to check the +defaults work; pinning it would delete that coverage. Examples: #34146, #33776, #33847, +#38221, #33772, #34300, #39411. + +**C3. The assertion matches the semantics under test.** +Spot: exit-code assertions on a process expected to abort; equality assertions where the +contract allows ties; an absolute tolerance on a value whose error scales with magnitude. +Examples: #34017, #37873, #32410, #34272, #35795. + +**C4. Thresholds come from a measured distribution; derived metrics are gated on the primary one.** +Spot: round-number thresholds with no recorded basis; a derived quantity asserted without +the accuracy result it depends on. Not this pattern: a deliberately loose bound whose job +is "the model did not fall apart", which is a design choice; flag the ones with no margin +or that keep being adjusted. Examples: #36570, #34145, #31702, #31748, #38725, #36290. + +**C5. Randomness is removed before a tolerance is loosened.** +Spot: a kernel test with unseeded random inputs whose bound was widened to make it pass; +a comparison that could be bit-exact but asserts within an epsilon. Seed the inputs, pin +the RNG, or make the guard exact. Not this pattern: a kernel that is itself +nondeterministic (atomic or split-K reduction order), where seeding does not help and a +tolerance is the correct assertion. Examples: #32126, #37343, #30026, #35787, #34356, +#34607. + +**C6. Shared test infrastructure has one implementation.** +Spot: the same eval, mixin or fixture reimplemented locally in several files, each with +its own threshold or drift. Not this pattern: a reference implementation written +independently inside a test, which C1 requires; this is about harnesses, not oracles. +Examples: #34477, #36979, #39906. + +--- + +## D. Is it stable? + +**D1. Resources are reused only after they are actually released, not after release was requested.** +Spot: a launch that follows a kill without waiting for the device memory to come back; a +teardown that sends SIGKILL without a graceful shutdown first; a container sharing the +host IPC namespace so `/dev/shm` leaks across jobs. Examples: #39545, #37194, #31871, +#32829, #36605, #37485, #38244, #38638. + +**D2. Each role in a test, and each concurrent run, has its own exclusive resources.** +Spot: two roles in one test sharing a device, NIC or port by default; concurrent runs +sharing a mount or namespace. Examples: #39584, #33044, #34755, #39611, #35500. + +**D3. Side artifacts do not decide the correctness verdict.** +Spot: a run that fails because a profile, metrics upload or perf dump was missing or +unreadable; profiling enabled in a run whose purpose is pass/fail. Examples: #33832, +#35489, #38629. + +--- + +## E. Pipeline structure + +**E1. Jobs and reruns come from one declarative definition, not hand-written or duplicated ones.** +Spot: a workflow where adding a runner means editing a matrix by hand; sharding sized by +a constant rather than by measured partition data; a rerun workflow that declares its own +image, deps or env instead of reusing the stage's. Examples: #34186, #33329, #31792, +#36257, #36775, #34324, #34195. + +**E2. An expensive artifact is built once per run and reused downstream.** +Spot: the same build or setup block appearing in several jobs of the same run; the same +compile happening in every job of a matrix. Examples: #33461, #33384, #33597, #33460, +#37258. + +**E3. Runner-pool availability is a single configuration switch, not a recurring workflow edit.** +Spot: the same job disabled and re-enabled by separate PRs, each touching the workflow +files, several times in a quarter. Examples: #31764, #32719, #35627, #35607, #36981, +#37522, #38770, #38842, #39044. + +**E4. Operational commands are gated on trust, scoped narrowly, and always answer.** +Spot: a command any commenter can trigger; a command that reruns far more than the caller +needs; an unrecognized command that is skipped silently; a reply that names the wrong +backend or run. Examples: #35750, #31980, #34057, #37618, #38734, #38736, #36778. + +**E5. Fail-fast cascades link only jobs whose failures are correlated.** +Spot: a scheduled or manually dispatched run whose later jobs are cancelled by an earlier +unrelated failure; another platform's lane cancelled by a CUDA failure. Examples: +#35392, #35238, #36146. + +**E6. Platform-specific CI triggers only on paths that affect it.** +Spot: a platform workflow that runs on every PR regardless of what changed; a lint stage +that runs a whole workspace's tests when none of its files moved. Examples: #36100, #34864. + +--- + +## F. Install and cache cost + +**F1. Build caches persist across jobs and runs, and every cache has a miss path.** +Spot: caches rebuilt per job; a cache miss that fails the job instead of falling back; a +cache key that does not change when its inputs do. Examples: #33361, #33619, #33460, +#34231, #33597, #35337, #33512, #32243. + +**F2. The environment is pinned: toolchain and critical dependencies are fixed, survive later installs, and the checkout is what runs.** +Spot: build steps with no explicit toolchain, so an image bump silently changes output; a +transitive dependency that a later `pip install` upgrades or downgrades; an image built +from whatever main was at build time rather than the workflow commit; a site-packages +copy of the package under test taking precedence over the working tree. Examples: +#33437, #37508, #34768, #34322, #33738, #33441. + +**F3. Repeated work inside the test process is cut.** +Spot: a checker that re-parses the whole package on every scan; a fixture that reloads a +tokenizer or model per case; independent subprocesses run serially; lint environments +rebuilt on every run. Examples: #36240, #36235, #36241, #37203, #34309. + +**F4. Work whose precondition already holds is skipped: installs, disk reclamation, cache refreshes.** +Spot: unconditional installs; a skip check whose matching is too strict to fire on real +package name variants; unconditional disk reclamation; cache-base refreshes on every ref. +Examples: #33757, #37689, #33637, #33644, #33904. + +**F5. A published package is preferred over a source clone; a required clone authenticates and retries.** +Spot: anonymous `git clone` in an install script; a dependency built from source that +ships a wheel. Examples: #37672, #37504, #37647, #34635. + +**F6. Build outputs match the platforms and ABIs of the runner fleet.** +Spot: single-architecture images consumed by heterogeneous runners; an extension built +for one ABI when jobs run on two. Examples: #34276, #34253, #33619, #37820. + +--- + +## G. Observability + +**G1. Job names and status surfaces state the configuration that actually ran.** +Spot: an image version or platform variant only discoverable in the log; a status block +that omits a lane the PR gate depends on. Examples: #35686, #34813, #34877. + +**G2. CI reporting carries queue time and runner health alongside pass/fail, and is shaped to be read.** +Spot: notifications that only say whether a run passed; a scheduled job missing from the +monitor that tracks its peers; a digest with no local timestamps, no filtering to +exceptions, and no trend over time. Examples: #37881, #33753, #37884, #38380. + +--- + +## H. Guardrails + +**H1. Every convention and every silent failure mode has a check that can fail.** +Spot: a rule that lives only in a review comment or a doc; a registered file whose tests +never run still reporting success; a suite whose job is never dispatched; a coverage job +that has failed on every run without anyone noticing; a sweep touching hundreds of test +files with no before/after registration count per backend. Silent non-execution is more +dangerous than a red test. Examples: #32735, #34074, #34147, #39697, #38581, #37436. diff --git a/.claude/skills/ci-workflow-guide/SKILL.md b/.claude/skills/ci-workflow-guide/SKILL.md index e660e93330bc..55aa382107e4 100644 --- a/.claude/skills/ci-workflow-guide/SKILL.md +++ b/.claude/skills/ci-workflow-guide/SKILL.md @@ -1,11 +1,11 @@ --- name: ci-workflow-guide -description: Guide to SGLang CI workflow orchestration — stage ordering, fast-fail, gating, partitioning, execution modes, and debugging CI failures. Use when modifying CI workflows, adding stages, debugging CI pipeline issues, or understanding how tests are dispatched and gated across stages. +description: Guide to SGLang CI workflow orchestration — stage ordering, fail-fast, gating, partitioning, execution modes, and debugging CI failures. Use when modifying CI workflows, adding stages, debugging CI pipeline issues, or understanding how tests are dispatched and gated across stages. --- # SGLang CI Workflow Orchestration Guide -This skill covers the CI **infrastructure** layer — how tests are dispatched, gated, and fast-failed across stages. For test authoring (templates, fixtures, registration, model selection), see the [write-sglang-test skill](../write-sglang-test/SKILL.md). +This skill covers the CI **infrastructure** layer — how tests are dispatched, gated, and aborted on failure across stages. For test authoring (templates, fixtures, registration, model selection), see the [write-sglang-test skill](../write-sglang-test/SKILL.md). --- @@ -24,9 +24,10 @@ This skill covers the CI **infrastructure** layer — how tests are dispatched, | `.github/workflows/pr-test.yml` | Main workflow — all stages, jobs, conditions, matrix definitions | | `.github/workflows/pr-test-extra.yml` | Extra workflow — gated by BOTH `run-ci` and `run-ci-extra` labels | | `.github/workflows/pr-gate.yml` | PR gating: draft check, `run-ci` label, per-user rate limiting | -| `.github/actions/check-pr-test-health/action.yml` | Cross-job fast-fail: queries API for any failed job | +| `.github/actions/check-pr-test-health/action.yml` | Cross-job fail-fast: queries API for any failed job | | `.github/actions/wait-for-jobs/action.yml` | Stage gating: polls API until stage jobs complete | | `.github/actions/check-maintenance/action.yml` | Maintenance mode check | +| `.github/scripts/ci-labels.cjs` | Resolves the four CI control labels into dispatch axes | | `test/run_suite.py` | Suite runner: collects, filters, partitions, executes tests | | `python/sglang/test/ci/ci_register.py` | Test registration (AST-parsed markers), LPT auto-partition | | `python/sglang/test/ci/ci_utils.py` | `run_unittest_files()`: execution, retry, continue-on-error | @@ -113,21 +114,21 @@ This skill covers the CI **infrastructure** layer — how tests are dispatched, └─────────────────────────────────────┘ ``` -**Every stage test job** includes a `check-pr-test-health` step after checkout — if any job in the run has already failed, the job fast-fails (red X) with a root cause annotation. +**Every stage test job** includes a `check-pr-test-health` step after checkout — if any job in the run has already failed, the job fails fast (red X) with a root cause annotation. -**Scheduled runs** skip `wait-for-base-*` jobs, running all stages in parallel. Fast-fail is also disabled. +**Scheduled runs** skip `wait-for-base-*` jobs, running all stages in parallel. Fail-fast is also disabled. --- -## Fast-Fail Layers +## Fail-Fast Layers -4 layers of fast-fail, from fine to coarse: +4 layers of fail-fast, from fine to coarse: | Layer | Mechanism | Granularity | Disabled on schedule? | |-------|-----------|-------------|----------------------| | **1. Test method → file** | `unittest -f` (failfast) | One test method fails → entire test file stops immediately | Yes | | **2. File → suite** | `run_unittest_files()` default | One test file fails → entire suite stops (`--continue-on-error` off) | Yes | -| **3. Job → job (same stage)** | `check-pr-test-health` action | One job fails → other waiting jobs in same stage fast-fail (red X) | Yes | +| **3. Job → job (same stage)** | `check-pr-test-health` action | One job fails → other waiting jobs in same stage fail-fast (red X) | Yes | | **4. Stage → stage (cross-stage)** | `wait-for-base-*` + `needs` | Base A fails → base B/C jobs skip entirely (never get a runner) | Yes (wait jobs skipped) | - **Layer 1**: `-f` flag appended to all `python3 -m pytest` / `unittest` invocations in `ci_utils.py` @@ -142,12 +143,17 @@ This skill covers the CI **infrastructure** layer — how tests are dispatched, | Aspect | PR (`pull_request`) | Scheduled (`cron`, every 6h) | Manual dispatch (`workflow_dispatch`) | |--------|---------------------|------------------------------|--------------------------------------| | **Stage ordering** | Sequential: A → B → C via `wait-for-base-*` | Parallel (all at once) | Single target stage only | -| **Cross-job fast-fail** | Yes (`check-pr-test-health`) | Yes | Yes | +| **Cross-job fail-fast** | Yes (`check-pr-test-health`) | Yes | Yes | | **continue-on-error** | No (stop at first failure within suite) | Yes (run all tests) | No | | **Retry** | Enabled | Enabled | Enabled | -| **max_parallel** | 3 (default), 14 if `high priority` label | 14 | 3 (default), 14 if `high priority` | +| **max_parallel** | 3 (default), 14 if `max-concurrency` label | 14 | 3 (default), 14 if `max-concurrency` | | **PR gate** | Yes (draft, label, rate limit) | Skipped | Skipped | -| **Concurrency** | `cancel-in-progress: true` per branch | Queue (no cancel) | Isolated per stage+SHA | +| **Concurrency** | `cancel-in-progress: true` per PR | Queue (no cancel) | Isolated per stage+SHA | + +Four labels relax these limits for one PR: `bypass-fail-fast`, `parallel-stages`, +`max-concurrency`, and `highest-priority` (all three). `.github/scripts/ci-labels.cjs` +resolves them; the [contribution guide](https://docs.sglang.io/developer_guide/contribution_guide.html#ci-control-labels) +describes what each one does. --- @@ -158,7 +164,7 @@ This skill covers the CI **infrastructure** layer — how tests are dispatched, **How it works:** 1. Calls `listJobsForWorkflowRun` to list all jobs in the current run 2. Matches jobs by exact name or prefix (for matrix jobs, e.g., `base-b-test-1-gpu-small (3)`) -3. If any matched job has `conclusion === 'failure'` → fail immediately (fast-fail) +3. If any matched job has `conclusion === 'failure'` → fail immediately (fail-fast) 4. If all matched jobs are completed and count matches `expected_count` → success 5. Otherwise → sleep `poll-interval-seconds` (default: 60s) and retry 6. Timeout after `max-wait-minutes` (240 min for base-a, 480 min for base-b) @@ -179,7 +185,7 @@ This skill covers the CI **infrastructure** layer — how tests are dispatched, --- -## Cross-Job Fast-Fail (`check-pr-test-health` action) +## Cross-Job Fail-Fast (`check-pr-test-health` action) Composite action called after checkout in every stage test job (21 jobs total across `pr-test.yml`, `pr-test-multimodal-gen.yml`, `pr-test-sgl-kernel.yml`, `pr-test-jit-kernel.yml`). @@ -189,7 +195,7 @@ Composite action called after checkout in every stage test job (21 jobs total ac 3. If root cause failures found → calls `core.setFailed()` with the list of root cause job names 4. If none → does nothing (step succeeds) -**Cascade filtering**: When job A fast-fails due to health check, it also has `conclusion: failure`. Without filtering, job B would list both the original failure AND job A's fast-fail. The filter checks each failed job's `steps` array — if the failing step name contains `check-pr-test-health` or `Check PR test health`, it's excluded from the root cause list. +**Cascade filtering**: When job A fails fast due to the health check, it also has `conclusion: failure`. Without filtering, job B would list both the original failure AND job A's fail-fast. The filter checks each failed job's `steps` array — if the failing step name contains `check-pr-test-health` or `Check PR test health`, it's excluded from the root cause list. **Usage pattern:** ```yaml @@ -210,11 +216,11 @@ steps: **Visual effect**: Job shows **red X** (failure) with error annotation showing root cause job names. Subsequent steps are naturally skipped (default `if: success()` is false after a failed step). No per-step `if` guards needed. -**No stage filtering**: Checks ALL jobs in the run, not just the current stage. Any failure anywhere triggers fast-fail. +**No stage filtering**: Checks ALL jobs in the run, not just the current stage. Any failure anywhere triggers fail-fast. **Error message example:** ``` -Fast-fail: skipping — root cause job(s): base-b-test-1-gpu-small (0), base-b-test-1-gpu-small (1) +Fail-fast: skipping — root cause job(s): base-b-test-1-gpu-small (0), base-b-test-1-gpu-small (1) ``` --- @@ -388,6 +394,8 @@ group: pr-test-{event_name}-{branch}-{pr_sha}-{stage} | `/rerun-failed-ci` | Reruns failed jobs in the latest workflow run | | `/tag-and-rerun-ci` | Adds `run-ci` label + reruns failed | | `/tag-and-rerun-ci extra` | Adds both `run-ci` and `run-ci-extra` labels + reruns failed | +| `/run-full-ci` | Short form of `/tag-and-rerun-ci extra` (baseline + extra). Alias: `/rerun-full-ci` | +| `/run-extra-ci` | Adds both labels + reruns **only** `PR Test Extra`, leaving baseline runs alone. Alias: `/rerun-extra-ci` | | `/rerun-test [ ...]` | Reruns specific test file(s) via `rerun-test.yml`. A file arg containing a glob metacharacter (`*`, `?`, `[...]`) expands against `test/registered/` and the multimodal test dir to every matching `test_*.py` (e.g. `/rerun-test test_*backend*.py` — wrap in backticks so GitHub doesn't italicize the `*`); matches are deduped, grouped by dispatch shape, and can't carry a `::test` selector. No match → single ⛔ reply, nothing dispatched. Each reply echoes its originating command (`Results for …`) so concurrent commands stay distinguishable | | `/rerun-group [ ...]` | Expands registered test groups, then reuses `/rerun-test` | diff --git a/.claude/skills/clean-startup-log/SKILL.md b/.claude/skills/clean-startup-log/SKILL.md index c1b9e886f952..7f50390a2d8e 100644 --- a/.claude/skills/clean-startup-log/SKILL.md +++ b/.claude/skills/clean-startup-log/SKILL.md @@ -1,282 +1,105 @@ --- name: clean-startup-log -description: Clean up noisy startup warnings and spurious prints in SGLang server logs. Use when users ask to clean up unwanted warnings, deprecation messages, or third-party noise in the server startup output. +description: Audit SGLang startup logs, save evidence, and propose cleanup for user review. With no arguments, run Qwen3-8B at TP1 and TP2 plus gpt-oss-20b at TP1. disable-model-invocation: true --- -# Clean Up SGLang Server Startup Logs - -Goal: ensure the server startup log is clean and minimal, with no spurious warnings, deprecation messages, or unformatted prints from third-party libraries. - -## Workflow - -### 1. Launch a server and capture the log - -```bash -uv run sglang serve --model-path Qwen/Qwen3-8B 2>&1 | tee /tmp/startup_log.txt -``` - -Wait until the server prints `The server is fired up and ready to roll!`, then Ctrl-C. - -For TP>1 testing: -```bash -uv run sglang serve --model-path Qwen/Qwen3-8B --tp 2 2>&1 | tee /tmp/startup_log.txt -``` - -For MoE / hybrid-SWA models (e.g. gpt-oss), test separately — they exercise different code paths: -```bash -uv run sglang serve --model-path openai/gpt-oss-20b 2>&1 | tee /tmp/startup_log.txt -``` - -### 2. Compare against the clean reference log - -Read `/tmp/startup_log.txt` and compare it against the reference log at the bottom of this file. Identify lines that: - -- Do NOT have the `[timestamp]` or `[timestamp TPx]` logger prefix -- Contain `WARNING`, `deprecated`, `is deprecated`, or similar noise -- Are printed by third-party libraries (transformers, torchao, NCCL, Gloo, tqdm, etc.) -- Are duplicate/redundant with information already logged by SGLang -- Appear multiple times due to `ModelConfig` being constructed in multiple processes - -### 3. Classify each noisy line - -For each noisy line, determine: - -| Category | Action | -|----------|--------| -| **SGLang code using wrong API** | Fix the SGLang code (e.g., replace deprecated API with new one) | -| **SGLang code logging at wrong level** | Change log level (e.g., warning -> debug for non-actionable messages) | -| **Duplicated across processes** | Downgrade to debug — info logged in one process becomes noise in 3-4 | -| **Third-party lib prints at import time** | Suppress the logger or redirect stdout during that import | -| **C-level print from .so library** | Redirect fd 1 during the specific C call, or accept it if too invasive | -| **Real warning the user should see** | Keep it | - -### 4. Present findings before fixing - -List all noisy lines with their source and proposed fix. Ask the user to review before making changes. - -### 5. Apply fixes and verify - -After approval, apply fixes one at a time, re-launch the server, and verify each fix works. - -## Key Architecture: Why Logs Repeat - -`ModelConfig` is constructed **3-4 times** during startup across different processes: -1. Main process: `ServerArgs.__post_init__()` → `get_model_config()` → `ModelConfig()` -2. Scheduler subprocess: `Scheduler.init_model_config()` → `ModelConfig.from_server_args()` -3. Scheduler subprocess: `TpModelWorker._init_model_config()` → `ModelConfig.from_server_args()` -4. Main process: `TokenizerManager.init_model_config()` → `ModelConfig.from_server_args()` - -Similarly, `get_tokenizer()` is called **5 times** across processes: -1. `resolve_auto_parsers` (main) — `template_detection.py` -2. `Scheduler.init_tokenizer()` (scheduler subprocess) — `scheduler.py` -3. `DetokenizerManager` (detokenizer subprocess) — `detokenizer_manager.py` -4. `TpModelWorker.__init__()` (scheduler subprocess) — `tp_worker.py` -5. `TokenizerManager` (main) — `tokenizer_manager.py` - -Any `logger.info()` or `logger.warning()` in `ModelConfig.__init__()` or `get_tokenizer()` will appear 3-5 times. **Keep these at `logger.debug()`.** - -## Known Noise Sources and Fixes (from past sessions) - -### 1. torchao "Skipping import of cpp extensions due to incompatible torch version" - -- **Source:** `torchao/__init__.py` — printed via `logger.warning()` when torch version < 2.11.0 -- **Trigger:** `sglang/__init__.py` -> `_apply_hf_patches()` -> `_patch_removed_symbols()` -> `from transformers.models.llama import modeling_llama` -> deep import chain -> `transformers/quantizers/auto.py` -> `from .quantizer_torchao import TorchAoHfQuantizer` -> imports torchao -- **Fix:** In `hf_transformers_patches.py::_patch_removed_symbols()`, temporarily set the `torchao` logger level to `ERROR` around the `modeling_llama` import: - ```python - _torchao_logger = logging.getLogger("torchao") - _prev_level = _torchao_logger.level - _torchao_logger.setLevel(logging.ERROR) - try: - from transformers.models.llama import modeling_llama - finally: - _torchao_logger.setLevel(_prev_level) - ``` - -### 2. "`torch_dtype` is deprecated! Use `dtype` instead!" (PARTIALLY FIXED) - -- **Source:** `transformers/configuration_utils.py` — the `torch_dtype` property warns via `logger.warning_once()` -- **Trigger:** Model files accessing `config.torch_dtype` instead of `config.dtype` -- **Fix applied so far:** Only `models/gpt_oss.py` (lines 222, 471) — tested with `openai/gpt-oss-20b`. -- **Remaining files that still use `config.torch_dtype`** (fix each only after testing with the corresponding model): - - `models/bailing_moe.py` (line 302) - - `models/llada2.py` (line 313) - - `models/qwen3_next.py` (lines 192, 209) - - `models/qwen3_5.py` (line 245) - - `models/nano_nemotron_vl.py` (lines 79, 102, 284) - - `models/llava.py` (lines 732, 734-737) - - `model_loader/loader.py` (line 649) -- **Note:** `common.py` was already fixed in a prior session. If new model files are added with `config.torch_dtype`, the warning will reappear — grep for `\.torch_dtype` to find them. -- **Important:** Only change `config.torch_dtype` → `config.dtype` for models you have actually tested. The `dtype` property should return the same value, but verify per-model to avoid regressions. - -### 3. "`BaseImageProcessorFast` is deprecated" - -- **Source:** `transformers/utils/import_utils.py` — the lazy module `__getattr__` warns when `BaseImageProcessorFast` is accessed -- **Trigger:** `base_processor.py` and `ernie45_vl.py` have `from transformers import BaseImageProcessorFast` at top level. These are imported eagerly via `tokenizer_manager.py` -> `multimodal_processor.py` -> `base_processor.py`, even for non-multimodal models. -- **Fix:** Replace `from transformers import BaseImageProcessorFast` with `from transformers import BaseImageProcessor` and update all `isinstance(..., BaseImageProcessorFast)` checks to `isinstance(..., BaseImageProcessor)` - -### 4. "No platform detected. Using base SRTPlatform with defaults." - -- **Source:** `sglang/srt/platforms/__init__.py` — `logger.warning()` -- **Fix:** Change to `logger.debug()` — this is expected on machines without a platform plugin and not actionable. - -### 5. `NCCL version 2.27.7+cuda13.0` - -- **Source:** C-level print from `libnccl.so` during `ncclCommInitRank()` call -- **Status:** Accepted as-is. SGLang already logs the version via `sglang is using nccl==X.Y.Z`. The C-level print cannot be suppressed without redirecting stdout fd, which is too invasive. `NCCL_DEBUG=WARN` does not suppress it in NCCL 2.27+. - -### 6. `[Gloo] Rank X is connected to Y peer ranks` - -- **Source:** C++ Gloo library print during process group init -- **Status:** Accepted as-is. From C++ code inside PyTorch's Gloo backend. - -### 7. `torchao SyntaxWarning: invalid escape sequence` - -- **Source:** `torchao/quantization/quant_api.py` — a raw string with unescaped `\.` -- **Status:** Upstream torchao bug. Cannot fix from SGLang side. - -### 8. tqdm progress bars (e.g., `Multi-thread loading shards`, `Capturing batches`) - -- **Status:** These are expected and useful. They show progress during weight loading and CUDA graph capture. Keep them. - -### 9. CUTE_DSL "Unexpected error during package walk" — double-logged (FIXED) - -- **Source:** `nvidia-cutlass-dsl` package at `.venv/.../cutlass/cutlass_dsl/cutlass.py`, line 391. Logger named `CUTE_DSL` with its own `StreamHandler`. -- **Trigger:** During CUDA graph capture, cutlass DSL walks packages and hits an unexpected error for `cutlass.cute.experimental`. -- **Root cause of double-logging:** The CUTE_DSL logger has `propagate=True` (default), so the warning is emitted by both the CUTE_DSL handler (with its format) and the root logger (SGLang's format). -- **Fix applied:** In `entrypoints/engine.py`, changed `CUTE_DSL_LOG_LEVEL` from `"30"` (WARNING) to `"40"` (ERROR). This suppresses the WARNING at both the CUTE_DSL logger and root propagation levels. The env var controls both `logger.setLevel()` and `console_handler.setLevel()` in cutlass's `setup_log()`. - -### 10. ModelConfig init logs repeated 3x (FIXED) - -- **Lines:** `"Downcasting torch.float32 to ..."`, `"Hybrid swa model: ..."`, `"DeepGemm is enabled but ..."` -- **Source:** `configs/model_config.py` — `_get_and_verify_dtype()` (line 1457), `_derive_hybrid_model()` (line 497), `_verify_quantization()` (line 1236) -- **Root cause:** `ModelConfig.__init__()` is called 3-4 times in different processes (see "Key Architecture" above). Each construction fires the same log lines. -- **Fix applied:** Downgraded all three from `logger.info()`/`logger.warning()` to `logger.debug()`. The dtype is already visible in `server_args` and `Load weight end`. Hybrid SWA info appears in `Tree cache initialized`. DeepGemm is not actionable. - -### 11. Tokenizer retry/fallback messages repeated 3-4x (FIXED) - -- **Lines:** `"Tokenizer loaded as generic TokenizersBackend ... retrying"`, `"Loading tokenizer ... directly as PreTrainedTokenizerFast"`, `"Tokenizer for ... loaded as generic TokenizersBackend. Set --trust-remote-code"` -- **Source:** `utils/hf_transformers/tokenizer.py` — `_resolve_tokenizers_backend()` (line 215), `_load_tokenizer_by_declared_class()` (line 110), final warning (line 244) -- **Root cause:** 5 separate `get_tokenizer()` calls across processes (see "Key Architecture" above). Each produces 3 log lines. Concurrent subprocess launches cause interleaved/doubled output. -- **Fix applied:** Downgraded all three from `logger.warning()`/`logger.info()` to `logger.debug()`. - -### 12. Template detection logs — 5 lines consolidated to 1 (FIXED) - -- **Lines:** `"Detected reasoning config '...' from template rule '...'"`, `"Detected reasoning parser '...' from template rule '...'"`, `"Detected tool-call parser '...' from template rule '...'"`, `"Auto-detected reasoning parser: ..."`, `"Auto-detected tool-call parser: ..."` -- **Source:** `managers/template_detection.py` (lines 337, 370) logged each detection rule match. `managers/template_manager.py` (lines 177-182) logged summary lines that duplicated the detection logs. -- **Fix applied:** Removed per-rule logs from `template_detection.py`. Consolidated the 5 lines in `template_manager.py` into a single summary: `"Auto-detected template features: reasoning_config=..., reasoning_parser=..., tool_call_parser=..."` - -### 13. KV cache dtype logged separately from allocation (FIXED) - -- **Lines:** `"Using KV cache dtype: torch.bfloat16"` then `"KV Cache is allocated. #tokens: ..., K size: ..., V size: ..."` -- **Source:** `model_executor/model_runner.py` (line 2217) and `mem_cache/memory_pool.py` (line 740) -- **Fix applied:** Removed the standalone dtype log from `model_runner.py`. Added `dtype` field to the allocation log in `memory_pool.py`: `"KV Cache is allocated. dtype: torch.bfloat16, #tokens: ..., K size: ..., V size: ..."` - -### 14. CUTLASS backend warning — B200 → SM100, warning → info (FIXED) - -- **Line:** `"CUTLASS backend is disabled when piecewise cuda graph is enabled due to TMA descriptor initialization issues on B200."` -- **Source:** `layers/attention/flashinfer_backend.py` (line 249) -- **Fix applied:** Changed "B200" to "SM100 GPUs" (the condition checks `is_sm100_supported()` which matches SM10x, not just B200). Downgraded from `logger.warning()` to `logger.info()` since it's an expected automatic fallback. - -### 15. `max_total_num_tokens` and `Tree cache initialized` log ordering - -- **Issue:** `max_total_num_tokens=...` appears before `Tree cache initialized:...` even though tree cache is conceptually part of memory setup. -- **Root cause:** `max_total_num_tokens` is logged inside `init_model_worker()` (scheduler.py:972), which runs before `build_kv_cache()` (scheduler.py:425) where tree cache is created. -- **Status:** Not fixed — reordering was reverted. Acceptable as-is. - -### 16. `Ignore import error when loading sglang.srt.models.midashenglm` - -- **Source:** `models/registry.py` (line 109) — `logger.warning()` during `import_model_classes()` which iterates all model modules via `pkgutil.iter_modules` -- **Trigger:** The `midashenglm` model depends on `torchaudio`, which fails to load -- **Status:** Should be downgraded to `logger.debug()` — not actionable when loading an unrelated model. Same pattern exists in `managers/multimodal_processor.py`, `dllm/algorithm/__init__.py`, `multimodal_gen/runtime/models/registry.py`. - -### 17. `Multiple NUMA nodes found for GPU X` - -- **Source:** `utils/numa_utils.py` (line 112) — `logger.warning()` -- **Status:** Could be downgraded to `logger.info()`. The situation is handled gracefully ("Using the first one") and not actionable. - -### 18. Warmup `/model_info` access log - -- **Source:** Uvicorn access log, triggered by SGLang's own warmup at `entrypoints/http_server.py` (line 1877) -- **Status:** SGLang talking to itself. Could suppress uvicorn access logger during warmup, or exclude `/model_info` from warmup access logging. - -## Investigation Techniques - -### Trace what triggers an import -```python -import sys -_real_import = __builtins__.__import__ -def _tracing_import(name, *args, **kwargs): - if 'TARGET_MODULE' in name: - import traceback - print(f'=== Importing {name} ===') - traceback.print_stack() - return _real_import(name, *args, **kwargs) -__builtins__.__import__ = _tracing_import -``` - -### Trace what triggers a logger warning -```python -import logging, traceback -class TraceHandler(logging.Handler): - def emit(self, record): - if 'SEARCH_STRING' in record.getMessage(): - traceback.print_stack() -h = TraceHandler() -h.setLevel(logging.WARNING) -logging.getLogger('TARGET_LOGGER_NAME').addHandler(h) -``` - -### Find C-level prints in .so files -```bash -strings /path/to/library.so | grep "SEARCH_STRING" -``` - -### Find all config.torch_dtype accesses (for deprecation warning) -```bash -grep -rn '\.torch_dtype' python/sglang/srt/models/ python/sglang/srt/model_loader/ python/sglang/srt/utils/hf_transformers/ -``` - -## Reference: Clean Startup Log (TP=1, Qwen3-8B) - -``` -[2026-05-24 00:52:39] Attention backend not specified. Use trtllm_mha backend by default. -[2026-05-24 00:52:39] TensorRT-LLM MHA only supports page_size of 16, 32 or 64, changing page_size from None to 64. -[2026-05-24 00:52:40] server_args=ServerArgs(model_path='Qwen/Qwen3-8B', ...) -[2026-05-24 00:52:40] Multiple NUMA nodes found for GPU 0: [...]. Using the first one. -[2026-05-24 00:52:42] Using default HuggingFace chat template with detected content format: string -[2026-05-24 00:52:42] Auto-detected template features: reasoning_config=..., reasoning_parser=qwen3, tool_call_parser=qwen -[2026-05-24 00:52:50] Init torch distributed begin. -[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 -[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 -[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 -[2026-05-24 00:52:50] Init torch distributed ends. elapsed=0.21 s, mem usage=0.10 GB -[2026-05-24 00:52:51] Load weight begin. avail mem=275.75 GB -[2026-05-24 00:52:51] Found local HF snapshot for Qwen/Qwen3-8B at ...; skipping download. -Multi-thread loading shards: 100% Completed | 5/5 [00:01<00:00, 2.62it/s] -[2026-05-24 00:52:54] Load weight end. elapsed=2.62 s, type=Qwen3ForCausalLM, avail mem=260.48 GB, mem usage=15.28 GB. -[2026-05-24 00:52:54] KV Cache is allocated. dtype: torch.bfloat16, #tokens: 1707904, K size: 117.28 GB, V size: 117.28 GB -[2026-05-24 00:52:54] Memory pool end. avail mem=25.28 GB -[2026-05-24 00:52:54] CUTLASS backend is disabled when piecewise cuda graph is enabled due to TMA descriptor initialization issues on SM100 GPUs. Using auto backend instead for stability. -[2026-05-24 00:52:54] Capture cuda graph begin. This can take up to several minutes. avail mem=24.16 GB -[2026-05-24 00:52:54] Capture cuda graph bs [1, 2, 4, ...] -Capturing batches (bs=1 avail_mem=23.56 GB): 100% | 52/52 [00:05<00:00, 10.36it/s] -[2026-05-24 00:53:00] Capture cuda graph end. Time elapsed: 5.38 s. mem usage=0.60 GB. avail mem=23.56 GB. -[2026-05-24 00:53:00] Capture piecewise CUDA graph begin. avail mem=23.56 GB -[2026-05-24 00:53:00] Capture cuda graph num tokens [4, 8, 12, ...] -Compiling num tokens (num_tokens=4): 100% | 74/74 [00:09<00:00, 7.44it/s] -Capturing num tokens (num_tokens=4 avail_mem=21.24 GB): 100% | 74/74 [00:07<00:00, 10.44it/s] -[2026-05-24 00:53:18] Capture piecewise CUDA graph end. Time elapsed: 18.18 s. mem usage=2.32 GB. avail mem=21.24 GB. -[2026-05-24 00:53:20] Tree cache initialized: source=default impl=RadixCache hybrid_swa=False hybrid_ssm=False hierarchical=False streaming_wrapped=False -[2026-05-24 00:53:20] max_total_num_tokens=1707904, chunked_prefill_size=16384, max_prefill_tokens=16384, max_running_requests=4096, context_len=40960, available_gpu_mem=21.24 GB -[2026-05-24 00:53:20] INFO: Started server process [1964249] -[2026-05-24 00:53:20] INFO: Waiting for application startup. -[2026-05-24 00:53:20] Using default chat sampling params from model generation config: {'temperature': 0.6, 'top_k': 20, 'top_p': 0.95} -[2026-05-24 00:53:20] INFO: Application startup complete. -[2026-05-24 00:53:20] INFO: Uvicorn running on http://127.0.0.1:30000 (Press CTRL+C to quit) -[2026-05-24 00:53:21] Prefill batch, #new-seq: 1, #new-token: 64, ... -[2026-05-24 00:53:21] INFO: 127.0.0.1:... - "POST /generate HTTP/1.1" 200 OK -[2026-05-24 00:53:21] The server is fired up and ready to roll! -``` - -Note: `[Gloo]` messages and tqdm progress bars are acceptable. The key is no warnings or deprecation messages from transformers, torchao, or other third-party libraries. The `CUTLASS backend is disabled` message is now `info` level, not a warning. +# Audit SGLang Startup Logs + +The default outcome is saved logs and a findings report. Apply runtime changes +only after the user selects them. A request to edit this skill does not itself +launch servers. + +## Default runs + +A bare `$clean-startup-log` invocation runs these cases sequentially without +asking for commands. Explicit commands, models, or TP sizes replace this matrix. + +| Case / log filename | Command | +|---|---| +| `qwen3-8b-tp1.log` | `uv run sglang serve --model-path Qwen/Qwen3-8B` | +| `qwen3-8b-tp2.log` | `uv run sglang serve --model-path Qwen/Qwen3-8B --tp 2` | +| `gpt-oss-20b-tp1.log` | `uv run sglang serve --model-path openai/gpt-oss-20b` | + +These cover dense, tensor-parallel, and MoE/hybrid sliding-window attention +startup. Reuse complete captures from the current audit when code and environment +have not changed. + +## Capture logs + +1. Check the checkout, free GPUs, and ports once. Use the requested command when + resources are free; otherwise select free GPUs with `CUDA_VISIBLE_DEVICES` and + an unused `--port`, recording the adjustments. Leave existing servers alone. +2. Create a unique directory with `mktemp -d /tmp/sglang-startup-audit-XXXXXX`. + Save raw stdout and stderr together in a separate log for each case, for + example with `set -o pipefail` and `COMMAND 2>&1 | tee LOG_PATH`. Record commands, + GPU IDs, ports, commit, relevant overrides, and readiness status. +3. Wait for `The server is fired up and ready to roll!`, then stop that server and + its workers before the next case. First runs can spend many minutes downloading + weights or compiling FlashInfer kernels; check download/compiler activity + before treating a quiet log as a hang. Preserve partial logs for failed or + stalled starts and report the last stage. Continue independent cases when possible. +4. Preserve the user's logging configuration, including `NCCL_DEBUG`. If NCCL + verbosity needs explaining, inspect relevant shell settings, `NCCL_CONF_FILE`, + and `/etc/nccl.conf`. Do not override intentional diagnostics or recommend + `NCCL_DEBUG=WARN` solely because the output is long. Avoid full environment dumps. + +## Investigate efficiently + +- Scan for deprecations, duplicate handler output, unrelated import failures, + unformatted prints, and unexpected warnings. Read representative excerpts and + counts instead of repeatedly dumping `server_args`, progress redraws, or NCCL + diagnostics. Normalize carriage returns for analysis only; preserve raw logs. +- Trace each candidate to its actual emitter with focused `rg` searches. Inspect + its log level: SGLang's formatter may omit severity. Group shared signatures + across cases and distinguish handler duplication from separate GPU/process calls. +- Repetition, WARNING severity, or a different third-party format alone does not + establish a cleanup need. Consider whether the message explains configuration, + progress, resource use, or an operational limitation. +- Consult [noise-source hints](references/noise-sources.md) only for a matching + signature or an unresolved emitter. Verify current code rather than trusting + historical line numbers, fix status, or assumptions about unrelated models. + +## Accepted output + +Preserve these reviewed messages unless the user requests a different policy: + +- NCCL diagnostics enabled by the user's environment or host configuration. +- NUMA permission warnings, including one check per GPU in TP runs. +- GPT-OSS MXFP4 backend-selection warnings and default page-size selection warnings. +- `Init Unified Radix Cache. Components: ... Tree Core: ...`, tree-cache summaries, + SWA allocation details, and per-rank memory/timing records. +- Useful progress bars, warmup HTTP access logs, uv synchronization messages, + isolated NCCL/Gloo startup lines, and one timestamped HF authentication warning. + +These can appear in a clean startup log. Do not repeatedly propose the declined +NUMA deduplication, backend/page-size level changes, or NCCL verbosity override. +Keep real operational warnings visible: for example, a Harmony vocabulary failure +can disable `/v1/responses` even when server readiness and `/generate` succeed. + +## Report before changing code + +Return a compact run table with readiness status and clickable raw-log links. +For each actual cleanup candidate, give an exact representative message, affected +cases/counts, source file/function, and specific proposed behavior. Distinguish +confirmed findings from suspicions and operational failures from logging noise. + +If there are no actionable cleanup findings, say the logs are clean and no +further cleanup is needed. Otherwise, ask which numbered changes to adopt and +wait for the user's selections before editing runtime code or preparing patches. +Honor existing approvals and declined items without asking again. + +## Apply selected changes + +- Batch compatible approved edits, then verify affected cases once. Repeat a + startup only for a new change, failure, or unresolved concern; do not relaunch + after every one-line edit. Save verification logs separately from baselines. +- Preserve useful warnings and application handlers. HF can warn during early + CLI model detection before `configure_logger()`, and spawned processes have + independent logger state. Keep `configure_hf_hub_logger()` in both + `suppress_noisy_warnings()` and `configure_logger()`; make repeated setup safe. +- Keep the legacy compiled-kernel cache migration notice at DEBUG. Avoid broad + library-level suppression or fd redirection for a narrow logging problem. +- Run relevant formatting and focused existing checks. Add tests only when they + verify meaningful behavior, not a log-level spelling. Report changes and + verification; create branches, commits, and PRs when requested. diff --git a/.claude/skills/clean-startup-log/references/noise-sources.md b/.claude/skills/clean-startup-log/references/noise-sources.md new file mode 100644 index 000000000000..f59d55f048dc --- /dev/null +++ b/.claude/skills/clean-startup-log/references/noise-sources.md @@ -0,0 +1,70 @@ +# Startup noise-source hints + +Read this only for an observed signature that needs investigation. Paths and +behaviors can change; search the current checkout and installed dependency before +proposing a fix. The accepted-output policy in `../SKILL.md` takes precedence. +These are investigation hints, not a list of changes to apply. + +## Common signatures + +| Observed output | Where to investigate | Decision to verify | +|---|---|---| +| HF warning printed in two formats | `utils/common.py`, HF logger handlers and propagation | A library handler plus root propagation can emit the same record twice. Keep one timestamped copy, including before full server logging setup. | +| `Skipping import of cpp extensions due to incompatible torch version` | `torchao/__init__.py`; imports through `hf_transformers_patches.py` and transformers quantizers | Determine whether torchao is needed for the requested model before proposing narrow import-time suppression. | +| `torch_dtype` is deprecated | Model code, loader, and HF utilities accessing `config.torch_dtype` | Prefer `config.dtype` where supported; change and verify only affected, tested models. | +| `BaseImageProcessorFast` is deprecated | Multimodal processor imports and type checks | Check the installed transformers replacement and compatible `isinstance` behavior. | +| `No platform detected. Using base SRTPlatform with defaults.` | Platform detection | Decide whether fallback is expected on this machine or indicates a missing plugin. | +| `Unexpected error during package walk` in CUTE_DSL | Installed cutlass package, its handler, root propagation, and `CUTE_DSL_LOG_LEVEL` | Inspect the actual exception and duplicate-handler path; do not assume the current log level or hide all warnings. | +| Repeated dtype, hybrid-model, or tokenizer fallback messages | Model-config and tokenizer construction in launcher, scheduler, and detokenizer | Multiple constructors may explain repetition. Retain meaningful per-process information and distinguish these from handler duplication. | +| Repeated template detection messages | `managers/template_detection.py` and `managers/template_manager.py` | Check whether a summary already contains the same information. | +| KV cache dtype logged separately from allocation | Model runner and memory pool | Check whether the allocation summary already carries dtype; retain useful allocation and SWA details. | +| CUTLASS backend disabled during graph capture | Attention backend selection | Expected fallback may be useful information; retain the reason and selected behavior. | +| `Ignore import error when loading ...` | Model, multimodal, or algorithm registries | Distinguish an unused optional model dependency from failure to load the requested model. | +| `Multiple NUMA nodes found for GPU ...` | `utils/numa_utils.py` | This differs from the accepted NUMA permission warning; evaluate its context separately. | +| `OpenAI Responses API (/v1/responses) disabled` | `entrypoints/http_server.py`, serving responses, and Harmony vocabulary loading | A real endpoint limitation should remain visible even if generation warmup succeeds. | +| `SyntaxWarning: invalid escape sequence` from a dependency | Installed package file named by the warning | Identify the upstream issue; avoid editing the installed package as a repository fix. | + +Repeated constructors and separate processes are not blanket reasons to downgrade +messages. Constructor counts and import paths vary by model and checkout. Likewise, +cache-summary ordering and native library formats need no change merely because +they differ from a historical reference log. + +## Targeted tracing + +Start with source searches. For example: + +```bash +rg -n 'SEARCH_STRING' python/sglang/srt/FOCUSED_DIRECTORY +rg -n '\.torch_dtype' python/sglang/srt/models/MODEL.py +``` + +If the emitter is clear but its call path is not, attach a temporary handler in an +isolated reproduction: + +```python +import logging +import traceback + + +class TraceHandler(logging.Handler): + def emit(self, record): + if "SEARCH_STRING" in record.getMessage(): + traceback.print_stack() + + +target = logging.getLogger("TARGET_LOGGER_NAME") +handler = TraceHandler() +target.addHandler(handler) +try: + reproduce_observed_warning() +finally: + target.removeHandler(handler) +``` + +For suspected native output, search only the identified shared library: + +```bash +strings /path/to/library.so | rg -F 'SEARCH_STRING' +``` + +Keep diagnostic instrumentation out of committed fixes and baseline logs. diff --git a/.claude/skills/cookbook-add-model/SKILL.md b/.claude/skills/cookbook-add-model/SKILL.md index c0d2e5988dbc..803ae11c7ede 100644 --- a/.claude/skills/cookbook-add-model/SKILL.md +++ b/.claude/skills/cookbook-add-model/SKILL.md @@ -164,8 +164,31 @@ least one real deployment or capability boundary. Put orthogonal runtime feature 4. **Fill `cells[]`** with the verified recipes from Phase 1 (replace every EXAMPLE cell; set `verified: true` only on tested combos), and `modelNames` with real HF slugs, `dockerImages` for your hw (use the Phase-1 tag, or default `lmsysorg/sglang:dev` — never - a guessed release; key by `hw`, or `hw|quant` when one quant on a shared GPU needs its own - image), `multiNodeHints` only for fabric-specific hw (e.g. gb200). + a guessed release; key by `hw`, or a more specific key — `hw|variant|quant`, `variant|quant`, + `hw|quant|strategy`, `hw|quant`, resolved in that order before `hw` — when one + variant/quant/strategy needs its own image), `multiNodeHints` only for fabric-specific hw + (e.g. gb200). + +5. **Diffusion pages: add the ComfyUI section.** Every diffusion cookbook page ends with + a `## . Run in ComfyUI` section so a reader never has to guess whether the model is + reachable from ComfyUI. It is one component; the per-model facts live in the component, + not the page: + + ```mdx + ## . Run in ComfyUI + + import { ComfyUISupport } from '/src/snippets/diffusion/comfyui-support.jsx'; + + + ``` + + Pick `model` from the table in + `docs/src/snippets/diffusion/comfyui-support.jsx`. Use the model's own key when it has + an entry (its executor or dedicated node differs); otherwise use the generic `image` or + `video`. A model gets its own key only when the plugin actually treats it specially — + an entry in `executor_class_dict` + (`python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/core/generator.py`) or a + dedicated node. Adding a key without the matching plugin support makes the page lie. ### Site-wiring (do all three) diff --git a/.claude/skills/cookbook-add-model/references/authoring-reference.md b/.claude/skills/cookbook-add-model/references/authoring-reference.md index 118ba83e6acb..d98d2d4feda3 100644 --- a/.claude/skills/cookbook-add-model/references/authoring-reference.md +++ b/.claude/skills/cookbook-add-model/references/authoring-reference.md @@ -44,7 +44,7 @@ the full contract): | Field | Type | Purpose | |---|---|---| | `multiNodeHints` | `{[hwId]: string[]}` | Lines prepended as `# ...` comments to multi-node commands (env-var hints). Per-hw, and only for hw whose **cluster fabric needs manual NIC config** (e.g. `gb200` NVL72/MNNVL → NVSHMEM/Gloo hints). NOT every multi-N hw needs an entry — standard-IB DeepEP (h200) auto-detects the HCA, and Marlin multi-node (h100) uses no DeepEP/NVSHMEM at all. Hints render above **both** run modes, so keep them mode-agnostic: `docker run` flags belong in the hardware entry's `multiNodeDockerFlags` (above), which the engine puts in the command itself. | -| `dockerImages` | `{[key]: string}` | Image for `docker run` framing, keyed by `hw\|quant` (most specific) then `hw`. Use a `hw\|quant` key only when one quant on a shared GPU needs a different image (e.g. an NVFP4 dev build on b300/gb300 while FP8/BF16 stay on the release image); otherwise key by plain `hw`. **Ask the user which sglang build the recipes ran on; don't guess a supporting release.** Falls back to `lmsysorg/sglang:dev` if missing — also the sensible default when unsure. | +| `dockerImages` | `{[key]: string}` | Image for `docker run` framing, keyed by `hw\|variant\|quant` (most specific), then `variant\|quant`, `hw\|quant\|strategy`, `hw\|quant`, `hw`. Key by plain `hw` unless a subset needs a different image: `hw\|quant` when one quant on a shared GPU needs its own build (e.g. an NVFP4 dev build on b300/gb300 while FP8/BF16 stay on the release image); the `variant` keys when one checkpoint needs its own build (e.g. a preview image for a not-yet-released variant); the `strategy` key when one tier needs one (e.g. a spec-decoding preview). **Ask the user which sglang build the recipes ran on; don't guess a supporting release.** Falls back to `lmsysorg/sglang:dev` if missing — also the sensible default when unsure. | | `playgroundFeatures` | `{[axisId]: {...}}` | Opts into the Playground widget. See §2.3. | | `benchmarkCommands` | `{speed: string, accuracy: {[accKey]: string \| {[variant]: string}}, numPromptsByConc?: {[c]: number}}` | Powers the benchmark card's **"⚡ Reproduce"** modal. `speed` is ONE `bench_serving` template; the engine fills `{{DATASET}}`/`{{ISL}}`/`{{OSL}}` from each cell's `speed[].workload`, the chip-picked `{{MAX_CONCURRENCY}}`, and `{{NUM_PROMPTS}}` (resolved `workload.num_prompts ?? numPromptsByConc[c] ?? max(c*2, 200)`). `accuracy` maps an accuracy field (e.g. `gsm8k_pct`) to a per-eval template — a string, OR a `{flash, pro, …}` object keyed by variant when the command differs per variant (e.g. GPQA/AIME `--max-tokens`). The modal renders a chip per eval (one command area, like Speed). Both also use `{{MODEL_NAME}}` + `{{CURL_HOST}}`/`{{CURL_PORT}}` like `curl`. `speed` should carry `--flush-cache` (bench_serving's `random` prompts are deterministic — warm reruns hit the radix cache and inflate throughput; measure cache-cold). Optional; the button only appears when this AND `benchmarks` are present. | | `defaultAccuracy` | `{[variant]: {[accKey]: number}}` | Model-level accuracy applied to **every** cell of a variant (e.g. GPQA Diamond / AIME25 — hardware-independent). Merged UNDER each cell's measured `accuracy` (a per-cell value wins), so you set a variant's score once instead of copying it onto every benchmark entry. Keys must match `accuracyLabels` (below) + `benchmarkCommands.accuracy`. | @@ -178,7 +178,7 @@ schemas (full reference in the `_playground.jsx` header): | `pdDisagg` | Mode + transfer backend (+ optional per-backend env via `envWhen` hw-gate) + IB device + optional `router{port, command}` | Model supports prefill/decode disaggregation. A `modes[]` entry may carry `flags` / `env` that only that role needs (`prefill`'s `--load-balance-method`, `decode`'s `--disaggregation-decode-polling-interval`, ...); they are emitted only while that role is selected, and a flag whose head the base cell already sets is replaced rather than duplicated. Put role-specific settings here, NOT in the cells — a cell carries one recipe, and the role is a Playground overlay. In `router.command` the ports MUST be `{{PREFILL_PORT}}` / `{{DECODE_PORT}}` / `{{ROUTER_PORT}}`: the engine substitutes them from its own `PD_PORTS` (prefill 30000, decode **30100**), so a literal port silently points the router at something the generated decode command never binds. When a PD role is active and `router` is set, the playground shows the router (SGLang Model Gateway) launch command as a companion block and retargets the cURL modal to `router.port`. | | `hicache` | Enable + storage + write policy | Model is large enough that hierarchical KV cache matters. | | `hisparse` | Enable + host-ratio select; whole card gated on the live PD-Disagg mode being `decode` | DSA-style model (DeepSeek-V3.2 / V4, GLM-5) that supports decode-side hierarchical sparse attention. | -| `flagSelects` | A config-declared **list** of single-selects, each `{ id, title, stripPrefixes, options }` (option = `{ id, label, flags?, hide?, disable?, disableReason? }`); a flagless option is the "none"/accuracy-safe choice | A titled single-select that picks one value of a flag family the other axes don't model — e.g. KV-cache dtype (`--kv-cache-dtype`), mamba scheduler strategy (`--mamba-scheduler-strategy`). Generic: no engine change to add another. | +| `flagSelects` | A config-declared **list** of single-selects, each `{ id, title, stripPrefixes, options }` (option = `{ id, label, flags?, hide?, disable?, disableReason? }`); a flagless option is the "none"/accuracy-safe choice | A titled single-select that picks one value of a flag family the other axes don't model — e.g. KV-cache dtype (`--kv-cache-dtype`), mamba scheduler strategy (`--mamba-radix-cache-strategy`). Generic: no engine change to add another. | **Per-chip constraints**: any chip entry in any axis can be wrapped with `hide` / `disable` constraint objects: diff --git a/.claude/skills/cookbook-add-model/templates/config.jsx.tmpl b/.claude/skills/cookbook-add-model/templates/config.jsx.tmpl index 6cfaf3d280a0..dfb5e537adc7 100644 --- a/.claude/skills/cookbook-add-model/templates/config.jsx.tmpl +++ b/.claude/skills/cookbook-add-model/templates/config.jsx.tmpl @@ -104,7 +104,7 @@ export const config = { // in _deployment.jsx + the per-cell/defaultAccuracy keys. accuracy: { gsm8k_pct: -`# To install sgl-eval: pip install git+https://github.com/sgl-project/sgl-eval +`# To install sgl-eval: pip install sgl-eval sgl-eval run gsm8k \\ --base-url http://{{CURL_HOST}}:{{CURL_PORT}}/v1 \\ --num-threads 32`, @@ -141,12 +141,14 @@ sgl-eval run gsm8k \\ ], }, - // Image for `docker run` framing, keyed by `hw` (or `hw|quant`, resolved first, when one - // quant on a shared GPU needs its own image — e.g. an FP4 dev build while FP8/BF16 use the - // release tag). ASK the user which sglang build the recipes ran on; don't guess a supporting - // release. Default below is :dev (nightly) — replace the tag with the user's release if they - // give one. NVIDIA share one image; AMD uses ROCm tags. GB200/GB300/B300 may need a - // `-cu130` (CUDA 13) tag — confirm per release. + // Image for `docker run` framing, keyed by `hw` (or, resolved before it: `hw|variant|quant`, + // `variant|quant`, `hw|quant|strategy`, `hw|quant` — use a more specific key only when a + // variant/quant/strategy subset needs its own build, e.g. an FP4 dev build while FP8/BF16 use + // the release tag, or a preview image for a not-yet-released variant). ASK the user which + // sglang build the recipes ran on; don't guess a supporting release. Default below is :dev + // (nightly) — replace the tag with the user's release if they give one. NVIDIA share one + // image; AMD uses ROCm tags. GB200/GB300/B300 may need a `-cu130` (CUDA 13) tag — confirm + // per release. dockerImages: { h100: "lmsysorg/sglang:dev", h200: "lmsysorg/sglang:dev", diff --git a/.claude/skills/cookbook-migrate-model/SKILL.md b/.claude/skills/cookbook-migrate-model/SKILL.md index eb1d514e84eb..ae9b55dad95f 100644 --- a/.claude/skills/cookbook-migrate-model/SKILL.md +++ b/.claude/skills/cookbook-migrate-model/SKILL.md @@ -73,7 +73,7 @@ your dispatch prompt, or ask for it. change in a migration PR. Model-specific features are config DATA consumed by generic axis handlers (MegaMoE precedent), so they need NO engine change. A **titled single-select that strips a flag family** — KV Cache DType - (`--kv-cache-dtype`), mamba (`--mamba-scheduler-strategy`), … — is already + (`--kv-cache-dtype`), mamba (`--mamba-radix-cache-strategy`), … — is already covered by the merged generic **`flagSelects`** axis: declare it in the config (a list of `{ id, title, stripPrefixes, options }`; see the Qwen3.5 mamba example), **no engine PR**. Only a genuinely new control *shape* that @@ -129,7 +129,7 @@ mention — and a model-specific control is **config data, not engine code** (MegaMoE W4A4 is all DSv4 config on the existing `moe` axis). It's pure config whenever it fits an existing axis's data schema. A **titled single-select that strips a flag family** (Nemotron3's "KV Cache DType", -mamba `--mamba-scheduler-strategy`, …) fits the merged generic **`flagSelects`** +mamba `--mamba-radix-cache-strategy`, …) fits the merged generic **`flagSelects`** axis — so it too is config-only (declare a `flagSelects` list). Only a control whose *shape* `flagSelects` still can't express would need a ONE-TIME generic primitive (never a model-named handler) on a separate PRIOR engine PR, keeping diff --git a/.claude/skills/cookbook-migrate-model/references/dimension-mapping.md b/.claude/skills/cookbook-migrate-model/references/dimension-mapping.md index acf87415dab3..3e34ae790321 100644 --- a/.claude/skills/cookbook-migrate-model/references/dimension-mapping.md +++ b/.claude/skills/cookbook-migrate-model/references/dimension-mapping.md @@ -15,7 +15,7 @@ this file is about the *mapping decisions*. | toggle that **couples** with other parts of the command (changes TP/mem/EP), OR one the legacy page labels with **operating-point words** | `strategies` | The Playground applies pure flag diffs — it cannot do coupled changes. Example: Qwen3.5's MTP toggle bumps TP on three H100 combos → strategies `low-latency` (MTP on) / `high-throughput` (MTP off). **Naming counts like coupling**: GLM-5.1's / Kimi-K2.6's `dpattention` adds only `--dp N --enable-dp-attention` (uncoupled), but its options are subtitled "Low Latency" / "High Throughput" — the page's own named operating-point split → strategies; a flag-only spec toggle riding alongside it stays a Playground axis and bakes per its legacy default. GPU-count radios (GLM-4.7, MiniMax-M2.5/2.7) → budget-tier strategies with the legacy SUPPORT matrix preserved by which cells exist. Strategy count follows the page's operating points: 1 → `balanced`, 2 → `low-latency`+`high-throughput`, 3 → the full trio (§4). | | toggle that only adds/removes its own flags | Playground axis (+ bake, EXCEPT parsers and accuracy-degrading flags) | **Parsers (`--reasoning-parser` / `--tool-call-parser`) are NEVER baked into cells** — Deployment commands ship without them regardless of the legacy default or the measured command; the `parsers` axis adds them on top (DSv4 convention; cells mirror the legacy generator's parsers-OFF output). Accuracy-degrading toggles are never baked either — §2 caveats (axis-only, accuracy-safe cells). Other flag-only toggles: legacy default ON → bake into cells AND declare the axis so users can strip (red strikethrough); default OFF → keep cells clean, axis preset only. MTP/EAGLE presets → `speculative` axis; dp-attention → a strategy when the legacy page labels it as the operating-point split or when coupled (see the row above), else `attention.dpAttn`. **EVERY legacy control survives as an interactive control** (a dimension or a Playground axis), never a tips-only mention — but a model-specific control is **config DATA, not engine code**: the axis handler reads options/flags/env/gating straight from `config.playgroundFeatures` (MegaMoE W4A4 is entirely DSv4 config data on the existing `moe` axis — no per-model engine edit). A control that fits an existing axis's data schema is therefore pure config, full stop. A **titled single-select that strips a flag family** (e.g. Nemotron3's "KV Cache DType" `--kv-cache-dtype`) is covered by the merged generic **`flagSelects`** axis → **config-only**: declare a `flagSelects` list of `{ id, title, stripPrefixes, options }` (see the Qwen3.5 mamba example), **no engine PR**. Only a control whose *shape* `flagSelects` still can't express would need a new ONE-TIME generic primitive (never a model-named handler) on a prior engine PR; the backward-compat reasoning (opt-in per key, not in the opt-out set) is in engine-axis.md. | | per-combo hidden option (e.g. spec hidden on Xeon) | absent cells | Don't create cells for combos the legacy widget couldn't produce; the engine greys them automatically. `# Error:` pseudo-commands → no cell + explanation in §2 tips and/or a chip `disable`/`disableReason`. | -| coupled secondary knob (e.g. mamba cache V1/V2) | cells + Playground axis | Bake the correct value per cell following the legacy coupling (Qwen3.5: MTP ⇒ `--mamba-scheduler-strategy extra_buffer` on NVIDIA; AMD/Xeon ⇒ V1/no flag) and document the coupling in §2 tips — AND surface the knob as a Playground axis like every other legacy feature (row above; add the axis when none fits). Baking alone is NOT enough — the every-feature rule supersedes the pilot's cells+prose-only treatment of Qwen3.5's mamba knob (retrofit pending). The mamba knob is the same single-select shape as KV Cache DType, so it rides the merged generic **`flagSelects`** axis — Qwen3.6 / Qwen3-Coder-Next declare it purely in config (a `flagSelects` block), **no engine PR**. | +| coupled secondary knob (e.g. mamba cache V1/V2) | cells + Playground axis | Bake the correct value per cell following the legacy coupling (Qwen3.5: MTP ⇒ `--mamba-radix-cache-strategy extra_buffer` on NVIDIA; AMD/Xeon ⇒ V1/no flag) and document the coupling in §2 tips — AND surface the knob as a Playground axis like every other legacy feature (row above; add the axis when none fits). Baking alone is NOT enough — the every-feature rule supersedes the pilot's cells+prose-only treatment of Qwen3.5's mamba knob (retrofit pending). The mamba knob is the same single-select shape as KV Cache DType, so it rides the merged generic **`flagSelects`** axis — Qwen3.6 / Qwen3-Coder-Next declare it purely in config (a `flagSelects` block), **no engine PR**. | ## 2. Command rewrite table (the ONLY allowed normalizations) diff --git a/.claude/skills/cookbook-review-pr/SKILL.md b/.claude/skills/cookbook-review-pr/SKILL.md index d7ccaf46a5de..27d29f087751 100644 --- a/.claude/skills/cookbook-review-pr/SKILL.md +++ b/.claude/skills/cookbook-review-pr/SKILL.md @@ -147,6 +147,18 @@ than restating. equal what the engine emits from the corresponding cell — same flags, same order. Drift here is the most common review miss. +### 5b. ComfyUI section (diffusion pages) +- A diffusion page ends with `## . Run in ComfyUI` rendering ``. A + reader must not have to guess whether the model is reachable from ComfyUI. +- The `model` prop is a key that exists in `docs/src/snippets/diffusion/comfyui-support.jsx`. + A model-specific key is only correct when the plugin really treats it specially — an entry + in `executor_class_dict` + (`python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion/core/generator.py`) or a dedicated + node. Otherwise the generic `image` / `video` key is the honest one; a model-specific key + without matching plugin support makes the page claim support that does not exist. +- Prose describing ComfyUI support inline instead of using the component is a finding: the + facts drift from the plugin. + ### 6. Commands / port - Launch uses `sglang serve` — flag any `python -m sglang.launch_server` / `python3 -m sglang.launch_server` (deprecated). The engine already emits `sglang serve`; diff --git a/.claude/skills/env-var-conventions/SKILL.md b/.claude/skills/env-var-conventions/SKILL.md index 048c6e26b06c..65fc008153dd 100644 --- a/.claude/skills/env-var-conventions/SKILL.md +++ b/.claude/skills/env-var-conventions/SKILL.md @@ -162,7 +162,7 @@ Picking between `ENABLE_FOO` and `DISABLE_FOO`: both verbs are valid. The only f For a rename where the old key must keep working with a warning: ```python -SGLANG_DSA_FUSE_TOPK = EnvBoolWithAlias(True, deprecated_name="SGLANG_NSA_FUSE_TOPK") +SGLANG_NEW_NAME = EnvBoolWithAlias(True, deprecated_name="SGLANG_OLD_NAME") ``` Use `EnvBoolWithAlias` / `EnvIntWithAlias`. The fallback emits a `DeprecationWarning` and copies the old value over. diff --git a/.claude/skills/llm-torch-profiler-analysis/references/fuse-overlap-catalog.md b/.claude/skills/llm-torch-profiler-analysis/references/fuse-overlap-catalog.md index ef953174d8ef..3b12b16ccf96 100644 --- a/.claude/skills/llm-torch-profiler-analysis/references/fuse-overlap-catalog.md +++ b/.claude/skills/llm-torch-profiler-analysis/references/fuse-overlap-catalog.md @@ -71,7 +71,7 @@ in-flight row as shipped. | Fused MoE sum + all-reduce | routed MoE followed by explicit sum-reduce kernels | `python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py`
`python/sglang/srt/layers/moe/fused_moe_triton/fused_moe_triton_kernels.py` | `fuse_sum_all_reduce=True` path in the second MoE GEMM | Before inventing a new MoE reduction fuse, check whether `enable_fused_moe_sum_all_reduce` is simply off or the quant path is incompatible. | | Fused MoE activation + quant / re-quant | `silu_and_mul_*quant*`
`npu_dequant_swiglu_quant`
`swiglu_quant` | `python/sglang/srt/layers/moe/ep_moe/kernels.py`
`python/sglang/kernels/ops/quantization/nvfp4_gemm_swiglu_nvfp4_quant.py`
`python/sglang/srt/layers/moe/cutlass_w4a8_moe.py`
`python/sglang/srt/hardware_backend/npu/quantization/moe_methods.py` | Quantized MoE backends fuse SwiGLU / SiLU-and-mul with FP8 / FP4 / NPU re-quant before the second expert GEMM | If MoE traces show standalone activation then quant kernels, first check whether the quantized fused path is missing. | | DeepSeek comm-prep fused RMSNorm + quant / flatten-quant | `fused_rms_fp8_group_quant`
`fused_rms_mxfp4_quant`
`fused_flatten_fp8_group_quant`
`fused_flatten_mxfp4_quant` | `python/sglang/srt/layers/communicator.py`
`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla.py`
`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mha.py` | DeepSeek MLA / MHA ROCm paths fuse RMSNorm or flatten with FP8 / MXFP4 quantization for comm / attention prep | On DeepSeek quant traces, split norm + quant or flatten + quant is an existing family, not a new idea. | -| NSA fused top-k transform / page-table build | `fast_topk_transform_fused`
`fast_topk_transform_ragged_fused` | `python/sglang/srt/layers/attention/nsa_backend.py` | NSA can fuse top-k selection with paged / ragged index transform instead of separate top-k plus metadata scatter | If NSA top-k metadata work is split, check `SGLANG_NSA_FUSE_TOPK` and backend support first. | +| NSA fused top-k transform / page-table build | `fast_topk_transform_fused`
`fast_topk_transform_ragged_fused` | `python/sglang/srt/layers/attention/dsa_backend.py` | NSA can fuse top-k selection with paged / ragged index transform instead of separate top-k plus metadata scatter | If NSA top-k metadata work is split, check `SGLANG_DSA_FUSE_TOPK` and backend support first. | | NSA fused quantize + indexed K-cache store | `fused_store_index_k_cache`
`act_quant`
`index_k_with_scale_buffer` | `python/sglang/kernels/ops/attention/fused_store_index_cache.py`
`python/sglang/srt/layers/attention/nsa/nsa_indexer.py` | Single JIT kernel quantizes bf16 K to fp8 + scale and writes directly into NSA index cache | Treat split `act_quant` + buffer-store on CUDA as missing an existing fused store path. | | Fused sampling temperature + softmax | `fused_temperature_softmax*` | `python/sglang/srt/layers/fused_sampling.py`
`python/sglang/srt/layers/sampler.py` | Triton single-pass / multi-pass kernels fuse temperature scaling and softmax during decode | Separate temp-divide + softmax at decode batch sizes is often a missed existing fusion. | | Fused logit softcap | `fused_softcap`
`final_logit_softcapping` | `python/sglang/srt/layers/elementwise.py`
`python/sglang/srt/layers/logits_processor.py` | Triton kernels fuse cast-to-float and softcap / tanh math for logits or generic elementwise softcapping | Treat exposed cast + softcap ladders as an existing Triton fuse family. | @@ -148,8 +148,8 @@ Stable entries should be folded into the mainline family rows above. | PR `#22392` CUTLASS FP8 GEMM replacing nvjet | `cutlass_scaled_mm`
`fp8_scaled_mm`
`nvjet`
`cudaMemsetAsync` | `PR #22392`
`python/sglang/kernels/aot/python/sgl_kernel/gemm.py`
`python/sglang/srt/layers/quantization/fp8_utils.py` | Runtime replacement swaps nvjet FP8 GEMMs for CUTLASS kernels, removing per-launch memset bubbles and extra output-copy kernels | Treat nvjet GEMM + memset bubble ladders as an in-flight SGLang linear-kernel family before calling them novel. | | PR `#18612` NVFP4 CUTLASS MoE fused SiLU+Mul+quant | `silu_and_mul_scaled_nvfp4`
`nvfp4 expert quant`
`cutlass moe` | `PR #18612`
`python/sglang/srt/layers/moe/cutlass_w4a8_moe.py`
`python/sglang/kernels/ops/quantization/nvfp4_gemm_swiglu_nvfp4_quant.py` | Fuses MoE activation epilogue and NVFP4 expert quantization before the CUTLASS MoE second GEMM | Treat split SiLU+Mul then NVFP4 expert quant in CUTLASS MoE traces as an in-flight upstream SGLang family. | | PR `#22918` FlashInfer per-token NVFP4 MoE | `per_token_nvfp4`
`trtllm_fp4_block_scale_moe`
`FlashInfer MoE` | `PR #22918`
`python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py` | Adds FlashInfer-backed per-token NVFP4 MoE execution so expert quant/dequant work can move into the fused MoE backend | Treat standalone per-token NVFP4 MoE support kernels as a candidate missing backend-selection path, not an automatically novel kernel idea. | -| PR `#22851` NSA top-k backend and FlashInfer / PyTorch top-k split | `nsa topk`
`flashinfer_topk`
`pytorch_topk`
`fast_topk_transform` | `PR #22851`
`python/sglang/srt/layers/attention/nsa_backend.py` | Makes NSA top-k backend selection explicit and aligns fused top-k transform with FlashInfer / PyTorch fallbacks | When NSA top-k dominates decode, first classify it as backend selection or fused-transform eligibility work. | -| PR `#24125` GLM5 NSA decode CatArrayBatchedCopy removal | `CatArrayBatchedCopy`
`GLM-5`
`NSA`
`TileLang decode` | `PR #24125`
`python/sglang/srt/layers/attention/nsa_backend.py` | Skips redundant cat/copy work in the GLM5 NSA TileLang decode path | Treat cat/copy bursts in GLM5 NSA decode as a concrete in-flight cleanup opportunity. | +| PR `#22851` NSA top-k backend and FlashInfer / PyTorch top-k split | `nsa topk`
`flashinfer_topk`
`pytorch_topk`
`fast_topk_transform` | `PR #22851`
`python/sglang/srt/layers/attention/dsa_backend.py` | Makes NSA top-k backend selection explicit and aligns fused top-k transform with FlashInfer / PyTorch fallbacks | When NSA top-k dominates decode, first classify it as backend selection or fused-transform eligibility work. | +| PR `#24125` GLM5 NSA decode CatArrayBatchedCopy removal | `CatArrayBatchedCopy`
`GLM-5`
`NSA`
`TileLang decode` | `PR #24125`
`python/sglang/srt/layers/attention/dsa_backend.py` | Skips redundant cat/copy work in the GLM5 NSA TileLang decode path | Treat cat/copy bursts in GLM5 NSA decode as a concrete in-flight cleanup opportunity. | | PR `#24007` MoE LoRA virtual experts for csgmv backend | `csgmv`
`virtual experts`
`MoE LoRA`
`fused_moe_lora` | `PR #24007`
`python/sglang/srt/layers/lora_backend.py`
`python/sglang/srt/layers/moe` | Routes MoE LoRA adapter work through virtual experts so csgmv-style kernels can batch it instead of launching fragmented adapter work | Treat MoE-LoRA tiny-kernel ladders as an in-flight batching/fusion family. | | PR `#24150` torch.compile local decode support | `enable_torch_compile`
`local compile`
`decode compile`
`torchinductor` | `PR #24150`
`python/sglang/srt` | Extends SGLang torch.compile coverage to local decode regions, so Inductor-generated fusion may replace hand-authored tiny kernels | When decode traces show compiler-generated kernels or missing named fused kernels, check this in-flight compile path before calling the shape unsupported. | @@ -319,7 +319,7 @@ only vLLM references to the TokenSpeed package. | `enable_single_batch_overlap` | `python/sglang/srt/server_args.py` | Enables the SBO family. | | `enable_fused_moe_sum_all_reduce` | `python/sglang/srt/server_args.py` | Enables fused MoE sum-reduce in the down path. | | `SGLANG_BLACKWELL_OVERLAP_SHARED_EXPERTS_OUTSIDE_SBO` | `python/sglang/srt/environ.py` | Alters how DeepSeek-style shared-expert overlap behaves on Blackwell. | -| `SGLANG_NSA_FUSE_TOPK` | `python/sglang/srt/environ.py` | Gates NSA fused top-k transform / page-table build. | +| `SGLANG_DSA_FUSE_TOPK` | `python/sglang/srt/environ.py` | Gates NSA fused top-k transform / page-table build. | | `SGLANG_DISAGG_STAGING_BUFFER` | `python/sglang/srt/environ.py` | Enables the heterogeneous-TP staging-buffer family and its overlap windows. | | `SGLANG_STAGING_USE_TORCH` | `python/sglang/srt/disaggregation/common/staging_buffer.py` | Forces torch fallback for staging gather / scatter, so Triton staging kernels may disappear by design. | | `SGLANG_VIT_ENABLE_CUDA_GRAPH` | `python/sglang/srt/environ.py` | Can intentionally disable vision `aux_stream` overlap. | diff --git a/.claude/skills/llm-torch-profiler-analysis/scripts/triage_kernel_helpers.py b/.claude/skills/llm-torch-profiler-analysis/scripts/triage_kernel_helpers.py index 81a71cb96377..de702db730ba 100644 --- a/.claude/skills/llm-torch-profiler-analysis/scripts/triage_kernel_helpers.py +++ b/.claude/skills/llm-torch-profiler-analysis/scripts/triage_kernel_helpers.py @@ -541,7 +541,7 @@ class FusionPatternSpec: "fused_flashmla_metadata", ), rationale_hint=( - "NSA replay metadata copies are already fused into one-kernel" " families." + "NSA replay metadata copies are already fused into one-kernel families." ), min_share=0.02, likely_share=0.2, @@ -787,7 +787,7 @@ class FusionPatternSpec: ("softmax", "sampling"), ), rationale_hint=( - "Decode-time sampling already has fused temperature and softmax" " kernels." + "Decode-time sampling already has fused temperature and softmax kernels." ), min_share=0.05, likely_share=0.5, @@ -1218,8 +1218,7 @@ class FusionPatternSpec: FusionPatternSpec( pattern="vLLM fused residual add + RMSNorm", candidate_path=( - "vllm/_custom_ops.py" - "
vllm/compilation/passes/fusion/rms_quant_fusion.py" + "vllm/_custom_ops.py
vllm/compilation/passes/fusion/rms_quant_fusion.py" ), active_keywords=( "fused_add_rms_norm", @@ -1236,8 +1235,7 @@ class FusionPatternSpec: FusionPatternSpec( pattern="vLLM fused activation-and-mul", candidate_path=( - "vllm/_custom_ops.py" - "
vllm/compilation/passes/fusion/act_quant_fusion.py" + "vllm/_custom_ops.py
vllm/compilation/passes/fusion/act_quant_fusion.py" ), active_keywords=( "silu_and_mul", diff --git a/.claude/skills/mechanical-refactor-verify/scripts/mechanical_refactor_proof_generator.py b/.claude/skills/mechanical-refactor-verify/scripts/mechanical_refactor_proof_generator.py index 598ac7a82644..0aeb0ea5f2a2 100644 --- a/.claude/skills/mechanical-refactor-verify/scripts/mechanical_refactor_proof_generator.py +++ b/.claude/skills/mechanical-refactor-verify/scripts/mechanical_refactor_proof_generator.py @@ -256,7 +256,9 @@ def _module_assign_names(text: str) -> set: targets = ( node.targets if isinstance(node, ast.Assign) - else [node.target] if isinstance(node, ast.AnnAssign) else [] + else [node.target] + if isinstance(node, ast.AnnAssign) + else [] ) names |= {t.id for t in targets if isinstance(t, ast.Name)} return names diff --git a/.claude/skills/mechanical-refactor-verify/scripts/mechanical_refactor_reproduction_utils.py b/.claude/skills/mechanical-refactor-verify/scripts/mechanical_refactor_reproduction_utils.py index fb0284dba510..8c2856a7ef73 100644 --- a/.claude/skills/mechanical-refactor-verify/scripts/mechanical_refactor_reproduction_utils.py +++ b/.claude/skills/mechanical-refactor-verify/scripts/mechanical_refactor_reproduction_utils.py @@ -173,9 +173,9 @@ def _find_unique_def( if isinstance(node, definition) and node.name == name ] assert matches, f"{name} not found in {where}" - assert ( - len(matches) == 1 - ), f"{len(matches)} defs named {name} in {where}; pass from_class to disambiguate" + assert len(matches) == 1, ( + f"{len(matches)} defs named {name} in {where}; pass from_class to disambiguate" + ) return matches[0] @@ -287,9 +287,9 @@ def _lowered_call_text(text: str, node: ast.Call) -> str: """ receiver = node.args[0] receiver_src = _node_slice(text, receiver) - assert ( - "\n" not in receiver_src and "#" not in receiver_src - ), f"receiver {receiver_src!r} must be single-line and comment-free" + assert "\n" not in receiver_src and "#" not in receiver_src, ( + f"receiver {receiver_src!r} must be single-line and comment-free" + ) opener = _slice_span( text, node.func.end_lineno, @@ -722,9 +722,9 @@ def op(root: Path) -> None: ) existing = [alias_text(a.name, a.asname) for a in node.names] added = alias_text(name, asname) - assert ( - added not in existing - ), f"{name!r} already imported from {module!r} in {rel}" + assert added not in existing, ( + f"{name!r} already imported from {module!r} in {rel}" + ) rebuilt = f"from {module} import " + ", ".join(existing + [added]) + nl lines[node.lineno - 1 : node.end_lineno] = [rebuilt] _write_source(path, "".join(lines)) @@ -825,9 +825,9 @@ def op(root: Path) -> None: for node in tree.body if isinstance(node, (ast.Import, ast.ImportFrom)) ] - assert ( - imports - ), f"no imports to anchor a new `if TYPE_CHECKING:` block in {rel}" + assert imports, ( + f"no imports to anchor a new `if TYPE_CHECKING:` block in {rel}" + ) insert_at = imports[-1].end_lineno lines[insert_at:insert_at] = [ nl, @@ -864,9 +864,9 @@ def op(root: Path) -> None: replaced = lines[node.lineno - 1].replace( f"from {spelled} import", f"from {new_module} import", 1 ) - assert ( - replaced != lines[node.lineno - 1] - ), f"import spelling {spelled!r} not found on its line in {rel}" + assert replaced != lines[node.lineno - 1], ( + f"import spelling {spelled!r} not found on its line in {rel}" + ) lines[node.lineno - 1] = replaced changed = True assert changed, f"nested import of {name} from {old_module} not in {rel}" @@ -974,9 +974,9 @@ def move_symbol( ``self: Target`` annotation is dropped (redundant inside the class). The body is moved verbatim; the formatter normalises the surrounding blank lines. """ - assert ( - before is None or after is None - ), "move_symbol: before and after are mutually exclusive" + assert before is None or after is None, ( + "move_symbol: before and after are mutually exclusive" + ) def op(root: Path) -> None: src_path = root / src @@ -1255,15 +1255,17 @@ def op(root: Path) -> None: targets = ( node.targets if isinstance(node, ast.Assign) - else [node.target] if isinstance(node, ast.AnnAssign) else [] + else [node.target] + if isinstance(node, ast.AnnAssign) + else [] ) names = {t.id for t in targets if isinstance(t, ast.Name)} hit = names & dropped if not hit: continue - assert len(names) == len( - targets - ), f"drop_assigns {sorted(hit)}: non-name targets in {src}" + assert len(names) == len(targets), ( + f"drop_assigns {sorted(hit)}: non-name targets in {src}" + ) value_src = ast.unparse(node.value) if node.value is not None else None for dropped_name in hit: removed_assigns[dropped_name] = value_src @@ -1289,15 +1291,17 @@ def op(root: Path) -> None: else: assign_spans.append((node.lineno, node.end_lineno)) found_assigns |= hit - assert ( - found_assigns == dropped - ), f"{dropped - found_assigns} not assigned in {src}" + assert found_assigns == dropped, ( + f"{dropped - found_assigns} not assigned in {src}" + ) rederivable: dict[str, str | None] = {} for node in tree.body: targets = ( node.targets if isinstance(node, ast.Assign) - else [node.target] if isinstance(node, ast.AnnAssign) else [] + else [node.target] + if isinstance(node, ast.AnnAssign) + else [] ) names = [t.id for t in targets if isinstance(t, ast.Name)] if not names or set(names) & dropped: @@ -1383,9 +1387,9 @@ def op(root: Path) -> None: src_text = _read_source(src_path) assert src_text.count(body) == 1, f"block not found uniquely in {src}" at = src_text.find(body) - assert ( - at == 0 or src_text[at - 1] == "\n" - ), f"block matches mid-line in {src}; it must start at a line boundary" + assert at == 0 or src_text[at - 1] == "\n", ( + f"block matches mid-line in {src}; it must start at a line boundary" + ) _write_source(src_path, src_text.replace(body, call, 1)) dst_path = root / dst diff --git a/.claude/skills/mechanical-refactor-verify/scripts/tests/proof_generator/test_infer_moves.py b/.claude/skills/mechanical-refactor-verify/scripts/tests/proof_generator/test_infer_moves.py index 0b9e5ead4077..8e5e4b601420 100644 --- a/.claude/skills/mechanical-refactor-verify/scripts/tests/proof_generator/test_infer_moves.py +++ b/.claude/skills/mechanical-refactor-verify/scripts/tests/proof_generator/test_infer_moves.py @@ -370,12 +370,7 @@ def test_infer_recipe_module_level_def_shadowed_by_method_name(repo: Path) -> No " return foo(x=self.x)\n" ), "util.py": ( - "def keep():\n" - " return 1\n" - "\n" - "\n" - "def foo(*, x):\n" - " return x + 1\n" + "def keep():\n return 1\n\n\ndef foo(*, x):\n return x + 1\n" ), }, ) @@ -437,9 +432,7 @@ def test_infer_recipe_move_leaving_a_forwarding_delegate(repo: Path) -> None: _write( repo, **{ - "model.py": ( - "class M:\n" " def work(self, x):\n" " return x + 1\n" - ), + "model.py": ("class M:\n def work(self, x):\n return x + 1\n"), "comp.py": "class C:\n def keep(self):\n return 1\n", }, ) @@ -448,9 +441,7 @@ def test_infer_recipe_move_leaving_a_forwarding_delegate(repo: Path) -> None: repo, **{ "model.py": ( - "class M:\n" - " def work(self, x):\n" - " return self.comp.work(x)\n" + "class M:\n def work(self, x):\n return self.comp.work(x)\n" ), "comp.py": ( "class C:\n" diff --git a/.claude/skills/mechanical-refactor-verify/scripts/tests/reproduction_cli/cli_testlib.py b/.claude/skills/mechanical-refactor-verify/scripts/tests/reproduction_cli/cli_testlib.py index fb1eef19b03d..e9490fdfa0ce 100644 --- a/.claude/skills/mechanical-refactor-verify/scripts/tests/reproduction_cli/cli_testlib.py +++ b/.claude/skills/mechanical-refactor-verify/scripts/tests/reproduction_cli/cli_testlib.py @@ -2,13 +2,9 @@ from pathlib import Path _PASSING_PROOF = ( - "import sys\n" - 'print("PASS: reproduces the commit byte-for-byte.")\n' - "sys.exit(0)\n" -) -_FAILING_PROOF = ( - "import sys\n" 'print("RESIDUAL (2 lines):\\n+x\\n-y")\n' "sys.exit(1)\n" + 'import sys\nprint("PASS: reproduces the commit byte-for-byte.")\nsys.exit(0)\n' ) +_FAILING_PROOF = 'import sys\nprint("RESIDUAL (2 lines):\\n+x\\n-y")\nsys.exit(1)\n' def _git(repo: Path, *args: str) -> str: diff --git a/.claude/skills/mechanical-refactor-verify/scripts/tests/reproduction_utils/test_add_imports.py b/.claude/skills/mechanical-refactor-verify/scripts/tests/reproduction_utils/test_add_imports.py index 0cf0bd58e717..3b63e3ad4862 100644 --- a/.claude/skills/mechanical-refactor-verify/scripts/tests/reproduction_utils/test_add_imports.py +++ b/.claude/skills/mechanical-refactor-verify/scripts/tests/reproduction_utils/test_add_imports.py @@ -110,13 +110,7 @@ def test_add_typechecking_import_inserts_in_block(tmp_path: Path) -> None: def test_add_typechecking_import_creates_missing_block(tmp_path: Path) -> None: """With no TYPE_CHECKING block, one is created after the trailing module import.""" (tmp_path / "m.py").write_text( - "from typing import TYPE_CHECKING\n" - "\n" - "from a import X\n" - "\n" - "\n" - "def f():\n" - " pass\n" + "from typing import TYPE_CHECKING\n\nfrom a import X\n\n\ndef f():\n pass\n" ) r = Repro("b", "t").add_typechecking_import("m.py", "from b import Y") _apply(r, tmp_path) @@ -215,12 +209,7 @@ def test_add_typechecking_import_raises_without_imports(tmp_path: Path) -> None: def test_add_typechecking_import_drops_a_lone_pass_placeholder(tmp_path: Path) -> None: """Populating a `pass`-only TYPE_CHECKING block replaces the placeholder.""" (tmp_path / "m.py").write_text( - "from typing import TYPE_CHECKING\n" - "\n" - "if TYPE_CHECKING:\n" - " pass\n" - "\n" - "x = 1\n" + "from typing import TYPE_CHECKING\n\nif TYPE_CHECKING:\n pass\n\nx = 1\n" ) r = Repro("b", "t").add_typechecking_import("m.py", "from b import Y") _apply(r, tmp_path) diff --git a/.claude/skills/mechanical-refactor-verify/scripts/tests/reproduction_utils/test_extract_symbols_to_new_module.py b/.claude/skills/mechanical-refactor-verify/scripts/tests/reproduction_utils/test_extract_symbols_to_new_module.py index 26d9fae2fa7b..7e0b1ebbfd74 100644 --- a/.claude/skills/mechanical-refactor-verify/scripts/tests/reproduction_utils/test_extract_symbols_to_new_module.py +++ b/.claude/skills/mechanical-refactor-verify/scripts/tests/reproduction_utils/test_extract_symbols_to_new_module.py @@ -111,11 +111,7 @@ def test_extract_symbols_to_new_module_drops_relocated_assigns(tmp_path: Path) - " return _FLAG\n" ) header = ( - "from __future__ import annotations\n" - "\n" - "import os\n" - "\n" - "_FLAG = os.cpu_count()\n" + "from __future__ import annotations\n\nimport os\n\n_FLAG = os.cpu_count()\n" ) r = Repro("b", "t").extract_symbols_to_new_module( "src.py", diff --git a/.claude/skills/mechanical-refactor-verify/scripts/tests/reproduction_utils/test_move_assign.py b/.claude/skills/mechanical-refactor-verify/scripts/tests/reproduction_utils/test_move_assign.py index 762eab97fd43..1ba4e192ea64 100644 --- a/.claude/skills/mechanical-refactor-verify/scripts/tests/reproduction_utils/test_move_assign.py +++ b/.claude/skills/mechanical-refactor-verify/scripts/tests/reproduction_utils/test_move_assign.py @@ -19,13 +19,7 @@ def test_move_assign_relocates_a_module_constant(tmp_path: Path) -> None: _apply(r, tmp_path) assert "LIMIT" not in (tmp_path / "src.py").read_text().split("def stay")[0] assert (tmp_path / "dst.py").read_text() == ( - "import sys\n" - "\n" - "LIMIT = 480 # seconds\n" - "\n" - "\n" - "def keep():\n" - " return 1\n" + "import sys\n\nLIMIT = 480 # seconds\n\n\ndef keep():\n return 1\n" ) @@ -50,13 +44,7 @@ def test_move_assign_relocates_an_annotated_constant(tmp_path: Path) -> None: _apply(r, tmp_path) assert "LIMIT" not in (tmp_path / "src.py").read_text().split("def stay")[0] assert (tmp_path / "dst.py").read_text() == ( - "import sys\n" - "\n" - "LIMIT: int = 480\n" - "\n" - "\n" - "def keep():\n" - " return 1\n" + "import sys\n\nLIMIT: int = 480\n\n\ndef keep():\n return 1\n" ) diff --git a/.claude/skills/mechanical-refactor-verify/scripts/tests/reproduction_utils/test_move_symbol.py b/.claude/skills/mechanical-refactor-verify/scripts/tests/reproduction_utils/test_move_symbol.py index a5f74a6876d1..fc808a9982ed 100644 --- a/.claude/skills/mechanical-refactor-verify/scripts/tests/reproduction_utils/test_move_symbol.py +++ b/.claude/skills/mechanical-refactor-verify/scripts/tests/reproduction_utils/test_move_symbol.py @@ -260,13 +260,7 @@ def test_move_symbol_dedent_leaves_string_literal_interior_lines( ) _apply(r, tmp_path) assert (tmp_path / "dst.py").read_text() == ( - "import os\n" - "\n" - "def foo(self):\n" - " s = '''raw\n" - " partial\n" - "'''\n" - " return s\n" + "import os\n\ndef foo(self):\n s = '''raw\n partial\n'''\n return s\n" ) diff --git a/.claude/skills/sglang-prod-incident-triage/scripts/incident_artifact_tool.py b/.claude/skills/sglang-prod-incident-triage/scripts/incident_artifact_tool.py index 2a9dacc69a63..fdc4aa875421 100755 --- a/.claude/skills/sglang-prod-incident-triage/scripts/incident_artifact_tool.py +++ b/.claude/skills/sglang-prod-incident-triage/scripts/incident_artifact_tool.py @@ -101,8 +101,7 @@ def format_summary_line(filename: str, result: Dict[str, Any]) -> str: if result.get("ok"): return f"{filename}: ok" return ( - f"{filename}: failed status={result.get('status')} " - f"error={result.get('error')}" + f"{filename}: failed status={result.get('status')} error={result.get('error')}" ) @@ -617,7 +616,9 @@ def summarize_dump_file(path: Path, max_requests: int, preview_chars: int) -> st time_span = ( max(timestamps) - min(timestamps) if len(timestamps) >= 2 - else 0.0 if len(timestamps) == 1 else None + else 0.0 + if len(timestamps) == 1 + else None ) lines = [ diff --git a/.claude/skills/sglang-runtime-context/SKILL.md b/.claude/skills/sglang-runtime-context/SKILL.md index 16de40f65fd1..6733281bc2b9 100644 --- a/.claude/skills/sglang-runtime-context/SKILL.md +++ b/.claude/skills/sglang-runtime-context/SKILL.md @@ -47,8 +47,11 @@ with what the operator typed, not with what resolution decided.** compilation disabled before restricting a role), and `=enforce` fails closed on bag reads outside the role's `ROLE_NAMESPACE_SETS` entry (`None` = full tree; only audited roles are restricted). -- Bag membership is metadata on the dataclass: every `ServerArgs` field carries - `NS("path")` (e.g. `NS("exec.moe")`); coverage is linted two-way +- Bag membership is **where the field is declared**: one class per namespace under + `arg_groups/fields/`, each carrying the `_NS_PATH` it stands for, and `ServerArgs` + is assembled from them (`collect_input_fields`). The per-field `NS("path")` marker + survives only for a class that cannot express this — an ad-hoc dataclass spanning + namespaces, which is what the config-bag tests build. Coverage is linted two-way (`test_server_args_namespaces.py`, `test_runtime_context_config_bags.py`). - **Reading config**: `get_()[.sub].field` — e.g. `get_exec().moe.moe_a2a_backend`, `get_schedule().max_running_requests`. Bag leaves @@ -70,11 +73,9 @@ with what the operator typed, not with what resolution decided.** `TokenizerManager.record_config_updates(source, **fields)`, a named wrapper over `get_context().override`. One process keeps one log: the request dumps ship `get_context().overrides_log()`, and `config_value(name)` / - `resolved_config_dict(base)` answer from the bags. The exposure ratchet - resolves the wrapper, so a field recorded through it joins the post-publish - override surface exactly like a direct `override` and needs the same ordering - judgment against any supplied-instance read of it - (`test_supplied_instance_exposure_ratchet.py`). + `resolved_config_dict(base)` answer from the bags. Fields recorded through + the wrapper follow the same ordering rules as a direct `override`: a later + read must observe the updated bag, not the startup record. - **`model_path` and `served_model_name` are answered off the manager.** Both are `NS` leaves and `override` accepts them, but the tokenizer-side weight reload records only `load_format` and writes the two path fields as `TokenizerManager` @@ -88,8 +89,11 @@ with what the operator typed, not with what resolution decided.** - **Late launcher-stage resolution (pre-publish)**: a few rules cannot run inside `__post_init__` — LoRA normalization, and the auto-parser detection that needs a tokenizer/chat-template load. They are resolution, not mutation, and they - **declare** via `arg_groups.overrides.declare_late_resolution(server_args, - source, **fields)`, which refuses the published instance. The declaration lands + **declare** via `arg_groups.overrides.declare_resolution(server_args, source, + **fields)`, the same call the rest of the pipeline makes; there is no + `declare_late_resolution` any more. *When* a declaration is made is not + something the code marks — the guardrails that used to read that marker + cover these sites through the ordinary keyword scan instead. The declaration lands in the stash on that very object, so every holder of it carries the decision — the HTTP server, the multi-tokenizer workers it is serialized for, the schedulers it forks — and each of them publishes bags projected from it. The @@ -148,8 +152,7 @@ bag to override at all. **retracted** — owner ruling (2026-08-15): a process holds at most one live config at a time (concurrent multi-Engine is unsupported; sequential rebuild stays legal, unit tests rely on it). Nothing in those files reads the instance - any more -- the exposure ratchet's pin set is empty, so the next such read is a - new entry that has to argue for itself. What + any more; review new instance reads against the raw-input contract. What genuinely stays per-instance is what differs per *worker* within one engine: `base_gpu_id` travels as a constructor argument (`MMEncoder(gpu_id=...)`; `BaseMultimodalProcessor._fast_image_processor_device` is the shape to copy). @@ -166,12 +169,9 @@ bag to override at all. attention pair and the encode-server `gpu_id` above are both this). The per-instance boundaries above are **not** exempt from this unless-clause (the multi-Engine exemption is retracted); each one gets its own disposition. - `test_supplied_instance_exposure_ratchet.py` - pins that set (empty today) — three spellings of the read: `server_args.field`, - literal-name `getattr(server_args, "field", default)`, and the parked form - (`self.x = server_args` in a method that takes the parameter, read as - `self.x.field` anywhere in the class) — and fails on a new one, so the - disposition gets picked when the read is written. Two shapes stay parameter-form on purpose: a helper the + Check direct attributes, `getattr`, and records stored on `self`; validate + the resolved value and any later overrides in behavior tests. + Two shapes stay parameter-form on purpose: a helper the *resolution pipeline* calls with a `resolved_view` (its parameter happens to be named `server_args`), and a factory whose contract is "build X from the record you are handed" (`create_kt_config_from_server_args`, `DllmConfig.from_server_args`). @@ -253,10 +253,18 @@ A process-global seed field-read of one of these sizes (`get_server_args().tp_size`, or an alias of it) is a read-ratchet failure. A `server_args` the object was *handed* is a different thing and not a ratchet matter — see "Reads that legitimately stay on a ServerArgs instance". -Fail-loud is narrower: before dist init, a live size/group read raises — except -the DCP pair, which degrades instead (`dcp_enabled` → `False`, -`attn_dcp_size` → `1` when no group is installed; -`test_attn_dcp_defaults_when_group_is_uninitialized` pins this). After init, +Fail-loud is narrower: before dist init, a live *rank/group* read raises. The six +parallel quotients are not live reads at all — `attn_tp_size`, `attn_dp_size`, +`attn_dcp_size`, `moe_ep_size`, `moe_tp_size`, `dcp_enabled` are a function of the +configured leaves, computed once at publish into bag leaves, and answered +override → stamp → published leaf. So `dcp_enabled` means "the launch configured +DCP" (`dcp_size > 1`), not "a DCP group is installed here"; in a scheduler the +stamp makes the two identical, in a process that publishes without dist init they +differ. `test_a_topology_is_stated_by_naming_the_width` and its neighbours in +`test_runtime_context.py` pin this; they replaced +`test_attn_dcp_defaults_when_group_is_uninitialized`. One consequence for tests: +overriding a leaf no longer moves its quotient — state a topology by publishing a +config, or by naming the width. After init, only the DCP group is optional (`_DCP` exists only when `dcp_size > 1`; attn-CP and moe-DP always install, as size-1 aliases if unused). The `config` hop is deliberately dynamo-traceable (a plain property over a slot, no @@ -283,13 +291,21 @@ where an object was handed one; it is not a global accessor. raises on a non-leaf. A call site that knows its field reads the bag leaf. - **the live topology** → `get_parallel()` (bare names). - **a value derived from published leaves** → an accessor in `runtime_context` that - derives it *from the bags*: `mamba_extra_buffer_enabled()` / - `mamba_extra_buffer_lazy_enabled()` read `get_memory()` and `get_exec()`, so - they see post-publish overrides. Prefer this shape whenever the inputs are - leaves; the same-named `ServerArgs` members are the pre-publish equivalents the - resolution pipeline uses, and wrapping one of those instead would quietly cost - you override visibility. `is_ep_joiner()` / `is_ep_scale_joiner()` are the same - shape over `exec.moe.ep_join_mode`, `attention_backends()` derives the + derives it *from the bags*. The strongest form of this is a `Derived(fn=...)` + declared beside the leaves it is computed from, in the namespace's own + `arg_groups/fields/` class: `publish` computes it once and stores it as an + ordinary bag leaf, so the read is a plain attribute load and it sees + post-publish overrides. `enable_mamba_extra_buffer`, `is_ep_joiner`, + `is_ep_scale_joiner` and `is_startup_weight_load_overlap` are declared that way + now — read them where they are declared: + `get_exec().mamba.enable_mamba_extra_buffer`, `get_exec().moe.is_ep_joiner`, + `get_model().is_startup_weight_load_overlap`. (The namespace is the class that + declares the field, not the namespaces its `fn` happens to read: the mamba one + spans `exec.mamba` and `memory`, which is exactly why it could not be a method + on either bag.) The + old `mamba_extra_buffer_enabled()` / `is_ep_joiner()` functions and the + same-named `ServerArgs` members are gone. The pre-publish helpers that remain + exist for resolution, which has no bag to read yet. `attention_backends()` derives the `(prefill, decode)` pair from the three `exec.kernel` leaves, and `max_speculative_num_draft_tokens()` / `cutedsl_moe_max_num_tokens()` derive theirs from `spec` / `schedule` / `exec.graph`. @@ -380,18 +396,45 @@ through a view instead: - `resolved_view(server_args)` — snapshots the overlay when built, which is what a post-process pass wants: it reads the state at *its* slot. -`test_resolution_reads_the_declarations` pins direct field reads at zero over the -two scopes it can derive exactly (every `arg_groups` function taking a config, -every `ServerArgs` handler the dispatcher reaches). Readers the pipeline calls -from elsewhere (`ModelConfig`, the platform defaults, the spec-algo hook) have -moved to the view as well — a field read there is the same bug, just one the -derivation cannot enumerate. +Resolution hooks and the helpers they call (`ModelConfig`, platform defaults, +the spec-algo hook) must read through these views too. Keep coverage in +`test_resolution_declarations.py`, `test_resolution_is_reproducible.py`, and +`test_record_holds_the_raw_input.py` focused on the values callers observe. One consequence worth knowing: because the fields are the raw input, resolving a bare `dataclasses.replace` copy lands in the same place as the parent — the -pipeline reads only its own input. `replace_resolved` is the way to copy a -resolved record (it carries the declarations and the `model_config` memo, so the -copy does not re-resolve at all). +pipeline reads only its own input. **So a resolved record is not copied at +all.** A caller that needs one field different for the process it is about to +hand the record to — the Ray paths and their `dist_init_addr` — declares it on +the record it holds (`declare_resolution`) and hands that over: the declaration +travels inside the object, the receiving process projects its bags from it, and +nothing re-resolves. There is no `ServerArgs.replace_resolved` any more, and the +`model_config`-memo bug that copying used to cause (a copy marked resolved but +arriving without the memo cannot refill it, because the guard refuses the write) +is gone by construction rather than guarded. + +A bag `override` cannot stand in for this. It is *not* because overriding needs +a publish — `set_server_args` is what projects the bags and `override` works as +soon as the context holds a record — but because `override` writes bag leaves +and by contract never touches the record, so its effect cannot travel inside an +object to another process. + +### The declaration stash has one writer + +Everything that decides configuration goes through +`declare_resolution(server_args, source, **fields)`. It validates the names, +refuses the published config (the stash is projected at publish and never +again, so a later declaration is a silent no-op), and appends. The other names +around it are spellings, not mechanisms: + +| name | what it adds | +|---|---| +| `run_post_process_pass` | runs a pass at its slot and validates its return; declares through `declare_resolution`. A pass returning an **empty** dict is a validation, not a declaration, and stays legal on the published instance — `Engine(server_args=sa)` after `Engine.shutdown()` re-runs `check_server_args` on the very instance the context holds | +| `record_foreign_defaults` | for a resolver this tree does not own (an out-of-tree platform plugin, a registered speculative algorithm), whose interface is to *assign* fields. It gets a stand-in whose reads fall through to `resolving_view`; what it assigned is declared. The record is never written, so the write seal has no exception. In-tree code does not go through it — `handle_platform_defaults` wraps the platform hook, and the in-tree speculative dispatcher is called directly, because handed the stand-in its own `declare_resolution` calls would stash on that instead | + +`resolution_projection` is gone; the whole-object readback is +`ServerArgs.resolved_dict()`, which is what `/server_info` and its gRPC and +in-process twins report. ### Adding a model-specific config adjustment @@ -512,8 +555,10 @@ ONE thread — do not design for TBO threads that don't exist. ## Guardrails (these fail CI; what to do when they fire) -1. **Strict mutation guard** (always on): bare `server_args.x = ...` after resolution - raises unconditionally in `ServerArgs.__setattr__` — this *is* the guarantee that +1. **Strict mutation guard** (always on, and with no exception): bare + `server_args.x = ...` after resolution raises unconditionally in + `ServerArgs.__setattr__` — the named lift that out-of-tree plugins used to + ask for is gone, they assign onto a stand-in instead — this *is* the guarantee that no writer can desync the bags, so there is no writer ratchet any more. Change resolved config with `get_context().override`; hand a per-runner value to its runner as a constructor argument. Projected bags are sealed the same way (leaf @@ -525,10 +570,12 @@ ONE thread — do not design for TBO threads that don't exist. `ServerArgs.override` nor `ServerArgs.derive` exists, and nothing in the package calls either form. Rerouting a writer to the bags means flipping **all its readers in the same commit** (no transitional dual-write). -4. **Legacy-accessor ratchet** (`test_legacy_global_ratchet.py`): `get_global_server_args` - call sites must not grow. The replacement for a *decision* is a bag leaf, a named - accessor, or the owning runner's stamp — not `get_server_args().field`, which the - read ratchet below pins at zero. `runtime_context.get_server_args()` is only for the +4. **The legacy accessor is retired** (`test_runtime_context.py`): every + `get_global_server_args()` call now raises, because it answered with the record -- + a caller reading a field resolution had decided got a stale value and no error. + The replacement for a *decision* is a bag leaf, a named accessor, or the owning + runner's stamp — not `get_server_args().field`, which the read ratchet below pins + at zero. `runtime_context.get_server_args()` is only for the whole-object shapes (dumps, provenance, a hand-off to a callee that takes a config). 5. **Global config read ratchet** (`test_global_config_read_ratchet.py`): baselines are **0** for both the direct `get_server_args().field` and the alias form (function-local @@ -548,8 +595,9 @@ ONE thread — do not design for TBO threads that don't exist. flag-owning layers are pinned by name. A new module-level runtime global belongs on a flags group / resources slot instead; migrating a pinned survivor must shrink the pin. 7. **Namespace coverage** (`test_server_args_namespaces.py`, - `test_runtime_context_config_bags.py`): every `ServerArgs` field carries `NS(...)` - metadata and the projected bags must cover the fields exactly (two-way). + `test_runtime_context_config_bags.py`): every `ServerArgs` field resolves to a + namespace — from the `arg_groups/fields/` class that declares it — and the + projected bags must cover the fields exactly (two-way). Never module-skip a test "until the migration settles" — seed the context instead (the deferral ratchet that once pinned this is retired; the rule stands). @@ -599,10 +647,10 @@ Never module-skip a test "until the migration settles" — seed the context inst Key source files: `python/sglang/srt/runtime_context.py` (the container, every tier, `publish`, `_ConfigBag`, `override_server_args`), `python/sglang/srt/arg_groups/overrides.py` (override registry, passes, -`declare_late_resolution`), `python/sglang/srt/server_args.py` (`NS` metadata, +`declare_resolution` and the spellings around it), `python/sglang/srt/server_args.py` (`NS` metadata, `Arg(..., resolvable=True)`, `__setattr__` strict guard), and the guardrail tests under `test/registered/unit/` (`test_server_args_mutation_ratchet.py`, -`test_global_config_read_ratchet.py`, `test_legacy_global_ratchet.py`, +`test_global_config_read_ratchet.py`, `test_module_state_ratchet.py`, `test_server_args_namespaces.py`, `test_runtime_context.py` — the last one doubles as executable documentation of every tier's semantics). diff --git a/.claude/skills/write-sglang-test/SKILL.md b/.claude/skills/write-sglang-test/SKILL.md index 1faaf2d9b197..c932523dd098 100644 --- a/.claude/skills/write-sglang-test/SKILL.md +++ b/.claude/skills/write-sglang-test/SKILL.md @@ -5,20 +5,34 @@ description: Guide for writing SGLang CI/UT tests. Covers CustomTestCase, CI reg # Writing SGLang CI / UT Tests -This skill covers **how to write and register tests**. For CI pipeline internals (stage ordering, fast-fail, gating, partitioning, debugging CI failures), see the [CI workflow guide](../ci-workflow-guide/SKILL.md). +This skill covers **how to write and register tests**. For CI pipeline internals (stage ordering, fail-fast, gating, partitioning, debugging CI failures), see the [CI workflow guide](../ci-workflow-guide/SKILL.md). Whether a case is worth adding at all is decided by [`unit-test-admission`](../../rules/unit-test-admission.md) — read it before writing the case, not after. ## Core Rules 1. **Always use `CustomTestCase`** — never raw `unittest.TestCase`. It ensures `tearDownClass` runs even when `setUpClass` fails, preventing resource leaks in CI. -2. **`tearDownClass` must be defensive** — use `hasattr`/null checks before accessing resources (e.g. `cls.process`) that `setUpClass` may not have finished allocating. -3. **Place tests in `test/registered//`** — including JIT kernel tests and benchmarks, which live in `test/registered/jit/` and `test/registered/jit/benchmark/` (nested subfolders are allowed) +2. **`tearDownClass` must shut the server down gracefully** — call `terminate_and_kill_process_tree(cls.process)`, never a bare `kill_process_tree`. SIGKILL alone skips the server's userspace cleanup and leaves its GPU memory charged to the dead process; the next class then OOMs while loading weights. Keep it defensive too: `hasattr`/null checks before accessing resources (e.g. `cls.process`) that `setUpClass` may not have finished allocating. +3. **Place non-kernel tests in `test/registered///`** — `` is `unit`, `e2e`, `accuracy`, `perf`, or `stress`; kernel tests use `test/registered/kernels/{ops,benchmark}//`; hardware belongs in registrations, not directory names 4. **Reuse server fixtures** — inherit from `DefaultServerBase` or write `setUpClass`/`tearDownClass` with `popen_launch_server` -5. **Prefer mock over real server** — when testing logic that doesn't need a server / engine launch (middleware, request routing, config validation, argument parsing), use `unittest.mock.patch` / `MagicMock` and place tests in `test/registered/unit/`. Only launch a real server when the test genuinely needs inference results or server lifecycle behavior. +5. **Mock boundaries, not SGLang behavior** — mock slow or external dependencies only when the assertion still checks an observable result, state transition, or error. A test whose evidence is only `assert_called*` mirrors its mock and is not admissible. Launch a real server only when inference results or lifecycle behavior are the contract under test. + +> **Existing files are not the reference.** About 290 test files still call the bare +> `kill_process_tree(cls.process.pid)`, against 43 on the current helper. They predate +> rule 2 and are being migrated, so grepping the repo for a teardown pattern finds the +> wrong one roughly seven times out of eight. The same goes for `register_cuda_ci(suite=...)`: +> four files still pass it, all of them under `test/registered/stress/`. + +```python +# Bad: kill_process_tree(cls.process.pid) # SIGKILL only; GPU memory lingers +# Good: terminate_and_kill_process_tree(cls.process) + +# Bad: register_cuda_ci(est_time=80, suite="base-b-test-1-gpu-small") +# Good: register_cuda_ci(est_time=80, stage="base-b", runner_config="1-gpu-small") +``` JIT kernel notes: - If the task is adding or updating code under `python/sglang/kernels/jit/`, prefer the `add-jit-kernel` skill first. -- JIT kernel correctness tests use `test/registered/jit/**/test_*.py`. -- JIT kernel benchmarks use `test/registered/jit/benchmark/**/bench_*.py`. +- JIT kernel correctness tests use `test/registered/kernels/ops//test_*.py`. +- JIT kernel benchmarks use `test/registered/kernels/benchmark//bench_*.py`. - Those files are executed by `test/run_suite.py` through dedicated kernel suites (`base-b-kernel-*`); a `register_*_ci(...)` call placed under `python/sglang/` is rejected by the `check-no-registered-tests-in-package` pre-commit hook. --- @@ -27,11 +41,11 @@ JIT kernel notes: | Scenario | Model | CI Registration | Suite | |----------|-------|-----------------|-------| -| **Unit tests** (no server / engine launch) | None | `register_cpu_ci` (prefer) or `register_cuda_ci` | `base-a-test-cpu` or `base-b-test-1-gpu-small` | +| **Unit tests** (no server / engine launch) | None | `register_cpu_ci` | `base-a-test-cpu` | | **Common / backend-independent** (middleware, abort, routing, config, arg parsing) | `DEFAULT_SMALL_MODEL_NAME_FOR_TEST` (1B) | `register_cuda_ci` only | `base-b-test-1-gpu-small` | | **Model-agnostic functionality** (sampling, session, OpenAI API features) | `DEFAULT_SMALL_MODEL_NAME_FOR_TEST` (1B) | `register_cuda_ci` (+ AMD if relevant) | `base-b-test-1-gpu-small` | | **General performance** (single node, no spec/DP/parallelism) | `DEFAULT_MODEL_NAME_FOR_TEST` (8B) | `register_cuda_ci` | `base-b-test-1-gpu-large` | -| **Bigger features** (spec, DP, TP, disaggregation) | Case by case | Case by case | See suite table below | +| **Bigger features** (spec, DP, TP, disaggregation) | Case by case | Case by case | See **Choosing a Suite** below | **Key principle for E2E tests**: Do NOT add `register_amd_ci` unless the test specifically exercises AMD/ROCm code paths. Common E2E tests just need any GPU to run — duplicating across backends wastes CI time with no extra coverage. @@ -60,74 +74,23 @@ A per-commit suite name is **generated** from registration metadata as `{stage}- ### All CI Suites -#### Per-commit (CUDA) - -| Suite | Runner (label) | Description | -|-------|----------------|-------------| -| `base-a-test-1-gpu-small` | `1-gpu-5090` | Quick checks on a small NVIDIA GPU before heavier stages | -| `base-a-test-cpu` | `ubuntu-latest` | CPU-only unit tests | -| `base-b-test-1-gpu-small` | `1-gpu-5090` | Core engine tests that fit a 5090-class card | -| `base-b-test-1-gpu-large` | `1-gpu-h100` | Tests that need H100-class memory or kernels (e.g. FA3) | -| `base-b-test-2-gpu-large` | `2-gpu-h100` | Two-GPU correctness and parallelism (TP/PP) on H100 | -| `base-b-test-4-gpu-b200` | `4-gpu-b200` | Early Blackwell coverage (SM100+ paths) on four GPUs | -| `base-b-kernel-unit-test-1-gpu-large` | `1-gpu-h100` | JIT kernel correctness tests under `test/registered/jit/` | -| `base-b-kernel-unit-test-4-gpu-b200` | `4-gpu-b200` | JIT kernel correctness tests for Blackwell / SM100-specific paths | -| `base-b-kernel-unit-test-8-gpu-h200` | `8-gpu-h200` | Multi-GPU JIT kernel correctness tests under `test/registered/jit/` | -| `base-b-kernel-benchmark-test-1-gpu-large` | `1-gpu-h100` | JIT kernel benchmark files under `test/registered/jit/benchmark/` | -| `base-c-test-4-gpu-h100` | `4-gpu-h100` | Large 4-GPU H100 integration and scaling tests | -| `base-c-test-8-gpu-h200` | `8-gpu-h200` | Large 8-GPU H200 runs for big models and parallelism | -| `base-c-test-8-gpu-h20` | `8-gpu-h20` | Large 8-GPU H20 runs for big models | -| `base-c-test-deepep-4-gpu-h100` | `4-gpu-h100` | DeepEP expert-parallel and networking on four H100s | -| `base-c-test-8-gpu-b200` | `8-gpu-b200` | 8-GPU B200 suite (registered but not yet wired to a workflow) | -| `base-c-test-4-gpu-b200` | `4-gpu-b200` | 4-GPU B200 suite for large models on Blackwell | -| `base-c-test-4-gpu-b200-small` | `4-gpu-b200` | Smaller 4-GPU B200 suite split onto low-disk B200 runners | -| `base-c-test-4-gpu-gb200` | `4-gpu-gb200` | 4-GPU GB200 suite for Grace Blackwell; registered in `run_suite.py`, but the PR workflow is currently disabled until a runner is provisioned | - -#### Per-commit (AMD) - -| Suite | Runner (label) | Description | -|-------|----------------|-------------| -| `stage-a-test-1-gpu-small-amd` | `linux-mi325-1gpu-sglang` | Quick checks on one MI325-class GPU | -| `stage-b-test-1-gpu-small-amd` | `linux-mi325-1gpu-sglang` | Core 1-GPU AMD tests (14 partitions) | -| `stage-b-test-1-gpu-small-amd-nondeterministic` | `linux-mi325-1gpu-sglang` | Non-deterministic 1-GPU AMD tests | -| `stage-b-test-1-gpu-small-amd-mi35x` | `linux-mi35x-gpu-1` | 1-GPU tests on MI35x hardware | -| `stage-b-test-1-gpu-large-amd` | `linux-mi325-1gpu-sglang` | Large 1-GPU AMD tests (2 partitions) | -| `stage-b-test-2-gpu-large-amd` | `linux-mi325-2gpu-sglang` | 2-GPU ROCm correctness and parallel setups | -| `stage-b-test-large-8-gpu-mi35x-disaggregation-amd` | `linux-mi35x-gpu-8.fabric` | PD disaggregation and RDMA on 8×MI35x fabric | -| `stage-c-test-4-gpu-amd` | `linux-mi325-4gpu-sglang` | 4-GPU AMD integration (2 partitions) | -| `stage-c-test-large-8-gpu-amd` | `linux-mi325-8gpu-sglang` | 8-GPU MI325 scaling and integration | -| `stage-c-test-large-8-gpu-amd-mi35x` | `linux-mi35x-gpu-8` | 8-GPU MI35x scaling (2 partitions) | - - -### Per-commit (Ascend NPU) - -| Suite | Runner (label) | Description | -| --- | --- | --- | -| `per-commit-1-npu-a2` | `linux-aarch64-a2-1` | 1-NPU LLM CI machine | -| `per-commit-2-npu-a2` | `linux-aarch64-a2-2` | 2-NPU LLM CI machine | -| `per-commit-4-npu-a3` | `linux-aarch64-a3-4` | 4-NPU LLM CI machine | -| `per-commit-16-npu-a3` | `linux-aarch64-a3-16` | 16-NPU LLM CI machine | -| `multimodal-gen-test-1-npu-a3` | `linux-aarch64-a3-2` | 1-NPU multimodal CI machine | -| `multimodal-gen-test-2-npu-a3` | `linux-aarch64-a3-16` | 2-NPU multimodal CI machine | -| `multimodal-gen-test-8-npu-a3` | `linux-aarch64-a3-16` | 8-NPU multimodal CI machine | - -#### Nightly - -Nightly suites are listed in `NIGHTLY_SUITES` in [`test/run_suite.py`](../../../test/run_suite.py). They run via `nightly-test-nvidia.yml`, `nightly-test-amd.yml`, and `nightly-test-npu.yml`, not `pr-test.yml`. - -CUDA nightly suites are named `nightly-test-{runner_config}` — one per machine type, holding everything that runs nightly on it. There is no per-purpose split (kernel / eval / perf / precision all share their machine's suite); `auto_partition` splits the work. Examples: - -- `nightly-test-1-gpu-large` (CUDA) -- `nightly-test-2-gpu-large` (CUDA) -- `nightly-test-8-gpu-h200` (CUDA) -- `nightly-test-4-gpu-gb300` (CUDA) -- `nightly-amd` (AMD) -- `nightly-amd-8-gpu-mi35x` (AMD) -- `nightly-1-npu-a3` (NPU) -- `nightly-2-npu-a3` (NPU) -- `nightly-4-npu-a3` (NPU) -- `nightly-8-npu-a3` (NPU) -- `nightly-16-npu-a3` (NPU) +Do not work from a list copied into this file; it goes stale silently. Read the +current one: + +```bash +grep -n "_SUITES = {" test/run_suite.py # PER_COMMIT_SUITES, NIGHTLY_SUITES, OTHER_SUITES +cat scripts/ci/runner_configs.yml # runner_config -> physical runner label +``` + +`scripts/ci/runner_configs.yml` calls itself the single source of truth for the +`runner_config` field, and `run_suite.py` is what actually dispatches, so those two +files settle any disagreement with prose anywhere else. + +Nightly suites live in `NIGHTLY_SUITES` and run via `nightly-test-nvidia.yml`, +`nightly-test-amd.yml`, and `nightly-test-npu.yml`, not `pr-test.yml`. CUDA nightly is +named `nightly-test-{runner_config}` — one suite per machine type, holding everything +that runs nightly on it, with `auto_partition` splitting the work. There is no +per-purpose split; kernel, eval, perf, and precision all share their machine's suite. > **Note**: Multimodal diffusion uses `python/sglang/multimodal_gen/test/run_suite.py`, not `test/run_suite.py`. @@ -162,7 +125,7 @@ from sglang.test.ci.ci_register import register_cpu_ci from sglang.test.test_utils import CustomTestCase register_cpu_ci(est_time=5, suite="base-a-test-cpu") -# Prefer CPU. Only use register_cuda_ci when the test truly needs a GPU. +# Unit tests are CPU-only. GPU operator tests belong under `kernel/`. class TestTargetClass(CustomTestCase): def test_basic_behavior(self): @@ -180,7 +143,13 @@ if __name__ == "__main__": unittest.main() ``` -Use `unittest.mock.patch` / `MagicMock` to mock dependencies and isolate the logic under test. If the module transitively imports GPU-only packages (e.g. `sgl_kernel`), they can be stubbed so the test runs on CPU CI. Do not modify `sys.modules` at module level — use `patch.dict` (as a class decorator or with `start`/`stop`) to ensure cleanup and avoid cross-test pollution. See `test/registered/unit/README.md` for details and examples. +Use `unittest.mock.patch` / `MagicMock` only at dependency boundaries. Assert the +resulting value, state, protocol output, or error—not merely that the mock was +called. If the module transitively imports GPU-only packages (e.g. `sgl_kernel`), +they can be stubbed so the test runs on CPU CI. Do not modify `sys.modules` at +module level—use `patch.dict` (as a class decorator or with `start`/`stop`) to +ensure cleanup and avoid cross-test pollution. See +`test/registered/unit/README.md` for details and examples. **Quality bar** — test real logic (validation boundaries, state transitions, error paths, branching, etc.). Skip tests that just verify Python itself works (e.g., "does calling an abstract method raise `NotImplementedError`?", "does a dataclass store the field I assigned?"). Consolidate repetitive patterns into parameterized tests. No production code changes in test PRs. @@ -191,7 +160,6 @@ import unittest import requests -from sglang.srt.utils import kill_process_tree from sglang.test.ci.ci_register import register_cuda_ci from sglang.test.test_utils import ( DEFAULT_SMALL_MODEL_NAME_FOR_TEST, @@ -199,9 +167,10 @@ from sglang.test.test_utils import ( DEFAULT_URL_FOR_TEST, CustomTestCase, popen_launch_server, + terminate_and_kill_process_tree, ) -register_cuda_ci(est_time=60, suite="base-b-test-1-gpu-small") +register_cuda_ci(est_time=60, stage="base-b", runner_config="1-gpu-small") class TestMyFeature(CustomTestCase): @@ -219,7 +188,7 @@ class TestMyFeature(CustomTestCase): @classmethod def tearDownClass(cls): if hasattr(cls, "process") and cls.process: - kill_process_tree(cls.process.pid) + terminate_and_kill_process_tree(cls.process) def test_basic_functionality(self): response = requests.post( @@ -233,6 +202,9 @@ if __name__ == "__main__": unittest.main(verbosity=3) ``` +Copy the `tearDownClass` above verbatim. Most existing E2E files still show the bare +`kill_process_tree`; that form is being migrated out and must not be reproduced. + ### E2E test (8B model, server needed, performance) ```python @@ -241,7 +213,6 @@ import unittest import requests -from sglang.srt.utils import kill_process_tree from sglang.test.ci.ci_register import register_cuda_ci from sglang.test.test_utils import ( DEFAULT_MODEL_NAME_FOR_TEST, @@ -249,9 +220,10 @@ from sglang.test.test_utils import ( DEFAULT_URL_FOR_TEST, CustomTestCase, popen_launch_server, + terminate_and_kill_process_tree, ) -register_cuda_ci(est_time=300, suite="base-b-test-1-gpu-large") +register_cuda_ci(est_time=300, stage="base-b", runner_config="1-gpu-large") class TestMyFeaturePerf(CustomTestCase): @@ -268,7 +240,7 @@ class TestMyFeaturePerf(CustomTestCase): @classmethod def tearDownClass(cls): if hasattr(cls, "process") and cls.process: - kill_process_tree(cls.process.pid) + terminate_and_kill_process_tree(cls.process) def test_latency(self): start = time.perf_counter() @@ -326,26 +298,29 @@ from sglang.test.ci.ci_register import ( ) # Per-commit test (small 1-gpu, runs on 5090) -register_cuda_ci(est_time=80, suite="base-b-test-1-gpu-small") +register_cuda_ci(est_time=80, stage="base-b", runner_config="1-gpu-small") # Per-commit test (large 1-gpu, runs on H100) -register_cuda_ci(est_time=120, suite="base-b-test-1-gpu-large") +register_cuda_ci(est_time=120, stage="base-b", runner_config="1-gpu-large") # Nightly-only test (same shape as per-commit, stage is just "nightly") register_cuda_ci(est_time=200, stage="nightly", runner_config="1-gpu-large") # Multi-backend test (only when testing backend-specific code paths) -register_cuda_ci(est_time=80, suite="base-a-test-1-gpu-small") +register_cuda_ci(est_time=80, stage="base-a", runner_config="1-gpu-small") register_amd_ci(est_time=120, suite="stage-a-test-1-gpu-small-amd") register_npu_ci(est_time=400, suite="nightly-8-npu-a3", nightly=True) # Temporarily disabled test -register_cuda_ci(est_time=80, suite="base-b-test-1-gpu-small", disabled="flaky - see #12345") +register_cuda_ci( + est_time=80, stage="base-b", runner_config="1-gpu-small", disabled="flaky - see #12345" +) ``` Parameters: - `est_time`: estimated runtime in seconds (used for CI partitioning) -- `suite`: which CI suite to run in (see suite tables above) +- `stage` + `runner_config`: the canonical pair for CUDA; the suite name is generated from them (see Naming Conventions) +- `suite`: legacy single-string form. Only `stress` and some AMD/CPU/NPU pools still take it; `register_cpu_ci(suite="base-a-test-cpu")` is correct and is not being migrated - `nightly=True`: legacy cadence flag, for non-CUDA nightly suites only. CUDA nightly uses `stage="nightly"` and must leave this unset - `disabled="reason"`: temporarily disable with explanation @@ -353,17 +328,18 @@ Parameters: ### JIT Kernel Registration -JIT kernel files live outside `test/registered/` but still use registration: +`run_suite.py` discovers every `test/registered/**/*.py`, JIT kernel files included. +They are ordinary registered tests; only their stage differs: ```python from sglang.test.ci.ci_register import register_cuda_ci -# Correctness tests in test/registered/jit/ +# Correctness tests in test/registered/kernels/ops// register_cuda_ci(est_time=30, stage="base-b-kernel-unit", runner_config="1-gpu-large") register_cuda_ci(est_time=30, stage="base-b-kernel-unit", runner_config="4-gpu-b200") register_cuda_ci(est_time=120, stage="base-b-kernel-unit", runner_config="8-gpu-h200") -# Benchmarks in test/registered/jit/benchmark/ +# Benchmarks in test/registered/kernels/benchmark// register_cuda_ci(est_time=6, stage="base-b-kernel-benchmark", runner_config="1-gpu-large") # Optional nightly registration — same form, stage is just "nightly" @@ -380,28 +356,28 @@ Every call generates a suite named `{stage}-test-{runner_config}`, e.g. `base-b- ``` test/ ├── registered/ # CI tests (auto-discovered by run_suite.py) -│ ├── unit/ # No server / engine launch (see test/registered/unit/README.md) -│ ├── kernels/ # CUDA kernel correctness (no server, GPU required) -│ ├── sampling/ # test_penalty.py, test_sampling_params.py ... -│ ├── sessions/ # test_session_control.py ... -│ ├── openai_server/ # basic/, features/, validation/ ... -│ ├── spec/ # eagle/, utils/ ... -│ ├── models/ # model-specific accuracy tests -│ ├── perf/ # performance benchmarks -│ └── / # create new category if needed +│ ├── unit// # CPU-only; no server or model weights +│ ├── kernel// # accelerator operator correctness/benchmarks +│ ├── e2e// # engine/server integration +│ ├── accuracy// # scheduled eval floors +│ ├── perf// # scheduled latency/throughput contracts +│ └── stress// # stress/weekly coverage ├── manual/ # Non-CI: debugging, one-off, manual verification -└── run_suite.py # CI runner (scans registered/ plus jit_kernel test/benchmark files) +└── run_suite.py # CI runner (globs test/registered/**/*.py; nothing outside it) -python/sglang/kernels/jit/ -├── tests/ # JIT kernel correctness tests (CI-discovered by test/run_suite.py) -└── benchmark/ # JIT kernel benchmarks (CI-discovered by test/run_suite.py) +python/sglang/kernels/jit/ # implementation + test-only helpers, never registered tests ``` +A `register_*_ci(...)` under `python/sglang/` is rejected by the +`check-no-registered-tests-in-package` pre-commit hook. + **Decision rule** (see also `test/registered/README.md`): -- Component logic, no server → `registered/unit/` -- JIT kernel correctness / benchmarks → `test/registered/jit/` or `test/registered/jit/benchmark/` -- Other kernel correctness → `registered/kernels/` -- Server needed → `registered//` +- CPU component logic, no server → `registered/unit//` +- JIT kernel correctness → `registered/kernels/ops//` +- JIT kernel benchmarks → `registered/kernels/benchmark//` +- Other accelerator operator correctness → `registered/kernels/ops//` +- Server needed → `registered/e2e//` +- Eval floor / performance contract → `registered/{accuracy,perf}//` - Local debugging → `manual/` --- @@ -428,11 +404,12 @@ class TestMyFeature(CustomTestCase, MMLUMixin): from sglang.test.test_utils import ( CustomTestCase, # base class with retry logic popen_launch_server, # launch server subprocess + terminate_and_kill_process_tree, # SIGTERM, then SIGKILL, then wait for + # the GPU memory to come back DEFAULT_URL_FOR_TEST, # auto-configured base URL DEFAULT_TIMEOUT_FOR_SERVER_LAUNCH, # 600s default run_bench_serving, # benchmark helper (launch + bench) ) -from sglang.srt.utils import kill_process_tree # cleanup server ``` --- @@ -443,11 +420,19 @@ Before submitting a test: - [ ] Inherits from `CustomTestCase` (not `unittest.TestCase`) - [ ] Has `register_*_ci(...)` call at module level -- [ ] Placed in `test/registered//` (JIT kernel test/benchmark → `test/registered/jit/` or `test/registered/jit/benchmark/`) -- [ ] JIT kernel work: test files live in `test/registered/jit/`; only test-only helpers stay under `python/sglang/kernels/jit/` +- [ ] Placed in `test/registered///` +- [ ] JIT kernel work: correctness tests live in `test/registered/kernels/ops//`, benchmarks live in `test/registered/kernels/benchmark//`, and only test-only helpers stay under `python/sglang/kernels/jit/` - [ ] Backend-independent tests: `register_cuda_ci` only + smallest model - [ ] Logic that doesn't need a server / engine launch → unit test in `registered/unit/` (see Unit Tests section) -- [ ] `setUpClass` launches server, `tearDownClass` kills it (if server-based) - [ ] `tearDownClass` is defensive — uses `hasattr`/null checks before accessing resources that may not have been allocated -- [ ] Has `if __name__ == "__main__": unittest.main()` +- [ ] Every case answers "what future diff turns this red?" — see [`unit-test-admission`](../../rules/unit-test-admission.md) - [ ] `est_time` is reasonable (measure locally) + +Run these against the new file and paste the output rather than self-attesting: + +```bash +f= +grep -n "kill_process_tree" $f # every hit must be terminate_and_kill_process_tree +grep -n "register_.*_ci(" $f # CUDA: stage= + runner_config=, never suite= +grep -n "CustomTestCase\|unittest.main" $f # both must appear +``` diff --git a/.codespellrc b/.codespellrc index e6d52acd3b6b..715171d21d76 100644 --- a/.codespellrc +++ b/.codespellrc @@ -1,3 +1,3 @@ [codespell] -ignore-words-list = ans, als, hel, boostrap, childs, te, vas, hsa, ment, cann, thi, makro, wil, rouge, PRIS, ather, MIS, medias, allready, inout, nd, fo, visibles, nothink, renderD, ond, tbe, CopyIn, notin, subtile, subtiles, dout, IST, kInf, datas +ignore-words-list = ans,als,hel,boostrap,childs,te,vas,hsa,ment,cann,thi,makro,wil,rouge,PRIS,ather,MIS,medias,allready,inout,nd,fo,visibles,nothink,renderD,ond,tbe,CopyIn,notin,subtile,subtiles,dout,IST,kInf,datas,abl,valu skip = *.json, *.jsonl, *.patch, *.txt, *.lock diff --git a/.git-blame-ignore-revs b/.git-blame-ignore-revs new file mode 100644 index 000000000000..eac93721ef67 --- /dev/null +++ b/.git-blame-ignore-revs @@ -0,0 +1,5 @@ +# Skipped by `git blame`. Enable locally: +# git config blame.ignoreRevsFile .git-blame-ignore-revs + +# [CI][RFC] Replace black-jupyter with ruff-format (#37210) +28262c20df6f2945c1b1cd7b27b0faef59cb0438 diff --git a/.github/CI_PERMISSIONS.json b/.github/CI_PERMISSIONS.json index 01d102904e85..217a5ab7f63d 100644 --- a/.github/CI_PERMISSIONS.json +++ b/.github/CI_PERMISSIONS.json @@ -95,6 +95,12 @@ "cooldown_interval_minutes": 0, "reason": "top contributor" }, + "Denji-kk": { + "can_tag_run_ci_label": true, + "can_rerun_failed_ci": true, + "cooldown_interval_minutes": 0, + "reason": "custom override" + }, "DevashishLal-CB": { "can_tag_run_ci_label": true, "can_rerun_failed_ci": true, @@ -278,7 +284,7 @@ "Shunkangz": { "can_tag_run_ci_label": true, "can_rerun_failed_ci": true, - "cooldown_interval_minutes": 60, + "cooldown_interval_minutes": 0, "reason": "custom override" }, "SimonCqk": { @@ -725,6 +731,12 @@ "cooldown_interval_minutes": 0, "reason": "top contributor" }, + "iyastreb": { + "can_tag_run_ci_label": true, + "can_rerun_failed_ci": true, + "cooldown_interval_minutes": 0, + "reason": "custom override" + }, "jason-fxz": { "can_tag_run_ci_label": true, "can_rerun_failed_ci": true, @@ -791,6 +803,13 @@ "cooldown_interval_minutes": 60, "reason": "custom override" }, + "kevin-mii": { + "can_tag_run_ci_label": true, + "can_rerun_failed_ci": true, + "can_rerun_stage": true, + "cooldown_interval_minutes": 0, + "reason": "custom override" + }, "kevin85421": { "can_tag_run_ci_label": true, "can_rerun_failed_ci": true, @@ -887,12 +906,24 @@ "cooldown_interval_minutes": 60, "reason": "custom override" }, + "lluki": { + "can_tag_run_ci_label": true, + "can_rerun_failed_ci": true, + "cooldown_interval_minutes": 0, + "reason": "custom override" + }, "luccafong": { "can_tag_run_ci_label": true, "can_rerun_failed_ci": true, "cooldown_interval_minutes": 60, "reason": "custom override" }, + "lucifer1004": { + "can_tag_run_ci_label": true, + "can_rerun_failed_ci": true, + "cooldown_interval_minutes": 60, + "reason": "custom override" + }, "maocheng23": { "can_tag_run_ci_label": true, "can_rerun_failed_ci": true, @@ -905,6 +936,12 @@ "cooldown_interval_minutes": 0, "reason": "top contributor" }, + "metamergebot": { + "can_tag_run_ci_label": true, + "can_rerun_failed_ci": true, + "cooldown_interval_minutes": 0, + "reason": "custom override" + }, "michaelzhang-ai": { "can_tag_run_ci_label": true, "can_rerun_failed_ci": true, @@ -953,6 +990,12 @@ "cooldown_interval_minutes": 0, "reason": "custom override" }, + "niehen6174": { + "can_tag_run_ci_label": true, + "can_rerun_failed_ci": true, + "cooldown_interval_minutes": 60, + "reason": "custom override" + }, "nvcastet": { "can_tag_run_ci_label": true, "can_rerun_failed_ci": true, @@ -989,6 +1032,12 @@ "cooldown_interval_minutes": 0, "reason": "custom override" }, + "ovidiusm": { + "can_tag_run_ci_label": true, + "can_rerun_failed_ci": true, + "cooldown_interval_minutes": 0, + "reason": "custom override" + }, "pansicheng": { "can_tag_run_ci_label": true, "can_rerun_failed_ci": true, @@ -1025,6 +1074,12 @@ "cooldown_interval_minutes": 0, "reason": "custom override" }, + "polisettyvarma": { + "can_tag_run_ci_label": true, + "can_rerun_failed_ci": true, + "cooldown_interval_minutes": 0, + "reason": "custom override" + }, "ppraneth": { "can_tag_run_ci_label": true, "can_rerun_failed_ci": true, @@ -1097,6 +1152,12 @@ "cooldown_interval_minutes": 0, "reason": "top contributor" }, + "rwang5203": { + "can_tag_run_ci_label": true, + "can_rerun_failed_ci": true, + "cooldown_interval_minutes": 0, + "reason": "custom override" + }, "saienduri": { "can_tag_run_ci_label": true, "can_rerun_failed_ci": true, @@ -1181,6 +1242,12 @@ "cooldown_interval_minutes": 0, "reason": "custom override" }, + "stepinto": { + "can_tag_run_ci_label": true, + "can_rerun_failed_ci": true, + "cooldown_interval_minutes": 0, + "reason": "custom override" + }, "stmatengss": { "can_tag_run_ci_label": true, "can_rerun_failed_ci": true, diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 5ce0a9f86730..08cccbf1dc13 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -5,9 +5,9 @@ /docs @wisclmy0611 @zijiexia @Richardczl98 @JustinTong0323 @sogalin /python/pyproject.toml @merrymercy @Fridge003 @ispobock /python/sglang/kernels @DarkSharpness @HaiShaw @BBuf @celve @HydraQYH @yuan-luo -/python/sglang/kernels/ops/diffusion @yingluosanqian @BBuf @mickqian @triple-mu +/python/sglang/kernels/ops/diffusion @yingluosanqian @BBuf @mickqian @triple-mu @niehen6174 /python/sglang/kernels/ops/attention/fla @yizhang2077 @hebiao064 @yuan-luo -/python/sglang/multimodal_gen @mickqian @ping1jing2 @HaiShaw @yichiche @AgainstEntropy @BBuf @triple-mu +/python/sglang/multimodal_gen @mickqian @ping1jing2 @HaiShaw @yichiche @AgainstEntropy @BBuf @triple-mu @kevin-mii @niehen6174 /python/sglang/multimodal_gen/runtime/cache @DefTruth /python/sglang/multimodal_gen/runtime/layers/quantization/ @HaiShaw @BowenBao @OrangeRedeng @TallMessiWu /python/sglang/multimodal_gen/test/server/ascend @ping1jing2 @ssshinigami @Makcum888e @e-martirosian @@ -18,9 +18,10 @@ /python/sglang/srt/disaggregation/ascend @ping1jing2 @iforgetmyname /python/sglang/srt/disaggregation/encoder @ShangmingCai @liusy58 @ZhengWG @gty111 /python/sglang/srt/disaggregation/mori @Duyi-Wang @kkHuang-amd @HaiShaw @Lzy17 @billishyahao +/python/sglang/srt/disaggregation/nixl @iyastreb @ovidiusm @lluki /python/sglang/srt/distributed @yizhang2077 @merrymercy @ch-wan /python/sglang/srt/distributed/device_communicators/mooncake_transfer_engine.py @ShangmingCai @stmatengss -/python/sglang/srt/dllm @ClawSeven @btw616 +/python/sglang/srt/dllm @ClawSeven @btw616 @rwang5203 /python/sglang/srt/entrypoints @ispobock @CatherineSue @slin1237 @merrymercy @JustinTong0323 /python/sglang/srt/entrypoints/anthropic @JustinTong0323 /python/sglang/srt/entrypoints/openai @JustinTong0323 @@ -38,18 +39,24 @@ /python/sglang/srt/layers @merrymercy @Ying1123 @Fridge003 @ispobock @HaiShaw @ch-wan @BBuf @Edwardf0t1 /python/sglang/srt/layers/attention @merrymercy @Fridge003 @ispobock @Qiaolin-Yu @hebiao064 @HaiShaw /python/sglang/srt/layers/attention/hybrid_linear_attn_backend.py @yizhang2077 @hebiao064 @hanming-lu @yuan-luo +/python/sglang/srt/layers/attention/linear @yizhang2077 @hebiao064 @yuan-luo @hanming-lu /python/sglang/srt/layers/attention/mamba @yizhang2077 @hebiao064 /python/sglang/srt/layers/attention/dsa @1am9trash @hubertlu-tw @kkHuang-amd @HaiShaw @Fridge003 @YAMY1234 @rainj-me +/python/sglang/srt/layers/attention/dsa/dsa_topk_backend.py @1am9trash @hubertlu-tw @kkHuang-amd @HaiShaw @Fridge003 @YAMY1234 @rainj-me @zianglih /python/sglang/srt/layers/attention/vision.py @mickqian @yuan-luo @yhyang201 /python/sglang/srt/layers/quantization @ch-wan @BBuf @Edwardf0t1 @FlamingoPg @AniZpZ @HaiShaw @b8zhong @OrangeRedeng @TamirBaydasov @Alisehen @mmangkad @TallMessiWu +/python/sglang/srt/layers/quantization/nvfp4_online.py @zianglih /python/sglang/srt/layers/quantization/quark @HaiShaw @kkHuang-amd @yichiche @hubertlu-tw @1am9trash @BowenBao /python/sglang/srt/lora @Ying1123 @Fridge003 @lifuhuang @yushengsu-thu @jybsuper /python/sglang/srt/managers @merrymercy @Ying1123 @hnyls2002 @xiezhq-hermann +/python/sglang/srt/managers/hisparse_coordinator.py @merrymercy @Ying1123 @hnyls2002 @xiezhq-hermann @hzh0425 @ispobock @alphabetc1 @huangtingwei9988 /python/sglang/srt/managers/scheduler_pp_mixin.py @ShangmingCai @XucSh /python/sglang/srt/managers/tokenizer_manager_score_mixin.py @sundar24295s @chanh @fortunecookiee /python/sglang/srt/mem_cache @merrymercy @Ying1123 @hnyls2002 @xiezhq-hermann @hanming-lu @yizhang2077 @hzh0425 @ispobock @alphabetc1 @huangtingwei9988 -/python/sglang/srt/mem_cache/allocator @hnyls2002 @ispobock @alphabetc1 -/python/sglang/srt/mem_cache/storage/mooncake_store @huangtingwei9988 +/python/sglang/srt/mem_cache/allocator @hnyls2002 @hzh0425 @xiezhq-hermann @ispobock @alphabetc1 @huangtingwei9988 +/python/sglang/srt/mem_cache/rust_tree_core @Jialin @hzh0425 @xiezhq-hermann @ispobock @alphabetc1 +/python/sglang/srt/mem_cache/storage/mooncake_store @huangtingwei9988 @stmatengss +/python/sglang/srt/mem_cache/storage/nixl @iyastreb @ovidiusm @lluki /python/sglang/srt/mem_cache/embedding_*.py @liusy58 /python/sglang/srt/mem_cache/storage/mooncake_store/mooncake_embedding_store.py @liusy58 /python/sglang/srt/model_executor @merrymercy @Ying1123 @hnyls2002 @Fridge003 @ispobock @@ -67,6 +74,8 @@ /python/sglang/srt/weight_cache @liusy58 @QiuMike @alexnails /python/sglang/kernels/aot @ispobock @BBuf @yizhang2077 @merrymercy @FlamingoPg @HaiShaw /python/sglang/kernels/aot/csrc/musa @yeahdongcn +/rust/sglang-radix-tree @Jialin @hzh0425 @xiezhq-hermann @ispobock @alphabetc1 +/rust/sglang-renderer @sagearc /sgl-model-gateway @slin1237 @CatherineSue /sgl-model-gateway/benches @slin1237 /sgl-model-gateway/bindings/python @CatherineSue @key4ng @slin1237 @@ -96,3 +105,4 @@ /python/sglang/kernels/jit/csrc/ngram_corpus @hnyls2002 @Qiaolin-Yu @kpham-sgl /python/sglang/srt/models/laguna.py @Jiminator /python/sglang/srt/configs/laguna.py @Jiminator +/tools/sglang-simulator @littlefatfat @hzh0425 @zhuzi-z @LinSiyuan814 diff --git a/.github/FOLDER_README.md b/.github/FOLDER_README.md index f987d4d92427..f951adfa80c3 100644 --- a/.github/FOLDER_README.md +++ b/.github/FOLDER_README.md @@ -12,8 +12,8 @@ Recognized permission keys: | Key | Grants | | --- | --- | -| `can_tag_run_ci_label` | `/tag-run-ci-label`, `/tag-and-rerun-ci` | -| `can_rerun_failed_ci` | `/rerun-failed-ci`, `/tag-and-rerun-ci` | +| `can_tag_run_ci_label` | `/tag-run-ci-label`, `/tag-and-rerun-ci`, `/run-full-ci`, `/run-extra-ci` | +| `can_rerun_failed_ci` | `/rerun-failed-ci`, `/tag-and-rerun-ci`, `/run-full-ci`, `/run-extra-ci` | | `cooldown_interval_minutes` | rate limit in `pr-gate.yml`; `0` also grants `/rerun-test`, `/rerun-group` | `/rerun-test` and `/rerun-group` are gated on the commenter alone: either diff --git a/.github/MAINTAINER.md b/.github/MAINTAINER.md index 4aac1716c0f5..42e863b48468 100644 --- a/.github/MAINTAINER.md +++ b/.github/MAINTAINER.md @@ -143,6 +143,17 @@ This section lists the oncalls for each hardware platform. The format is @github This list is based on the current situation. If you or someone you know would like to donate machines for CI, they can serve as the CI oncalls for their machines. Please ping [Lianmin Zheng](https://github.com/merrymercy) and [Ying Sheng](https://github.com/Ying1123) in the Slack channel. They will start a nomination and internal review process. +## CI Control Labels + +`bypass-fail-fast`, `parallel-stages`, `max-concurrency` and `highest-priority` +each relax one of the limits that keep a single PR from monopolizing the +self-hosted GPU runners; `highest-priority` relaxes all of them at once. The +[contribution guide](https://docs.sglang.io/developer_guide/contribution_guide.html#ci-control-labels) +describes what each one does. + +Applying one spends other PRs' runner capacity. `parallel-stages` is the +expensive one: a PR that cannot pass now runs its whole matrix. + ## CI Maintenance Mode When the CI is unhealthy (e.g., the scheduled pr-test on `main` is broken for consecutive runs), the project enters **CI Maintenance Mode** by opening [issue #21065](https://github.com/sgl-project/sglang/issues/21065). While active: - All PR CI runs are paused. Resources are allocated to PRs that fix the CI. diff --git a/.github/actions/check-pr-test-health/action.test.cjs b/.github/actions/check-pr-test-health/action.test.cjs new file mode 100644 index 000000000000..d31120f2cc9d --- /dev/null +++ b/.github/actions/check-pr-test-health/action.test.cjs @@ -0,0 +1,76 @@ +const assert = require('node:assert/strict'); +const fs = require('node:fs'); +const path = require('node:path'); +const { test } = require('node:test'); + +const yaml = fs.readFileSync(path.join(__dirname, 'action.yml'), 'utf8'); +const script = yaml.split(' script: |\n')[1] + .split('\n').map(line => line.replace(/^ /, '')).join('\n'); +// `require` and GITHUB_WORKSPACE mirror what actions/github-script injects. +const WORKSPACE = path.join(__dirname, '..', '..', '..'); +const run = new (Object.getPrototypeOf(async function () {}).constructor)( + 'github', 'context', 'core', 'process', 'require', script, +); + +async function check({ snapshot = [], live = [], lint = 'success', event = 'pull_request' } = {}) { + const failures = []; + let labelReads = 0; + let jobReads = 0; + const labels = names => names.map(name => ({ name })); + const github = { + rest: { + checks: { listForRef: async () => ({ data: { check_runs: [ + { app: { slug: 'github-actions' }, status: 'completed', conclusion: lint }, + ] } }) }, + pulls: { get: async args => { + assert.equal(args.pull_number, 42); + labelReads++; + return { data: { labels: labels(live) } }; + } }, + repos: { listPullRequestsAssociatedWithCommit: async () => { + labelReads++; + return { data: [{ labels: labels(live) }] }; + } }, + actions: { listJobsForWorkflowRun: () => {} }, + }, + paginate: async () => { + jobReads++; + return [{ name: 'model-test', status: 'completed', conclusion: 'failure', steps: [] }]; + }, + }; + await run(github, { + eventName: event, repo: { owner: 'owner', repo: 'repo' }, sha: 'head', runId: 1, + payload: event === 'pull_request' + ? { pull_request: { number: 42, head: { sha: 'head' }, labels: labels(snapshot) } } + : {}, + }, { info: () => {}, setFailed: message => failures.push(message) }, + { env: { GITHUB_WORKSPACE: WORKSPACE } }, require); + return { failures, labelReads, jobReads }; +} + +test('a label added after the event bypasses sibling failures on rerun', async () => { + assert.deepEqual(await check({ live: ['bypass-fail-fast'] }), + { failures: [], labelReads: 1, jobReads: 0 }); +}); + +test('a removed label does not continue bypassing sibling failures', async () => { + const result = await check({ snapshot: ['bypass-fail-fast'] }); + assert.equal(result.labelReads, 1); + assert.equal(result.jobReads, 1); + assert.match(result.failures[0], /root cause job\(s\): model-test/); +}); + +test('bypass never skips a failed lint check', async () => { + assert.deepEqual(await check({ live: ['bypass-fail-fast'], lint: 'failure' }), + { failures: ['Fail-fast: lint check failed'], labelReads: 0, jobReads: 0 }); +}); + +test('non-PR events retain associated-PR label lookup', async () => { + assert.deepEqual(await check({ event: 'workflow_dispatch', live: ['bypass-fail-fast'] }), + { failures: [], labelReads: 1, jobReads: 0 }); +}); + +test('scheduled runs remain exempt', async () => { + assert.deepEqual(await check({ event: 'schedule' }), + { failures: [], labelReads: 0, jobReads: 0 }); +}); diff --git a/.github/actions/check-pr-test-health/action.yml b/.github/actions/check-pr-test-health/action.yml index 8cafb66e96d4..4b1e42f79106 100644 --- a/.github/actions/check-pr-test-health/action.yml +++ b/.github/actions/check-pr-test-health/action.yml @@ -1,5 +1,5 @@ name: Check PR Test Health -description: Fail fast if any job in the current workflow run has already failed, or if the lint check (from lint.yml) has failed. Auto-skips for scheduled runs. The jobs-failed check (but not the lint check) is bypassed when the PR carries the `bypass-fastfail` label. +description: Fail fast if any job in the current workflow run has already failed, or if the lint check (from lint.yml) has failed. Auto-skips for scheduled runs. The jobs-failed check (but not the lint check) is bypassed when the PR carries the `bypass-fail-fast` label (or `highest-priority`, which implies it). inputs: github-token: @@ -17,15 +17,17 @@ runs: with: github-token: ${{ inputs.github-token }} script: | + core.info(`[health-check] START -- event=${context.eventName}, runId=${context.runId}`); + // Skip when explicitly requested via env var (e.g. release branch cut) if (process.env.SKIP_PR_TEST_HEALTH_CHECK === 'true') { - core.info('Skipping health check (SKIP_PR_TEST_HEALTH_CHECK=true)'); + core.info('[health-check] SKIP: SKIP_PR_TEST_HEALTH_CHECK=true'); return; } - // Skip for scheduled runs — they should collect all failures, not fast-fail + // Skip for scheduled runs -- they should collect all failures, not fail-fast if (context.eventName === 'schedule') { - core.info('Skipping health check for scheduled run'); + core.info('[health-check] SKIP: scheduled run'); return; } @@ -33,6 +35,7 @@ runs: // listJobsForWorkflowRun only sees jobs within the SAME run, so we use // checks.listForRef which queries by commit SHA across ALL workflows. const ref = context.payload.pull_request?.head?.sha || context.sha; + core.info(`[health-check] Checking lint for ref=${ref}`); const { data } = await github.rest.checks.listForRef({ owner: context.repo.owner, repo: context.repo.repo, @@ -42,28 +45,20 @@ runs: const lintRun = data.check_runs.find( cr => cr.app?.slug === 'github-actions' ); + core.info(`[health-check] Lint check: status=${lintRun?.status}, conclusion=${lintRun?.conclusion}`); if (lintRun?.status === 'completed' && lintRun?.conclusion === 'failure') { - core.setFailed('Fast-fail: lint check failed'); + core.setFailed('Fail-fast: lint check failed'); return; } - // Skip the jobs-failed check when the PR carries the bypass-fastfail label. - // Lint check above still runs. - let labels = []; - if (context.payload.pull_request?.labels) { - labels = context.payload.pull_request.labels.map(l => l.name); - } else { - const { data: prs } = await github.rest.repos.listPullRequestsAssociatedWithCommit({ - owner: context.repo.owner, - repo: context.repo.repo, - commit_sha: ref, - }); - if (prs.length > 0) { - labels = prs[0].labels.map(l => l.name); - } - } - if (labels.includes('bypass-fastfail')) { - core.info('Skipping jobs-failed check (bypass-fastfail label present)'); + // The lint check above is never bypassed; only sibling failures are. + const { resolveCiLabels } = require( + `${process.env.GITHUB_WORKSPACE}/.github/scripts/ci-labels.cjs` + ); + const axes = await resolveCiLabels(github, context); + core.info(`[health-check] PR labels: [${axes.labels.join(', ')}]`); + if (axes.bypassFailFast) { + core.info('[health-check] SKIP jobs-failed check: bypass-fail-fast label present'); return; } @@ -73,26 +68,41 @@ runs: run_id: context.runId, per_page: 100, }); - // Find jobs that failed from a real error, not from fast-fail cascade + core.info(`[health-check] Total jobs in run: ${jobs.length}`); + const failedJobs = jobs.filter(j => j.status === 'completed' && j.conclusion === 'failure'); + core.info(`[health-check] Failed jobs (before filtering): ${failedJobs.map(j => `${j.name}(${j.conclusion})`).join(', ') || 'none'}`); + + // Find jobs that failed from a real error, not from fail-fast cascade const rootCauseFailures = jobs.filter(j => { if (j.status !== 'completed' || j.conclusion !== 'failure') return false; // h20 runners are flaky (dirty GPU state from prior runs); their failures - // should not cascade fast-fail to other stages. j.name shape from + // should not cascade fail-fast to other stages. j.name shape from // listJobsForWorkflowRun: "" + optional " / " // + optional " ()". Split off the base job key before exact // match so we cover both inline + reusable forms without confusing // 'h20' with the 'h200' prefix. const baseName = j.name.split(/[ /]/)[0]; if (baseName === 'base-c-test-8-gpu-h20') { + core.info(`[health-check] Filtered out h20 job: ${j.name}`); + return false; + } + // multimodal-gen NPU tests should not cascade fail-fast to perf/accuracy stages. + if (baseName === 'multimodal-gen-test-1-npu-a3' || baseName === 'multimodal-gen-test-2-npu-a3') { + core.info(`[health-check] Filtered out multimodal-gen NPU job: ${j.name}`); return false; } - // If the failing step is the health check, it's a cascade — skip it + // If the failing step is the health check, it's a cascade -- skip it const failedStep = (j.steps || []).find(s => s.conclusion === 'failure'); if (failedStep && (failedStep.name.includes('check-pr-test-health') || failedStep.name.includes('Check PR test health'))) { + core.info(`[health-check] Filtered out cascade failure: ${j.name} (failed step: ${failedStep.name})`); return false; } return true; }); + core.info(`[health-check] Root cause failures (after filtering): ${rootCauseFailures.map(j => j.name).join(', ') || 'none'}`); + if (rootCauseFailures.length > 0) { - core.setFailed(`Fast-fail: skipping — root cause job(s): ${rootCauseFailures.map(j => j.name).join(', ')}`); + core.setFailed(`Fail-fast: skipping — root cause job(s): ${rootCauseFailures.map(j => j.name).join(', ')}`); + } else { + core.info('[health-check] PASS: no root cause failures detected'); } diff --git a/.github/actions/download-rust-ext/action.yml b/.github/actions/download-rust-ext/action.yml index 78893e1ad8ad..7809793cefcb 100644 --- a/.github/actions/download-rust-ext/action.yml +++ b/.github/actions/download-rust-ext/action.yml @@ -1,8 +1,9 @@ name: 'Download prebuilt Rust extensions' description: > Put rust-ext-build's PyO3 extension modules in the checkout and set - SGLANG_BUILD_RUST_EXTS=none for the job, so install skips the cargo build. - Sets nothing when neither source has them, leaving install to compile. + SGLANG_BUILD_RUST_EXTS=none and SGLANG_RUST_BUILD_MODE=never for the job, so + install skips the cargo build and runtime trusts these fingerprinted modules. + Sets nothing when neither source has them, leaving source builds enabled. inputs: artifact_name: @@ -49,8 +50,10 @@ runs: if: steps.artifact.outcome != 'success' uses: actions/cache/restore@v4 with: - path: python/sglang/srt/rust_extensions/_*.so - key: ${{ inputs.cache_key_prefix }}-${{ hashFiles('rust/**', 'python/setup.py') }} + path: | + python/sglang/srt/rust_extensions/_*.so + python/sglang/srt/mem_cache/rust_tree_core/mem_cache*.so + key: ${{ inputs.cache_key_prefix }}-${{ hashFiles('rust/**', 'proto/**', 'python/setup.py', 'python/pyproject.toml', 'python/sglang/srt/rust_extensions/torch_build.py', '.github/workflows/_pr-test-rust-ext-build.yml', 'scripts/ci/utils/stage_rust_ext_modules.sh') }} # Job-wide, but only setup.py reads it, and only while building. # Whether the modules suit this interpreter is not decided here: @@ -65,6 +68,9 @@ runs: || [ "${{ steps.cache.outputs.cache-hit }}" = "true" ]; then echo "hit=true" >> "$GITHUB_OUTPUT" echo "SGLANG_BUILD_RUST_EXTS=none" >> "$GITHUB_ENV" + # A source checkout normally ignores in-package native artifacts + # because they may be stale. These are fingerprint-keyed CI bytes. + echo "SGLANG_RUST_BUILD_MODE=never" >> "$GITHUB_ENV" else echo "hit=false" >> "$GITHUB_OUTPUT" fi diff --git a/.github/actions/wait-for-jobs/action.yml b/.github/actions/wait-for-jobs/action.yml index dab8fa364e1f..846bd9118b12 100644 --- a/.github/actions/wait-for-jobs/action.yml +++ b/.github/actions/wait-for-jobs/action.yml @@ -1,5 +1,5 @@ name: Wait for Jobs -description: Poll and wait for specified jobs in the current workflow run to complete. Returns success immediately when the PR carries the `bypass-fastfail` label, letting downstream stages dispatch in parallel (same effect as scheduled runs). +description: Poll and wait for specified jobs in the current workflow run to complete. Returns success immediately when the PR carries the `parallel-stages` label (or `highest-priority`, which implies it), letting downstream stages dispatch in parallel (same effect as scheduled runs). inputs: stage-name: @@ -49,28 +49,12 @@ runs: const pollIntervalSeconds = parseInt(process.env.INPUT_POLL_INTERVAL_SECONDS); const maxAttempts = (maxWaitMinutes * 60) / pollIntervalSeconds; - // bypass-fastfail label opts the PR out of stage-to-stage waiting, - // letting all stages dispatch in parallel like scheduled runs do. - let labels = []; - if (context.payload.pull_request?.labels) { - labels = context.payload.pull_request.labels.map(l => l.name); - } else { - const ref = context.payload.pull_request?.head?.sha || context.sha; - try { - const { data: prs } = await github.rest.repos.listPullRequestsAssociatedWithCommit({ - owner: context.repo.owner, - repo: context.repo.repo, - commit_sha: ref, - }); - if (prs.length > 0) { - labels = prs[0].labels.map(l => l.name); - } - } catch (e) { - console.log(`Could not fetch PR labels for ${ref}: ${e.message}`); - } - } - if (labels.includes('bypass-fastfail')) { - console.log(`Skipping ${stageName} wait (bypass-fastfail label present)`); + const { resolveCiLabels } = require( + `${process.env.GITHUB_WORKSPACE}/.github/scripts/ci-labels.cjs` + ); + const { parallelStages } = await resolveCiLabels(github, context); + if (parallelStages) { + console.log(`Skipping ${stageName} wait (parallel-stages label present)`); core.setOutput('result', 'success'); return; } @@ -182,15 +166,15 @@ runs: if (matchingJobs.length < spec.expected_count) { // Job-level `if:` is evaluated before matrix expansion. When it // evaluates false, GitHub emits exactly one "skipped" entry - // without a "(shard)" suffix instead of N matrix entries. Two - // shapes are possible: + // instead of N matrix entries. Shapes include: // - inline matrix: `` // - reusable workflow: ` / ` - // Both lack the ` (` shard marker. Detect that precise shape so - // we don't poll forever — and so we don't mistake a partially - // materialized dynamic/reusable matrix for a skipped one. + // Reusable workflows can also retain an unevaluated matrix + // expression in the shard suffix. Recognize that placeholder, + // while still waiting for missing shards of an expanded matrix. const unexpandedSkip = matchingJobs.length === 1 && - !matchingJobs[0].name.includes(' (') && + (!matchingJobs[0].name.includes(' (') || + /\(\$\{\{\s*matrix\.partition\s*\}\}\)$/.test(matchingJobs[0].name)) && matchingJobs[0].status === 'completed' && matchingJobs[0].conclusion === 'skipped'; if (unexpandedSkip) { @@ -198,7 +182,7 @@ runs: totalCount += missing; completedCount += missing; if (!cached) { - console.log(`${spec.prefix}: job-level skip (bare entry, conclusion=skipped); treating as all ${spec.expected_count} skipped`); + console.log(`${spec.prefix}: job-level skip (unexpanded entry, conclusion=skipped); treating as all ${spec.expected_count} skipped`); } } else { console.log(`${spec.prefix}: found ${matchingJobs.length}/${spec.expected_count} jobs (waiting for more)`); diff --git a/.github/labeler.yml b/.github/labeler.yml index 2f1a9291583e..377b148ef222 100644 --- a/.github/labeler.yml +++ b/.github/labeler.yml @@ -92,7 +92,6 @@ blackwell: - changed-files: - any-glob-to-any-file: - '**/*nvfp4*' - - 'python/sglang/kernels/aot/csrc/attention/cutlass_sm100_mla/**/*' - 'python/sglang/srt/layers/attention/trtllm_mla_backend.py' - 'python/sglang/srt/layers/attention/trtllm_mha_backend.py' @@ -115,7 +114,6 @@ unified-radix-cache: - changed-files: - any-glob-to-any-file: - 'python/sglang/srt/mem_cache/unified_cache/**/*' - - 'python/sglang/srt/mem_cache/cpp_radix_tree/**/*' - 'python/sglang/srt/mem_cache/**/*radix*' - 'python/sglang/srt/mem_cache/chunk_cache.py' - 'python/sglang/srt/mem_cache/evict_policy.py' @@ -126,7 +124,6 @@ hicache: - changed-files: - any-glob-to-any-file: - '**/*hicache*' - - '**/*hiradix*' - '**/HiCache/**/*' - 'benchmark/hicache/**/*' - 'test/registered/hicache/**/*' diff --git a/.github/scripts/ci-labels.cjs b/.github/scripts/ci-labels.cjs new file mode 100644 index 000000000000..81b1cb55f2f6 --- /dev/null +++ b/.github/scripts/ci-labels.cjs @@ -0,0 +1,62 @@ +"use strict"; + +/** + * Labels come from the API, not `context.payload`: a rerun replays the original + * event, so the payload carries the label set from when the run was created. + */ + +const BYPASS_FAIL_FAST = "bypass-fail-fast"; +const PARALLEL_STAGES = "parallel-stages"; +const MAX_CONCURRENCY = "max-concurrency"; +const HIGHEST_PRIORITY = "highest-priority"; + +// Callers gate the whole run on this, so a transient API error must not fail them. +async function readLabels(github, context) { + try { + return await fetchLabels(github, context); + } catch (e) { + console.warn(`Could not read PR labels: ${e.message}`); + return []; + } +} + +async function fetchLabels(github, context) { + const prNumber = context.payload.pull_request?.number; + if (prNumber) { + const { data: pr } = await github.rest.pulls.get({ + owner: context.repo.owner, + repo: context.repo.repo, + pull_number: prNumber, + }); + return pr.labels.map((l) => l.name); + } + const sha = context.payload.pull_request?.head?.sha || context.sha; + const { data: prs } = + await github.rest.repos.listPullRequestsAssociatedWithCommit({ + owner: context.repo.owner, + repo: context.repo.repo, + commit_sha: sha, + }); + return prs.length > 0 ? prs[0].labels.map((l) => l.name) : []; +} + +async function resolveCiLabels(github, context) { + const labels = await readLabels(github, context); + const has = (name) => labels.includes(name); + const highestPriority = has(HIGHEST_PRIORITY); + return { + labels, + bypassFailFast: highestPriority || has(BYPASS_FAIL_FAST), + parallelStages: highestPriority || has(PARALLEL_STAGES), + maxConcurrency: highestPriority || has(MAX_CONCURRENCY), + highestPriority, + }; +} + +module.exports = { + resolveCiLabels, + BYPASS_FAIL_FAST, + PARALLEL_STAGES, + MAX_CONCURRENCY, + HIGHEST_PRIORITY, +}; diff --git a/.github/workflows/_docker-build-and-publish.yml b/.github/workflows/_docker-build-and-publish.yml index 6a84a7b19451..22f21ef703ae 100644 --- a/.github/workflows/_docker-build-and-publish.yml +++ b/.github/workflows/_docker-build-and-publish.yml @@ -1,7 +1,7 @@ name: Build and Publish Multi-Arch Docker Images -# Reusable workflow: builds CUDA 12 + CUDA 13 images for amd64 and arm64, -# then creates multi-arch manifests with caller-specified tags. +# Reusable workflow: builds CUDA 13 images for amd64 and arm64, then creates +# multi-arch manifests with caller-specified tags. on: workflow_call: @@ -26,7 +26,7 @@ on: type: string default: "" tag_config: - description: 'JSON array of {"cuda":"cu129|cu130","tags":["tag1","tag2"]}. Tags support {version} substitution.' + description: 'JSON array of {"cuda":"cu130","tags":["tag1","tag2"]}. Tags support {version} substitution.' required: true type: string use_environment: @@ -50,7 +50,6 @@ jobs: SGL_VERSION: ${{ inputs.sgl_version }} IMAGE_REPO: ${{ inputs.image_repo }} outputs: - digest-cu129: ${{ steps.build-cu129.outputs.digest }} digest-cu130: ${{ steps.build-cu130.outputs.digest }} steps: - name: Delete huge unnecessary tools folder @@ -69,16 +68,14 @@ jobs: set -euo pipefail BUILD_COMMIT="$(git rev-parse HEAD)" BUILD_URL="${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}" - for CUDA_VARIANT in cu129 cu130; do - python3 scripts/ci/utils/docker_build_metadata_args.py \ - --cuda "${CUDA_VARIANT}" \ - --tag-config "${TAG_CONFIG}" \ - --image-repo "${IMAGE_REPO}" \ - --sgl-version "${SGL_VERSION}" \ - --build-commit "${BUILD_COMMIT}" \ - --build-url "${BUILD_URL}" \ - > "/tmp/docker-metadata-${CUDA_VARIANT}.args" - done + python3 scripts/ci/utils/docker_build_metadata_args.py \ + --cuda cu130 \ + --tag-config "${TAG_CONFIG}" \ + --image-repo "${IMAGE_REPO}" \ + --sgl-version "${SGL_VERSION}" \ + --build-commit "${BUILD_COMMIT}" \ + --build-url "${BUILD_URL}" \ + > /tmp/docker-metadata-cu130.args - name: Free disk space uses: jlumbroso/free-disk-space@main @@ -106,34 +103,6 @@ jobs: username: ${{ secrets.DOCKERHUB_USERNAME }} password: ${{ secrets.DOCKERHUB_TOKEN }} - - name: Build and push AMD64 image (CUDA 12) - id: build-cu129 - run: | - VERSION_ARG="" - if [ -n "${SGL_VERSION}" ]; then - VERSION_ARG="--build-arg SGL_VERSION=${SGL_VERSION}" - fi - mapfile -t METADATA_ARGS < /tmp/docker-metadata-cu129.args - - docker buildx build \ - --target ${{ inputs.docker_target }} \ - --platform linux/amd64 \ - --output type=image,name=${{ inputs.image_repo }},push-by-digest=true,name-canonical=true,push=true \ - -f docker/Dockerfile \ - --build-arg CUDA_VERSION=12.9.2 \ - --build-arg BUILD_TYPE=all \ - --build-arg INSTALL_FLASHINFER_JIT_CACHE=1 \ - "${METADATA_ARGS[@]}" \ - ${VERSION_ARG} \ - ${{ inputs.extra_build_args }} \ - --metadata-file /tmp/metadata-cu129.json \ - --no-cache \ - . - - DIGEST=$(python3 -c "import json; print(json.load(open('/tmp/metadata-cu129.json'))['containerimage.digest'])") - echo "Pushed digest: ${DIGEST}" - echo "digest=${DIGEST}" >> $GITHUB_OUTPUT - - name: Build and push AMD64 image (CUDA 13) id: build-cu130 run: | @@ -171,7 +140,6 @@ jobs: SGL_VERSION: ${{ inputs.sgl_version }} IMAGE_REPO: ${{ inputs.image_repo }} outputs: - digest-cu129: ${{ steps.build-cu129.outputs.digest }} digest-cu130: ${{ steps.build-cu130.outputs.digest }} steps: - name: Delete huge unnecessary tools folder @@ -190,16 +158,14 @@ jobs: set -euo pipefail BUILD_COMMIT="$(git rev-parse HEAD)" BUILD_URL="${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}" - for CUDA_VARIANT in cu129 cu130; do - python3 scripts/ci/utils/docker_build_metadata_args.py \ - --cuda "${CUDA_VARIANT}" \ - --tag-config "${TAG_CONFIG}" \ - --image-repo "${IMAGE_REPO}" \ - --sgl-version "${SGL_VERSION}" \ - --build-commit "${BUILD_COMMIT}" \ - --build-url "${BUILD_URL}" \ - > "/tmp/docker-metadata-${CUDA_VARIANT}.args" - done + python3 scripts/ci/utils/docker_build_metadata_args.py \ + --cuda cu130 \ + --tag-config "${TAG_CONFIG}" \ + --image-repo "${IMAGE_REPO}" \ + --sgl-version "${SGL_VERSION}" \ + --build-commit "${BUILD_COMMIT}" \ + --build-url "${BUILD_URL}" \ + > /tmp/docker-metadata-cu130.args - name: Prune Docker to reclaim disk space run: | @@ -216,34 +182,6 @@ jobs: username: ${{ secrets.DOCKERHUB_USERNAME }} password: ${{ secrets.DOCKERHUB_TOKEN }} - - name: Build and push ARM64 image (CUDA 12) - id: build-cu129 - run: | - VERSION_ARG="" - if [ -n "${SGL_VERSION}" ]; then - VERSION_ARG="--build-arg SGL_VERSION=${SGL_VERSION}" - fi - mapfile -t METADATA_ARGS < /tmp/docker-metadata-cu129.args - - docker buildx build \ - --target ${{ inputs.docker_target }} \ - --platform linux/arm64 \ - --output type=image,name=${{ inputs.image_repo }},push-by-digest=true,name-canonical=true,push=true \ - -f docker/Dockerfile \ - --build-arg CUDA_VERSION=12.9.2 \ - --build-arg BUILD_TYPE=all \ - --build-arg INSTALL_FLASHINFER_JIT_CACHE=1 \ - "${METADATA_ARGS[@]}" \ - ${VERSION_ARG} \ - ${{ inputs.extra_build_args }} \ - --metadata-file /tmp/metadata-cu129.json \ - --no-cache \ - . - - DIGEST=$(python3 -c "import json; print(json.load(open('/tmp/metadata-cu129.json'))['containerimage.digest'])") - echo "Pushed digest: ${DIGEST}" - echo "digest=${DIGEST}" >> $GITHUB_OUTPUT - - name: Build and push ARM64 image (CUDA 13) id: build-cu130 run: | @@ -292,22 +230,13 @@ jobs: TAG_CONFIG: ${{ inputs.tag_config }} SGL_VERSION: ${{ inputs.sgl_version }} IMAGE_REPO: ${{ inputs.image_repo }} - X86_CU129: ${{ needs.build-x86.outputs.digest-cu129 }} X86_CU130: ${{ needs.build-x86.outputs.digest-cu130 }} - ARM64_CU129: ${{ needs.build-arm64.outputs.digest-cu129 }} ARM64_CU130: ${{ needs.build-arm64.outputs.digest-cu130 }} SHORT_SHA: ${{ github.sha }} run: | echo "${TAG_CONFIG}" | jq -c '.[]' | while read -r entry; do - CUDA=$(echo "${entry}" | jq -r '.cuda') - - if [ "${CUDA}" = "cu129" ]; then - X86_DIGEST="${X86_CU129}" - ARM64_DIGEST="${ARM64_CU129}" - else - X86_DIGEST="${X86_CU130}" - ARM64_DIGEST="${ARM64_CU130}" - fi + X86_DIGEST="${X86_CU130}" + ARM64_DIGEST="${ARM64_CU130}" TAG_ARGS="" for tag in $(echo "${entry}" | jq -r '.tags[]'); do diff --git a/.github/workflows/_npu-analyze-failure.yml b/.github/workflows/_npu-analyze-failure.yml new file mode 100644 index 000000000000..f801f45575c0 --- /dev/null +++ b/.github/workflows/_npu-analyze-failure.yml @@ -0,0 +1,96 @@ +# Reusable workflow: cross-reference CI test failures with coverage-based +# test recommendations. Called by pr-test-npu.yml's analyze-failure-report job. +name: Analyze Failure Report + +on: + workflow_call: + inputs: + log_artifact_pattern: + type: string + required: true + description: 'Glob pattern for test log artifacts (e.g. selected-test-logs-*)' + recommendations_content: + type: string + required: false + default: '' + description: 'Content of recommendations from upstream job output' + +defaults: + run: + shell: bash -el {0} + +jobs: + analyze: + name: cross-reference failures with recommendations + continue-on-error: true # Non-blocking on parse/read failure. + runs-on: ubuntu-latest + steps: + - name: Checkout sglang repo + uses: actions/checkout@v4 + + # ----- Step 1: Locate recommendations ----- + - name: Locate recommendations file + id: locate-recs + run: | + mkdir -p ./test-logs + + if [ -n "${{ inputs.recommendations_content }}" ]; then + echo "has_recommendations=true" >> $GITHUB_OUTPUT + echo "Using workflow output recommendations" + echo "${{ inputs.recommendations_content }}" > ./test-logs/_recommended.txt + echo "RECS_FILE=./test-logs/_recommended.txt" >> $GITHUB_ENV + echo "RECS_SOURCE=output" >> $GITHUB_ENV + COUNT=$(wc -l < ./test-logs/_recommended.txt) + echo "Recommendations count: ${COUNT}" + else + echo "has_recommendations=false" >> $GITHUB_OUTPUT + echo "::notice::No recommended test cases detected, skipping subsequent analysis tasks." + fi + + # ----- Step 2: Download test logs ----- + - name: Download test logs + uses: actions/download-artifact@v7 + with: + pattern: ${{ inputs.log_artifact_pattern }} + path: ./test-logs + merge-multiple: true + continue-on-error: true + + - name: Show downloaded logs structure + run: | + echo "Downloaded log files:" + find ./test-logs -type f | head -50 || echo " (no logs found)" + + # ----- Step 3: Run analysis ----- + - name: Ensure regex is available + if: steps.locate-recs.outputs.has_recommendations != 'false' + run: | + set -euo pipefail + if ! python3 -c "import regex" >/dev/null 2>&1; then + echo "regex not found for $(python3 -V); bootstrapping pip and installing regex" + curl -fsSL https://bootstrap.pypa.io/get-pip.py -o get-pip.py + python3 get-pip.py --break-system-packages + python3 -m pip install --break-system-packages regex + fi + python3 -c "import regex; print('regex ok:', regex.__file__)" + + - name: Run failure analysis + if: steps.locate-recs.outputs.has_recommendations != 'false' + id: analysis + continue-on-error: true + run: | + python3 scripts/ci/npu/precise-test/analyze_failure_report.py \ + --log-dir ./test-logs \ + --recommendations-file "${RECS_FILE}" \ + --recommendations-source "${RECS_SOURCE:-none}" \ + --output ./test-logs/failure_report.md + + # ----- Step 4: Upload report ----- + - name: Upload failure analysis report + if: always() + uses: actions/upload-artifact@v7 + with: + name: failure-analysis-report + path: ./test-logs/failure_report.md + if-no-files-found: ignore + retention-days: 14 diff --git a/.github/workflows/_npu-pr-test-stage.yml b/.github/workflows/_npu-pr-test-stage.yml index d36933d8d2d7..ee08a54fe248 100644 --- a/.github/workflows/_npu-pr-test-stage.yml +++ b/.github/workflows/_npu-pr-test-stage.yml @@ -1,5 +1,7 @@ name: PR Test Stage for NPU -# Reusable workflow for one CUDA test stage. Caller pr-test-npu.yml forwards +# Reusable workflow for one NPU test stage. Caller pr-test-npu.yml forwards +# job parameters (runner, image, partitions) and optionally delegates test +# execution to the coverage runner (use_coverage_runner=true). on: workflow_call: @@ -17,7 +19,7 @@ on: type: string required: true npu_device_type: - description: 'NPU device type, e.g. 910b, a3.' + description: 'NPU device type, e.g. a3, 950.' type: string default: 'a3' run_timeout_minutes: @@ -33,17 +35,25 @@ on: type: string default: '{"size":1,"arr":[0]}' ref: - description: 'Git ref (branch, tag, or SHA) to test. If not provided, uses the default branch.' + description: 'Git ref (branch, tag, or SHA) to test. If not provided, uses the event commit SHA.' type: string default: '' skip_pr_test_health_check: - description: 'Git ref (branch, tag, or SHA) to test. If not provided, uses the default branch.' + description: 'Set to true to skip the PR test health check (fast-fail gate).' type: string default: 'false' is_nightly_pipeline_job: description: 'Run the test suite with --nightly (collects nightly-registered tests) and --continue-on-error' type: boolean default: false + use_coverage_runner: + description: 'If true, delegate test execution to scripts/ci/npu/precise-test/run_tests_with_coverage.sh instead of running via run_suite.py directly' + type: boolean + default: false + upload_test_logs: + description: 'If true, upload the test logs as a GitHub Actions artifact (only enabled by pr-test-npu)' + type: boolean + default: false github-token: description: 'GitHub token for API calls' type: string @@ -72,7 +82,9 @@ jobs: - name: Checkout code uses: actions/checkout@v4 with: - ref: ${{ inputs.ref || github.ref }} + # Pin to the event SHA (not the branch name) so every job in this run + # checks out the same commit even if the branch advances mid-run. + ref: ${{ inputs.ref || github.sha }} - name: Mark repository safe run: | @@ -186,26 +198,39 @@ jobs: } - name: Install dependencies - # Only PR jobs install dependencies - if: ${{ inputs.is_nightly_pipeline_job != true }} env: TORCH_CACHE_URL: "http://cache-service.nginx-pypi-cache.svc.cluster.local/whl/cpu" PYPI_CACHE_URL: "http://cache-service.nginx-pypi-cache.svc.cluster.local/pypi/simple" UV_INDEX_URL: "http://cache-service.nginx-pypi-cache.svc.cluster.local/pypi/simple" GITHUB_PROXY_URL: "https://gh-proxy.test.osinfra.cn/" RUSTUP_CACHE_URL: "http://cache-service.nginx-pypi-cache.svc.cluster.local:8082" + shell: bash run: | - # speed up by using infra cache services + set -euo pipefail + cp ~/.cache/modelscope/hub/datasets/otavia/ShareGPT_Vicuna_unfiltered/ShareGPT_V3_unfiltered_cleaned_split.json /tmp + cp ~/.cache/modelscope/hub/datasets/tmp/test.jsonl /tmp + CACHING_URL="cache-service.nginx-pypi-cache.svc.cluster.local" sed -Ei "s@(ports|archive).ubuntu.com@${CACHING_URL}:8081@g" /etc/apt/sources.list pip config set global.index-url http://${CACHING_URL}/pypi/simple pip config set global.trusted-host "${CACHING_URL}" - bash scripts/ci/npu/npu_ci_install_dependency.sh ${{ inputs.npu_device_type }} - # copy required file from our daily cache - cp ~/.cache/modelscope/hub/datasets/otavia/ShareGPT_Vicuna_unfiltered/ShareGPT_V3_unfiltered_cleaned_split.json /tmp - # copy gsm8k dataset - cp ~/.cache/modelscope/hub/datasets/tmp/test.jsonl /tmp + if [[ "${{ inputs.is_nightly_pipeline_job }}" != "true" ]]; then + bash scripts/ci/npu/npu_ci_install_dependency.sh ${{ inputs.npu_device_type }} + + else + sglang_pkg_path=/sgl-workspace/sglang/python + ascend_test_util_path=${sglang_pkg_path}/sglang/test/ascend + mkdir -p ${ascend_test_util_path} + mv ${ascend_test_util_path} ${ascend_test_util_path}_bak + cp -r $(pwd)/python/sglang/test/ascend ${ascend_test_util_path} + + pip install sentence_transformers zss "wandb>=0.16.0" tenacity==8.3.0 loguru openpyxl latex2sympy2 zstandard transformers-stream-generator tqdm-multiprocess pycocoevalcap + pip install yt-dlp sentencepiece==0.1.99 nltk av ftfy sqlitedict==2.1.0 "sacrebleu>=1.5.0" pytablewriter "peft>=0.2.0" "accelerate>=0.29.1" + pip install jsonlines "evaluate>=0.4.0" numexpr dotenv + + hf download MMMU/MMMU --repo-type dataset + fi # install sglang_router apt-get install -y libssl-dev @@ -215,10 +240,12 @@ jobs: timeout-minutes: ${{ fromJson(inputs.run_timeout_minutes) }} env: CONTINUE_ON_ERROR_FLAG: ${{ inputs.continue_on_error == 'true' && '--continue-on-error' || '' }} + TEST_LOG_DIR: /tmp/test-logs shell: bash run: | # Fail fast on any command error, undefined variable, or pipe failure. set -euo pipefail + mkdir -p "${TEST_LOG_DIR}" # Install missing python deps (skip if already importable). PYTHON_FOR_SGLANG="python" @@ -242,6 +269,12 @@ jobs: install_pkg "${pkg}" done + # Fallback for images that predate the pin in pyproject_npu.toml. Checking + # the version, not the command, so a stale wheel is replaced rather than kept. + SGL_EVAL_VERSION=0.1.2 + ${PYTHON_FOR_SGLANG} -c "import sgl_eval, sys; sys.exit(sgl_eval.__version__ != '${SGL_EVAL_VERSION}')" 2>/dev/null \ + || install_pkg "sgl-eval==${SGL_EVAL_VERSION}" + cd test # Append `--nightly` and `--continue-on-error` when the job belongs to the nightly pipeline. # --nightly: Collect test cases registered for nightly runs. @@ -250,8 +283,38 @@ jobs: if [ "${{ inputs.is_nightly_pipeline_job }}" = "true" ]; then NIGHTLY_FLAG="--nightly --continue-on-error" fi + + # Optional: delegate execution to scripts/ci/npu/precise-test/run_tests_with_coverage.sh when the + # caller sets use_coverage_runner=true. list_tests.py (same directory) enumerates the + # selected test files; the original run_suite.py execution below is preserved + # unchanged for the default (false) path. + LOG_FILE="${TEST_LOG_DIR}/${{ inputs.self_name }}-part${{ matrix.partition }}.log" + + if [[ "${{ inputs.use_coverage_runner }}" == "true" ]]; then + pip install coverage==7.8.* + python3 ../scripts/ci/npu/precise-test/list_tests.py --hw npu --suite ${{ inputs.self_name }} \ + --auto-partition-id ${{ matrix.partition }} \ + --auto-partition-size ${{ fromJson(inputs.partitions).size }} \ + -o /tmp/selected_tests.txt + echo "Selected test files:" + cat /tmp/selected_tests.txt + if [ -s /tmp/selected_tests.txt ]; then + bash ../scripts/ci/npu/precise-test/run_tests_with_coverage.sh $(cat /tmp/selected_tests.txt) + fi + exit 0 + fi + python3 run_suite.py --hw npu --suite ${{ inputs.self_name }} \ --auto-partition-id ${{ matrix.partition }} \ --auto-partition-size ${{ fromJson(inputs.partitions).size }} \ ${{ inputs.timeout_per_file && format('--timeout-per-file {0}', inputs.timeout_per_file) || '' }} \ - $NIGHTLY_FLAG $CONTINUE_ON_ERROR_FLAG + $NIGHTLY_FLAG $CONTINUE_ON_ERROR_FLAG 2>&1 | tee "${LOG_FILE}" + + - name: Upload test logs + if: always() && inputs.use_coverage_runner != true && inputs.upload_test_logs == true + uses: actions/upload-artifact@v7 + with: + name: selected-test-logs-${{ inputs.self_name }}-part${{ matrix.partition }} + path: /tmp/test-logs/ + if-no-files-found: ignore + retention-days: 14 diff --git a/.github/workflows/_npu-single-node-test-stage.yml b/.github/workflows/_npu-single-node-test-stage.yml index c0895c29aac8..7fe82bd73fb8 100644 --- a/.github/workflows/_npu-single-node-test-stage.yml +++ b/.github/workflows/_npu-single-node-test-stage.yml @@ -21,7 +21,7 @@ on: required: true type: string description: image for pods - default: "swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:main-cann9.0.0-a3" + default: "swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:main-cann9.1.0-a3" install_sglang_deps: required: false type: boolean @@ -31,7 +31,7 @@ on: required: false type: string default: 'a3' - description: device type for dependency installation (a3 or 910b) + description: device type for dependency installation (a3 or 950) transformers_version: required: false type: string @@ -43,7 +43,7 @@ on: default: '{}' description: 'JSON run metadata {branch_label, workflow_name, create_time}, recorded once at workflow start' skip_pr_test_health_check: - description: 'Git ref (branch, tag, or SHA) to test. If not provided, uses the default branch.' + description: 'Set to true to skip the PR test health check (fast-fail gate).' type: string default: 'false' is_nightly_pipeline_job: @@ -54,6 +54,10 @@ on: description: 'Partition config to parallelize a suite across jobs, e.g. {"size":3,"arr":[0,1,2]}. Each element of arr is a partition id; the template expands one matrix job per element. size is passed to run_suite.py --auto-partition-size.' type: string default: '{"size":1,"arr":[0]}' + ref: + description: 'Git ref (branch, tag, or SHA) to test. If not provided, uses the event SHA.' + type: string + default: '' github-token: description: 'GitHub token for API calls' type: string @@ -63,6 +67,14 @@ on: type: string default: '300' description: 'timeout-minutes for the Run test step' + use_coverage_runner: + description: 'If true, delegate test execution to scripts/ci/npu/precise-test/run_tests_with_coverage.sh instead of running via run_suite.py directly' + type: boolean + default: false + upload_test_logs: + description: 'If true, upload the test logs as a GitHub Actions artifact (only enabled by pr-test-npu)' + type: boolean + default: false env: SKIP_PR_TEST_HEALTH_CHECK: ${{ inputs.skip_pr_test_health_check }} @@ -84,6 +96,10 @@ jobs: steps: - name: Checkout code uses: actions/checkout@v4 + with: + # Pin to the event SHA (not the branch name) so every job in this run + # checks out the same commit even if the branch advances mid-run. + ref: ${{ inputs.ref || github.sha }} - name: Check PR test health uses: actions/github-script@v8 @@ -227,6 +243,9 @@ jobs: # Fail fast on any command error, undefined variable, or pipe failure. set -euo pipefail + TEST_LOG_DIR="/tmp/test-logs" + mkdir -p "${TEST_LOG_DIR}" + sglang_source_path=$(pwd) echo "Source code path: ${sglang_source_path}" ln -sf ${sglang_source_path} /root/sglang @@ -258,6 +277,11 @@ jobs: workflow_name=$(echo "${{ fromJson(inputs.run_start_metadata).workflow_name }}" | tr ' ' '_' | tr -d '()') create_time="${{ fromJson(inputs.run_start_metadata).create_time }}" if [ -z "${create_time}" ]; then create_time=$(date -u -d '+8 hours' +%Y%m%d-%H%M); fi + # Append the image tag to the workflow-name segment so that runs with + # different images (CANN 9.0.0/9.1.0, a3/a5) persist into distinct, + # image-tagged output/log directories. + image_label=$(echo "${{ inputs.image }}" | sed 's#.*[/:]##') + workflow_name="${workflow_name}--${image_label}" # Print persistence‑directory‑prefix variables echo "Persistence prefix: run_id=${run_id} run_attempt=${run_attempt} branch_label=${branch_label} workflow_name=${workflow_name} create_time=${create_time}" @@ -345,6 +369,7 @@ jobs: source /usr/local/Ascend/nnal/atb/set_env.sh || true source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash || true source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/custom_transformer/bin/set_env.bash || true + source /usr/local/Ascend/ascend-toolkit/latest/set_env.sh || true set -u # Nightly log path mirrors the output layout; PR keeps the original {date}/{testcase}. @@ -384,14 +409,29 @@ jobs: fi # Nightly additionally persists the full suite log to ${log_path}/${tc_name}.log - # under the structured path; PR runs only tee to /tmp/test_output.log. + # under the structured path; PR runs tee to ${TEST_LOG_DIR}/${tc_name}.log. # Use an array so the extra target is omitted when empty: passing an empty # string arg makes GNU tee fail (exit code 1) and flips the pipeline exit code. - LOG_TEE_TARGETS=("/tmp/test_output.log") + LOG_TEE_TARGETS=("${TEST_LOG_DIR}/${tc_name}.log") if [ "${{ inputs.is_nightly_pipeline_job }}" = "true" ]; then LOG_TEE_TARGETS+=("${log_path}/${tc_name}.log") fi + # Optional: delegate execution to scripts/ci/npu/precise-test/run_tests_with_coverage.sh when the + # caller sets use_coverage_runner=true. list_tests.py (same directory) enumerates the + # selected test files; the original run_suite.py execution below is preserved + # unchanged for the default (false) path. + if [[ "${{ inputs.use_coverage_runner }}" == "true" ]]; then + pip install coverage==7.8.* + ${PYTHON_FOR_SGLANG} -u ../scripts/ci/npu/precise-test/list_tests.py --hw npu --suite ${test_suite} ${PARTITION_ARGS} -o /tmp/selected_tests.txt + echo "Selected test files:" + cat /tmp/selected_tests.txt + if [ -s /tmp/selected_tests.txt ]; then + bash ../scripts/ci/npu/precise-test/run_tests_with_coverage.sh $(cat /tmp/selected_tests.txt) + fi + exit 0 + fi + # Run NPU test suite with run_suite.py. # Capture mode (--enable-retry --max-attempts 1) keeps test subprocesses off the # tee pipe, so leftover server processes cannot block the pipeline on exit. @@ -411,18 +451,28 @@ jobs: export test_status tc_name echo "## ${tc_name} ${status_icon} ${test_status}" >> $GITHUB_STEP_SUMMARY echo "" >> $GITHUB_STEP_SUMMARY - metric_count=$(grep -c '\[METRIC\]' /tmp/test_output.log 2>/dev/null || echo 0) - if [ "${metric_count}" -gt 0 ]; then - echo "| Metric | Value | Pass |" >> $GITHUB_STEP_SUMMARY - echo "|--------|-------|------|" >> $GITHUB_STEP_SUMMARY - grep '\[METRIC\]' /tmp/test_output.log | while IFS= read -r line; do - metric_name=$(echo "$line" | sed -E 's/.*\[METRIC\] ([^=]+)=.*/\1/') - metric_value=$(echo "$line" | sed -E 's/.*\[METRIC\] [^=]+=([^ ]+).*/\1/') - echo "| ${metric_name} | ${metric_value} | ${status_icon} |" >> $GITHUB_STEP_SUMMARY - done - else - echo "No metrics collected (test may have failed before producing results)." >> $GITHUB_STEP_SUMMARY + if [ "${{ inputs.is_nightly_pipeline_job }}" != "true" ]; then + metric_count=$(grep -c '\[METRIC\]' "${TEST_LOG_DIR}/${tc_name}.log" 2>/dev/null || echo 0) + if [ "${metric_count}" -gt 0 ]; then + echo "| Metric | Value | Pass |" >> $GITHUB_STEP_SUMMARY + echo "|--------|-------|------|" >> $GITHUB_STEP_SUMMARY + grep '\[METRIC\]' "${TEST_LOG_DIR}/${tc_name}.log" | while IFS= read -r line; do + metric_name=$(echo "$line" | sed -E 's/.*\[METRIC\] ([^=]+)=.*/\1/') + metric_value=$(echo "$line" | sed -E 's/.*\[METRIC\] [^=]+=([^ ]+).*/\1/') + echo "| ${metric_name} | ${metric_value} | ${status_icon} |" >> $GITHUB_STEP_SUMMARY + done + else + echo "No metrics collected (test may have failed before producing results)." >> $GITHUB_STEP_SUMMARY + fi + echo "" >> $GITHUB_STEP_SUMMARY fi - echo "" >> $GITHUB_STEP_SUMMARY - exit ${test_exit_code} + + - name: Upload test logs + if: always() && inputs.use_coverage_runner != true && inputs.upload_test_logs == true + uses: actions/upload-artifact@v7 + with: + name: selected-test-logs-${{ inputs.test_suite }}-part${{ matrix.partition }} + path: /tmp/test-logs/ + if-no-files-found: ignore + retention-days: 14 diff --git a/.github/workflows/_pr-test-check-changes.yml b/.github/workflows/_pr-test-check-changes.yml index 2dfae1948c91..9a7fa183afa7 100644 --- a/.github/workflows/_pr-test-check-changes.yml +++ b/.github/workflows/_pr-test-check-changes.yml @@ -19,12 +19,16 @@ on: outputs: main_package: value: ${{ jobs.run.outputs.main_package }} + simulator: + value: ${{ jobs.run.outputs.simulator }} sgl_kernel: value: ${{ jobs.run.outputs.sgl_kernel }} jit_kernel: value: ${{ jobs.run.outputs.jit_kernel }} multimodal_gen: value: ${{ jobs.run.outputs.multimodal_gen }} + renderer: + value: ${{ jobs.run.outputs.renderer }} rust_workspace: value: ${{ jobs.run.outputs.rust_workspace }} partitions: @@ -44,9 +48,11 @@ jobs: runs-on: ubuntu-latest outputs: main_package: ${{ steps.filter.outputs.main_package || steps.run-mode.outputs.run_all_tests }} + simulator: ${{ steps.filter.outputs.simulator || steps.run-mode.outputs.run_all_tests }} sgl_kernel: ${{ steps.filter.outputs.sgl_kernel }} jit_kernel: ${{ steps.filter.outputs.jit_kernel || steps.run-mode.outputs.run_all_tests }} multimodal_gen: ${{ steps.filter.outputs.multimodal_gen || steps.run-mode.outputs.run_all_tests }} + renderer: ${{ steps.filter.outputs.renderer || steps.run-mode.outputs.run_all_tests }} rust_workspace: ${{ steps.filter.outputs.rust_workspace || steps.run-mode.outputs.run_all_tests }} partitions: ${{ steps.partitions.outputs.partitions }} partition_model_sha: ${{ steps.partition-model-sha.outputs.sha }} @@ -86,14 +92,23 @@ jobs: - ".github/workflows/pr-gate.yml" - ".github/actions/**" - "python/pyproject.toml" + - "python/setup.py" - "python/sglang/!(multimodal_gen|kernels)/**/!(*.md)" - "python/sglang/kernels/!(*.md)" - "python/sglang/kernels/!(aot)/**/!(*.md)" - "scripts/ci/cuda/*" - "scripts/ci/utils/*" - "test/**/!(*.md)" - - "rust/**" + - "rust/*" + - "rust/!(sglang-renderer)/**" - "proto/sglang/runtime/v1/sglang.proto" + renderer: + - "rust/sglang-renderer/**" + simulator: + - ".github/workflows/pr-test-extra.yml" + - ".github/workflows/_pr-test-simulator-cpu.yml" + - "benchmark/simulator/**/!(*.md)" + - "tools/sglang-simulator/**/!(*.md)" multimodal_gen: - ".github/workflows/pr-test.yml" - ".github/workflows/pr-test-multimodal-gen.yml" @@ -104,6 +119,7 @@ jobs: - "python/sglang/kernels/ops/diffusion/**" - "test/registered/kernels/ops/diffusion/**" - "test/registered/kernels/benchmark/diffusion/**" + - "test/registered/unit/diffusion/**" - "python/sglang/cli/**" jit_kernel: - ".github/workflows/pr-test.yml" @@ -126,15 +142,24 @@ jobs: - "test/registered/rust/**" - ".github/workflows/_pr-test-*.yml" + - name: Resolve CI labels + id: ci-labels + if: github.event_name == 'pull_request' + uses: actions/github-script@v8 + with: + script: | + const { resolveCiLabels } = require( + `${process.env.GITHUB_WORKSPACE}/.github/scripts/ci-labels.cjs` + ); + const axes = await resolveCiLabels(github, context); + core.info(`CI labels: [${axes.labels.join(', ')}]`); + core.setOutput('max_concurrency', String(axes.maxConcurrency)); + - name: Determine full-parallel mode id: parallel-mode run: | # `full=true` lifts the matrix-fanout throttle so each suite's - # max_parallel = size. Conditions: - # 1. Scheduled cron run. - # 2. run_all_tests run (manual full dispatch / release) -- a full - # run should mirror the cron's parallelism, not just its test set. - # 3. pull_request event with the `high priority` label. + # max_parallel = size. FULL=false if [[ "${{ github.event_name }}" == "schedule" ]]; then FULL=true @@ -142,9 +167,9 @@ jobs: elif [[ "${{ inputs.run_all_tests }}" == "true" ]]; then FULL=true echo "run_all_tests -> full parallelism" - elif [[ "${{ github.event_name }}" == "pull_request" && "${{ contains(github.event.pull_request.labels.*.name, 'high priority') }}" == "true" ]]; then + elif [[ "${{ steps.ci-labels.outputs.max_concurrency }}" == "true" ]]; then FULL=true - echo "high priority PR -> full parallelism" + echo "max-concurrency PR -> full parallelism" fi echo "full=$FULL" >> "$GITHUB_OUTPUT" @@ -179,10 +204,7 @@ jobs: id: partitions run: | # Emit a single JSON output `partitions` keyed by suite name with - # {size, arr, max_parallel} fields per suite. Replaces the prior - # full/low max-parallel presets; `--full-parallel` keeps the - # `high priority` PR / scheduled cron escape hatch. - # See scripts/ci/utils/compute_partitions.py. + # {size, arr, max_parallel} fields per suite. python3 scripts/ci/utils/compute_partitions.py \ --full-parallel ${{ steps.parallel-mode.outputs.full }} \ --partition-model-file /tmp/partition-model.json \ diff --git a/.github/workflows/_pr-test-rust-ext-build.yml b/.github/workflows/_pr-test-rust-ext-build.yml index 20a9cfae901e..9c2034f6a40d 100644 --- a/.github/workflows/_pr-test-rust-ext-build.yml +++ b/.github/workflows/_pr-test-rust-ext-build.yml @@ -14,21 +14,29 @@ name: PR Test - Build Rust Extensions # - The artifact hands it to the stages, which the cache cannot: it is best-effort, # evictable, and this repo is at its 10 GB limit. Artifacts are durable per run. # -# Two jobs, because only compiling needs the build node: exactly one runner carries -# that label and it also serves the sgl-kernel and docker builds, so queueing there -# on a cache hit would put its wait in front of every stage. Republishing bytes -# needs no particular host - the glibc a module requires is recorded in the module, -# not decided by whoever uploads it. +# Compiling is split off from restoring and collecting because only it needs the +# build node, and queueing there on a cache hit would put that wait in front of +# every stage. Republishing bytes needs no particular host - the glibc a module +# requires is recorded in the module, not decided by whoever uploads it. on: workflow_call: inputs: runs_on: - description: 'Runner label for the compile job. Must be self-hosted: its glibc has to satisfy max_glibc, and the cargo build cache needs a persistent ~/.cache.' + description: 'Runner label for the compile job. Its glibc has to satisfy max_glibc.' type: string required: true restore_runs_on: - description: 'Runner label for the cache-hit path. Wants free capacity and nothing else, since it neither compiles nor imports the modules.' + description: > + Runner label for the cache-hit path. Only republishes bytes, so it wants + free capacity and is the one job here with no architecture requirement. + type: string + default: ubuntu-latest + collect_runs_on: + description: > + Runner label for the collect job. Must be runs_on's ARCHITECTURE: its GLIBC + gate reads the .so with objdump, and the hosted images ship a single-target + binutils. type: string default: ubuntu-latest artifact_name: @@ -40,7 +48,10 @@ on: type: string default: rust-ext-x86_64-cp310-cp312 max_glibc: - description: 'Highest GLIBC symbol version the built .so files may require. Set by the oldest test runner image, jammy at glibc 2.35 - the pools are not all on one image.' + description: > + Highest GLIBC symbol version the built .so files may require. A ceiling set + by the oldest image that has to LOAD them, not by whoever builds them; the + pools are not all on one image, and x86_64's oldest is jammy at 2.35. type: string default: '2.35' git_ref: @@ -53,7 +64,7 @@ on: outputs: artifact_name: description: 'Artifact holding the built modules. Empty when neither job published one, whether skipped or failed; consumers that still run compile them during install.' - value: ${{ jobs.restore.outputs.artifact_name || jobs.compile.outputs.artifact_name }} + value: ${{ jobs.restore.outputs.artifact_name || jobs.collect.outputs.artifact_name }} # Reusable workflows do not inherit the caller's env; mirror what # check-maintenance reads. @@ -77,11 +88,15 @@ jobs: ref: ${{ inputs.git_ref || github.sha }} # Just what the cache key hashes, plus the action and script this job # runs: the workspace is cold here and the rest of the tree is mostly - # docs. Both jobs must hash the same rust/** set, which this preserves. + # docs. Both jobs must hash the same Rust extension inputs, which this + # preserves. # Cone mode off is what allows naming a single file. sparse-checkout: | rust + proto python/setup.py + python/pyproject.toml + python/sglang/srt/rust_extensions/torch_build.py .github scripts/ci/utils sparse-checkout-cone-mode: false @@ -93,14 +108,16 @@ jobs: - name: Ensure zstd so the saved entry is readable run: bash scripts/ci/utils/ensure_zstd.sh - # setup.py counts because it selects which crates get built. pyproject.toml - # is left out - it churns on bumps that cannot affect these modules. + # The setup hook and torch helper select and configure what gets built; + # pyproject.toml pins the libtorch ABI used by sglang-radix-tree. - name: Restore built modules id: cache uses: actions/cache/restore@v4 with: - path: python/sglang/srt/rust_extensions/_*.so - key: ${{ inputs.cache_key_prefix }}-${{ hashFiles('rust/**', 'python/setup.py') }} + path: | + python/sglang/srt/rust_extensions/_*.so + python/sglang/srt/mem_cache/rust_tree_core/mem_cache*.so + key: ${{ inputs.cache_key_prefix }}-${{ hashFiles('rust/**', 'proto/**', 'python/setup.py', 'python/pyproject.toml', 'python/sglang/srt/rust_extensions/torch_build.py', '.github/workflows/_pr-test-rust-ext-build.yml', 'scripts/ci/utils/stage_rust_ext_modules.sh') }} # On a miss: different hash = rust/setup.py moved; no entries = evicted. - name: Report cache lookup @@ -112,7 +129,8 @@ jobs: run: | if [ -n "${MATCHED_KEY}" ]; then echo "hit: ${MATCHED_KEY}" - ls -l python/sglang/srt/rust_extensions/_*.so + ls -l python/sglang/srt/rust_extensions/_*.so \ + python/sglang/srt/mem_cache/rust_tree_core/mem_cache*.so else echo "miss: ${PRIMARY_KEY}" echo "entries under ${KEY_PREFIX}- (created / ref / size / key):" @@ -122,7 +140,7 @@ jobs: || echo "(gh cache list unavailable: token lacks actions:read)" fi - # No MAX_GLIBC: these are the bytes the compile job already checked before + # No MAX_GLIBC: these are the bytes the collect job already checked before # saving them under this key. The module count is still worth re-checking, # so a truncated entry fails here rather than as a test import error. - name: Stage modules for upload @@ -134,7 +152,7 @@ jobs: uses: actions/upload-artifact@v4 with: name: ${{ inputs.artifact_name }} - # Archive holds rust_extensions/_*.so, so it unpacks into python/sglang/srt/. + # Archive holds package-relative paths, so it unpacks into python/sglang/srt/. path: rust-ext-staging/ if-no-files-found: error retention-days: 1 @@ -150,8 +168,16 @@ jobs: runs-on: ${{ inputs.runs_on }} timeout-minutes: 60 name: Build Rust Ext - outputs: - artifact_name: ${{ steps.publish.outputs.name }} + # No crate sets abi3, so the pools' interpreters (h100 ships 3.10, h20 ships + # 3.12) each need their own module set, and each its own target dir, since + # PyO3's fingerprint tracks the interpreter. The two passes therefore share + # nothing, which is why they run as parallel jobs rather than back to back. + strategy: + # Both legs run to completion, so a failure in one still leaves the other's + # log to compare against. + fail-fast: false + matrix: + python-version: ['3.10', '3.12'] steps: - uses: actions/checkout@v4 with: @@ -159,19 +185,11 @@ jobs: - uses: ./.github/actions/check-maintenance - # No crate sets abi3, so build one module set per interpreter the pools - # run (h100 ships 3.10, h20 ships 3.12); EXT_SUFFIX keeps them apart. - - name: Set up Python 3.10 - id: py310 + - name: Set up Python ${{ matrix.python-version }} + id: py uses: actions/setup-python@v5 with: - python-version: '3.10' - - - name: Set up Python 3.12 - id: py312 - uses: actions/setup-python@v5 - with: - python-version: '3.12' + python-version: ${{ matrix.python-version }} - name: Install protoc and Rust toolchain run: bash scripts/ci/utils/install_rust_protoc.sh @@ -180,37 +198,152 @@ jobs: run: | set -euxo pipefail export PATH="${CARGO_HOME:-$HOME/.cargo}/bin:${PATH}" - # Per-interpreter subdirs (set in the loop): PyO3's fingerprint tracks - # the interpreter, so a shared dir rebuilds on every ABI switch. - cargo_target_root="${HOME}/.cache/sglang-cargo-target" - mkdir -p "${cargo_target_root}" - used="$(df --output=pcent "${cargo_target_root}" 2>/dev/null | tr -dc '0-9')" - if [ "${used:-0}" -ge 85 ]; then - echo "cargo target dir filesystem at ${used}%; dropping ${cargo_target_root}" - rm -rf "${cargo_target_root}" - mkdir -p "${cargo_target_root}" - fi + # A hosted runner is fresh per job, so there is no shared tree to reuse or + # to guard against a concurrent dropper. + export CARGO_TARGET_DIR="${RUNNER_TEMP:-/tmp}/sglang-cargo-target-${GITHUB_RUN_ID:-norun}-$$" + mkdir -p "${CARGO_TARGET_DIR}" python3 -m pip install --upgrade pip command -v uv >/dev/null 2>&1 || pip install uv - # build_rust needs only the build backend, not sglang's ~294 runtime deps. - # Per-job path: these runners are persistent and shared, so a fixed one - # both inherits the previous job's venv and races a concurrent build. - venv_root="${RUNNER_TEMP:-/tmp}/sglang-ci-rust-ext-${GITHUB_RUN_ID:-norun}-$$" + # Per-job path: a self-hosted runs_on is persistent and shared, so a fixed + # one both inherits the previous job's venv and races a concurrent build. + venv="${RUNNER_TEMP:-/tmp}/sglang-ci-rust-ext-${GITHUB_RUN_ID:-norun}-$$" # Best-effort, like ci_cleanup_venv.sh: under set -e a failing EXIT trap # would fail the step, and nothing here is worth keeping for a postmortem. - trap 'rm -rf "${venv_root}" || true' EXIT - for python_bin in "${{ steps.py310.outputs.python-path }}" "${{ steps.py312.outputs.python-path }}"; do - minor="$("${python_bin}" -c 'import sys; print(f"{sys.version_info.major}.{sys.version_info.minor}")')" - export CARGO_TARGET_DIR="${cargo_target_root}/py${minor}" - venv="${venv_root}/py${minor}" - uv venv "${venv}" --python "${python_bin}" --seed - # shellcheck disable=SC1091 - source "${venv}/bin/activate" - uv pip install "setuptools>=61.0" "setuptools-rust>=1.10" "setuptools-scm>=8.0" wheel - (cd python && SGLANG_BUILD_RUST_EXTS=all python setup.py build_rust --inplace) - deactivate - done + trap 'rm -rf "${venv}" "${CARGO_TARGET_DIR}" || true' EXIT + # build_rust needs the build backend and torch, not sglang's ~294 other runtime deps. + uv venv "${venv}" --python "${{ steps.py.outputs.python-path }}" --seed + # shellcheck disable=SC1091 + source "${venv}/bin/activate" + uv pip install "setuptools>=61.0" "setuptools-rust>=1.11" "setuptools-scm>=8.0" "torch==2.13.0" wheel + # torch-sys resolves libtorch from the active interpreter and bakes that + # path into the cargo cache. venv is per-run and deleted on exit, so a + # persistent target dir would reuse a "Fresh torch-sys" whose -L points at + # a gone directory and fail with "unable to find library -ltorch". Pin + # LIBTORCH and drop the torch-shim units so they rebuild against this venv. + LIBTORCH="$(python -c 'import pathlib, torch; print(pathlib.Path(torch.__file__).parent)')" + export LIBTORCH + export LD_LIBRARY_PATH="${LIBTORCH}/lib:${LD_LIBRARY_PATH:-}" + cargo clean --release --manifest-path rust/sglang-radix-tree/Cargo.toml \ + -p torch-sys -p sglang-radix-tree 2>/dev/null || true + (cd python && SGLANG_BUILD_RUST_EXTS=all python setup.py build_rust --inplace) + python - <<'PY' + import importlib.util + import os + import pathlib + import runpy + import shutil + import subprocess + import sysconfig + + import torch # noqa: F401 - preload libtorch before the extension + + root = pathlib.Path.cwd() + suffix = sysconfig.get_config_var("EXT_SUFFIX") + production_path = pathlib.Path( + "python/sglang/srt/mem_cache/rust_tree_core/mem_cache" + suffix + ).resolve() + inspection_path = production_path.with_name( + "mem_cache_inspection" + suffix + ) + + helper = runpy.run_path( + root / "python/sglang/srt/rust_extensions/torch_build.py" + ) + build = helper["torch_build_configuration"]( + compat_header=root / "rust/sglang-radix-tree/torch_2_13_compat.h", + python_module="sglang.srt.mem_cache.rust_tree_core.mem_cache", + torch_module=torch, + include_absolute_rpath=False, + ) + subprocess.run( + [ + "cargo", + "build", + "--release", + "--locked", + "--manifest-path", + "rust/sglang-radix-tree/Cargo.toml", + "--features", + "python-extension,inspection", + ], + env=build.environment, + check=True, + ) + release_dir = pathlib.Path(os.environ["CARGO_TARGET_DIR"]) / "release" + if target := os.environ.get("CARGO_BUILD_TARGET"): + release_dir = pathlib.Path(os.environ["CARGO_TARGET_DIR"]) / target / "release" + shutil.copy2(release_dir / "libmem_cache.so", inspection_path) + + def load(name, path): + spec = importlib.util.spec_from_file_location(name, path) + assert spec is not None and spec.loader is not None + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + production = load( + "sglang.srt.mem_cache.rust_tree_core.mem_cache", production_path + ) + inspection = load( + "sglang.srt.mem_cache.rust_tree_core.mem_cache_inspection", + inspection_path, + ) + assert not hasattr( + production.RustUnifiedTreeCoreBinding, "inspect_contains_node" + ) + assert hasattr( + inspection.RustUnifiedTreeCoreBinding, "inspect_contains_node" + ) + PY + + # Both globs sit under python/sglang/srt/, which upload-artifact strips as the + # common prefix, so the collect job restores them by unpacking back into it. + - name: Upload this interpreter's modules + uses: actions/upload-artifact@v4 + with: + name: ${{ inputs.artifact_name }}-build-py${{ matrix.python-version }} + path: | + python/sglang/srt/rust_extensions/_*.so + python/sglang/srt/mem_cache/rust_tree_core/mem_cache*.so + if-no-files-found: error + retention-days: 1 + + # The interpreters compile independently, but the suffix-set check and the cache + # entry both cover the whole set, so they join in one job that sees all of it. + collect: + needs: compile + runs-on: ${{ inputs.collect_runs_on }} + timeout-minutes: 15 + name: Collect Rust Ext + outputs: + # From the last step, so a failed job publishes no name for always() consumers. + artifact_name: ${{ steps.publish.outputs.name }} + steps: + - uses: actions/checkout@v4 + with: + ref: ${{ inputs.git_ref || github.sha }} + # Mirrors the restore job: this one writes the entry that job reads, so + # both have to check out the same set for hashFiles to agree. + sparse-checkout: | + rust + proto + python/setup.py + python/pyproject.toml + python/sglang/srt/rust_extensions/torch_build.py + .github + scripts/ci/utils + sparse-checkout-cone-mode: false + + - name: Collect the per-interpreter modules + uses: actions/download-artifact@v4 + with: + pattern: ${{ inputs.artifact_name }}-build-py* + merge-multiple: true + path: python/sglang/srt + # First point that sees every interpreter, which is what the suffix-set check + # compares. MAX_GLIBC belongs here rather than in the restore job: these are + # freshly compiled bytes, not ones that already passed under this key. - name: Verify modules and stage for upload env: MAX_GLIBC: ${{ inputs.max_glibc }} @@ -226,13 +359,16 @@ jobs: - name: Save built modules uses: actions/cache/save@v4 with: - path: python/sglang/srt/rust_extensions/_*.so - key: ${{ inputs.cache_key_prefix }}-${{ hashFiles('rust/**', 'python/setup.py') }} + path: | + python/sglang/srt/rust_extensions/_*.so + python/sglang/srt/mem_cache/rust_tree_core/mem_cache*.so + key: ${{ inputs.cache_key_prefix }}-${{ hashFiles('rust/**', 'proto/**', 'python/setup.py', 'python/pyproject.toml', 'python/sglang/srt/rust_extensions/torch_build.py', '.github/workflows/_pr-test-rust-ext-build.yml', 'scripts/ci/utils/stage_rust_ext_modules.sh') }} - name: Upload extension modules uses: actions/upload-artifact@v4 with: name: ${{ inputs.artifact_name }} + # Archive holds package-relative paths, so it unpacks into python/sglang/srt/. path: rust-ext-staging/ if-no-files-found: error retention-days: 1 diff --git a/.github/workflows/_pr-test-simulator-cpu.yml b/.github/workflows/_pr-test-simulator-cpu.yml new file mode 100644 index 000000000000..6c47ce9c5eca --- /dev/null +++ b/.github/workflows/_pr-test-simulator-cpu.yml @@ -0,0 +1,97 @@ +name: PR Test SGLang Simulator (CPU) + +on: + workflow_call: + inputs: + check_changes: + description: 'toJson(needs.check-changes.outputs).' + type: string + required: true + caller_inputs: + description: 'toJson(inputs) from the caller workflow.' + type: string + required: true + rust_ext_artifact: + description: 'Artifact of prebuilt Rust extension modules.' + type: string + default: '' + +env: + SGLANG_IS_IN_CI: true + SKIP_PR_TEST_HEALTH_CHECK: ${{ (fromJson(inputs.caller_inputs).skip_pr_test_health_check || fromJson(inputs.caller_inputs).test_parallel_dispatch || fromJson(inputs.caller_inputs).run_all_tests) && 'true' || 'false' }} + PR_TEST_BYPASS_MAINTENANCE_ON_MAIN: ${{ github.ref == 'refs/heads/main' && 'true' || 'false' }} + USE_VENV: false + +jobs: + run: + name: simulator-test-cpu + if: fromJson(inputs.check_changes).main_package == 'true' || fromJson(inputs.check_changes).simulator == 'true' + runs-on: ubuntu-latest + timeout-minutes: 40 + steps: + - name: Checkout code + uses: actions/checkout@v4 + with: + ref: ${{ fromJson(inputs.caller_inputs).git_ref || github.sha }} + + - uses: ./.github/actions/check-pr-test-health + + - uses: ./.github/actions/check-maintenance + + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: '3.10' + + - name: Install uv + uses: astral-sh/setup-uv@v5 + + - uses: ./.github/actions/download-rust-ext + id: rust_ext + with: + artifact_name: ${{ inputs.rust_ext_artifact }} + + - name: Install protoc + Rust toolchain + if: ${{ steps.rust_ext.outputs.hit != 'true' }} + timeout-minutes: 10 + run: bash scripts/ci/utils/install_rust_protoc.sh + + - name: Rust cache (rust/ workspace) + if: ${{ steps.rust_ext.outputs.hit != 'true' }} + uses: Swatinem/rust-cache@v2 + with: + workspaces: rust + shared-key: "sglang-grpc-cpu" + + - name: Install dependencies + timeout-minutes: 20 + env: + UV_SYSTEM_PYTHON: "1" + run: | + uv pip install -e "python" --index-strategy unsafe-best-match --prerelease allow + uv pip install pytest + uv pip install -e "tools/sglang-simulator[aic]" + + - name: Prebuild HiCache native hash extension + timeout-minutes: 5 + env: + MALLOC_ARENA_MAX: "2" + MAX_JOBS: "1" + TORCH_EXTENSIONS_DIR: ${{ runner.temp }}/torch-extensions + run: | + python3 -c \ + 'from sglang.srt.mem_cache.cpp_utils.native_hash import get_native_hash; get_native_hash([1], None)' + + - name: Run SGLang Simulator compatibility tests + timeout-minutes: 10 + env: + MALLOC_ARENA_MAX: "2" + MAX_JOBS: "1" + PYTEST_DISABLE_PLUGIN_AUTOLOAD: "1" + PYTEST_ADDOPTS: "-s" + TORCH_EXTENSIONS_DIR: ${{ runner.temp }}/torch-extensions + run: | + python3 -m pytest -q tools/sglang-simulator/test/test_simulation_sglang_runner.py + python3 -m pytest -q tools/sglang-simulator/test/test_simulation_sglang_serving.py + python3 -m pytest -q tools/sglang-simulator/test/test_simulation_cache_hit_ratio.py + python3 -m pytest -q tools/sglang-simulator/test/test_simulation_offline_blocking.py diff --git a/.github/workflows/_pr-test-stage-cpu.yml b/.github/workflows/_pr-test-stage-cpu.yml index 651a0aa8d1fa..9206c8761d42 100644 --- a/.github/workflows/_pr-test-stage-cpu.yml +++ b/.github/workflows/_pr-test-stage-cpu.yml @@ -13,7 +13,7 @@ on: type: string required: true check_changes: - description: 'toJson(needs.check-changes.outputs). Read via fromJson(...).main_package / rust_workspace / continue_on_error / partition_model_sha.' + description: 'toJson(needs.check-changes.outputs). Read via fromJson(...).main_package / renderer / rust_workspace / continue_on_error / partition_model_sha.' type: string required: true caller_inputs: @@ -49,7 +49,7 @@ jobs: # emits a skipped entry for wait-for-jobs to match. The schedule / # parallel-dispatch / !failure() half stays on the caller, where the # `needs` context it guards actually lives. - if: fromJson(inputs.check_changes).main_package == 'true' + if: fromJson(inputs.check_changes).main_package == 'true' || fromJson(inputs.check_changes).renderer == 'true' runs-on: ubuntu-latest timeout-minutes: 240 env: @@ -119,8 +119,6 @@ jobs: UV_SYSTEM_PYTHON: "1" run: | uv pip install -e "python[dev]" --index-strategy unsafe-best-match --prerelease allow - source scripts/ci/utils/sgl_eval_ref.sh - uv pip install "$SGL_EVAL_SPEC" --index-strategy unsafe-best-match # Hosted runners are ephemeral, so models are re-fetched every run and the # Hub occasionally returns 429s. Persist the HF cache in GitHub's cache @@ -158,7 +156,7 @@ jobs: SGLANG_SKIP_RUST_TESTS: ${{ fromJson(inputs.check_changes).rust_workspace == 'true' && 'false' || 'true' }} run: | cd test/ - python3 run_suite.py --hw cpu --suite ${{ inputs.self_name }} --auto-partition-id ${{ matrix.partition }} --auto-partition-size ${{ fromJson(inputs.partitions)[inputs.self_name].size }} --partition-model-file /tmp/partition-model.json $CONTINUE_ON_ERROR_FLAG + python3 run_suite.py --hw cpu --suite ${{ inputs.self_name }} --auto-partition-id ${{ matrix.partition }} --auto-partition-size ${{ fromJson(inputs.partitions)[inputs.self_name].size }} --partition-model-file /tmp/partition-model.json --fork-worker-batch-size 20 $CONTINUE_ON_ERROR_FLAG # PRs save only what the base lacks; the implicit if: success() keeps a # half-downloaded cache unpublished. diff --git a/.github/workflows/_pr-test-stage.yml b/.github/workflows/_pr-test-stage.yml index 4475e8ba0e51..42c22191acb2 100644 --- a/.github/workflows/_pr-test-stage.yml +++ b/.github/workflows/_pr-test-stage.yml @@ -59,7 +59,10 @@ on: type: string default: '' skip_prebuilt_rust_ext: - description: 'Take neither the artifact nor the cache, and compile during install. For a stage no prebuild targets: the cache key is arch-blind, so an aarch64 stage would be handed x86_64 modules.' + description: > + Take neither the artifact nor the cache, and compile during install. No + stage sets it: both architectures have a prebuild. Kept as the opt-out for + a pool that gets one later than its stages do. type: boolean default: false @@ -100,8 +103,16 @@ jobs: run: name: ${{ inputs.self_name }} (${{ matrix.partition }}) # Runs on schedule / parallel-dispatch / non-failed PR with main_package or sgl_kernel changes. + # `runner_filter` is matched here rather than in each caller's job `if`: a + # caller that declares its runners as a matrix cannot, since job-level `if` + # sees github/needs/vars/inputs but not `matrix`. Callers that never declare + # the input read back null and match every runner_config. if: | always() && + (fromJson(inputs.caller_inputs).runner_filter == null || + fromJson(inputs.caller_inputs).runner_filter == '' || + fromJson(inputs.caller_inputs).runner_filter == 'all' || + fromJson(inputs.caller_inputs).runner_filter == inputs.runner_config) && ((github.event_name == 'schedule' || fromJson(inputs.caller_inputs).test_parallel_dispatch == true) || (!failure() && !cancelled())) && (fromJson(inputs.check_changes).main_package == 'true' || fromJson(inputs.check_changes).sgl_kernel == 'true') # runs-on resolved from runs_on_map; check-changes already substituted diff --git a/.github/workflows/amd-aiter-scout.yml b/.github/workflows/amd-aiter-scout.yml index 8365856adf9b..19e6cf06f2d1 100644 --- a/.github/workflows/amd-aiter-scout.yml +++ b/.github/workflows/amd-aiter-scout.yml @@ -12,7 +12,7 @@ on: type: string default: 'main' job_filter: - description: 'Workflows: nightly-amd, nightly-amd-rocm720, pr-test-amd (ROCm 7.0 shadow), pr-test-amd-rocm720 (ROCm 7.2 gate). Default: all' + description: 'Workflows: nightly-amd, pr-test-amd. Default: all' required: false type: string default: 'all' @@ -38,9 +38,7 @@ jobs: outputs: aiter_sha: ${{ steps.resolve.outputs.sha }} run_nightly_amd: ${{ steps.parse.outputs.run_nightly_amd }} - run_nightly_amd_rocm720: ${{ steps.parse.outputs.run_nightly_amd_rocm720 }} run_pr_test_amd: ${{ steps.parse.outputs.run_pr_test_amd }} - run_pr_test_amd_rocm720: ${{ steps.parse.outputs.run_pr_test_amd_rocm720 }} steps: - name: Resolve AITER commit id: resolve @@ -73,63 +71,39 @@ jobs: if [[ "$FILTER" == "all" ]]; then echo "run_nightly_amd=true" >> $GITHUB_OUTPUT - echo "run_nightly_amd_rocm720=true" >> $GITHUB_OUTPUT echo "run_pr_test_amd=true" >> $GITHUB_OUTPUT - echo "run_pr_test_amd_rocm720=true" >> $GITHUB_OUTPUT else - # Wrap with commas for exact substring matching (avoids "nightly-amd" matching "nightly-amd-rocm720") + # Wrap with commas for exact substring matching. PADDED=",${FILTER// /}," echo "run_nightly_amd=$(echo "$PADDED" | grep -q ',nightly-amd,' && echo true || echo false)" >> $GITHUB_OUTPUT - echo "run_nightly_amd_rocm720=$(echo "$PADDED" | grep -q ',nightly-amd-rocm720,' && echo true || echo false)" >> $GITHUB_OUTPUT echo "run_pr_test_amd=$(echo "$PADDED" | grep -q ',pr-test-amd,' && echo true || echo false)" >> $GITHUB_OUTPUT - echo "run_pr_test_amd_rocm720=$(echo "$PADDED" | grep -q ',pr-test-amd-rocm720,' && echo true || echo false)" >> $GITHUB_OUTPUT fi echo "### Job Filter" >> $GITHUB_STEP_SUMMARY echo "- **Filter:** \`${FILTER}\`" >> $GITHUB_STEP_SUMMARY call-nightly-amd: + name: Nightly AMD (all ROCm versions) if: needs.resolve-aiter.outputs.run_nightly_amd == 'true' needs: resolve-aiter uses: ./.github/workflows/nightly-test-amd.yml secrets: inherit with: ref: amd/aiter-ci - aiter_ref: ${{ needs.resolve-aiter.outputs.aiter_sha }} - job_filter: 'all' - continue_on_error: ${{ inputs.continue_on_error == '' && true || inputs.continue_on_error }} - - call-nightly-amd-rocm720: - if: needs.resolve-aiter.outputs.run_nightly_amd_rocm720 == 'true' - needs: resolve-aiter - uses: ./.github/workflows/nightly-test-amd-rocm720.yml - secrets: inherit - with: - ref: amd/aiter-ci - rocm_version: rocm724 + rocm_version: all aiter_ref: ${{ needs.resolve-aiter.outputs.aiter_sha }} job_filter: 'all' continue_on_error: ${{ inputs.continue_on_error == '' && true || inputs.continue_on_error }} call-pr-test-amd: + name: PR Test AMD (all ROCm versions) if: needs.resolve-aiter.outputs.run_pr_test_amd == 'true' needs: resolve-aiter uses: ./.github/workflows/pr-test-amd.yml secrets: inherit with: ref: amd/aiter-ci - run_all_tests: true - aiter_ref: ${{ needs.resolve-aiter.outputs.aiter_sha }} - continue_on_error: ${{ inputs.continue_on_error == '' && true || inputs.continue_on_error }} - - call-pr-test-amd-rocm720: - if: needs.resolve-aiter.outputs.run_pr_test_amd_rocm720 == 'true' - needs: resolve-aiter - uses: ./.github/workflows/pr-test-amd-rocm720.yml - secrets: inherit - with: - ref: amd/aiter-ci - rocm_version: rocm724 + rocm_version: all run_all_tests: true aiter_ref: ${{ needs.resolve-aiter.outputs.aiter_sha }} continue_on_error: ${{ inputs.continue_on_error == '' && true || inputs.continue_on_error }} @@ -139,9 +113,7 @@ jobs: needs: - resolve-aiter - call-nightly-amd - - call-nightly-amd-rocm720 - call-pr-test-amd - - call-pr-test-amd-rocm720 runs-on: ubuntu-latest steps: - name: Summary @@ -154,10 +126,8 @@ jobs: echo "" >> $GITHUB_STEP_SUMMARY echo "| Workflow | Result |" >> $GITHUB_STEP_SUMMARY echo "|----------|--------|" >> $GITHUB_STEP_SUMMARY - echo "| Nightly AMD (AITER Latest) | \`${{ needs.call-nightly-amd.result }}\` |" >> $GITHUB_STEP_SUMMARY - echo "| Nightly AMD ROCm 7.2 | \`${{ needs.call-nightly-amd-rocm720.result }}\` |" >> $GITHUB_STEP_SUMMARY - echo "| PR Test AMD ROCm 7.0 Shadow | \`${{ needs.call-pr-test-amd.result }}\` |" >> $GITHUB_STEP_SUMMARY - echo "| PR Test AMD ROCm 7.2 Gate | \`${{ needs.call-pr-test-amd-rocm720.result }}\` |" >> $GITHUB_STEP_SUMMARY + echo "| Nightly AMD ROCm10 + ROCm7.2.4 + ROCm7.2.0 | \`${{ needs.call-nightly-amd.result }}\` |" >> $GITHUB_STEP_SUMMARY + echo "| PR Test AMD ROCm10 + ROCm7.2.4 + ROCm7.2.0 Gate | \`${{ needs.call-pr-test-amd.result }}\` |" >> $GITHUB_STEP_SUMMARY - name: Check if any job failed run: | diff --git a/.github/workflows/amd-ci-job-monitor.yml b/.github/workflows/amd-ci-job-monitor.yml index e1dd2a1984ba..e091fc04e43b 100644 --- a/.github/workflows/amd-ci-job-monitor.yml +++ b/.github/workflows/amd-ci-job-monitor.yml @@ -53,9 +53,9 @@ jobs: id: select-workflows run: | if [[ -n "${{ inputs.job_filter }}" ]]; then - echo "workflows=pr-test-amd-rocm720.yml" >> "$GITHUB_OUTPUT" + echo "workflows=pr-test-amd.yml" >> "$GITHUB_OUTPUT" else - echo "workflows=pr-test-amd.yml,nightly-test-amd.yml,pr-test-amd-rocm720.yml,nightly-test-amd-rocm720.yml,nightly-amd-mi355x-disagg.yml" >> "$GITHUB_OUTPUT" + echo "workflows=pr-test-amd.yml,nightly-test-amd.yml,nightly-amd-mi355x-disagg.yml" >> "$GITHUB_OUTPUT" fi - name: Fetch Actions data snapshot @@ -106,7 +106,7 @@ jobs: python scripts/ci/utils/query_job_status.py \ --repo ${{ github.repository }} \ --job "${{ inputs.job_filter }}" \ - --workflow "pr-test-amd-rocm720.yml" \ + --workflow "pr-test-amd.yml" \ --hours ${{ inputs.hours || '24' }} \ --input-data-file ci-data/actions-job-snapshot.json \ --summary @@ -119,8 +119,6 @@ jobs: outputs: pr_jobs: ${{ steps.parse.outputs.pr_jobs }} nightly_jobs: ${{ steps.parse.outputs.nightly_jobs }} - pr_rocm700_jobs: ${{ steps.parse.outputs.pr_rocm700_jobs }} - nightly_rocm700_jobs: ${{ steps.parse.outputs.nightly_rocm700_jobs }} disagg_jobs: ${{ steps.parse.outputs.disagg_jobs }} steps: - name: Checkout code @@ -131,55 +129,34 @@ jobs: run: | set -euo pipefail - # Parse the default ROCm 7.2 PR gate (exclude utility jobs) + # Parse the canonical multi-version AMD PR gate (exclude utility jobs) # Excluded: call-gate, check-changes, finish, cancel, check-all-jobs, # plus the stage waits -- they run on ubuntu-latest, so the # self-hosted-only snapshot has nothing for them. Keep the extra call: # its delegated GPU jobs use the explicit caller name as their prefix. - # Job ids keep the -rocm720 lane suffix but display names no longer do, - # so strip it to get the prefix query_job_status.py matches against. - pr_jobs=$(yq -r '.jobs | keys | .[]' .github/workflows/pr-test-amd-rocm720.yml | \ - grep -v -E '^(call-gate|check-changes|pr-test-amd-rocm720-finish|cancel|check-all-jobs|wait-for-stage-[ab]-amd-rocm720)$' | \ - sed 's/-rocm720$//' | \ + pr_jobs=$(yq -r '.jobs | keys | .[]' .github/workflows/pr-test-amd.yml | \ + grep -v -E '^(call-gate|check-changes|pr-test-amd-finish|cancel|check-all-jobs|wait-for-stage-[ab]-amd)$' | \ jq -R -s -c 'split("\n") | map(select(length > 0))') echo "pr_jobs=$pr_jobs" >> $GITHUB_OUTPUT echo "PR jobs: $pr_jobs" - # Parse the default ROCm 7.2 nightly (exclude utility jobs) + # Parse the canonical multi-version AMD nightly (exclude utility jobs) # Excluded: check-all-jobs # One nightly run covers every flavor, so ask for each by its full name: # a stem-only prefix matches every flavor and merges them into a single # row, hiding a regression that only reproduces on one image. Nightly # names carry the runner too, so pair each job id with its runs-on. # Keep in sync with the rocm_version matrix in the nightly workflow. - rocm_flavors="rocm724 rocm720" - nightly_jobs=$(yq -r '.jobs | to_entries[] | .key + " " + .value."runs-on"' .github/workflows/nightly-test-amd-rocm720.yml | \ + rocm_flavors="rocm10 rocm724 rocm720" + nightly_jobs=$(yq -r '.jobs | to_entries[] | .key + " " + .value."runs-on"' .github/workflows/nightly-test-amd.yml | \ grep -v -E '^check-all-jobs ' | \ while read -r id runner; do - stem=${id%-rocm720} - for f in $rocm_flavors; do echo "$stem ($f, $runner)"; done + for f in $rocm_flavors; do echo "$id ($f, $runner)"; done done | \ jq -R -s -c 'split("\n") | map(select(length > 0))') echo "nightly_jobs=$nightly_jobs" >> $GITHUB_OUTPUT echo "Nightly jobs: $nightly_jobs" - # Parse the ROCm 7.0 rollback shadow (exclude utility jobs) - # Excluded: call-gate, check-changes, finish, and the stage waits. - # Keep the reusable extra caller so its nested GPU jobs remain covered. - pr_rocm700_jobs=$(yq -r '.jobs | keys | .[]' .github/workflows/pr-test-amd.yml | \ - grep -v -E '^(call-gate|check-changes|pr-test-amd-finish|wait-for-stage-[ab]-amd)$' | \ - jq -R -s -c 'split("\n") | map(select(length > 0))') - echo "pr_rocm700_jobs=$pr_rocm700_jobs" >> $GITHUB_OUTPUT - echo "PR ROCm 7.0 shadow jobs: $pr_rocm700_jobs" - - # Parse the ROCm 7.0 rollback shadow nightly (exclude utility jobs) - # Excluded: check-all-jobs - nightly_rocm700_jobs=$(yq -r '.jobs | keys | .[]' .github/workflows/nightly-test-amd.yml | \ - grep -v -E '^(check-all-jobs)$' | \ - jq -R -s -c 'split("\n") | map(select(length > 0))') - echo "nightly_rocm700_jobs=$nightly_rocm700_jobs" >> $GITHUB_OUTPUT - echo "Nightly ROCm 7.0 shadow jobs: $nightly_rocm700_jobs" - # Parse nightly-amd-mi355x-disagg.yml (exclude utility jobs) # Excluded: setup, collect-results -- both run on ubuntu-latest and the # snapshot only keeps self-hosted jobs, so they would report as empty. @@ -223,7 +200,7 @@ jobs: python scripts/ci/utils/query_job_status.py \ --repo ${{ github.repository }} \ --job "${{ matrix.job_name }}" \ - --workflow "pr-test-amd-rocm720.yml" \ + --workflow "pr-test-amd.yml" \ --hours ${{ inputs.hours || '24' }} \ --input-data-file ci-data/actions-job-snapshot.json \ --summary @@ -257,84 +234,6 @@ jobs: path: ci-data - name: Generate Nightly Report - timeout-minutes: 15 - run: | - python scripts/ci/utils/query_job_status.py \ - --repo ${{ github.repository }} \ - --job "${{ matrix.job_name }}" \ - --workflow "nightly-test-amd-rocm720.yml" \ - --hours ${{ inputs.hours || '24' }} \ - --input-data-file ci-data/actions-job-snapshot.json \ - --summary - - # ROCm 7.0 rollback-shadow reports using dynamic matrix - pr-rocm700-ci-reports: - name: PR ROCm700 Shadow - ${{ matrix.job_name }} - needs: [parse-workflows, fetch-actions-data] - if: ${{ !inputs.job_filter }} - runs-on: ubuntu-latest - strategy: - fail-fast: false - matrix: - job_name: ${{ fromJson(needs.parse-workflows.outputs.pr_rocm700_jobs) }} - steps: - - name: Checkout code - uses: actions/checkout@v4 - - - name: Set up Python - uses: actions/setup-python@v5 - with: - python-version: '3.10' - - - name: Install dependencies - run: pip install tabulate - - - name: Download Actions data snapshot - uses: actions/download-artifact@v4 - with: - name: actions-job-snapshot - path: ci-data - - - name: Generate PR ROCm 7.0 Shadow Report - timeout-minutes: 15 - run: | - python scripts/ci/utils/query_job_status.py \ - --repo ${{ github.repository }} \ - --job "${{ matrix.job_name }}" \ - --workflow "pr-test-amd.yml" \ - --hours ${{ inputs.hours || '24' }} \ - --input-data-file ci-data/actions-job-snapshot.json \ - --summary - - # ROCm 7.0 rollback-shadow nightly reports using dynamic matrix - nightly-rocm700-reports: - name: Nightly ROCm700 Shadow - ${{ matrix.job_name }} - needs: [parse-workflows, fetch-actions-data] - if: ${{ !inputs.job_filter }} - runs-on: ubuntu-latest - strategy: - fail-fast: false - matrix: - job_name: ${{ fromJson(needs.parse-workflows.outputs.nightly_rocm700_jobs) }} - steps: - - name: Checkout code - uses: actions/checkout@v4 - - - name: Set up Python - uses: actions/setup-python@v5 - with: - python-version: '3.10' - - - name: Install dependencies - run: pip install tabulate - - - name: Download Actions data snapshot - uses: actions/download-artifact@v4 - with: - name: actions-job-snapshot - path: ci-data - - - name: Generate Nightly ROCm 7.0 Shadow Report timeout-minutes: 15 run: | python scripts/ci/utils/query_job_status.py \ @@ -414,7 +313,7 @@ jobs: python scripts/ci/utils/query_job_status.py \ --repo ${{ github.repository }} \ --runner-report \ - --workflow "pr-test-amd.yml,nightly-test-amd.yml,pr-test-amd-rocm720.yml,nightly-test-amd-rocm720.yml,nightly-amd-mi355x-disagg.yml" \ + --workflow "pr-test-amd.yml,nightly-test-amd.yml,nightly-amd-mi355x-disagg.yml" \ --hours ${{ inputs.hours || '24' }} \ --input-data-file ci-data/actions-job-snapshot.json \ --summary diff --git a/.github/workflows/bot-bump-sglang-version.yml b/.github/workflows/bot-bump-sglang-version.yml index b764d0b50075..125229b30369 100644 --- a/.github/workflows/bot-bump-sglang-version.yml +++ b/.github/workflows/bot-bump-sglang-version.yml @@ -62,10 +62,10 @@ jobs: run-nightly-tests-amd: needs: bump-sglang-version - uses: ./.github/workflows/nightly-test-amd-rocm720.yml + uses: ./.github/workflows/nightly-test-amd.yml with: ref: ${{ needs.bump-sglang-version.outputs.branch_name }} - rocm_version: rocm724 + rocm_version: rocm10 secrets: inherit run-nightly-tests-npu: diff --git a/.github/workflows/cancel-unfinished-pr-tests.yml b/.github/workflows/cancel-unfinished-pr-tests.yml index dc08ed9acb4c..4b0218da802f 100644 --- a/.github/workflows/cancel-unfinished-pr-tests.yml +++ b/.github/workflows/cancel-unfinished-pr-tests.yml @@ -8,8 +8,8 @@ on: required: true type: string default: 'pr-test.yml pr-test-extra.yml' - include_high_priority: - description: 'Also cancel runs from high-priority PRs' + include_highest_priority: + description: 'Also cancel runs from PRs labelled highest-priority' required: false type: boolean default: false @@ -36,7 +36,7 @@ jobs: GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} REPO: ${{ github.repository }} WORKFLOWS: ${{ github.event.inputs.workflows }} - INCLUDE_HIGH_PRIORITY: ${{ github.event.inputs.include_high_priority }} + INCLUDE_HIGHEST_PRIORITY: ${{ github.event.inputs.include_highest_priority }} INCLUDE_RERUN_TEST: ${{ github.event.inputs.include_rerun_test }} shell: bash run: | @@ -49,7 +49,7 @@ jobs: fi echo "Targeting ${#WORKFLOW_FILES[@]} workflow(s): ${WORKFLOW_FILES[*]}" - echo "include_high_priority=$INCLUDE_HIGH_PRIORITY, include_rerun_test=$INCLUDE_RERUN_TEST" + echo "include_highest_priority=$INCLUDE_HIGHEST_PRIORITY, include_rerun_test=$INCLUDE_RERUN_TEST" echo "" # Decide whether to cancel run_id given a PR-lookup endpoint. @@ -99,12 +99,12 @@ jobs: return fi - if echo "$labels" | grep -Fxq "high priority"; then - if [ "$INCLUDE_HIGH_PRIORITY" != "true" ]; then - echo " 🛑 Skipping (high priority label)" + if echo "$labels" | grep -Fxq "highest-priority"; then + if [ "$INCLUDE_HIGHEST_PRIORITY" != "true" ]; then + echo " 🛑 Skipping (highest-priority label)" return fi - echo " ⚠️ High priority PR, but include_high_priority is enabled" + echo " ⚠️ highest-priority PR, but include_highest_priority is enabled" fi echo " 🚫 Cancelling..." diff --git a/.github/workflows/ci-coverage-overview.yml b/.github/workflows/ci-coverage-overview.yml index 121a68764f29..96c4ce57210f 100644 --- a/.github/workflows/ci-coverage-overview.yml +++ b/.github/workflows/ci-coverage-overview.yml @@ -63,10 +63,28 @@ jobs: run: | python scripts/ci/utils/ci_coverage_report.py --section by-suite + by-gpu-count: + name: Tests by GPU Count + runs-on: ubuntu-latest + steps: + - name: Checkout code + uses: actions/checkout@v4 + + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: '3.10' + + - name: Generate Tests by GPU Count Report + run: | + python scripts/ci/utils/ci_coverage_report.py --section by-gpu-count + unit-test-coverage: name: Unit Test Code Coverage runs-on: 1-gpu-h100 - timeout-minutes: 30 + # ~6 min install + ~22 min of tests, and the suite keeps growing + # (7.7k collected items in Aug, 11k now). + timeout-minutes: 60 steps: - name: Checkout code uses: actions/checkout@v4 @@ -88,13 +106,19 @@ jobs: pip install -e "python/[test]" - name: Run unit tests with coverage - timeout-minutes: 10 + # Non-gating on purpose: running the whole tree in one process + # cross-contaminates tests that CI proper runs one file per + # process. `| tee` used to hide that by accident; pipefail plus an + # explicit `|| echo` makes it deliberate and reports the counts. + timeout-minutes: 45 run: | + set -o pipefail pytest test/registered/unit/ \ --cov --cov-config=.coveragerc \ --cov-report=term-missing:skip-covered \ --continue-on-collection-errors \ - -v | tee coverage_output.txt + -v | tee coverage_output.txt \ + || echo "pytest exited $? -- reported below, not gating this job" - name: Write coverage to summary if: always() @@ -106,7 +130,8 @@ jobs: # Test result line (e.g., "== 42 passed, 1 failed in 23.5s ==") echo '```' >> $GITHUB_STEP_SUMMARY - grep -E '^=+.*passed' coverage_output.txt >> $GITHUB_STEP_SUMMARY || true + grep -E '^=+.*passed' coverage_output.txt >> $GITHUB_STEP_SUMMARY \ + || echo "pytest did not reach its summary line (step killed or crashed)" >> $GITHUB_STEP_SUMMARY echo "" >> $GITHUB_STEP_SUMMARY # Coverage total, reformatted awk '/^TOTAL / { for(i=1;i<=NF;i++) if($i~/^[0-9]+$/ || $i~/^[0-9]+%$/) a[++n]=$i; if(n>=3) printf "TOTAL Stmts: %s Miss: %s Cover: %s\n", a[1], a[2], a[3] }' coverage_output.txt >> $GITHUB_STEP_SUMMARY || true diff --git a/.github/workflows/ci-lark-notify.yml b/.github/workflows/ci-lark-notify.yml new file mode 100644 index 000000000000..c520ae765d2b --- /dev/null +++ b/.github/workflows/ci-lark-notify.yml @@ -0,0 +1,103 @@ +name: CI Lark Notify + +# Lark cards for CUDA CI health; see scripts/ci_monitor/README.md. + +on: + workflow_run: + workflows: ["Nightly Test (Nvidia)", "Weekly Test (Nvidia)", "PR Test Base"] + types: [completed] + branches: [main] + schedule: + - cron: '*/15 * * * *' # runner-health + workflow_dispatch: + inputs: + task: + description: 'Which report to run' + required: true + type: choice + options: [ci-status, runner-health] + run_id: + description: 'Workflow run id (ci-status only)' + required: false + type: string + dry_run: + description: 'Print the card instead of posting it' + required: false + type: boolean + default: false + +permissions: + contents: read + actions: read + +env: + PYTHONUNBUFFERED: 1 + DRY_RUN_FLAG: ${{ inputs.dry_run && '--dry-run' || '' }} + +jobs: + ci-status: + if: >- + github.repository == 'sgl-project/sglang' && ( + (github.event_name == 'workflow_run' && github.event.workflow_run.event == 'schedule') || + (github.event_name == 'workflow_dispatch' && inputs.task == 'ci-status') + ) + runs-on: ubuntu-latest + timeout-minutes: 15 + steps: + - uses: actions/checkout@v4 + with: + sparse-checkout: scripts/ci_monitor + - uses: actions/setup-python@v5 + with: + python-version: '3.12' + - name: Post CI status card + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + LARK_WEBHOOK: ${{ secrets.LARK_WEBHOOK }} + RUN_ID: ${{ github.event.workflow_run.id || inputs.run_id }} + run: | + python scripts/ci_monitor/lark_notify.py $DRY_RUN_FLAG ci-status \ + --run-id "$RUN_ID" \ + ${{ github.event_name == 'workflow_dispatch' && '--any-event' || '' }} + + runner-health: + if: >- + github.repository == 'sgl-project/sglang' && ( + (github.event_name == 'schedule' && github.event.schedule == '*/15 * * * *') || + (github.event_name == 'workflow_dispatch' && inputs.task == 'runner-health') + ) + runs-on: ubuntu-latest + timeout-minutes: 15 + # Serialize so the state file is never written by two runs at once. + concurrency: + group: ci-lark-notify-runner-health + cancel-in-progress: false + steps: + - uses: actions/checkout@v4 + with: + sparse-checkout: scripts/ci_monitor + - uses: actions/setup-python@v5 + with: + python-version: '3.12' + # Key never matches (run_id is unique) so restore-keys picks the most + # recent state and the save step writes a fresh entry every run. + - name: Restore runner-health state + uses: actions/cache/restore@v4 + with: + path: runner-health-state.json + key: lark-runner-health-${{ github.run_id }} + restore-keys: lark-runner-health- + - name: Check runner pools + env: + GITHUB_TOKEN: ${{ secrets.GH_PAT_FOR_RUNNER_ADMIN }} + LARK_WEBHOOK: ${{ secrets.LARK_WEBHOOK }} + run: | + python scripts/ci_monitor/lark_notify.py $DRY_RUN_FLAG runner-health \ + --state-file runner-health-state.json \ + --remind-hours 6 + - name: Save runner-health state + if: ${{ !inputs.dry_run }} + uses: actions/cache/save@v4 + with: + path: runner-health-state.json + key: lark-runner-health-${{ github.run_id }} diff --git a/.github/workflows/close-stale-prs.yml b/.github/workflows/close-stale-prs.yml index 96f83c806aff..1965b9960d89 100644 --- a/.github/workflows/close-stale-prs.yml +++ b/.github/workflows/close-stale-prs.yml @@ -38,7 +38,7 @@ jobs: const QUOTA_IDLE = 7; const STALE_DAYS = 90; - const KEEP_LABELS = new Set(['high priority', 'keep-open', 'good first issue']); + const KEEP_LABELS = new Set(['highest-priority', 'keep-open', 'good first issue']); const WIP_MARKER = /^\s*\[?\s*(wip|do[ _-]?not[ _-]?merge|dnm|draft)\s*\]?/i; // Scheduled runs always act; only manual runs can be dry. diff --git a/.github/workflows/coverage-collection-npu.yml b/.github/workflows/coverage-collection-npu.yml new file mode 100644 index 000000000000..8493bf30372f --- /dev/null +++ b/.github/workflows/coverage-collection-npu.yml @@ -0,0 +1,27 @@ +# Coverage data collection pipeline for the NPU precision test selection. +# Delegates to pr-test-npu.yml with coverage_mode=true, which switches: +# - Runners to the coverage pool (shared /root/.cache with setup-covstub) +# - use_coverage_runner=true on all test jobs +# - Skips pr-gate, multimodal-gen, recommend, analyze +# - Runs setup-covstub to assemble coverage data and publish it to the +# shared disk at a fixed path (/root/.cache/tests/precise-test/coverage) +# pr-test-npu-finish is the final gate in the called workflow. +# +# Manually triggered only (workflow_dispatch): PRs run the standard +# pr-test-npu.yml instead; this pipeline builds the full coverage baseline +# on demand. +name: Coverage Collection (NPU) + +on: + workflow_dispatch: + +concurrency: + group: npu-coverage-collection-${{ github.ref }} + cancel-in-progress: true + +jobs: + coverage: + uses: ./.github/workflows/pr-test-npu.yml + with: + coverage_mode: true + secrets: inherit diff --git a/.github/workflows/diffusion-ci-gt-gen-npu.yml b/.github/workflows/diffusion-ci-gt-gen-npu.yml index 36e56971462d..b4eed2f8db02 100644 --- a/.github/workflows/diffusion-ci-gt-gen-npu.yml +++ b/.github/workflows/diffusion-ci-gt-gen-npu.yml @@ -94,7 +94,7 @@ jobs: matrix: ${{ fromJson(needs.compute-diffusion-partitions.outputs.matrix-1npu) }} timeout-minutes: 150 container: - image: swr.cn-southwest-2.myhuaweicloud.com/base_image/ascend-ci/cann:9.0.0-a3-ubuntu22.04-py3.11 + image: swr.cn-southwest-2.myhuaweicloud.com/base_image/ascend-ci/cann:9.1.0-a3-ubuntu22.04-py3.12 steps: - name: Checkout code uses: actions/checkout@v4 @@ -166,7 +166,7 @@ jobs: matrix: ${{ fromJson(needs.compute-diffusion-partitions.outputs.matrix-2npu) }} timeout-minutes: 150 container: - image: swr.cn-southwest-2.myhuaweicloud.com/base_image/ascend-ci/cann:9.0.0-a3-ubuntu22.04-py3.11 + image: swr.cn-southwest-2.myhuaweicloud.com/base_image/ascend-ci/cann:9.1.0-a3-ubuntu22.04-py3.12 steps: - name: Checkout code uses: actions/checkout@v4 diff --git a/.github/workflows/diffusion-ci-gt-gen.yml b/.github/workflows/diffusion-ci-gt-gen.yml index 6bdcf8dad9af..75c46d96c89e 100644 --- a/.github/workflows/diffusion-ci-gt-gen.yml +++ b/.github/workflows/diffusion-ci-gt-gen.yml @@ -48,6 +48,11 @@ on: required: false default: 'main' type: string + publish: + description: 'Publish the generated GT to sgl-project/ci-data-diffusion. false only uploads the run artifacts, which needs no GH_PAT_FOR_NIGHTLY_CI_DATA.' + required: false + default: true + type: boolean concurrency: group: diffusion-ci-gt-gen-${{ github.ref }}-${{ inputs.output_name || inputs.case_ids || inputs.official_case_ids || inputs.official_source_group || inputs.run_official_cases || 'default' }} @@ -73,6 +78,7 @@ jobs: case-count: ${{ steps.compute.outputs.case-count }} steps: - name: Verify write access to sgl-project/ci-data-diffusion + if: ${{ inputs.publish }} env: GH_TOKEN: ${{ secrets.GH_PAT_FOR_NIGHTLY_CI_DATA }} run: | @@ -307,7 +313,7 @@ jobs: - name: Validate generated GT images env: - GITHUB_TOKEN: ${{ secrets.GH_PAT_FOR_NIGHTLY_CI_DATA }} + GITHUB_TOKEN: ${{ inputs.publish && secrets.GH_PAT_FOR_NIGHTLY_CI_DATA || github.token }} run: | python scripts/ci/utils/diffusion/publish_diffusion_gt.py \ --source-dir python/${{ env.OUTPUT_NAME }} \ @@ -315,6 +321,7 @@ jobs: --check-only - name: Publish official GT images to sgl-project/ci-data-diffusion + if: ${{ inputs.publish }} env: GITHUB_TOKEN: ${{ secrets.GH_PAT_FOR_NIGHTLY_CI_DATA }} run: | @@ -347,6 +354,7 @@ jobs: python-version: '3.10' - name: Verify write access to sgl-project/ci-data-diffusion + if: ${{ inputs.publish }} env: GH_TOKEN: ${{ secrets.GH_PAT_FOR_NIGHTLY_CI_DATA }} run: | @@ -427,7 +435,7 @@ jobs: - name: Validate generated GT images env: - GITHUB_TOKEN: ${{ secrets.GH_PAT_FOR_NIGHTLY_CI_DATA }} + GITHUB_TOKEN: ${{ inputs.publish && secrets.GH_PAT_FOR_NIGHTLY_CI_DATA || github.token }} run: | python scripts/ci/utils/diffusion/publish_diffusion_gt.py \ --source-dir python/${{ env.OUTPUT_NAME }} \ @@ -435,6 +443,7 @@ jobs: --check-only - name: Publish GT images to sgl-project/ci-data-diffusion + if: ${{ inputs.publish }} env: GITHUB_TOKEN: ${{ secrets.GH_PAT_FOR_NIGHTLY_CI_DATA }} run: | @@ -504,7 +513,7 @@ jobs: - name: Validate generated GT images env: - GITHUB_TOKEN: ${{ secrets.GH_PAT_FOR_NIGHTLY_CI_DATA }} + GITHUB_TOKEN: ${{ inputs.publish && secrets.GH_PAT_FOR_NIGHTLY_CI_DATA || github.token }} run: | python scripts/ci/utils/diffusion/publish_diffusion_gt.py \ --source-dir python/${{ env.OUTPUT_NAME }} \ @@ -512,6 +521,7 @@ jobs: --check-only - name: Publish GT images to sgl-project/ci-data-diffusion + if: ${{ inputs.publish }} env: GITHUB_TOKEN: ${{ secrets.GH_PAT_FOR_NIGHTLY_CI_DATA }} run: | @@ -578,8 +588,78 @@ jobs: retention-days: 7 - name: Validate generated GT images + env: + GITHUB_TOKEN: ${{ inputs.publish && secrets.GH_PAT_FOR_NIGHTLY_CI_DATA || github.token }} + run: | + python scripts/ci/utils/diffusion/publish_diffusion_gt.py \ + --source-dir python/${{ env.OUTPUT_NAME }} \ + --target-dir "${{ env.PUBLISH_TARGET_DIR }}" \ + --check-only + + - name: Publish GT images to sgl-project/ci-data-diffusion + if: ${{ inputs.publish }} env: GITHUB_TOKEN: ${{ secrets.GH_PAT_FOR_NIGHTLY_CI_DATA }} + run: | + python scripts/ci/utils/diffusion/publish_diffusion_gt.py \ + --source-dir python/${{ env.OUTPUT_NAME }} \ + --target-dir "${{ env.PUBLISH_TARGET_DIR }}" + + multimodal-diffusion-gen-5090: + # The 5090 canary suite is six cases; it is generated whole rather than + # partitioned, so it needs no entry in the partition plan. + needs: compute-diffusion-partitions + if: needs.compute-diffusion-partitions.result == 'success' + runs-on: 1-gpu-5090 + timeout-minutes: 150 + steps: + - name: Checkout code + uses: actions/checkout@v4 + with: + ref: ${{ inputs.ref || github.ref }} + + - name: Prepare AOT dist for prebuilt wheel + if: inputs.kernel_artifact_run_id != '' + run: | + ls -alh python/sglang/kernels/aot/dist || true + rm -rf python/sglang/kernels/aot/dist/* || true + + - name: Download prebuilt sgl-kernel wheel + if: inputs.kernel_artifact_run_id != '' + uses: actions/download-artifact@v4 + with: + path: python/sglang/kernels/aot/dist/ + merge-multiple: true + name: wheel-python3.10-cuda13.0 + run-id: ${{ inputs.kernel_artifact_run_id }} + github-token: ${{ secrets.GITHUB_TOKEN }} + + - name: Install dependencies + run: | + CUSTOM_BUILD_SGL_KERNEL="${{ inputs.kernel_artifact_run_id != '' && 'true' || 'false' }}" \ + bash scripts/ci/cuda/ci_install_dependency.sh diffusion + + - name: Generate outputs + env: + RUNAI_STREAMER_MEMORY_LIMIT: 0 + run: | + cd python + python -m sglang.multimodal_gen.test.scripts.gen_diffusion_ci_outputs \ + --suite 1-gpu-5090 \ + --out-dir ./${{ env.OUTPUT_NAME }} \ + ${{ inputs.case_ids != '' && format('--case-ids {0}', inputs.case_ids) || '' }} + + - name: Upload artifact + uses: actions/upload-artifact@v4 + with: + name: ${{ env.OUTPUT_NAME }}-5090 + path: python/${{ env.OUTPUT_NAME }} + retention-days: 7 + if-no-files-found: ignore + + - name: Validate generated GT images + env: + GITHUB_TOKEN: ${{ inputs.publish && secrets.GH_PAT_FOR_NIGHTLY_CI_DATA || github.token }} run: | python scripts/ci/utils/diffusion/publish_diffusion_gt.py \ --source-dir python/${{ env.OUTPUT_NAME }} \ @@ -587,6 +667,7 @@ jobs: --check-only - name: Publish GT images to sgl-project/ci-data-diffusion + if: ${{ inputs.publish }} env: GITHUB_TOKEN: ${{ secrets.GH_PAT_FOR_NIGHTLY_CI_DATA }} run: | diff --git a/.github/workflows/full-test-npu.yml b/.github/workflows/full-test-npu.yml index 42360ff67d58..77226434044f 100644 --- a/.github/workflows/full-test-npu.yml +++ b/.github/workflows/full-test-npu.yml @@ -19,15 +19,10 @@ on: type: string default: 'all' image_a3: - description: 'The a3 running docker image of the test task.' + description: 'The a3 running docker image of the test task. Must be an image with sglang preinstalled (e.g. lmsysorg/sglang:*a3*).' required: false type: string - default: 'swr.cn-southwest-2.myhuaweicloud.com/base_image/ascend-ci/cann:9.0.0-a3-ubuntu22.04-py3.11' - skip_install_flag: - description: 'Indicates whether to skip the installation of sglang, defaulting to false.' - required: false - type: string - default: 'false' + default: 'swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:main-cann9.1.0-a3' concurrency: group: full-test-npu-${{ inputs.ref || github.ref }} @@ -40,7 +35,6 @@ jobs: ref: ${{ steps.set-vars.outputs.ref }} job_filter: ${{ steps.set-vars.outputs.job_filter }} image_a3: ${{ steps.set-vars.outputs.image_a3 }} - skip_install_flag: ${{ steps.set-vars.outputs.skip_install_flag }} steps: # When triggered by PR, no inputs parameters are used. The latest community code is tested by default. - name: Set image config @@ -59,17 +53,11 @@ jobs: fi if [ -z "${{ inputs.image_a3 }}" ]; then - echo "image_a3=swr.cn-southwest-2.myhuaweicloud.com/base_image/ascend-ci/cann:8.5.0-a3-ubuntu22.04-py3.11" >> $GITHUB_OUTPUT + echo "image_a3=swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:main-cann9.1.0-a3" >> $GITHUB_OUTPUT else echo "image_a3=${{ inputs.image_a3 }}" >> $GITHUB_OUTPUT fi - if [ -z "${{ inputs.skip_install_flag }}" ]; then - echo "skip_install_flag=false" >> $GITHUB_OUTPUT - else - echo "skip_install_flag=${{ inputs.skip_install_flag }}" >> $GITHUB_OUTPUT - fi - nighly-test-npu: needs: [set-image-config] name: nightly-test-npu @@ -79,260 +67,225 @@ jobs: ref: ${{ needs.set-image-config.outputs.ref }} job_filter: ${{ needs.set-image-config.outputs.job_filter }} image_a3: ${{ needs.set-image-config.outputs.image_a3 }} - skip_install_flag: ${{ needs.set-image-config.outputs.skip_install_flag }} secrets: inherit full-1-npu-a3: - needs: [set-image-config] + name: full-1-npu-a3 if: ${{ (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') }} - runs-on: linux-aarch64-a3-2 - container: + needs: [set-image-config] + uses: ./.github/workflows/_npu-pr-test-stage.yml + with: + self_name: full-1-npu-a3 + runner_config: linux-aarch64-a3-2 image: ${{ needs.set-image-config.outputs.image_a3 }} - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ needs.set-image-config.outputs.ref || github.ref }} - - - name: Install dependencies - env: - TORCH_CACHE_URL: "http://cache-service.nginx-pypi-cache.svc.cluster.local/whl/cpu" - PYPI_CACHE_URL: "http://cache-service.nginx-pypi-cache.svc.cluster.local/pypi/simple" - UV_INDEX_URL: "http://cache-service.nginx-pypi-cache.svc.cluster.local/pypi/simple" - GITHUB_PROXY_URL: "https://gh-proxy.test.osinfra.cn/" - run: | - # speed up by using infra cache services - CACHING_URL="cache-service.nginx-pypi-cache.svc.cluster.local" - sed -Ei "s@(ports|archive).ubuntu.com@${CACHING_URL}:8081@g" /etc/apt/sources.list - pip config set global.index-url http://${CACHING_URL}/pypi/simple - pip config set global.trusted-host "${CACHING_URL}" - - if [ ${{ needs.set-image-config.outputs.skip_install_flag }} != "true" ];then - bash scripts/ci/npu/npu_ci_install_dependency.sh a3 - fi - - # copy required file from our daily cache - cp ~/.cache/modelscope/hub/datasets/otavia/ShareGPT_Vicuna_unfiltered/ShareGPT_V3_unfiltered_cleaned_split.json /tmp - # copy gsm8k dataset - cp ~/.cache/modelscope/hub/datasets/tmp/test.jsonl /tmp - - - name: Print Log Information - run: | - bash scripts/ci/npu/npu_log_print.sh - - - name: Run test - timeout-minutes: 240 - env: - SGLANG_USE_MODELSCOPE: true - SGLANG_IS_IN_CI: true - HF_ENDPOINT: https://hf-mirror.com - TORCH_EXTENSIONS_DIR: /tmp/torch_extensions - PYTORCH_NPU_ALLOC_CONF: "expandable_segments:True" - STREAMS_PER_DEVICE: 32 - run: | - pip install sglang_router - hf download lmms-lab/MMMU --repo-type dataset - pip install sentence_transformers torchaudio==2.8.0 - pip install protobuf==6.31.1 zss pre-commit wandb>=0.16.0 tenacity==8.3.0 loguru openpyxl latex2sympy2 zstandard transformers-stream-generator tqdm-multiprocess pycocoevalcap - pip install yt-dlp sentencepiece==0.1.99 nltk av ftfy sqlitedict==2.1.0 sacrebleu>=1.5.0 pytablewriter black==24.1.0 isort==5.13.2 peft>=0.2.0 accelerate>=0.29.1 - pip install jsonlines httpx==0.25.0 evaluate>=0.4.0 datasets==2.16.1 numexpr xgrammar==0.2.1 numpy==1.26.4 dotenv - git clone --branch v0.3.3 --depth 1 https://github.com/EvolvingLMMs-Lab/lmms-eval.git - cd ./lmms-eval - nohup pip install . > lmmslog.txt 2>&1 & - sleep 120 - export PYTHONPATH=$PYTHONPATH:$(pwd) - cd ../ - cd test - python3 run_suite.py --hw npu --suite full-1-npu-a3 --nightly --continue-on-error --timeout-per-file 3600 + run_timeout_minutes: '240' + timeout_per_file: '3600' + ref: ${{ needs.set-image-config.outputs.ref }} + is_nightly_pipeline_job: true + skip_pr_test_health_check: 'true' + secrets: inherit full-2-npu-a3: - needs: [set-image-config] + name: full-2-npu-a3 if: ${{ (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') }} - runs-on: linux-aarch64-a3-2 - container: + needs: [set-image-config] + uses: ./.github/workflows/_npu-pr-test-stage.yml + with: + self_name: full-2-npu-a3 + runner_config: linux-aarch64-a3-2 image: ${{ needs.set-image-config.outputs.image_a3 }} - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ needs.set-image-config.outputs.ref || github.ref }} - - - name: Install dependencies - env: - TORCH_CACHE_URL: "http://cache-service.nginx-pypi-cache.svc.cluster.local/whl/cpu" - PYPI_CACHE_URL: "http://cache-service.nginx-pypi-cache.svc.cluster.local/pypi/simple" - UV_INDEX_URL: "http://cache-service.nginx-pypi-cache.svc.cluster.local/pypi/simple" - GITHUB_PROXY_URL: "https://gh-proxy.test.osinfra.cn/" - run: | - # speed up by using infra cache services - CACHING_URL="cache-service.nginx-pypi-cache.svc.cluster.local" - sed -Ei "s@(ports|archive).ubuntu.com@${CACHING_URL}:8081@g" /etc/apt/sources.list - pip config set global.index-url http://${CACHING_URL}/pypi/simple - pip config set global.trusted-host "${CACHING_URL}" - - if [ ${{ needs.set-image-config.outputs.skip_install_flag }} != "true" ];then - bash scripts/ci/npu/npu_ci_install_dependency.sh a3 - fi - - # copy required file from our daily cache - cp ~/.cache/modelscope/hub/datasets/otavia/ShareGPT_Vicuna_unfiltered/ShareGPT_V3_unfiltered_cleaned_split.json /tmp - # copy gsm8k dataset - cp ~/.cache/modelscope/hub/datasets/tmp/test.jsonl /tmp - - - name: Print Log Information - run: | - bash scripts/ci/npu/npu_log_print.sh - - - name: Run test - timeout-minutes: 240 - env: - SGLANG_USE_MODELSCOPE: true - SGLANG_IS_IN_CI: true - HF_ENDPOINT: https://hf-mirror.com - TORCH_EXTENSIONS_DIR: /tmp/torch_extensions - PYTORCH_NPU_ALLOC_CONF: "expandable_segments:True" - STREAMS_PER_DEVICE: 32 - run: | - pip install sglang_router - hf download lmms-lab/MMMU --repo-type dataset - pip install sentence_transformers torchaudio==2.8.0 - pip install protobuf==6.31.1 zss pre-commit wandb>=0.16.0 tenacity==8.3.0 loguru openpyxl latex2sympy2 zstandard transformers-stream-generator tqdm-multiprocess pycocoevalcap - pip install yt-dlp sentencepiece==0.1.99 nltk av ftfy sqlitedict==2.1.0 sacrebleu>=1.5.0 pytablewriter black==24.1.0 isort==5.13.2 peft>=0.2.0 accelerate>=0.29.1 - pip install jsonlines httpx==0.25.0 evaluate>=0.4.0 datasets==2.16.1 numexpr xgrammar==0.2.1 numpy==1.26.4 dotenv - git clone --branch v0.3.3 --depth 1 https://github.com/EvolvingLMMs-Lab/lmms-eval.git - cd ./lmms-eval - nohup pip install . > lmmslog.txt 2>&1 & - sleep 120 - export PYTHONPATH=$PYTHONPATH:$(pwd) - cd ../ - cd test - python3 run_suite.py --hw npu --suite full-2-npu-a3 --nightly --continue-on-error --timeout-per-file 3600 + run_timeout_minutes: '240' + timeout_per_file: '3600' + ref: ${{ needs.set-image-config.outputs.ref }} + is_nightly_pipeline_job: true + skip_pr_test_health_check: 'true' + secrets: inherit full-4-npu-a3: - needs: [set-image-config] + name: full-4-npu-a3 if: ${{ (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') }} - runs-on: linux-aarch64-a3-4 - container: + needs: [set-image-config] + uses: ./.github/workflows/_npu-pr-test-stage.yml + with: + self_name: full-4-npu-a3 + runner_config: linux-aarch64-a3-4 image: ${{ needs.set-image-config.outputs.image_a3 }} - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ needs.set-image-config.outputs.ref || github.ref }} - - - name: Install dependencies - env: - TORCH_CACHE_URL: "http://cache-service.nginx-pypi-cache.svc.cluster.local/whl/cpu" - PYPI_CACHE_URL: "http://cache-service.nginx-pypi-cache.svc.cluster.local/pypi/simple" - UV_INDEX_URL: "http://cache-service.nginx-pypi-cache.svc.cluster.local/pypi/simple" - GITHUB_PROXY_URL: "https://gh-proxy.test.osinfra.cn/" - run: | - # speed up by using infra cache services - CACHING_URL="cache-service.nginx-pypi-cache.svc.cluster.local" - sed -Ei "s@(ports|archive).ubuntu.com@${CACHING_URL}:8081@g" /etc/apt/sources.list - pip config set global.index-url http://${CACHING_URL}/pypi/simple - pip config set global.trusted-host "${CACHING_URL}" - - if [ ${{ needs.set-image-config.outputs.skip_install_flag }} != "true" ];then - bash scripts/ci/npu/npu_ci_install_dependency.sh a3 - fi + run_timeout_minutes: '240' + timeout_per_file: '3600' + ref: ${{ needs.set-image-config.outputs.ref }} + is_nightly_pipeline_job: true + skip_pr_test_health_check: 'true' + secrets: inherit - # copy required file from our daily cache - cp ~/.cache/modelscope/hub/datasets/otavia/ShareGPT_Vicuna_unfiltered/ShareGPT_V3_unfiltered_cleaned_split.json /tmp - # copy gsm8k dataset - cp ~/.cache/modelscope/hub/datasets/tmp/test.jsonl /tmp + full-8-npu-a3: + name: full-8-npu-a3 + if: ${{ (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') }} + needs: [set-image-config] + uses: ./.github/workflows/_npu-pr-test-stage.yml + with: + self_name: full-8-npu-a3 + runner_config: linux-aarch64-a3-8 + image: ${{ needs.set-image-config.outputs.image_a3 }} + run_timeout_minutes: '240' + timeout_per_file: '3600' + ref: ${{ needs.set-image-config.outputs.ref }} + is_nightly_pipeline_job: true + skip_pr_test_health_check: 'true' + secrets: inherit - - name: Print Log Information - run: | - bash scripts/ci/npu/npu_log_print.sh + full-16-npu-a3: + name: full-16-npu-a3 + if: ${{ (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') }} + needs: [set-image-config] + uses: ./.github/workflows/_npu-pr-test-stage.yml + with: + self_name: full-16-npu-a3 + runner_config: linux-aarch64-a3-16 + image: ${{ needs.set-image-config.outputs.image_a3 }} + run_timeout_minutes: '240' + timeout_per_file: '3600' + ref: ${{ needs.set-image-config.outputs.ref }} + is_nightly_pipeline_job: true + skip_pr_test_health_check: 'true' + secrets: inherit - - name: Run test - timeout-minutes: 240 - env: - SGLANG_USE_MODELSCOPE: true - SGLANG_IS_IN_CI: true - HF_ENDPOINT: https://hf-mirror.com - TORCH_EXTENSIONS_DIR: /tmp/torch_extensions - PYTORCH_NPU_ALLOC_CONF: "expandable_segments:True" - STREAMS_PER_DEVICE: 32 - run: | - pip install sglang_router - hf download lmms-lab/MMMU --repo-type dataset - pip install sentence_transformers torchaudio==2.8.0 - pip install protobuf==6.31.1 zss pre-commit wandb>=0.16.0 tenacity==8.3.0 loguru openpyxl latex2sympy2 zstandard transformers-stream-generator tqdm-multiprocess pycocoevalcap - pip install yt-dlp sentencepiece==0.1.99 nltk av ftfy sqlitedict==2.1.0 sacrebleu>=1.5.0 pytablewriter black==24.1.0 isort==5.13.2 peft>=0.2.0 accelerate>=0.29.1 - pip install jsonlines httpx==0.25.0 evaluate>=0.4.0 datasets==2.16.1 numexpr xgrammar==0.2.1 numpy==1.26.4 dotenv - git clone --branch v0.3.3 --depth 1 https://github.com/EvolvingLMMs-Lab/lmms-eval.git - cd ./lmms-eval - nohup pip install . > lmmslog.txt 2>&1 & - sleep 120 - export PYTHONPATH=$PYTHONPATH:$(pwd) - cd ../ - cd test - python3 run_suite.py --hw npu --suite full-4-npu-a3 --nightly --continue-on-error --timeout-per-file 3600 + full-acc-2-npu-a3: + name: full-acc-2-npu-a3 + if: ${{ (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') }} + needs: [set-image-config] + uses: ./.github/workflows/_npu-single-node-test-stage.yml + with: + runner: linux-aarch64-a3-2- + test_type: 'accuracy' + test_suite: full-acc-2-npu-a3 + image: ${{ needs.set-image-config.outputs.image_a3 }} + is_nightly_pipeline_job: true + install_sglang_deps: true + device_type_for_deps: 'a3' + skip_pr_test_health_check: 'true' + secrets: inherit - full-16-npu-a3: + full-acc-4-npu-a3: + name: full-acc-4-npu-a3 + if: ${{ (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') }} needs: [set-image-config] + uses: ./.github/workflows/_npu-single-node-test-stage.yml + with: + runner: linux-aarch64-a3-4- + test_type: 'accuracy' + test_suite: full-acc-4-npu-a3 + image: ${{ needs.set-image-config.outputs.image_a3 }} + is_nightly_pipeline_job: true + install_sglang_deps: true + device_type_for_deps: 'a3' + skip_pr_test_health_check: 'true' + secrets: inherit + + full-acc-16-npu-a3: + name: full-acc-16-npu-a3 if: ${{ (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') }} - runs-on: linux-aarch64-a3-16 - container: + needs: [set-image-config] + uses: ./.github/workflows/_npu-single-node-test-stage.yml + with: + runner: linux-aarch64-a3-16- + test_type: 'accuracy' + test_suite: full-acc-16-npu-a3 image: ${{ needs.set-image-config.outputs.image_a3 }} - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ needs.set-image-config.outputs.ref || github.ref }} + is_nightly_pipeline_job: true + install_sglang_deps: true + device_type_for_deps: 'a3' + skip_pr_test_health_check: 'true' + secrets: inherit - - name: Install dependencies - env: - TORCH_CACHE_URL: "http://cache-service.nginx-pypi-cache.svc.cluster.local/whl/cpu" - PYPI_CACHE_URL: "http://cache-service.nginx-pypi-cache.svc.cluster.local/pypi/simple" - UV_INDEX_URL: "http://cache-service.nginx-pypi-cache.svc.cluster.local/pypi/simple" - GITHUB_PROXY_URL: "https://gh-proxy.test.osinfra.cn/" - run: | - # speed up by using infra cache services - CACHING_URL="cache-service.nginx-pypi-cache.svc.cluster.local" - sed -Ei "s@(ports|archive).ubuntu.com@${CACHING_URL}:8081@g" /etc/apt/sources.list - pip config set global.index-url http://${CACHING_URL}/pypi/simple - pip config set global.trusted-host "${CACHING_URL}" + full-perf-2-npu-a3: + name: full-perf-2-npu-a3 + if: ${{ (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') }} + needs: [set-image-config] + uses: ./.github/workflows/_npu-pr-test-stage.yml + with: + self_name: full-perf-2-npu-a3 + runner_config: linux-aarch64-a3-800t-2 + image: ${{ needs.set-image-config.outputs.image_a3 }} + run_timeout_minutes: '240' + timeout_per_file: '3600' + ref: ${{ needs.set-image-config.outputs.ref }} + is_nightly_pipeline_job: true + skip_pr_test_health_check: 'true' + secrets: inherit - if [ ${{ needs.set-image-config.outputs.skip_install_flag }} != "true" ];then - bash scripts/ci/npu/npu_ci_install_dependency.sh a3 - fi + full-perf-4-npu-a3: + name: full-perf-4-npu-a3 + if: ${{ (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') }} + needs: [set-image-config] + uses: ./.github/workflows/_npu-pr-test-stage.yml + with: + self_name: full-perf-4-npu-a3 + runner_config: linux-aarch64-a3-800t-4 + image: ${{ needs.set-image-config.outputs.image_a3 }} + run_timeout_minutes: '240' + timeout_per_file: '3600' + ref: ${{ needs.set-image-config.outputs.ref }} + is_nightly_pipeline_job: true + skip_pr_test_health_check: 'true' + secrets: inherit - # copy required file from our daily cache - cp ~/.cache/modelscope/hub/datasets/otavia/ShareGPT_Vicuna_unfiltered/ShareGPT_V3_unfiltered_cleaned_split.json /tmp - # copy gsm8k dataset - cp ~/.cache/modelscope/hub/datasets/tmp/test.jsonl /tmp + full-perf-8-npu-a3: + name: full-perf-8-npu-a3 + if: ${{ (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') }} + needs: [set-image-config] + uses: ./.github/workflows/_npu-pr-test-stage.yml + with: + self_name: full-perf-8-npu-a3 + runner_config: linux-aarch64-a3-800t-8 + image: ${{ needs.set-image-config.outputs.image_a3 }} + run_timeout_minutes: '240' + timeout_per_file: '3600' + ref: ${{ needs.set-image-config.outputs.ref }} + is_nightly_pipeline_job: true + skip_pr_test_health_check: 'true' + secrets: inherit - - name: Print Log Information - run: | - bash scripts/ci/npu/npu_log_print.sh + full-perf-16-npu-a3: + name: full-perf-16-npu-a3 + if: ${{ (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') }} + needs: [set-image-config] + uses: ./.github/workflows/_npu-pr-test-stage.yml + with: + self_name: full-perf-16-npu-a3 + runner_config: linux-aarch64-a3-800t-16 + image: ${{ needs.set-image-config.outputs.image_a3 }} + run_timeout_minutes: '240' + timeout_per_file: '3600' + ref: ${{ needs.set-image-config.outputs.ref }} + is_nightly_pipeline_job: true + skip_pr_test_health_check: 'true' + secrets: inherit - - name: Run test - timeout-minutes: 240 - env: - SGLANG_USE_MODELSCOPE: true - SGLANG_IS_IN_CI: true - HF_ENDPOINT: https://hf-mirror.com - TORCH_EXTENSIONS_DIR: /tmp/torch_extensions - PYTORCH_NPU_ALLOC_CONF: "expandable_segments:True" - STREAMS_PER_DEVICE: 32 - run: | - pip install sglang_router - hf download lmms-lab/MMMU --repo-type dataset - pip install sentence_transformers torchaudio==2.8.0 - pip install protobuf==6.31.1 zss pre-commit wandb>=0.16.0 tenacity==8.3.0 loguru openpyxl latex2sympy2 zstandard transformers-stream-generator tqdm-multiprocess pycocoevalcap - pip install yt-dlp sentencepiece==0.1.99 nltk av ftfy sqlitedict==2.1.0 sacrebleu>=1.5.0 pytablewriter black==24.1.0 isort==5.13.2 peft>=0.2.0 accelerate>=0.29.1 - pip install jsonlines httpx==0.25.0 evaluate>=0.4.0 datasets==2.16.1 numexpr xgrammar==0.2.1 numpy==1.26.4 dotenv - git clone --branch v0.3.3 --depth 1 https://github.com/EvolvingLMMs-Lab/lmms-eval.git - cd ./lmms-eval - nohup pip install . > lmmslog.txt 2>&1 & - sleep 120 - export PYTHONPATH=$PYTHONPATH:$(pwd) - cd ../ - cd test - python3 run_suite.py --hw npu --suite full-16-npu-a3 --nightly --continue-on-error --timeout-per-file 3600 + full-poc-multi-node-mix-tests: + name: multi-node-mix-poc + if: ${{ (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') }} + needs: [set-image-config] + strategy: + fail-fast: false + max-parallel: 1 + matrix: + test_config: + # kimi_k2_6 performance tests + - name: kimi_k2_6_w4a8_16p_in64k_out1k_100ms_aime25 + node_size: 2 + test_case: test/registered/npu/accuracy/kimi_k2_6/test_npu_kimi_k2_6_w4a8_16p_in64k_out1k_100ms_aime25.py + test_type: 'accuracy' + uses: ./.github/workflows/nightly-test-npu-e2e-multi-node.yml + with: + runner: linux-amd64-cpu-4 + test_type: ${{ matrix.test_config.test_type || 'perf' }} + test_config_name: ${{ matrix.test_config.name }} + node_size: ${{ matrix.test_config.node_size }} + test_case: ${{ matrix.test_config.test_case }} + image: ${{ needs.set-image-config.outputs.image_a3 }} + install_sglang_from_source: false + prefill_decode_deployment: 'mix' + transformers_version: '' check-all-jobs: if: github.repository == 'sgl-project/sglang' && always() @@ -341,7 +294,16 @@ jobs: - full-1-npu-a3 - full-2-npu-a3 - full-4-npu-a3 + - full-8-npu-a3 - full-16-npu-a3 + - full-acc-2-npu-a3 + - full-acc-4-npu-a3 + - full-acc-16-npu-a3 + - full-perf-2-npu-a3 + - full-perf-4-npu-a3 + - full-perf-8-npu-a3 + - full-perf-16-npu-a3 + - full-poc-multi-node-mix-tests runs-on: ubuntu-latest container: image: docker.m.daocloud.io/ubuntu:22.04 diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml index 0b24876192c8..2041ea817a8a 100644 --- a/.github/workflows/lint.yml +++ b/.github/workflows/lint.yml @@ -8,6 +8,8 @@ on: jobs: lint: runs-on: ubuntu-latest + env: + MINT_VERSION: 4.2.559 steps: - uses: actions/checkout@v4 with: @@ -33,12 +35,20 @@ jobs: uses: actions/setup-python@v4 with: python-version: "3.12" + cache: pip + cache-dependency-path: .pre-commit-config.yaml - name: Install pre-commit hook run: | python -m pip install pre-commit pre-commit install + - name: Cache pre-commit environments + uses: actions/cache@v4 + with: + path: ~/.cache/pre-commit + key: pre-commit-3|${{ env.pythonLocation }}|${{ hashFiles('.pre-commit-config.yaml') }} + # The clippy-rust-workspace pre-commit hook compiles the rust/ workspace # (debug profile); cache cargo registry + target across runs. - name: Rust cache (rust/ workspace) @@ -66,19 +76,23 @@ jobs: - name: Check cookbook authoring contracts run: node docs/scripts/check_cookbook_configs.mjs + - name: Test PR health action + run: node --test .github/actions/check-pr-test-health/action.test.cjs + + - name: Cache mint + uses: actions/cache@v4 + with: + path: ~/.mint-cli + key: mint-${{ runner.os }}-${{ env.MINT_VERSION }} + # Hard gate: docs internal links/anchors/redirects must resolve. # mint is pinned for reproducibility (its heading-slug rules are # version-sensitive); bump deliberately after re-verifying locally. - name: Check docs links (Mintlify broken-links) run: | - npm install -g mint@4.2.559 + if [ ! -x "$HOME/.mint-cli/node_modules/.bin/mint" ]; then + npm install --prefix "$HOME/.mint-cli" "mint@${MINT_VERSION}" + fi + export PATH="$HOME/.mint-cli/node_modules/.bin:$PATH" cd docs mint broken-links --check-anchors --check-redirects - - - name: Run sgl-kernel clang-format checks - uses: DoozyX/clang-format-lint-action@v0.20 - with: - source: python/sglang/kernels/aot - extensions: h,c,cpp,hpp,cu,cuh,cc - clangFormatVersion: 20 - style: file diff --git a/.github/workflows/list-active-pr-runs.yml b/.github/workflows/list-active-pr-runs.yml index 10deab8374cf..4a289af43537 100644 --- a/.github/workflows/list-active-pr-runs.yml +++ b/.github/workflows/list-active-pr-runs.yml @@ -127,23 +127,23 @@ jobs: # Get unique PR numbers (exclude NO_PR entries) pr_numbers=$(cut -d'|' -f1 < "$pr_data_file" | grep -v '^NO_PR$' | sort -u || true) - # Separate high priority and normal PRs - high_priority_prs=() + # Separate highest-priority and normal PRs + highest_priority_prs=() normal_prs=() for pr_num in $pr_numbers; do labels=$(gh pr view "$pr_num" --repo "$REPO" --json labels \ | jq -r '.labels[].name' 2>/dev/null || true) - if echo "$labels" | grep -Fxq "high priority"; then - high_priority_prs+=($pr_num) + if echo "$labels" | grep -Fxq "highest-priority"; then + highest_priority_prs+=($pr_num) else normal_prs+=($pr_num) fi done - # Combine: high priority first, then normal - sorted_pr_numbers=("${high_priority_prs[@]}" "${normal_prs[@]}") + # Combine: highest-priority first, then normal + sorted_pr_numbers=("${highest_priority_prs[@]}" "${normal_prs[@]}") pr_count=0 total_running=0 @@ -170,8 +170,8 @@ jobs: # Add priority indicator priority_indicator="" - if echo "$pr_labels" | grep -q "high priority"; then - priority_indicator="🔴 [HIGH PRIORITY] " + if echo "$pr_info" | jq -e '[.labels[].name] | index("highest-priority")' >/dev/null; then + priority_indicator="🔴 [HIGHEST PRIORITY] " fi echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" diff --git a/.github/workflows/nightly-amd-mi355x-disagg.yml b/.github/workflows/nightly-amd-mi355x-disagg.yml index bee816beca74..8d5f5f18ed1f 100644 --- a/.github/workflows/nightly-amd-mi355x-disagg.yml +++ b/.github/workflows/nightly-amd-mi355x-disagg.yml @@ -1,4 +1,4 @@ -name: Nightly Test (AMD MI355X 2N 1P1D Disagg) +name: Nightly Test (AMD MI355X Disagg) # Self-hosted runners on the amd-sglang login node salloc two or four MI355X # nodes and run the Docker harness (prefill + decode over MORI, a standalone @@ -10,7 +10,7 @@ on: workflow_dispatch: inputs: image: - description: 'Optional. ROCm SGLang Docker image to benchmark. Leave empty for the recipe default.' + description: 'Optional. ROCm SGLang Docker image to benchmark. Leave empty for the latest MI35x ROCm 10 image.' required: false default: '' configs: @@ -42,29 +42,38 @@ jobs: - name: Checkout code uses: actions/checkout@v4 - - name: Resolve latest MI35x nightly image + - name: Resolve latest MI35x ROCm 10 nightly image id: image # Track the newest published image so the nightly actually catches # regressions instead of re-testing one pinned build. We resolve on the # GitHub-hosted runner (clean Docker Hub access, no rate limits) and pass # the tag to the self-hosted runner. Scope is deliberately tight: only - # the clean main MI35x ROCm 7.2 line - # `lmsysorg/sglang-rocm:v*-rocm720-mi35x-YYYYMMDD` (excludes -test-, - # miles-, mori-, vllm-, rocm7_14 variants). On any failure we emit an - # empty string and the launcher falls back to the recipe's pinned - # `runtime.image`. + # the clean main MI35x ROCm 10 line + # `lmsysorg/sglang-rocm:v*-rocm10-mi35x-YYYYMMDD` (excludes -test-, + # miles-, mori-, vllm- variants). If resolution fails, use the pinned + # ROCm 10 image from the last validated full workflow run. run: | set -o pipefail + FALLBACK_IMAGE="lmsysorg/sglang-rocm:v0.5.20-rocm10-mi35x-20260922" RESOLVED=$(curl -fsSL "https://hub.docker.com/v2/repositories/lmsysorg/sglang-rocm/tags?page_size=100&ordering=last_updated" \ - | python3 -c "import json,re,sys;p=re.compile(r'^v[0-9][0-9A-Za-z.]*-rocm720-mi35x-([0-9]{8})\$');c=sorted((m.group(1),t['name']) for t in json.load(sys.stdin).get('results',[]) for m in [p.match(t.get('name',''))] if m);print(c[-1][1] if c else '')" || true) + | python3 -c "import json,re,sys;p=re.compile(r'^v[0-9][0-9A-Za-z.]*-rocm10-mi35x-([0-9]{8})\$');c=sorted((m.group(1),t['name']) for t in json.load(sys.stdin).get('results',[]) for m in [p.match(t.get('name',''))] if m);print(c[-1][1] if c else '')" || true) if [ -n "$RESOLVED" ]; then echo "Resolved latest MI35x image: lmsysorg/sglang-rocm:$RESOLVED" echo "image=lmsysorg/sglang-rocm:$RESOLVED" >> $GITHUB_OUTPUT else - echo "Could not resolve a latest image; launcher will use recipe default." - echo "image=" >> $GITHUB_OUTPUT + echo "Could not resolve a latest image; using fallback: $FALLBACK_IMAGE" + echo "image=$FALLBACK_IMAGE" >> $GITHUB_OUTPUT fi + - name: Record which image this run uses + env: + DISPATCH_IMAGE: ${{ inputs.image }} + RESOLVED_IMAGE: ${{ steps.image.outputs.image }} + run: | + IMAGE="${DISPATCH_IMAGE:-$RESOLVED_IMAGE}" + echo "### Image" >> "$GITHUB_STEP_SUMMARY" + echo "\`$IMAGE\`" >> "$GITHUB_STEP_SUMMARY" + - name: Generate benchmark matrix id: generate env: @@ -153,11 +162,14 @@ jobs: # mi355x-ci--- # Here we clear THIS runner's leftovers from a crashed previous run, so # we match the RUNNER_NAME prefix (older runs have a different run id). - # Never a blanket `squeue --me`, which would kill a concurrent leg. + # Never a blanket match on all jobs, which would kill a concurrent leg. # %200j: squeue truncates the job name (%j) by default -- widen it so # the grep sees the full name. + # No --me: Spur's squeue does not accept it. The RUNNER_NAME prefix is + # what makes this selective, and that is unique per runner, so the + # filter is equally precise without it. if [ -z "${RUNNER_NAME:-}" ]; then echo "RUNNER_NAME unset; skipping"; exit 0; fi - STALE_JOBS=$(squeue --me --noheader --format="%i %200j" | grep -F "mi355x-ci-${RUNNER_NAME}-" | awk '{print $1}' || true) + STALE_JOBS=$(squeue --noheader --format="%i %200j" | grep -F "mi355x-ci-${RUNNER_NAME}-" | awk '{print $1}' || true) if [ -n "$STALE_JOBS" ]; then echo "Cancelling stale jobs for ${RUNNER_NAME}: $STALE_JOBS" scancel $STALE_JOBS @@ -234,7 +246,7 @@ jobs: # config are unique per matrix leg, so a concurrent leg is never hit. if [ -z "${RUNNER_NAME:-}" ]; then echo "RUNNER_NAME unset; skipping"; exit 0; fi JOB_TAG="mi355x-ci-${RUNNER_NAME}-${GITHUB_RUN_ID}-${MATRIX_CONFIG_NAME}" - ACTIVE_JOBS=$(squeue --me --noheader --format="%i %200j" | grep -F "$JOB_TAG" | awk '{print $1}' || true) + ACTIVE_JOBS=$(squeue --noheader --format="%i %200j" | grep -F "$JOB_TAG" | awk '{print $1}' || true) if [ -n "$ACTIVE_JOBS" ]; then echo "Cancelling jobs for ${JOB_TAG}: $ACTIVE_JOBS" scancel $ACTIVE_JOBS diff --git a/.github/workflows/nightly-test-amd-miles-rocm10.yml b/.github/workflows/nightly-test-amd-miles-rocm10.yml new file mode 100644 index 000000000000..e13b810d43bf --- /dev/null +++ b/.github/workflows/nightly-test-amd-miles-rocm10.yml @@ -0,0 +1,93 @@ +name: Nightly Test Miles (AMD ROCm 10) + +on: + schedule: + # Runs after the 12:00 UTC miles image build has pushed the dated image. + - cron: "30 17 * * *" + workflow_dispatch: + inputs: + image_tag: + description: "Override miles image tag (e.g. miles-rocm10-mi35x-20260901); empty = newest by date" + required: false + type: string + default: "" + continue_on_error: + description: "Continue on error (do not fail the workflow on test failures)" + required: false + type: boolean + default: true + +concurrency: + group: ${{ github.workflow }}-${{ github.event_name == 'workflow_dispatch' && format('manual-{0}', github.run_id) || github.ref }} + cancel-in-progress: false + +jobs: + test-miles-mi35x: + if: github.repository == 'sgl-project/sglang' + strategy: + fail-fast: false + matrix: + include: + - suite: nightly-stage-c-2-gpu-mi350 + runner: linux-mi35x-gpu-2 + # Use 8-GPU MI35X pool for 4-GPU suite. + - suite: nightly-stage-c-4-gpu-mi350 + runner: linux-mi35x-gpu-8 + - suite: nightly-stage-c-8-gpu-mi350 + runner: linux-mi35x-gpu-8 + runs-on: ${{ matrix.runner }} + steps: + - name: Checkout code + uses: actions/checkout@v4 + + - name: Login to Docker Hub (AMD) + uses: docker/login-action@v2 + with: + username: ${{ secrets.DOCKERHUB_AMD_USERNAME }} + password: ${{ secrets.DOCKERHUB_AMD_TOKEN }} + + - name: Resolve newest miles ROCm 10 image + run: | + repo="rocm/sgl-dev" + base="miles-rocm10-mi35x" + tag="${{ inputs.image_tag }}" + if [ -z "${tag}" ]; then + # Walk back from today; the build job pushes a dated tag at 12:00 UTC. + # Probe with `manifest inspect` so we don't pull a multi-GB image just + # to test existence; the real pull happens in amd_ci_start_container.sh. + for i in $(seq 0 6); do + d=$(date -u -d "-${i} day" +%Y%m%d) + candidate="${base}-${d}" + if docker manifest inspect "${repo}:${candidate}" >/dev/null 2>&1; then + tag="${candidate}" + break + fi + done + fi + if [ -z "${tag}" ]; then + echo "::error::No ${repo}:${base}- image found on Docker Hub for the last 7 days" + exit 1 + fi + echo "MILES_IMAGE=${repo}:${tag}" >> "$GITHUB_ENV" + echo "Using miles image: ${repo}:${tag}" + + - name: Ensure VRAM is clear + run: bash scripts/ci/amd/ensure_vram_clear.sh rocm + + - name: Start container from the miles image + run: bash scripts/ci/amd/amd_ci_start_container.sh --custom-image "${MILES_IMAGE}" + env: + GITHUB_WORKSPACE: ${{ github.workspace }} + + - name: Run miles CI suite ${{ matrix.suite }} + timeout-minutes: 180 + run: | + touch "${GITHUB_WORKSPACE}/github_summary.md" + TEST_EXIT_CODE=0 + bash scripts/ci/amd/amd_ci_exec.sh -w /root/miles \ + -e PYTHONPATH=/root/miles:/opt/tilelang \ + -e GITHUB_STEP_SUMMARY=/sglang-checkout/github_summary.md \ + python3 tests/ci/run_suite.py --hw rocm --suite ${{ matrix.suite }} --cadence nightly \ + ${{ (github.event_name != 'workflow_dispatch' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? + cat "${GITHUB_WORKSPACE}/github_summary.md" >> "$GITHUB_STEP_SUMMARY" || true + exit ${TEST_EXIT_CODE} diff --git a/.github/workflows/nightly-test-amd-miles-rocm720.yml b/.github/workflows/nightly-test-amd-miles-rocm720.yml deleted file mode 100644 index 813fae11a2e1..000000000000 --- a/.github/workflows/nightly-test-amd-miles-rocm720.yml +++ /dev/null @@ -1,93 +0,0 @@ -name: Nightly Test Miles (AMD ROCm 7.2) - -on: - schedule: - # Runs after the 12:00 UTC miles image build has pushed the dated image. - - cron: "30 17 * * *" - workflow_dispatch: - inputs: - image_tag: - description: "Override miles image tag (e.g. miles-rocm720-mi35x-20260621); empty = newest by date" - required: false - type: string - default: "" - continue_on_error: - description: "Continue on error (do not fail the workflow on test failures)" - required: false - type: boolean - default: true - -concurrency: - group: ${{ github.workflow }}-${{ github.event_name == 'workflow_dispatch' && format('manual-{0}', github.run_id) || github.ref }} - cancel-in-progress: false - -jobs: - test-miles-mi35x: - if: github.repository == 'sgl-project/sglang' - strategy: - fail-fast: false - matrix: - include: - - suite: nightly-stage-c-2-gpu-mi350 - runner: linux-mi35x-gpu-2 - # Use 8-GPU MI35X pool for 4-GPU suite. - - suite: nightly-stage-c-4-gpu-mi350 - runner: linux-mi35x-gpu-8 - - suite: nightly-stage-c-8-gpu-mi350 - runner: linux-mi35x-gpu-8 - runs-on: ${{ matrix.runner }} - steps: - - name: Checkout code - uses: actions/checkout@v4 - - - name: Login to Docker Hub (AMD) - uses: docker/login-action@v2 - with: - username: ${{ secrets.DOCKERHUB_AMD_USERNAME }} - password: ${{ secrets.DOCKERHUB_AMD_TOKEN }} - - - name: Resolve newest miles ROCm 7.2 image - run: | - repo="rocm/sgl-dev" - base="miles-rocm720-mi35x" - tag="${{ inputs.image_tag }}" - if [ -z "${tag}" ]; then - # Walk back from today; the build job pushes a dated tag at 12:00 UTC. - # Probe with `manifest inspect` so we don't pull a multi-GB image just - # to test existence; the real pull happens in amd_ci_start_container.sh. - for i in $(seq 0 6); do - d=$(date -u -d "-${i} day" +%Y%m%d) - candidate="${base}-${d}" - if docker manifest inspect "${repo}:${candidate}" >/dev/null 2>&1; then - tag="${candidate}" - break - fi - done - fi - if [ -z "${tag}" ]; then - echo "::error::No ${repo}:${base}- image found on Docker Hub for the last 7 days" - exit 1 - fi - echo "MILES_IMAGE=${repo}:${tag}" >> "$GITHUB_ENV" - echo "Using miles image: ${repo}:${tag}" - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Start container from the miles image - run: bash scripts/ci/amd/amd_ci_start_container.sh --custom-image "${MILES_IMAGE}" - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - - - name: Run miles CI suite ${{ matrix.suite }} - timeout-minutes: 180 - run: | - touch "${GITHUB_WORKSPACE}/github_summary.md" - TEST_EXIT_CODE=0 - bash scripts/ci/amd/amd_ci_exec.sh -w /root/miles \ - -e PYTHONPATH=/root/miles:/opt/tilelang \ - -e GITHUB_STEP_SUMMARY=/sglang-checkout/github_summary.md \ - python3 tests/ci/run_suite.py --hw rocm --suite ${{ matrix.suite }} --cadence nightly \ - ${{ (github.event_name != 'workflow_dispatch' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - cat "${GITHUB_WORKSPACE}/github_summary.md" >> "$GITHUB_STEP_SUMMARY" || true - exit ${TEST_EXIT_CODE} diff --git a/.github/workflows/nightly-test-amd-rocm720.yml b/.github/workflows/nightly-test-amd-rocm720.yml deleted file mode 100644 index e7081ebc64e3..000000000000 --- a/.github/workflows/nightly-test-amd-rocm720.yml +++ /dev/null @@ -1,2355 +0,0 @@ -name: Nightly Test (AMD ROCm 7.2) - -on: - schedule: - - cron: '30 17 * * *' - push: - branches: - - main - paths: - - "python/sglang/version.py" - workflow_dispatch: - inputs: - rocm_version: - description: 'ROCm image version ("all" runs the scheduled ROCm 7.2 matrix; select rocm10 explicitly)' - required: false - type: choice - default: 'all' - options: - - 'all' - - rocm10 - - rocm724 - - rocm720 - aiter_ref: - description: 'Override AITER commit (optional, leave empty to use Dockerfile default)' - required: false - type: string - default: '' - continue_on_error: - description: 'Continue on error (do not fail the workflow on test failures)' - required: false - type: boolean - default: true - job_select: - description: 'Select a job to run from dropdown (choose "all" to run all jobs)' - required: false - type: choice - default: 'all' - options: - - 'all' - # 1-GPU Unit Tests (MI30x + MI35x) - - nightly-test-1-gpu-unit-rocm720 - - nightly-test-1-gpu-kernel-rocm720 - - nightly-test-1-gpu-mi35x-rocm720 - # 2-GPU and 4-GPU Tests (MI30x + MI35x) - - nightly-accuracy-2-gpu-rocm720 - - nightly-accuracy-2-gpu-vlm-rocm720 - - nightly-perf-2-gpu-text-rocm720 - - nightly-perf-2-gpu-vlm-rocm720 - - nightly-4-gpu-rocm720 - # 2-GPU GLM-5.1-MXFP4 (MI35x only) - - nightly-2-gpu-mi35x-glm51-mxfp4-rocm720 - # 2-GPU DeepSeek-R1-MXFP4 TP2 (MI35x only) - - nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2-rocm720 - # 8-GPU GPT-OSS (MI35x mixes Qwen3-Coder-Next) - - nightly-accuracy-8-gpu-rocm720 - - nightly-accuracy-8-gpu-mi35x-rocm720 - # 8-GPU Grok1-INT4 (MI30x + MI35x) - - nightly-8-gpu-grok1-int4-rocm720 - - nightly-8-gpu-mi35x-grok1-int4-rocm720 - # 8-GPU Grok2 (MI30x + MI35x) - - nightly-8-gpu-grok2-rocm720 - - nightly-8-gpu-mi35x-grok2-rocm720 - # 8-GPU DeepSeek-V3.2 (MI30x) - - nightly-8-gpu-deepseek-v32-rocm720 - - nightly-8-gpu-deepseek-v32-mtp-rocm720 - # 8-GPU DeepSeek-V3.2 (MI35x) - - nightly-8-gpu-mi35x-deepseek-v32-rocm720 - - nightly-8-gpu-mi35x-deepseek-v32-mtp-rocm720 - # 8-GPU DeepSeek-R1 (MI35x only) - - nightly-8-gpu-mi35x-deepseek-r1-mxfp4-rocm720 - - nightly-8-gpu-mi35x-deepseek-r1-mxfp4-kv-fp8-rocm720 - - nightly-8-gpu-mi35x-deepseek-r1-mxfp4-ar-fusion-rocm720 - - nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4-rocm720 - - nightly-8-gpu-mi35x-deepseek-r1-hicache-rocm720 - # 8-GPU DeepSeek-V4-Flash (MI30x) - - nightly-8-gpu-deepseek-v4-flash-rocm720 - # 8-GPU DeepSeek-V4 (MI35x) - - nightly-8-gpu-mi35x-deepseek-v4-flash-rocm720 - - nightly-8-gpu-mi35x-deepseek-v4-pro-rocm720 - - nightly-8-gpu-mi35x-deepseek-v4-pro-mtp-rocm720 - - nightly-8-gpu-mi35x-deepseek-v4-pro-dspark-rocm720 - # 8-GPU Kimi-K2.6 (MI30x) - - nightly-8-gpu-kimi-k26-rocm720 - # 8-GPU Kimi-K3 (MI35x only - native MXFP4 needs gfx95x) - - nightly-8-gpu-mi35x-kimi-k3-rocm720 - # 8-GPU Qwen3.8-MXFP4 (MI35x only - MXFP4 needs gfx95x) - - nightly-8-gpu-mi35x-qwen38-mxfp4-rocm720 - # 8-GPU Qwen 3.5 (MI30x + MI35x) - - nightly-8-gpu-qwen35-rocm720 - - nightly-8-gpu-mi35x-qwen35-rocm720 - - nightly-8-gpu-mi35x-qwen35-triton-dcp-rocm720 - # 8-GPU GLM-5.1 (MI30x) - - nightly-8-gpu-glm51-rocm720 - # 8-GPU GLM-5.2-FP8 (MI35x accuracy + performance) - - nightly-8-gpu-mi35x-glm52-fp8-rocm720 - # 8-GPU GLM-5-MXFP4 (MI35x only) - - nightly-8-gpu-mi35x-glm5-mxfp4-rocm720 - # 4-GPU MiniMax-M2.5 (MI35x) - - nightly-4-gpu-mi35x-minimax-m25-rocm720 - # 4-GPU MiniMax-M3 MXFP8 (MI35x accuracy + performance) - - nightly-4-gpu-mi35x-minimax-m3-rocm720 - # 8-GPU MiniMax-M2.7 (MI30x only) - - nightly-8-gpu-minimax-m27-rocm720 - # Diffusion (MI30x) - - nightly-1-gpu-zimage-turbo-rocm720 - job_filter: - description: 'Or type comma-separated job names (overrides dropdown if non-empty)' - required: false - type: string - default: '' - workflow_call: - inputs: - rocm_version: - description: 'ROCm image version ("all" runs the scheduled ROCm 7.2 matrix; pass rocm10 explicitly)' - required: false - type: string - # A single flavor, unlike the schedule and the dispatch form: a caller - # that says nothing should not silently get double the GPU cost. - default: rocm724 - ref: - description: 'Git ref (branch, tag, or SHA) to test. If not provided, uses the default branch.' - required: false - type: string - default: '' - aiter_ref: - description: 'Override AITER commit (optional, leave empty to use Dockerfile default)' - required: false - type: string - default: '' - job_filter: - description: 'Select which job to run (leave empty or "all" to run all jobs)' - required: false - type: string - default: 'all' - continue_on_error: - description: 'Continue on error (do not fail the workflow on test failures)' - required: false - type: boolean - default: true - -env: - AITER_COMMIT_OVERRIDE: ${{ inputs.aiter_ref }} - DOCKERHUB_AMD_USERNAME: ${{ secrets.DOCKERHUB_AMD_USERNAME }} - DOCKERHUB_AMD_TOKEN: ${{ secrets.DOCKERHUB_AMD_TOKEN }} - -concurrency: - # When called via workflow_call with ref set, use a unique group per caller run to avoid - # collisions with direct schedule/push triggers. We use inputs.ref (not github.event_name) - # to detect this, because github.event_name inherits from the caller in workflow_call. - # Manual dispatch runs also get unique groups so they never cancel each other. - group: nightly-test-amd-rocm720-${{ github.event_name == 'workflow_dispatch' && format('manual-{0}', github.run_id) || inputs.ref && format('caller-{0}', github.run_id) || github.ref }} - cancel-in-progress: ${{ !inputs.ref && github.event_name != 'workflow_call' && github.event_name != 'workflow_dispatch' }} - -jobs: - # ============================================== MI30x ROCm 7.2 Unit Tests ============================================== - # 1-GPU Unit Tests - LoRA, debug utils, scheduler, etc. (MI30x ROCm 7.2) - - # ============================================================================== - # 1-GPU Unit Tests (MI30x + MI35x) - # ============================================================================== - - nightly-test-1-gpu-unit-rocm720: - name: ${{ format('nightly-test-1-gpu-unit ({0}, linux-mi300-1gpu-sglang)', matrix.rocm_version) }} - strategy: - fail-fast: false - matrix: - rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm724", "rocm720"]') }} - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-test-1-gpu-unit-rocm720,')) - runs-on: linux-mi300-1gpu-sglang - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Setup docker (${{ matrix.rocm_version }}) - run: | - touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - - - name: Install dependencies - run: bash scripts/ci/amd/amd_ci_install_dependency.sh - - name: Nightly Unit Test ROCm 7.2 (1-GPU) - timeout-minutes: 90 - run: | - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-1-gpu --nightly --timeout-per-file 900 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - nightly-test-1-gpu-kernel-rocm720: - name: ${{ format('nightly-test-1-gpu-kernel ({0}, linux-mi300-1gpu-sglang)', matrix.rocm_version) }} - strategy: - fail-fast: false - matrix: - rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm724", "rocm720"]') }} - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-test-1-gpu-kernel-rocm720,')) - runs-on: linux-mi300-1gpu-sglang - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Setup docker (${{ matrix.rocm_version }}) - run: | - touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - - - name: Install dependencies - run: bash scripts/ci/amd/amd_ci_install_dependency.sh - - name: Nightly Kernel Benchmark ROCm 7.2 (1-GPU) - timeout-minutes: 60 - run: | - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-kernel-1-gpu --nightly --timeout-per-file 900 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - nightly-test-1-gpu-mi35x-rocm720: - name: ${{ format('nightly-test-1-gpu-mi35x ({0}, linux-mi35x-gpu-1)', matrix.rocm_version) }} - strategy: - fail-fast: false - matrix: - rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm724", "rocm720"]') }} - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-test-1-gpu-mi35x-rocm720,')) - runs-on: linux-mi35x-gpu-1 - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Setup docker (${{ matrix.rocm_version }}) - run: | - touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - - - name: Install dependencies - run: | - bash scripts/ci/amd/amd_ci_install_dependency.sh - - name: Nightly Test MI35x ROCm 7.2 (1-GPU) - timeout-minutes: 90 - run: | - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-1-gpu-mi35x --nightly --timeout-per-file 900 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - # ============================================================================== - # 2-GPU and 4-GPU Tests (MI30x + MI35x) - # ============================================================================== - - nightly-accuracy-2-gpu-rocm720: - name: ${{ format('nightly-accuracy-2-gpu ({0}, linux-mi300-2gpu-sglang)', matrix.rocm_version) }} - strategy: - fail-fast: false - matrix: - rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm724", "rocm720"]') }} - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-accuracy-2-gpu-rocm720,')) - runs-on: linux-mi300-2gpu-sglang - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Setup docker (${{ matrix.rocm_version }}) - run: | - touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - - - name: Install dependencies - run: bash scripts/ci/amd/amd_ci_install_dependency.sh - - name: Nightly Test ROCm 7.2 (2-GPU) - run: | - > github_summary.md # Clear summary file - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - nightly-2-gpu-mi35x-glm51-mxfp4-rocm720: - name: ${{ format('nightly-2-gpu-mi35x-glm51-mxfp4 ({0}, linux-mi35x-gpu-2)', matrix.rocm_version) }} - strategy: - fail-fast: false - matrix: - rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm724", "rocm720"]') }} - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-2-gpu-mi35x-glm51-mxfp4-rocm720,')) - runs-on: linux-mi35x-gpu-2 - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Setup docker (${{ matrix.rocm_version }}) - run: | - touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - - - name: Install dependencies - run: | - bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps - bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate - - - name: Accuracy Test MI35x ROCm 7.2 (2-GPU GLM-5.1-MXFP4 GSM8K) - timeout-minutes: 120 - run: | - > github_summary.md - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-2-gpu-mi35x-glm51-mxfp4 --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2-rocm720: - name: ${{ format('nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2 ({0}, linux-mi35x-gpu-2)', matrix.rocm_version) }} - strategy: - fail-fast: false - matrix: - rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm724", "rocm720"]') }} - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2-rocm720,')) - runs-on: linux-mi35x-gpu-2 - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Setup docker (${{ matrix.rocm_version }}) - run: | - touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - ENABLE_CACHE_HOST: "1" - - - name: Install dependencies - run: | - bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps - bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate - - - name: Accuracy Test MI35x ROCm 7.2 (2-GPU DeepSeek-R1-MXFP4 TP2) - timeout-minutes: 180 - run: | - > github_summary.md - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-2-gpu-mi35x-deepseek-r1-mxfp4-tp2 --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - nightly-accuracy-2-gpu-vlm-rocm720: - name: ${{ format('nightly-accuracy-2-gpu-vlm ({0}, linux-mi300-2gpu-sglang)', matrix.rocm_version) }} - strategy: - fail-fast: false - matrix: - rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm724", "rocm720"]') }} - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-accuracy-2-gpu-vlm-rocm720,')) - runs-on: linux-mi300-2gpu-sglang - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Setup docker (${{ matrix.rocm_version }}) - run: | - touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - - - name: Install dependencies - run: bash scripts/ci/amd/amd_ci_install_dependency.sh - - name: Nightly Accuracy Test ROCm 7.2 (2-GPU VLM MMMU) - timeout-minutes: 180 - run: | - > github_summary.md # Clear summary file - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-accuracy-2-gpu-vlm --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - nightly-perf-2-gpu-text-rocm720: - name: ${{ format('nightly-perf-2-gpu-text ({0}, linux-mi300-2gpu-sglang)', matrix.rocm_version) }} - strategy: - fail-fast: false - matrix: - rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm724", "rocm720"]') }} - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-perf-2-gpu-text-rocm720,')) - runs-on: linux-mi300-2gpu-sglang - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Setup docker (${{ matrix.rocm_version }}) - run: | - touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - - - name: Install dependencies - run: bash scripts/ci/amd/amd_ci_install_dependency.sh - - name: Performance Test (2-GPU Text Models) - timeout-minutes: 120 - run: | - > github_summary.md # Clear summary file - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e SGLANG_USE_AITER=1 \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-perf-text-2-gpu --nightly --timeout-per-file 3600 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - nightly-perf-2-gpu-vlm-rocm720: - name: ${{ format('nightly-perf-2-gpu-vlm ({0}, linux-mi300-2gpu-sglang)', matrix.rocm_version) }} - strategy: - fail-fast: false - matrix: - rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm724", "rocm720"]') }} - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-perf-2-gpu-vlm-rocm720,')) - runs-on: linux-mi300-2gpu-sglang - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Setup docker (${{ matrix.rocm_version }}) - run: | - touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - - - name: Install dependencies - run: bash scripts/ci/amd/amd_ci_install_dependency.sh - - name: Performance Test ROCm 7.2 (2-GPU VLM Models) - timeout-minutes: 180 - run: | - > github_summary.md # Clear summary file - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e SGLANG_USE_AITER=1 \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-perf-vlm-2-gpu --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - nightly-4-gpu-rocm720: - name: ${{ format('nightly-4-gpu ({0}, linux-mi300-4gpu-sglang)', matrix.rocm_version) }} - strategy: - fail-fast: false - matrix: - rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm724", "rocm720"]') }} - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-4-gpu-rocm720,')) - runs-on: linux-mi300-4gpu-sglang - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Setup docker (${{ matrix.rocm_version }}) - run: | - touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - - - name: Install dependencies - run: bash scripts/ci/amd/amd_ci_install_dependency.sh - - - name: Nightly Test ROCm 7.2 (4-GPU) - timeout-minutes: 120 - run: | - > github_summary.md - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-4-gpu --nightly --timeout-per-file 3600 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - # ============================================================================== - # 8-GPU GPT-OSS (MI35x mixes Qwen3-Coder-Next) - # ============================================================================== - - nightly-accuracy-8-gpu-rocm720: - name: ${{ format('nightly-accuracy-8-gpu ({0}, linux-mi300-8gpu-sglang)', matrix.rocm_version) }} - strategy: - fail-fast: false - matrix: - rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm724", "rocm720"]') }} - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-accuracy-8-gpu-rocm720,')) - runs-on: linux-mi300-8gpu-sglang - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Setup docker (${{ matrix.rocm_version }}) - run: | - touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - - - name: Install dependencies - run: bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps - - - name: Accuracy Test ROCm 7.2 (8-GPU GPT-OSS) - timeout-minutes: 180 - run: | - > github_summary.md # Clear summary file - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-accuracy-8-gpu-gpt-oss --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - # Shares this job with the accuracy step above: it already serves the same - # two models, and an MI30x job spends ~49 min pulling the image and - # installing dependencies before it runs anything. - - name: Performance Test ROCm 7.2 (8-GPU GPT-OSS) - if: ${{ !cancelled() }} - timeout-minutes: 120 - run: | - > github_summary.md # Clear summary file - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-perf-8-gpu-gpt-oss --nightly --timeout-per-file 5400 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - nightly-accuracy-8-gpu-mi35x-rocm720: - name: ${{ format('nightly-accuracy-8-gpu-mi35x ({0}, linux-mi35x-gpu-8)', matrix.rocm_version) }} - strategy: - fail-fast: false - matrix: - rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm724", "rocm720"]') }} - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-accuracy-8-gpu-mi35x-rocm720,')) - runs-on: linux-mi35x-gpu-8 - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Setup docker (${{ matrix.rocm_version }}) - run: | - touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - - - name: Install dependencies - run: | - bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps - # Install tabulate for run_suite.py (missing in MI35x container) - bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate - - - name: Accuracy Test MI35x ROCm 7.2 (8-GPU GPT-OSS) - timeout-minutes: 180 - run: | - > github_summary.md # Clear summary file - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-8-gpu-mi35x --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - # Shares this job with the accuracy step above rather than taking its own, - # so the container setup and the GPT-OSS weight cache are paid for once. - - name: Performance Test MI35x ROCm 7.2 (8-GPU GPT-OSS) - if: ${{ !cancelled() }} - timeout-minutes: 120 - run: | - > github_summary.md # Clear summary file - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-perf-8-gpu-mi35x-gpt-oss --nightly --timeout-per-file 5400 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - # ============================================================================== - # 8-GPU Grok1-INT4 (MI30x + MI35x) - # ============================================================================== - - nightly-8-gpu-grok1-int4-rocm720: - name: ${{ format('nightly-8-gpu-grok1-int4 ({0}, linux-mi300-8gpu-sglang)', matrix.rocm_version) }} - strategy: - fail-fast: false - matrix: - rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm724", "rocm720"]') }} - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-grok1-int4-rocm720,')) - runs-on: linux-mi300-8gpu-sglang - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Setup docker (${{ matrix.rocm_version }}) - run: | - touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - - - name: Install dependencies - run: bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps - - - name: Accuracy Test ROCm 7.2 (8-GPU Grok1-INT4) - timeout-minutes: 60 - run: | - > github_summary.md # Clear summary file - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e RCCL_MSCCL_ENABLE=0 \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-accuracy-8-gpu-grok1-int4 --nightly --timeout-per-file 3600 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - - name: Performance Test ROCm 7.2 (8-GPU Grok1-INT4) - timeout-minutes: 60 - continue-on-error: true - run: | - > github_summary.md # Clear summary file - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e RCCL_MSCCL_ENABLE=0 \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-perf-8-gpu-grok1-int4 --nightly --timeout-per-file 3600 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - nightly-8-gpu-mi35x-grok1-int4-rocm720: - name: ${{ format('nightly-8-gpu-mi35x-grok1-int4 ({0}, linux-mi35x-gpu-8)', matrix.rocm_version) }} - strategy: - fail-fast: false - matrix: - rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm724", "rocm720"]') }} - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-mi35x-grok1-int4-rocm720,')) - runs-on: linux-mi35x-gpu-8 - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Setup docker (${{ matrix.rocm_version }}) - run: | - touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - - - name: Install dependencies - run: | - bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps - # Install tabulate for run_suite.py (missing in MI35x container) - bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate - - - name: Accuracy Test MI35x ROCm 7.2 (8-GPU Grok1-INT4) - timeout-minutes: 60 - run: | - > github_summary.md # Clear summary file - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e RCCL_MSCCL_ENABLE=0 \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-accuracy-8-gpu-mi35x-grok1-int4 --nightly --timeout-per-file 3600 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - - name: Performance Test MI35x ROCm 7.2 (8-GPU Grok1-INT4) - timeout-minutes: 60 - continue-on-error: true - run: | - > github_summary.md # Clear summary file - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e RCCL_MSCCL_ENABLE=0 \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-perf-8-gpu-mi35x-grok1-int4 --nightly --timeout-per-file 3600 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - # ============================================================================== - # 8-GPU Grok2 (MI30x + MI35x) - # ============================================================================== - - nightly-8-gpu-grok2-rocm720: - name: ${{ format('nightly-8-gpu-grok2 ({0}, linux-mi300-8gpu-sglang)', matrix.rocm_version) }} - strategy: - fail-fast: false - matrix: - rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm724", "rocm720"]') }} - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-grok2-rocm720,')) - runs-on: linux-mi300-8gpu-sglang - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Setup docker (${{ matrix.rocm_version }}) - run: | - touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - - - name: Install dependencies - run: bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps - - - name: Accuracy Test ROCm 7.2 (8-GPU Grok2) - timeout-minutes: 60 - run: | - > github_summary.md # Clear summary file - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e RCCL_MSCCL_ENABLE=0 \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-accuracy-8-gpu-grok2 --nightly --timeout-per-file 3600 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - - name: Performance Test ROCm 7.2 (8-GPU Grok2) - timeout-minutes: 60 - continue-on-error: true - run: | - > github_summary.md # Clear summary file - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e RCCL_MSCCL_ENABLE=0 \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-perf-8-gpu-grok2 --nightly --timeout-per-file 3600 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - nightly-8-gpu-mi35x-grok2-rocm720: - name: ${{ format('nightly-8-gpu-mi35x-grok2 ({0}, linux-mi35x-gpu-8)', matrix.rocm_version) }} - strategy: - fail-fast: false - matrix: - rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm724", "rocm720"]') }} - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-mi35x-grok2-rocm720,')) - runs-on: linux-mi35x-gpu-8 - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Setup docker (${{ matrix.rocm_version }}) - run: | - touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - - - name: Install dependencies - run: | - bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps - # Install tabulate for run_suite.py (missing in MI35x container) - bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate - - - name: Accuracy Test MI35x ROCm 7.2 (8-GPU Grok2) - timeout-minutes: 60 - run: | - > github_summary.md # Clear summary file - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e RCCL_MSCCL_ENABLE=0 \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-accuracy-8-gpu-mi35x-grok2 --nightly --timeout-per-file 3600 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - - name: Performance Test MI35x ROCm 7.2 (8-GPU Grok2) - timeout-minutes: 60 - continue-on-error: true - run: | - > github_summary.md # Clear summary file - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e RCCL_MSCCL_ENABLE=0 \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-perf-8-gpu-mi35x-grok2 --nightly --timeout-per-file 3600 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - # ============================================================================== - # 8-GPU DeepSeek-V3.2 (MI30x) - # - # V3-0324 and V3.1 are no longer scheduled here: V3.2 covers the same MI30x - # paths (aiter MLA, EAGLE MTP, multithread weight load) on a current - # checkpoint, and DeepSeek-R1/V4 carry the MI35x side. - # ============================================================================== - - nightly-8-gpu-deepseek-v32-rocm720: - name: ${{ format('nightly-8-gpu-deepseek-v32 ({0}, linux-mi300-8gpu-sglang)', matrix.rocm_version) }} - strategy: - fail-fast: false - matrix: - rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm724", "rocm720"]') }} - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-deepseek-v32-rocm720,')) - runs-on: linux-mi300-8gpu-sglang - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Setup docker (${{ matrix.rocm_version }}) - run: | - touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - ENABLE_CACHE_HOST: "1" - - - name: Install dependencies - run: bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps - - - name: Accuracy Test ROCm 7.2 (8-GPU DeepSeek-V3.2 Basic) - timeout-minutes: 120 - run: | - > github_summary.md # Clear summary file - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-accuracy-8-gpu-deepseek-v32 --nightly --timeout-per-file 3600 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - - name: Performance Test ROCm 7.2 (8-GPU DeepSeek-V3.2 Basic) - timeout-minutes: 150 - continue-on-error: true - run: | - > github_summary.md # Clear summary file - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-perf-8-gpu-deepseek-v32-basic --nightly --timeout-per-file 5400 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - nightly-8-gpu-deepseek-v32-mtp-rocm720: - name: ${{ format('nightly-8-gpu-deepseek-v32-mtp ({0}, linux-mi300-8gpu-sglang)', matrix.rocm_version) }} - strategy: - fail-fast: false - matrix: - rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm724", "rocm720"]') }} - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-deepseek-v32-mtp-rocm720,')) - runs-on: linux-mi300-8gpu-sglang - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Setup docker (${{ matrix.rocm_version }}) - run: | - touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - ENABLE_CACHE_HOST: "1" - - - name: Install dependencies - run: bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps - - - name: Accuracy Test ROCm 7.2 (8-GPU DeepSeek-V3.2 MTP) - timeout-minutes: 120 - run: | - > github_summary.md # Clear summary file - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-accuracy-8-gpu-deepseek-v32-mtp --nightly --timeout-per-file 3600 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - - name: Performance Test ROCm 7.2 (8-GPU DeepSeek-V3.2 MTP) - timeout-minutes: 180 - continue-on-error: true - run: | - > github_summary.md # Clear summary file - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-perf-8-gpu-deepseek-v32-mtp --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - # ============================================================================== - # 8-GPU DeepSeek-V3.2 (MI35x) - # - # Accuracy and performance share one job per config, as the MI30x V3.2 jobs - # above already do: both steps serve the same weights, so splitting them cost - # a second container setup and a second cold weight load for nothing. - # ============================================================================== - - nightly-8-gpu-mi35x-deepseek-v32-rocm720: - name: ${{ format('nightly-8-gpu-mi35x-deepseek-v32 ({0}, linux-mi35x-gpu-8)', matrix.rocm_version) }} - strategy: - fail-fast: false - matrix: - rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm724", "rocm720"]') }} - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-mi35x-deepseek-v32-rocm720,')) - runs-on: linux-mi35x-gpu-8 - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Setup docker (${{ matrix.rocm_version }}) - run: | - touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - ENABLE_CACHE_HOST: "1" - - - name: Install dependencies - run: | - bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps - # Install tabulate for run_suite.py (missing in MI35x container) - bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate - - - name: Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-V3.2) - timeout-minutes: 120 - run: | - > github_summary.md # Clear summary file - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-8-gpu-mi35x-deepseek-v32 --nightly --timeout-per-file 3600 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - - name: Performance Test MI35x (8-GPU DeepSeek-V3.2 Basic) - timeout-minutes: 150 - continue-on-error: true - run: | - > github_summary.md # Clear summary file - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-perf-8-gpu-mi35x-deepseek-v32-basic --nightly --timeout-per-file 5400 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - nightly-8-gpu-mi35x-deepseek-v32-mtp-rocm720: - name: ${{ format('nightly-8-gpu-mi35x-deepseek-v32-mtp ({0}, linux-mi35x-gpu-8)', matrix.rocm_version) }} - strategy: - fail-fast: false - matrix: - rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm724", "rocm720"]') }} - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-mi35x-deepseek-v32-mtp-rocm720,')) - runs-on: linux-mi35x-gpu-8 - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Setup docker (${{ matrix.rocm_version }}) - run: | - touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - ENABLE_CACHE_HOST: "1" - - - name: Install dependencies - run: | - bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps - # Install tabulate for run_suite.py (missing in MI35x container) - bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate - - - name: Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-V3.2 TP+MTP) - timeout-minutes: 120 - run: | - > github_summary.md # Clear summary file - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-accuracy-8-gpu-mi35x-deepseek-v32-mtp --nightly --timeout-per-file 3600 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - - name: Performance Test MI35x ROCm 7.2 (8-GPU DeepSeek-V3.2 MTP) - timeout-minutes: 180 - continue-on-error: true - run: | - > github_summary.md # Clear summary file - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-perf-8-gpu-mi35x-deepseek-v32-mtp --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - # ============================================================================== - # 8-GPU DeepSeek-R1 (MI35x only) - # ============================================================================== - - nightly-8-gpu-mi35x-deepseek-r1-mxfp4-rocm720: - name: ${{ format('nightly-8-gpu-mi35x-deepseek-r1-mxfp4 ({0}, linux-mi35x-gpu-8)', matrix.rocm_version) }} - strategy: - fail-fast: false - matrix: - rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm724", "rocm720"]') }} - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-mi35x-deepseek-r1-mxfp4-rocm720,')) - runs-on: linux-mi35x-gpu-8 - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Setup docker (${{ matrix.rocm_version }}) - run: | - touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - ENABLE_CACHE_HOST: "1" - - - name: Install dependencies - run: | - bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps - # Install tabulate for run_suite.py (missing in MI35x container) - bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate - - - name: Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-R1-MXFP4) - timeout-minutes: 180 - run: | - > github_summary.md # Clear summary file - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-8-gpu-mi35x-deepseek-r1-mxfp4 --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - - name: Performance Test MI35x ROCm 7.2 (8-GPU DeepSeek-R1-MXFP4) - timeout-minutes: 300 - continue-on-error: true - run: | - > github_summary.md # Clear summary file - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 registered/amd/perf/mi35x/test_deepseek_r1_mxfp4_perf_mi35x.py || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4-rocm720: - name: ${{ format('nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4 ({0}, linux-mi35x-gpu-8)', matrix.rocm_version) }} - strategy: - fail-fast: false - matrix: - rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm724", "rocm720"]') }} - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4-rocm720,')) - runs-on: linux-mi35x-gpu-8 - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Setup docker (${{ matrix.rocm_version }}) - run: | - touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - ENABLE_CACHE_HOST: "1" - - - name: Install dependencies - run: | - bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps - bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate - - - name: Accuracy Test MI35x ROCm 7.2 (DeepSeek-R1-MXFP4 TP4) - timeout-minutes: 180 - run: | - > github_summary.md - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 registered/amd/accuracy/mi35x/test_deepseek_r1_mxfp4_tp4_mi35x.py || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - - name: Accuracy Test MI35x ROCm 7.2 (DeepSeek-R1-MXFP4 TP4 MTP) - if: ${{ !cancelled() }} - timeout-minutes: 180 - run: | - > github_summary.md - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 registered/amd/accuracy/mi35x/test_deepseek_r1_mxfp4_tp4_mtp_mi35x.py || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - nightly-8-gpu-mi35x-deepseek-r1-mxfp4-kv-fp8-rocm720: - name: ${{ format('nightly-8-gpu-mi35x-deepseek-r1-mxfp4-kv-fp8 ({0}, linux-mi35x-gpu-8)', matrix.rocm_version) }} - strategy: - fail-fast: false - matrix: - rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm724", "rocm720"]') }} - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-mi35x-deepseek-r1-mxfp4-kv-fp8-rocm720,')) - runs-on: linux-mi35x-gpu-8 - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Setup docker (${{ matrix.rocm_version }}) - run: | - touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - ENABLE_CACHE_HOST: "1" - - - name: Install dependencies - run: | - bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps - # Install tabulate for run_suite.py (missing in MI35x container) - bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate - - - name: Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-R1-MXFP4 KV FP8) - timeout-minutes: 180 - run: | - > github_summary.md # Clear summary file - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-8-gpu-mi35x-deepseek-r1-mxfp4-kv-fp8 --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - - name: Performance Test MI35x ROCm 7.2 (8-GPU DeepSeek-R1-MXFP4 KV FP8) - timeout-minutes: 300 - continue-on-error: true - run: | - > github_summary.md # Clear summary file - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 registered/amd/perf/mi35x/test_deepseek_r1_mxfp4_kv_fp8_perf_mi35x.py || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - nightly-8-gpu-mi35x-deepseek-r1-mxfp4-ar-fusion-rocm720: - name: ${{ format('nightly-8-gpu-mi35x-deepseek-r1-mxfp4-ar-fusion ({0}, linux-mi35x-gpu-8)', matrix.rocm_version) }} - strategy: - fail-fast: false - matrix: - rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm724", "rocm720"]') }} - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-mi35x-deepseek-r1-mxfp4-ar-fusion-rocm720,')) - runs-on: linux-mi35x-gpu-8 - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Setup docker (${{ matrix.rocm_version }}) - run: | - touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - ENABLE_CACHE_HOST: "1" - - - name: Install dependencies - run: | - bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps - # Install tabulate for run_suite.py (missing in MI35x container) - bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate - - - name: Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-R1-MXFP4 AllReduce Fusion) - timeout-minutes: 180 - run: | - > github_summary.md # Clear summary file - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-8-gpu-mi35x-deepseek-r1-mxfp4-ar-fusion --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - - name: Performance Test MI35x ROCm 7.2 (8-GPU DeepSeek-R1-MXFP4 AllReduce Fusion) - timeout-minutes: 300 - continue-on-error: true - run: | - > github_summary.md # Clear summary file - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 registered/amd/perf/mi35x/test_deepseek_r1_mxfp4_ar_fusion_perf_mi35x.py || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - nightly-8-gpu-mi35x-deepseek-r1-hicache-rocm720: - name: ${{ format('nightly-8-gpu-mi35x-deepseek-r1-hicache ({0}, linux-mi35x-gpu-8)', matrix.rocm_version) }} - strategy: - fail-fast: false - matrix: - rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm724", "rocm720"]') }} - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-mi35x-deepseek-r1-hicache-rocm720,')) - runs-on: linux-mi35x-gpu-8 - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Setup docker (${{ matrix.rocm_version }}) - run: | - touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - - - name: Install dependencies - run: | - bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps - bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate - - - name: Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-R1 HiCache) - timeout-minutes: 180 - run: | - > github_summary.md - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-8-gpu-mi35x-deepseek-r1-hicache --nightly --timeout-per-file 3600 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - # ============================================================================== - # 8-GPU DeepSeek-V4-Flash (MI30x) - # - # Accuracy only: the MI35x jobs below carry the DSV4 perf numbers, and gfx942 - # has no DSV4 perf baseline to regress against yet. Runs the cookbook's - # MI300X Flash FP8 low-latency cell, so it also covers MLA + KV-FP8 on gfx942. - # ============================================================================== - - nightly-8-gpu-deepseek-v4-flash-rocm720: - name: ${{ format('nightly-8-gpu-deepseek-v4-flash ({0}, linux-mi300-8gpu-sglang)', matrix.rocm_version) }} - strategy: - fail-fast: false - matrix: - rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm724", "rocm720"]') }} - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-deepseek-v4-flash-rocm720,')) - runs-on: linux-mi300-8gpu-sglang - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Setup docker (${{ matrix.rocm_version }}) - run: | - touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - ENABLE_CACHE_HOST: "1" - - - name: Install dependencies - run: bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps - - - name: Accuracy Test ROCm 7.2 (8-GPU DeepSeek-V4-Flash FP8) - timeout-minutes: 180 - run: | - > github_summary.md # Clear summary file - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-accuracy-8-gpu-deepseek-v4-flash --nightly --timeout-per-file 5400 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - # ============================================================================== - # 8-GPU DeepSeek-V4 (MI35x) - # ============================================================================== - - nightly-8-gpu-mi35x-deepseek-v4-flash-rocm720: - name: ${{ format('nightly-8-gpu-mi35x-deepseek-v4-flash ({0}, linux-mi35x-gpu-8)', matrix.rocm_version) }} - strategy: - fail-fast: false - matrix: - rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm724", "rocm720"]') }} - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-mi35x-deepseek-v4-flash-rocm720,')) - runs-on: linux-mi35x-gpu-8 - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Setup docker (${{ matrix.rocm_version }}) - run: | - touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - ENABLE_CACHE_HOST: "1" - - - name: Install dependencies - run: | - # --skip-test-time-deps: GSM8K + bench_one_batch_server don't need lmms-eval / human-eval. - bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps - bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate - - - name: Accuracy + Performance Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Flash FP8 + FP4, unified_kv_triton) - timeout-minutes: 300 - run: | - > github_summary.md # Clear summary file - echo "## SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton" >> github_summary.md - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton \ - -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-8-gpu-mi35x-deepseek-v4-flash --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} - echo "$(> $GITHUB_STEP_SUMMARY || true - - - name: Accuracy + Performance Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Flash FP8 + FP4, triton) - if: ${{ !cancelled() }} - timeout-minutes: 300 - run: | - > github_summary.md # Clear summary file - echo "## SGLANG_HACK_FLASHMLA_BACKEND=triton" >> github_summary.md - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e SGLANG_HACK_FLASHMLA_BACKEND=triton \ - -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-8-gpu-mi35x-deepseek-v4-flash --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} - echo "$(> $GITHUB_STEP_SUMMARY || true - - nightly-8-gpu-mi35x-deepseek-v4-pro-rocm720: - name: ${{ format('nightly-8-gpu-mi35x-deepseek-v4-pro ({0}, linux-mi35x-gpu-8)', matrix.rocm_version) }} - strategy: - fail-fast: false - matrix: - rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm724", "rocm720"]') }} - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-mi35x-deepseek-v4-pro-rocm720,')) - runs-on: linux-mi35x-gpu-8 - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Setup docker (${{ matrix.rocm_version }}) - run: | - touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - ENABLE_CACHE_HOST: "1" - - - name: Install dependencies - run: | - # --skip-test-time-deps: GSM8K + bench_one_batch_server don't need lmms-eval / human-eval. - bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps - bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate - - - name: Accuracy + Performance Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Pro FP8 + FP4, unified_kv_triton) - timeout-minutes: 480 - run: | - > github_summary.md # Clear summary file - echo "## SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton" >> github_summary.md - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton \ - -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-8-gpu-mi35x-deepseek-v4-pro --nightly --timeout-per-file 14400 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} - echo "$(> $GITHUB_STEP_SUMMARY || true - - - name: Accuracy + Performance Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Pro FP8 + FP4, triton) - if: ${{ !cancelled() }} - timeout-minutes: 480 - run: | - > github_summary.md # Clear summary file - echo "## SGLANG_HACK_FLASHMLA_BACKEND=triton" >> github_summary.md - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e SGLANG_HACK_FLASHMLA_BACKEND=triton \ - -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-8-gpu-mi35x-deepseek-v4-pro --nightly --timeout-per-file 14400 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} - echo "$(> $GITHUB_STEP_SUMMARY || true - - nightly-8-gpu-mi35x-deepseek-v4-pro-mtp-rocm720: - name: ${{ format('nightly-8-gpu-mi35x-deepseek-v4-pro-mtp ({0}, linux-mi35x-gpu-8)', matrix.rocm_version) }} - strategy: - fail-fast: false - matrix: - rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm724", "rocm720"]') }} - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-mi35x-deepseek-v4-pro-mtp-rocm720,')) - runs-on: linux-mi35x-gpu-8 - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Setup docker (${{ matrix.rocm_version }}) - run: | - touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - ENABLE_CACHE_HOST: "1" - - - name: Install dependencies - run: | - # --skip-test-time-deps: GSM8K + send_one don't need lmms-eval / human-eval. - bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps - bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate - - - name: Accuracy + MTP Accept Length Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Pro FP8 MTP, unified_kv_triton) - timeout-minutes: 240 - run: | - > github_summary.md # Clear summary file - echo "## SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton" >> github_summary.md - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton \ - -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-8-gpu-mi35x-deepseek-v4-pro-mtp --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} - echo "$(> $GITHUB_STEP_SUMMARY || true - - - name: Accuracy + MTP Accept Length Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Pro FP8 MTP, triton) - if: ${{ !cancelled() }} - timeout-minutes: 240 - run: | - > github_summary.md # Clear summary file - echo "## SGLANG_HACK_FLASHMLA_BACKEND=triton" >> github_summary.md - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e SGLANG_HACK_FLASHMLA_BACKEND=triton \ - -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-8-gpu-mi35x-deepseek-v4-pro-mtp --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} - echo "$(> $GITHUB_STEP_SUMMARY || true - - nightly-8-gpu-mi35x-deepseek-v4-pro-dspark-rocm720: - name: ${{ format('nightly-8-gpu-mi35x-deepseek-v4-pro-dspark ({0}, linux-mi35x-gpu-8)', matrix.rocm_version) }} - strategy: - fail-fast: false - matrix: - rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm724", "rocm720"]') }} - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-mi35x-deepseek-v4-pro-dspark-rocm720,')) - runs-on: linux-mi35x-gpu-8 - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Setup docker (${{ matrix.rocm_version }}) - run: | - touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - ENABLE_CACHE_HOST: "1" - - - name: Install dependencies - run: | - # --skip-test-time-deps: GSM8K doesn't need lmms-eval / human-eval. - bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps - bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate - - - name: Accuracy + DSpark Accept Length Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Pro-DSpark FP4, unified_kv_triton) - timeout-minutes: 300 - run: | - > github_summary.md # Clear summary file - echo "## SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton" >> github_summary.md - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton \ - -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-8-gpu-mi35x-deepseek-v4-pro-dspark --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} - echo "$(> $GITHUB_STEP_SUMMARY || true - - # ============================================================================== - # 8-GPU Kimi-K2.6 (MI30x) - # - # Kept on MI30x: Kimi-K3 needs gfx95x for its native MXFP4 weights, so it - # cannot take over here the way it does on MI35x. - # ============================================================================== - - nightly-8-gpu-kimi-k26-rocm720: - name: ${{ format('nightly-8-gpu-kimi-k26 ({0}, linux-mi300-8gpu-sglang)', matrix.rocm_version) }} - strategy: - fail-fast: false - matrix: - rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm724", "rocm720"]') }} - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-kimi-k26-rocm720,')) - runs-on: linux-mi300-8gpu-sglang - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Setup docker (${{ matrix.rocm_version }}) - run: | - touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - ENABLE_CACHE_HOST: "1" - - - name: Install dependencies - run: bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps - - - name: Accuracy Test ROCm 7.2 (8-GPU Kimi-K2.6) - timeout-minutes: 120 - run: | - > github_summary.md # Clear summary file - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-accuracy-8-gpu-kimi-k26 --nightly --timeout-per-file 3600 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - # ============================================================================== - # 8-GPU Kimi-K3 (MI35x only - native MXFP4 needs gfx95x) - # ============================================================================== - - nightly-8-gpu-mi35x-kimi-k3-rocm720: - name: ${{ format('nightly-8-gpu-mi35x-kimi-k3 ({0}, linux-mi35x-gpu-8)', matrix.rocm_version) }} - strategy: - fail-fast: false - matrix: - rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm724", "rocm720"]') }} - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-mi35x-kimi-k3-rocm720,')) - runs-on: linux-mi35x-gpu-8 - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Setup docker (${{ matrix.rocm_version }}) - run: | - touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - ENABLE_CACHE_HOST: "1" - - - name: Install dependencies - run: | - bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps - # Install tabulate for run_suite.py (missing in MI35x container) - bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate - - # The 2.8T MXFP4 checkpoint is ~1.56 TB, so weight download and load - # dominate wall time well before the eval itself starts. - - name: Accuracy Test MI35x ROCm 7.2 (8-GPU Kimi-K3) - timeout-minutes: 300 - run: | - > github_summary.md # Clear summary file - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-accuracy-8-gpu-mi35x-kimi-k3 --nightly --timeout-per-file 14400 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - # Shares the job with the eval above so the checkpoint is already cached - # and only one 8-GPU MI35x slot is consumed. Step ordering is also the - # accuracy gate: a failed eval fails the job before this runs, so - # throughput is never measured on a build that got the tokens wrong. - # continue-on-error matches every other combined accuracy + perf job here, - # so a perf hiccup cannot turn the accuracy result red. - - name: Performance Test MI35x ROCm 7.2 (8-GPU Kimi-K3) - timeout-minutes: 300 - continue-on-error: true - run: | - > github_summary.md # Clear summary file - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 registered/amd/perf/mi35x/test_kimi_k3_perf_mi35x.py || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - # ============================================================================== - # 8-GPU Qwen 3.5 (MI30x + MI35x) - # ============================================================================== - - nightly-8-gpu-qwen35-rocm720: - name: ${{ format('nightly-8-gpu-qwen35 ({0}, linux-mi300-8gpu-sglang)', matrix.rocm_version) }} - strategy: - fail-fast: false - matrix: - rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm724", "rocm720"]') }} - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-qwen35-rocm720,')) - runs-on: linux-mi300-8gpu-sglang - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Setup docker (${{ matrix.rocm_version }}) - run: | - touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - ENABLE_CACHE_HOST: "1" - - - name: Install dependencies - run: | - bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-aiter-build --skip-test-time-deps - bash scripts/ci/amd/amd_ci_exec.sh pip install mistral-common "lm-eval[api]" - - - name: Accuracy Test ROCm 7.2 (8-GPU Qwen 3.5) - timeout-minutes: 120 - run: | - > github_summary.md # Clear summary file - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-accuracy-8-gpu-qwen35 --nightly --timeout-per-file 3600 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - - name: Performance Test ROCm 7.2 (8-GPU Qwen 3.5 FP8) - timeout-minutes: 120 - continue-on-error: true - run: | - > github_summary.md # Clear summary file - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e SGLANG_USE_AITER=1 \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-perf-8-gpu-qwen35-fp8 --nightly --timeout-per-file 5400 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - nightly-8-gpu-mi35x-qwen35-rocm720: - name: ${{ format('nightly-8-gpu-mi35x-qwen35 ({0}, linux-mi35x-gpu-8)', matrix.rocm_version) }} - strategy: - fail-fast: false - matrix: - rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm724", "rocm720"]') }} - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-mi35x-qwen35-rocm720,')) - runs-on: linux-mi35x-gpu-8 - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Setup docker (${{ matrix.rocm_version }}) - run: | - touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - - - name: Install dependencies - run: | - bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-aiter-build --skip-test-time-deps - bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate - bash scripts/ci/amd/amd_ci_exec.sh pip install mistral-common "lm-eval[api]" - - - name: Accuracy Test MI35x ROCm 7.2 (8-GPU Qwen 3.5) - timeout-minutes: 120 - run: | - > github_summary.md # Clear summary file - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-accuracy-8-gpu-mi35x-qwen35 --nightly --timeout-per-file 3600 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - - name: Performance Test MI35x ROCm 7.2 (8-GPU Qwen 3.5 FP8) - timeout-minutes: 120 - continue-on-error: true - run: | - > github_summary.md # Clear summary file - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e SGLANG_USE_AITER=1 \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-perf-8-gpu-mi35x-qwen35-fp8 --nightly --timeout-per-file 5400 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - nightly-8-gpu-mi35x-qwen35-triton-dcp-rocm720: - name: ${{ format('nightly-8-gpu-mi35x-qwen35-triton-dcp ({0}, linux-mi35x-gpu-8)', matrix.rocm_version) }} - strategy: - fail-fast: false - matrix: - rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm724", "rocm720"]') }} - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-mi35x-qwen35-triton-dcp-rocm720,')) - runs-on: linux-mi35x-gpu-8 - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Setup docker (${{ matrix.rocm_version }}) - run: | - touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - - - name: Install dependencies - run: | - bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-aiter-build --skip-test-time-deps - bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate - - - name: Accuracy Test MI35x ROCm 7.2 (8-GPU Qwen 3.5 Triton DCP) - timeout-minutes: 120 - run: | - > github_summary.md # Clear summary file - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-accuracy-8-gpu-mi35x-qwen35-triton-dcp --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - # ============================================================================== - # 8-GPU Qwen3.8-MXFP4 (MI35x only - MXFP4 needs gfx95x) - # ============================================================================== - - nightly-8-gpu-mi35x-qwen38-mxfp4-rocm720: - name: ${{ format('nightly-8-gpu-mi35x-qwen38-mxfp4 ({0}, linux-mi35x-gpu-8)', matrix.rocm_version) }} - strategy: - fail-fast: false - matrix: - rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm724", "rocm720"]') }} - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-mi35x-qwen38-mxfp4-rocm720,')) - runs-on: linux-mi35x-gpu-8 - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Setup docker (${{ matrix.rocm_version }}) - run: | - touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - ENABLE_CACHE_HOST: "1" - - - name: Install dependencies - run: | - bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps - # Install tabulate for run_suite.py (missing in MI35x container) - bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate - - # ~1.2 TB of weights against a fleet-shared cache that runs at 100% used. - # Run 32196787596 died 40 minutes into the download on ENOSPC with no - # free-space figure anywhere in its log; record one up front instead. - - name: Check model cache space - run: | - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout \ - bash scripts/ci/amd/check_hf_cache_space.sh \ - amd/Qwen3.8-2.4T-A95B-Quark-MXFP4 1250 - - # One step, not the usual accuracy/perf pair: both phases live in the - # same suite so they launch from one server config. The 2.4T MXFP4 - # checkpoint is ~1.2 TB and is loaded once per phase, which dominates - # wall time well before either the eval or the benchmark starts. - - name: Accuracy Test + Performance Test MI35x ROCm 7.2 (8-GPU Qwen3.8-MXFP4) - timeout-minutes: 420 - run: | - > github_summary.md # Clear summary file - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-8-gpu-mi35x-qwen38-mxfp4 --nightly --timeout-per-file 21600 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - # ============================================================================== - # 8-GPU GLM-5.1 (MI30x + MI35x) - # ============================================================================== - - nightly-8-gpu-glm51-rocm720: - name: ${{ format('nightly-8-gpu-glm51 ({0}, linux-mi300-8gpu-sglang)', matrix.rocm_version) }} - strategy: - fail-fast: false - matrix: - rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm724", "rocm720"]') }} - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-glm51-rocm720,')) - runs-on: linux-mi300-8gpu-sglang - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Setup docker (${{ matrix.rocm_version }}) - run: | - touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - - - name: Install dependencies - run: | - bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps - - - name: Accuracy Test ROCm 7.2 (8-GPU GLM-5.1 DSA) - timeout-minutes: 120 - run: | - > github_summary.md # Clear summary file - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-accuracy-8-gpu-glm51 --nightly --timeout-per-file 3600 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - - name: Performance Test ROCm 7.2 (8-GPU GLM-5.1) - timeout-minutes: 120 - continue-on-error: true - run: | - > github_summary.md # Clear summary file - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e SGLANG_USE_AITER=1 \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-perf-8-gpu-glm51 --nightly --timeout-per-file 5400 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - # ============================================================================== - # 8-GPU GLM-5.2-FP8 (MI35x accuracy + performance) - # ============================================================================== - - nightly-8-gpu-mi35x-glm52-fp8-rocm720: - name: ${{ format('nightly-8-gpu-mi35x-glm52-fp8 ({0}, linux-mi35x-gpu-8)', matrix.rocm_version) }} - strategy: - fail-fast: false - matrix: - rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm724", "rocm720"]') }} - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-mi35x-glm52-fp8-rocm720,')) - runs-on: linux-mi35x-gpu-8 - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Setup docker (${{ matrix.rocm_version }}) - run: | - touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - ENABLE_CACHE_HOST: "1" - - - name: Install dependencies - run: | - bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps - bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate - - # The ~700 GB FP8 checkpoint means weight download and load dominate wall - # time well before the eval itself starts. - - name: Accuracy Test MI35x ROCm 7.2 (8-GPU GLM-5.2-FP8 DSA) - timeout-minutes: 180 - run: | - > github_summary.md # Clear summary file - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-8-gpu-mi35x-glm52-fp8 --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - # Run perf only after correctness passes. Keeping both steps in this job - # reuses the cached ~700 GB checkpoint and one scarce 8-GPU MI35x slot. - - name: Performance Test MI35x ROCm 7.2 (8-GPU GLM-5.2-FP8) - timeout-minutes: 120 - continue-on-error: true - run: | - > github_summary.md # Clear summary file - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-perf-8-gpu-mi35x-glm52-fp8 --nightly --timeout-per-file 5400 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - # ============================================================================== - # 8-GPU GLM-5-MXFP4 (MI35x only) - # ============================================================================== - - nightly-8-gpu-mi35x-glm5-mxfp4-rocm720: - name: ${{ format('nightly-8-gpu-mi35x-glm5-mxfp4 ({0}, linux-mi35x-gpu-8)', matrix.rocm_version) }} - strategy: - fail-fast: false - matrix: - rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm724", "rocm720"]') }} - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-mi35x-glm5-mxfp4-rocm720,')) - runs-on: linux-mi35x-gpu-8 - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Setup docker (${{ matrix.rocm_version }}) - run: | - touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - - - name: Install dependencies - run: | - bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps - bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate - - - name: Accuracy Test MI35x ROCm 7.2 (8-GPU GLM-5-MXFP4) - timeout-minutes: 180 - run: | - > github_summary.md - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e SGLANG_USE_AITER=1 \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-8-gpu-mi35x-glm5-mxfp4 --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - - name: Performance Test MI35x ROCm 7.2 (8-GPU GLM-5-MXFP4) - timeout-minutes: 300 - continue-on-error: true - run: | - > github_summary.md - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e SGLANG_USE_AITER=1 \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 registered/amd/perf/mi35x/test_glm5_mxfp4_perf_mi35x.py || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - # ============================================================================== - # 4-GPU MiniMax-M2.5 (MI35x) - # ============================================================================== - - nightly-4-gpu-mi35x-minimax-m25-rocm720: - name: ${{ format('nightly-4-gpu-mi35x-minimax-m25 ({0}, linux-mi35x-gpu-8)', matrix.rocm_version) }} - strategy: - fail-fast: false - matrix: - rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm724", "rocm720"]') }} - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-4-gpu-mi35x-minimax-m25-rocm720,')) - runs-on: linux-mi35x-gpu-8 - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Setup docker (${{ matrix.rocm_version }}) - run: | - touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - ENABLE_CACHE_HOST: "1" - - - name: Install dependencies - run: bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps - - - name: Accuracy Test MI35x ROCm 7.2 (4-GPU MiniMax-M2.5) - timeout-minutes: 120 - run: | - > github_summary.md - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e SGLANG_USE_AITER=1 \ - -e SGLANG_USE_AITER_UNIFIED_ATTN=1 \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-4-gpu-mi35x-minimax-m25-tp4 --nightly --timeout-per-file 5400 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - # ============================================================================== - # 4-GPU MiniMax-M3 MXFP8 (MI35x accuracy + performance) - # ============================================================================== - - nightly-4-gpu-mi35x-minimax-m3-rocm720: - name: ${{ format('nightly-4-gpu-mi35x-minimax-m3 ({0}, linux-mi35x-gpu-8)', matrix.rocm_version) }} - strategy: - fail-fast: false - matrix: - rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm724", "rocm720"]') }} - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-4-gpu-mi35x-minimax-m3-rocm720,')) - runs-on: linux-mi35x-gpu-8 - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Setup docker (${{ matrix.rocm_version }}) - run: | - touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - ENABLE_CACHE_HOST: "1" - - - name: Install dependencies - run: bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps - - - name: Accuracy Test MI35x ROCm 7.2 (4-GPU MiniMax-M3 MXFP8) - timeout-minutes: 120 - run: | - > github_summary.md - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e SGLANG_USE_AITER=1 \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-4-gpu-mi35x-minimax-m3-tp4 --nightly --timeout-per-file 5400 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - # Run perf only after correctness passes. Keeping both steps in this job - # reuses the cached checkpoint and one scarce MI35x slot. - - name: Performance Test MI35x ROCm 7.2 (4-GPU MiniMax-M3 MXFP8) - timeout-minutes: 120 - continue-on-error: true # Perf test failure doesn't fail the job if accuracy passed - run: | - > github_summary.md - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e SGLANG_USE_AITER=1 \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-perf-4-gpu-mi35x-minimax-m3 --nightly --timeout-per-file 5400 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - # ============================================================================== - # 8-GPU MiniMax-M2.7 (MI30x only) - # ============================================================================== - - nightly-8-gpu-minimax-m27-rocm720: - name: ${{ format('nightly-8-gpu-minimax-m27 ({0}, linux-mi300-8gpu-sglang)', matrix.rocm_version) }} - strategy: - fail-fast: false - matrix: - rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm724", "rocm720"]') }} - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-minimax-m27-rocm720,')) - runs-on: linux-mi300-8gpu-sglang - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Setup docker (${{ matrix.rocm_version }}) - run: | - touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - ENABLE_CACHE_HOST: "1" - - - name: Install dependencies - run: bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps - - - name: Accuracy Test ROCm 7.2 (8-GPU MiniMax-M2.7) - timeout-minutes: 120 - run: | - > github_summary.md # Clear summary file - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e SGLANG_USE_AITER=1 \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-accuracy-8-gpu-minimax-m27 --nightly --timeout-per-file 3600 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - - name: Performance Test ROCm 7.2 (8-GPU MiniMax-M2.7) - timeout-minutes: 120 - continue-on-error: true # Perf test failure doesn't fail the job if accuracy passed - run: | - > github_summary.md # Clear summary file - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e SGLANG_USE_AITER=1 \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-perf-8-gpu-minimax-m27 --nightly --timeout-per-file 5400 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - # ============================================================================== - # Diffusion (MI30x) - # ============================================================================== - - nightly-1-gpu-zimage-turbo-rocm720: - name: ${{ format('nightly-1-gpu-zimage-turbo ({0}, linux-mi300-1gpu-sglang)', matrix.rocm_version) }} - strategy: - fail-fast: false - matrix: - rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm724", "rocm720"]') }} - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-1-gpu-zimage-turbo-rocm720,')) - runs-on: linux-mi300-1gpu-sglang - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Setup docker (${{ matrix.rocm_version }}) - run: | - touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - - - name: Install dependencies - run: bash scripts/ci/amd/amd_ci_install_dependency.sh - - - name: Z-Image-Turbo Diffusion Test ROCm 7.2 (1-GPU) - timeout-minutes: 45 - run: | - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - -e SGLANG_DIFFUSION_ARTIFACT_DIR="/sglang-checkout/diffusion-artifacts" \ - pytest test/registered/amd/test_zimage_turbo.py -v -s ${{ inputs.continue_on_error && '|| true' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - - name: Upload generated images - if: always() - uses: actions/upload-artifact@v4 - with: - name: zimage-turbo-outputs-rocm720 - path: diffusion-artifacts/ - if-no-files-found: ignore - retention-days: 30 - - check-all-jobs: - if: always() && (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request' || github.event_name == 'workflow_dispatch') - needs: - # 1-GPU Unit Tests (MI30x + MI35x) - - nightly-test-1-gpu-unit-rocm720 - - nightly-test-1-gpu-mi35x-rocm720 - # 2-GPU and 4-GPU Tests (MI30x + MI35x) - - nightly-accuracy-2-gpu-rocm720 - - nightly-accuracy-2-gpu-vlm-rocm720 - - nightly-perf-2-gpu-text-rocm720 - - nightly-perf-2-gpu-vlm-rocm720 - - nightly-4-gpu-rocm720 - - nightly-2-gpu-mi35x-glm51-mxfp4-rocm720 - - nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2-rocm720 - # 8-GPU GPT-OSS (MI35x mixes Qwen3-Coder-Next) - - nightly-accuracy-8-gpu-rocm720 - - nightly-accuracy-8-gpu-mi35x-rocm720 - # 8-GPU Grok1-INT4 (MI30x + MI35x) - - nightly-8-gpu-grok1-int4-rocm720 - - nightly-8-gpu-mi35x-grok1-int4-rocm720 - # 8-GPU Grok2 (MI30x + MI35x) - - nightly-8-gpu-grok2-rocm720 - - nightly-8-gpu-mi35x-grok2-rocm720 - # 8-GPU DeepSeek-V3.2 (MI30x) - - nightly-8-gpu-deepseek-v32-rocm720 - - nightly-8-gpu-deepseek-v32-mtp-rocm720 - # 8-GPU DeepSeek-V3.2 (MI35x) - - nightly-8-gpu-mi35x-deepseek-v32-rocm720 - - nightly-8-gpu-mi35x-deepseek-v32-mtp-rocm720 - # 8-GPU DeepSeek-R1 (MI35x only) - - nightly-8-gpu-mi35x-deepseek-r1-mxfp4-rocm720 - - nightly-8-gpu-mi35x-deepseek-r1-mxfp4-kv-fp8-rocm720 - - nightly-8-gpu-mi35x-deepseek-r1-mxfp4-ar-fusion-rocm720 - - nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4-rocm720 - - nightly-8-gpu-mi35x-deepseek-r1-hicache-rocm720 - # 8-GPU DeepSeek-V4-Flash (MI30x) - - nightly-8-gpu-deepseek-v4-flash-rocm720 - # 8-GPU DeepSeek-V4 (MI35x) - - nightly-8-gpu-mi35x-deepseek-v4-flash-rocm720 - - nightly-8-gpu-mi35x-deepseek-v4-pro-rocm720 - - nightly-8-gpu-mi35x-deepseek-v4-pro-mtp-rocm720 - - nightly-8-gpu-mi35x-deepseek-v4-pro-dspark-rocm720 - # 8-GPU Kimi-K2.6 (MI30x) - - nightly-8-gpu-kimi-k26-rocm720 - # 8-GPU Kimi-K3 (MI35x only - native MXFP4 needs gfx95x) - - nightly-8-gpu-mi35x-kimi-k3-rocm720 - # 8-GPU Qwen3.8-MXFP4 (MI35x only - MXFP4 needs gfx95x) - - nightly-8-gpu-mi35x-qwen38-mxfp4-rocm720 - # 8-GPU Qwen 3.5 (MI30x + MI35x) - - nightly-8-gpu-qwen35-rocm720 - - nightly-8-gpu-mi35x-qwen35-rocm720 - - nightly-8-gpu-mi35x-qwen35-triton-dcp-rocm720 - # 8-GPU GLM-5.1 (MI30x) - - nightly-8-gpu-glm51-rocm720 - # 8-GPU GLM-5.2-FP8 (MI35x accuracy + performance) - - nightly-8-gpu-mi35x-glm52-fp8-rocm720 - # 8-GPU GLM-5-MXFP4 (MI35x only) - - nightly-8-gpu-mi35x-glm5-mxfp4-rocm720 - # 4-GPU MiniMax-M2.5 (MI35x) - - nightly-4-gpu-mi35x-minimax-m25-rocm720 - # 4-GPU MiniMax-M3 MXFP8 (MI35x accuracy + performance) - - nightly-4-gpu-mi35x-minimax-m3-rocm720 - # 8-GPU MiniMax-M2.7 (MI30x only) - - nightly-8-gpu-minimax-m27-rocm720 - # Diffusion (MI30x) - - nightly-1-gpu-zimage-turbo-rocm720 - runs-on: ubuntu-latest - steps: - - name: Check if any job failed - run: | - if [[ "${{ contains(needs.*.result, 'failure') }}" == "true" ]]; then - echo "One or more ROCm 7.2 nightly test jobs failed" - exit 1 - fi - if [[ "${{ contains(needs.*.result, 'cancelled') }}" == "true" ]]; then - echo "One or more ROCm 7.2 nightly test jobs were cancelled" - exit 1 - fi - echo "All ROCm 7.2 nightly test jobs passed" diff --git a/.github/workflows/nightly-test-amd.yml b/.github/workflows/nightly-test-amd.yml index 863ebc726b0a..c2a931bec850 100644 --- a/.github/workflows/nightly-test-amd.yml +++ b/.github/workflows/nightly-test-amd.yml @@ -1,4 +1,4 @@ -name: Nightly Test (AMD ROCm 7.0) +name: Nightly Test (AMD) on: schedule: @@ -10,6 +10,16 @@ on: - "python/sglang/version.py" workflow_dispatch: inputs: + rocm_version: + description: 'ROCm image version ("all" runs rocm10, rocm724, and rocm720)' + required: false + type: choice + default: rocm10 + options: + - 'all' + - rocm10 + - rocm724 + - rocm720 aiter_ref: description: 'Override AITER commit (optional, leave empty to use Dockerfile default)' required: false @@ -50,40 +60,44 @@ on: # 8-GPU Grok2 (MI30x + MI35x) - nightly-8-gpu-grok2 - nightly-8-gpu-mi35x-grok2 - # 8-GPU DeepSeek-V3.x (MI30x) - - nightly-8-gpu-deepseek-v31 + # 8-GPU DeepSeek-V3.2 (MI30x) - nightly-8-gpu-deepseek-v32 - nightly-8-gpu-deepseek-v32-mtp - - nightly-8-gpu-deepseek-v3-kv-fp8 # 8-GPU DeepSeek-V3.2 (MI35x) - - nightly-accuracy-8-gpu-mi35x-deepseek-v32 - - nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp - - nightly-perf-8-gpu-mi35x-deepseek-v32-basic - - nightly-perf-8-gpu-mi35x-deepseek-v32-mtp + - nightly-8-gpu-mi35x-deepseek-v32 + - nightly-8-gpu-mi35x-deepseek-v32-mtp # 8-GPU DeepSeek-R1 (MI35x only) - nightly-8-gpu-mi35x-deepseek-r1-mxfp4 - nightly-8-gpu-mi35x-deepseek-r1-mxfp4-kv-fp8 - nightly-8-gpu-mi35x-deepseek-r1-mxfp4-ar-fusion - nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4 - nightly-8-gpu-mi35x-deepseek-r1-hicache - # 8-GPU Kimi-K2.6 (MI30x + MI35x) + # 8-GPU DeepSeek-V4-Flash (MI30x) + - nightly-8-gpu-deepseek-v4-flash + # 8-GPU DeepSeek-V4 (MI35x) + - nightly-8-gpu-mi35x-deepseek-v4-flash + - nightly-8-gpu-mi35x-deepseek-v4-pro + - nightly-8-gpu-mi35x-deepseek-v4-pro-mtp + - nightly-8-gpu-mi35x-deepseek-v4-pro-dspark + # 8-GPU Kimi-K2.6 (MI30x) - nightly-8-gpu-kimi-k26 - - nightly-8-gpu-mi35x-kimi-k26 - # 8-GPU Qwen3-235B (MI30x + MI35x MXFP4) - - nightly-8-gpu-qwen3-235b - - nightly-8-gpu-mi35x-qwen3-235b-mxfp4 + # 8-GPU Kimi-K3 (MI35x only - native MXFP4 needs gfx95x) + - nightly-8-gpu-mi35x-kimi-k3 + # 8-GPU Qwen3.8-MXFP4 (MI35x only - MXFP4 needs gfx95x) + - nightly-8-gpu-mi35x-qwen38-mxfp4 # 8-GPU Qwen 3.5 (MI30x + MI35x) - nightly-8-gpu-qwen35 - nightly-8-gpu-mi35x-qwen35 - nightly-8-gpu-mi35x-qwen35-triton-dcp - # 8-GPU GLM-5.1 (MI30x + MI35x) + # 8-GPU GLM-5.1 (MI30x) - nightly-8-gpu-glm51 - - nightly-8-gpu-mi35x-glm51 + # 8-GPU GLM-5.2-FP8 (MI35x accuracy + performance) + - nightly-8-gpu-mi35x-glm52-fp8 # 8-GPU GLM-5-MXFP4 (MI35x only) - nightly-8-gpu-mi35x-glm5-mxfp4 # 4-GPU MiniMax-M2.5 (MI35x) - nightly-4-gpu-mi35x-minimax-m25 - # 4-GPU MiniMax-M3 MXFP8 (MI35x) + # 4-GPU MiniMax-M3 MXFP8 (MI35x accuracy + performance) - nightly-4-gpu-mi35x-minimax-m3 # 8-GPU MiniMax-M2.7 (MI30x only) - nightly-8-gpu-minimax-m27 @@ -96,6 +110,13 @@ on: default: '' workflow_call: inputs: + rocm_version: + description: 'ROCm image version ("all" runs rocm10, rocm724, and rocm720)' + required: false + type: string + # A caller that says nothing gets the same single-flavor default as a + # manual dispatch; pass "all" explicitly for multi-version GPU coverage. + default: rocm10 ref: description: 'Git ref (branch, tag, or SHA) to test. If not provided, uses the default branch.' required: false @@ -126,20 +147,25 @@ concurrency: # When called via workflow_call with ref set, use a unique group per caller run to avoid # collisions with direct schedule/push triggers. We use inputs.ref (not github.event_name) # to detect this, because github.event_name inherits from the caller in workflow_call. - # Manual dispatch runs also get unique groups so they never cancel each other. - group: nightly-test-amd-${{ github.event_name == 'workflow_dispatch' && format('manual-{0}', github.run_id) || inputs.ref && format('caller-{0}', github.run_id) || github.ref }} - cancel-in-progress: ${{ !inputs.ref && github.event_name != 'workflow_call' && github.event_name != 'workflow_dispatch' }} + # Manual dispatch and scheduled runs also get unique groups so a version bump + # or the next daily run cannot cancel an in-progress three-version nightly. + group: nightly-test-amd-${{ github.event_name == 'workflow_dispatch' && format('manual-{0}', github.run_id) || github.event_name == 'schedule' && format('scheduled-{0}', github.run_id) || inputs.ref && format('caller-{0}', github.run_id) || github.ref }} + cancel-in-progress: ${{ !inputs.ref && github.event_name != 'workflow_call' && github.event_name != 'workflow_dispatch' && github.event_name != 'schedule' }} jobs: - # ============================================== MI30x Unit Tests ============================================== - # 1-GPU Unit Tests - LoRA, debug utils, scheduler, etc. (MI30x only) + # ============================================== MI30x ROCm Unit Tests ============================================== + # 1-GPU Unit Tests - LoRA, debug utils, scheduler, etc. (MI30x ROCm) # ============================================================================== # 1-GPU Unit Tests (MI30x + MI35x) # ============================================================================== nightly-test-1-gpu-unit: - name: nightly-test-1-gpu-unit (rocm700, linux-mi300-1gpu-sglang) + name: ${{ format('nightly-test-1-gpu-unit ({0}, linux-mi300-1gpu-sglang)', matrix.rocm_version) }} + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm10", "rocm724", "rocm720"]') }} if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-test-1-gpu-unit,')) runs-on: linux-mi300-1gpu-sglang steps: @@ -151,17 +177,16 @@ jobs: - name: Ensure VRAM is clear run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - name: Setup docker + - name: Setup docker (${{ matrix.rocm_version }}) run: | touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh + bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} env: GITHUB_WORKSPACE: ${{ github.workspace }} - name: Install dependencies run: bash scripts/ci/amd/amd_ci_install_dependency.sh - - - name: Nightly Unit Test (1-GPU) + - name: Nightly Unit Test ROCm (1-GPU) timeout-minutes: 90 run: | bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ @@ -171,7 +196,11 @@ jobs: exit ${TEST_EXIT_CODE:-0} nightly-test-1-gpu-kernel: - name: nightly-test-1-gpu-kernel (rocm700, linux-mi300-1gpu-sglang) + name: ${{ format('nightly-test-1-gpu-kernel ({0}, linux-mi300-1gpu-sglang)', matrix.rocm_version) }} + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm10", "rocm724", "rocm720"]') }} if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-test-1-gpu-kernel,')) runs-on: linux-mi300-1gpu-sglang steps: @@ -183,17 +212,16 @@ jobs: - name: Ensure VRAM is clear run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - name: Setup docker + - name: Setup docker (${{ matrix.rocm_version }}) run: | touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh + bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} env: GITHUB_WORKSPACE: ${{ github.workspace }} - name: Install dependencies run: bash scripts/ci/amd/amd_ci_install_dependency.sh - - - name: Nightly Kernel Benchmark (1-GPU) + - name: Nightly Kernel Benchmark ROCm (1-GPU) timeout-minutes: 60 run: | bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ @@ -203,7 +231,11 @@ jobs: exit ${TEST_EXIT_CODE:-0} nightly-test-1-gpu-mi35x: - name: nightly-test-1-gpu-mi35x (rocm700, linux-mi35x-gpu-1) + name: ${{ format('nightly-test-1-gpu-mi35x ({0}, linux-mi35x-gpu-1)', matrix.rocm_version) }} + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm10", "rocm724", "rocm720"]') }} if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-test-1-gpu-mi35x,')) runs-on: linux-mi35x-gpu-1 steps: @@ -215,20 +247,17 @@ jobs: - name: Ensure VRAM is clear run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - name: Setup docker + - name: Setup docker (${{ matrix.rocm_version }}) run: | touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh + bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} env: GITHUB_WORKSPACE: ${{ github.workspace }} - name: Install dependencies run: | bash scripts/ci/amd/amd_ci_install_dependency.sh - # Install tabulate for run_suite.py (missing in MI35x container) - bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate - - - name: Nightly Test MI35x (1-GPU) + - name: Nightly Test MI35x ROCm (1-GPU) timeout-minutes: 90 run: | bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ @@ -242,7 +271,11 @@ jobs: # ============================================================================== nightly-accuracy-2-gpu: - name: nightly-accuracy-2-gpu (rocm700, linux-mi300-2gpu-sglang) + name: ${{ format('nightly-accuracy-2-gpu ({0}, linux-mi300-2gpu-sglang)', matrix.rocm_version) }} + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm10", "rocm724", "rocm720"]') }} if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-accuracy-2-gpu,')) runs-on: linux-mi300-2gpu-sglang steps: @@ -254,27 +287,31 @@ jobs: - name: Ensure VRAM is clear run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - name: Setup docker + - name: Setup docker (${{ matrix.rocm_version }}) run: | touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh + bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} env: GITHUB_WORKSPACE: ${{ github.workspace }} - name: Install dependencies run: bash scripts/ci/amd/amd_ci_install_dependency.sh - - - name: Nightly Test (2-GPU) + - name: Nightly Test ROCm (2-GPU) run: | > github_summary.md # Clear summary file bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ + -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ python3 run_suite.py --hw amd --suite nightly-amd --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? echo "$(> $GITHUB_STEP_SUMMARY || true exit ${TEST_EXIT_CODE:-0} nightly-2-gpu-mi35x-glm51-mxfp4: - name: nightly-2-gpu-mi35x-glm51-mxfp4 (rocm700, linux-mi35x-gpu-2) + name: ${{ format('nightly-2-gpu-mi35x-glm51-mxfp4 ({0}, linux-mi35x-gpu-2)', matrix.rocm_version) }} + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm10", "rocm724", "rocm720"]') }} if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-2-gpu-mi35x-glm51-mxfp4,')) runs-on: linux-mi35x-gpu-2 steps: @@ -286,30 +323,35 @@ jobs: - name: Ensure VRAM is clear run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - name: Setup docker + - name: Setup docker (${{ matrix.rocm_version }}) run: | touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh + bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} env: GITHUB_WORKSPACE: ${{ github.workspace }} - name: Install dependencies run: | - bash scripts/ci/amd/amd_ci_install_dependency.sh + bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate - - name: Accuracy Test MI35x (2-GPU GLM-5.1-MXFP4 GSM8K) + - name: Accuracy Test MI35x ROCm (2-GPU GLM-5.1-MXFP4 GSM8K) timeout-minutes: 120 run: | > github_summary.md bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ + -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ python3 run_suite.py --hw amd --suite nightly-amd-2-gpu-mi35x-glm51-mxfp4 --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? echo "$(> $GITHUB_STEP_SUMMARY || true exit ${TEST_EXIT_CODE:-0} nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2: - name: nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2 (rocm700, linux-mi35x-gpu-2) + name: ${{ format('nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2 ({0}, linux-mi35x-gpu-2)', matrix.rocm_version) }} + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm10", "rocm724", "rocm720"]') }} if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2,')) runs-on: linux-mi35x-gpu-2 steps: @@ -321,31 +363,36 @@ jobs: - name: Ensure VRAM is clear run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - name: Setup docker + - name: Setup docker (${{ matrix.rocm_version }}) run: | touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh + bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} env: GITHUB_WORKSPACE: ${{ github.workspace }} ENABLE_CACHE_HOST: "1" - name: Install dependencies run: | - bash scripts/ci/amd/amd_ci_install_dependency.sh + bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate - - name: Accuracy Test MI35x (2-GPU DeepSeek-R1-MXFP4 TP2) + - name: Accuracy Test MI35x ROCm (2-GPU DeepSeek-R1-MXFP4 TP2) timeout-minutes: 180 run: | > github_summary.md bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ + -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ python3 run_suite.py --hw amd --suite nightly-amd-2-gpu-mi35x-deepseek-r1-mxfp4-tp2 --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? echo "$(> $GITHUB_STEP_SUMMARY || true exit ${TEST_EXIT_CODE:-0} nightly-accuracy-2-gpu-vlm: - name: nightly-accuracy-2-gpu-vlm (rocm700, linux-mi300-2gpu-sglang) + name: ${{ format('nightly-accuracy-2-gpu-vlm ({0}, linux-mi300-2gpu-sglang)', matrix.rocm_version) }} + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm10", "rocm724", "rocm720"]') }} if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-accuracy-2-gpu-vlm,')) runs-on: linux-mi300-2gpu-sglang steps: @@ -357,28 +404,32 @@ jobs: - name: Ensure VRAM is clear run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - name: Setup docker + - name: Setup docker (${{ matrix.rocm_version }}) run: | touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh + bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} env: GITHUB_WORKSPACE: ${{ github.workspace }} - name: Install dependencies run: bash scripts/ci/amd/amd_ci_install_dependency.sh - - - name: Nightly Accuracy Test (2-GPU VLM MMMU) + - name: Nightly Accuracy Test ROCm (2-GPU VLM MMMU) timeout-minutes: 180 run: | > github_summary.md # Clear summary file bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ + -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ python3 run_suite.py --hw amd --suite nightly-amd-accuracy-2-gpu-vlm --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? echo "$(> $GITHUB_STEP_SUMMARY || true exit ${TEST_EXIT_CODE:-0} nightly-perf-2-gpu-text: - name: nightly-perf-2-gpu-text (rocm700, linux-mi300-2gpu-sglang) + name: ${{ format('nightly-perf-2-gpu-text ({0}, linux-mi300-2gpu-sglang)', matrix.rocm_version) }} + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm10", "rocm724", "rocm720"]') }} if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-perf-2-gpu-text,')) runs-on: linux-mi300-2gpu-sglang steps: @@ -390,21 +441,21 @@ jobs: - name: Ensure VRAM is clear run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - name: Setup docker + - name: Setup docker (${{ matrix.rocm_version }}) run: | touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh + bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} env: GITHUB_WORKSPACE: ${{ github.workspace }} - name: Install dependencies run: bash scripts/ci/amd/amd_ci_install_dependency.sh - - name: Performance Test (2-GPU Text Models) timeout-minutes: 120 run: | > github_summary.md # Clear summary file bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ + -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ -e SGLANG_USE_AITER=1 \ -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ python3 run_suite.py --hw amd --suite nightly-amd-perf-text-2-gpu --nightly --timeout-per-file 3600 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? @@ -412,7 +463,11 @@ jobs: exit ${TEST_EXIT_CODE:-0} nightly-perf-2-gpu-vlm: - name: nightly-perf-2-gpu-vlm (rocm700, linux-mi300-2gpu-sglang) + name: ${{ format('nightly-perf-2-gpu-vlm ({0}, linux-mi300-2gpu-sglang)', matrix.rocm_version) }} + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm10", "rocm724", "rocm720"]') }} if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-perf-2-gpu-vlm,')) runs-on: linux-mi300-2gpu-sglang steps: @@ -424,21 +479,21 @@ jobs: - name: Ensure VRAM is clear run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - name: Setup docker + - name: Setup docker (${{ matrix.rocm_version }}) run: | touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh + bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} env: GITHUB_WORKSPACE: ${{ github.workspace }} - name: Install dependencies run: bash scripts/ci/amd/amd_ci_install_dependency.sh - - - name: Performance Test (2-GPU VLM Models) + - name: Performance Test ROCm (2-GPU VLM Models) timeout-minutes: 180 run: | > github_summary.md # Clear summary file bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ + -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ -e SGLANG_USE_AITER=1 \ -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ python3 run_suite.py --hw amd --suite nightly-amd-perf-vlm-2-gpu --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? @@ -446,7 +501,11 @@ jobs: exit ${TEST_EXIT_CODE:-0} nightly-4-gpu: - name: nightly-4-gpu (rocm700, linux-mi300-4gpu-sglang) + name: ${{ format('nightly-4-gpu ({0}, linux-mi300-4gpu-sglang)', matrix.rocm_version) }} + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm10", "rocm724", "rocm720"]') }} if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-4-gpu,')) runs-on: linux-mi300-4gpu-sglang steps: @@ -458,17 +517,17 @@ jobs: - name: Ensure VRAM is clear run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - name: Setup docker + - name: Setup docker (${{ matrix.rocm_version }}) run: | touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh + bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} env: GITHUB_WORKSPACE: ${{ github.workspace }} - name: Install dependencies run: bash scripts/ci/amd/amd_ci_install_dependency.sh - - name: Nightly Test (4-GPU) + - name: Nightly Test ROCm (4-GPU) timeout-minutes: 120 run: | > github_summary.md @@ -483,7 +542,11 @@ jobs: # ============================================================================== nightly-accuracy-8-gpu: - name: nightly-accuracy-8-gpu (rocm700, linux-mi300-8gpu-sglang) + name: ${{ format('nightly-accuracy-8-gpu ({0}, linux-mi300-8gpu-sglang)', matrix.rocm_version) }} + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm10", "rocm724", "rocm720"]') }} if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-accuracy-8-gpu,')) runs-on: linux-mi300-8gpu-sglang steps: @@ -495,27 +558,48 @@ jobs: - name: Ensure VRAM is clear run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - name: Setup docker + - name: Setup docker (${{ matrix.rocm_version }}) run: | touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh + bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} env: GITHUB_WORKSPACE: ${{ github.workspace }} - name: Install dependencies - run: bash scripts/ci/amd/amd_ci_install_dependency.sh + run: bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps - - name: Accuracy Test (8-GPU GPT-OSS) + - name: Accuracy Test ROCm (8-GPU GPT-OSS) timeout-minutes: 180 run: | + > github_summary.md # Clear summary file bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ + -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ python3 run_suite.py --hw amd --suite nightly-amd-accuracy-8-gpu-gpt-oss --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? echo "$(> $GITHUB_STEP_SUMMARY || true exit ${TEST_EXIT_CODE:-0} + # Shares this job with the accuracy step above: it already serves the same + # two models, and an MI30x job spends ~49 min pulling the image and + # installing dependencies before it runs anything. + - name: Performance Test ROCm (8-GPU GPT-OSS) + if: ${{ !cancelled() }} + timeout-minutes: 120 + run: | + > github_summary.md # Clear summary file + bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ + -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ + -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ + python3 run_suite.py --hw amd --suite nightly-perf-8-gpu-gpt-oss --nightly --timeout-per-file 5400 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? + echo "$(> $GITHUB_STEP_SUMMARY || true + exit ${TEST_EXIT_CODE:-0} + nightly-accuracy-8-gpu-mi35x: - name: nightly-accuracy-8-gpu-mi35x (rocm700, linux-mi35x-gpu-8) + name: ${{ format('nightly-accuracy-8-gpu-mi35x ({0}, linux-mi35x-gpu-8)', matrix.rocm_version) }} + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm10", "rocm724", "rocm720"]') }} if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-accuracy-8-gpu-mi35x,')) runs-on: linux-mi35x-gpu-8 steps: @@ -527,34 +611,54 @@ jobs: - name: Ensure VRAM is clear run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - name: Setup docker + - name: Setup docker (${{ matrix.rocm_version }}) run: | touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh + bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} env: GITHUB_WORKSPACE: ${{ github.workspace }} - name: Install dependencies run: | - bash scripts/ci/amd/amd_ci_install_dependency.sh + bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps # Install tabulate for run_suite.py (missing in MI35x container) bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate - - name: Accuracy Test MI35x (8-GPU GPT-OSS) + - name: Accuracy Test MI35x ROCm (8-GPU GPT-OSS) timeout-minutes: 180 run: | + > github_summary.md # Clear summary file bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ + -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ python3 run_suite.py --hw amd --suite nightly-amd-8-gpu-mi35x --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? echo "$(> $GITHUB_STEP_SUMMARY || true exit ${TEST_EXIT_CODE:-0} + # Shares this job with the accuracy step above rather than taking its own, + # so the container setup and the GPT-OSS weight cache are paid for once. + - name: Performance Test MI35x ROCm (8-GPU GPT-OSS) + if: ${{ !cancelled() }} + timeout-minutes: 120 + run: | + > github_summary.md # Clear summary file + bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ + -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ + -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ + python3 run_suite.py --hw amd --suite nightly-perf-8-gpu-mi35x-gpt-oss --nightly --timeout-per-file 5400 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? + echo "$(> $GITHUB_STEP_SUMMARY || true + exit ${TEST_EXIT_CODE:-0} + # ============================================================================== # 8-GPU Grok1-INT4 (MI30x + MI35x) # ============================================================================== nightly-8-gpu-grok1-int4: - name: nightly-8-gpu-grok1-int4 (rocm700, linux-mi300-8gpu-sglang) + name: ${{ format('nightly-8-gpu-grok1-int4 ({0}, linux-mi300-8gpu-sglang)', matrix.rocm_version) }} + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm10", "rocm724", "rocm720"]') }} if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-grok1-int4,')) runs-on: linux-mi300-8gpu-sglang steps: @@ -566,17 +670,17 @@ jobs: - name: Ensure VRAM is clear run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - name: Setup docker + - name: Setup docker (${{ matrix.rocm_version }}) run: | touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh + bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} env: GITHUB_WORKSPACE: ${{ github.workspace }} - name: Install dependencies - run: bash scripts/ci/amd/amd_ci_install_dependency.sh + run: bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps - - name: Accuracy Test (8-GPU Grok1-INT4) + - name: Accuracy Test ROCm (8-GPU Grok1-INT4) timeout-minutes: 60 run: | > github_summary.md # Clear summary file @@ -587,9 +691,9 @@ jobs: echo "$(> $GITHUB_STEP_SUMMARY || true exit ${TEST_EXIT_CODE:-0} - - name: Performance Test (8-GPU Grok1-INT4) + - name: Performance Test ROCm (8-GPU Grok1-INT4) timeout-minutes: 60 - continue-on-error: true # Perf test failure doesn't fail the job if accuracy passed + continue-on-error: true run: | > github_summary.md # Clear summary file bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ @@ -600,7 +704,11 @@ jobs: exit ${TEST_EXIT_CODE:-0} nightly-8-gpu-mi35x-grok1-int4: - name: nightly-8-gpu-mi35x-grok1-int4 (rocm700, linux-mi35x-gpu-8) + name: ${{ format('nightly-8-gpu-mi35x-grok1-int4 ({0}, linux-mi35x-gpu-8)', matrix.rocm_version) }} + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm10", "rocm724", "rocm720"]') }} if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-mi35x-grok1-int4,')) runs-on: linux-mi35x-gpu-8 steps: @@ -612,33 +720,33 @@ jobs: - name: Ensure VRAM is clear run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - name: Setup docker + - name: Setup docker (${{ matrix.rocm_version }}) run: | touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh + bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} env: GITHUB_WORKSPACE: ${{ github.workspace }} - name: Install dependencies run: | - bash scripts/ci/amd/amd_ci_install_dependency.sh + bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps # Install tabulate for run_suite.py (missing in MI35x container) bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate - - name: Accuracy Test MI35x (8-GPU Grok1-INT4) - timeout-minutes: 90 + - name: Accuracy Test MI35x ROCm (8-GPU Grok1-INT4) + timeout-minutes: 60 run: | > github_summary.md # Clear summary file bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ -e RCCL_MSCCL_ENABLE=0 \ -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-accuracy-8-gpu-mi35x-grok1-int4 --nightly --timeout-per-file 5400 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? + python3 run_suite.py --hw amd --suite nightly-amd-accuracy-8-gpu-mi35x-grok1-int4 --nightly --timeout-per-file 3600 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? echo "$(> $GITHUB_STEP_SUMMARY || true exit ${TEST_EXIT_CODE:-0} - - name: Performance Test MI35x (8-GPU Grok1-INT4) + - name: Performance Test MI35x ROCm (8-GPU Grok1-INT4) timeout-minutes: 60 - continue-on-error: true # Perf test failure doesn't fail the job if accuracy passed + continue-on-error: true run: | > github_summary.md # Clear summary file bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ @@ -653,7 +761,11 @@ jobs: # ============================================================================== nightly-8-gpu-grok2: - name: nightly-8-gpu-grok2 (rocm700, linux-mi300-8gpu-sglang) + name: ${{ format('nightly-8-gpu-grok2 ({0}, linux-mi300-8gpu-sglang)', matrix.rocm_version) }} + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm10", "rocm724", "rocm720"]') }} if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-grok2,')) runs-on: linux-mi300-8gpu-sglang steps: @@ -665,17 +777,17 @@ jobs: - name: Ensure VRAM is clear run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - name: Setup docker + - name: Setup docker (${{ matrix.rocm_version }}) run: | touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh + bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} env: GITHUB_WORKSPACE: ${{ github.workspace }} - name: Install dependencies - run: bash scripts/ci/amd/amd_ci_install_dependency.sh + run: bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps - - name: Accuracy Test (8-GPU Grok2) + - name: Accuracy Test ROCm (8-GPU Grok2) timeout-minutes: 60 run: | > github_summary.md # Clear summary file @@ -686,9 +798,9 @@ jobs: echo "$(> $GITHUB_STEP_SUMMARY || true exit ${TEST_EXIT_CODE:-0} - - name: Performance Test (8-GPU Grok2) + - name: Performance Test ROCm (8-GPU Grok2) timeout-minutes: 60 - continue-on-error: true # Perf test failure doesn't fail the job if accuracy passed + continue-on-error: true run: | > github_summary.md # Clear summary file bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ @@ -699,7 +811,11 @@ jobs: exit ${TEST_EXIT_CODE:-0} nightly-8-gpu-mi35x-grok2: - name: nightly-8-gpu-mi35x-grok2 (rocm700, linux-mi35x-gpu-8) + name: ${{ format('nightly-8-gpu-mi35x-grok2 ({0}, linux-mi35x-gpu-8)', matrix.rocm_version) }} + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm10", "rocm724", "rocm720"]') }} if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-mi35x-grok2,')) runs-on: linux-mi35x-gpu-8 steps: @@ -711,20 +827,20 @@ jobs: - name: Ensure VRAM is clear run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - name: Setup docker + - name: Setup docker (${{ matrix.rocm_version }}) run: | touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh + bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} env: GITHUB_WORKSPACE: ${{ github.workspace }} - name: Install dependencies run: | - bash scripts/ci/amd/amd_ci_install_dependency.sh + bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps # Install tabulate for run_suite.py (missing in MI35x container) bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate - - name: Accuracy Test MI35x (8-GPU Grok2) + - name: Accuracy Test MI35x ROCm (8-GPU Grok2) timeout-minutes: 60 run: | > github_summary.md # Clear summary file @@ -735,9 +851,9 @@ jobs: echo "$(> $GITHUB_STEP_SUMMARY || true exit ${TEST_EXIT_CODE:-0} - - name: Performance Test MI35x (8-GPU Grok2) + - name: Performance Test MI35x ROCm (8-GPU Grok2) timeout-minutes: 60 - continue-on-error: true # Perf test failure doesn't fail the job if accuracy passed + continue-on-error: true run: | > github_summary.md # Clear summary file bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ @@ -748,12 +864,20 @@ jobs: exit ${TEST_EXIT_CODE:-0} # ============================================================================== - # 8-GPU DeepSeek-V3.x (MI30x) + # 8-GPU DeepSeek-V3.2 (MI30x) + # + # V3-0324 and V3.1 are no longer scheduled here: V3.2 covers the same MI30x + # paths (aiter MLA, EAGLE MTP, multithread weight load) on a current + # checkpoint, and DeepSeek-R1/V4 carry the MI35x side. # ============================================================================== - nightly-8-gpu-deepseek-v31: - name: nightly-8-gpu-deepseek-v31 (rocm700, linux-mi300-8gpu-sglang) - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-deepseek-v31,')) + nightly-8-gpu-deepseek-v32: + name: ${{ format('nightly-8-gpu-deepseek-v32 ({0}, linux-mi300-8gpu-sglang)', matrix.rocm_version) }} + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm10", "rocm724", "rocm720"]') }} + if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-deepseek-v32,')) runs-on: linux-mi300-8gpu-sglang steps: - name: Checkout code @@ -764,43 +888,45 @@ jobs: - name: Ensure VRAM is clear run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - name: Setup docker + - name: Setup docker (${{ matrix.rocm_version }}) run: | touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh + bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} env: GITHUB_WORKSPACE: ${{ github.workspace }} ENABLE_CACHE_HOST: "1" - name: Install dependencies - run: bash scripts/ci/amd/amd_ci_install_dependency.sh + run: bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps - - name: Accuracy Test (8-GPU DeepSeek-V3.1) + - name: Accuracy Test ROCm (8-GPU DeepSeek-V3.2 Basic) timeout-minutes: 120 run: | > github_summary.md # Clear summary file bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e SGLANG_USE_AITER=1 \ -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-accuracy-8-gpu-deepseek-v31 --nightly --timeout-per-file 3600 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? + python3 run_suite.py --hw amd --suite nightly-amd-accuracy-8-gpu-deepseek-v32 --nightly --timeout-per-file 3600 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? echo "$(> $GITHUB_STEP_SUMMARY || true exit ${TEST_EXIT_CODE:-0} - - name: Performance Test (8-GPU DeepSeek-V3.1) - timeout-minutes: 300 - continue-on-error: true # Perf test failure doesn't fail the job if accuracy passed + - name: Performance Test ROCm (8-GPU DeepSeek-V3.2 Basic) + timeout-minutes: 150 + continue-on-error: true run: | > github_summary.md # Clear summary file bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e SGLANG_USE_ROCM700A=1 \ -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-perf-8-gpu-deepseek-v31 --nightly --timeout-per-file 18000 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? + python3 run_suite.py --hw amd --suite nightly-perf-8-gpu-deepseek-v32-basic --nightly --timeout-per-file 5400 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? echo "$(> $GITHUB_STEP_SUMMARY || true exit ${TEST_EXIT_CODE:-0} - nightly-8-gpu-deepseek-v32: - name: nightly-8-gpu-deepseek-v32 (rocm700, linux-mi300-8gpu-sglang) - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-deepseek-v32,')) + nightly-8-gpu-deepseek-v32-mtp: + name: ${{ format('nightly-8-gpu-deepseek-v32-mtp ({0}, linux-mi300-8gpu-sglang)', matrix.rocm_version) }} + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm10", "rocm724", "rocm720"]') }} + if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-deepseek-v32-mtp,')) runs-on: linux-mi300-8gpu-sglang steps: - name: Checkout code @@ -811,42 +937,54 @@ jobs: - name: Ensure VRAM is clear run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - name: Setup docker + - name: Setup docker (${{ matrix.rocm_version }}) run: | touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh + bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} env: GITHUB_WORKSPACE: ${{ github.workspace }} ENABLE_CACHE_HOST: "1" - name: Install dependencies - run: bash scripts/ci/amd/amd_ci_install_dependency.sh + run: bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps - - name: Accuracy Test (8-GPU DeepSeek-V3.2 Basic) + - name: Accuracy Test ROCm (8-GPU DeepSeek-V3.2 MTP) timeout-minutes: 120 run: | > github_summary.md # Clear summary file bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-accuracy-8-gpu-deepseek-v32 --nightly --timeout-per-file 3600 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? + python3 run_suite.py --hw amd --suite nightly-amd-accuracy-8-gpu-deepseek-v32-mtp --nightly --timeout-per-file 3600 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? echo "$(> $GITHUB_STEP_SUMMARY || true exit ${TEST_EXIT_CODE:-0} - - name: Performance Test (8-GPU DeepSeek-V3.2 Basic) - timeout-minutes: 150 - continue-on-error: true # Perf test failure doesn't fail the job if accuracy passed + - name: Performance Test ROCm (8-GPU DeepSeek-V3.2 MTP) + timeout-minutes: 180 + continue-on-error: true run: | > github_summary.md # Clear summary file bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-perf-8-gpu-deepseek-v32-basic --nightly --timeout-per-file 5400 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? + python3 run_suite.py --hw amd --suite nightly-perf-8-gpu-deepseek-v32-mtp --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? echo "$(> $GITHUB_STEP_SUMMARY || true exit ${TEST_EXIT_CODE:-0} - nightly-8-gpu-deepseek-v32-mtp: - name: nightly-8-gpu-deepseek-v32-mtp (rocm700, linux-mi300-8gpu-sglang) - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-deepseek-v32-mtp,')) - runs-on: linux-mi300-8gpu-sglang + # ============================================================================== + # 8-GPU DeepSeek-V3.2 (MI35x) + # + # Accuracy and performance share one job per config, as the MI30x V3.2 jobs + # above already do: both steps serve the same weights, so splitting them cost + # a second container setup and a second cold weight load for nothing. + # ============================================================================== + + nightly-8-gpu-mi35x-deepseek-v32: + name: ${{ format('nightly-8-gpu-mi35x-deepseek-v32 ({0}, linux-mi35x-gpu-8)', matrix.rocm_version) }} + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm10", "rocm724", "rocm720"]') }} + if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-mi35x-deepseek-v32,')) + runs-on: linux-mi35x-gpu-8 steps: - name: Checkout code uses: actions/checkout@v4 @@ -856,42 +994,51 @@ jobs: - name: Ensure VRAM is clear run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - name: Setup docker + - name: Setup docker (${{ matrix.rocm_version }}) run: | touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh + bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} env: GITHUB_WORKSPACE: ${{ github.workspace }} ENABLE_CACHE_HOST: "1" - name: Install dependencies - run: bash scripts/ci/amd/amd_ci_install_dependency.sh + run: | + bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps + # Install tabulate for run_suite.py (missing in MI35x container) + bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate - - name: Accuracy Test (8-GPU DeepSeek-V3.2 MTP) + - name: Accuracy Test MI35x ROCm (8-GPU DeepSeek-V3.2) timeout-minutes: 120 run: | > github_summary.md # Clear summary file bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ + -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-accuracy-8-gpu-deepseek-v32-mtp --nightly --timeout-per-file 3600 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? + python3 run_suite.py --hw amd --suite nightly-amd-8-gpu-mi35x-deepseek-v32 --nightly --timeout-per-file 3600 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? echo "$(> $GITHUB_STEP_SUMMARY || true exit ${TEST_EXIT_CODE:-0} - - name: Performance Test (8-GPU DeepSeek-V3.2 MTP) - timeout-minutes: 180 - continue-on-error: true # Perf test failure doesn't fail the job if accuracy passed + - name: Performance Test MI35x (8-GPU DeepSeek-V3.2 Basic) + timeout-minutes: 150 + continue-on-error: true run: | > github_summary.md # Clear summary file bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ + -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-perf-8-gpu-deepseek-v32-mtp --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? + python3 run_suite.py --hw amd --suite nightly-perf-8-gpu-mi35x-deepseek-v32-basic --nightly --timeout-per-file 5400 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? echo "$(> $GITHUB_STEP_SUMMARY || true exit ${TEST_EXIT_CODE:-0} - nightly-8-gpu-deepseek-v3-kv-fp8: - name: nightly-8-gpu-deepseek-v3-kv-fp8 (rocm700, linux-mi300-8gpu-sglang) - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-deepseek-v3-kv-fp8,')) - runs-on: linux-mi300-8gpu-sglang + nightly-8-gpu-mi35x-deepseek-v32-mtp: + name: ${{ format('nightly-8-gpu-mi35x-deepseek-v32-mtp ({0}, linux-mi35x-gpu-8)', matrix.rocm_version) }} + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm10", "rocm724", "rocm720"]') }} + if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-mi35x-deepseek-v32-mtp,')) + runs-on: linux-mi35x-gpu-8 steps: - name: Checkout code uses: actions/checkout@v4 @@ -901,34 +1048,52 @@ jobs: - name: Ensure VRAM is clear run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - name: Setup docker + - name: Setup docker (${{ matrix.rocm_version }}) run: | touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh + bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} env: GITHUB_WORKSPACE: ${{ github.workspace }} ENABLE_CACHE_HOST: "1" - name: Install dependencies - run: bash scripts/ci/amd/amd_ci_install_dependency.sh + run: | + bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps + # Install tabulate for run_suite.py (missing in MI35x container) + bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate - - name: DeepSeek-V3 KV FP8 Test (8-GPU Basic + MTP) + - name: Accuracy Test MI35x ROCm (8-GPU DeepSeek-V3.2 TP+MTP) timeout-minutes: 120 run: | > github_summary.md # Clear summary file bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-8-gpu-deepseek-v3-kv-fp8 --nightly --timeout-per-file 3600 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? + python3 run_suite.py --hw amd --suite nightly-amd-accuracy-8-gpu-mi35x-deepseek-v32-mtp --nightly --timeout-per-file 3600 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? + echo "$(> $GITHUB_STEP_SUMMARY || true + exit ${TEST_EXIT_CODE:-0} + + - name: Performance Test MI35x ROCm (8-GPU DeepSeek-V3.2 MTP) + timeout-minutes: 180 + continue-on-error: true + run: | + > github_summary.md # Clear summary file + bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ + -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ + python3 run_suite.py --hw amd --suite nightly-perf-8-gpu-mi35x-deepseek-v32-mtp --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? echo "$(> $GITHUB_STEP_SUMMARY || true exit ${TEST_EXIT_CODE:-0} # ============================================================================== - # 8-GPU DeepSeek-V3.2 (MI35x) + # 8-GPU DeepSeek-R1 (MI35x only) # ============================================================================== - nightly-accuracy-8-gpu-mi35x-deepseek-v32: - name: nightly-accuracy-8-gpu-mi35x-deepseek-v32 (rocm700, linux-mi35x-gpu-8) - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-accuracy-8-gpu-mi35x-deepseek-v32,')) + nightly-8-gpu-mi35x-deepseek-r1-mxfp4: + name: ${{ format('nightly-8-gpu-mi35x-deepseek-r1-mxfp4 ({0}, linux-mi35x-gpu-8)', matrix.rocm_version) }} + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm10", "rocm724", "rocm720"]') }} + if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-mi35x-deepseek-r1-mxfp4,')) runs-on: linux-mi35x-gpu-8 steps: - name: Checkout code @@ -939,70 +1104,50 @@ jobs: - name: Ensure VRAM is clear run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - name: Setup docker + - name: Setup docker (${{ matrix.rocm_version }}) run: | touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh + bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} env: GITHUB_WORKSPACE: ${{ github.workspace }} ENABLE_CACHE_HOST: "1" - name: Install dependencies run: | - bash scripts/ci/amd/amd_ci_install_dependency.sh + bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps # Install tabulate for run_suite.py (missing in MI35x container) bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate - - name: Accuracy Test MI35x (8-GPU DeepSeek-V3.2) - timeout-minutes: 120 + - name: Accuracy Test MI35x ROCm (8-GPU DeepSeek-R1-MXFP4) + timeout-minutes: 180 run: | > github_summary.md # Clear summary file bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ + -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-8-gpu-mi35x-deepseek-v32 --nightly --timeout-per-file 3600 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? + python3 run_suite.py --hw amd --suite nightly-amd-8-gpu-mi35x-deepseek-r1-mxfp4 --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? echo "$(> $GITHUB_STEP_SUMMARY || true exit ${TEST_EXIT_CODE:-0} - nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp: - name: nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp (rocm700, linux-mi35x-gpu-8) - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp,')) - runs-on: linux-mi35x-gpu-8 - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Setup docker - run: | - touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - ENABLE_CACHE_HOST: "1" - - - name: Install dependencies - run: | - bash scripts/ci/amd/amd_ci_install_dependency.sh - # Install tabulate for run_suite.py (missing in MI35x container) - bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate - - - name: Accuracy Test MI35x (8-GPU DeepSeek-V3.2 TP+MTP) - timeout-minutes: 120 + - name: Performance Test MI35x ROCm (8-GPU DeepSeek-R1-MXFP4) + timeout-minutes: 300 + continue-on-error: true run: | > github_summary.md # Clear summary file bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ + -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-accuracy-8-gpu-mi35x-deepseek-v32-mtp --nightly --timeout-per-file 3600 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? + python3 registered/amd/perf/mi35x/test_deepseek_r1_mxfp4_perf_mi35x.py || TEST_EXIT_CODE=$? echo "$(> $GITHUB_STEP_SUMMARY || true exit ${TEST_EXIT_CODE:-0} - nightly-perf-8-gpu-mi35x-deepseek-v32-basic: - name: nightly-perf-8-gpu-mi35x-deepseek-v32-basic (rocm700, linux-mi35x-gpu-8) - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-perf-8-gpu-mi35x-deepseek-v32-basic,')) + nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4: + name: ${{ format('nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4 ({0}, linux-mi35x-gpu-8)', matrix.rocm_version) }} + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm10", "rocm724", "rocm720"]') }} + if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4,')) runs-on: linux-mi35x-gpu-8 steps: - name: Checkout code @@ -1013,33 +1158,49 @@ jobs: - name: Ensure VRAM is clear run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - name: Setup docker + - name: Setup docker (${{ matrix.rocm_version }}) run: | touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh + bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} env: GITHUB_WORKSPACE: ${{ github.workspace }} ENABLE_CACHE_HOST: "1" - name: Install dependencies run: | - bash scripts/ci/amd/amd_ci_install_dependency.sh - # Install tabulate for run_suite.py (missing in MI35x container) + bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate - - name: Performance Test MI35x (8-GPU DeepSeek-V3.2 Basic) - timeout-minutes: 150 + - name: Accuracy Test MI35x ROCm (DeepSeek-R1-MXFP4 TP4) + timeout-minutes: 180 run: | - > github_summary.md # Clear summary file + > github_summary.md bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ + -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-perf-8-gpu-mi35x-deepseek-v32-basic --nightly --timeout-per-file 5400 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? + python3 registered/amd/accuracy/mi35x/test_deepseek_r1_mxfp4_tp4_mi35x.py || TEST_EXIT_CODE=$? + echo "$(> $GITHUB_STEP_SUMMARY || true + exit ${TEST_EXIT_CODE:-0} + + - name: Accuracy Test MI35x ROCm (DeepSeek-R1-MXFP4 TP4 MTP) + if: ${{ !cancelled() }} + timeout-minutes: 180 + run: | + > github_summary.md + bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ + -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ + -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ + python3 registered/amd/accuracy/mi35x/test_deepseek_r1_mxfp4_tp4_mtp_mi35x.py || TEST_EXIT_CODE=$? echo "$(> $GITHUB_STEP_SUMMARY || true exit ${TEST_EXIT_CODE:-0} - nightly-perf-8-gpu-mi35x-deepseek-v32-mtp: - name: nightly-perf-8-gpu-mi35x-deepseek-v32-mtp (rocm700, linux-mi35x-gpu-8) - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-perf-8-gpu-mi35x-deepseek-v32-mtp,')) + nightly-8-gpu-mi35x-deepseek-r1-mxfp4-kv-fp8: + name: ${{ format('nightly-8-gpu-mi35x-deepseek-r1-mxfp4-kv-fp8 ({0}, linux-mi35x-gpu-8)', matrix.rocm_version) }} + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm10", "rocm724", "rocm720"]') }} + if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-mi35x-deepseek-r1-mxfp4-kv-fp8,')) runs-on: linux-mi35x-gpu-8 steps: - name: Checkout code @@ -1050,37 +1211,50 @@ jobs: - name: Ensure VRAM is clear run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - name: Setup docker + - name: Setup docker (${{ matrix.rocm_version }}) run: | touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh + bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} env: GITHUB_WORKSPACE: ${{ github.workspace }} ENABLE_CACHE_HOST: "1" - name: Install dependencies run: | - bash scripts/ci/amd/amd_ci_install_dependency.sh + bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps # Install tabulate for run_suite.py (missing in MI35x container) bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate - - name: Performance Test MI35x (8-GPU DeepSeek-V3.2 MTP) + - name: Accuracy Test MI35x ROCm (8-GPU DeepSeek-R1-MXFP4 KV FP8) timeout-minutes: 180 run: | > github_summary.md # Clear summary file bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ + -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-perf-8-gpu-mi35x-deepseek-v32-mtp --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? + python3 run_suite.py --hw amd --suite nightly-amd-8-gpu-mi35x-deepseek-r1-mxfp4-kv-fp8 --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? echo "$(> $GITHUB_STEP_SUMMARY || true exit ${TEST_EXIT_CODE:-0} - # ============================================================================== - # 8-GPU DeepSeek-R1 (MI35x only) - # ============================================================================== + - name: Performance Test MI35x ROCm (8-GPU DeepSeek-R1-MXFP4 KV FP8) + timeout-minutes: 300 + continue-on-error: true + run: | + > github_summary.md # Clear summary file + bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ + -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ + -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ + python3 registered/amd/perf/mi35x/test_deepseek_r1_mxfp4_kv_fp8_perf_mi35x.py || TEST_EXIT_CODE=$? + echo "$(> $GITHUB_STEP_SUMMARY || true + exit ${TEST_EXIT_CODE:-0} - nightly-8-gpu-mi35x-deepseek-r1-mxfp4: - name: nightly-8-gpu-mi35x-deepseek-r1-mxfp4 (rocm700, linux-mi35x-gpu-8) - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-mi35x-deepseek-r1-mxfp4,')) + nightly-8-gpu-mi35x-deepseek-r1-mxfp4-ar-fusion: + name: ${{ format('nightly-8-gpu-mi35x-deepseek-r1-mxfp4-ar-fusion ({0}, linux-mi35x-gpu-8)', matrix.rocm_version) }} + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm10", "rocm724", "rocm720"]') }} + if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-mi35x-deepseek-r1-mxfp4-ar-fusion,')) runs-on: linux-mi35x-gpu-8 steps: - name: Checkout code @@ -1091,44 +1265,50 @@ jobs: - name: Ensure VRAM is clear run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - name: Setup docker + - name: Setup docker (${{ matrix.rocm_version }}) run: | touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh + bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} env: GITHUB_WORKSPACE: ${{ github.workspace }} ENABLE_CACHE_HOST: "1" - name: Install dependencies run: | - bash scripts/ci/amd/amd_ci_install_dependency.sh + bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps # Install tabulate for run_suite.py (missing in MI35x container) bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate - - name: Accuracy Test MI35x (8-GPU DeepSeek-R1-MXFP4) + - name: Accuracy Test MI35x ROCm (8-GPU DeepSeek-R1-MXFP4 AllReduce Fusion) timeout-minutes: 180 run: | > github_summary.md # Clear summary file bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ + -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-8-gpu-mi35x-deepseek-r1-mxfp4 --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? + python3 run_suite.py --hw amd --suite nightly-amd-8-gpu-mi35x-deepseek-r1-mxfp4-ar-fusion --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? echo "$(> $GITHUB_STEP_SUMMARY || true exit ${TEST_EXIT_CODE:-0} - - name: Performance Test MI35x (8-GPU DeepSeek-R1-MXFP4) + - name: Performance Test MI35x ROCm (8-GPU DeepSeek-R1-MXFP4 AllReduce Fusion) timeout-minutes: 300 - continue-on-error: true # Perf test failure doesn't fail the job if accuracy passed + continue-on-error: true run: | > github_summary.md # Clear summary file bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ + -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 registered/amd/perf/mi35x/test_deepseek_r1_mxfp4_perf_mi35x.py || TEST_EXIT_CODE=$? + python3 registered/amd/perf/mi35x/test_deepseek_r1_mxfp4_ar_fusion_perf_mi35x.py || TEST_EXIT_CODE=$? echo "$(> $GITHUB_STEP_SUMMARY || true exit ${TEST_EXIT_CODE:-0} - nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4: - name: nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4 (rocm700, linux-mi35x-gpu-8) - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4,')) + nightly-8-gpu-mi35x-deepseek-r1-hicache: + name: ${{ format('nightly-8-gpu-mi35x-deepseek-r1-hicache ({0}, linux-mi35x-gpu-8)', matrix.rocm_version) }} + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm10", "rocm724", "rocm720"]') }} + if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-mi35x-deepseek-r1-hicache,')) runs-on: linux-mi35x-gpu-8 steps: - name: Checkout code @@ -1139,44 +1319,44 @@ jobs: - name: Ensure VRAM is clear run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - name: Setup docker + - name: Setup docker (${{ matrix.rocm_version }}) run: | touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh + bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} env: GITHUB_WORKSPACE: ${{ github.workspace }} - ENABLE_CACHE_HOST: "1" - name: Install dependencies run: | - bash scripts/ci/amd/amd_ci_install_dependency.sh + bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate - - name: Accuracy Test MI35x (DeepSeek-R1-MXFP4 TP4) + - name: Accuracy Test MI35x ROCm (8-GPU DeepSeek-R1 HiCache) timeout-minutes: 180 run: | > github_summary.md bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 registered/amd/accuracy/mi35x/test_deepseek_r1_mxfp4_tp4_mi35x.py || TEST_EXIT_CODE=$? + python3 run_suite.py --hw amd --suite nightly-amd-8-gpu-mi35x-deepseek-r1-hicache --nightly --timeout-per-file 3600 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? echo "$(> $GITHUB_STEP_SUMMARY || true exit ${TEST_EXIT_CODE:-0} - - name: Accuracy Test MI35x (DeepSeek-R1-MXFP4 TP4 MTP) - if: ${{ !cancelled() }} - timeout-minutes: 180 - run: | - > github_summary.md - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 registered/amd/accuracy/mi35x/test_deepseek_r1_mxfp4_tp4_mtp_mi35x.py || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} + # ============================================================================== + # 8-GPU DeepSeek-V4-Flash (MI30x) + # + # Accuracy only: the MI35x jobs below carry the DSV4 perf numbers, and gfx942 + # has no DSV4 perf baseline to regress against yet. Runs the cookbook's + # MI300X Flash FP8 low-latency cell, so it also covers MLA + KV-FP8 on gfx942. + # ============================================================================== - nightly-8-gpu-mi35x-deepseek-r1-mxfp4-kv-fp8: - name: nightly-8-gpu-mi35x-deepseek-r1-mxfp4-kv-fp8 (rocm700, linux-mi35x-gpu-8) - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-mi35x-deepseek-r1-mxfp4-kv-fp8,')) - runs-on: linux-mi35x-gpu-8 + nightly-8-gpu-deepseek-v4-flash: + name: ${{ format('nightly-8-gpu-deepseek-v4-flash ({0}, linux-mi300-8gpu-sglang)', matrix.rocm_version) }} + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm10", "rocm724", "rocm720"]') }} + if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-deepseek-v4-flash,')) + runs-on: linux-mi300-8gpu-sglang steps: - name: Checkout code uses: actions/checkout@v4 @@ -1186,44 +1366,39 @@ jobs: - name: Ensure VRAM is clear run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - name: Setup docker + - name: Setup docker (${{ matrix.rocm_version }}) run: | touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh + bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} env: GITHUB_WORKSPACE: ${{ github.workspace }} ENABLE_CACHE_HOST: "1" - name: Install dependencies - run: | - bash scripts/ci/amd/amd_ci_install_dependency.sh - # Install tabulate for run_suite.py (missing in MI35x container) - bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate + run: bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps - - name: Accuracy Test MI35x (8-GPU DeepSeek-R1-MXFP4 KV FP8) + - name: Accuracy Test ROCm (8-GPU DeepSeek-V4-Flash FP8) timeout-minutes: 180 run: | > github_summary.md # Clear summary file bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ + -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-8-gpu-mi35x-deepseek-r1-mxfp4-kv-fp8 --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? + python3 run_suite.py --hw amd --suite nightly-amd-accuracy-8-gpu-deepseek-v4-flash --nightly --timeout-per-file 5400 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? echo "$(> $GITHUB_STEP_SUMMARY || true exit ${TEST_EXIT_CODE:-0} - - name: Performance Test MI35x (8-GPU DeepSeek-R1-MXFP4 KV FP8) - timeout-minutes: 300 - continue-on-error: true # Perf test failure doesn't fail the job if accuracy passed - run: | - > github_summary.md # Clear summary file - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 registered/amd/perf/mi35x/test_deepseek_r1_mxfp4_kv_fp8_perf_mi35x.py || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} + # ============================================================================== + # 8-GPU DeepSeek-V4 (MI35x) + # ============================================================================== - nightly-8-gpu-mi35x-deepseek-r1-mxfp4-ar-fusion: - name: nightly-8-gpu-mi35x-deepseek-r1-mxfp4-ar-fusion (rocm700, linux-mi35x-gpu-8) - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-mi35x-deepseek-r1-mxfp4-ar-fusion,')) + nightly-8-gpu-mi35x-deepseek-v4-flash: + name: ${{ format('nightly-8-gpu-mi35x-deepseek-v4-flash ({0}, linux-mi35x-gpu-8)', matrix.rocm_version) }} + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm10", "rocm724", "rocm720"]') }} + if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-mi35x-deepseek-v4-flash,')) runs-on: linux-mi35x-gpu-8 steps: - name: Checkout code @@ -1234,44 +1409,52 @@ jobs: - name: Ensure VRAM is clear run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - name: Setup docker + - name: Setup docker (${{ matrix.rocm_version }}) run: | touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh + bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} env: GITHUB_WORKSPACE: ${{ github.workspace }} ENABLE_CACHE_HOST: "1" - name: Install dependencies run: | - bash scripts/ci/amd/amd_ci_install_dependency.sh - # Install tabulate for run_suite.py (missing in MI35x container) + # --skip-test-time-deps: GSM8K + bench_one_batch_server don't need lmms-eval / human-eval. + bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate - - name: Accuracy Test MI35x (8-GPU DeepSeek-R1-MXFP4 AllReduce Fusion) - timeout-minutes: 180 + - name: Accuracy + Performance Test MI35x ROCm (8-GPU DeepSeek-V4-Flash FP8 + FP4, unified_kv_triton) + timeout-minutes: 300 run: | > github_summary.md # Clear summary file + echo "## SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton" >> github_summary.md bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ + -e SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton \ + -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-8-gpu-mi35x-deepseek-r1-mxfp4-ar-fusion --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? + python3 run_suite.py --hw amd --suite nightly-amd-8-gpu-mi35x-deepseek-v4-flash --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - name: Performance Test MI35x (8-GPU DeepSeek-R1-MXFP4 AllReduce Fusion) + - name: Accuracy + Performance Test MI35x ROCm (8-GPU DeepSeek-V4-Flash FP8 + FP4, triton) + if: ${{ !cancelled() }} timeout-minutes: 300 - continue-on-error: true # Perf test failure doesn't fail the job if accuracy passed run: | > github_summary.md # Clear summary file + echo "## SGLANG_HACK_FLASHMLA_BACKEND=triton" >> github_summary.md bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ + -e SGLANG_HACK_FLASHMLA_BACKEND=triton \ + -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 registered/amd/perf/mi35x/test_deepseek_r1_mxfp4_ar_fusion_perf_mi35x.py || TEST_EXIT_CODE=$? + python3 run_suite.py --hw amd --suite nightly-amd-8-gpu-mi35x-deepseek-v4-flash --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - nightly-8-gpu-mi35x-deepseek-r1-hicache: - name: nightly-8-gpu-mi35x-deepseek-r1-hicache (rocm700, linux-mi35x-gpu-8) - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-mi35x-deepseek-r1-hicache,')) + nightly-8-gpu-mi35x-deepseek-v4-pro: + name: ${{ format('nightly-8-gpu-mi35x-deepseek-v4-pro ({0}, linux-mi35x-gpu-8)', matrix.rocm_version) }} + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm10", "rocm724", "rocm720"]') }} + if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-mi35x-deepseek-v4-pro,')) runs-on: linux-mi35x-gpu-8 steps: - name: Checkout code @@ -1282,36 +1465,53 @@ jobs: - name: Ensure VRAM is clear run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - name: Setup docker + - name: Setup docker (${{ matrix.rocm_version }}) run: | touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh + bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} env: GITHUB_WORKSPACE: ${{ github.workspace }} + ENABLE_CACHE_HOST: "1" - name: Install dependencies run: | - bash scripts/ci/amd/amd_ci_install_dependency.sh + # --skip-test-time-deps: GSM8K + bench_one_batch_server don't need lmms-eval / human-eval. + bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate - - name: Accuracy Test MI35x (8-GPU DeepSeek-R1 HiCache) - timeout-minutes: 180 + - name: Accuracy + Performance Test MI35x ROCm (8-GPU DeepSeek-V4-Pro FP8 + FP4, unified_kv_triton) + timeout-minutes: 480 run: | - > github_summary.md + > github_summary.md # Clear summary file + echo "## SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton" >> github_summary.md bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ + -e SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton \ + -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-8-gpu-mi35x-deepseek-r1-hicache --nightly --timeout-per-file 3600 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? + python3 run_suite.py --hw amd --suite nightly-amd-8-gpu-mi35x-deepseek-v4-pro --nightly --timeout-per-file 14400 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - # ============================================================================== - # 8-GPU Kimi-K2.6 (MI30x + MI35x) - # ============================================================================== + - name: Accuracy + Performance Test MI35x ROCm (8-GPU DeepSeek-V4-Pro FP8 + FP4, triton) + if: ${{ !cancelled() }} + timeout-minutes: 480 + run: | + > github_summary.md # Clear summary file + echo "## SGLANG_HACK_FLASHMLA_BACKEND=triton" >> github_summary.md + bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ + -e SGLANG_HACK_FLASHMLA_BACKEND=triton \ + -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ + -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ + python3 run_suite.py --hw amd --suite nightly-amd-8-gpu-mi35x-deepseek-v4-pro --nightly --timeout-per-file 14400 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} + echo "$(> $GITHUB_STEP_SUMMARY || true - nightly-8-gpu-kimi-k26: - name: nightly-8-gpu-kimi-k26 (rocm700, linux-mi300-8gpu-sglang) - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-kimi-k26,')) - runs-on: linux-mi300-8gpu-sglang + nightly-8-gpu-mi35x-deepseek-v4-pro-mtp: + name: ${{ format('nightly-8-gpu-mi35x-deepseek-v4-pro-mtp ({0}, linux-mi35x-gpu-8)', matrix.rocm_version) }} + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm10", "rocm724", "rocm720"]') }} + if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-mi35x-deepseek-v4-pro-mtp,')) + runs-on: linux-mi35x-gpu-8 steps: - name: Checkout code uses: actions/checkout@v4 @@ -1321,30 +1521,52 @@ jobs: - name: Ensure VRAM is clear run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - name: Setup docker + - name: Setup docker (${{ matrix.rocm_version }}) run: | touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh + bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} env: GITHUB_WORKSPACE: ${{ github.workspace }} ENABLE_CACHE_HOST: "1" - name: Install dependencies - run: bash scripts/ci/amd/amd_ci_install_dependency.sh + run: | + # --skip-test-time-deps: GSM8K + send_one don't need lmms-eval / human-eval. + bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps + bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate - - name: Accuracy Test (8-GPU Kimi-K2.6) - timeout-minutes: 120 + - name: Accuracy + MTP Accept Length Test MI35x ROCm (8-GPU DeepSeek-V4-Pro FP8 MTP, unified_kv_triton) + timeout-minutes: 240 run: | > github_summary.md # Clear summary file + echo "## SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton" >> github_summary.md bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ + -e SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton \ + -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-accuracy-8-gpu-kimi-k26 --nightly --timeout-per-file 3600 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? + python3 run_suite.py --hw amd --suite nightly-amd-8-gpu-mi35x-deepseek-v4-pro-mtp --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - nightly-8-gpu-mi35x-kimi-k26: - name: nightly-8-gpu-mi35x-kimi-k26 (rocm700, linux-mi35x-gpu-8) - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-mi35x-kimi-k26,')) + - name: Accuracy + MTP Accept Length Test MI35x ROCm (8-GPU DeepSeek-V4-Pro FP8 MTP, triton) + if: ${{ !cancelled() }} + timeout-minutes: 240 + run: | + > github_summary.md # Clear summary file + echo "## SGLANG_HACK_FLASHMLA_BACKEND=triton" >> github_summary.md + bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ + -e SGLANG_HACK_FLASHMLA_BACKEND=triton \ + -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ + -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ + python3 run_suite.py --hw amd --suite nightly-amd-8-gpu-mi35x-deepseek-v4-pro-mtp --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} + echo "$(> $GITHUB_STEP_SUMMARY || true + + nightly-8-gpu-mi35x-deepseek-v4-pro-dspark: + name: ${{ format('nightly-8-gpu-mi35x-deepseek-v4-pro-dspark ({0}, linux-mi35x-gpu-8)', matrix.rocm_version) }} + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm10", "rocm724", "rocm720"]') }} + if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-mi35x-deepseek-v4-pro-dspark,')) runs-on: linux-mi35x-gpu-8 steps: - name: Checkout code @@ -1355,37 +1577,46 @@ jobs: - name: Ensure VRAM is clear run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - name: Setup docker + - name: Setup docker (${{ matrix.rocm_version }}) run: | touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh + bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} env: GITHUB_WORKSPACE: ${{ github.workspace }} ENABLE_CACHE_HOST: "1" - name: Install dependencies run: | - bash scripts/ci/amd/amd_ci_install_dependency.sh - # Install tabulate for run_suite.py (missing in MI35x container) + # --skip-test-time-deps: GSM8K doesn't need lmms-eval / human-eval. + bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate - - name: Accuracy Test MI35x (8-GPU Kimi-K2.6) - timeout-minutes: 180 + - name: Accuracy + DSpark Accept Length Test MI35x ROCm (8-GPU DeepSeek-V4-Pro-DSpark FP4, unified_kv_triton) + timeout-minutes: 300 run: | > github_summary.md # Clear summary file + echo "## SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton" >> github_summary.md bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ + -e SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton \ + -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-accuracy-8-gpu-mi35x-kimi-k26 --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? + python3 run_suite.py --hw amd --suite nightly-amd-8-gpu-mi35x-deepseek-v4-pro-dspark --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} # ============================================================================== - # 8-GPU Qwen3-235B (MI30x + MI35x MXFP4) + # 8-GPU Kimi-K2.6 (MI30x) + # + # Kept on MI30x: Kimi-K3 needs gfx95x for its native MXFP4 weights, so it + # cannot take over here the way it does on MI35x. # ============================================================================== - nightly-8-gpu-qwen3-235b: - name: nightly-8-gpu-qwen3-235b (rocm700, linux-mi300-8gpu-sglang) - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-qwen3-235b,')) + nightly-8-gpu-kimi-k26: + name: ${{ format('nightly-8-gpu-kimi-k26 ({0}, linux-mi300-8gpu-sglang)', matrix.rocm_version) }} + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm10", "rocm724", "rocm720"]') }} + if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-kimi-k26,')) runs-on: linux-mi300-8gpu-sglang steps: - name: Checkout code @@ -1396,30 +1627,39 @@ jobs: - name: Ensure VRAM is clear run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - name: Setup docker + - name: Setup docker (${{ matrix.rocm_version }}) run: | touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh + bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} env: GITHUB_WORKSPACE: ${{ github.workspace }} ENABLE_CACHE_HOST: "1" - name: Install dependencies - run: bash scripts/ci/amd/amd_ci_install_dependency.sh + run: bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps - - name: Accuracy Test + Performance Test (8-GPU Qwen3) + - name: Accuracy Test ROCm (8-GPU Kimi-K2.6) timeout-minutes: 120 run: | > github_summary.md # Clear summary file bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ + -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-8-gpu-qwen3-235b --nightly --timeout-per-file 3600 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? + python3 run_suite.py --hw amd --suite nightly-amd-accuracy-8-gpu-kimi-k26 --nightly --timeout-per-file 3600 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? echo "$(> $GITHUB_STEP_SUMMARY || true exit ${TEST_EXIT_CODE:-0} - nightly-8-gpu-mi35x-qwen3-235b-mxfp4: - name: nightly-8-gpu-mi35x-qwen3-235b-mxfp4 (rocm700, linux-mi35x-gpu-8) - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-mi35x-qwen3-235b-mxfp4,')) + # ============================================================================== + # 8-GPU Kimi-K3 (MI35x only - native MXFP4 needs gfx95x) + # ============================================================================== + + nightly-8-gpu-mi35x-kimi-k3: + name: ${{ format('nightly-8-gpu-mi35x-kimi-k3 ({0}, linux-mi35x-gpu-8)', matrix.rocm_version) }} + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm10", "rocm724", "rocm720"]') }} + if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-mi35x-kimi-k3,')) runs-on: linux-mi35x-gpu-8 steps: - name: Checkout code @@ -1430,26 +1670,48 @@ jobs: - name: Ensure VRAM is clear run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - name: Setup docker + - name: Setup docker (${{ matrix.rocm_version }}) run: | touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh + bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} env: GITHUB_WORKSPACE: ${{ github.workspace }} + ENABLE_CACHE_HOST: "1" - name: Install dependencies run: | - bash scripts/ci/amd/amd_ci_install_dependency.sh + bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps # Install tabulate for run_suite.py (missing in MI35x container) bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate - - name: Accuracy Test + Performance Test MI35x (8-GPU Qwen3-235B-MXFP4) - timeout-minutes: 120 + # The 2.8T MXFP4 checkpoint is ~1.56 TB, so weight download and load + # dominate wall time well before the eval itself starts. + - name: Accuracy Test MI35x ROCm (8-GPU Kimi-K3) + timeout-minutes: 300 + run: | + > github_summary.md # Clear summary file + bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ + -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ + -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ + python3 run_suite.py --hw amd --suite nightly-amd-accuracy-8-gpu-mi35x-kimi-k3 --nightly --timeout-per-file 14400 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? + echo "$(> $GITHUB_STEP_SUMMARY || true + exit ${TEST_EXIT_CODE:-0} + + # Shares the job with the eval above so the checkpoint is already cached + # and only one 8-GPU MI35x slot is consumed. Step ordering is also the + # accuracy gate: a failed eval fails the job before this runs, so + # throughput is never measured on a build that got the tokens wrong. + # continue-on-error matches every other combined accuracy + perf job here, + # so a perf hiccup cannot turn the accuracy result red. + - name: Performance Test MI35x ROCm (8-GPU Kimi-K3) + timeout-minutes: 300 + continue-on-error: true run: | > github_summary.md # Clear summary file bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ + -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-8-gpu-mi35x-qwen3-235b-mxfp4 --nightly --timeout-per-file 3600 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? + python3 registered/amd/perf/mi35x/test_kimi_k3_perf_mi35x.py || TEST_EXIT_CODE=$? echo "$(> $GITHUB_STEP_SUMMARY || true exit ${TEST_EXIT_CODE:-0} @@ -1458,7 +1720,11 @@ jobs: # ============================================================================== nightly-8-gpu-qwen35: - name: nightly-8-gpu-qwen35 (rocm700, linux-mi300-8gpu-sglang) + name: ${{ format('nightly-8-gpu-qwen35 ({0}, linux-mi300-8gpu-sglang)', matrix.rocm_version) }} + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm10", "rocm724", "rocm720"]') }} if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-qwen35,')) runs-on: linux-mi300-8gpu-sglang steps: @@ -1470,20 +1736,20 @@ jobs: - name: Ensure VRAM is clear run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - name: Setup docker + - name: Setup docker (${{ matrix.rocm_version }}) run: | touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh + bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} env: GITHUB_WORKSPACE: ${{ github.workspace }} ENABLE_CACHE_HOST: "1" - name: Install dependencies run: | - bash scripts/ci/amd/amd_ci_install_dependency.sh + bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-aiter-build --skip-test-time-deps bash scripts/ci/amd/amd_ci_exec.sh pip install mistral-common "lm-eval[api]" - - name: Accuracy Test (8-GPU Qwen 3.5) + - name: Accuracy Test ROCm (8-GPU Qwen 3.5) timeout-minutes: 120 run: | > github_summary.md # Clear summary file @@ -1494,12 +1760,13 @@ jobs: echo "$(> $GITHUB_STEP_SUMMARY || true exit ${TEST_EXIT_CODE:-0} - - name: Performance Test (8-GPU Qwen 3.5 FP8) + - name: Performance Test ROCm (8-GPU Qwen 3.5 FP8) timeout-minutes: 120 continue-on-error: true run: | > github_summary.md # Clear summary file bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ + -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ -e SGLANG_USE_AITER=1 \ -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ python3 run_suite.py --hw amd --suite nightly-perf-8-gpu-qwen35-fp8 --nightly --timeout-per-file 5400 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? @@ -1507,7 +1774,11 @@ jobs: exit ${TEST_EXIT_CODE:-0} nightly-8-gpu-mi35x-qwen35: - name: nightly-8-gpu-mi35x-qwen35 (rocm700, linux-mi35x-gpu-8) + name: ${{ format('nightly-8-gpu-mi35x-qwen35 ({0}, linux-mi35x-gpu-8)', matrix.rocm_version) }} + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm10", "rocm724", "rocm720"]') }} if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-mi35x-qwen35,')) runs-on: linux-mi35x-gpu-8 steps: @@ -1519,43 +1790,49 @@ jobs: - name: Ensure VRAM is clear run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - name: Setup docker + - name: Setup docker (${{ matrix.rocm_version }}) run: | touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh + bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} env: GITHUB_WORKSPACE: ${{ github.workspace }} - name: Install dependencies run: | - bash scripts/ci/amd/amd_ci_install_dependency.sh + bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-aiter-build --skip-test-time-deps bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate bash scripts/ci/amd/amd_ci_exec.sh pip install mistral-common "lm-eval[api]" - - name: Accuracy Test MI35x (8-GPU Qwen 3.5) + - name: Accuracy Test MI35x ROCm (8-GPU Qwen 3.5) timeout-minutes: 120 run: | > github_summary.md # Clear summary file bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ + -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ python3 run_suite.py --hw amd --suite nightly-amd-accuracy-8-gpu-mi35x-qwen35 --nightly --timeout-per-file 3600 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? echo "$(> $GITHUB_STEP_SUMMARY || true exit ${TEST_EXIT_CODE:-0} - - name: Performance Test MI35x (8-GPU Qwen 3.5 FP8) + - name: Performance Test MI35x ROCm (8-GPU Qwen 3.5 FP8) timeout-minutes: 120 continue-on-error: true run: | > github_summary.md # Clear summary file bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ -e SGLANG_USE_AITER=1 \ + -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ python3 run_suite.py --hw amd --suite nightly-perf-8-gpu-mi35x-qwen35-fp8 --nightly --timeout-per-file 5400 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? echo "$(> $GITHUB_STEP_SUMMARY || true exit ${TEST_EXIT_CODE:-0} nightly-8-gpu-mi35x-qwen35-triton-dcp: - name: nightly-8-gpu-mi35x-qwen35-triton-dcp (rocm700, linux-mi35x-gpu-8) + name: ${{ format('nightly-8-gpu-mi35x-qwen35-triton-dcp ({0}, linux-mi35x-gpu-8)', matrix.rocm_version) }} + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm10", "rocm724", "rocm720"]') }} if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-mi35x-qwen35-triton-dcp,')) runs-on: linux-mi35x-gpu-8 steps: @@ -1567,34 +1844,98 @@ jobs: - name: Ensure VRAM is clear run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - name: Setup docker + - name: Setup docker (${{ matrix.rocm_version }}) run: | touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh + bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} env: GITHUB_WORKSPACE: ${{ github.workspace }} - name: Install dependencies run: | - bash scripts/ci/amd/amd_ci_install_dependency.sh + bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-aiter-build --skip-test-time-deps bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate - - name: Accuracy Test MI35x (8-GPU Qwen 3.5 Triton DCP) + - name: Accuracy Test MI35x ROCm (8-GPU Qwen 3.5 Triton DCP) timeout-minutes: 120 run: | > github_summary.md # Clear summary file bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ + -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ python3 run_suite.py --hw amd --suite nightly-amd-accuracy-8-gpu-mi35x-qwen35-triton-dcp --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? echo "$(> $GITHUB_STEP_SUMMARY || true exit ${TEST_EXIT_CODE:-0} + # ============================================================================== + # 8-GPU Qwen3.8-MXFP4 (MI35x only - MXFP4 needs gfx95x) + # ============================================================================== + + nightly-8-gpu-mi35x-qwen38-mxfp4: + name: ${{ format('nightly-8-gpu-mi35x-qwen38-mxfp4 ({0}, linux-mi35x-gpu-8)', matrix.rocm_version) }} + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm10", "rocm724", "rocm720"]') }} + if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-mi35x-qwen38-mxfp4,')) + runs-on: linux-mi35x-gpu-8 + steps: + - name: Checkout code + uses: actions/checkout@v4 + with: + ref: ${{ inputs.ref || github.sha }} + + - name: Ensure VRAM is clear + run: bash scripts/ci/amd/ensure_vram_clear.sh rocm + + - name: Setup docker (${{ matrix.rocm_version }}) + run: | + touch github_summary.md + bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} + env: + GITHUB_WORKSPACE: ${{ github.workspace }} + ENABLE_CACHE_HOST: "1" + + - name: Install dependencies + run: | + bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps + # Install tabulate for run_suite.py (missing in MI35x container) + bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate + + # ~1.2 TB of weights against a fleet-shared cache that runs at 100% used. + # Run 32196787596 died 40 minutes into the download on ENOSPC with no + # free-space figure anywhere in its log; record one up front instead. + - name: Check model cache space + run: | + bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout \ + bash scripts/ci/amd/check_hf_cache_space.sh \ + amd/Qwen3.8-2.4T-A95B-Quark-MXFP4 1250 + + # One step, not the usual accuracy/perf pair: both phases live in the + # same suite so they launch from one server config. The 2.4T MXFP4 + # checkpoint is ~1.2 TB and is loaded once per phase, which dominates + # wall time well before either the eval or the benchmark starts. + - name: Accuracy Test + Performance Test MI35x ROCm (8-GPU Qwen3.8-MXFP4) + timeout-minutes: 420 + run: | + > github_summary.md # Clear summary file + bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ + -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ + -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ + python3 run_suite.py --hw amd --suite nightly-amd-8-gpu-mi35x-qwen38-mxfp4 --nightly --timeout-per-file 21600 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? + echo "$(> $GITHUB_STEP_SUMMARY || true + exit ${TEST_EXIT_CODE:-0} + # ============================================================================== # 8-GPU GLM-5.1 (MI30x + MI35x) # ============================================================================== nightly-8-gpu-glm51: - name: nightly-8-gpu-glm51 (rocm700, linux-mi300-8gpu-sglang) + name: ${{ format('nightly-8-gpu-glm51 ({0}, linux-mi300-8gpu-sglang)', matrix.rocm_version) }} + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm10", "rocm724", "rocm720"]') }} if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-glm51,')) runs-on: linux-mi300-8gpu-sglang steps: @@ -1606,42 +1947,52 @@ jobs: - name: Ensure VRAM is clear run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - name: Setup docker + - name: Setup docker (${{ matrix.rocm_version }}) run: | touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh + bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} env: GITHUB_WORKSPACE: ${{ github.workspace }} - name: Install dependencies run: | - bash scripts/ci/amd/amd_ci_install_dependency.sh + bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps - - name: Accuracy Test (8-GPU GLM-5.1 DSA) + - name: Accuracy Test ROCm (8-GPU GLM-5.1 DSA) timeout-minutes: 120 run: | > github_summary.md # Clear summary file bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ + -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ python3 run_suite.py --hw amd --suite nightly-amd-accuracy-8-gpu-glm51 --nightly --timeout-per-file 3600 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? echo "$(> $GITHUB_STEP_SUMMARY || true exit ${TEST_EXIT_CODE:-0} - - name: Performance Test (8-GPU GLM-5.1) + - name: Performance Test ROCm (8-GPU GLM-5.1) timeout-minutes: 120 continue-on-error: true run: | > github_summary.md # Clear summary file bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ + -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ -e SGLANG_USE_AITER=1 \ -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ python3 run_suite.py --hw amd --suite nightly-perf-8-gpu-glm51 --nightly --timeout-per-file 5400 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? echo "$(> $GITHUB_STEP_SUMMARY || true exit ${TEST_EXIT_CODE:-0} - nightly-8-gpu-mi35x-glm51: - name: nightly-8-gpu-mi35x-glm51 (rocm700, linux-mi35x-gpu-8) - if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-mi35x-glm51,')) + # ============================================================================== + # 8-GPU GLM-5.2-FP8 (MI35x accuracy + performance) + # ============================================================================== + + nightly-8-gpu-mi35x-glm52-fp8: + name: ${{ format('nightly-8-gpu-mi35x-glm52-fp8 ({0}, linux-mi35x-gpu-8)', matrix.rocm_version) }} + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm10", "rocm724", "rocm720"]') }} + if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-mi35x-glm52-fp8,')) runs-on: linux-mi35x-gpu-8 steps: - name: Checkout code @@ -1652,36 +2003,43 @@ jobs: - name: Ensure VRAM is clear run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - name: Setup docker + - name: Setup docker (${{ matrix.rocm_version }}) run: | touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh + bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} env: GITHUB_WORKSPACE: ${{ github.workspace }} + ENABLE_CACHE_HOST: "1" - name: Install dependencies run: | - bash scripts/ci/amd/amd_ci_install_dependency.sh + bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate - - name: Accuracy Test MI35x (8-GPU GLM-5.1 DSA) + # The ~700 GB FP8 checkpoint means weight download and load dominate wall + # time well before the eval itself starts. + - name: Accuracy Test MI35x ROCm (8-GPU GLM-5.2-FP8 DSA) timeout-minutes: 180 run: | > github_summary.md # Clear summary file bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ + -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-amd-8-gpu-mi35x-glm51 --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? + python3 run_suite.py --hw amd --suite nightly-amd-8-gpu-mi35x-glm52-fp8 --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? echo "$(> $GITHUB_STEP_SUMMARY || true exit ${TEST_EXIT_CODE:-0} - - name: Performance Test MI35x (8-GPU GLM-5.1) + # Run perf only after correctness passes. Keeping both steps in this job + # reuses the cached ~700 GB checkpoint and one scarce 8-GPU MI35x slot. + - name: Performance Test MI35x ROCm (8-GPU GLM-5.2-FP8) timeout-minutes: 120 continue-on-error: true run: | > github_summary.md # Clear summary file bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ + -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - python3 run_suite.py --hw amd --suite nightly-perf-8-gpu-mi35x-glm51 --nightly --timeout-per-file 5400 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? + python3 run_suite.py --hw amd --suite nightly-perf-8-gpu-mi35x-glm52-fp8 --nightly --timeout-per-file 5400 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? echo "$(> $GITHUB_STEP_SUMMARY || true exit ${TEST_EXIT_CODE:-0} @@ -1690,7 +2048,11 @@ jobs: # ============================================================================== nightly-8-gpu-mi35x-glm5-mxfp4: - name: nightly-8-gpu-mi35x-glm5-mxfp4 (rocm700, linux-mi35x-gpu-8) + name: ${{ format('nightly-8-gpu-mi35x-glm5-mxfp4 ({0}, linux-mi35x-gpu-8)', matrix.rocm_version) }} + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm10", "rocm724", "rocm720"]') }} if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-mi35x-glm5-mxfp4,')) runs-on: linux-mi35x-gpu-8 steps: @@ -1702,36 +2064,38 @@ jobs: - name: Ensure VRAM is clear run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - name: Setup docker + - name: Setup docker (${{ matrix.rocm_version }}) run: | touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh + bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} env: GITHUB_WORKSPACE: ${{ github.workspace }} - name: Install dependencies run: | - bash scripts/ci/amd/amd_ci_install_dependency.sh + bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate - - name: Accuracy Test MI35x (8-GPU GLM-5-MXFP4) + - name: Accuracy Test MI35x ROCm (8-GPU GLM-5-MXFP4) timeout-minutes: 180 run: | > github_summary.md bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ -e SGLANG_USE_AITER=1 \ + -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ python3 run_suite.py --hw amd --suite nightly-amd-8-gpu-mi35x-glm5-mxfp4 --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? echo "$(> $GITHUB_STEP_SUMMARY || true exit ${TEST_EXIT_CODE:-0} - - name: Performance Test MI35x (8-GPU GLM-5-MXFP4) + - name: Performance Test MI35x ROCm (8-GPU GLM-5-MXFP4) timeout-minutes: 300 continue-on-error: true run: | > github_summary.md bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ -e SGLANG_USE_AITER=1 \ + -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ python3 registered/amd/perf/mi35x/test_glm5_mxfp4_perf_mi35x.py || TEST_EXIT_CODE=$? echo "$(> $GITHUB_STEP_SUMMARY || true @@ -1742,7 +2106,11 @@ jobs: # ============================================================================== nightly-4-gpu-mi35x-minimax-m25: - name: nightly-4-gpu-mi35x-minimax-m25 (rocm700, linux-mi35x-gpu-8) + name: ${{ format('nightly-4-gpu-mi35x-minimax-m25 ({0}, linux-mi35x-gpu-8)', matrix.rocm_version) }} + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm10", "rocm724", "rocm720"]') }} if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-4-gpu-mi35x-minimax-m25,')) runs-on: linux-mi35x-gpu-8 steps: @@ -1754,10 +2122,10 @@ jobs: - name: Ensure VRAM is clear run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - name: Setup docker (ROCm 7.0) + - name: Setup docker (${{ matrix.rocm_version }}) run: | touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh + bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} env: GITHUB_WORKSPACE: ${{ github.workspace }} ENABLE_CACHE_HOST: "1" @@ -1765,7 +2133,7 @@ jobs: - name: Install dependencies run: bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps - - name: Accuracy Test MI35x (4-GPU MiniMax-M2.5) + - name: Accuracy Test MI35x ROCm (4-GPU MiniMax-M2.5) timeout-minutes: 120 run: | > github_summary.md @@ -1778,11 +2146,15 @@ jobs: exit ${TEST_EXIT_CODE:-0} # ============================================================================== - # 4-GPU MiniMax-M3 MXFP8 (MI35x) + # 4-GPU MiniMax-M3 MXFP8 (MI35x accuracy + performance) # ============================================================================== nightly-4-gpu-mi35x-minimax-m3: - name: nightly-4-gpu-mi35x-minimax-m3 (rocm700, linux-mi35x-gpu-8) + name: ${{ format('nightly-4-gpu-mi35x-minimax-m3 ({0}, linux-mi35x-gpu-8)', matrix.rocm_version) }} + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm10", "rocm724", "rocm720"]') }} if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-4-gpu-mi35x-minimax-m3,')) runs-on: linux-mi35x-gpu-8 steps: @@ -1794,10 +2166,10 @@ jobs: - name: Ensure VRAM is clear run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - name: Setup docker (ROCm 7.0) + - name: Setup docker (${{ matrix.rocm_version }}) run: | touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh + bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} env: GITHUB_WORKSPACE: ${{ github.workspace }} ENABLE_CACHE_HOST: "1" @@ -1805,23 +2177,43 @@ jobs: - name: Install dependencies run: bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps - - name: Accuracy Test MI35x (4-GPU MiniMax-M3 MXFP8) + - name: Accuracy Test MI35x ROCm (4-GPU MiniMax-M3 MXFP8) timeout-minutes: 120 run: | > github_summary.md bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ -e SGLANG_USE_AITER=1 \ + -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ python3 run_suite.py --hw amd --suite nightly-amd-4-gpu-mi35x-minimax-m3-tp4 --nightly --timeout-per-file 5400 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? echo "$(> $GITHUB_STEP_SUMMARY || true exit ${TEST_EXIT_CODE:-0} + # Run perf only after correctness passes. Keeping both steps in this job + # reuses the cached checkpoint and one scarce MI35x slot. + - name: Performance Test MI35x ROCm (4-GPU MiniMax-M3 MXFP8) + timeout-minutes: 120 + continue-on-error: true # Perf test failure doesn't fail the job if accuracy passed + run: | + > github_summary.md + bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ + -e SGLANG_USE_AITER=1 \ + -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ + -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ + python3 run_suite.py --hw amd --suite nightly-perf-4-gpu-mi35x-minimax-m3 --nightly --timeout-per-file 5400 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? + echo "$(> $GITHUB_STEP_SUMMARY || true + exit ${TEST_EXIT_CODE:-0} + # ============================================================================== # 8-GPU MiniMax-M2.7 (MI30x only) # ============================================================================== nightly-8-gpu-minimax-m27: - name: nightly-8-gpu-minimax-m27 (rocm700, linux-mi300-8gpu-sglang) + name: ${{ format('nightly-8-gpu-minimax-m27 ({0}, linux-mi300-8gpu-sglang)', matrix.rocm_version) }} + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm10", "rocm724", "rocm720"]') }} if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-8-gpu-minimax-m27,')) runs-on: linux-mi300-8gpu-sglang steps: @@ -1833,34 +2225,36 @@ jobs: - name: Ensure VRAM is clear run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - name: Setup docker + - name: Setup docker (${{ matrix.rocm_version }}) run: | touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh + bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} env: GITHUB_WORKSPACE: ${{ github.workspace }} ENABLE_CACHE_HOST: "1" - name: Install dependencies - run: bash scripts/ci/amd/amd_ci_install_dependency.sh + run: bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps - - name: Accuracy Test (8-GPU MiniMax-M2.7) + - name: Accuracy Test ROCm (8-GPU MiniMax-M2.7) timeout-minutes: 120 run: | > github_summary.md # Clear summary file bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ + -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ -e SGLANG_USE_AITER=1 \ -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ python3 run_suite.py --hw amd --suite nightly-amd-accuracy-8-gpu-minimax-m27 --nightly --timeout-per-file 3600 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? echo "$(> $GITHUB_STEP_SUMMARY || true exit ${TEST_EXIT_CODE:-0} - - name: Performance Test (8-GPU MiniMax-M2.7) + - name: Performance Test ROCm (8-GPU MiniMax-M2.7) timeout-minutes: 120 continue-on-error: true # Perf test failure doesn't fail the job if accuracy passed run: | > github_summary.md # Clear summary file bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ + -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ -e SGLANG_USE_AITER=1 \ -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ python3 run_suite.py --hw amd --suite nightly-perf-8-gpu-minimax-m27 --nightly --timeout-per-file 5400 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? @@ -1872,7 +2266,11 @@ jobs: # ============================================================================== nightly-1-gpu-zimage-turbo: - name: nightly-1-gpu-zimage-turbo (rocm700, linux-mi300-1gpu-sglang) + name: ${{ format('nightly-1-gpu-zimage-turbo ({0}, linux-mi300-1gpu-sglang)', matrix.rocm_version) }} + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(inputs.rocm_version && inputs.rocm_version != 'all' && format('["{0}"]', inputs.rocm_version) || '["rocm10", "rocm724", "rocm720"]') }} if: (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') && (!(inputs.job_filter || inputs.job_select) || (inputs.job_filter || inputs.job_select) == 'all' || contains(format(',{0},', inputs.job_filter || inputs.job_select), ',nightly-1-gpu-zimage-turbo,')) runs-on: linux-mi300-1gpu-sglang steps: @@ -1884,17 +2282,17 @@ jobs: - name: Ensure VRAM is clear run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - name: Setup docker + - name: Setup docker (${{ matrix.rocm_version }}) run: | touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh + bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ matrix.rocm_version }} env: GITHUB_WORKSPACE: ${{ github.workspace }} - name: Install dependencies run: bash scripts/ci/amd/amd_ci_install_dependency.sh - - name: Z-Image-Turbo Diffusion Test (1-GPU) + - name: Z-Image-Turbo Diffusion Test ROCm (1-GPU) timeout-minutes: 45 run: | bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout \ @@ -1908,7 +2306,7 @@ jobs: if: always() uses: actions/upload-artifact@v4 with: - name: zimage-turbo-outputs + name: zimage-turbo-outputs-${{ matrix.rocm_version }} path: diffusion-artifacts/ if-no-files-found: ignore retention-days: 30 @@ -1918,12 +2316,13 @@ jobs: needs: # 1-GPU Unit Tests (MI30x + MI35x) - nightly-test-1-gpu-unit + - nightly-test-1-gpu-kernel - nightly-test-1-gpu-mi35x # 2-GPU and 4-GPU Tests (MI30x + MI35x) - nightly-accuracy-2-gpu - nightly-accuracy-2-gpu-vlm - # - nightly-perf-2-gpu-text # excluded: perf failures don't block CI - # - nightly-perf-2-gpu-vlm # excluded: perf failures don't block CI + - nightly-perf-2-gpu-text + - nightly-perf-2-gpu-vlm - nightly-4-gpu - nightly-2-gpu-mi35x-glm51-mxfp4 - nightly-2-gpu-mi35x-deepseek-r1-mxfp4-tp2 @@ -1936,40 +2335,44 @@ jobs: # 8-GPU Grok2 (MI30x + MI35x) - nightly-8-gpu-grok2 - nightly-8-gpu-mi35x-grok2 - # 8-GPU DeepSeek-V3.x (MI30x) - - nightly-8-gpu-deepseek-v31 + # 8-GPU DeepSeek-V3.2 (MI30x) - nightly-8-gpu-deepseek-v32 - nightly-8-gpu-deepseek-v32-mtp - - nightly-8-gpu-deepseek-v3-kv-fp8 # 8-GPU DeepSeek-V3.2 (MI35x) - - nightly-accuracy-8-gpu-mi35x-deepseek-v32 - - nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp - # - nightly-perf-8-gpu-mi35x-deepseek-v32-basic # excluded: perf failures don't block CI - # - nightly-perf-8-gpu-mi35x-deepseek-v32-mtp # excluded: perf failures don't block CI + - nightly-8-gpu-mi35x-deepseek-v32 + - nightly-8-gpu-mi35x-deepseek-v32-mtp # 8-GPU DeepSeek-R1 (MI35x only) - nightly-8-gpu-mi35x-deepseek-r1-mxfp4 - nightly-8-gpu-mi35x-deepseek-r1-mxfp4-kv-fp8 - nightly-8-gpu-mi35x-deepseek-r1-mxfp4-ar-fusion - nightly-8-gpu-mi35x-deepseek-r1-mxfp4-tp4 - nightly-8-gpu-mi35x-deepseek-r1-hicache - # 8-GPU Kimi-K2.6 (MI30x + MI35x) + # 8-GPU DeepSeek-V4-Flash (MI30x) + - nightly-8-gpu-deepseek-v4-flash + # 8-GPU DeepSeek-V4 (MI35x) + - nightly-8-gpu-mi35x-deepseek-v4-flash + - nightly-8-gpu-mi35x-deepseek-v4-pro + - nightly-8-gpu-mi35x-deepseek-v4-pro-mtp + - nightly-8-gpu-mi35x-deepseek-v4-pro-dspark + # 8-GPU Kimi-K2.6 (MI30x) - nightly-8-gpu-kimi-k26 - - nightly-8-gpu-mi35x-kimi-k26 - # 8-GPU Qwen3-235B (MI30x + MI35x MXFP4) - - nightly-8-gpu-qwen3-235b - - nightly-8-gpu-mi35x-qwen3-235b-mxfp4 + # 8-GPU Kimi-K3 (MI35x only - native MXFP4 needs gfx95x) + - nightly-8-gpu-mi35x-kimi-k3 + # 8-GPU Qwen3.8-MXFP4 (MI35x only - MXFP4 needs gfx95x) + - nightly-8-gpu-mi35x-qwen38-mxfp4 # 8-GPU Qwen 3.5 (MI30x + MI35x) - nightly-8-gpu-qwen35 - nightly-8-gpu-mi35x-qwen35 - nightly-8-gpu-mi35x-qwen35-triton-dcp - # 8-GPU GLM-5.1 (MI30x + MI35x) + # 8-GPU GLM-5.1 (MI30x) - nightly-8-gpu-glm51 - - nightly-8-gpu-mi35x-glm51 + # 8-GPU GLM-5.2-FP8 (MI35x accuracy + performance) + - nightly-8-gpu-mi35x-glm52-fp8 # 8-GPU GLM-5-MXFP4 (MI35x only) - nightly-8-gpu-mi35x-glm5-mxfp4 # 4-GPU MiniMax-M2.5 (MI35x) - nightly-4-gpu-mi35x-minimax-m25 - # 4-GPU MiniMax-M3 MXFP8 (MI35x) + # 4-GPU MiniMax-M3 MXFP8 (MI35x accuracy + performance) - nightly-4-gpu-mi35x-minimax-m3 # 8-GPU MiniMax-M2.7 (MI30x only) - nightly-8-gpu-minimax-m27 @@ -1980,11 +2383,11 @@ jobs: - name: Check if any job failed run: | if [[ "${{ contains(needs.*.result, 'failure') }}" == "true" ]]; then - echo "One or more nightly test jobs failed" + echo "One or more ROCm nightly test jobs failed" exit 1 fi if [[ "${{ contains(needs.*.result, 'cancelled') }}" == "true" ]]; then - echo "One or more nightly test jobs were cancelled" + echo "One or more ROCm nightly test jobs were cancelled" exit 1 fi - echo "All nightly test jobs passed" + echo "All ROCm nightly test jobs passed" diff --git a/.github/workflows/nightly-test-intel.yml b/.github/workflows/nightly-test-intel.yml index 2d16a5df9d9b..15e496ba53e1 100644 --- a/.github/workflows/nightly-test-intel.yml +++ b/.github/workflows/nightly-test-intel.yml @@ -59,6 +59,17 @@ jobs: env: GITHUB_WORKSPACE: ${{ github.workspace }} + - name: Overlay in-tree sglang test helpers onto image site-packages + run: | + # Bridges the up-to-24h window between merge and nightly image rebuild. + dest=$(docker exec ci_sglang_xpu /opt/venv/bin/python3 -c \ + "import sglang, os; print(os.path.dirname(sglang.__file__))") + docker exec ci_sglang_xpu cp \ + /sglang-checkout/python/sglang/test/ci/ci_utils.py \ + "$dest/test/ci/ci_utils.py" + docker exec ci_sglang_xpu /opt/venv/bin/python3 -c \ + "import sglang.test.ci.ci_utils; print('overlay ok')" + - name: HF login + install run_suite extras timeout-minutes: 10 run: | @@ -95,17 +106,28 @@ jobs: - name: Nightly Test (1-GPU XPU) timeout-minutes: 240 run: | - touch github_summary.md + touch github_summary.md nightly-xpu-1-gpu-metrics.jsonl docker exec ci_sglang_xpu bash -c " source /opt/venv/bin/activate && cd /sglang-checkout/test && OLMOCR_BENCH_DIR=/sglang-checkout/olmOCR-bench/bench_data \ GITHUB_STEP_SUMMARY=/sglang-checkout/github_summary.md \ + SGLANG_IS_IN_CI=true \ + SGLANG_TEST_METRICS_FILE=/sglang-checkout/nightly-xpu-1-gpu-metrics.jsonl \ python3 run_suite.py --hw xpu --suite nightly-xpu-1-gpu --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} " || TEST_EXIT_CODE=$? echo "$(> $GITHUB_STEP_SUMMARY || true exit ${TEST_EXIT_CODE:-0} + - name: Upload per-model metrics jsonl + if: always() + uses: actions/upload-artifact@v4 + with: + name: nightly-xpu-1-gpu-metrics + path: nightly-xpu-1-gpu-metrics.jsonl + if-no-files-found: warn + retention-days: 30 + - name: Cleanup container if: always() run: | @@ -144,6 +166,17 @@ jobs: env: GITHUB_WORKSPACE: ${{ github.workspace }} + - name: Overlay in-tree sglang test helpers onto image site-packages + run: | + # Bridges the up-to-24h window between merge and nightly image rebuild. + dest=$(docker exec ci_sglang_xpu /opt/venv/bin/python3 -c \ + "import sglang, os; print(os.path.dirname(sglang.__file__))") + docker exec ci_sglang_xpu cp \ + /sglang-checkout/python/sglang/test/ci/ci_utils.py \ + "$dest/test/ci/ci_utils.py" + docker exec ci_sglang_xpu /opt/venv/bin/python3 -c \ + "import sglang.test.ci.ci_utils; print('overlay ok')" + - name: HF login + install run_suite extras timeout-minutes: 10 run: | @@ -170,16 +203,27 @@ jobs: - name: Nightly Test (2-GPU XPU) timeout-minutes: 240 run: | - touch github_summary.md + touch github_summary.md nightly-xpu-2-gpu-metrics.jsonl docker exec ci_sglang_xpu bash -c " source /opt/venv/bin/activate && cd /sglang-checkout/test && GITHUB_STEP_SUMMARY=/sglang-checkout/github_summary.md \ + SGLANG_IS_IN_CI=true \ + SGLANG_TEST_METRICS_FILE=/sglang-checkout/nightly-xpu-2-gpu-metrics.jsonl \ python3 run_suite.py --hw xpu --suite nightly-xpu-2-gpu --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} " || TEST_EXIT_CODE=$? echo "$(> $GITHUB_STEP_SUMMARY || true exit ${TEST_EXIT_CODE:-0} + - name: Upload per-model metrics jsonl + if: always() + uses: actions/upload-artifact@v4 + with: + name: nightly-xpu-2-gpu-metrics + path: nightly-xpu-2-gpu-metrics.jsonl + if-no-files-found: warn + retention-days: 30 + - name: Cleanup container if: always() run: | @@ -218,6 +262,17 @@ jobs: env: GITHUB_WORKSPACE: ${{ github.workspace }} + - name: Overlay in-tree sglang test helpers onto image site-packages + run: | + # Bridges the up-to-24h window between merge and nightly image rebuild. + dest=$(docker exec ci_sglang_xpu /opt/venv/bin/python3 -c \ + "import sglang, os; print(os.path.dirname(sglang.__file__))") + docker exec ci_sglang_xpu cp \ + /sglang-checkout/python/sglang/test/ci/ci_utils.py \ + "$dest/test/ci/ci_utils.py" + docker exec ci_sglang_xpu /opt/venv/bin/python3 -c \ + "import sglang.test.ci.ci_utils; print('overlay ok')" + - name: HF login + install run_suite extras timeout-minutes: 10 run: | @@ -244,16 +299,123 @@ jobs: - name: Nightly Test (4-GPU XPU) timeout-minutes: 480 run: | - touch github_summary.md + touch github_summary.md nightly-xpu-4-gpu-metrics.jsonl docker exec ci_sglang_xpu bash -c " source /opt/venv/bin/activate && cd /sglang-checkout/test && GITHUB_STEP_SUMMARY=/sglang-checkout/github_summary.md \ + SGLANG_IS_IN_CI=true \ + SGLANG_TEST_METRICS_FILE=/sglang-checkout/nightly-xpu-4-gpu-metrics.jsonl \ python3 run_suite.py --hw xpu --suite nightly-xpu-4-gpu --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} " || TEST_EXIT_CODE=$? echo "$(> $GITHUB_STEP_SUMMARY || true exit ${TEST_EXIT_CODE:-0} + - name: Upload per-model metrics jsonl + if: always() + uses: actions/upload-artifact@v4 + with: + name: nightly-xpu-4-gpu-metrics + path: nightly-xpu-4-gpu-metrics.jsonl + if-no-files-found: warn + retention-days: 30 + + - name: Cleanup container + if: always() + run: | + docker run --rm -v "${{ github.workspace }}:/w" busybox:latest \ + chown -R "$(id -u):$(id -g)" /w || true + rm -rf test/result.jsonl test/results test/.pytest_cache .pytest_cache || true + find . -type d -name "__pycache__" -prune -exec rm -rf {} + || true + find . -type f -name "*.pyc" -delete || true + docker rm -f ci_sglang_xpu || true + if [[ -n "${CI_SGLANG_XPU_IMAGE:-}" ]]; then + docker rmi -f "${CI_SGLANG_XPU_IMAGE}" || true + fi + + nightly-xpu-8-gpu: + if: github.repository == 'sgl-project/sglang' + runs-on: intel-bmg-nightly + env: + DOCKERHUB_INTEL_USERNAME: ${{ secrets.DOCKERHUB_INTEL_USERNAME }} + DOCKERHUB_INTEL_TOKEN: ${{ secrets.DOCKERHUB_INTEL_TOKEN }} + steps: + - name: Reset workspace ownership + run: | + docker run --rm -v "${{ github.workspace }}:/w" busybox:latest \ + chown -R "$(id -u):$(id -g)" /w || true + + - name: Checkout code + uses: actions/checkout@v4 + with: + fetch-depth: 0 + ref: ${{ inputs.ref || github.sha }} + + - name: Start CI container (pull intel/sglang-dev:latest) + run: | + export HF_TOKEN="$(cat ~/huggingface_token.txt)" + bash scripts/ci/xpu/xpu_ci_start_container.sh + env: + GITHUB_WORKSPACE: ${{ github.workspace }} + + - name: Overlay in-tree sglang test helpers onto image site-packages + run: | + # Bridges the up-to-24h window between merge and nightly image rebuild. + dest=$(docker exec ci_sglang_xpu /opt/venv/bin/python3 -c \ + "import sglang, os; print(os.path.dirname(sglang.__file__))") + docker exec ci_sglang_xpu cp \ + /sglang-checkout/python/sglang/test/ci/ci_utils.py \ + "$dest/test/ci/ci_utils.py" + docker exec ci_sglang_xpu /opt/venv/bin/python3 -c \ + "import sglang.test.ci.ci_utils; print('overlay ok')" + + - name: HF login + install run_suite extras + timeout-minutes: 10 + run: | + docker exec ci_sglang_xpu /opt/venv/bin/python3 -m pip install --no-cache-dir tabulate + docker exec ci_sglang_xpu /opt/venv/bin/python3 -m pip install --no-cache-dir --no-deps xgrammar==0.1.33 + docker exec ci_sglang_xpu /opt/venv/bin/python3 -m pip install --no-cache-dir "lm-eval==0.4.9" + docker exec ci_sglang_xpu /opt/venv/bin/python3 -m pip install --no-cache-dir pytest + # Diffusion extras + # Pins mirror python/pyproject_xpu.toml [project.optional-dependencies.diffusion]. + docker exec ci_sglang_xpu /opt/venv/bin/python3 -m pip install --no-cache-dir "sglang[diffusion]" + # HF /whoami-v2 is aggressively rate-limited and the three matrix + # jobs (1/2/4/8-GPU) all hit it at the same second under a scheduled + # run. Retry with linear backoff + jitter to desynchronize them. + docker exec ci_sglang_xpu /bin/bash -c ' + for i in 1 2 3 4 5; do + /opt/venv/bin/hf auth login --token "${HF_TOKEN}" && exit 0 + sleep_secs=$((i * 15 + RANDOM % 15)) + echo "hf auth login failed (attempt $i), retrying in ${sleep_secs}s..." + sleep "${sleep_secs}" + done + exit 1 + ' + + - name: Nightly Test (8-GPU XPU) + timeout-minutes: 480 + run: | + touch github_summary.md nightly-xpu-8-gpu-metrics.jsonl + docker exec ci_sglang_xpu bash -c " + source /opt/venv/bin/activate && + cd /sglang-checkout/test && + GITHUB_STEP_SUMMARY=/sglang-checkout/github_summary.md \ + SGLANG_IS_IN_CI=true \ + SGLANG_TEST_METRICS_FILE=/sglang-checkout/nightly-xpu-8-gpu-metrics.jsonl \ + python3 run_suite.py --hw xpu --suite nightly-xpu-8-gpu --nightly --timeout-per-file 7200 ${{ (github.event_name == 'schedule' || inputs.continue_on_error) && '--continue-on-error' || '' }} + " || TEST_EXIT_CODE=$? + echo "$(> $GITHUB_STEP_SUMMARY || true + exit ${TEST_EXIT_CODE:-0} + + - name: Upload per-model metrics jsonl + if: always() + uses: actions/upload-artifact@v4 + with: + name: nightly-xpu-8-gpu-metrics + path: nightly-xpu-8-gpu-metrics.jsonl + if-no-files-found: warn + retention-days: 30 + - name: Cleanup container if: always() run: | @@ -273,6 +435,7 @@ jobs: - nightly-xpu-1-gpu - nightly-xpu-2-gpu - nightly-xpu-4-gpu + - nightly-xpu-8-gpu runs-on: ubuntu-latest steps: - name: Check if any job failed diff --git a/.github/workflows/nightly-test-musa.yml b/.github/workflows/nightly-test-musa.yml index 23a257ae086e..15e6c939d409 100644 --- a/.github/workflows/nightly-test-musa.yml +++ b/.github/workflows/nightly-test-musa.yml @@ -74,9 +74,9 @@ jobs: if: steps.gate.outputs.run_job == 'true' timeout-minutes: 30 run: | - pytest python/sglang/kernels/aot/tests/speculative/test_eagle_utils.py - pytest python/sglang/kernels/aot/tests/speculative/test_ngram_utils.py - pytest python/sglang/kernels/aot/tests/speculative/test_speculative_sampling.py + pytest test/registered/kernels/ops/speculative/test_eagle_utils.py + pytest test/registered/kernels/ops/speculative/test_ngram_utils.py + pytest test/registered/kernels/ops/speculative/test_speculative_sampling.py pytest python/sglang/kernels/aot/tests/test_torch_defaults_reset.py # ==================== General: multimodal layer ==================== diff --git a/.github/workflows/nightly-test-npu-e2e-multi-node.yml b/.github/workflows/nightly-test-npu-e2e-multi-node.yml index d591b1c2726c..d54167a20e64 100644 --- a/.github/workflows/nightly-test-npu-e2e-multi-node.yml +++ b/.github/workflows/nightly-test-npu-e2e-multi-node.yml @@ -79,7 +79,7 @@ jobs: KUBECONFIG: /root/.cache/.cache/kb.yaml KUBECTL: /root/.cache/.cache/kubectl NAMESPACE: sgl-project - KUBE_JOB_NAME: ascend-sglang-${{ inputs.test_type }}-test + KUBE_JOB_NAME: ascend-${{ github.run_id }} SGLANG_IS_IN_CI: true ASCEND_E2E_TEST_CONFIG_PATH: python/sglang/test/ascend/e2e SGLANG_USE_MODELSCOPE: true @@ -101,8 +101,9 @@ jobs: - name: Prepare code for testing run: | # copy source code to shared-disk + # Append run_id to isolate concurrent runs sharing the same PVC. current_path=$(pwd) - target_path=/root/.cache/tests/sglang + target_path=/root/.cache/tests/sglang-${{ github.run_id }} rm -rf ${target_path} mkdir -p ${target_path} cp -r ${current_path}/* ${target_path}/ @@ -129,7 +130,8 @@ jobs: timeout-minutes: 300 run: | # sglang_source_relative_path is shared-disk path - sglang_source_relative_path=tests/sglang + # Append run_id so concurrent runs do not overwrite each other on the shared PVC. + sglang_source_relative_path=tests/sglang-${{ github.run_id }} sglang_source_path=/root/.cache/${sglang_source_relative_path} echo "Source code path: ${sglang_source_path}" @@ -155,6 +157,11 @@ jobs: workflow_name=$(echo "${{ fromJson(inputs.run_start_metadata).workflow_name }}" | tr ' ' '_' | tr -d '()') create_time="${{ fromJson(inputs.run_start_metadata).create_time }}" if [ -z "${create_time}" ]; then create_time=$(date -u -d '+8 hours' +%Y%m%d-%H%M); fi + # Append the image tag to the workflow-name segment so that runs with + # different images (CANN 9.0.0/9.1.0, a3/a5) persist into distinct, + # image-tagged output/log directories. + image_label=$(echo "${{ inputs.image }}" | sed 's#.*[/:]##') + workflow_name="${workflow_name}--${image_label}" # Print persistence‑directory‑prefix variables echo "Persistence prefix: run_id=${run_id} run_attempt=${run_attempt} branch_label=${branch_label} workflow_name=${workflow_name} create_time=${create_time}" @@ -184,7 +191,8 @@ jobs: --test-case ${test_case} \ --kube-name-space ${NAMESPACE} \ --kube-job-type ${kube_job_type} \ - --kube-job-name-prefix ${KUBE_JOB_NAME}" + --kube-job-name-prefix ${KUBE_JOB_NAME} \ + --run-id ${{ github.run_id }}" if [ "$prefill_decode_deployment" = "separation" ];then CMD="${CMD} \ @@ -222,14 +230,6 @@ jobs: echo "Run command: ${CMD}" eval "${CMD}" - - name: Upload metrics - if: always() - uses: actions/upload-artifact@v4 - with: - name: metrics-${{ inputs.test_config_name }} - path: /tmp/metrics.json - retention-days: 7 - - name: Post process if: always() run: | @@ -249,3 +249,5 @@ jobs: break fi done + # Cleanup the run-scoped source code directory on the shared PVC. + rm -rf /root/.cache/tests/sglang-${{ github.run_id }} || true diff --git a/.github/workflows/nightly-test-npu.yml b/.github/workflows/nightly-test-npu.yml index 6cde2e662bbe..7104e744bc3f 100644 --- a/.github/workflows/nightly-test-npu.yml +++ b/.github/workflows/nightly-test-npu.yml @@ -19,16 +19,16 @@ on: required: false type: string default: 'all' - image_a2: - description: 'The a2 running docker image of the test task.' - required: false - type: string - default: 'swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:main-cann9.0.0-910b' image_a3: description: 'The a3 running docker image of the test task.' required: false type: string - default: 'swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:main-cann9.0.0-a3' + default: 'swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:main-cann9.1.0-a3' + image_a5: + description: 'The a5 running docker image of the test task.' + required: false + type: string + default: 'swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:main-cann9.1.0-950' skip_install_flag: description: 'Indicates whether to skip the installation of sglang, defaulting to false.' required: false @@ -46,13 +46,13 @@ on: required: false type: string default: 'all' - image_a2: - description: 'The a2 running docker image of the test task.' + image_a3: + description: 'The a3 running docker image of the test task.' required: false type: string default: '' - image_a3: - description: 'The a3 running docker image of the test task.' + image_a5: + description: 'The a5 running docker image of the test task.' required: false type: string default: '' @@ -70,8 +70,8 @@ jobs: outputs: ref: ${{ steps.set-vars.outputs.ref }} job_filter: ${{ steps.set-vars.outputs.job_filter }} - image_a2: ${{ steps.set-vars.outputs.image_a2 }} image_a3: ${{ steps.set-vars.outputs.image_a3 }} + image_a5: ${{ steps.set-vars.outputs.image_a5 }} skip_install_flag: ${{ steps.set-vars.outputs.skip_install_flag }} run_start_metadata: ${{ steps.set-vars.outputs.run_start_metadata }} steps: @@ -93,16 +93,16 @@ jobs: else echo "job_filter=${{ inputs.job_filter }}" >> $GITHUB_OUTPUT fi - if [ -z "${{ inputs.image_a2 }}" ]; then - echo "image_a2=swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:main-cann9.0.0-910b" >> $GITHUB_OUTPUT - else - echo "image_a2=${{ inputs.image_a2 }}" >> $GITHUB_OUTPUT - fi if [ -z "${{ inputs.image_a3 }}" ]; then - echo "image_a3=swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:main-cann9.0.0-a3" >> $GITHUB_OUTPUT + echo "image_a3=swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:main-cann9.1.0-a3" >> $GITHUB_OUTPUT else echo "image_a3=${{ inputs.image_a3 }}" >> $GITHUB_OUTPUT fi + if [ -z "${{ inputs.image_a5 }}" ]; then + echo "image_a5=swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:main-cann9.1.0-950" >> $GITHUB_OUTPUT + else + echo "image_a5=${{ inputs.image_a5 }}" >> $GITHUB_OUTPUT + fi if [ -z "${{ inputs.skip_install_flag }}" ]; then echo "skip_install_flag=false" >> $GITHUB_OUTPUT else @@ -122,22 +122,6 @@ jobs: # Write to GITHUB_OUTPUT and print to the log in one command. echo "run_start_metadata=${RUN_START_METADATA}" | tee -a $GITHUB_OUTPUT - nightly-1-npu-a2: - name: nightly-1-npu-a2 - if: ${{ !cancelled() }} - needs: [set-image-config] - uses: ./.github/workflows/_npu-single-node-test-stage.yml - with: - runner: linux-aarch64-a2-4 - test_type: 'perf' - test_suite: nightly-1-npu-a2 - is_nightly_pipeline_job: true - skip_pr_test_health_check: 'true' - image: ${{ needs.set-image-config.outputs.image_a2 }} - install_sglang_deps: false - device_type_for_deps: '910b' - run_start_metadata: ${{ needs.set-image-config.outputs.run_start_metadata }} - nightly-1-npu-a3: name: nightly-1-npu-a3 if: ${{ !cancelled() }} @@ -147,7 +131,7 @@ jobs: self_name: nightly-1-npu-a3 runner_config: linux-aarch64-a3-2- image: ${{ needs.set-image-config.outputs.image_a3 }} - run_timeout_minutes: '60' + run_timeout_minutes: '90' timeout_per_file: '3600' is_nightly_pipeline_job: true skip_pr_test_health_check: 'true' @@ -162,7 +146,7 @@ jobs: self_name: nightly-2-npu-a3 runner_config: linux-aarch64-a3-2- image: ${{ needs.set-image-config.outputs.image_a3 }} - run_timeout_minutes: '60' + run_timeout_minutes: '90' timeout_per_file: '3600' is_nightly_pipeline_job: true skip_pr_test_health_check: 'true' @@ -247,22 +231,6 @@ jobs: device_type_for_deps: 'a3' run_start_metadata: ${{ needs.set-image-config.outputs.run_start_metadata }} - nightly-perf-8-npu-a3: - name: nightly-perf-8-npu-a3 - if: ${{ !cancelled() }} - needs: [set-image-config] - uses: ./.github/workflows/_npu-single-node-test-stage.yml - with: - runner: linux-aarch64-a3-800t-8 - test_type: 'perf' - test_suite: nightly-perf-8-npu-a3 - is_nightly_pipeline_job: true - skip_pr_test_health_check: 'true' - image: ${{ needs.set-image-config.outputs.image_a3 }} - install_sglang_deps: false - device_type_for_deps: 'a3' - run_start_metadata: ${{ needs.set-image-config.outputs.run_start_metadata }} - nightly-perf-16-npu-a3: name: nightly-perf-16-npu-a3 if: ${{ !cancelled() }} @@ -298,15 +266,15 @@ jobs: device_type_for_deps: 'a3' run_start_metadata: ${{ needs.set-image-config.outputs.run_start_metadata }} - nightly-acc-4-npu-a3: - name: nightly-acc-4-npu-a3 + nightly-acc-16-npu-a3: + name: nightly-acc-16-npu-a3 if: ${{ !cancelled() }} needs: [set-image-config] uses: ./.github/workflows/_npu-single-node-test-stage.yml with: - runner: linux-aarch64-a3-4- + runner: linux-aarch64-a3-16- test_type: 'accuracy' - test_suite: nightly-acc-4-npu-a3 + test_suite: nightly-acc-16-npu-a3 is_nightly_pipeline_job: true skip_pr_test_health_check: 'true' image: ${{ needs.set-image-config.outputs.image_a3 }} @@ -314,61 +282,46 @@ jobs: device_type_for_deps: 'a3' run_start_metadata: ${{ needs.set-image-config.outputs.run_start_metadata }} - nightly-acc-16-npu-a3: - name: nightly-acc-16-npu-a3 + nightly-acc-4-npu-a5: + name: nightly-acc-4-npu-a5 if: ${{ !cancelled() }} needs: [set-image-config] uses: ./.github/workflows/_npu-single-node-test-stage.yml with: - runner: linux-aarch64-a3-16- + runner: linux-amd64-a5-4 test_type: 'accuracy' - test_suite: nightly-acc-16-npu-a3 + test_suite: nightly-acc-4-npu-a5 is_nightly_pipeline_job: true skip_pr_test_health_check: 'true' - image: ${{ needs.set-image-config.outputs.image_a3 }} + image: ${{ needs.set-image-config.outputs.image_a5 }} install_sglang_deps: false - device_type_for_deps: 'a3' + device_type_for_deps: 'a5' run_start_metadata: ${{ needs.set-image-config.outputs.run_start_metadata }} - nightly-poc-multi-node-tests: name: multi-node-poc if: ${{ !cancelled() }} - needs: [set-image-config, nightly-perf-2-npu-a3, nightly-perf-4-npu-a3, nightly-perf-16-npu-a3, nightly-acc-2-npu-a3, nightly-acc-4-npu-a3, nightly-acc-16-npu-a3] + needs: [set-image-config, nightly-perf-2-npu-a3, nightly-perf-4-npu-a3, nightly-perf-16-npu-a3, nightly-acc-2-npu-a3, nightly-acc-16-npu-a3] strategy: fail-fast: false max-parallel: 1 matrix: test_config: # glm5_1 performance tests - - name: glm5_1_w4a8_1p1d_32p_in64k_out1k_50ms - prefill_size: 2 - decode_size: 2 - router_size: 1 - test_case: test/registered/npu/performance/glm5_1/test_npu_glm5_1_w4a8_1p1d_32p_in64k_out1k_50ms.py - test_type: 'perf' - prefill_decode_deployment: 'separation' - - name: glm5_1_w4a8_1p1d_32p_in64k_out1k_50ms_aime26 - prefill_size: 2 - decode_size: 2 - router_size: 1 - test_case: test/registered/npu/accuracy/glm5_1/test_npu_glm5_1_w4a8_1p1d_32p_in64k_out1k_50ms_aime26.py - test_type: 'accuracy' - prefill_decode_deployment: 'separation' - # mimo_v2_flash performance tests - - name: test_npu_mimo_v2_flash_1p1d_12p_in16k_out1_ttft_5s - prefill_size: 1 - decode_size: 1 - router_size: 1 - test_case: test/registered/npu/performance/mimo_v2_flash/test_npu_mimo_v2_flash_1p1d_12p_in16k_out1_ttft_5s.py - test_type: 'perf' - prefill_decode_deployment: 'separation' - - name: test_npu_mimo_v2_flash_1p1d_12p_in16k_out1k_tpot_20ms - prefill_size: 1 - decode_size: 1 - router_size: 1 - test_case: test/registered/npu/performance/mimo_v2_flash/test_npu_mimo_v2_flash_1p1d_12p_in16k_out1k_tpot_20ms.py - test_type: 'perf' - prefill_decode_deployment: 'separation' + # Disable test case temporarily due to feature refactor in PR #38293. + # - name: glm5_1_w4a8_1p1d_32p_in64k_out1k_50ms + # prefill_size: 2 + # decode_size: 2 + # router_size: 1 + # test_case: test/registered/npu/performance/glm5_1/test_npu_glm5_1_w4a8_1p1d_32p_in64k_out1k_50ms.py + # test_type: 'perf' + # prefill_decode_deployment: 'separation' + # - name: glm5_1_w4a8_1p1d_32p_in64k_out1k_50ms_aime26 + # prefill_size: 2 + # decode_size: 2 + # router_size: 1 + # test_case: test/registered/npu/accuracy/glm5_1/test_npu_glm5_1_w4a8_1p1d_32p_in64k_out1k_50ms_aime26.py + # test_type: 'accuracy' + # prefill_decode_deployment: 'separation' # deepseek_v4_flash performance tests - name: deepseek_v4_flash_w8a8_1p1d_16p_in8k_out1k_50ms prefill_size: 1 @@ -395,25 +348,21 @@ jobs: nightly-poc-multi-node-mix-tests: name: multi-node-mix-poc if: ${{ !cancelled() }} - needs: [set-image-config, nightly-perf-2-npu-a3, nightly-perf-4-npu-a3, nightly-perf-16-npu-a3, nightly-acc-2-npu-a3, nightly-acc-4-npu-a3, nightly-acc-16-npu-a3, nightly-poc-multi-node-tests] + needs: [set-image-config, nightly-perf-2-npu-a3, nightly-perf-4-npu-a3, nightly-perf-16-npu-a3, nightly-acc-2-npu-a3, nightly-acc-16-npu-a3, nightly-poc-multi-node-tests] strategy: fail-fast: false max-parallel: 1 matrix: test_config: - # kimi_k2_6 performance tests - - name: kimi_k2_6_w4a8_16p_in64k_out1k_100ms - node_size: 2 - test_case: test/registered/npu/performance/kimi_k2_6/test_npu_kimi_k2_6_w4a8_16p_in64k_out1k_100ms.py - test_type: 'perf' - - name: kimi_k2_6_w4a8_16p_in64k_out1k_100ms_aime25 - node_size: 2 - test_case: test/registered/npu/accuracy/kimi_k2_6/test_npu_kimi_k2_6_w4a8_16p_in64k_out1k_100ms_aime25.py - test_type: 'accuracy' # glm_5_2 accuracy tests - - name: glm_5_2_w4a8_16p_gpqa + - name: glm_5_2_w8a8_16p_gpqa node_size: 2 - test_case: test/registered/npu/accuracy/glm5_2/test_npu_glm_5_2_w4a8_16p_gpqa.py + test_case: test/registered/npu/accuracy/glm5_2/test_npu_glm_5_2_w8a8_16p_gpqa.py + test_type: 'accuracy' + # kimi_k3 accuracy tests + - name: kimi_k3_w4a8_32p_gpqa + node_size: 4 + test_case: test/registered/npu/accuracy/kimi_k3/test_npu_kimi_k3_w4a8_32p_gpqa.py test_type: 'accuracy' uses: ./.github/workflows/nightly-test-npu-e2e-multi-node.yml with: @@ -431,7 +380,6 @@ jobs: check-all-jobs: if: ${{ !cancelled() }} needs: - - nightly-1-npu-a2 - nightly-1-npu-a3 - nightly-2-npu-a3 - nightly-4-npu-a3 @@ -439,25 +387,16 @@ jobs: - nightly-16-npu-a3 - nightly-perf-2-npu-a3 - nightly-perf-4-npu-a3 - - nightly-perf-8-npu-a3 - nightly-perf-16-npu-a3 - nightly-acc-2-npu-a3 - - nightly-acc-4-npu-a3 - nightly-acc-16-npu-a3 + - nightly-acc-4-npu-a5 - nightly-poc-multi-node-tests - nightly-poc-multi-node-mix-tests runs-on: ubuntu-latest steps: - - name: Download all metrics - uses: actions/download-artifact@v4 - with: - pattern: metrics-* - path: /tmp/metrics - merge-multiple: false - - name: Generate results table run: | - single_result_a2="${{ needs.nightly-1-npu-a2.result }}" multi_result="${{ needs.nightly-poc-multi-node-tests.result }}" mix_result="${{ needs.nightly-poc-multi-node-mix-tests.result }}" @@ -471,11 +410,10 @@ jobs: "${{ needs.nightly-16-npu-a3.result }}" \ "${{ needs.nightly-perf-2-npu-a3.result }}" \ "${{ needs.nightly-perf-4-npu-a3.result }}" \ - "${{ needs.nightly-perf-8-npu-a3.result }}" \ "${{ needs.nightly-perf-16-npu-a3.result }}" \ "${{ needs.nightly-acc-2-npu-a3.result }}" \ - "${{ needs.nightly-acc-4-npu-a3.result }}" \ - "${{ needs.nightly-acc-16-npu-a3.result }}"; do + "${{ needs.nightly-acc-16-npu-a3.result }}" \ + "${{ needs.nightly-acc-4-npu-a5.result }}"; do if [ "${r}" != "success" ] && [ "${r}" != "skipped" ]; then single_result="failure" fi @@ -495,7 +433,6 @@ jobs: echo "" >> $GITHUB_STEP_SUMMARY echo "| Group | Status |" >> $GITHUB_STEP_SUMMARY echo "|-------|--------|" >> $GITHUB_STEP_SUMMARY - echo "| nightly-1-npu-a2 | $(group_icon ${single_result_a2}) ${single_result_a2} |" >> $GITHUB_STEP_SUMMARY for entry in \ "nightly-1-npu-a3:${{ needs.nightly-1-npu-a3.result }}" \ "nightly-2-npu-a3:${{ needs.nightly-2-npu-a3.result }}" \ @@ -504,11 +441,10 @@ jobs: "nightly-16-npu-a3:${{ needs.nightly-16-npu-a3.result }}" \ "nightly-perf-2-npu-a3:${{ needs.nightly-perf-2-npu-a3.result }}" \ "nightly-perf-4-npu-a3:${{ needs.nightly-perf-4-npu-a3.result }}" \ - "nightly-perf-8-npu-a3:${{ needs.nightly-perf-8-npu-a3.result }}" \ "nightly-perf-16-npu-a3:${{ needs.nightly-perf-16-npu-a3.result }}" \ "nightly-acc-2-npu-a3:${{ needs.nightly-acc-2-npu-a3.result }}" \ - "nightly-acc-4-npu-a3:${{ needs.nightly-acc-4-npu-a3.result }}" \ - "nightly-acc-16-npu-a3:${{ needs.nightly-acc-16-npu-a3.result }}"; do + "nightly-acc-16-npu-a3:${{ needs.nightly-acc-16-npu-a3.result }}" \ + "nightly-acc-4-npu-a5:${{ needs.nightly-acc-4-npu-a5.result }}"; do suite="${entry%%:*}" r="${entry##*:}" echo "| ${suite} | $(group_icon ${r}) ${r} |" >> $GITHUB_STEP_SUMMARY @@ -517,44 +453,8 @@ jobs: echo "| multi-node-mix-poc | $(group_icon ${mix_result}) ${mix_result} |" >> $GITHUB_STEP_SUMMARY echo "" >> $GITHUB_STEP_SUMMARY - echo "## Per-Test Metrics" >> $GITHUB_STEP_SUMMARY - echo "" >> $GITHUB_STEP_SUMMARY - - has_metrics=false - # Each suite job uploads its METRICS_DATA_FILE directory, which - # contains per-case subdirectories with their own metrics.json. - # Recursively find every metrics.json and render one table row per - # test case. - for file in $(find /tmp/metrics -name metrics.json 2>/dev/null); do - if [ -f "$file" ]; then - has_metrics=true - echo "import json" > /tmp/parse_metrics.py - echo "with open('$file') as f:" >> /tmp/parse_metrics.py - echo " d = json.load(f)" >> /tmp/parse_metrics.py - echo "tc = d.get('test_case', '-')" >> /tmp/parse_metrics.py - echo "tp = d.get('test_type', '-')" >> /tmp/parse_metrics.py - echo "st = d.get('status', 'unknown')" >> /tmp/parse_metrics.py - echo "icon = '✅' if st == 'pass' else '❌'" >> /tmp/parse_metrics.py - echo "metrics = d.get('metrics', {})" >> /tmp/parse_metrics.py - echo "baselines = d.get('baselines', {})" >> /tmp/parse_metrics.py - echo "mstr = ', '.join(f'{k}={v}' for k,v in metrics.items()) if metrics else '-'" >> /tmp/parse_metrics.py - echo "bstr = ', '.join(f'{k}={v}' for k,v in baselines.items()) if baselines else '-'" >> /tmp/parse_metrics.py - echo "print(f'| {tc} | {tp} | {icon} {st} | {mstr} | {bstr} |')" >> /tmp/parse_metrics.py - python3 /tmp/parse_metrics.py - fi - done > /tmp/metrics_table.txt - - if [ "${has_metrics}" = "true" ]; then - echo "| Test Case | Type | Status | Metrics | Baseline |" >> $GITHUB_STEP_SUMMARY - echo "|-----------|------|--------|---------|----------|" >> $GITHUB_STEP_SUMMARY - cat /tmp/metrics_table.txt >> $GITHUB_STEP_SUMMARY - else - echo "No per-test metrics available (artifacts not found)." >> $GITHUB_STEP_SUMMARY - fi - echo "" >> $GITHUB_STEP_SUMMARY - FAIL=0 - for r in "${single_result_a2}" "${single_result}" "${multi_result}" "${mix_result}"; do + for r in "${single_result}" "${multi_result}" "${mix_result}"; do if [ "${r}" != "success" ] && [ "${r}" != "skipped" ]; then FAIL=1; fi done exit $FAIL diff --git a/.github/workflows/nightly-test-nvidia.yml b/.github/workflows/nightly-test-nvidia.yml index d8eeff7fe93d..a3e4e2c5294e 100644 --- a/.github/workflows/nightly-test-nvidia.yml +++ b/.github/workflows/nightly-test-nvidia.yml @@ -152,8 +152,6 @@ jobs: run_timeout_minutes: '360' job_timeout_minutes: '420' scheduled: true - # aarch64: the rust-ext cache key is x86_64-only, same reason base-c skips it. - skip_prebuilt_rust_ext: true secrets: inherit nightly-8-gpu-h200: diff --git a/.github/workflows/patch-docker-dev.yml b/.github/workflows/patch-docker-dev.yml index d46d21e176fc..7ecd5b674134 100644 --- a/.github/workflows/patch-docker-dev.yml +++ b/.github/workflows/patch-docker-dev.yml @@ -8,7 +8,7 @@ on: required: false default: "" image_tag: - description: "Base image tag to patch (e.g. dev, dev-cu13, dev-cu12)" + description: "Base image tag to patch (e.g. dev, dev-cu13)" required: true output_tag: description: "Tag to publish as. Overwrites it if it already exists." diff --git a/.github/workflows/pr-gate.yml b/.github/workflows/pr-gate.yml index 53241f6c337f..cebc98bdcbee 100644 --- a/.github/workflows/pr-gate.yml +++ b/.github/workflows/pr-gate.yml @@ -9,6 +9,10 @@ on: description: "Whether the PR must also have the run-ci-extra label (in addition to run-ci)" type: boolean default: false + require-label: + description: "Additional label the PR must carry besides run-ci (empty = disabled)" + type: string + default: '' cool-down-minutes: description: "Cooldown period in minutes for low-permission users; 0 disables rate limiting" type: number @@ -18,10 +22,15 @@ jobs: pr-gate: # 1. for commits on main: no gating needed # 2. for workflow_dispatch: this can only be triggered by users with write access + # 3. for pull_request and pull_request_target: apply the checks below. + # This job only reads PR metadata through the API and never checks out + # or executes PR code, so it is safe for pull_request_target callers. runs-on: ubuntu-latest + env: + IS_PR_EVENT: ${{ github.event_name == 'pull_request' || github.event_name == 'pull_request_target' }} steps: - name: Fetch latest PR info - if: github.event_name == 'pull_request' + if: env.IS_PR_EVENT == 'true' id: pr uses: actions/github-script@v8 with: @@ -37,7 +46,7 @@ jobs: core.setOutput("user", pr.data.user.login); - name: Log PR info - if: github.event_name == 'pull_request' + if: env.IS_PR_EVENT == 'true' run: | echo "===== PR Info =====" echo "PR Event: ${{ github.event_name }}" @@ -46,17 +55,18 @@ jobs: echo "PR User: ${{ steps.pr.outputs.user }}" echo "Require run-ci: ${{ inputs.require-run-ci }}" echo "Require run-ci-extra: ${{ inputs.require-run-ci-extra }}" + echo "Require label: ${{ inputs.require-label }}" echo "Cool down minutes: ${{ inputs.cool-down-minutes }}" echo "===================" - name: Block draft PR - if: github.event_name == 'pull_request' && fromJson(steps.pr.outputs.draft) + if: env.IS_PR_EVENT == 'true' && fromJson(steps.pr.outputs.draft) run: | echo "PR is draft. Blocking CI." exit 1 - name: Require run-ci label (optional) - if: github.event_name == 'pull_request' && inputs.require-run-ci == true + if: env.IS_PR_EVENT == 'true' && inputs.require-run-ci == true run: | labels='${{ steps.pr.outputs.labels }}' if [[ "${{ contains(fromJson(steps.pr.outputs.labels), 'run-ci') }}" == "false" ]]; then @@ -70,15 +80,28 @@ jobs: # then reruns) could never un-skip. A runtime step using gh-api-fetched # labels lets reruns pick up the new label set. - name: Require run-ci-extra label (optional) - if: github.event_name == 'pull_request' && inputs.require-run-ci-extra == true + if: env.IS_PR_EVENT == 'true' && inputs.require-run-ci-extra == true run: | if [[ "${{ contains(fromJson(steps.pr.outputs.labels), 'run-ci-extra') }}" == "false" ]]; then echo "Missing required label 'run-ci-extra'. Add the label (e.g. via /tag-and-rerun-ci extra) to opt this PR into the extra test workflow." exit 1 fi + # Generic opt-in label gate (same live-fetch semantics as run-ci-extra). + # Lets a workflow require its own label on top of run-ci without adding + # a dedicated input per label. + - name: Require additional label (optional) + if: env.IS_PR_EVENT == 'true' && inputs.require-label != '' + env: + REQUIRED_LABEL: ${{ inputs.require-label }} + run: | + if [[ "${{ contains(fromJson(steps.pr.outputs.labels), inputs.require-label) }}" == "false" ]]; then + echo "Missing required label '${REQUIRED_LABEL}' (in addition to 'run-ci'). Add the label to opt this PR into this workflow." + exit 1 + fi + - name: Enforce rate limit for low-permission actors (optional) - if: github.event_name == 'pull_request' && inputs.cool-down-minutes > 0 + if: env.IS_PR_EVENT == 'true' && inputs.cool-down-minutes > 0 uses: actions/github-script@v8 with: github-token: ${{ secrets.GITHUB_TOKEN }} @@ -87,6 +110,11 @@ jobs: const owner = context.repo.owner; const repo = context.repo.repo; const eventName = context.eventName; + // pr-test.yml runs on pull_request, so pull_request_target callers must + // query that event to share the same rate-limit history. + const rateLimitEventName = eventName === "pull_request_target" + ? "pull_request" + : eventName; const curRun = await github.rest.actions.getWorkflowRun({ owner, repo, run_id: context.runId }); @@ -161,14 +189,14 @@ jobs: const cutoff = new Date(Date.now() - effectiveCooldownMinutes * 60 * 1000); core.info( - `Checking for workflow runs since ${cutoff.toISOString()} (last ${effectiveCooldownMinutes} minutes) for event '${eventName}'.` + `Checking for workflow runs since ${cutoff.toISOString()} (last ${effectiveCooldownMinutes} minutes) for event '${rateLimitEventName}'.` ); const { data } = await github.rest.actions.listWorkflowRuns({ owner, repo, workflow_id: 'pr-test.yml', - event: eventName, + event: rateLimitEventName, per_page: 100, }); diff --git a/.github/workflows/pr-states.yml b/.github/workflows/pr-states.yml index 3a6da1eff9be..9a053d000227 100644 --- a/.github/workflows/pr-states.yml +++ b/.github/workflows/pr-states.yml @@ -14,7 +14,7 @@ on: pull_request_target: types: [opened, synchronize, reopened, labeled, unlabeled] workflow_run: - workflows: ["PR Test Base", "PR Test Extra", "PR Test ROCm 7.2 (AMD)"] + workflows: ["PR Test Base", "PR Test Extra", "PR Test (AMD)"] types: [requested, completed] workflow_dispatch: inputs: @@ -163,7 +163,7 @@ jobs: 'pr-test-extra.yml', (hasCI && hasExtra) ? labelOnOpts : labelOffOpts, ), - findRun('pr-test-amd-rocm720.yml', labelOffOpts), + findRun('pr-test-amd.yml', labelOffOpts), ]); // Skipped run = "no real run" -- happens when a label is added @@ -214,8 +214,8 @@ jobs: const ptEnd = ''; const peStart = ''; const peEnd = ''; - const amdStart = ''; - const amdEnd = ''; + const amdStart = ''; + const amdEnd = ''; const newBlock = [ outerStart, @@ -224,7 +224,7 @@ jobs: '', `Latest PR Test (Base): ${ptStart}${ptText}${ptEnd}`, `Latest PR Test (Extra): ${peStart}${peText}${peEnd}`, - `Latest PR Test (AMD ROCm 7.2): ${amdStart}${amdText}${amdEnd}`, + `Latest PR Test (AMD ROCm 10): ${amdStart}${amdText}${amdEnd}`, outerEnd, ].join('\n'); diff --git a/.github/workflows/pr-test-amd-extra.yml b/.github/workflows/pr-test-amd-extra.yml index 931aa5f0a7d1..2096b61145f8 100644 --- a/.github/workflows/pr-test-amd-extra.yml +++ b/.github/workflows/pr-test-amd-extra.yml @@ -37,15 +37,14 @@ on: - mi300 - mi325 rocm_version: - description: 'ROCm container variant (ROCm 7.2.4 by default)' + description: 'ROCm container variant (ROCm 10 by default)' required: false type: choice - default: rocm724 + default: rocm10 options: - rocm10 - rocm724 - rocm720 - - rocm700 aiter_ref: description: 'Override AITER commit (optional, leave empty to use Dockerfile default)' required: false @@ -74,10 +73,10 @@ on: type: string default: mi300 rocm_version: - description: 'ROCm container variant (ROCm 7.2.4 by default)' + description: 'ROCm container variant (ROCm 10 by default)' required: false type: string - default: rocm724 + default: rocm10 aiter_ref: description: 'Override AITER commit (optional, leave empty to use Dockerfile default)' required: false @@ -101,11 +100,11 @@ env: DOCKERHUB_AMD_TOKEN: ${{ secrets.DOCKERHUB_AMD_TOKEN }} concurrency: - # Mirror pr-test-amd.yml: schedule/dispatch runs key on run_id (unique group, - # never cancel each other); PR runs share a per-branch group so pushes cancel - # stale runs. (In a reusable workflow github.event_name is the originating - # event, never workflow_call, so the guard keys off the real events.) - group: pr-test-amd-extra-${{ (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') && format('full-{0}', github.run_id) || github.head_ref || github.ref_name || inputs.ref || 'default' }} + # Schedule/dispatch runs key on run_id plus ROCm version so the three + # scheduled reusable-workflow matrix calls cannot replace each other's + # pending runs. PR runs still share a per-branch group so pushes cancel stale + # runs. In a reusable workflow github.event_name is the originating event. + group: pr-test-amd-extra-${{ (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') && format('full-{0}-{1}', github.run_id, inputs.rocm_version || 'rocm10') || github.head_ref || github.ref_name || inputs.ref || 'default' }} cancel-in-progress: ${{ github.event_name == 'pull_request' }} permissions: @@ -143,7 +142,7 @@ jobs: # setup across scarce AMD GPUs to shave only a couple minutes of test time, # so one GPU running the whole suite sequentially is the better trade. extra-a-test-1-gpu-small-amd: - name: ${{ format('extra-a-test-1-gpu-small-amd ({0}, linux-{1}-1gpu-sglang)', inputs.rocm_version || 'rocm724', inputs.runner_arch || 'mi300') }} + name: ${{ format('extra-a-test-1-gpu-small-amd ({0}, linux-{1}-1gpu-sglang)', inputs.rocm_version || 'rocm10', inputs.runner_arch || 'mi300') }} needs: [call-gate] if: ${{ !cancelled() && needs.call-gate.result == 'success' }} runs-on: ${{ format('linux-{0}-1gpu-sglang', inputs.runner_arch || 'mi300') }} @@ -158,8 +157,8 @@ jobs: - name: Start CI container # `inputs` is empty on pull_request events, so fall back explicitly to - # the ROCm 7.2 PR default. Reusable callers can request another variant. - run: bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ inputs.rocm_version || 'rocm724' }} + # the ROCm 10 PR default. Reusable callers can request another variant. + run: bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ inputs.rocm_version || 'rocm10' }} env: GITHUB_WORKSPACE: ${{ github.workspace }} @@ -176,7 +175,7 @@ jobs: # pool as small (AMD GPUs carry enough VRAM that "large" here is a CUDA # memory-tier label, not a separate AMD runner pool). extra-a-test-1-gpu-large-amd: - name: ${{ format('extra-a-test-1-gpu-large-amd ({0}, linux-{1}-1gpu-sglang)', inputs.rocm_version || 'rocm724', inputs.runner_arch || 'mi300') }} + name: ${{ format('extra-a-test-1-gpu-large-amd ({0}, linux-{1}-1gpu-sglang)', inputs.rocm_version || 'rocm10', inputs.runner_arch || 'mi300') }} needs: [call-gate] if: ${{ !cancelled() && needs.call-gate.result == 'success' }} runs-on: ${{ format('linux-{0}-1gpu-sglang', inputs.runner_arch || 'mi300') }} @@ -190,7 +189,7 @@ jobs: run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - name: Start CI container - run: bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ inputs.rocm_version || 'rocm724' }} + run: bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ inputs.rocm_version || 'rocm10' }} env: GITHUB_WORKSPACE: ${{ github.workspace }} @@ -206,7 +205,7 @@ jobs: # Multi-GPU TP / PP / PD mock-model + kv_canary e2e tests. Mirrors CUDA's # extra-a 2-gpu-large stage; runs on the 2-GPU AMD pool. extra-a-test-2-gpu-large-amd: - name: ${{ format('extra-a-test-2-gpu-large-amd ({0}, linux-{1}-2gpu-sglang)', inputs.rocm_version || 'rocm724', inputs.runner_arch || 'mi300') }} + name: ${{ format('extra-a-test-2-gpu-large-amd ({0}, linux-{1}-2gpu-sglang)', inputs.rocm_version || 'rocm10', inputs.runner_arch || 'mi300') }} needs: [call-gate] if: ${{ !cancelled() && needs.call-gate.result == 'success' }} runs-on: ${{ format('linux-{0}-2gpu-sglang', inputs.runner_arch || 'mi300') }} @@ -220,7 +219,7 @@ jobs: run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - name: Start CI container - run: bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ inputs.rocm_version || 'rocm724' }} + run: bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version ${{ inputs.rocm_version || 'rocm10' }} env: GITHUB_WORKSPACE: ${{ github.workspace }} @@ -244,7 +243,14 @@ jobs: extra-a-test-1-gpu-large-amd, extra-a-test-2-gpu-large-amd, ] - if: always() + # Same `labeled` guard as call-gate: an unrelated label would otherwise finish + # green with nothing executed, over the real run's result. + if: | + always() && + (github.event_name != 'pull_request' || + github.event.action != 'labeled' || + github.event.label.name == 'run-ci' || + github.event.label.name == 'run-ci-extra') runs-on: ubuntu-latest steps: - name: Check all dependent job statuses diff --git a/.github/workflows/pr-test-amd-mori.yml b/.github/workflows/pr-test-amd-mori.yml index fde156b6e7d7..e43d4ee9a1c9 100644 --- a/.github/workflows/pr-test-amd-mori.yml +++ b/.github/workflows/pr-test-amd-mori.yml @@ -53,7 +53,7 @@ jobs: fi - name: Start disaggregation container - run: bash scripts/ci/amd/amd_ci_start_container_disagg.sh --rocm-version rocm724 + run: bash scripts/ci/amd/amd_ci_start_container_disagg.sh --rocm-version rocm10 env: GITHUB_WORKSPACE: ${{ github.workspace }} @@ -84,7 +84,7 @@ jobs: run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - name: Start CI container - run: bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version rocm724 + run: bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version rocm10 env: GITHUB_WORKSPACE: ${{ github.workspace }} ENABLE_CACHE_HOST: "1" diff --git a/.github/workflows/pr-test-amd-rocm720.yml b/.github/workflows/pr-test-amd-rocm720.yml deleted file mode 100644 index 367b9170f53a..000000000000 --- a/.github/workflows/pr-test-amd-rocm720.yml +++ /dev/null @@ -1,1460 +0,0 @@ -name: PR Test ROCm 7.2 (AMD) -# Dynamic run-name for manual stage dispatches to enable URL lookup -# Format: "[stage-name] sha" for fork PRs, "[stage-name]" for non-fork, default for normal runs -run-name: ${{ (inputs.target_stage || inputs.target_stage_select) && (inputs.pr_head_sha && format('[{0}] {1}', inputs.target_stage || inputs.target_stage_select, inputs.pr_head_sha) || format('[{0}]', inputs.target_stage || inputs.target_stage_select)) || '' }} - -on: - schedule: - - cron: '0 */12 * * *' # Run every 12 hours (UTC) - pull_request: - paths: - - "python/**" - - "scripts/ci/**" - - "test/**" - - "python/sglang/kernels/aot/**" - - ".github/workflows/pr-test-amd-rocm720.yml" - - "docker/rocm.Dockerfile" - workflow_dispatch: - inputs: - rocm_version: - description: 'ROCm image version' - required: false - type: choice - default: rocm724 - options: - - rocm10 - - rocm724 - - rocm720 - target_stage_select: - description: "Select a stage to run from dropdown (leave empty for auto-detect)" - required: false - type: choice - default: '' - options: - - '' - - sgl-kernel-unit-test-amd-rocm720 - - sgl-kernel-unit-test-2-gpu-amd-rocm720 - - stage-a-test-1-gpu-small-amd-rocm720 - - jit-kernel-unit-test-amd-rocm720 - - jit-kernel-benchmark-test-amd-rocm720 - - stage-b-test-1-gpu-small-amd-rocm720 - - stage-b-test-1-gpu-small-amd-nondeterministic-rocm720 - - stage-b-test-1-gpu-small-amd-mi35x-rocm720 - - stage-b-test-1-gpu-large-amd-rocm720 - - stage-b-test-2-gpu-large-amd-rocm720 - - multimodal-gen-test-1-gpu-amd-rocm720 - - multimodal-gen-test-2-gpu-amd-rocm720 - - stage-c-test-4-gpu-amd-rocm720 - - stage-c-dsv4-flash-fp4-fp8-amd-mi35x-rocm720 - - stage-c-dsv4-pro-fp4-amd-mi35x-rocm720 - - stage-c-test-large-8-gpu-amd-rocm720 - - stage-c-test-large-8-gpu-amd-mi35x-rocm720 - - stage-b-test-large-8-gpu-mi35x-disaggregation-amd-rocm720 - target_stage: - description: "Or type comma-separated stage names (overrides dropdown if non-empty)" - required: false - type: string - default: "" - pr_head_sha: - description: "PR head SHA to checkout (for stage dispatches on fork PRs)" - required: false - type: string - default: "" - aiter_ref: - description: 'Override AITER commit (optional, leave empty to use Dockerfile default)' - required: false - type: string - default: '' - amd_ci_image: - description: 'Override AMD CI Docker image (optional, e.g. rocm/sgl-dev:tag)' - required: false - type: string - default: '' - continue_on_error: - description: 'Continue on error (do not fail the workflow on test failures)' - required: false - type: boolean - default: false - runner_arch: - description: 'AMD runner pool to dispatch GPU jobs to' - required: false - type: choice - default: mi300 - options: - - mi300 - - mi325 - run_all_tests: - description: 'Run all tests (skip change detection). Ignored when target_stage / target_stage_select is set.' - required: false - type: boolean - default: false - workflow_call: - inputs: - rocm_version: - description: 'ROCm image version' - required: false - type: string - default: rocm724 - ref: - description: 'Git ref (branch, tag, or SHA) to test. If not provided, uses the default branch.' - required: false - type: string - default: '' - runner_arch: - description: 'AMD runner pool to dispatch GPU jobs to' - required: false - type: string - default: mi300 - run_all_tests: - description: "Run all tests (for releasing or testing purpose)" - required: false - type: boolean - default: false - aiter_ref: - description: 'Override AITER commit (optional, leave empty to use Dockerfile default)' - required: false - type: string - default: '' - amd_ci_image: - description: 'Override AMD CI Docker image (optional, e.g. rocm/sgl-dev:tag)' - required: false - type: string - default: '' - continue_on_error: - description: 'Continue on error (do not fail the workflow on test failures)' - required: false - type: boolean - default: false - -# Mirror pr-test.yml: the chained extra suite (call-pr-test-amd-extra-rocm720 -> -# pr-test-amd-extra.yml) declares actions: write / issues: read / -# pull-requests: read. A called reusable workflow can only use scopes the -# caller already holds, so without this block the call fails workflow -# validation ("requesting actions: write... but only allowed ...none"). -permissions: - actions: write - contents: read - issues: read - pull-requests: read - -env: - AITER_COMMIT_OVERRIDE: ${{ inputs.aiter_ref }} - AMD_CI_IMAGE: ${{ inputs.amd_ci_image }} - ROCM_VERSION: ${{ inputs.rocm_version || 'rocm724' }} - DOCKERHUB_AMD_USERNAME: ${{ secrets.DOCKERHUB_AMD_USERNAME }} - DOCKERHUB_AMD_TOKEN: ${{ secrets.DOCKERHUB_AMD_TOKEN }} - -concurrency: - # Scheduled, run_all_tests, and manual dispatch runs get unique groups (never cancel each other). - # PR runs share a group per branch so new pushes cancel stale runs. - group: pr-test-amd-rocm720-${{ (inputs.run_all_tests || github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') && format('full-{0}', github.run_id) || inputs.pr_head_sha || inputs.ref || github.ref }} - cancel-in-progress: ${{ !inputs.run_all_tests && github.event_name != 'workflow_call' && github.event_name != 'schedule' && github.event_name != 'workflow_dispatch' }} - -jobs: - call-gate: - if: github.event_name != 'schedule' - uses: ./.github/workflows/pr-gate.yml - secrets: inherit - check-changes: - needs: [call-gate] - if: always() - runs-on: ubuntu-latest - outputs: - main_package: ${{ steps.filter.outputs.main_package || steps.run-mode.outputs.run_all_tests }} - sgl_kernel: ${{ steps.filter.outputs.sgl_kernel || steps.run-mode.outputs.run_all_tests }} - jit_kernel: ${{ steps.filter.outputs.jit_kernel || steps.run-mode.outputs.run_all_tests }} - multimodal_gen: ${{ steps.filter.outputs.multimodal_gen || steps.run-mode.outputs.run_all_tests }} - continue_on_error: ${{ steps.set-continue-on-error.outputs.continue_on_error }} - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.pr_head_sha || inputs.ref || github.sha }} - - - name: Determine run mode - id: run-mode - run: | - if [[ "${{ inputs.run_all_tests }}" == "true" || "${{ github.event_name }}" == "schedule" ]]; then - echo "run_all_tests=true" >> $GITHUB_OUTPUT - echo "Run mode: ALL TESTS (run_all_tests=${{ inputs.run_all_tests }}, event=${{ github.event_name }})" - else - echo "run_all_tests=false" >> $GITHUB_OUTPUT - echo "Run mode: FILTERED (triggered by ${{ github.event_name }})" - fi - - - name: Set continue-on-error for schedule/full runs - id: set-continue-on-error - env: - # `bypass-fastfail` PR label: also disable within-suite fast-fail - # here. The shared actions/wait-for-jobs already honors the same - # label to skip cross-stage waits. - BYPASS_FASTFAIL_LABEL: ${{ contains(github.event.pull_request.labels.*.name, 'bypass-fastfail') }} - run: | - if [[ "${{ steps.run-mode.outputs.run_all_tests }}" == "true" \ - || "${{ inputs.continue_on_error }}" == "true" \ - || "$BYPASS_FASTFAIL_LABEL" == "true" ]]; then - echo "continue_on_error=true" >> $GITHUB_OUTPUT - echo "Continue-on-error: ENABLED (run_all_tests=${{ steps.run-mode.outputs.run_all_tests }}, input=${{ inputs.continue_on_error }}, bypass-fastfail=$BYPASS_FASTFAIL_LABEL)" - else - echo "continue_on_error=false" >> $GITHUB_OUTPUT - echo "Continue-on-error: DISABLED" - fi - - - name: Detect file changes - id: filter - uses: dorny/paths-filter@v3 - if: steps.run-mode.outputs.run_all_tests != 'true' - with: - filters: | - main_package: - - "python/sglang/!(multimodal_gen|kernels)/**/!(*.md)" - - "python/sglang/kernels/!(*.md)" - - "python/sglang/kernels/!(aot)/**/!(*.md)" - - "python/pyproject_rocm.toml" - - "python/pyproject_other.toml" - - "scripts/ci/amd/*" - - "scripts/ci/utils/*" - - "test/**/!(*.md)" - - ".github/workflows/pr-test-amd-rocm720.yml" - sgl_kernel: - - "python/sglang/kernels/aot/**/!(*.md|THIRDPARTYNOTICES.txt|LICENSE)" - - ".github/workflows/pr-test-amd-rocm720.yml" - jit_kernel: - - "python/sglang/kernels/!(*.md)" - - "python/sglang/kernels/!(aot)/**" - - "test/registered/kernels/**" - - ".github/workflows/pr-test-amd-rocm720.yml" - multimodal_gen: - - "python/sglang/multimodal_gen/**/!(*.md|*.ipynb)" - - "python/sglang/cli/**" - - "python/sglang/srt/observability/**" - - "python/sglang/kernels/ops/diffusion/**" - - "test/registered/kernels/ops/diffusion/**" - - "test/registered/kernels/benchmark/diffusion/**" - - "python/pyproject_rocm.toml" - - "python/pyproject_other.toml" - - # =============================================== extra (scheduled) ==================================================== - # Chain the label-gated AMD extra tier into the scheduled run, mirroring - # pr-test.yml's `call-pr-test-extra`. On `schedule` (and run_all_tests - # dispatch) the extra suite runs on `main` without needing the - # `run-ci-extra` label (pr-gate.yml only enforces labels on pull_request - # events). Targeted stage dispatches (target_stage set) are excluded. - # Not added to `pr-test-amd-rocm720-finish` so the base AMD gate never depends on - # the opt-in extra suite. - call-pr-test-amd-extra-rocm720: - name: call-pr-test-amd-extra - if: | - (github.event_name == 'schedule' || inputs.run_all_tests == true) && - !(inputs.target_stage || inputs.target_stage_select) - uses: ./.github/workflows/pr-test-amd-extra.yml - with: - ref: ${{ inputs.pr_head_sha || inputs.ref || '' }} - runner_arch: ${{ inputs.runner_arch || 'mi300' }} - rocm_version: ${{ inputs.rocm_version || 'rocm724' }} - aiter_ref: ${{ inputs.aiter_ref }} - amd_ci_image: ${{ inputs.amd_ci_image }} - continue_on_error: true - secrets: inherit - - # =============================================== sgl-kernel ==================================================== - sgl-kernel-unit-test-amd-rocm720: - name: ${{ format('sgl-kernel-unit-test-amd ({0}, linux-{1}-1gpu-sglang)', inputs.rocm_version || 'rocm724', inputs.runner_arch || 'mi300') }} - needs: [check-changes, call-gate] - if: | - always() && !cancelled() && - ( - (contains(format(',{0},', inputs.target_stage || inputs.target_stage_select), ',sgl-kernel-unit-test-amd-rocm720,')) || - ( - !(inputs.target_stage || inputs.target_stage_select) && - (needs.call-gate.result == 'success' || needs.call-gate.result == 'skipped') && - needs.check-changes.outputs.sgl_kernel == 'true' - ) - ) - runs-on: ${{ format('linux-{0}-1gpu-sglang', inputs.runner_arch || 'mi300') }} - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.pr_head_sha || inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Start CI container - run: bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version "$ROCM_VERSION" - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - - - name: Install dependencies - run: | - bash scripts/ci/amd/amd_ci_install_dependency.sh - - - name: Run test - timeout-minutes: 30 - env: - CONTINUE_ON_ERROR: ${{ needs.check-changes.outputs.continue_on_error }} - run: | - # In continue-on-error mode (schedule/full runs), keep running all pytest - # files and aggregate the exit code. In PR mode, preserve fail-fast. - failures=0 - run_pytest() { - if [[ "$CONTINUE_ON_ERROR" == "true" ]]; then - "$@" || failures=$((failures + 1)) - else - "$@" - fi - } - run_pytest docker exec -w /sglang-checkout/python/sglang/kernels/aot/tests ci_sglang python3 -m pytest test_moe_align.py - run_pytest docker exec -w /sglang-checkout/python/sglang/kernels/aot/tests ci_sglang python3 -m pytest test_moe_topk_softmax.py - run_pytest docker exec -w /sglang-checkout/python/sglang/kernels/aot/tests/speculative ci_sglang python3 -m pytest test_eagle_utils.py - run_pytest docker exec -w /sglang-checkout/python/sglang/kernels/aot/tests ci_sglang python3 -m pytest test_apply_token_bitmask_inplace.py - run_pytest docker exec -w /sglang-checkout/python/sglang/kernels/aot/tests ci_sglang python3 -m pytest test_activation.py - run_pytest docker exec -w /sglang-checkout/python/sglang/kernels/aot/tests ci_sglang python3 -m pytest test_topk.py - run_pytest docker exec -w /sglang-checkout/python/sglang/kernels/aot/tests ci_sglang python3 -m pytest test_kvcacheio.py - run_pytest docker exec -w /sglang-checkout/python/sglang/kernels/aot/tests ci_sglang python3 -m pytest test_moe_topk_sigmoid.py - run_pytest docker exec -w /sglang-checkout/python/sglang/kernels/aot/tests ci_sglang python3 -m pytest test_torch_defaults_reset.py - exit $failures - - sgl-kernel-unit-test-2-gpu-amd-rocm720: - name: ${{ format('sgl-kernel-unit-test-2-gpu-amd ({0}, linux-{1}-2gpu-sglang)', inputs.rocm_version || 'rocm724', inputs.runner_arch || 'mi300') }} - needs: [check-changes, call-gate] - if: | - always() && !cancelled() && - ( - (contains(format(',{0},', inputs.target_stage || inputs.target_stage_select), ',sgl-kernel-unit-test-2-gpu-amd-rocm720,')) || - ( - !(inputs.target_stage || inputs.target_stage_select) && - (needs.call-gate.result == 'success' || needs.call-gate.result == 'skipped') && - needs.check-changes.outputs.sgl_kernel == 'true' - ) - ) - runs-on: ${{ format('linux-{0}-2gpu-sglang', inputs.runner_arch || 'mi300') }} - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.pr_head_sha || inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Start CI container - run: bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version "$ROCM_VERSION" - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - - - name: Install dependencies - run: | - bash scripts/ci/amd/amd_ci_install_dependency.sh - - - name: Run test - timeout-minutes: 30 - env: - CONTINUE_ON_ERROR: ${{ needs.check-changes.outputs.continue_on_error }} - run: | - bash scripts/ci/amd/amd_ci_exec.sh -w "/sglang-checkout/test" python3 run_suite.py --hw amd --suite sgl-kernel-unit-test-2-gpu-amd ${{ needs.check-changes.outputs.continue_on_error == 'true' && '--continue-on-error' || '' }} - - # =============================================== primary ==================================================== - - stage-a-test-1-gpu-small-amd-rocm720: - name: ${{ format('stage-a-test-1-gpu-small-amd ({0}, linux-{1}-1gpu-sglang)', inputs.rocm_version || 'rocm724', inputs.runner_arch || 'mi300') }} - needs: [check-changes, call-gate] - if: | - always() && !cancelled() && - ( - (contains(format(',{0},', inputs.target_stage || inputs.target_stage_select), ',stage-a-test-1-gpu-small-amd-rocm720,')) || - ( - !(inputs.target_stage || inputs.target_stage_select) && - (needs.call-gate.result == 'success' || needs.call-gate.result == 'skipped') && - ((needs.check-changes.outputs.main_package == 'true') || (needs.check-changes.outputs.sgl_kernel == 'true')) - ) - ) - runs-on: ${{ format('linux-{0}-1gpu-sglang', inputs.runner_arch || 'mi300') }} - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.pr_head_sha || inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Start CI container - run: bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version "$ROCM_VERSION" - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - - - name: Install dependencies - run: | - bash scripts/ci/amd/amd_ci_install_dependency.sh - - - name: Run test - timeout-minutes: 30 - run: | - bash scripts/ci/amd/amd_ci_exec.sh -w "/sglang-checkout/test" python3 run_suite.py --hw amd --suite stage-a-test-1-gpu-small-amd ${{ needs.check-changes.outputs.continue_on_error == 'true' && '--continue-on-error' || '' }} - - jit-kernel-unit-test-amd-rocm720: - name: ${{ format('jit-kernel-unit-test-amd ({0}, linux-{1}-1gpu-sglang)', inputs.rocm_version || 'rocm724', inputs.runner_arch || 'mi300') }} - needs: [check-changes, call-gate] - if: | - always() && !cancelled() && - ( - (contains(format(',{0},', inputs.target_stage || inputs.target_stage_select), ',jit-kernel-unit-test-amd-rocm720,')) || - ( - !(inputs.target_stage || inputs.target_stage_select) && - (needs.call-gate.result == 'success' || needs.call-gate.result == 'skipped') && - needs.check-changes.outputs.jit_kernel == 'true' - ) - ) - runs-on: ${{ format('linux-{0}-1gpu-sglang', inputs.runner_arch || 'mi300') }} - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.pr_head_sha || inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Start CI container - run: bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version "$ROCM_VERSION" - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - - - name: Install dependencies - run: | - bash scripts/ci/amd/amd_ci_install_dependency.sh - - - name: Run JIT kernel unit tests - timeout-minutes: 30 - run: | - bash scripts/ci/amd/amd_ci_exec.sh -w "/sglang-checkout/test" python3 run_suite.py --hw amd --suite jit-kernel-unit-test-amd ${{ needs.check-changes.outputs.continue_on_error == 'true' && '--continue-on-error' || '' }} - - jit-kernel-benchmark-test-amd-rocm720: - name: ${{ format('jit-kernel-benchmark-test-amd ({0}, linux-{1}-1gpu-sglang)', inputs.rocm_version || 'rocm724', inputs.runner_arch || 'mi300') }} - needs: [check-changes, call-gate] - if: | - always() && !cancelled() && - ( - (contains(format(',{0},', inputs.target_stage || inputs.target_stage_select), ',jit-kernel-benchmark-test-amd-rocm720,')) || - ( - !(inputs.target_stage || inputs.target_stage_select) && - (needs.call-gate.result == 'success' || needs.call-gate.result == 'skipped') && - needs.check-changes.outputs.jit_kernel == 'true' - ) - ) - runs-on: ${{ format('linux-{0}-1gpu-sglang', inputs.runner_arch || 'mi300') }} - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.pr_head_sha || inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Start CI container - run: bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version "$ROCM_VERSION" - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - - - name: Install dependencies - run: | - bash scripts/ci/amd/amd_ci_install_dependency.sh - - - name: Run JIT kernel benchmarks - timeout-minutes: 30 - run: | - bash scripts/ci/amd/amd_ci_exec.sh -w "/sglang-checkout/test" python3 run_suite.py --hw amd --suite jit-kernel-benchmark-test-amd ${{ needs.check-changes.outputs.continue_on_error == 'true' && '--continue-on-error' || '' }} - - # =============================================== Wait Jobs for Sequential PR Execution ==================================================== - # These jobs poll GitHub API to wait for previous stages to complete. - # For PR runs: wait jobs run and enforce sequential execution via polling. - # For scheduled runs: wait jobs are skipped, enabling parallel execution of all stages. - - wait-for-stage-a-amd-rocm720: - name: wait-for-stage-a-amd - needs: [check-changes, call-gate] - if: | - always() && - !cancelled() && - github.event_name == 'pull_request' && - !(inputs.target_stage || inputs.target_stage_select) && - (needs.check-changes.outputs.main_package == 'true' || needs.check-changes.outputs.sgl_kernel == 'true') && - (needs.call-gate.result == 'success' || needs.call-gate.result == 'skipped') - runs-on: ubuntu-latest - outputs: - stage_a_result: ${{ steps.wait.outputs.result }} - steps: - - uses: actions/checkout@v4 - - uses: ./.github/actions/wait-for-jobs - id: wait - with: - stage-name: stage-a-amd - jobs: '[{"prefix": "stage-a-test-1-gpu-small-amd", "expected_count": 1}]' - max-wait-minutes: '240' - - stage-b-test-1-gpu-small-amd-rocm720: - name: ${{ format('stage-b-test-1-gpu-small-amd ({0}, linux-{1}-1gpu-sglang, {2})', inputs.rocm_version || 'rocm724', inputs.runner_arch || 'mi300', matrix.part) }} - needs: [check-changes, wait-for-stage-a-amd-rocm720] - if: | - always() && - ( - (contains(format(',{0},', inputs.target_stage || inputs.target_stage_select), ',stage-b-test-1-gpu-small-amd-rocm720,')) || - ( - !(inputs.target_stage || inputs.target_stage_select) && - ((github.event_name == 'schedule') || (!failure() && !cancelled())) && - ((needs.check-changes.outputs.main_package == 'true') || (needs.check-changes.outputs.sgl_kernel == 'true')) - ) - ) - strategy: - fail-fast: false - max-parallel: 4 - matrix: - part: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13] - runs-on: ${{ format('linux-{0}-1gpu-sglang', inputs.runner_arch || 'mi300') }} - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.pr_head_sha || inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Start CI container - run: bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version "$ROCM_VERSION" - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - - - name: Install dependencies - run: bash scripts/ci/amd/amd_ci_install_dependency.sh - - - name: Run test - timeout-minutes: ${{ (inputs.runner_arch || 'mi300') == 'mi300' && 75 || 60 }} - run: | - bash scripts/ci/amd/amd_ci_exec.sh -w "/sglang-checkout/test" python3 run_suite.py --hw amd --suite stage-b-test-1-gpu-small-amd --auto-partition-id ${{ matrix.part }} --auto-partition-size 14 --timeout-per-file 2400 ${{ needs.check-changes.outputs.continue_on_error == 'true' && '--continue-on-error' || '' }} - - stage-b-test-1-gpu-small-amd-nondeterministic-rocm720: - name: ${{ format('stage-b-test-1-gpu-small-amd-nondeterministic ({0}, linux-{1}-1gpu-sglang)', inputs.rocm_version || 'rocm724', inputs.runner_arch || 'mi300') }} - needs: [check-changes, wait-for-stage-a-amd-rocm720] - if: | - always() && - ( - (contains(format(',{0},', inputs.target_stage || inputs.target_stage_select), ',stage-b-test-1-gpu-small-amd-nondeterministic-rocm720,')) || - ( - !(inputs.target_stage || inputs.target_stage_select) && - ((github.event_name == 'schedule') || (!failure() && !cancelled())) && - ((needs.check-changes.outputs.main_package == 'true') || (needs.check-changes.outputs.sgl_kernel == 'true')) - ) - ) - runs-on: ${{ format('linux-{0}-1gpu-sglang', inputs.runner_arch || 'mi300') }} - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.pr_head_sha || inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Start CI container - run: bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version "$ROCM_VERSION" - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - - - name: Install dependencies - run: bash scripts/ci/amd/amd_ci_install_dependency.sh - - - name: Run test - timeout-minutes: 75 - run: | - bash scripts/ci/amd/amd_ci_exec.sh -w "/sglang-checkout/test" python3 run_suite.py --hw amd --suite stage-b-test-1-gpu-small-amd-nondeterministic --timeout-per-file 3600 ${{ needs.check-changes.outputs.continue_on_error == 'true' && '--continue-on-error' || '' }} - - stage-b-test-1-gpu-small-amd-mi35x-rocm720: - name: ${{ format('stage-b-test-1-gpu-small-amd-mi35x ({0}, {1})', inputs.rocm_version || 'rocm724', matrix.runner) }} - needs: [check-changes, wait-for-stage-a-amd-rocm720] - if: | - always() && - ( - (contains(format(',{0},', inputs.target_stage || inputs.target_stage_select), ',stage-b-test-1-gpu-small-amd-mi35x-rocm720,')) || - ( - !(inputs.target_stage || inputs.target_stage_select) && - ((github.event_name == 'schedule') || (!failure() && !cancelled())) && - ((needs.check-changes.outputs.main_package == 'true') || (needs.check-changes.outputs.sgl_kernel == 'true')) - ) - ) - strategy: - fail-fast: false - matrix: - runner: [linux-mi35x-gpu-1] - runs-on: ${{matrix.runner}} - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.pr_head_sha || inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Start CI container - run: bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version "$ROCM_VERSION" - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - - - name: Install dependencies - run: bash scripts/ci/amd/amd_ci_install_dependency.sh - - - name: Run test - timeout-minutes: 30 - run: | - bash scripts/ci/amd/amd_ci_exec.sh -w "/sglang-checkout/test" python3 run_suite.py --hw amd --suite stage-b-test-1-gpu-small-amd-mi35x ${{ needs.check-changes.outputs.continue_on_error == 'true' && '--continue-on-error' || '' }} - - stage-b-test-1-gpu-large-amd-rocm720: - name: ${{ format('stage-b-test-1-gpu-large-amd ({0}, linux-{1}-1gpu-sglang, {2})', inputs.rocm_version || 'rocm724', inputs.runner_arch || 'mi300', matrix.part) }} - needs: [check-changes, wait-for-stage-a-amd-rocm720] - if: | - always() && - ( - (contains(format(',{0},', inputs.target_stage || inputs.target_stage_select), ',stage-b-test-1-gpu-large-amd-rocm720,')) || - ( - !(inputs.target_stage || inputs.target_stage_select) && - ((github.event_name == 'schedule') || (!failure() && !cancelled())) && - ((needs.check-changes.outputs.main_package == 'true') || (needs.check-changes.outputs.sgl_kernel == 'true')) - ) - ) - strategy: - fail-fast: false - max-parallel: 1 - matrix: - part: [0, 1, 2] - runs-on: ${{ format('linux-{0}-1gpu-sglang', inputs.runner_arch || 'mi300') }} - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.pr_head_sha || inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Start CI container - run: bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version "$ROCM_VERSION" - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - - - name: Install dependencies - run: bash scripts/ci/amd/amd_ci_install_dependency.sh - - - name: Run test - timeout-minutes: ${{ (inputs.runner_arch || 'mi300') == 'mi300' && 60 || 45 }} - run: | - bash scripts/ci/amd/amd_ci_exec.sh -w "/sglang-checkout/test" python3 run_suite.py --hw amd --suite stage-b-test-1-gpu-large-amd --auto-partition-id ${{ matrix.part }} --auto-partition-size 3 --timeout-per-file 2700 ${{ needs.check-changes.outputs.continue_on_error == 'true' && '--continue-on-error' || '' }} - - stage-b-test-2-gpu-large-amd-rocm720: - name: ${{ format('stage-b-test-2-gpu-large-amd ({0}, linux-{1}-2gpu-sglang, {2})', inputs.rocm_version || 'rocm724', inputs.runner_arch || 'mi300', matrix.part) }} - needs: [check-changes, wait-for-stage-a-amd-rocm720] - if: | - always() && - ( - (contains(format(',{0},', inputs.target_stage || inputs.target_stage_select), ',stage-b-test-2-gpu-large-amd-rocm720,')) || - ( - !(inputs.target_stage || inputs.target_stage_select) && - ((github.event_name == 'schedule') || (!failure() && !cancelled())) && - ((needs.check-changes.outputs.main_package == 'true') || (needs.check-changes.outputs.sgl_kernel == 'true')) - ) - ) - strategy: - fail-fast: false - max-parallel: 1 - matrix: - part: [0, 1] - runs-on: ${{ format('linux-{0}-2gpu-sglang', inputs.runner_arch || 'mi300') }} - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.pr_head_sha || inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Start CI container - run: bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version "$ROCM_VERSION" - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - - - name: Install dependencies - run: bash scripts/ci/amd/amd_ci_install_dependency.sh - - - name: Run test - timeout-minutes: ${{ (inputs.runner_arch || 'mi300') == 'mi300' && 120 || 45 }} - run: | - bash scripts/ci/amd/amd_ci_exec.sh -w "/sglang-checkout/test" python3 run_suite.py --hw amd --suite stage-b-test-2-gpu-large-amd --auto-partition-id ${{ matrix.part }} --auto-partition-size 2 --timeout-per-file ${{ (inputs.runner_arch || 'mi300') == 'mi300' && 5400 || 2700 }} ${{ needs.check-changes.outputs.continue_on_error == 'true' && '--continue-on-error' || '' }} - - multimodal-gen-test-1-gpu-amd-rocm720: - name: ${{ format('multimodal-gen-test-1-gpu-amd ({0}, linux-{1}-1gpu-sglang, {2})', inputs.rocm_version || 'rocm724', inputs.runner_arch || 'mi300', matrix.part) }} - needs: [check-changes, call-gate] - if: | - always() && !cancelled() && - ( - (contains(format(',{0},', inputs.target_stage || inputs.target_stage_select), ',multimodal-gen-test-1-gpu-amd-rocm720,')) || - ( - !(inputs.target_stage || inputs.target_stage_select) && - (needs.call-gate.result == 'success' || needs.call-gate.result == 'skipped') && - needs.check-changes.outputs.multimodal_gen == 'true' - ) - ) - strategy: - fail-fast: false - max-parallel: 1 - matrix: - part: [0, 1, 2, 3] - runs-on: ${{ format('linux-{0}-1gpu-sglang', inputs.runner_arch || 'mi300') }} - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.pr_head_sha || inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Download artifacts - if: needs.check-changes.outputs.sgl_kernel == 'true' - uses: actions/download-artifact@v4 - with: - path: python/sglang/kernels/aot/dist/ - merge-multiple: true - pattern: wheel-python3.10-cuda12.9 - - - name: Start CI container - run: bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version "$ROCM_VERSION" - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - - - name: Install dependencies - run: | - bash scripts/ci/amd/amd_ci_install_dependency.sh diffusion - - - name: Setup kernel caches - run: | - docker exec ci_sglang mkdir -p /sgl-data/miopen-cache /sgl-data/hf-cache/hub - - # Create persistent cache marker if /sgl-data is a real mount (not ephemeral) - # This tells the test cleanup code to NOT delete downloaded models - if docker exec ci_sglang test -d /sgl-data && docker exec ci_sglang mountpoint -q /sgl-data 2>/dev/null; then - docker exec ci_sglang touch /sgl-data/hf-cache/.persistent_cache - echo "Created .persistent_cache marker - HF cache will persist" - else - echo "WARNING: /sgl-data is not a mount point - models will be cleaned up after each test" - fi - - # Check MIOpen cache (VAE convolution kernels) - miopen_files=$(docker exec ci_sglang find /sgl-data/miopen-cache -name "*.udb" 2>/dev/null | wc -l || echo "0") - echo "Found ${miopen_files} MIOpen cache files" - - - name: Diagnose HF cache and system resources - run: | - echo "=== System Memory Status ===" - free -h - echo "" - echo "=== Disk Space ===" - df -h /home/runner/sglang-data 2>/dev/null || df -h - echo "" - echo "=== HF Cache Directory Structure ===" - docker exec ci_sglang ls -la /sgl-data/hf-cache/ 2>/dev/null || echo "HF cache dir not found" - docker exec ci_sglang ls -la /sgl-data/hf-cache/hub/ 2>/dev/null || echo "HF hub cache not found" - echo "" - echo "=== Checking for cached diffusion models (1-GPU tests) ===" - # Models used in 1-GPU tests: Wan2.1-T2V-1.3B, HunyuanVideo, Qwen-Image, FLUX.1, FLUX.2 - for model in "Wan-AI--Wan2.1-T2V-1.3B-Diffusers" "tencent--HunyuanVideo" "Qwen--Qwen-Image" "black-forest-labs--FLUX.1-dev" "black-forest-labs--FLUX.2-dev"; do - cache_path="/sgl-data/hf-cache/hub/models--${model}" - if docker exec ci_sglang test -d "$cache_path"; then - size=$(docker exec ci_sglang du -sh "$cache_path" 2>/dev/null | cut -f1) - echo "✓ CACHED: $model ($size)" - else - echo "✗ NOT CACHED: $model" - fi - done - echo "" - echo "=== GPU Memory Status ===" - docker exec ci_sglang rocm-smi --showmeminfo vram 2>/dev/null || echo "rocm-smi not available" - - - name: Run diffusion server tests (1-GPU) - timeout-minutes: 90 - run: | - # AMD CI: All 1-GPU tests except FLUX.2 (FLUX.1 covers same code path) - # Tests: T2V, T2I, I2V, LoRA - # - # HF download env vars: - # - HF_HUB_ENABLE_HF_TRANSFER=1: Use faster hf_transfer for downloads (if available) - # - HF_HUB_DISABLE_SYMLINKS_WARNING=1: Suppress symlink warnings - docker exec \ - -e SGLANG_E2E_TOLERANCE=0.3 \ - -e SGLANG_STAGE_TIME_TOLERANCE=0.2 \ - -e SGLANG_NON_DENOISE_STAGE_TIME_TOLERANCE=0.6 \ - -e SGLANG_DENOISE_STEP_TOLERANCE=0.6 \ - -e SGLANG_DENOISE_AGG_TOLERANCE=0.3 \ - -e SGLANG_SKIP_CONSISTENCY=1 \ - -e SGLANG_TEST_NUM_INFERENCE_STEPS=5 \ - -e SGLANG_DIFFUSION_ARTIFACT_DIR=/sglang-checkout/diffusion-failures \ - -e MIOPEN_USER_DB_PATH=/sgl-data/miopen-cache \ - -e HF_HUB_ENABLE_HF_TRANSFER=1 \ - -e HF_HUB_DISABLE_SYMLINKS_WARNING=1 \ - -w /sglang-checkout/python \ - ci_sglang python3 sglang/multimodal_gen/test/run_suite.py \ - --suite 1-gpu \ - --partition-id ${{ matrix.part }} \ - --total-partitions 4 \ - -k "not flux_2" \ - ${{ needs.check-changes.outputs.continue_on_error == 'true' && '--continue-on-error' || '' }} - - # Post-test diagnostics - echo "=== Post-test System Memory Status ===" - free -h - - # The mm_gen `unit` suite is portable CPU-style tests that finish in a - # few minutes, which does not justify a job of its own: on mi300 the - # container pull plus dependency install costs ~2h before any test runs. - # Ride along on the shard that is already warm. Runs after the diffusion - # tests, and on failure too, so a diffusion regression cannot hide it. - # Skips the CUDA-only ltx2_vae_channels_last memory-format assertion. - - name: Run diffusion unit tests - if: matrix.part == 0 && (success() || failure()) - timeout-minutes: 30 - run: | - docker exec \ - -e MIOPEN_USER_DB_PATH=/sgl-data/miopen-cache \ - -w /sglang-checkout/python \ - ci_sglang python3 sglang/multimodal_gen/test/run_suite.py \ - --suite unit \ - -k "not ltx2_vae_channels_last" \ - ${{ needs.check-changes.outputs.continue_on_error == 'true' && '--continue-on-error' || '' }} - - - name: Upload diffusion failure artifacts - if: always() - uses: actions/upload-artifact@v4 - with: - name: diffusion-failures-amd-1gpu-${{ matrix.part }}-${{ github.run_attempt }} - path: diffusion-failures/ - if-no-files-found: ignore - retention-days: 7 - - multimodal-gen-test-2-gpu-amd-rocm720: - name: ${{ format('multimodal-gen-test-2-gpu-amd ({0}, linux-{1}-2gpu-sglang, {2})', inputs.rocm_version || 'rocm724', inputs.runner_arch || 'mi300', matrix.part) }} - needs: [check-changes, call-gate] - if: | - always() && !cancelled() && - ( - (contains(format(',{0},', inputs.target_stage || inputs.target_stage_select), ',multimodal-gen-test-2-gpu-amd-rocm720,')) || - ( - !(inputs.target_stage || inputs.target_stage_select) && - (needs.call-gate.result == 'success' || needs.call-gate.result == 'skipped') && - needs.check-changes.outputs.multimodal_gen == 'true' - ) - ) - strategy: - fail-fast: false - max-parallel: 1 - matrix: - # run_suite.py load-balances the suite's parametrized cases and - # standalone files over these partitions, so the count is free to - # choose. Two shards measured 45-65 min of case work each, inside the - # 180 min step timeout, and halve the ~2h per-job container pull plus - # dependency install. - part: [0, 1] - runs-on: ${{ format('linux-{0}-2gpu-sglang', inputs.runner_arch || 'mi300') }} - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.pr_head_sha || inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Download artifacts - if: needs.check-changes.outputs.sgl_kernel == 'true' - uses: actions/download-artifact@v4 - with: - path: python/sglang/kernels/aot/dist/ - merge-multiple: true - pattern: wheel-python3.10-cuda12.9 - - - name: Start CI container - run: bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version "$ROCM_VERSION" - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - - - name: Install dependencies - run: | - bash scripts/ci/amd/amd_ci_install_dependency.sh diffusion - - - name: Setup kernel caches - run: | - docker exec ci_sglang mkdir -p /sgl-data/miopen-cache /sgl-data/hf-cache/hub - - # Create persistent cache marker if /sgl-data is a real mount (not ephemeral) - # This tells the test cleanup code to NOT delete downloaded models - if docker exec ci_sglang test -d /sgl-data && docker exec ci_sglang mountpoint -q /sgl-data 2>/dev/null; then - docker exec ci_sglang touch /sgl-data/hf-cache/.persistent_cache - echo "Created .persistent_cache marker - HF cache will persist" - else - echo "WARNING: /sgl-data is not a mount point - models will be cleaned up after each test" - fi - - # Check MIOpen cache (VAE convolution kernels) - miopen_files=$(docker exec ci_sglang find /sgl-data/miopen-cache -name "*.udb" 2>/dev/null | wc -l || echo "0") - echo "Found ${miopen_files} MIOpen cache files" - - - name: Diagnose HF cache and system resources - run: | - echo "=== System Memory Status ===" - free -h - echo "" - echo "=== Disk Space ===" - df -h /home/runner/sglang-data 2>/dev/null || df -h - echo "" - echo "=== HF Cache Directory Structure ===" - docker exec ci_sglang ls -la /sgl-data/hf-cache/ 2>/dev/null || echo "HF cache dir not found" - docker exec ci_sglang ls -la /sgl-data/hf-cache/hub/ 2>/dev/null || echo "HF hub cache not found" - echo "" - echo "=== Checking for cached diffusion models (2-GPU tests) ===" - # Models used in 2-GPU tests: Wan2.2-T2V-A14B, Wan2.1-T2V-14B, Qwen-Image, FLUX.1 - for model in "Wan-AI--Wan2.2-T2V-A14B-Diffusers" "Wan-AI--Wan2.1-T2V-14B-Diffusers" "Qwen--Qwen-Image" "black-forest-labs--FLUX.1-dev"; do - cache_path="/sgl-data/hf-cache/hub/models--${model}" - if docker exec ci_sglang test -d "$cache_path"; then - size=$(docker exec ci_sglang du -sh "$cache_path" 2>/dev/null | cut -f1) - echo "✓ CACHED: $model ($size)" - else - echo "✗ NOT CACHED: $model" - fi - done - echo "" - echo "=== GPU Memory Status ===" - docker exec ci_sglang rocm-smi --showmeminfo vram 2>/dev/null || echo "rocm-smi not available" - - - name: Run diffusion server tests (2-GPU) - timeout-minutes: ${{ (inputs.runner_arch || 'mi300') == 'mi300' && 180 || 150 }} - run: | - # AMD CI: All 2-GPU tests including LoRA - # Tests: T2V, T2I, I2V, LoRA - # - # HF download env vars: - # - HF_HUB_ENABLE_HF_TRANSFER=1: Use faster hf_transfer for downloads (if available) - # - HF_HUB_DISABLE_SYMLINKS_WARNING=1: Suppress symlink warnings - docker exec \ - -e SGLANG_E2E_TOLERANCE=0.3 \ - -e SGLANG_STAGE_TIME_TOLERANCE=0.2 \ - -e SGLANG_NON_DENOISE_STAGE_TIME_TOLERANCE=0.6 \ - -e SGLANG_DENOISE_STEP_TOLERANCE=0.6 \ - -e SGLANG_DENOISE_AGG_TOLERANCE=0.3 \ - -e SGLANG_SKIP_CONSISTENCY=1 \ - -e SGLANG_TEST_NUM_INFERENCE_STEPS=5 \ - -e SGLANG_DIFFUSION_ARTIFACT_DIR=/sglang-checkout/diffusion-failures \ - -e MIOPEN_USER_DB_PATH=/sgl-data/miopen-cache \ - -e HF_HUB_ENABLE_HF_TRANSFER=1 \ - -e HF_HUB_DISABLE_SYMLINKS_WARNING=1 \ - -w /sglang-checkout/python \ - ci_sglang python3 sglang/multimodal_gen/test/run_suite.py \ - --suite 2-gpu \ - --partition-id ${{ matrix.part }} \ - --total-partitions 2 \ - ${{ needs.check-changes.outputs.continue_on_error == 'true' && '--continue-on-error' || '' }} - - # Post-test diagnostics - echo "=== Post-test System Memory Status ===" - free -h - - - name: Upload diffusion failure artifacts - if: always() - uses: actions/upload-artifact@v4 - with: - name: diffusion-failures-amd-2gpu-${{ matrix.part }}-${{ github.run_attempt }} - path: diffusion-failures/ - if-no-files-found: ignore - retention-days: 7 - - wait-for-stage-b-amd-rocm720: - name: wait-for-stage-b-amd - needs: [check-changes, call-gate, wait-for-stage-a-amd-rocm720] - if: | - always() && - !cancelled() && - github.event_name == 'pull_request' && - !(inputs.target_stage || inputs.target_stage_select) && - (needs.check-changes.outputs.main_package == 'true' || needs.check-changes.outputs.sgl_kernel == 'true') && - (needs.wait-for-stage-a-amd-rocm720.result == 'success' || needs.wait-for-stage-a-amd-rocm720.result == 'skipped') && - (needs.call-gate.result == 'success' || needs.call-gate.result == 'skipped') - runs-on: ubuntu-latest - outputs: - stage_b_result: ${{ steps.wait.outputs.result }} - steps: - - uses: actions/checkout@v4 - - uses: ./.github/actions/wait-for-jobs - id: wait - with: - stage-name: stage-b-amd - jobs: | - [ - {"prefix": "stage-b-test-1-gpu-small-amd", "expected_count": 14}, - {"prefix": "stage-b-test-1-gpu-large-amd", "expected_count": 3}, - {"prefix": "stage-b-test-2-gpu-large-amd", "expected_count": 2} - ] - max-wait-minutes: '480' - - stage-c-test-4-gpu-amd-rocm720: - name: ${{ format('stage-c-test-4-gpu-amd ({0}, linux-{1}-4gpu-sglang, {2})', inputs.rocm_version || 'rocm724', inputs.runner_arch || 'mi300', matrix.part) }} - needs: [check-changes, call-gate, wait-for-stage-b-amd-rocm720] - if: | - always() && - ( - (contains(format(',{0},', inputs.target_stage || inputs.target_stage_select), ',stage-c-test-4-gpu-amd-rocm720,')) || - ( - !(inputs.target_stage || inputs.target_stage_select) && - ((github.event_name == 'schedule') || (!failure() && !cancelled())) && - ((needs.check-changes.outputs.main_package == 'true') || (needs.check-changes.outputs.sgl_kernel == 'true')) - ) - ) - strategy: - fail-fast: false - matrix: - part: [0] - runs-on: ${{ format('linux-{0}-4gpu-sglang', inputs.runner_arch || 'mi300') }} - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.pr_head_sha || inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Start CI container - run: bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version "$ROCM_VERSION" - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - - - name: Install dependencies - run: bash scripts/ci/amd/amd_ci_install_dependency.sh - - - name: Run test - timeout-minutes: 90 - run: | - bash scripts/ci/amd/amd_ci_exec.sh \ - -e NCCL_CUMEM_ENABLE=0 \ - -e NCCL_NVLS_ENABLE=0 \ - -e RCCL_MSCCL_ENABLE=0 \ - -e SGLANG_USE_ROCM700A=1 \ - -w "/sglang-checkout/test" \ - python3 run_suite.py \ - --hw amd \ - --suite stage-c-test-4-gpu-amd \ - --auto-partition-id ${{ matrix.part }} \ - --auto-partition-size 1 \ - --timeout-per-file 5400 \ - --enable-retry \ - --max-attempts 2 \ - --retry-wait-seconds 120 \ - --retry-timeout-increase 0 \ - ${{ needs.check-changes.outputs.continue_on_error == 'true' && '--continue-on-error' || '' }} - - stage-c-test-large-8-gpu-amd-rocm720: - name: ${{ format('stage-c-test-large-8-gpu-amd ({0}, linux-{1}-8gpu-sglang, {2})', inputs.rocm_version || 'rocm724', inputs.runner_arch || 'mi300', matrix.part) }} - needs: [check-changes, call-gate, wait-for-stage-b-amd-rocm720] - if: | - always() && - ( - (contains(format(',{0},', inputs.target_stage || inputs.target_stage_select), ',stage-c-test-large-8-gpu-amd-rocm720,')) || - ( - !(inputs.target_stage || inputs.target_stage_select) && - ((github.event_name == 'schedule') || (!failure() && !cancelled())) && - ((needs.check-changes.outputs.main_package == 'true') || (needs.check-changes.outputs.sgl_kernel == 'true')) - ) - ) - env: - RUNNER_LABELS: ${{ format('linux-{0}-8gpu-sglang', inputs.runner_arch || 'mi300') }} - strategy: - fail-fast: false - max-parallel: 2 - matrix: - part: [0, 1, 2, 3] - runs-on: ${{ format('linux-{0}-8gpu-sglang', inputs.runner_arch || 'mi300') }} - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.pr_head_sha || inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Start CI container - run: bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version "$ROCM_VERSION" - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - - - name: Install dependencies - run: bash scripts/ci/amd/amd_ci_install_dependency.sh - - - name: Test RCCL multi-GPU communication - timeout-minutes: 5 - run: | - echo "Testing RCCL multi-GPU communication with debug info..." - docker exec ci_sglang bash -c "cd /sglang-checkout && NCCL_DEBUG=INFO RCCL_DEBUG=INFO torchrun --nproc_per_node=8 scripts/ci/amd/test_rccl_multi_gpu.py" - - - name: Run test - timeout-minutes: 120 - run: | - bash scripts/ci/amd/amd_ci_exec.sh -w "/sglang-checkout/test" python3 run_suite.py --hw amd --suite stage-c-test-large-8-gpu-amd --auto-partition-id ${{ matrix.part }} --auto-partition-size 4 --timeout-per-file 5400 ${{ needs.check-changes.outputs.continue_on_error == 'true' && '--continue-on-error' || '' }} - - stage-c-test-large-8-gpu-amd-mi35x-rocm720: - name: ${{ format('stage-c-test-large-8-gpu-amd-mi35x ({0}, {1}, {2})', inputs.rocm_version || 'rocm724', matrix.runner, matrix.part) }} - needs: [check-changes, call-gate, wait-for-stage-b-amd-rocm720] - if: | - always() && - ( - (contains(format(',{0},', inputs.target_stage || inputs.target_stage_select), ',stage-c-test-large-8-gpu-amd-mi35x-rocm720,')) || - ( - !(inputs.target_stage || inputs.target_stage_select) && - ((github.event_name == 'schedule') || (!failure() && !cancelled())) && - ((needs.check-changes.outputs.main_package == 'true') || (needs.check-changes.outputs.sgl_kernel == 'true')) - ) - ) - strategy: - fail-fast: false - matrix: - runner: [linux-mi35x-gpu-8] - part: [0, 1, 2] - runs-on: ${{matrix.runner}} - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.pr_head_sha || inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Start CI container - run: bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version "$ROCM_VERSION" - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - - - name: Install dependencies - run: bash scripts/ci/amd/amd_ci_install_dependency.sh - - - name: Run test - timeout-minutes: 60 - run: | - bash scripts/ci/amd/amd_ci_exec.sh -w "/sglang-checkout/test" python3 run_suite.py --hw amd --suite stage-c-test-large-8-gpu-amd-mi35x --auto-partition-id ${{ matrix.part }} --auto-partition-size 3 --timeout-per-file 3600 ${{ needs.check-changes.outputs.continue_on_error == 'true' && '--continue-on-error' || '' }} - - # =============================================== DeepSeek-V4 (MI35x, 8-GPU) ==================================================== - # GSM8K accuracy on the nightly dsv4 suites, ~20min each. Gated as stage-C so - # every PR that touches the main package covers DeepSeek-V4. - stage-c-dsv4-flash-fp4-fp8-amd-mi35x-rocm720: - name: ${{ format('stage-c-dsv4-flash-fp4-fp8-amd-mi35x ({0}, {1})', inputs.rocm_version || 'rocm724', matrix.runner) }} - needs: [check-changes, call-gate, wait-for-stage-b-amd-rocm720] - if: | - always() && - ( - (contains(format(',{0},', inputs.target_stage || inputs.target_stage_select), ',stage-c-dsv4-flash-fp4-fp8-amd-mi35x-rocm720,')) || - ( - !(inputs.target_stage || inputs.target_stage_select) && - ((github.event_name == 'schedule') || (!failure() && !cancelled())) && - ((needs.check-changes.outputs.main_package == 'true') || (needs.check-changes.outputs.sgl_kernel == 'true')) - ) - ) - strategy: - fail-fast: false - matrix: - runner: [linux-mi35x-gpu-8] - runs-on: ${{matrix.runner}} - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.pr_head_sha || inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Setup docker (ROCm 7.2) - run: | - touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version "$ROCM_VERSION" - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - ENABLE_CACHE_HOST: "1" - - - name: Install dependencies - run: | - # --skip-test-time-deps: GSM8K + bench_one_batch_server don't need lmms-eval / human-eval. - bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps - bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate - - - name: Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Flash FP4 + FP8) - timeout-minutes: 60 - run: | - > github_summary.md # Clear summary file - # SGLANG_DSV4_ACCURACY_ONLY=1 makes the dsv4 test files skip their perf test - # (test_b_perf_8k_1k); only the GSM8K accuracy test runs in this workflow. - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - -e SGLANG_DSV4_ACCURACY_ONLY=1 \ - -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ - python3 run_suite.py --hw amd --suite nightly-amd-8-gpu-mi35x-deepseek-v4-flash --nightly --timeout-per-file 3600 ${{ needs.check-changes.outputs.continue_on_error == 'true' && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - stage-c-dsv4-pro-fp4-amd-mi35x-rocm720: - name: ${{ format('stage-c-dsv4-pro-fp4-amd-mi35x ({0}, {1})', inputs.rocm_version || 'rocm724', matrix.runner) }} - needs: [check-changes, call-gate, wait-for-stage-b-amd-rocm720] - if: | - always() && - ( - (contains(format(',{0},', inputs.target_stage || inputs.target_stage_select), ',stage-c-dsv4-pro-fp4-amd-mi35x-rocm720,')) || - ( - !(inputs.target_stage || inputs.target_stage_select) && - ((github.event_name == 'schedule') || (!failure() && !cancelled())) && - ((needs.check-changes.outputs.main_package == 'true') || (needs.check-changes.outputs.sgl_kernel == 'true')) - ) - ) - strategy: - fail-fast: false - matrix: - runner: [linux-mi35x-gpu-8] - runs-on: ${{matrix.runner}} - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.pr_head_sha || inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Setup docker (ROCm 7.2) - run: | - touch github_summary.md - bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version "$ROCM_VERSION" - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - ENABLE_CACHE_HOST: "1" - - - name: Install dependencies - run: | - # --skip-test-time-deps: GSM8K + bench_one_batch_server don't need lmms-eval / human-eval. - bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps - bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate - - - name: Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-V4-Pro FP4) - timeout-minutes: 60 - run: | - > github_summary.md # Clear summary file - # SGLANG_DSV4_ACCURACY_ONLY=1 makes the dsv4 test files skip their perf test - # (test_b_perf_8k_1k); only the GSM8K accuracy test runs in this workflow. - bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ - -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ - -e SGLANG_DSV4_ACCURACY_ONLY=1 \ - -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ - python3 registered/amd/test_deepseek_v4_pro_fp4.py || TEST_EXIT_CODE=$? - echo "$(> $GITHUB_STEP_SUMMARY || true - exit ${TEST_EXIT_CODE:-0} - - # =============================================== Disaggregation ==================================================== - stage-b-test-large-8-gpu-mi35x-disaggregation-amd-rocm720: - name: ${{ format('stage-b-test-large-8-gpu-mi35x-disaggregation-amd ({0}, {1})', inputs.rocm_version || 'rocm724', matrix.runner) }} - needs: [check-changes, wait-for-stage-a-amd-rocm720] - if: | - always() && - ( - (contains(format(',{0},', inputs.target_stage || inputs.target_stage_select), ',stage-b-test-large-8-gpu-mi35x-disaggregation-amd-rocm720,')) || - ( - !(inputs.target_stage || inputs.target_stage_select) && - ((github.event_name == 'schedule') || (!failure() && !cancelled())) && - ((needs.check-changes.outputs.main_package == 'true') || (needs.check-changes.outputs.sgl_kernel == 'true')) - ) - ) - strategy: - fail-fast: false - matrix: - runner: [linux-mi35x-gpu-8.fabric] - - runs-on: ${{matrix.runner}} - - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.pr_head_sha || inputs.ref || github.sha }} - - - name: Ensure VRAM is clear - run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - - - name: Check Host RDMA Environment - id: rdma_detect - run: | - set +e - echo "=== Checking Host RDMA Environment ===" - - echo "" - echo "=== 1. Ionic driver library check ===" - ls -l /usr/lib/x86_64-linux-gnu/libibverbs/libionic* 2>/dev/null || echo "libionic not found in standard path" - - echo "" - echo "=== 2. Infiniband devices ===" - ls -la /dev/infiniband/ 2>/dev/null || echo "/dev/infiniband not found" - ls -la /sys/class/infiniband/ 2>/dev/null || echo "/sys/class/infiniband not found" - - echo "" - echo "=== 3. ibv_devinfo ===" - which ibv_devinfo 2>/dev/null && ibv_devinfo 2>&1 || echo "ibv_devinfo not available" - - echo "" - echo "=== 4. Kernel modules ===" - lsmod 2>/dev/null | grep -E "ib_|rdma|ionic" || echo "No RDMA kernel modules loaded" - - echo "" - echo "=== 5. Detect RDMA Devices for test environment ===" - if [ -d "/sys/class/infiniband" ]; then - # RDMA_DEVS=$(ls /sys/class/infiniband | paste -sd "," -) - RDMA_DEVS=rdma0 - echo "Detected RDMA Devices: $RDMA_DEVS" - echo "SGLANG_TEST_RDMA_DEVICE=$RDMA_DEVS" >> $GITHUB_ENV - else - echo "No RDMA devices found in /sys/class/infiniband" - echo "SGLANG_TEST_RDMA_DEVICE=" >> $GITHUB_ENV - fi - - echo "" - echo "=== Host RDMA Check Complete ===" - - - name: Start Special Container - run: bash scripts/ci/amd/amd_ci_start_container_disagg.sh --rocm-version "$ROCM_VERSION" - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - - - name: Install dependencies - run: bash scripts/ci/amd/amd_ci_install_dependency.sh - - - name: Verify NIXL in Container - run: | - docker exec ci_sglang bash -lc ' - set -eux - export LD_LIBRARY_PATH=/opt/ucx/lib:/opt/rocm/lib:${LD_LIBRARY_PATH:-} - python -c "import nixl; print(\"nixl OK:\", nixl.__file__)" - test -d /opt/ucx/lib - UCX_INFO=$(command -v ucx_info || true) - if [ -z "$UCX_INFO" ]; then - UCX_INFO=/opt/ucx/bin/ucx_info - fi - test -x "$UCX_INFO" - "$UCX_INFO" -v - "$UCX_INFO" -d | grep -iq rocm - python -c "from nixl._api import nixl_agent, nixl_agent_config; nixl_agent(\"ci-smoke\", nixl_agent_config(backends=[\"UCX\"])); print(\"nixl UCX backend init OK\")" - ' - - - name: Verify RDMA in Container - run: | - docker exec -u root ci_sglang bash -c ' - echo "=== Container RDMA Verification ===" - echo "Device nodes:" - ls -la /dev/infiniband/ - echo "" - echo "Provider libraries:" - ls /usr/lib/x86_64-linux-gnu/libibverbs/ | grep -E "ionic|mlx" || echo "No Ionic/Mellanox providers" - echo "" - echo "HCA devices:" - HCA_COUNT=$(ibv_devinfo -list 2>&1 | grep -oE "^[0-9]+ HCAs? found" | grep -oE "^[0-9]+" || echo "0") - ibv_devinfo -list - if [ "$HCA_COUNT" -gt 0 ]; then - echo "" - echo "=== SUCCESS: RDMA setup complete. Found $HCA_COUNT HCA(s) ===" - else - echo "" - echo "=== WARNING: No HCAs detected. RDMA tests may fail ===" - fi - ' - - - name: Run Aiter Op Test (RMSNorm) - timeout-minutes: 10 - run: | - echo "Running pre-check: test_rmsnorm2d.py" - docker exec \ - -e MAX_JOBS=192 \ - ci_sglang \ - python /sgl-workspace/aiter/op_tests/test_rmsnorm2d.py - - - name: Run test_disaggregation - timeout-minutes: 60 - run: | - docker exec ci_sglang rocm-smi --showtopotype || true - docker exec ci_sglang bash -c 'strings $(python3 -c "import mooncake,os;print(os.path.dirname(mooncake.__file__))")/engine*.so | grep -iE "rdma,hip|Unsupported multi-protocol"' || true - bash scripts/ci/amd/amd_ci_exec.sh \ - -e SGLANG_TEST_RDMA_DEVICE="${{ env.SGLANG_TEST_RDMA_DEVICE }}" \ - -e MC_LOG_LEVEL=TRACE \ - -e GLOG_logtostderr=1 \ - -w "/sglang-checkout/test" python3 run_suite.py --hw amd --suite stage-b-test-large-8-gpu-mi35x-disaggregation-amd --timeout-per-file 1800 ${{ needs.check-changes.outputs.continue_on_error == 'true' && '--continue-on-error' || '' }} - - pr-test-amd-rocm720-finish: - name: pr-test-amd-finish - needs: - [ - call-gate, - check-changes, - - sgl-kernel-unit-test-amd-rocm720, - sgl-kernel-unit-test-2-gpu-amd-rocm720, - multimodal-gen-test-1-gpu-amd-rocm720, - multimodal-gen-test-2-gpu-amd-rocm720, - - wait-for-stage-a-amd-rocm720, - stage-a-test-1-gpu-small-amd-rocm720, - jit-kernel-unit-test-amd-rocm720, - jit-kernel-benchmark-test-amd-rocm720, - wait-for-stage-b-amd-rocm720, - stage-b-test-1-gpu-small-amd-rocm720, - stage-b-test-1-gpu-small-amd-nondeterministic-rocm720, - stage-b-test-1-gpu-small-amd-mi35x-rocm720, - stage-b-test-1-gpu-large-amd-rocm720, - stage-b-test-2-gpu-large-amd-rocm720, - stage-b-test-large-8-gpu-mi35x-disaggregation-amd-rocm720, - stage-c-test-4-gpu-amd-rocm720, - stage-c-test-large-8-gpu-amd-rocm720, - stage-c-test-large-8-gpu-amd-mi35x-rocm720, - - stage-c-dsv4-flash-fp4-fp8-amd-mi35x-rocm720, - stage-c-dsv4-pro-fp4-amd-mi35x-rocm720, - ] - if: always() - runs-on: ubuntu-latest - steps: - - name: Check all dependent job statuses - run: | - # Convert the 'needs' context to a JSON string - json_needs='${{ toJson(needs) }}' - - # Get a list of all job names from the JSON keys - job_names=$(echo "$json_needs" | jq -r 'keys_unsorted[]') - - for job in $job_names; do - # For each job, extract its result - result=$(echo "$json_needs" | jq -r --arg j "$job" '.[$j].result') - - # Print the job name and its result - echo "$job: $result" - - # Check for failure or cancellation and exit if found - if [[ "$result" == "failure" || "$result" == "cancelled" ]]; then - echo "The above jobs failed." - exit 1 - fi - done - - # If the loop completes, all jobs were successful - echo "All jobs completed successfully" - exit 0 diff --git a/.github/workflows/pr-test-amd.yml b/.github/workflows/pr-test-amd.yml index edc6fcb606b4..cf92aa5fab59 100644 --- a/.github/workflows/pr-test-amd.yml +++ b/.github/workflows/pr-test-amd.yml @@ -1,18 +1,32 @@ -name: PR Test ROCm 7.0 (AMD) +name: PR Test (AMD) # Dynamic run-name for manual stage dispatches to enable URL lookup # Format: "[stage-name] sha" for fork PRs, "[stage-name]" for non-fork, default for normal runs run-name: ${{ (inputs.target_stage || inputs.target_stage_select) && (inputs.pr_head_sha && format('[{0}] {1}', inputs.target_stage || inputs.target_stage_select, inputs.pr_head_sha) || format('[{0}]', inputs.target_stage || inputs.target_stage_select)) || '' }} on: schedule: - # ROCm 7.0 rollback shadow. pr-test-amd-rocm720.yml is the PR gate; this - # workflow now only proves ROCm 7.0 still works, so it fires once a day - # alongside nightly-test-amd.yml and chases coverage rather than latency: - # no matrix parallelism caps, no stage fast-fail (the `schedule` escape in - # each job's `if`), and continue-on-error via check-changes. - - cron: '30 17 * * *' + - cron: '0 */12 * * *' # rocm10: twice daily (UTC) + - cron: '30 17 * * *' # rocm724/rocm720: once daily (UTC) + pull_request: + paths: + - "python/**" + - "scripts/ci/**" + - "test/**" + - "python/sglang/kernels/aot/**" + - ".github/workflows/pr-test-amd.yml" + - "docker/rocm.Dockerfile" workflow_dispatch: inputs: + rocm_version: + description: 'ROCm image version ("all" runs rocm10, rocm724, and rocm720)' + required: false + type: choice + default: rocm10 + options: + - all + - rocm10 + - rocm724 + - rocm720 target_stage_select: description: "Select a stage to run from dropdown (leave empty for auto-detect)" required: false @@ -33,6 +47,8 @@ on: - multimodal-gen-test-1-gpu-amd - multimodal-gen-test-2-gpu-amd - stage-c-test-4-gpu-amd + - stage-c-dsv4-flash-fp4-fp8-amd-mi35x + - stage-c-dsv4-pro-fp4-amd-mi35x - stage-c-test-large-8-gpu-amd - stage-c-test-large-8-gpu-amd-mi35x - stage-b-test-large-8-gpu-mi35x-disaggregation-amd @@ -51,6 +67,11 @@ on: required: false type: string default: '' + amd_ci_image: + description: 'Override AMD CI Docker image (optional, e.g. rocm/sgl-dev:tag)' + required: false + type: string + default: '' continue_on_error: description: 'Continue on error (do not fail the workflow on test failures)' required: false @@ -71,6 +92,11 @@ on: default: false workflow_call: inputs: + rocm_version: + description: 'ROCm image version ("all" runs rocm10, rocm724, and rocm720)' + required: false + type: string + default: rocm10 ref: description: 'Git ref (branch, tag, or SHA) to test. If not provided, uses the default branch.' required: false @@ -91,6 +117,11 @@ on: required: false type: string default: '' + amd_ci_image: + description: 'Override AMD CI Docker image (optional, e.g. rocm/sgl-dev:tag)' + required: false + type: string + default: '' continue_on_error: description: 'Continue on error (do not fail the workflow on test failures)' required: false @@ -110,6 +141,7 @@ permissions: env: AITER_COMMIT_OVERRIDE: ${{ inputs.aiter_ref }} + AMD_CI_IMAGE: ${{ inputs.amd_ci_image }} DOCKERHUB_AMD_USERNAME: ${{ secrets.DOCKERHUB_AMD_USERNAME }} DOCKERHUB_AMD_TOKEN: ${{ secrets.DOCKERHUB_AMD_TOKEN }} @@ -134,6 +166,10 @@ jobs: jit_kernel: ${{ steps.filter.outputs.jit_kernel || steps.run-mode.outputs.run_all_tests }} multimodal_gen: ${{ steps.filter.outputs.multimodal_gen || steps.run-mode.outputs.run_all_tests }} continue_on_error: ${{ steps.set-continue-on-error.outputs.continue_on_error }} + # Keep rocm10 on the twice-daily cadence. The 17:30 UTC schedule runs + # rocm724 and rocm720 once daily. Dispatch/call inputs select one version + # or all three, while a pull_request falls through to the rocm10 default. + rocm_versions: ${{ inputs.rocm_version == 'all' && '["rocm10","rocm724","rocm720"]' || inputs.rocm_version && format('["{0}"]', inputs.rocm_version) || github.event.schedule == '30 17 * * *' && '["rocm724","rocm720"]' || '["rocm10"]' }} steps: - name: Checkout code uses: actions/checkout@v4 @@ -151,19 +187,29 @@ jobs: echo "Run mode: FILTERED (triggered by ${{ github.event_name }})" fi + - name: Resolve CI labels + id: ci-labels + if: github.event_name == 'pull_request' + uses: actions/github-script@v8 + with: + script: | + const { resolveCiLabels } = require( + `${process.env.GITHUB_WORKSPACE}/.github/scripts/ci-labels.cjs` + ); + const axes = await resolveCiLabels(github, context); + core.info(`CI labels: [${axes.labels.join(', ')}]`); + core.setOutput('bypass_fail_fast', String(axes.bypassFailFast)); + - name: Set continue-on-error for schedule/full runs id: set-continue-on-error env: - # `bypass-fastfail` PR label: also disable within-suite fast-fail - # here. The shared actions/wait-for-jobs already honors the same - # label to skip cross-stage waits. - BYPASS_FASTFAIL_LABEL: ${{ contains(github.event.pull_request.labels.*.name, 'bypass-fastfail') }} + BYPASS_FAIL_FAST: ${{ steps.ci-labels.outputs.bypass_fail_fast }} run: | if [[ "${{ steps.run-mode.outputs.run_all_tests }}" == "true" \ || "${{ inputs.continue_on_error }}" == "true" \ - || "$BYPASS_FASTFAIL_LABEL" == "true" ]]; then + || "$BYPASS_FAIL_FAST" == "true" ]]; then echo "continue_on_error=true" >> $GITHUB_OUTPUT - echo "Continue-on-error: ENABLED (run_all_tests=${{ steps.run-mode.outputs.run_all_tests }}, input=${{ inputs.continue_on_error }}, bypass-fastfail=$BYPASS_FASTFAIL_LABEL)" + echo "Continue-on-error: ENABLED (run_all_tests=${{ steps.run-mode.outputs.run_all_tests }}, input=${{ inputs.continue_on_error }}, bypass-fail-fast=$BYPASS_FAIL_FAST)" else echo "continue_on_error=false" >> $GITHUB_OUTPUT echo "Continue-on-error: DISABLED" @@ -212,21 +258,32 @@ jobs: # Not added to `pr-test-amd-finish` so the base AMD gate never depends on # the opt-in extra suite. call-pr-test-amd-extra: + name: call-pr-test-amd-extra + needs: [check-changes] if: | (github.event_name == 'schedule' || inputs.run_all_tests == true) && !(inputs.target_stage || inputs.target_stage_select) + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(needs.check-changes.outputs.rocm_versions) }} uses: ./.github/workflows/pr-test-amd-extra.yml with: ref: ${{ inputs.pr_head_sha || inputs.ref || '' }} runner_arch: ${{ inputs.runner_arch || 'mi300' }} - rocm_version: rocm700 + rocm_version: ${{ matrix.rocm_version }} aiter_ref: ${{ inputs.aiter_ref }} + amd_ci_image: ${{ inputs.amd_ci_image }} continue_on_error: true secrets: inherit # =============================================== sgl-kernel ==================================================== sgl-kernel-unit-test-amd: - name: ${{ format('sgl-kernel-unit-test-amd (rocm700, linux-{0}-1gpu-sglang)', inputs.runner_arch || 'mi300') }} + name: ${{ format('sgl-kernel-unit-test-amd ({0}, linux-{1}-1gpu-sglang)', matrix.rocm_version, inputs.runner_arch || 'mi300') }} + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(needs.check-changes.outputs.rocm_versions) }} needs: [check-changes, call-gate] if: | always() && !cancelled() && @@ -249,7 +306,7 @@ jobs: run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - name: Start CI container - run: bash scripts/ci/amd/amd_ci_start_container.sh + run: bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version "${{ matrix.rocm_version }}" env: GITHUB_WORKSPACE: ${{ github.workspace }} @@ -274,7 +331,7 @@ jobs: } run_pytest docker exec -w /sglang-checkout/python/sglang/kernels/aot/tests ci_sglang python3 -m pytest test_moe_align.py run_pytest docker exec -w /sglang-checkout/python/sglang/kernels/aot/tests ci_sglang python3 -m pytest test_moe_topk_softmax.py - run_pytest docker exec -w /sglang-checkout/python/sglang/kernels/aot/tests/speculative ci_sglang python3 -m pytest test_eagle_utils.py + run_pytest docker exec -w /sglang-checkout/test/registered/kernels/ops/speculative ci_sglang python3 -m pytest test_eagle_utils.py run_pytest docker exec -w /sglang-checkout/python/sglang/kernels/aot/tests ci_sglang python3 -m pytest test_apply_token_bitmask_inplace.py run_pytest docker exec -w /sglang-checkout/python/sglang/kernels/aot/tests ci_sglang python3 -m pytest test_activation.py run_pytest docker exec -w /sglang-checkout/python/sglang/kernels/aot/tests ci_sglang python3 -m pytest test_topk.py @@ -284,7 +341,11 @@ jobs: exit $failures sgl-kernel-unit-test-2-gpu-amd: - name: ${{ format('sgl-kernel-unit-test-2-gpu-amd (rocm700, linux-{0}-2gpu-sglang)', inputs.runner_arch || 'mi300') }} + name: ${{ format('sgl-kernel-unit-test-2-gpu-amd ({0}, linux-{1}-2gpu-sglang)', matrix.rocm_version, inputs.runner_arch || 'mi300') }} + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(needs.check-changes.outputs.rocm_versions) }} needs: [check-changes, call-gate] if: | always() && !cancelled() && @@ -307,7 +368,7 @@ jobs: run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - name: Start CI container - run: bash scripts/ci/amd/amd_ci_start_container.sh + run: bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version "${{ matrix.rocm_version }}" env: GITHUB_WORKSPACE: ${{ github.workspace }} @@ -325,7 +386,11 @@ jobs: # =============================================== primary ==================================================== stage-a-test-1-gpu-small-amd: - name: ${{ format('stage-a-test-1-gpu-small-amd (rocm700, linux-{0}-1gpu-sglang)', inputs.runner_arch || 'mi300') }} + name: ${{ format('stage-a-test-1-gpu-small-amd ({0}, linux-{1}-1gpu-sglang)', matrix.rocm_version, inputs.runner_arch || 'mi300') }} + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(needs.check-changes.outputs.rocm_versions) }} needs: [check-changes, call-gate] if: | always() && !cancelled() && @@ -348,7 +413,7 @@ jobs: run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - name: Start CI container - run: bash scripts/ci/amd/amd_ci_start_container.sh + run: bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version "${{ matrix.rocm_version }}" env: GITHUB_WORKSPACE: ${{ github.workspace }} @@ -362,7 +427,11 @@ jobs: bash scripts/ci/amd/amd_ci_exec.sh -w "/sglang-checkout/test" python3 run_suite.py --hw amd --suite stage-a-test-1-gpu-small-amd ${{ needs.check-changes.outputs.continue_on_error == 'true' && '--continue-on-error' || '' }} jit-kernel-unit-test-amd: - name: ${{ format('jit-kernel-unit-test-amd (rocm700, linux-{0}-1gpu-sglang)', inputs.runner_arch || 'mi300') }} + name: ${{ format('jit-kernel-unit-test-amd ({0}, linux-{1}-1gpu-sglang)', matrix.rocm_version, inputs.runner_arch || 'mi300') }} + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(needs.check-changes.outputs.rocm_versions) }} needs: [check-changes, call-gate] if: | always() && !cancelled() && @@ -385,7 +454,7 @@ jobs: run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - name: Start CI container - run: bash scripts/ci/amd/amd_ci_start_container.sh + run: bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version "${{ matrix.rocm_version }}" env: GITHUB_WORKSPACE: ${{ github.workspace }} @@ -399,7 +468,11 @@ jobs: bash scripts/ci/amd/amd_ci_exec.sh -w "/sglang-checkout/test" python3 run_suite.py --hw amd --suite jit-kernel-unit-test-amd ${{ needs.check-changes.outputs.continue_on_error == 'true' && '--continue-on-error' || '' }} jit-kernel-benchmark-test-amd: - name: ${{ format('jit-kernel-benchmark-test-amd (rocm700, linux-{0}-1gpu-sglang)', inputs.runner_arch || 'mi300') }} + name: ${{ format('jit-kernel-benchmark-test-amd ({0}, linux-{1}-1gpu-sglang)', matrix.rocm_version, inputs.runner_arch || 'mi300') }} + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(needs.check-changes.outputs.rocm_versions) }} needs: [check-changes, call-gate] if: | always() && !cancelled() && @@ -422,7 +495,7 @@ jobs: run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - name: Start CI container - run: bash scripts/ci/amd/amd_ci_start_container.sh + run: bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version "${{ matrix.rocm_version }}" env: GITHUB_WORKSPACE: ${{ github.workspace }} @@ -441,6 +514,7 @@ jobs: # For scheduled runs: wait jobs are skipped, enabling parallel execution of all stages. wait-for-stage-a-amd: + name: wait-for-stage-a-amd needs: [check-changes, call-gate] if: | always() && @@ -462,7 +536,7 @@ jobs: max-wait-minutes: '240' stage-b-test-1-gpu-small-amd: - name: ${{ format('stage-b-test-1-gpu-small-amd (rocm700, linux-{0}-1gpu-sglang, {1})', inputs.runner_arch || 'mi300', matrix.part) }} + name: ${{ format('stage-b-test-1-gpu-small-amd ({0}, linux-{1}-1gpu-sglang, {2})', matrix.rocm_version, inputs.runner_arch || 'mi300', matrix.part) }} needs: [check-changes, wait-for-stage-a-amd] if: | always() && @@ -476,7 +550,9 @@ jobs: ) strategy: fail-fast: false + max-parallel: 4 matrix: + rocm_version: ${{ fromJson(needs.check-changes.outputs.rocm_versions) }} part: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13] runs-on: ${{ format('linux-{0}-1gpu-sglang', inputs.runner_arch || 'mi300') }} steps: @@ -489,7 +565,7 @@ jobs: run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - name: Start CI container - run: bash scripts/ci/amd/amd_ci_start_container.sh + run: bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version "${{ matrix.rocm_version }}" env: GITHUB_WORKSPACE: ${{ github.workspace }} @@ -502,7 +578,11 @@ jobs: bash scripts/ci/amd/amd_ci_exec.sh -w "/sglang-checkout/test" python3 run_suite.py --hw amd --suite stage-b-test-1-gpu-small-amd --auto-partition-id ${{ matrix.part }} --auto-partition-size 14 --timeout-per-file 2400 ${{ needs.check-changes.outputs.continue_on_error == 'true' && '--continue-on-error' || '' }} stage-b-test-1-gpu-small-amd-nondeterministic: - name: ${{ format('stage-b-test-1-gpu-small-amd-nondeterministic (rocm700, linux-{0}-1gpu-sglang)', inputs.runner_arch || 'mi300') }} + name: ${{ format('stage-b-test-1-gpu-small-amd-nondeterministic ({0}, linux-{1}-1gpu-sglang)', matrix.rocm_version, inputs.runner_arch || 'mi300') }} + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(needs.check-changes.outputs.rocm_versions) }} needs: [check-changes, wait-for-stage-a-amd] if: | always() && @@ -525,7 +605,7 @@ jobs: run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - name: Start CI container - run: bash scripts/ci/amd/amd_ci_start_container.sh + run: bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version "${{ matrix.rocm_version }}" env: GITHUB_WORKSPACE: ${{ github.workspace }} @@ -538,7 +618,7 @@ jobs: bash scripts/ci/amd/amd_ci_exec.sh -w "/sglang-checkout/test" python3 run_suite.py --hw amd --suite stage-b-test-1-gpu-small-amd-nondeterministic --timeout-per-file 3600 ${{ needs.check-changes.outputs.continue_on_error == 'true' && '--continue-on-error' || '' }} stage-b-test-1-gpu-small-amd-mi35x: - name: ${{ format('stage-b-test-1-gpu-small-amd-mi35x (rocm700, {0})', matrix.runner) }} + name: ${{ format('stage-b-test-1-gpu-small-amd-mi35x ({0}, {1})', matrix.rocm_version, matrix.runner) }} needs: [check-changes, wait-for-stage-a-amd] if: | always() && @@ -553,6 +633,7 @@ jobs: strategy: fail-fast: false matrix: + rocm_version: ${{ fromJson(needs.check-changes.outputs.rocm_versions) }} runner: [linux-mi35x-gpu-1] runs-on: ${{matrix.runner}} steps: @@ -565,7 +646,7 @@ jobs: run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - name: Start CI container - run: bash scripts/ci/amd/amd_ci_start_container.sh + run: bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version "${{ matrix.rocm_version }}" env: GITHUB_WORKSPACE: ${{ github.workspace }} @@ -578,7 +659,7 @@ jobs: bash scripts/ci/amd/amd_ci_exec.sh -w "/sglang-checkout/test" python3 run_suite.py --hw amd --suite stage-b-test-1-gpu-small-amd-mi35x ${{ needs.check-changes.outputs.continue_on_error == 'true' && '--continue-on-error' || '' }} stage-b-test-1-gpu-large-amd: - name: ${{ format('stage-b-test-1-gpu-large-amd (rocm700, linux-{0}-1gpu-sglang, {1})', inputs.runner_arch || 'mi300', matrix.part) }} + name: ${{ format('stage-b-test-1-gpu-large-amd ({0}, linux-{1}-1gpu-sglang, {2})', matrix.rocm_version, inputs.runner_arch || 'mi300', matrix.part) }} needs: [check-changes, wait-for-stage-a-amd] if: | always() && @@ -592,7 +673,9 @@ jobs: ) strategy: fail-fast: false + max-parallel: 1 matrix: + rocm_version: ${{ fromJson(needs.check-changes.outputs.rocm_versions) }} part: [0, 1, 2] runs-on: ${{ format('linux-{0}-1gpu-sglang', inputs.runner_arch || 'mi300') }} steps: @@ -605,7 +688,7 @@ jobs: run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - name: Start CI container - run: bash scripts/ci/amd/amd_ci_start_container.sh + run: bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version "${{ matrix.rocm_version }}" env: GITHUB_WORKSPACE: ${{ github.workspace }} @@ -618,7 +701,7 @@ jobs: bash scripts/ci/amd/amd_ci_exec.sh -w "/sglang-checkout/test" python3 run_suite.py --hw amd --suite stage-b-test-1-gpu-large-amd --auto-partition-id ${{ matrix.part }} --auto-partition-size 3 --timeout-per-file 2700 ${{ needs.check-changes.outputs.continue_on_error == 'true' && '--continue-on-error' || '' }} stage-b-test-2-gpu-large-amd: - name: ${{ format('stage-b-test-2-gpu-large-amd (rocm700, linux-{0}-2gpu-sglang, {1})', inputs.runner_arch || 'mi300', matrix.part) }} + name: ${{ format('stage-b-test-2-gpu-large-amd ({0}, linux-{1}-2gpu-sglang, {2})', matrix.rocm_version, inputs.runner_arch || 'mi300', matrix.part) }} needs: [check-changes, wait-for-stage-a-amd] if: | always() && @@ -632,7 +715,9 @@ jobs: ) strategy: fail-fast: false + max-parallel: 1 matrix: + rocm_version: ${{ fromJson(needs.check-changes.outputs.rocm_versions) }} part: [0, 1] runs-on: ${{ format('linux-{0}-2gpu-sglang', inputs.runner_arch || 'mi300') }} steps: @@ -645,7 +730,7 @@ jobs: run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - name: Start CI container - run: bash scripts/ci/amd/amd_ci_start_container.sh + run: bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version "${{ matrix.rocm_version }}" env: GITHUB_WORKSPACE: ${{ github.workspace }} @@ -658,7 +743,7 @@ jobs: bash scripts/ci/amd/amd_ci_exec.sh -w "/sglang-checkout/test" python3 run_suite.py --hw amd --suite stage-b-test-2-gpu-large-amd --auto-partition-id ${{ matrix.part }} --auto-partition-size 2 --timeout-per-file ${{ (inputs.runner_arch || 'mi300') == 'mi300' && 5400 || 2700 }} ${{ needs.check-changes.outputs.continue_on_error == 'true' && '--continue-on-error' || '' }} multimodal-gen-test-1-gpu-amd: - name: ${{ format('multimodal-gen-test-1-gpu-amd (rocm700, linux-{0}-1gpu-sglang, {1})', inputs.runner_arch || 'mi300', matrix.part) }} + name: ${{ format('multimodal-gen-test-1-gpu-amd ({0}, linux-{1}-1gpu-sglang, {2})', matrix.rocm_version, inputs.runner_arch || 'mi300', matrix.part) }} needs: [check-changes, call-gate] if: | always() && !cancelled() && @@ -672,7 +757,9 @@ jobs: ) strategy: fail-fast: false + max-parallel: 1 matrix: + rocm_version: ${{ fromJson(needs.check-changes.outputs.rocm_versions) }} part: [0, 1, 2, 3] runs-on: ${{ format('linux-{0}-1gpu-sglang', inputs.runner_arch || 'mi300') }} steps: @@ -693,7 +780,7 @@ jobs: pattern: wheel-python3.10-cuda12.9 - name: Start CI container - run: bash scripts/ci/amd/amd_ci_start_container.sh + run: bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version "${{ matrix.rocm_version }}" env: GITHUB_WORKSPACE: ${{ github.workspace }} @@ -800,13 +887,13 @@ jobs: if: always() uses: actions/upload-artifact@v4 with: - name: diffusion-failures-amd-1gpu-${{ matrix.part }}-${{ github.run_attempt }} + name: diffusion-failures-amd-1gpu-${{ matrix.rocm_version }}-${{ matrix.part }}-${{ github.run_attempt }} path: diffusion-failures/ if-no-files-found: ignore retention-days: 7 multimodal-gen-test-2-gpu-amd: - name: ${{ format('multimodal-gen-test-2-gpu-amd (rocm700, linux-{0}-2gpu-sglang, {1})', inputs.runner_arch || 'mi300', matrix.part) }} + name: ${{ format('multimodal-gen-test-2-gpu-amd ({0}, linux-{1}-2gpu-sglang, {2})', matrix.rocm_version, inputs.runner_arch || 'mi300', matrix.part) }} needs: [check-changes, call-gate] if: | always() && !cancelled() && @@ -820,7 +907,9 @@ jobs: ) strategy: fail-fast: false + max-parallel: 1 matrix: + rocm_version: ${{ fromJson(needs.check-changes.outputs.rocm_versions) }} # run_suite.py load-balances the suite's parametrized cases and # standalone files over these partitions, so the count is free to # choose. Two shards measured 45-65 min of case work each, inside the @@ -846,7 +935,7 @@ jobs: pattern: wheel-python3.10-cuda12.9 - name: Start CI container - run: bash scripts/ci/amd/amd_ci_start_container.sh + run: bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version "${{ matrix.rocm_version }}" env: GITHUB_WORKSPACE: ${{ github.workspace }} @@ -934,12 +1023,13 @@ jobs: if: always() uses: actions/upload-artifact@v4 with: - name: diffusion-failures-amd-2gpu-${{ matrix.part }}-${{ github.run_attempt }} + name: diffusion-failures-amd-2gpu-${{ matrix.rocm_version }}-${{ matrix.part }}-${{ github.run_attempt }} path: diffusion-failures/ if-no-files-found: ignore retention-days: 7 wait-for-stage-b-amd: + name: wait-for-stage-b-amd needs: [check-changes, call-gate, wait-for-stage-a-amd] if: | always() && @@ -967,7 +1057,7 @@ jobs: max-wait-minutes: '480' stage-c-test-4-gpu-amd: - name: ${{ format('stage-c-test-4-gpu-amd (rocm700, linux-{0}-4gpu-sglang, {1})', inputs.runner_arch || 'mi300', matrix.part) }} + name: ${{ format('stage-c-test-4-gpu-amd ({0}, linux-{1}-4gpu-sglang, {2})', matrix.rocm_version, inputs.runner_arch || 'mi300', matrix.part) }} needs: [check-changes, call-gate, wait-for-stage-b-amd] if: | always() && @@ -982,6 +1072,7 @@ jobs: strategy: fail-fast: false matrix: + rocm_version: ${{ fromJson(needs.check-changes.outputs.rocm_versions) }} part: [0] runs-on: ${{ format('linux-{0}-4gpu-sglang', inputs.runner_arch || 'mi300') }} steps: @@ -994,7 +1085,7 @@ jobs: run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - name: Start CI container - run: bash scripts/ci/amd/amd_ci_start_container.sh + run: bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version "${{ matrix.rocm_version }}" env: GITHUB_WORKSPACE: ${{ github.workspace }} @@ -1023,7 +1114,7 @@ jobs: ${{ needs.check-changes.outputs.continue_on_error == 'true' && '--continue-on-error' || '' }} stage-c-test-large-8-gpu-amd: - name: ${{ format('stage-c-test-large-8-gpu-amd (rocm700, linux-{0}-8gpu-sglang, {1})', inputs.runner_arch || 'mi300', matrix.part) }} + name: ${{ format('stage-c-test-large-8-gpu-amd ({0}, linux-{1}-8gpu-sglang, {2})', matrix.rocm_version, inputs.runner_arch || 'mi300', matrix.part) }} needs: [check-changes, call-gate, wait-for-stage-b-amd] if: | always() && @@ -1039,7 +1130,9 @@ jobs: RUNNER_LABELS: ${{ format('linux-{0}-8gpu-sglang', inputs.runner_arch || 'mi300') }} strategy: fail-fast: false + max-parallel: 2 matrix: + rocm_version: ${{ fromJson(needs.check-changes.outputs.rocm_versions) }} part: [0, 1, 2, 3] runs-on: ${{ format('linux-{0}-8gpu-sglang', inputs.runner_arch || 'mi300') }} steps: @@ -1052,7 +1145,7 @@ jobs: run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - name: Start CI container - run: bash scripts/ci/amd/amd_ci_start_container.sh + run: bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version "${{ matrix.rocm_version }}" env: GITHUB_WORKSPACE: ${{ github.workspace }} @@ -1071,7 +1164,7 @@ jobs: bash scripts/ci/amd/amd_ci_exec.sh -w "/sglang-checkout/test" python3 run_suite.py --hw amd --suite stage-c-test-large-8-gpu-amd --auto-partition-id ${{ matrix.part }} --auto-partition-size 4 --timeout-per-file 5400 ${{ needs.check-changes.outputs.continue_on_error == 'true' && '--continue-on-error' || '' }} stage-c-test-large-8-gpu-amd-mi35x: - name: ${{ format('stage-c-test-large-8-gpu-amd-mi35x (rocm700, {0}, {1})', matrix.runner, matrix.part) }} + name: ${{ format('stage-c-test-large-8-gpu-amd-mi35x ({0}, {1}, {2})', matrix.rocm_version, matrix.runner, matrix.part) }} needs: [check-changes, call-gate, wait-for-stage-b-amd] if: | always() && @@ -1086,6 +1179,7 @@ jobs: strategy: fail-fast: false matrix: + rocm_version: ${{ fromJson(needs.check-changes.outputs.rocm_versions) }} runner: [linux-mi35x-gpu-8] part: [0, 1, 2] runs-on: ${{matrix.runner}} @@ -1099,7 +1193,7 @@ jobs: run: bash scripts/ci/amd/ensure_vram_clear.sh rocm - name: Start CI container - run: bash scripts/ci/amd/amd_ci_start_container.sh + run: bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version "${{ matrix.rocm_version }}" env: GITHUB_WORKSPACE: ${{ github.workspace }} @@ -1111,9 +1205,124 @@ jobs: run: | bash scripts/ci/amd/amd_ci_exec.sh -w "/sglang-checkout/test" python3 run_suite.py --hw amd --suite stage-c-test-large-8-gpu-amd-mi35x --auto-partition-id ${{ matrix.part }} --auto-partition-size 3 --timeout-per-file 3600 ${{ needs.check-changes.outputs.continue_on_error == 'true' && '--continue-on-error' || '' }} + # =============================================== DeepSeek-V4 (MI35x, 8-GPU) ==================================================== + # GSM8K accuracy on the nightly dsv4 suites, ~20min each. Gated as stage-C so + # every PR that touches the main package covers DeepSeek-V4. + stage-c-dsv4-flash-fp4-fp8-amd-mi35x: + name: ${{ format('stage-c-dsv4-flash-fp4-fp8-amd-mi35x ({0}, {1})', matrix.rocm_version, matrix.runner) }} + needs: [check-changes, call-gate, wait-for-stage-b-amd] + if: | + always() && + ( + (contains(format(',{0},', inputs.target_stage || inputs.target_stage_select), ',stage-c-dsv4-flash-fp4-fp8-amd-mi35x,')) || + ( + !(inputs.target_stage || inputs.target_stage_select) && + ((github.event_name == 'schedule') || (!failure() && !cancelled())) && + ((needs.check-changes.outputs.main_package == 'true') || (needs.check-changes.outputs.sgl_kernel == 'true')) + ) + ) + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(needs.check-changes.outputs.rocm_versions) }} + runner: [linux-mi35x-gpu-8] + runs-on: ${{matrix.runner}} + steps: + - name: Checkout code + uses: actions/checkout@v4 + with: + ref: ${{ inputs.pr_head_sha || inputs.ref || github.sha }} + + - name: Ensure VRAM is clear + run: bash scripts/ci/amd/ensure_vram_clear.sh rocm + + - name: Setup docker (ROCm) + run: | + touch github_summary.md + bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version "${{ matrix.rocm_version }}" + env: + GITHUB_WORKSPACE: ${{ github.workspace }} + ENABLE_CACHE_HOST: "1" + + - name: Install dependencies + run: | + # --skip-test-time-deps: GSM8K + bench_one_batch_server don't need lmms-eval / human-eval. + bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps + bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate + + - name: Accuracy Test MI35x ROCm (8-GPU DeepSeek-V4-Flash FP4 + FP8) + timeout-minutes: 60 + run: | + > github_summary.md # Clear summary file + # SGLANG_DSV4_ACCURACY_ONLY=1 makes the dsv4 test files skip their perf test + # (test_b_perf_8k_1k); only the GSM8K accuracy test runs in this workflow. + bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ + -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ + -e SGLANG_DSV4_ACCURACY_ONLY=1 \ + -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ + python3 run_suite.py --hw amd --suite nightly-amd-8-gpu-mi35x-deepseek-v4-flash --nightly --timeout-per-file 3600 ${{ needs.check-changes.outputs.continue_on_error == 'true' && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$? + echo "$(> $GITHUB_STEP_SUMMARY || true + exit ${TEST_EXIT_CODE:-0} + + stage-c-dsv4-pro-fp4-amd-mi35x: + name: ${{ format('stage-c-dsv4-pro-fp4-amd-mi35x ({0}, {1})', matrix.rocm_version, matrix.runner) }} + needs: [check-changes, call-gate, wait-for-stage-b-amd] + if: | + always() && + ( + (contains(format(',{0},', inputs.target_stage || inputs.target_stage_select), ',stage-c-dsv4-pro-fp4-amd-mi35x,')) || + ( + !(inputs.target_stage || inputs.target_stage_select) && + ((github.event_name == 'schedule') || (!failure() && !cancelled())) && + ((needs.check-changes.outputs.main_package == 'true') || (needs.check-changes.outputs.sgl_kernel == 'true')) + ) + ) + strategy: + fail-fast: false + matrix: + rocm_version: ${{ fromJson(needs.check-changes.outputs.rocm_versions) }} + runner: [linux-mi35x-gpu-8] + runs-on: ${{matrix.runner}} + steps: + - name: Checkout code + uses: actions/checkout@v4 + with: + ref: ${{ inputs.pr_head_sha || inputs.ref || github.sha }} + + - name: Ensure VRAM is clear + run: bash scripts/ci/amd/ensure_vram_clear.sh rocm + + - name: Setup docker (ROCm) + run: | + touch github_summary.md + bash scripts/ci/amd/amd_ci_start_container.sh --rocm-version "${{ matrix.rocm_version }}" + env: + GITHUB_WORKSPACE: ${{ github.workspace }} + ENABLE_CACHE_HOST: "1" + + - name: Install dependencies + run: | + # --skip-test-time-deps: GSM8K + bench_one_batch_server don't need lmms-eval / human-eval. + bash scripts/ci/amd/amd_ci_install_dependency.sh --skip-test-time-deps + bash scripts/ci/amd/amd_ci_exec.sh pip install tabulate + + - name: Accuracy Test MI35x ROCm (8-GPU DeepSeek-V4-Pro FP4) + timeout-minutes: 60 + run: | + > github_summary.md # Clear summary file + # SGLANG_DSV4_ACCURACY_ONLY=1 makes the dsv4 test files skip their perf test + # (test_b_perf_8k_1k); only the GSM8K accuracy test runs in this workflow. + bash scripts/ci/amd/amd_ci_exec.sh -w /sglang-checkout/test \ + -e GITHUB_STEP_SUMMARY="/sglang-checkout/github_summary.md" \ + -e SGLANG_DSV4_ACCURACY_ONLY=1 \ + -e SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D=1 \ + python3 registered/amd/test_deepseek_v4_pro_fp4.py || TEST_EXIT_CODE=$? + echo "$(> $GITHUB_STEP_SUMMARY || true + exit ${TEST_EXIT_CODE:-0} + # =============================================== Disaggregation ==================================================== stage-b-test-large-8-gpu-mi35x-disaggregation-amd: - name: ${{ format('stage-b-test-large-8-gpu-mi35x-disaggregation-amd (rocm700, {0})', matrix.runner) }} + name: ${{ format('stage-b-test-large-8-gpu-mi35x-disaggregation-amd ({0}, {1})', matrix.rocm_version, matrix.runner) }} needs: [check-changes, wait-for-stage-a-amd] if: | always() && @@ -1128,6 +1337,7 @@ jobs: strategy: fail-fast: false matrix: + rocm_version: ${{ fromJson(needs.check-changes.outputs.rocm_versions) }} runner: [linux-mi35x-gpu-8.fabric] runs-on: ${{matrix.runner}} @@ -1180,7 +1390,7 @@ jobs: echo "=== Host RDMA Check Complete ===" - name: Start Special Container - run: bash scripts/ci/amd/amd_ci_start_container_disagg.sh + run: bash scripts/ci/amd/amd_ci_start_container_disagg.sh --rocm-version "${{ matrix.rocm_version }}" env: GITHUB_WORKSPACE: ${{ github.workspace }} @@ -1247,6 +1457,7 @@ jobs: -w "/sglang-checkout/test" python3 run_suite.py --hw amd --suite stage-b-test-large-8-gpu-mi35x-disaggregation-amd --timeout-per-file 1800 ${{ needs.check-changes.outputs.continue_on_error == 'true' && '--continue-on-error' || '' }} pr-test-amd-finish: + name: pr-test-amd-finish needs: [ call-gate, @@ -1271,6 +1482,9 @@ jobs: stage-c-test-4-gpu-amd, stage-c-test-large-8-gpu-amd, stage-c-test-large-8-gpu-amd-mi35x, + + stage-c-dsv4-flash-fp4-fp8-amd-mi35x, + stage-c-dsv4-pro-fp4-amd-mi35x, ] if: always() runs-on: ubuntu-latest diff --git a/.github/workflows/pr-test-extra.yml b/.github/workflows/pr-test-extra.yml index fc15908c112f..cd0faf07231a 100644 --- a/.github/workflows/pr-test-extra.yml +++ b/.github/workflows/pr-test-extra.yml @@ -50,13 +50,14 @@ on: type: boolean default: false skip_pr_test_health_check: - description: "Skip PR test health check fast-fail (e.g. for release branch cuts)" + description: "Skip PR test health check fail-fast (e.g. for release branch cuts)" required: false type: boolean default: false concurrency: - group: pr-test-extra-${{ github.event_name }}-${{ github.head_ref || github.ref_name || 'default' }}-${{ inputs.git_ref || 'all' }} + # Keys on the PR number: two forks can share a head_ref and would cancel each other. + group: pr-test-extra-${{ github.event_name }}-${{ github.event.pull_request.number || github.ref_name || 'default' }}-${{ inputs.git_ref || 'all' }} cancel-in-progress: ${{ github.event_name != 'workflow_call' }} env: @@ -143,98 +144,48 @@ jobs: (needs.call-gate.result == 'success' || needs.call-gate.result == 'skipped') uses: ./.github/workflows/_pr-test-rust-ext-build.yml with: - runs_on: 1-gpu-5090 + runs_on: ubuntu-22.04 artifact_name: rust-ext-x86_64-extra git_ref: ${{ inputs.git_ref || '' }} skip_pr_test_health_check: ${{ inputs.skip_pr_test_health_check == true }} secrets: inherit - # =============================================== extra-a (1-/2-gpu) =============================================== - extra-a-test-1-gpu-small: - needs: [check-changes, call-gate, sgl-kernel-build-wheels, rust-ext-build] + simulator-test-cpu: + needs: [check-changes, call-gate, rust-ext-build] if: ${{ !failure() && !cancelled() && needs.check-changes.result == 'success' && (needs.call-gate.result == 'success' || needs.call-gate.result == 'skipped') }} - uses: ./.github/workflows/_pr-test-stage.yml - with: - self_name: extra-a-test-1-gpu-small - runner_config: 1-gpu-small - check_changes: ${{ toJson(needs.check-changes.outputs) }} - caller_inputs: ${{ toJson(inputs) }} - partitions: ${{ needs.check-changes.outputs.partitions }} - run_timeout_minutes: '60' - rust_ext_artifact: ${{ needs.rust-ext-build.outputs.artifact_name }} - secrets: inherit - - extra-a-test-1-gpu-large: - needs: [check-changes, call-gate, sgl-kernel-build-wheels, rust-ext-build] - if: ${{ !failure() && !cancelled() && needs.check-changes.result == 'success' && (needs.call-gate.result == 'success' || needs.call-gate.result == 'skipped') }} - uses: ./.github/workflows/_pr-test-stage.yml + uses: ./.github/workflows/_pr-test-simulator-cpu.yml with: - self_name: extra-a-test-1-gpu-large - runner_config: 1-gpu-large check_changes: ${{ toJson(needs.check-changes.outputs) }} caller_inputs: ${{ toJson(inputs) }} - partitions: ${{ needs.check-changes.outputs.partitions }} - run_timeout_minutes: '60' - timeout_per_file: '1800' rust_ext_artifact: ${{ needs.rust-ext-build.outputs.artifact_name }} - secrets: inherit + # No `secrets: inherit`: this hosted CPU job has no secret consumer. - extra-a-test-2-gpu-large: + # =============================================== extra-a / extra-b =============================================== + # `name:` keeps the job names wait-for-jobs and check-pr-test-health match on. + extra-test: + name: ${{ matrix.stage }}-test-${{ matrix.runner_config }} needs: [check-changes, call-gate, sgl-kernel-build-wheels, rust-ext-build] if: ${{ !failure() && !cancelled() && needs.check-changes.result == 'success' && (needs.call-gate.result == 'success' || needs.call-gate.result == 'skipped') }} + strategy: + fail-fast: false + matrix: + include: + - { stage: extra-a, runner_config: 1-gpu-small, timeout_per_file: '' } + - { stage: extra-a, runner_config: 1-gpu-large, timeout_per_file: '1800' } + - { stage: extra-a, runner_config: 2-gpu-large, timeout_per_file: '' } + - { stage: extra-b, runner_config: 4-gpu-h100, timeout_per_file: '3600' } + - { stage: extra-b, runner_config: 4-gpu-b200, timeout_per_file: '1800' } + - { stage: extra-b, runner_config: 8-gpu-h200, timeout_per_file: '' } + - { stage: extra-b, runner_config: 8-gpu-b300, timeout_per_file: '3600' } uses: ./.github/workflows/_pr-test-stage.yml with: - self_name: extra-a-test-2-gpu-large - runner_config: 2-gpu-large - check_changes: ${{ toJson(needs.check-changes.outputs) }} - caller_inputs: ${{ toJson(inputs) }} - partitions: ${{ needs.check-changes.outputs.partitions }} - run_timeout_minutes: '60' - rust_ext_artifact: ${{ needs.rust-ext-build.outputs.artifact_name }} - secrets: inherit - - # =============================================== extra-b (4-/8-gpu) =============================================== - extra-b-test-4-gpu-h100: - needs: [check-changes, call-gate, sgl-kernel-build-wheels, rust-ext-build] - if: ${{ !failure() && !cancelled() && needs.check-changes.result == 'success' && (needs.call-gate.result == 'success' || needs.call-gate.result == 'skipped') }} - uses: ./.github/workflows/_pr-test-stage.yml - with: - self_name: extra-b-test-4-gpu-h100 - runner_config: 4-gpu-h100 - check_changes: ${{ toJson(needs.check-changes.outputs) }} - caller_inputs: ${{ toJson(inputs) }} - partitions: ${{ needs.check-changes.outputs.partitions }} - run_timeout_minutes: '60' - timeout_per_file: '3600' - rust_ext_artifact: ${{ needs.rust-ext-build.outputs.artifact_name }} - secrets: inherit - - extra-b-test-4-gpu-b200: - needs: [check-changes, call-gate, sgl-kernel-build-wheels, rust-ext-build] - if: ${{ !failure() && !cancelled() && needs.check-changes.result == 'success' && (needs.call-gate.result == 'success' || needs.call-gate.result == 'skipped') }} - uses: ./.github/workflows/_pr-test-stage.yml - with: - self_name: extra-b-test-4-gpu-b200 - runner_config: 4-gpu-b200 - check_changes: ${{ toJson(needs.check-changes.outputs) }} - caller_inputs: ${{ toJson(inputs) }} - partitions: ${{ needs.check-changes.outputs.partitions }} - run_timeout_minutes: '60' - timeout_per_file: '1800' - rust_ext_artifact: ${{ needs.rust-ext-build.outputs.artifact_name }} - secrets: inherit - - extra-b-test-8-gpu-h200: - needs: [check-changes, call-gate, sgl-kernel-build-wheels, rust-ext-build] - if: ${{ !failure() && !cancelled() && needs.check-changes.result == 'success' && (needs.call-gate.result == 'success' || needs.call-gate.result == 'skipped') }} - uses: ./.github/workflows/_pr-test-stage.yml - with: - self_name: extra-b-test-8-gpu-h200 - runner_config: 8-gpu-h200 + self_name: ${{ matrix.stage }}-test-${{ matrix.runner_config }} + runner_config: ${{ matrix.runner_config }} check_changes: ${{ toJson(needs.check-changes.outputs) }} caller_inputs: ${{ toJson(inputs) }} partitions: ${{ needs.check-changes.outputs.partitions }} run_timeout_minutes: '60' + timeout_per_file: ${{ matrix.timeout_per_file }} rust_ext_artifact: ${{ needs.rust-ext-build.outputs.artifact_name }} secrets: inherit @@ -249,14 +200,17 @@ jobs: call-gate, sgl-kernel-build-wheels, rust-ext-build, - extra-a-test-1-gpu-small, - extra-a-test-1-gpu-large, - extra-a-test-2-gpu-large, - extra-b-test-4-gpu-h100, - extra-b-test-4-gpu-b200, - extra-b-test-8-gpu-h200, + simulator-test-cpu, + extra-test, ] - if: always() + # Without this guard an unrelated label starts a run that finishes green with + # nothing executed, over the real failure; `skipped` is what pr-states.yml reads. + if: | + always() && + (github.event_name != 'pull_request' || + github.event.action != 'labeled' || + github.event.label.name == 'run-ci' || + github.event.label.name == 'run-ci-extra') runs-on: ubuntu-latest steps: - name: Check all dependent job statuses @@ -286,10 +240,14 @@ jobs: # there (initial completion) and pull_request_target (push / label). notify-pr-states: needs: [pr-test-extra-finish] + # Same guard; pr-states.yml subscribes to labeled/unlabeled itself, so nothing is lost. if: | always() && github.event_name == 'pull_request' && - github.event.pull_request.head.repo.full_name == github.repository + github.event.pull_request.head.repo.full_name == github.repository && + (github.event.action != 'labeled' || + github.event.label.name == 'run-ci' || + github.event.label.name == 'run-ci-extra') runs-on: ubuntu-latest steps: - name: Dispatch pr-states refresh diff --git a/.github/workflows/pr-test-jit-kernel.yml b/.github/workflows/pr-test-jit-kernel.yml index f94ccb9d25b6..e8685122fbd8 100644 --- a/.github/workflows/pr-test-jit-kernel.yml +++ b/.github/workflows/pr-test-jit-kernel.yml @@ -13,9 +13,6 @@ on: description: 'Artifact of prebuilt Rust extension modules, from rust-ext-build. Empty, or a download that fails, falls back to the cache; a miss there compiles during install.' type: string default: '' - runner_config: - required: true - type: string runs_on_map: required: true type: string @@ -43,15 +40,37 @@ env: SKIP_PR_TEST_HEALTH_CHECK: ${{ inputs.skip_pr_test_health_check == true && 'true' || 'false' }} jobs: - jit-kernel-unit-test: + # `name:` reproduces the job names this workflow published before the table. + jit-kernel-test: + # The job name is the suite it runs, with the stage prefix swapped for this + # workflow's, so a row declares a (suite, runner) pair and nothing else. + name: jit-kernel-${{ matrix.suite }}${{ matrix.shard }} # Runs whenever call-jit-kernel-tests dispatches this workflow. That caller is the # single gate (PR jit_kernel changes, or scheduled/parallel-dispatch full runs), so # the sub-jobs no longer re-exclude schedule/parallel-dispatch here. strategy: fail-fast: false matrix: - partition: [0, 1] - runs-on: 1-gpu-h100 + include: + - suite: unit-test-1-gpu-large + runner_config: 1-gpu-large + shard: " (0)" + suite_args: --auto-partition-id 0 --auto-partition-size 2 --fork-worker-batch-size 20 + - suite: unit-test-1-gpu-large + runner_config: 1-gpu-large + shard: " (1)" + suite_args: --auto-partition-id 1 --auto-partition-size 2 --fork-worker-batch-size 20 + - suite: unit-test-1-gpu-small + runner_config: 1-gpu-small + - suite: unit-test-4-gpu-b200 + runner_config: 4-gpu-b200 + - suite: unit-test-8-gpu-h200 + runner_config: 8-gpu-h200 + # Alone among these, it has never run the health check. + skip_health_check: true + - suite: benchmark-test-1-gpu-large + runner_config: 1-gpu-large + runs-on: ${{ fromJson(inputs.runs_on_map)[matrix.runner_config] }} timeout-minutes: 240 steps: - uses: actions/checkout@v4 @@ -59,6 +78,7 @@ jobs: ref: ${{ inputs.git_ref || github.sha }} - uses: ./.github/actions/check-pr-test-health + if: ${{ !matrix.skip_health_check }} - uses: ./.github/actions/check-maintenance @@ -86,141 +106,7 @@ jobs: CUSTOM_BUILD_SGL_KERNEL=${{ inputs.sgl_kernel }} bash scripts/ci/cuda/ci_install_dependency.sh diffusion - name: Run test - timeout-minutes: 30 - run: | - cd test/ - python3 run_suite.py --hw cuda \ - --suite base-b-kernel-unit-test-1-gpu-large \ - --auto-partition-id ${{ matrix.partition }} \ - --auto-partition-size 2 \ - --fork-worker-batch-size 20 - - jit-kernel-multigpu-unit-test: - # Runs whenever call-jit-kernel-tests dispatches this workflow. That caller is the - # single gate (PR jit_kernel changes, or scheduled/parallel-dispatch full runs), so - # the sub-jobs no longer re-exclude schedule/parallel-dispatch here. - runs-on: 8-gpu-h200 - timeout-minutes: 240 - steps: - - uses: actions/checkout@v4 - with: - ref: ${{ inputs.git_ref || github.sha }} - - - uses: ./.github/actions/check-maintenance - - - name: Cleanup - if: inputs.sgl_kernel == 'true' - run: | - ls -alh python/sglang/kernels/aot/dist || true - rm -rf python/sglang/kernels/aot/dist/* || true - - - name: Download artifacts - if: inputs.sgl_kernel == 'true' - uses: actions/download-artifact@v4 - with: - path: python/sglang/kernels/aot/dist/ - merge-multiple: true - pattern: wheel-python3.10-cuda13.0 - - - uses: ./.github/actions/download-rust-ext - with: - artifact_name: ${{ inputs.rust_ext_artifact }} - - - name: Install dependencies - timeout-minutes: 20 - run: | - CUSTOM_BUILD_SGL_KERNEL=${{ inputs.sgl_kernel }} bash scripts/ci/cuda/ci_install_dependency.sh diffusion - - - name: Run multi-GPU test - timeout-minutes: 45 - run: | - cd test/ - python3 run_suite.py --hw cuda --suite base-b-kernel-unit-test-8-gpu-h200 - - jit-kernel-benchmark-test: - # Runs whenever call-jit-kernel-tests dispatches this workflow. That caller is the - # single gate (PR jit_kernel changes, or scheduled/parallel-dispatch full runs), so - # the sub-jobs no longer re-exclude schedule/parallel-dispatch here. - runs-on: 1-gpu-h100 - timeout-minutes: 240 - steps: - - uses: actions/checkout@v4 - with: - ref: ${{ inputs.git_ref || github.sha }} - - - uses: ./.github/actions/check-pr-test-health - - - uses: ./.github/actions/check-maintenance - - - name: Cleanup - if: inputs.sgl_kernel == 'true' - run: | - ls -alh python/sglang/kernels/aot/dist || true - rm -rf python/sglang/kernels/aot/dist/* || true - - - name: Download artifacts - if: inputs.sgl_kernel == 'true' - uses: actions/download-artifact@v4 - with: - path: python/sglang/kernels/aot/dist/ - merge-multiple: true - pattern: wheel-python3.10-cuda13.0 - - - uses: ./.github/actions/download-rust-ext - with: - artifact_name: ${{ inputs.rust_ext_artifact }} - - - name: Install dependencies - timeout-minutes: 20 - run: | - CUSTOM_BUILD_SGL_KERNEL=${{ inputs.sgl_kernel }} bash scripts/ci/cuda/ci_install_dependency.sh diffusion - - - name: Run benchmark tests - timeout-minutes: 45 - run: | - cd test/ - python3 run_suite.py --hw cuda --suite base-b-kernel-benchmark-test-1-gpu-large - - jit-kernel-b200-test: - # Runs whenever call-jit-kernel-tests dispatches this workflow. That caller is the - # single gate (PR jit_kernel changes, or scheduled/parallel-dispatch full runs), so - # the sub-jobs no longer re-exclude schedule/parallel-dispatch here. - runs-on: ${{ fromJson(inputs.runs_on_map)[inputs.runner_config] }} - timeout-minutes: 240 - steps: - - uses: actions/checkout@v4 - with: - ref: ${{ inputs.git_ref || github.sha }} - - - uses: ./.github/actions/check-pr-test-health - - - uses: ./.github/actions/check-maintenance - - - name: Cleanup - if: inputs.sgl_kernel == 'true' - run: | - ls -alh python/sglang/kernels/aot/dist || true - rm -rf python/sglang/kernels/aot/dist/* || true - - - name: Download artifacts - if: inputs.sgl_kernel == 'true' - uses: actions/download-artifact@v4 - with: - path: python/sglang/kernels/aot/dist/ - merge-multiple: true - pattern: wheel-python3.10-cuda13.0 - - - uses: ./.github/actions/download-rust-ext - with: - artifact_name: ${{ inputs.rust_ext_artifact }} - - - name: Install dependencies - timeout-minutes: 20 - run: | - CUSTOM_BUILD_SGL_KERNEL=${{ inputs.sgl_kernel }} bash scripts/ci/cuda/ci_install_dependency.sh diffusion - - - name: Run B200 diffusion test - timeout-minutes: 30 + timeout-minutes: 60 run: | cd test/ - python3 run_suite.py --hw cuda --suite base-b-kernel-unit-test-4-gpu-b200 + python3 run_suite.py --hw cuda --suite base-b-kernel-${{ matrix.suite }} ${{ matrix.suite_args }} diff --git a/.github/workflows/pr-test-multimodal-gen.yml b/.github/workflows/pr-test-multimodal-gen.yml index c06f1eabb82c..f15e8a45b7d2 100644 --- a/.github/workflows/pr-test-multimodal-gen.yml +++ b/.github/workflows/pr-test-multimodal-gen.yml @@ -9,10 +9,6 @@ on: sgl_kernel: required: true type: string - rust_ext_artifact: - description: 'Artifact of prebuilt Rust extension modules, from rust-ext-build. Empty, or a download that fails, falls back to the cache; a miss there compiles during install.' - type: string - default: '' runner_config: required: true type: string @@ -107,10 +103,6 @@ jobs: merge-multiple: true pattern: wheel-python3.10-cuda* - - uses: ./.github/actions/download-rust-ext - with: - artifact_name: ${{ inputs.rust_ext_artifact }} - - name: Install dependencies timeout-minutes: 20 run: | @@ -119,17 +111,16 @@ jobs: timeout-minutes: 240 env: RUNAI_STREAMER_MEMORY_LIMIT: 0 - CONTINUE_ON_ERROR_FLAG: ${{ inputs.continue_on_error == 'true' && '--continue-on-error' || '' }} - PARTITION_PLAN_JSON: ${{ needs.compute-diffusion-partitions.outputs.plan-1gpu }} + DIFFUSION_CONTINUE_ON_ERROR: ${{ inputs.continue_on_error }} + DIFFUSION_PARTITION_ID: ${{ matrix.part }} + DIFFUSION_TOTAL_PARTITIONS: ${{ needs.compute-diffusion-partitions.outputs['partition-count-1gpu'] }} + DIFFUSION_PARTITION_PLAN_JSON: ${{ needs.compute-diffusion-partitions.outputs.plan-1gpu }} SGLANG_DIFFUSION_ARTIFACT_DIR: ${{ github.workspace }}/diffusion-failures run: | - cd python - python3 sglang/multimodal_gen/test/run_suite.py \ - --suite 1-gpu \ - --partition-id ${{ matrix.part }} \ - --total-partitions ${{ needs.compute-diffusion-partitions.outputs['partition-count-1gpu'] }} \ - --partition-plan-json "$PARTITION_PLAN_JSON" \ - $CONTINUE_ON_ERROR_FLAG + python3 test/run_suite.py \ + --hw cuda \ + --suite base-b-test-diffusion-1-gpu-h100 \ + --timeout-per-file 14400 - name: Upload execution report if: always() @@ -139,7 +130,7 @@ jobs: path: | python/sglang/multimodal_gen/test/execution_report_*.json python/diffusion-results.json - retention-days: 1 + retention-days: 7 - name: Upload diffusion failure artifacts if: always() @@ -179,10 +170,6 @@ jobs: merge-multiple: true pattern: wheel-python3.10-cuda* - - uses: ./.github/actions/download-rust-ext - with: - artifact_name: ${{ inputs.rust_ext_artifact }} - - name: Install dependencies timeout-minutes: 20 run: | @@ -192,13 +179,13 @@ jobs: timeout-minutes: 120 env: RUNAI_STREAMER_MEMORY_LIMIT: 0 - CONTINUE_ON_ERROR_FLAG: ${{ inputs.continue_on_error == 'true' && '--continue-on-error' || '' }} + DIFFUSION_CONTINUE_ON_ERROR: ${{ inputs.continue_on_error }} SGLANG_DIFFUSION_ARTIFACT_DIR: ${{ github.workspace }}/diffusion-failures run: | - cd python - python3 sglang/multimodal_gen/test/run_suite.py \ - --suite 1-gpu-5090 \ - $CONTINUE_ON_ERROR_FLAG + python3 test/run_suite.py \ + --hw cuda \ + --suite base-b-test-diffusion-1-gpu-5090 \ + --timeout-per-file 7200 - name: Upload execution report if: always() @@ -248,10 +235,6 @@ jobs: merge-multiple: true pattern: wheel-python3.10-cuda* - - uses: ./.github/actions/download-rust-ext - with: - artifact_name: ${{ inputs.rust_ext_artifact }} - - name: Install dependencies timeout-minutes: 20 run: | @@ -261,13 +244,13 @@ jobs: timeout-minutes: 60 env: RUNAI_STREAMER_MEMORY_LIMIT: 0 - CONTINUE_ON_ERROR_FLAG: ${{ inputs.continue_on_error == 'true' && '--continue-on-error' || '' }} + DIFFUSION_CONTINUE_ON_ERROR: ${{ inputs.continue_on_error }} SGLANG_DIFFUSION_ARTIFACT_DIR: ${{ github.workspace }}/diffusion-bcg-artifacts run: | - cd python - python3 sglang/multimodal_gen/test/run_suite.py \ - --suite bcg-diffusion \ - $CONTINUE_ON_ERROR_FLAG + python3 test/run_suite.py \ + --hw cuda \ + --suite base-b-test-diffusion-bcg-1-gpu-h100 \ + --timeout-per-file 3600 - name: Upload BCG diffusion artifacts if: always() @@ -292,11 +275,14 @@ jobs: ((github.event_name == 'schedule' || inputs.test_parallel_dispatch == 'true') || (inputs.caller_needs_failure != 'true' && !cancelled())) && inputs.multimodal_gen == 'true' runs-on: 2-gpu-h100 - timeout-minutes: 240 + timeout-minutes: 45 strategy: fail-fast: false matrix: ${{ fromJson(needs.compute-diffusion-partitions.outputs.matrix-2gpu) }} steps: + - name: Record retry deadline + run: echo "SGLANG_DIFFUSION_RETRY_DEADLINE=$(( $(date +%s) + 2400 ))" >> "$GITHUB_ENV" + - name: Checkout code uses: actions/checkout@v4 with: @@ -314,32 +300,27 @@ jobs: merge-multiple: true pattern: wheel-python3.10-cuda* - - uses: ./.github/actions/download-rust-ext - with: - artifact_name: ${{ inputs.rust_ext_artifact }} - - name: Install dependencies timeout-minutes: 20 run: | CUSTOM_BUILD_SGL_KERNEL=${{inputs.sgl_kernel}} bash scripts/ci/cuda/ci_install_dependency.sh diffusion - name: Run diffusion server tests - timeout-minutes: 240 + timeout-minutes: 45 env: - HF_TOKEN: ${{ secrets.MINIMAX_H3_HF_TOKEN || secrets.SGLANG_DIFFUSION_CI_HF_TOKEN || secrets.HF_TOKEN }} - HUGGING_FACE_HUB_TOKEN: ${{ secrets.MINIMAX_H3_HF_TOKEN || secrets.SGLANG_DIFFUSION_CI_HF_TOKEN || secrets.HF_TOKEN }} + HF_TOKEN: ${{ secrets.SGLANG_DIFFUSION_CI_HF_TOKEN || secrets.HF_TOKEN }} + HUGGING_FACE_HUB_TOKEN: ${{ secrets.SGLANG_DIFFUSION_CI_HF_TOKEN || secrets.HF_TOKEN }} RUNAI_STREAMER_MEMORY_LIMIT: 0 - CONTINUE_ON_ERROR_FLAG: ${{ inputs.continue_on_error == 'true' && '--continue-on-error' || '' }} - PARTITION_PLAN_JSON: ${{ needs.compute-diffusion-partitions.outputs.plan-2gpu }} + DIFFUSION_CONTINUE_ON_ERROR: ${{ inputs.continue_on_error }} + DIFFUSION_PARTITION_ID: ${{ matrix.part }} + DIFFUSION_TOTAL_PARTITIONS: ${{ needs.compute-diffusion-partitions.outputs['partition-count-2gpu'] }} + DIFFUSION_PARTITION_PLAN_JSON: ${{ needs.compute-diffusion-partitions.outputs.plan-2gpu }} SGLANG_DIFFUSION_ARTIFACT_DIR: ${{ github.workspace }}/diffusion-failures run: | - cd python - python3 sglang/multimodal_gen/test/run_suite.py \ - --suite 2-gpu \ - --partition-id ${{ matrix.part }} \ - --total-partitions ${{ needs.compute-diffusion-partitions.outputs['partition-count-2gpu'] }} \ - --partition-plan-json "$PARTITION_PLAN_JSON" \ - $CONTINUE_ON_ERROR_FLAG + python3 test/run_suite.py \ + --hw cuda \ + --suite base-b-test-diffusion-2-gpu-h100 \ + --timeout-per-file 14400 - name: Upload execution report if: always() @@ -349,7 +330,7 @@ jobs: path: | python/sglang/multimodal_gen/test/execution_report_*.json python/diffusion-results.json - retention-days: 1 + retention-days: 7 - name: Upload diffusion failure artifacts if: always() @@ -389,10 +370,6 @@ jobs: merge-multiple: true pattern: wheel-python3.10-cuda* - - uses: ./.github/actions/download-rust-ext - with: - artifact_name: ${{ inputs.rust_ext_artifact }} - - name: Install dependencies timeout-minutes: 20 run: | @@ -402,12 +379,12 @@ jobs: timeout-minutes: 240 env: RUNAI_STREAMER_MEMORY_LIMIT: 0 - CONTINUE_ON_ERROR_FLAG: ${{ inputs.continue_on_error == 'true' && '--continue-on-error' || '' }} + DIFFUSION_CONTINUE_ON_ERROR: ${{ inputs.continue_on_error }} run: | - cd python - python3 sglang/multimodal_gen/test/run_suite.py \ - --suite component-accuracy \ - $CONTINUE_ON_ERROR_FLAG + python3 test/run_suite.py \ + --hw cuda \ + --suite base-b-test-diffusion-component-2-gpu-h100 \ + --timeout-per-file 14400 - uses: ./.github/actions/upload-cuda-coredumps if: always() @@ -438,10 +415,6 @@ jobs: merge-multiple: true pattern: wheel-python3.10-cuda* - - uses: ./.github/actions/download-rust-ext - with: - artifact_name: ${{ inputs.rust_ext_artifact }} - - name: Install dependencies timeout-minutes: 20 env: @@ -453,13 +426,13 @@ jobs: timeout-minutes: 240 env: RUNAI_STREAMER_MEMORY_LIMIT: 0 - CONTINUE_ON_ERROR_FLAG: ${{ inputs.continue_on_error == 'true' && '--continue-on-error' || '' }} + DIFFUSION_CONTINUE_ON_ERROR: ${{ inputs.continue_on_error }} SGLANG_DIFFUSION_ARTIFACT_DIR: ${{ github.workspace }}/diffusion-failures run: | - cd python - python3 sglang/multimodal_gen/test/run_suite.py \ - --suite 1-gpu-b200 \ - $CONTINUE_ON_ERROR_FLAG + python3 test/run_suite.py \ + --hw cuda \ + --suite base-b-test-diffusion-1-gpu-b200 \ + --timeout-per-file 14400 - name: Upload diffusion failure artifacts if: always() @@ -507,10 +480,6 @@ jobs: merge-multiple: true pattern: wheel-python3.10-cuda* - - uses: ./.github/actions/download-rust-ext - with: - artifact_name: ${{ inputs.rust_ext_artifact }} - - name: Install dependencies timeout-minutes: 20 run: | @@ -519,8 +488,10 @@ jobs: - name: Run diffusion unit tests timeout-minutes: 60 run: | - cd python - python3 sglang/multimodal_gen/test/run_suite.py --suite unit + python3 test/run_suite.py \ + --hw cuda \ + --suite base-b-test-diffusion-unit-1-gpu-h100 \ + --timeout-per-file 3600 diffusion-coverage-check: needs: [multimodal-gen-test-1-gpu, multimodal-gen-test-2-gpu] diff --git a/.github/workflows/pr-test-musa.yml b/.github/workflows/pr-test-musa.yml index c85273d3a585..c8543df0fa86 100644 --- a/.github/workflows/pr-test-musa.yml +++ b/.github/workflows/pr-test-musa.yml @@ -222,9 +222,9 @@ jobs: - name: Run sgl-kernel test timeout-minutes: 20 run: | - pytest python/sglang/kernels/aot/tests/speculative/test_eagle_utils.py - pytest python/sglang/kernels/aot/tests/speculative/test_ngram_utils.py - pytest python/sglang/kernels/aot/tests/speculative/test_speculative_sampling.py + pytest test/registered/kernels/ops/speculative/test_eagle_utils.py + pytest test/registered/kernels/ops/speculative/test_ngram_utils.py + pytest test/registered/kernels/ops/speculative/test_speculative_sampling.py pytest python/sglang/kernels/aot/tests/test_torch_defaults_reset.py diff --git a/.github/workflows/pr-test-npu.yml b/.github/workflows/pr-test-npu.yml index f677db7d9ef1..afb2e531ab56 100644 --- a/.github/workflows/pr-test-npu.yml +++ b/.github/workflows/pr-test-npu.yml @@ -3,12 +3,20 @@ name: PR Test (NPU) on: push: branches: [ main ] + schedule: + - cron: '0 12 * * *' # Run daily at 12:00 UTC pull_request: workflow_dispatch: + inputs: + coverage_mode: + description: 'Run in coverage collection mode (uses coverage runners, enables coverage instrumentation, runs setup-covstub)' + required: false + type: boolean + default: false workflow_call: inputs: ref: - description: 'Git ref (branch, tag, or SHA) to test. If not provided, uses the default branch.' + description: 'Git ref (branch, tag, or SHA) to test. If not provided, uses the event commit SHA.' required: false type: string default: '' @@ -17,9 +25,14 @@ on: required: false type: boolean default: false + coverage_mode: + description: 'Run in coverage collection mode (uses coverage runners, enables coverage instrumentation, runs setup-covstub)' + required: false + type: boolean + default: false concurrency: - group: pr-test-npu-${{ inputs.ref || github.ref }} + group: pr-test-npu-${{ inputs.coverage_mode == true && 'coverage-' || '' }}${{ inputs.ref || github.ref }} cancel-in-progress: ${{ github.event_name != 'workflow_call' }} jobs: @@ -34,16 +47,18 @@ jobs: - name: Checkout code uses: actions/checkout@v4 with: - ref: ${{ inputs.ref || github.ref }} + # Pin to the event SHA (not the branch name) so every job in this run + # checks out the same commit even if the branch advances mid-run. + ref: ${{ inputs.ref || github.sha }} - name: Determine run mode id: run-mode run: | - # Run all tests for workflow_call (when ref input is provided) + # Run all tests for scheduled runs and workflow_call (when ref input is provided) # Note: github.event_name is inherited from caller, so we detect workflow_call by checking inputs.ref - if [[ "${{ inputs.run_all_tests }}" == "true" ]]; then + if [[ "${{ github.event_name }}" == "schedule" || "${{ inputs.run_all_tests }}" == "true" ]]; then echo "run_all_tests=true" >> $GITHUB_OUTPUT - echo "Run mode: ALL TESTS (run_all_tests=${{ inputs.run_all_tests }})" + echo "Run mode: ALL TESTS (schedule=${{ github.event_name == 'schedule' }}, run_all_tests=${{ inputs.run_all_tests }})" else echo "run_all_tests=false" >> $GITHUB_OUTPUT echo "Run mode: FILTERED (triggered by ${{ github.event_name }})" @@ -55,8 +70,18 @@ jobs: if: steps.run-mode.outputs.run_all_tests != 'true' with: filters: | + # Everything under python/sglang except multimodal_gen/ and kernels/ops/diffusion/. + # Exclusions have to be extglobs inside a single pattern: paths-filter ORs the + # patterns of a filter (predicate-quantifier defaults to "some"), so a standalone + # "!some/path/**" entry matches every file outside that path and makes the whole + # filter unconditionally true. main_package: - - "python/sglang/!(multimodal_gen)/**/!(*.md)" + - "python/sglang/!(*.md)" + - "python/sglang/!(multimodal_gen|kernels)/**/!(*.md)" + - "python/sglang/kernels/!(*.md)" + - "python/sglang/kernels/!(ops)/**/!(*.md)" + - "python/sglang/kernels/ops/!(*.md)" + - "python/sglang/kernels/ops/!(diffusion)/**/!(*.md)" - "python/pyproject_npu.toml" - "scripts/ci/npu/npu_ci_install_dependency.sh" - "test/registered/npu/**" @@ -64,7 +89,7 @@ jobs: - ".github/workflows/pr-test-npu.yml" multimodal_gen: - "python/sglang/multimodal_gen/**/!(*.md|*.ipynb)" - - "python/sglang/kernels/ops/diffusion/triton/npu_fallback.py" + - "python/sglang/kernels/ops/diffusion/**" - "python/sglang/srt/**" - "python/pyproject_npu.toml" - "scripts/ci/npu/npu_ci_install_dependency.sh" @@ -73,7 +98,7 @@ jobs: # ==================== PR Gate ==================== # pr-gate: needs: check-changes - if: needs.check-changes.outputs.changes_exist == 'true' + if: ${{ inputs.coverage_mode != true && needs.check-changes.outputs.changes_exist == 'true' }} uses: ./.github/workflows/pr-gate.yml secrets: inherit @@ -81,91 +106,143 @@ jobs: runs-on: ubuntu-latest outputs: CANN_image_a3: ${{ steps.set-vars.outputs.CANN_image_a3 }} - CANN_image_910b: ${{ steps.set-vars.outputs.CANN_image_910b }} steps: # When triggered by PR, no inputs parameters are used. The latest community code is tested by default. - name: Set image config id: set-vars run: | - echo "CANN_image_a3=swr.cn-southwest-2.myhuaweicloud.com/base_image/ascend-ci/cann:9.0.0-a3-ubuntu22.04-py3.11" >> $GITHUB_OUTPUT - echo "CANN_image_910b=swr.cn-southwest-2.myhuaweicloud.com/base_image/ascend-ci/cann:9.0.0-910b-ubuntu22.04-py3.11" >> $GITHUB_OUTPUT + echo "CANN_image_a3=swr.cn-southwest-2.myhuaweicloud.com/base_image/ascend-ci/cann:9.1.0-a3-ubuntu22.04-py3.12" >> $GITHUB_OUTPUT - base-a-test-1-npu-a2: - needs: [check-changes, pr-gate, set-image-config] - if: ${{ !failure() && !cancelled() && needs.check-changes.outputs.main_package == 'true' }} - uses: ./.github/workflows/_npu-pr-test-stage.yml - with: - self_name: base-a-test-1-npu-a2 - runner_config: linux-aarch64-a2-1 - image: ${{ needs.set-image-config.outputs.CANN_image_910b }} - npu_device_type: 910b - run_timeout_minutes: '15' - secrets: inherit + # Stage-A gate: CPU-only unit tests. The NPU attention-backend UTs stub out + # torch_npu / sgl_kernel_npu before importing, so a hosted runner suffices. + base-a-test-npu: + needs: [check-changes, pr-gate] + if: ${{ !failure() && !cancelled() && inputs.coverage_mode != true && needs.check-changes.outputs.main_package == 'true' }} + runs-on: ubuntu-latest + timeout-minutes: 45 + env: + SGLANG_IS_IN_CI: true + steps: + - name: Checkout code + uses: actions/checkout@v4 + with: + # Pin to the event SHA so this job tests the same commit as the rest of the run. + ref: ${{ inputs.ref || github.sha }} + + # Same gate the NPU stages run first: fail before the NPU jobs are scheduled. + - uses: ./.github/actions/check-pr-test-health + + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: '3.12' + + - name: Install uv + uses: astral-sh/setup-uv@v5 + with: + enable-cache: true + cache-dependency-glob: 'python/pyproject_npu.toml' + + - name: Install dependencies + timeout-minutes: 30 + env: + # uv pip targets a venv by default; setup-python has no venv. + UV_SYSTEM_PYTHON: "1" + run: | + # Resolve the same dependency set as the NPU stages, minus Ascend-only wheels. + rm -f python/pyproject.toml + mv python/pyproject_npu.toml python/pyproject.toml + # Versions mirror npu_ci_install_dependency.sh's defaults; keep them in sync. + # triton: sglang.srt.utils.common imports it unconditionally and the CPU + # wheel does not pull it in (the NPU image uses triton-ascend instead). + # unsafe-best-match: --index-url is the lowest-priority index, so the + # default strategy would resolve PyPI's CUDA torch, not the +cpu build. + uv pip install torch==2.10.0 torchvision==0.25.0 torchaudio==2.10.0 triton \ + --index-url https://download.pytorch.org/whl/cpu \ + --extra-index-url https://pypi.org/simple/ \ + --index-strategy unsafe-best-match + uv pip install -e "python[dev_npu]" --index-strategy unsafe-best-match + + - name: Run NPU unit tests + timeout-minutes: 10 + run: | + cd test + python3 run_suite.py --hw npu --suite base-a-test-npu base-b-test-1-npu-a3: - needs: [check-changes, pr-gate, set-image-config] - if: ${{ !failure() && !cancelled() && needs.check-changes.outputs.main_package == 'true' }} + needs: [check-changes, pr-gate, set-image-config, base-a-test-npu] + if: ${{ !failure() && !cancelled() && (inputs.coverage_mode == true || needs.check-changes.outputs.main_package == 'true') }} uses: ./.github/workflows/_npu-pr-test-stage.yml with: self_name: base-b-test-1-npu-a3 - runner_config: linux-aarch64-a3-2- + runner_config: ${{ inputs.coverage_mode == true && 'linux-aarch64-a3-800t-2' || 'linux-aarch64-a3-2-' }} image: ${{ needs.set-image-config.outputs.CANN_image_a3 }} run_timeout_minutes: '60' timeout_per_file: '3600' + use_coverage_runner: ${{ inputs.coverage_mode == true }} + upload_test_logs: ${{ inputs.coverage_mode != true }} secrets: inherit base-b-test-2-npu-a3: - needs: [check-changes, pr-gate, set-image-config] - if: ${{ !failure() && !cancelled() && needs.check-changes.outputs.main_package == 'true' }} + needs: [check-changes, pr-gate, set-image-config, base-a-test-npu] + if: ${{ !failure() && !cancelled() && (inputs.coverage_mode == true || needs.check-changes.outputs.main_package == 'true') }} uses: ./.github/workflows/_npu-pr-test-stage.yml with: self_name: base-b-test-2-npu-a3 - runner_config: linux-aarch64-a3-2- + runner_config: ${{ inputs.coverage_mode == true && 'linux-aarch64-a3-800t-2' || 'linux-aarch64-a3-2-' }} image: ${{ needs.set-image-config.outputs.CANN_image_a3 }} run_timeout_minutes: '60' timeout_per_file: '3600' + use_coverage_runner: ${{ inputs.coverage_mode == true }} + upload_test_logs: ${{ inputs.coverage_mode != true }} secrets: inherit base-b-test-4-npu-a3: - needs: [check-changes, pr-gate, set-image-config] - if: ${{ !failure() && !cancelled() && needs.check-changes.outputs.main_package == 'true' }} + needs: [check-changes, pr-gate, set-image-config, base-a-test-npu] + if: ${{ !failure() && !cancelled() && (inputs.coverage_mode == true || needs.check-changes.outputs.main_package == 'true') }} uses: ./.github/workflows/_npu-pr-test-stage.yml with: self_name: base-b-test-4-npu-a3 - runner_config: linux-aarch64-a3-4- + runner_config: ${{ inputs.coverage_mode == true && 'linux-aarch64-a3-800t-4' || 'linux-aarch64-a3-4-' }} image: ${{ needs.set-image-config.outputs.CANN_image_a3 }} run_timeout_minutes: '120' timeout_per_file: '3600' partitions: '{"size":2,"arr":[0, 1]}' + use_coverage_runner: ${{ inputs.coverage_mode == true }} + upload_test_logs: ${{ inputs.coverage_mode != true }} secrets: inherit base-b-test-8-npu-a3: - needs: [check-changes, pr-gate, set-image-config] - if: ${{ !failure() && !cancelled() && needs.check-changes.outputs.main_package == 'true' }} + needs: [check-changes, pr-gate, set-image-config, base-a-test-npu] + if: ${{ !failure() && !cancelled() && (inputs.coverage_mode == true || needs.check-changes.outputs.main_package == 'true') }} uses: ./.github/workflows/_npu-pr-test-stage.yml with: self_name: base-b-test-8-npu-a3 - runner_config: linux-aarch64-a3-8- + runner_config: ${{ inputs.coverage_mode == true && 'linux-aarch64-a3-800t-8' || 'linux-aarch64-a3-8-' }} image: ${{ needs.set-image-config.outputs.CANN_image_a3 }} run_timeout_minutes: '60' timeout_per_file: '3600' + use_coverage_runner: ${{ inputs.coverage_mode == true }} + upload_test_logs: ${{ inputs.coverage_mode != true }} secrets: inherit base-b-test-16-npu-a3: - needs: [check-changes, pr-gate, set-image-config] - if: ${{ !failure() && !cancelled() && needs.check-changes.outputs.main_package == 'true' }} + needs: [check-changes, pr-gate, set-image-config, base-a-test-npu] + if: ${{ !failure() && !cancelled() && (inputs.coverage_mode == true || needs.check-changes.outputs.main_package == 'true') }} uses: ./.github/workflows/_npu-pr-test-stage.yml with: self_name: base-b-test-16-npu-a3 - runner_config: linux-aarch64-a3-16- + runner_config: ${{ inputs.coverage_mode == true && 'linux-aarch64-a3-800t-16' || 'linux-aarch64-a3-16-' }} image: ${{ needs.set-image-config.outputs.CANN_image_a3 }} run_timeout_minutes: '120' timeout_per_file: '3600' + use_coverage_runner: ${{ inputs.coverage_mode == true }} + upload_test_logs: ${{ inputs.coverage_mode != true }} secrets: inherit multimodal-gen-test-1-npu-a3: - needs: [check-changes, pr-gate, set-image-config] - if: ${{ !failure() && !cancelled() && needs.check-changes.outputs.multimodal_gen == 'true' }} + needs: [check-changes, pr-gate, set-image-config, base-a-test-npu] + if: ${{ !failure() && !cancelled() && inputs.coverage_mode != true && needs.check-changes.outputs.multimodal_gen == 'true' }} runs-on: linux-aarch64-a3-800t-2 strategy: fail-fast: false @@ -227,10 +304,10 @@ jobs: if-no-files-found: ignore retention-days: 7 - multimodal-gen-test-2-npu-a3: - needs: [check-changes, pr-gate, set-image-config] - if: ${{ !failure() && !cancelled() && needs.check-changes.outputs.multimodal_gen == 'true' }} - runs-on: linux-aarch64-a3-800t-16 + multimodal-gen-test-4-npu-a3: + needs: [check-changes, pr-gate, set-image-config, base-a-test-npu] + if: ${{ !failure() && !cancelled() && inputs.coverage_mode != true && needs.check-changes.outputs.multimodal_gen == 'true' }} + runs-on: linux-aarch64-a3-800t-4 strategy: fail-fast: false matrix: @@ -293,61 +370,39 @@ jobs: base-c-test-acc-2-npu-a3: name: base-c-test-acc-2-npu-a3 - needs: [ check-changes, pr-gate, set-image-config, base-a-test-1-npu-a2, base-b-test-1-npu-a3, base-b-test-2-npu-a3, base-b-test-4-npu-a3, base-b-test-8-npu-a3, base-b-test-16-npu-a3 ] - if: ${{ !failure() && !cancelled() && needs.check-changes.outputs.main_package == 'true' }} + needs: [ check-changes, pr-gate, set-image-config, base-b-test-1-npu-a3, base-b-test-2-npu-a3, base-b-test-4-npu-a3, base-b-test-8-npu-a3, base-b-test-16-npu-a3 ] + if: ${{ !failure() && !cancelled() && (inputs.coverage_mode == true || needs.check-changes.outputs.main_package == 'true') }} uses: ./.github/workflows/_npu-single-node-test-stage.yml with: - runner: linux-aarch64-a3-800t-2 + runner: ${{ inputs.coverage_mode == true && 'linux-aarch64-a3-800t-2' || 'linux-aarch64-a3-2-' }} test_type: 'accuracy' test_suite: base-c-test-acc-2-npu-a3 image: ${{ needs.set-image-config.outputs.CANN_image_a3 }} install_sglang_deps: true device_type_for_deps: 'a3' partitions: '{"size":2,"arr":[0,1]}' - - base-c-test-acc-4-npu-a3: - name: base-c-test-acc-4-npu-a3 - needs: [ check-changes, pr-gate, set-image-config, base-a-test-1-npu-a2, base-b-test-1-npu-a3, base-b-test-2-npu-a3, base-b-test-4-npu-a3, base-b-test-8-npu-a3, base-b-test-16-npu-a3 ] - if: ${{ !failure() && !cancelled() && needs.check-changes.outputs.main_package == 'true' }} - uses: ./.github/workflows/_npu-single-node-test-stage.yml - with: - runner: linux-aarch64-a3-800t-4 - test_type: 'accuracy' - test_suite: base-c-test-acc-4-npu-a3 - image: ${{ needs.set-image-config.outputs.CANN_image_a3 }} - install_sglang_deps: true - device_type_for_deps: 'a3' - - base-c-test-acc-8-npu-a3: - name: base-c-test-acc-8-npu-a3 - needs: [ check-changes, pr-gate, set-image-config, base-a-test-1-npu-a2, base-b-test-1-npu-a3, base-b-test-2-npu-a3, base-b-test-4-npu-a3, base-b-test-8-npu-a3, base-b-test-16-npu-a3 ] - if: ${{ !failure() && !cancelled() && needs.check-changes.outputs.main_package == 'true' }} - uses: ./.github/workflows/_npu-single-node-test-stage.yml - with: - runner: linux-aarch64-a3-800t-8 - test_type: 'accuracy' - test_suite: base-c-test-acc-8-npu-a3 - image: ${{ needs.set-image-config.outputs.CANN_image_a3 }} - install_sglang_deps: true - device_type_for_deps: 'a3' + use_coverage_runner: ${{ inputs.coverage_mode == true }} + upload_test_logs: ${{ inputs.coverage_mode != true }} base-c-test-acc-16-npu-a3: name: base-c-test-acc-16-npu-a3 - needs: [ check-changes, pr-gate, set-image-config, base-a-test-1-npu-a2, base-b-test-1-npu-a3, base-b-test-2-npu-a3, base-b-test-4-npu-a3, base-b-test-8-npu-a3, base-b-test-16-npu-a3 ] - if: ${{ !failure() && !cancelled() && needs.check-changes.outputs.main_package == 'true' }} + needs: [ check-changes, pr-gate, set-image-config, base-b-test-1-npu-a3, base-b-test-2-npu-a3, base-b-test-4-npu-a3, base-b-test-8-npu-a3, base-b-test-16-npu-a3 ] + if: ${{ !failure() && !cancelled() && (inputs.coverage_mode == true || needs.check-changes.outputs.main_package == 'true') }} uses: ./.github/workflows/_npu-single-node-test-stage.yml with: - runner: linux-aarch64-a3-800t-16 + runner: ${{ inputs.coverage_mode == true && 'linux-aarch64-a3-800t-16' || 'linux-aarch64-a3-16-' }} test_type: 'accuracy' test_suite: base-c-test-acc-16-npu-a3 image: ${{ needs.set-image-config.outputs.CANN_image_a3 }} install_sglang_deps: true device_type_for_deps: 'a3' + use_coverage_runner: ${{ inputs.coverage_mode == true }} + upload_test_logs: ${{ inputs.coverage_mode != true }} base-c-test-perf-2-npu-a3: name: base-c-test-perf-2-npu-a3 - needs: [ check-changes, pr-gate, set-image-config, base-a-test-1-npu-a2, base-b-test-1-npu-a3, base-b-test-2-npu-a3, base-b-test-4-npu-a3, base-b-test-8-npu-a3, base-b-test-16-npu-a3, base-c-test-acc-2-npu-a3, base-c-test-acc-4-npu-a3, base-c-test-acc-8-npu-a3, base-c-test-acc-16-npu-a3 ] - if: ${{ !failure() && !cancelled() && needs.check-changes.outputs.main_package == 'true' }} + needs: [ check-changes, pr-gate, set-image-config, base-c-test-acc-2-npu-a3, base-c-test-acc-16-npu-a3 ] + if: ${{ !failure() && !cancelled() && (inputs.coverage_mode == true || needs.check-changes.outputs.main_package == 'true') }} uses: ./.github/workflows/_npu-single-node-test-stage.yml with: runner: linux-aarch64-a3-800t-2 @@ -356,37 +411,13 @@ jobs: image: ${{ needs.set-image-config.outputs.CANN_image_a3 }} install_sglang_deps: true device_type_for_deps: 'a3' - - base-c-test-perf-4-npu-a3: - name: base-c-test-perf-4-npu-a3 - needs: [ check-changes, pr-gate, set-image-config, base-a-test-1-npu-a2, base-b-test-1-npu-a3, base-b-test-2-npu-a3, base-b-test-4-npu-a3, base-b-test-8-npu-a3, base-b-test-16-npu-a3, base-c-test-acc-2-npu-a3, base-c-test-acc-4-npu-a3, base-c-test-acc-8-npu-a3, base-c-test-acc-16-npu-a3 ] - if: ${{ !failure() && !cancelled() && needs.check-changes.outputs.main_package == 'true' }} - uses: ./.github/workflows/_npu-single-node-test-stage.yml - with: - runner: linux-aarch64-a3-800t-4 - test_type: 'perf' - test_suite: base-c-test-perf-4-npu-a3 - image: ${{ needs.set-image-config.outputs.CANN_image_a3 }} - install_sglang_deps: true - device_type_for_deps: 'a3' - - base-c-test-perf-8-npu-a3: - name: base-c-test-perf-8-npu-a3 - needs: [ check-changes, pr-gate, set-image-config, base-a-test-1-npu-a2, base-b-test-1-npu-a3, base-b-test-2-npu-a3, base-b-test-4-npu-a3, base-b-test-8-npu-a3, base-b-test-16-npu-a3, base-c-test-acc-2-npu-a3, base-c-test-acc-4-npu-a3, base-c-test-acc-8-npu-a3, base-c-test-acc-16-npu-a3 ] - if: ${{ !failure() && !cancelled() && needs.check-changes.outputs.main_package == 'true' }} - uses: ./.github/workflows/_npu-single-node-test-stage.yml - with: - runner: linux-aarch64-a3-800t-8 - test_type: 'perf' - test_suite: base-c-test-perf-8-npu-a3 - image: ${{ needs.set-image-config.outputs.CANN_image_a3 }} - install_sglang_deps: true - device_type_for_deps: 'a3' + use_coverage_runner: ${{ inputs.coverage_mode == true }} + upload_test_logs: ${{ inputs.coverage_mode != true }} base-c-test-perf-16-npu-a3: name: base-c-test-perf-16-npu-a3 - needs: [ check-changes, pr-gate, set-image-config, base-a-test-1-npu-a2, base-b-test-1-npu-a3, base-b-test-2-npu-a3, base-b-test-4-npu-a3, base-b-test-8-npu-a3, base-b-test-16-npu-a3, base-c-test-acc-2-npu-a3, base-c-test-acc-4-npu-a3, base-c-test-acc-8-npu-a3, base-c-test-acc-16-npu-a3 ] - if: ${{ !failure() && !cancelled() && needs.check-changes.outputs.main_package == 'true' }} + needs: [ check-changes, pr-gate, set-image-config, base-c-test-acc-2-npu-a3, base-c-test-acc-16-npu-a3 ] + if: ${{ !failure() && !cancelled() && (inputs.coverage_mode == true || needs.check-changes.outputs.main_package == 'true') }} uses: ./.github/workflows/_npu-single-node-test-stage.yml with: runner: linux-aarch64-a3-800t-16 @@ -396,14 +427,268 @@ jobs: install_sglang_deps: true device_type_for_deps: 'a3' test_timeout_minutes: '180' + use_coverage_runner: ${{ inputs.coverage_mode == true }} + upload_test_logs: ${{ inputs.coverage_mode != true }} + + # ==================== Recommend Tests from Coverage ==================== # + # Recommends PR-affected test cases from the pre-built coverage baseline. + # continue-on-error so it never blocks the PR gate. + recommend-tests-from-coverage: + name: Recommend tests from coverage + runs-on: ubuntu-latest + continue-on-error: true + if: ${{ !cancelled() && github.event_name == 'pull_request' && inputs.coverage_mode != true }} + outputs: + coverage_paths: ${{ steps.export-paths.outputs.coverage_paths }} + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + steps: + - name: Checkout sglang scripts + uses: actions/checkout@v4 + with: + sparse-checkout: | + scripts + sparse-checkout-cone-mode: false + + - name: Download test case map + run: | + set -euo pipefail + curl -fsSL \ + --retry 3 \ + --retry-delay 5 \ + --retry-all-errors \ + "https://sglang-npu.obs.cn-southwest-2.myhuaweicloud.com/coverage/test_case_map.json" \ + -o test_case_map.json + test -s test_case_map.json + echo "Downloaded test_case_map.json" + du -h test_case_map.json + + - name: Download coverage package and extract covstub + run: | + set -euo pipefail + curl -fsSL \ + --retry 3 \ + --retry-delay 5 \ + --retry-all-errors \ + "https://sglang-npu.obs.cn-southwest-2.myhuaweicloud.com/coverage/outputs.tar.gz" \ + -o outputs.tar.gz + test -s outputs.tar.gz + echo "Downloaded outputs.tar.gz" + du -h outputs.tar.gz + tar xzf outputs.tar.gz outputs/covstub + test -d outputs/covstub + echo "Extracted outputs/covstub" + find outputs/covstub -maxdepth 2 -type d | head -20 || true + + - name: Recommend tests for current PR + run: | + set -euo pipefail + if ! python3 -c "import regex" >/dev/null 2>&1; then + echo "regex not found for $(python3 -V); bootstrapping pip and installing regex" + curl -fsSL https://bootstrap.pypa.io/get-pip.py -o get-pip.py + python3 get-pip.py --break-system-packages + python3 -m pip install --break-system-packages regex + fi + python3 -c "import regex; print('regex ok:', regex.__file__)" + PR_SPEC="${{ github.repository }}#${{ github.event.pull_request.number }}" + echo "Selecting tests for PR: ${PR_SPEC}" + # test_selector.py resolves relative paths against the script's own dir, + # so pass absolute paths for files downloaded to the workspace root. + python3 scripts/ci/npu/precise-test/test_selector.py \ + --github-pr "${PR_SPEC}" \ + --source-dir "$PWD/outputs/covstub" \ + --map-file "$PWD/test_case_map.json" + + - name: Export coverage paths output + id: export-paths + if: always() + run: | + FILE="scripts/ci/npu/precise-test/recommended_pytest_paths.txt" + if [ -f "$FILE" ]; then + { + echo "coverage_paths<> "$GITHUB_OUTPUT" + echo "Exported $(wc -l < "$FILE") recommended paths" + else + echo "coverage_paths=" >> "$GITHUB_OUTPUT" + echo "::notice::recommended_pytest_paths.txt not found" + fi + + # ==================== Analyze Failure Report ==================== # + # Cross-references test failures with coverage recommendations. + # !cancelled() only: skipped jobs in needs poison success(). + analyze-failure-report: + name: Analyze failure report + needs: + [ + base-a-test-npu, + base-b-test-1-npu-a3, + base-b-test-2-npu-a3, + base-b-test-4-npu-a3, + base-b-test-8-npu-a3, + base-b-test-16-npu-a3, + + multimodal-gen-test-1-npu-a3, + multimodal-gen-test-4-npu-a3, + + base-c-test-acc-2-npu-a3, + base-c-test-acc-16-npu-a3, + base-c-test-perf-2-npu-a3, + base-c-test-perf-16-npu-a3, + + recommend-tests-from-coverage, + ] + if: ${{ !cancelled() && inputs.coverage_mode != true }} + uses: ./.github/workflows/_npu-analyze-failure.yml + with: + log_artifact_pattern: selected-test-logs-* + recommendations_content: ${{ needs.recommend-tests-from-coverage.outputs.coverage_paths || '' }} + + # ==================== Coverage Assembly & Publish ==================== # + # Assembles coverage data and publishes the baseline to the shared disk. + # !failure() && !cancelled() (not success()): skipped upstream poisons success(). + setup-covstub: + needs: + [ + set-image-config, + base-b-test-1-npu-a3, + base-b-test-2-npu-a3, + base-b-test-4-npu-a3, + base-b-test-8-npu-a3, + base-b-test-16-npu-a3, + base-c-test-acc-2-npu-a3, + base-c-test-acc-16-npu-a3, + base-c-test-perf-2-npu-a3, + base-c-test-perf-16-npu-a3, + ] + if: ${{ !failure() && !cancelled() && inputs.coverage_mode == true }} + runs-on: linux-aarch64-a3-800t-2 + container: + image: ${{ needs.set-image-config.outputs.CANN_image_a3 }} + steps: + - name: Checkout code + uses: actions/checkout@v4 + with: + # Pin to the event SHA (not the branch name) so every job in this run + # checks out the same commit even if the branch advances mid-run. + ref: ${{ inputs.ref || github.sha }} + + - name: Copy sglang source to covstub + run: | + RUN_DIR="/root/.cache/tests/precise-test/${{ github.run_id }}-attempt-${{ github.run_attempt }}" + COVSTUB="${RUN_DIR}/outputs/covstub" + mkdir -p "${COVSTUB}" + cp -r python/sglang "${COVSTUB}/" + + - name: Build test case map + shell: bash + run: | + set -euo pipefail + + RUN_DIR="/root/.cache/tests/precise-test/${{ github.run_id }}-attempt-${{ github.run_attempt }}" + # Date tag derivation must match scripts/ci/npu/precise-test/run_tests_with_coverage.sh. + # The custom k8s runner only injects explicitly-set env vars, so + # GITHUB_RUN_STARTED_AT may be absent (set -u would abort); fall back + # to local date exactly like run_tests_with_coverage.sh does, so both + # sides derive the same tag. + if [ -n "${GITHUB_RUN_STARTED_AT:-}" ]; then + COV_DATE_TAG="${GITHUB_RUN_STARTED_AT:0:10}" + COV_DATE_TAG="${COV_DATE_TAG//-/}" + else + COV_DATE_TAG="$(date +%Y%m%d)" + fi + COV_ROOT="${RUN_DIR}/outputs/sglang@${COV_DATE_TAG}" + COVSTUB="${RUN_DIR}/outputs/covstub" + # Placed outside outputs/ so it is NOT included in outputs.tar.gz. + MAP_FILE="${RUN_DIR}/test_case_map.json" + + echo "Coverage dir: ${COV_ROOT}" + echo "Source dir: ${COVSTUB}" + + # Coverage data may be absent when all test jobs were skipped + # (e.g. by the changes filter). Skip map building in that case. + if [ ! -d "${COV_ROOT}" ]; then + echo "::warning::Coverage dir not found: ${COV_ROOT}, skip building test case map" + exit 0 + fi + + COVERAGE_FILE_COUNT=$(find "${COV_ROOT}" -name 'coverage.*' -type f | wc -l) + echo "Found ${COVERAGE_FILE_COUNT} coverage files" + if [ "${COVERAGE_FILE_COUNT}" -eq 0 ]; then + echo "::warning::No coverage files found under ${COV_ROOT}, skip building test case map" + exit 0 + fi + + if ! python3 -c "import regex" 2>/dev/null; then + pip install regex \ + -i http://cache-service.nginx-pypi-cache.svc.cluster.local/pypi/simple \ + --trusted-host cache-service.nginx-pypi-cache.svc.cluster.local \ + --retries 3 --timeout 60 \ + || pip install regex -i https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple + fi + + python3 scripts/ci/npu/precise-test/test_selector.py \ + --build-map \ + --coverage-dir "${COV_ROOT}" \ + --source-dir "${COVSTUB}" \ + --map-file "${MAP_FILE}" + test -s "${MAP_FILE}" + du -h "${MAP_FILE}" + + - name: Archive outputs + run: | + RUN_DIR="/root/.cache/tests/precise-test/${{ github.run_id }}-attempt-${{ github.run_attempt }}" + tar -czf "${RUN_DIR}/outputs.tar.gz" -C "${RUN_DIR}" outputs + + - name: Publish outputs and map to shared disk + shell: bash + run: | + set -euo pipefail + + RUN_DIR="/root/.cache/tests/precise-test/${{ github.run_id }}-attempt-${{ github.run_attempt }}" + # Fixed publish dir with no RUN_DIR layer: other pipelines have a + # different GITHUB_RUN_ID and must be able to read the baseline. + # Must match PUBLISH_DIR in the recommend-tests-from-coverage job. + PUBLISH_DIR="/root/.cache/tests/precise-test/coverage" + + # Skip publish when no coverage data was collected this run (map + # build was skipped), so we never overwrite the historical + # baseline on the shared disk with an incomplete one. + if [ ! -s "${RUN_DIR}/test_case_map.json" ]; then + echo "::warning::test_case_map.json missing (no coverage data this run), skip publish" + exit 0 + fi + + mkdir -p "${PUBLISH_DIR}" + + # mv (rename) within the same shared filesystem is atomic, so a + # concurrent reader sees either the old or the new baseline, + # never a partial file. + mv "${RUN_DIR}/outputs.tar.gz" "${PUBLISH_DIR}/outputs.tar.gz" + mv "${RUN_DIR}/test_case_map.json" "${PUBLISH_DIR}/test_case_map.json" + + echo "Published baseline to ${PUBLISH_DIR}" + du -h "${PUBLISH_DIR}/outputs.tar.gz" "${PUBLISH_DIR}/test_case_map.json" + + - name: Clean up per-run directory + # Skipped on publish failure, keeping RUN_DIR for manual recovery. + shell: bash + run: | + set -euo pipefail + RUN_DIR="/root/.cache/tests/precise-test/${{ github.run_id }}-attempt-${{ github.run_attempt }}" + rm -rf "${RUN_DIR}" + echo "Removed ${RUN_DIR}" pr-test-npu-finish: needs: [ check-changes, - base-a-test-1-npu-a2, + base-a-test-npu, base-b-test-1-npu-a3, base-b-test-2-npu-a3, base-b-test-4-npu-a3, @@ -411,16 +696,14 @@ jobs: base-b-test-16-npu-a3, multimodal-gen-test-1-npu-a3, - multimodal-gen-test-2-npu-a3, + multimodal-gen-test-4-npu-a3, base-c-test-acc-2-npu-a3, - base-c-test-acc-4-npu-a3, - base-c-test-acc-8-npu-a3, base-c-test-acc-16-npu-a3, base-c-test-perf-2-npu-a3, - base-c-test-perf-4-npu-a3, - base-c-test-perf-8-npu-a3, base-c-test-perf-16-npu-a3, + + setup-covstub, ] if: always() runs-on: ubuntu-latest diff --git a/.github/workflows/pr-test-rust.yml b/.github/workflows/pr-test-rust.yml index 5c8f7fbf4f04..b2c225920e35 100644 --- a/.github/workflows/pr-test-rust.yml +++ b/.github/workflows/pr-test-rust.yml @@ -8,6 +8,12 @@ on: - ".github/workflows/pr-test-rust.yml" - "scripts/ci/cuda/ci_install_dependency.sh" - "scripts/ci/cuda/ci_install_gateway_dependencies.sh" + # gateway-e2e resolves the SMG runtime deps (the smg-grpc-servicer + # floor) from here, via ci_install_dependency.sh's + # `pip install -e python[...]` — a bump there must re-run this workflow. + - "python/pyproject.toml" + # The SMG gRPC entrypoint; only this workflow exercises it. + - "python/sglang/srt/entrypoints/grpc_server.py" pull_request: branches: [ main ] types: [opened, synchronize, reopened, labeled] @@ -16,6 +22,12 @@ on: - ".github/workflows/pr-test-rust.yml" - "scripts/ci/cuda/ci_install_dependency.sh" - "scripts/ci/cuda/ci_install_gateway_dependencies.sh" + # gateway-e2e resolves the SMG runtime deps (the smg-grpc-servicer + # floor) from here, via ci_install_dependency.sh's + # `pip install -e python[...]` — a bump there must re-run this workflow. + - "python/pyproject.toml" + # The SMG gRPC entrypoint; only this workflow exercises it. + - "python/sglang/srt/entrypoints/grpc_server.py" workflow_dispatch: concurrency: diff --git a/.github/workflows/pr-test-sgl-router.yml b/.github/workflows/pr-test-sgl-router.yml index 45f17cac0710..285edf968a15 100644 --- a/.github/workflows/pr-test-sgl-router.yml +++ b/.github/workflows/pr-test-sgl-router.yml @@ -102,38 +102,12 @@ jobs: # of self-hosted-runner pip installs that compile Rust extensions. RUSTC_WRAPPER: sccache SCCACHE_GHA_ENABLED: "true" - # Supply-chain guard: deny.toml allowlists the dynamo git source but - # cannot pin a SHA. Bumps to this constant must be a deliberate PR. - DYNAMO_TOKENIZERS_EXPECTED_REV: "1efdd4dcb901caeae636131321094090d252c8d6" steps: - uses: actions/checkout@v4 - name: Install protoc run: sudo bash scripts/ci/utils/install_protoc.sh - - name: Verify dynamo-* SHA pin - run: | - set -euo pipefail - revs=$(grep -E '^dynamo-[a-z]+[[:space:]]*=.*\brev[[:space:]]*=' experimental/sgl-router/Cargo.toml \ - | sed -E 's/.*rev[[:space:]]*=[[:space:]]*"([^"]+)".*/\1/') - if [ -z "${revs:-}" ]; then - echo "::error::no dynamo-* git deps with rev= found in experimental/sgl-router/Cargo.toml" - exit 1 - fi - count=0 - while IFS= read -r r; do - count=$((count + 1)) - if [ "$r" != "$DYNAMO_TOKENIZERS_EXPECTED_REV" ]; then - echo "::error::dynamo-* SHA pin mismatch in experimental/sgl-router/Cargo.toml" - echo " expected: $DYNAMO_TOKENIZERS_EXPECTED_REV" - echo " actual: $r" - echo "All revs found:" - echo "$revs" | sed 's/^/ /' - exit 1 - fi - done <<< "$revs" - echo "dynamo-* SHA pin OK: $count entries all at $DYNAMO_TOKENIZERS_EXPECTED_REV" - # sccache install is best-effort: the GitHub release CDN intermittently # returns 5xx and the action's built-in retry is shallow. If it fails, # we unset RUSTC_WRAPPER below and proceed without compile caching — @@ -163,11 +137,11 @@ jobs: - name: cargo check working-directory: experimental/sgl-router - run: cargo check --workspace --all-targets + run: cargo check --locked --workspace --all-targets - name: cargo clippy working-directory: experimental/sgl-router - run: cargo clippy --workspace --all-targets -- -D warnings + run: cargo clippy --locked --workspace --all-targets -- -D warnings - name: cargo fmt working-directory: experimental/sgl-router @@ -238,7 +212,7 @@ jobs: working-directory: experimental/sgl-router run: | START=$(date +%s) - cargo build --release --all-targets + cargo build --locked --release --all-targets END=$(date +%s) BUILD_SECONDS=$((END - START)) echo "tier_2_build_duration_seconds=${BUILD_SECONDS}" >> "$GITHUB_ENV" @@ -252,7 +226,7 @@ jobs: # the Qwen3-0.6B tokenizer.json. Filter is matched against the # full test path `tokenizer::parity::parity_matrix`; substring # `parity_matrix` is unique to that test. - run: cargo test --release --workspace -- --skip parity_matrix + run: cargo test --locked --release --workspace -- --skip parity_matrix # Regenerate the cross-impl block-hash parity fixture and fail if it # differs from the committed file. The Python script replicates @@ -372,7 +346,7 @@ jobs: working-directory: experimental/sgl-router run: | source "$HOME/.cargo/env" - cargo build --release --workspace + cargo build --locked --release --workspace # Install SGLang from the local checkout in editable mode (no PyPI # version pin) — mirrors `pr-test-rust.yml` so the router e2e runs @@ -411,7 +385,7 @@ jobs: HF_TOKEN: ${{ secrets.HF_TOKEN }} run: | source "$HOME/.cargo/env" - cargo test --release --test component tokenizer::parity + cargo test --locked --release --test component tokenizer::parity sgl-router-finish: name: finish diff --git a/.github/workflows/pr-test-xeon.yml b/.github/workflows/pr-test-xeon.yml index 683e98e47f89..558d2cd65503 100644 --- a/.github/workflows/pr-test-xeon.yml +++ b/.github/workflows/pr-test-xeon.yml @@ -90,7 +90,7 @@ jobs: # full machine, no partitioning. 4 files: 3 TP + autoround. - runner: xeon-gnr role: gnr - suite: "base-b-tp-test-cpu" + suite: "stage-a-tp-test-cpu-intel" name: gnr # Three SPR boxes each run ONE full-machine container taking one @@ -102,19 +102,19 @@ jobs: # per box keeps each server within the box's memory budget. - runner: xeon-spr role: spr - suite: "base-b-test-cpu,base-c-test-cpu" + suite: "stage-a-test-cpu-intel,stage-b-test-cpu-intel" name: spr1 part_id: 0 part_size: 3 - runner: xeon-spr role: spr - suite: "base-b-test-cpu,base-c-test-cpu" + suite: "stage-a-test-cpu-intel,stage-b-test-cpu-intel" name: spr2 part_id: 1 part_size: 3 - runner: xeon-spr role: spr - suite: "base-b-test-cpu,base-c-test-cpu" + suite: "stage-a-test-cpu-intel,stage-b-test-cpu-intel" name: spr3 part_id: 2 part_size: 3 @@ -150,7 +150,7 @@ jobs: run: | docker rm -f ci_sglang_${{ matrix.name }} || true docker run -dt \ - -v ${{ github.workspace }}:/sglang-checkout/ --ipc=host \ + -v ${{ github.workspace }}:/sglang-checkout/ --shm-size=64g \ -v $HOME/.cache/huggingface:/root/.cache/huggingface \ -v $HOME/.cache/sgl_eval:/root/.cache/sgl_eval \ -e HF_TOKEN="$(cat ~/huggingface_token.txt)" \ @@ -177,7 +177,7 @@ jobs: run: | docker rm -f ci_sglang_${{ matrix.name }} || true docker run -dt \ - -v ${{ github.workspace }}:/sglang-checkout/ --ipc=host \ + -v ${{ github.workspace }}:/sglang-checkout/ --shm-size=64g \ -v $HOME/.cache/huggingface:/root/.cache/huggingface \ -v $HOME/.cache/sgl_eval:/root/.cache/sgl_eval \ -e HF_TOKEN="$(cat ~/huggingface_token.txt)" \ diff --git a/.github/workflows/pr-test-xpu.yml b/.github/workflows/pr-test-xpu.yml index ff284c96ad1a..b98e05687b6b 100644 --- a/.github/workflows/pr-test-xpu.yml +++ b/.github/workflows/pr-test-xpu.yml @@ -56,17 +56,56 @@ jobs: with: filters: | main_package: - - "python/sglang/!(multimodal_gen)/**/!(*.md)" + # Extend test/registered/ entries when adding a non-nightly register_xpu_ci. + - "python/sglang/!(multimodal_gen|kernels|cli|test)/**/!(*.md)" + - "python/sglang/test/*.py" + - "python/sglang/test/!(ascend|observability|mock_model|manual|external_models|kernels)/**/!(*.md)" - "python/pyproject_xpu.toml" - - "test/**/!(*.md)" - - "python/sglang/kernels/aot/**/!(*.md|THIRDPARTYNOTICES.txt|LICENSE)" + - "python/sglang/kernels/ops/speculative/@(reconstruct_tree|ngram_corpus).py" + - "python/sglang/kernels/jit/csrc/ngram_corpus/**" + - "test/registered/xpu/**/!(*.md)" + - "test/registered/e2e/xpu/**/!(*.md)" + - "test/registered/disaggregation/test_disaggregation_xpu.py" + - "test/registered/attention/test_chunk_gated_delta_rule.py" + - "test/registered/attention/test_deterministic.py" + - "test/registered/kernels/ops/kvcache/test_hisparse.py" + - "test/registered/kv_canary/test_self_unit_future_tensor.py" + - "test/registered/kv_canary/test_self_unit_plan_input.py" + - "test/registered/kv_canary/test_self_unit_req_to_expected_token_ids_manager.py" + - "test/registered/kv_canary/test_self_unit_runner_swa_divergence.py" + - "test/registered/layers/mamba/test_mamba_slot_fused.py" + - "test/registered/lora/test_moe_lora_info.py" + - "test/registered/lora/test_virtual_experts_kernels.py" + - "test/registered/unit/layers/test_mamba_state_scatter_triton.py" + - "test/registered/unit/managers/test_hisparse_unit.py" + - "test/registered/unit/mem_cache/test_mamba_unittest.py" + - "test/registered/unit/sampling/test_sampling_params.py" + - "test/registered/unit/spec/test_adaptive_spec_params.py" + - "test/registered/unit/spec/test_ngram_corpus.py" + - "test/run_suite.py" + - "test/pytest.ini" - ".github/workflows/pr-test-xpu.yml" - "docker/xpu.Dockerfile" + - "scripts/ci/xpu/**" multimodal_gen: - - "python/sglang/multimodal_gen/**/!(*.md|*.ipynb)" + # Only paths imported by the 1-gpu-xpu suite; other-vendor and off-suite files are excluded. + - "python/sglang/multimodal_gen/*.py" + - "python/sglang/multimodal_gen/runtime/!(pipelines)/**/!(*.md|*.ipynb)" + - "python/sglang/multimodal_gen/runtime/*.py" + - "python/sglang/multimodal_gen/runtime/pipelines/__init__.py" + - "python/sglang/multimodal_gen/runtime/pipelines/@(zimage_pipeline|flux_2_klein|flux_2|wan_pipeline|diffusers_pipeline).py" + - "python/sglang/multimodal_gen/configs/**/!(*.md|*.ipynb)" + - "python/sglang/multimodal_gen/apps/webui/**/!(*.md|*.ipynb)" + - "python/sglang/multimodal_gen/benchmarks/compare_perf.py" + - "python/sglang/multimodal_gen/test/*.py" + - "python/sglang/multimodal_gen/test/runner/**/!(*.md|*.ipynb)" + - "python/sglang/multimodal_gen/test/server/!(test_server_1_gpu_5090|test_server_4_gpu_h100|test_server_b200|test_server_2_gpu|test_request_logger).py" + - "python/sglang/multimodal_gen/test/server/common/**/!(*.md|*.ipynb)" + - "python/sglang/multimodal_gen/test/server/perf_baselines/xpu_b60.json" - "python/pyproject_xpu.toml" - ".github/workflows/pr-test-xpu.yml" - "docker/xpu.Dockerfile" + - "scripts/ci/xpu/**" # ==================== PR Gate ==================== # pr-gate: @@ -75,7 +114,7 @@ jobs: uses: ./.github/workflows/pr-gate.yml secrets: inherit - # ==================== Stage A ==================== # + # Runs the entire per-commit XPU test suite in one job (single install, single queue). stage-a-test-1-gpu-xpu: needs: [check-changes, pr-gate] if: needs.check-changes.outputs.main_package == 'true' @@ -109,8 +148,14 @@ jobs: - name: Install Dependency timeout-minutes: 60 run: | + # Bridge fix: install libssl-dev in the running container so the + # JIT build of hicache_hash_cpp (needs ) succeeds. + # Can be dropped once intel/sglang-dev:latest is rebuilt from a + # docker/xpu.Dockerfile that already ships libssl-dev. + docker exec -e DEBIAN_FRONTEND=noninteractive ci_sglang_xpu apt-get update + docker exec -e DEBIAN_FRONTEND=noninteractive ci_sglang_xpu apt-get install -y libssl-dev docker exec ci_sglang_xpu /opt/venv/bin/python3 -m pip install --upgrade pip - docker exec ci_sglang_xpu /opt/venv/bin/python3 -m pip install pytest expecttest ray huggingface_hub tabulate "lmcache>=0.3.9" + docker exec ci_sglang_xpu /opt/venv/bin/python3 -m pip install pytest expecttest ray huggingface_hub tabulate "lmcache>=0.3.9" accelerate docker exec ci_sglang_xpu /opt/venv/bin/python3 -m pip uninstall -y flashinfer-python sgl-kernel sglang docker exec ci_sglang_xpu cp /sglang-checkout/python/pyproject_xpu.toml /sglang-checkout/python/pyproject.toml # Fetch tags so setuptools_scm resolves a real version instead of @@ -120,102 +165,16 @@ jobs: docker exec ci_sglang_xpu /opt/venv/bin/python3 -m pip install --no-cache-dir --no-deps xgrammar==0.1.33 docker exec ci_sglang_xpu /bin/bash -c '/opt/venv/bin/hf auth login --token ${HF_TOKEN}' - - name: Run stage-a tests - timeout-minutes: 30 + - name: Install checkpoint-engine extra (optional; tests skip if unavailable) + continue-on-error: true run: | - docker exec ci_sglang_xpu bash -c "source /opt/venv/bin/activate && cd /sglang-checkout/test && python3 run_suite.py --hw xpu --suite stage-a-test-1-gpu-xpu --enable-retry" + docker exec -w /sglang-checkout/python ci_sglang_xpu /opt/venv/bin/python3 -m pip install --no-cache-dir ".[checkpoint-engine]" --extra-index-url https://download.pytorch.org/whl/xpu - - name: Cleanup container - if: always() + - name: Run tests + # Old stage-a (30) + stage-b (120) = 150; +20% headroom for --enable-retry. + timeout-minutes: 180 run: | - # pip install ran as root inside the container against the - # bind-mounted workspace, so build artifacts are root-owned on - # the host. Chown them back before rm to avoid Permission denied. - docker run --rm -v "${{ github.workspace }}:/w" busybox:latest \ - chown -R "$(id -u):$(id -g)" /w || true - # Wipe everything the run wrote into the workspace so the next - # job starts from a clean tree. - rm -rf \ - python/build \ - python/dist \ - python/sglang.egg-info \ - python/sglang/*.egg-info \ - test/result.jsonl \ - test/results \ - test/.pytest_cache \ - .pytest_cache || true - find . -type d -name "__pycache__" -prune -exec rm -rf {} + || true - find . -type f -name "*.pyc" -delete || true - # SIGTERM sglang and drain GPU context before `docker rm -f`; - # SIGKILL leaves the xe/GuC exec queue registered and triggers a - # GT reset (+ devcoredump) on B580. - if docker ps --format '{{.Names}}' | grep -qx ci_sglang_xpu; then - docker exec ci_sglang_xpu bash -c ' - pkill -TERM -f "sglang|run_suite|python3.*test_" 2>/dev/null || true - for _ in $(seq 1 30); do - pgrep -f "sglang::|sglang.launch_server" >/dev/null || break - sleep 1 - done - pkill -KILL -f "sglang|run_suite" 2>/dev/null || true - ' || true - fi - docker rm -f ci_sglang_xpu || true - if [[ -n "${CI_SGLANG_XPU_IMAGE:-}" ]]; then - docker rmi -f "${CI_SGLANG_XPU_IMAGE}" || true - fi - - # ==================== Wait for Stage A ==================== # - wait-for-stage-a: - needs: [stage-a-test-1-gpu-xpu] - runs-on: ubuntu-latest - steps: - - run: echo "stage-a passed" - - # ==================== Stage B ==================== # - stage-b-test-1-gpu-xpu: - needs: [check-changes, pr-gate, wait-for-stage-a] - if: needs.check-changes.outputs.main_package == 'true' - runs-on: intel-bmg - env: - DOCKERHUB_INTEL_USERNAME: ${{ secrets.DOCKERHUB_INTEL_USERNAME }} - DOCKERHUB_INTEL_TOKEN: ${{ secrets.DOCKERHUB_INTEL_TOKEN }} - steps: - - name: Reset workspace ownership - run: | - docker run --rm -v "${{ github.workspace }}:/w" busybox:latest \ - chown -R "$(id -u):$(id -g)" /w || true - - - name: Checkout code - uses: actions/checkout@v4 - with: - fetch-depth: 0 - ref: ${{ inputs.ref || github.ref }} - - - name: Start CI container - run: | - export HF_TOKEN="$(cat ~/huggingface_token.txt)" - bash scripts/ci/xpu/xpu_ci_start_container.sh - env: - GITHUB_WORKSPACE: ${{ github.workspace }} - - - name: Install Dependency - timeout-minutes: 60 - run: | - docker exec ci_sglang_xpu /opt/venv/bin/python3 -m pip install --upgrade pip - docker exec ci_sglang_xpu /opt/venv/bin/python3 -m pip install pytest expecttest ray huggingface_hub tabulate "lmcache>=0.3.9" - docker exec ci_sglang_xpu /opt/venv/bin/python3 -m pip uninstall -y flashinfer-python sgl-kernel sglang - docker exec ci_sglang_xpu cp /sglang-checkout/python/pyproject_xpu.toml /sglang-checkout/python/pyproject.toml - # Fetch tags so setuptools_scm resolves a real version instead of - # falling back to 0.0.0 on a shallow/tag-less checkout. - docker exec -w /sglang-checkout ci_sglang_xpu git fetch origin '+refs/tags/*:refs/tags/*' --force - docker exec -w /sglang-checkout/python ci_sglang_xpu /opt/venv/bin/python3 -m pip install --no-cache-dir . --extra-index-url https://download.pytorch.org/whl/xpu - docker exec ci_sglang_xpu /opt/venv/bin/python3 -m pip install --no-cache-dir --no-deps xgrammar==0.1.33 - docker exec ci_sglang_xpu /bin/bash -c '/opt/venv/bin/hf auth login --token ${HF_TOKEN}' - - - name: Run stage-b tests - timeout-minutes: 120 - run: | - docker exec ci_sglang_xpu bash -c "source /opt/venv/bin/activate && cd /sglang-checkout/test && python3 run_suite.py --hw xpu --suite stage-b-test-1-gpu-xpu --enable-retry" + docker exec ci_sglang_xpu bash -c "source /opt/venv/bin/activate && cd /sglang-checkout/test && python3 run_suite.py --hw xpu --suite stage-a-test-1-gpu-xpu,stage-b-test-1-gpu-xpu --enable-retry" - name: Cleanup container if: always() @@ -286,6 +245,12 @@ jobs: - name: Install Dependency timeout-minutes: 60 run: | + # Bridge fix: install libssl-dev in the running container so the + # JIT build of hicache_hash_cpp (needs ) succeeds. + # Can be dropped once intel/sglang-dev:latest is rebuilt from a + # docker/xpu.Dockerfile that already ships libssl-dev. + docker exec -e DEBIAN_FRONTEND=noninteractive ci_sglang_xpu apt-get update + docker exec -e DEBIAN_FRONTEND=noninteractive ci_sglang_xpu apt-get install -y libssl-dev docker exec ci_sglang_xpu /opt/venv/bin/python3 -m pip install --upgrade pip docker exec ci_sglang_xpu /opt/venv/bin/python3 -m pip install pytest expecttest ray huggingface_hub tabulate "lmcache>=0.3.9" docker exec ci_sglang_xpu /opt/venv/bin/python3 -m pip uninstall -y flashinfer-python sgl-kernel sglang @@ -300,7 +265,9 @@ jobs: - name: Run diffusion server tests (1-GPU) timeout-minutes: 60 run: | - docker exec ci_sglang_xpu bash -c "source /opt/venv/bin/activate && cd /sglang-checkout/python && python3 sglang/multimodal_gen/test/run_suite.py --suite 1-gpu-xpu" + # xpu_b60.json is seeded with this flag on; without it the SYCL queue + # is deep enough that step timings record host enqueue, not device time. + docker exec -e SGLANG_DIFFUSION_SYNC_STAGE_PROFILING=1 ci_sglang_xpu bash -c "source /opt/venv/bin/activate && cd /sglang-checkout/python && python3 sglang/multimodal_gen/test/run_suite.py --suite 1-gpu-xpu" - name: Cleanup container if: always() @@ -338,22 +305,17 @@ jobs: finish: if: always() - needs: [stage-a-test-1-gpu-xpu, stage-b-test-1-gpu-xpu, multimodal-gen-test-1-gpu-xpu, pr-gate] + needs: [stage-a-test-1-gpu-xpu, multimodal-gen-test-1-gpu-xpu, pr-gate] runs-on: ubuntu-latest steps: - name: Check job status run: | stage_a="${{ needs.stage-a-test-1-gpu-xpu.result }}" - stage_b="${{ needs.stage-b-test-1-gpu-xpu.result }}" multimodal_gen="${{ needs.multimodal-gen-test-1-gpu-xpu.result }}" if [ "$stage_a" != "success" ] && [ "$stage_a" != "skipped" ]; then echo "stage-a failed with result: $stage_a" exit 1 fi - if [ "$stage_b" != "success" ] && [ "$stage_b" != "skipped" ]; then - echo "stage-b failed with result: $stage_b" - exit 1 - fi if [ "$multimodal_gen" != "success" ] && [ "$multimodal_gen" != "skipped" ]; then echo "multimodal-gen failed with result: $multimodal_gen" exit 1 diff --git a/.github/workflows/pr-test.yml b/.github/workflows/pr-test.yml index 5ebae988b9b6..92ac2038e446 100644 --- a/.github/workflows/pr-test.yml +++ b/.github/workflows/pr-test.yml @@ -41,17 +41,17 @@ on: type: boolean default: false skip_pr_test_health_check: - description: "Skip PR test health check fast-fail (e.g. for release branch cuts)" + description: "Skip PR test health check fail-fast (e.g. for release branch cuts)" required: false type: boolean default: false concurrency: - # Concurrency group structure: pr-test-{event}-{branch}-{git_ref} # - event_name prevents scheduled runs from colliding with fork PRs whose branch is named 'main' # (without it, both resolve the branch segment to 'main' and block each other) - # - github.head_ref (pull_request) or github.ref_name (workflow_dispatch) normalizes to branch name - group: pr-test-${{ github.event_name }}-${{ github.head_ref || github.ref_name || 'default' }}-${{ inputs.git_ref || 'all' }} + # - a PR keys on its number: github.head_ref is a bare branch name with no owner, + # so two forks using the same name would share one group and cancel each other + group: pr-test-${{ github.event_name }}-${{ github.event.pull_request.number || github.ref_name || 'default' }}-${{ inputs.git_ref || 'all' }} cancel-in-progress: ${{ github.event_name != 'workflow_call' }} env: @@ -88,11 +88,6 @@ jobs: secrets: inherit # =============================================== Wait Jobs for Sequential PR Execution ==================================================== - # These jobs poll GitHub API to wait for previous stages to complete. - # For PR runs: wait jobs run and enforce sequential execution via polling. - # For scheduled runs: wait jobs are skipped, enabling parallel execution for easier retry. - # For PRs with the `bypass-fastfail` label: wait jobs run but return success immediately - # (handled inside the wait-for-jobs action), so downstream stages dispatch in parallel. wait-for-base-a: needs: [check-changes, call-gate] @@ -166,7 +161,8 @@ jobs: needs.check-changes.outputs.main_package == 'true' || needs.check-changes.outputs.sgl_kernel == 'true' || needs.check-changes.outputs.jit_kernel == 'true' || - needs.check-changes.outputs.multimodal_gen == 'true' + needs.check-changes.outputs.multimodal_gen == 'true' || + needs.check-changes.outputs.renderer == 'true' ) uses: ./.github/workflows/pr-gate.yml secrets: inherit @@ -187,7 +183,7 @@ jobs: runs_on: x64-kernel-build-node job_display_name: Build Wheel git_ref: ${{ inputs.git_ref || '' }} - skip_pr_test_health_check: ${{ inputs.skip_pr_test_health_check == true }} + skip_pr_test_health_check: ${{ inputs.skip_pr_test_health_check == true || inputs.test_parallel_dispatch == true || inputs.run_all_tests == true }} secrets: inherit sgl-kernel-build-wheels-arm: @@ -205,13 +201,13 @@ jobs: job_display_name: Build Wheel Arm arch_suffix: '-aarch64' git_ref: ${{ inputs.git_ref || '' }} - skip_pr_test_health_check: ${{ inputs.skip_pr_test_health_check == true }} + skip_pr_test_health_check: ${{ inputs.skip_pr_test_health_check == true || inputs.test_parallel_dispatch == true || inputs.run_all_tests == true }} secrets: inherit # =============================================== Rust extensions ==================================================== - # Unlike sgl-kernel, not gated on source changes: the cache key is the source - # hash, so the job always runs but only compiles on a miss. + # diffusion-only PRs do not consume SRT Rust extensions; other runs seed or + # reuse the source-hash cache rust-ext-build: needs: [check-changes, call-gate] # Ungated on the event, unlike the jobs above: a cache entry is visible to @@ -223,12 +219,42 @@ jobs: if: | !cancelled() && needs.check-changes.result == 'success' && + (github.event_name != 'pull_request' || + needs.check-changes.outputs.multimodal_gen != 'true' || + needs.check-changes.outputs.main_package == 'true' || + needs.check-changes.outputs.jit_kernel == 'true' || + needs.check-changes.outputs.sgl_kernel == 'true') && (needs.call-gate.result == 'success' || needs.call-gate.result == 'skipped') uses: ./.github/workflows/_pr-test-rust-ext-build.yml with: - runs_on: 1-gpu-5090 + runs_on: ubuntu-22.04 git_ref: ${{ inputs.git_ref || '' }} - skip_pr_test_health_check: ${{ inputs.skip_pr_test_health_check == true }} + skip_pr_test_health_check: ${{ inputs.skip_pr_test_health_check == true || inputs.test_parallel_dispatch == true || inputs.run_all_tests == true }} + secrets: inherit + + # gb300 is Grace, and PyO3 modules are portable across neither arch nor + # interpreter, so it can use nothing the job above builds. + rust-ext-build-aarch64: + needs: [check-changes, call-gate] + # Same gate as rust-ext-build: every main-ref run has to seed the cache, and a + # docs-only PR still installs sglang on gb300. + if: | + !cancelled() && + needs.check-changes.result == 'success' && + (github.event_name != 'pull_request' || + needs.check-changes.outputs.multimodal_gen != 'true' || + needs.check-changes.outputs.main_package == 'true' || + needs.check-changes.outputs.jit_kernel == 'true' || + needs.check-changes.outputs.sgl_kernel == 'true') && + (needs.call-gate.result == 'success' || needs.call-gate.result == 'skipped') + uses: ./.github/workflows/_pr-test-rust-ext-build.yml + with: + runs_on: ubuntu-22.04-arm + collect_runs_on: ubuntu-22.04-arm + artifact_name: rust-ext-aarch64 + cache_key_prefix: rust-ext-aarch64-cp310-cp312 + git_ref: ${{ inputs.git_ref || '' }} + skip_pr_test_health_check: ${{ inputs.skip_pr_test_health_check == true || inputs.test_parallel_dispatch == true || inputs.run_all_tests == true }} secrets: inherit call-sgl-kernel-tests: @@ -243,7 +269,7 @@ jobs: runs_on_map: ${{ needs.check-changes.outputs.runs_on_map }} sgl_kernel: ${{ needs.check-changes.outputs.sgl_kernel }} git_ref: ${{ inputs.git_ref || '' }} - skip_pr_test_health_check: ${{ inputs.skip_pr_test_health_check == true }} + skip_pr_test_health_check: ${{ inputs.skip_pr_test_health_check == true || inputs.test_parallel_dispatch == true || inputs.run_all_tests == true }} rust_ext_artifact: ${{ needs.rust-ext-build.outputs.artifact_name }} secrets: inherit @@ -260,7 +286,6 @@ jobs: needs.check-changes.outputs.jit_kernel == 'true' uses: ./.github/workflows/pr-test-jit-kernel.yml with: - runner_config: 4-gpu-b200 runs_on_map: ${{ needs.check-changes.outputs.runs_on_map }} jit_kernel: ${{ needs.check-changes.outputs.jit_kernel }} # On scheduled/parallel-dispatch runs sgl-kernel-build-wheels is skipped, so the wheel @@ -269,24 +294,33 @@ jobs: sgl_kernel: ${{ (github.event_name == 'schedule' || inputs.test_parallel_dispatch == true) && 'false' || needs.check-changes.outputs.sgl_kernel }} git_ref: ${{ inputs.git_ref || '' }} test_parallel_dispatch: ${{ inputs.test_parallel_dispatch == true && 'true' || 'false' }} - skip_pr_test_health_check: ${{ inputs.skip_pr_test_health_check == true }} + skip_pr_test_health_check: ${{ inputs.skip_pr_test_health_check == true || inputs.test_parallel_dispatch == true || inputs.run_all_tests == true }} rust_ext_artifact: ${{ needs.rust-ext-build.outputs.artifact_name }} secrets: inherit # =============================================== primary ==================================================== - # Runs on 5090 (32GB, SM120) - base-a-test-1-gpu-small: + # `name:` is load-bearing: wait-for-jobs gates a stage by job-name prefix, + # and check-pr-test-health carves h20 out by the same name. + base-a-test: + name: base-a-test-${{ matrix.runner_config }} needs: [check-changes, call-gate, sgl-kernel-build-wheels, rust-ext-build] if: ${{ !failure() && !cancelled() }} + strategy: + fail-fast: false + matrix: + include: + # 5090 (32GB, SM120) + - { runner_config: 1-gpu-small, run_timeout: '10', timeout_per_file: '' } uses: ./.github/workflows/_pr-test-stage.yml with: - self_name: base-a-test-1-gpu-small - runner_config: 1-gpu-small + self_name: base-a-test-${{ matrix.runner_config }} + runner_config: ${{ matrix.runner_config }} check_changes: ${{ toJson(needs.check-changes.outputs) }} caller_inputs: ${{ toJson(inputs) }} partitions: ${{ needs.check-changes.outputs.partitions }} - run_timeout_minutes: '10' + run_timeout_minutes: ${{ matrix.run_timeout }} + timeout_per_file: ${{ matrix.timeout_per_file }} rust_ext_artifact: ${{ needs.rust-ext-build.outputs.artifact_name }} secrets: inherit @@ -307,67 +341,33 @@ jobs: # No `secrets: inherit`: this stage has no secret consumer, unlike the GPU # stages' coredump upload. GITHUB_TOKEN and permissions inherit regardless. - # Runs on 5090 (32GB, SM120) - base-b-test-1-gpu-small: + base-b-test: + name: base-b-test-${{ matrix.runner_config }} needs: [check-changes, call-gate, wait-for-base-a, sgl-kernel-build-wheels, rust-ext-build] if: ${{ !failure() && !cancelled() }} + strategy: + fail-fast: false + matrix: + include: + # 5090 (32GB, SM120) + - { runner_config: 1-gpu-small, run_timeout: '30', timeout_per_file: '' } + - { runner_config: 1-gpu-large, run_timeout: '60', timeout_per_file: '1800' } + - { runner_config: 2-gpu-large, run_timeout: '30', timeout_per_file: '' } + - { runner_config: 4-gpu-b200, run_timeout: '40', timeout_per_file: '' } uses: ./.github/workflows/_pr-test-stage.yml with: - self_name: base-b-test-1-gpu-small - runner_config: 1-gpu-small + self_name: base-b-test-${{ matrix.runner_config }} + runner_config: ${{ matrix.runner_config }} check_changes: ${{ toJson(needs.check-changes.outputs) }} caller_inputs: ${{ toJson(inputs) }} partitions: ${{ needs.check-changes.outputs.partitions }} - run_timeout_minutes: '30' - rust_ext_artifact: ${{ needs.rust-ext-build.outputs.artifact_name }} - secrets: inherit - - # Runs on H100 (80GB, SM90) - tests that don't pass on 5090 (FA3, FP8, high VRAM, etc.) - base-b-test-1-gpu-large: - needs: [check-changes, call-gate, wait-for-base-a, sgl-kernel-build-wheels, rust-ext-build] - if: ${{ !failure() && !cancelled() }} - uses: ./.github/workflows/_pr-test-stage.yml - with: - self_name: base-b-test-1-gpu-large - runner_config: 1-gpu-large - check_changes: ${{ toJson(needs.check-changes.outputs) }} - caller_inputs: ${{ toJson(inputs) }} - partitions: ${{ needs.check-changes.outputs.partitions }} - run_timeout_minutes: '30' - timeout_per_file: '1800' - rust_ext_artifact: ${{ needs.rust-ext-build.outputs.artifact_name }} - secrets: inherit - - base-b-test-2-gpu-large: - needs: [check-changes, call-gate, wait-for-base-a, sgl-kernel-build-wheels, rust-ext-build] - if: ${{ !failure() && !cancelled() }} - uses: ./.github/workflows/_pr-test-stage.yml - with: - self_name: base-b-test-2-gpu-large - runner_config: 2-gpu-large - check_changes: ${{ toJson(needs.check-changes.outputs) }} - caller_inputs: ${{ toJson(inputs) }} - partitions: ${{ needs.check-changes.outputs.partitions }} - run_timeout_minutes: '30' - rust_ext_artifact: ${{ needs.rust-ext-build.outputs.artifact_name }} - secrets: inherit - - base-b-test-4-gpu-b200: - needs: [check-changes, call-gate, wait-for-base-a, sgl-kernel-build-wheels, rust-ext-build] - if: ${{ !failure() && !cancelled() }} - uses: ./.github/workflows/_pr-test-stage.yml - with: - self_name: base-b-test-4-gpu-b200 - runner_config: 4-gpu-b200 - check_changes: ${{ toJson(needs.check-changes.outputs) }} - caller_inputs: ${{ toJson(inputs) }} - partitions: ${{ needs.check-changes.outputs.partitions }} - run_timeout_minutes: '40' + run_timeout_minutes: ${{ matrix.run_timeout }} + timeout_per_file: ${{ matrix.timeout_per_file }} rust_ext_artifact: ${{ needs.rust-ext-build.outputs.artifact_name }} secrets: inherit call-multimodal-gen-tests: - needs: [check-changes, call-gate, sgl-kernel-build-wheels, rust-ext-build] + needs: [check-changes, call-gate, sgl-kernel-build-wheels] if: | always() && !cancelled() && @@ -383,107 +383,71 @@ jobs: git_ref: ${{ inputs.git_ref || '' }} test_parallel_dispatch: ${{ inputs.test_parallel_dispatch == true && 'true' || 'false' }} caller_needs_failure: ${{ (needs.call-gate.result == 'failure' || needs.sgl-kernel-build-wheels.result == 'failure' || needs.check-changes.result == 'failure') && 'true' || 'false' }} - skip_pr_test_health_check: ${{ inputs.skip_pr_test_health_check == true && 'true' || 'false' }} - rust_ext_artifact: ${{ needs.rust-ext-build.outputs.artifact_name }} - secrets: inherit - - base-c-test-4-gpu-h100: - needs: [check-changes, call-gate, wait-for-base-b, sgl-kernel-build-wheels, rust-ext-build] - if: ${{ !failure() && !cancelled() }} - uses: ./.github/workflows/_pr-test-stage.yml - with: - self_name: base-c-test-4-gpu-h100 - runner_config: 4-gpu-h100 - check_changes: ${{ toJson(needs.check-changes.outputs) }} - caller_inputs: ${{ toJson(inputs) }} - partitions: ${{ needs.check-changes.outputs.partitions }} - run_timeout_minutes: '30' - warmup_deep_gemm_models: 'lmsys/sglang-ci-dsv3-test:4' - warmup_server_models: 'lmsys/sglang-ci-dsv3-test:4' - rust_ext_artifact: ${{ needs.rust-ext-build.outputs.artifact_name }} + skip_pr_test_health_check: ${{ (inputs.skip_pr_test_health_check == true || inputs.test_parallel_dispatch == true || inputs.run_all_tests == true) && 'true' || 'false' }} secrets: inherit - base-c-test-8-gpu-h200: + base-c-test: + name: base-c-test-${{ matrix.runner_config }} needs: [check-changes, call-gate, wait-for-base-b, sgl-kernel-build-wheels, rust-ext-build] if: ${{ !failure() && !cancelled() }} + strategy: + fail-fast: false + matrix: + include: + - { + runner_config: 4-gpu-h100, + run_timeout: '30', + timeout_per_file: '', + warmup_deep_gemm_models: 'lmsys/sglang-ci-dsv3-test:4', + warmup_server_models: 'lmsys/sglang-ci-dsv3-test:4', + } + - { + runner_config: 8-gpu-h200, + run_timeout: '30', + timeout_per_file: '1800', + warmup_deep_gemm_models: 'deepseek-ai/DeepSeek-V3-0324:8 deepseek-ai/DeepSeek-V3.2:8 zai-org/GLM-5-FP8:8 XiaomiMiMo/MiMo-V2-Flash:4 XiaomiMiMo/MiMo-V2.5:8', + warmup_timeout: '60', + } + - { runner_config: 8-gpu-h20, run_timeout: '30', timeout_per_file: '' } + - { runner_config: 4-gpu-b200, run_timeout: '30', timeout_per_file: '1800' } + - { runner_config: 8-gpu-b300, run_timeout: '60', timeout_per_file: '3600' } uses: ./.github/workflows/_pr-test-stage.yml with: - self_name: base-c-test-8-gpu-h200 - runner_config: 8-gpu-h200 + self_name: base-c-test-${{ matrix.runner_config }} + runner_config: ${{ matrix.runner_config }} check_changes: ${{ toJson(needs.check-changes.outputs) }} caller_inputs: ${{ toJson(inputs) }} partitions: ${{ needs.check-changes.outputs.partitions }} - run_timeout_minutes: '30' - timeout_per_file: '1800' - # Per-model TP must match the test's launch in test/registered/ -- see - # FALLBACK_ARGS in scripts/ci/cuda/warmup_deep_gemm.py for extra dp/ep - # flags. Only models that actually invoke DeepGEMM kernels at runtime - # are listed. Cold-cache ~13 min; warm-cache <=30 s via marker file. - # Server CUDA Graph warmup is combined into this step (warmup_server_models unset). - warmup_deep_gemm_models: 'deepseek-ai/DeepSeek-V3-0324:8 deepseek-ai/DeepSeek-V3.2:8 zai-org/GLM-5-FP8:8 XiaomiMiMo/MiMo-V2-Flash:4 XiaomiMiMo/MiMo-V2.5:8' - warmup_timeout_minutes: '60' + run_timeout_minutes: ${{ matrix.run_timeout }} + timeout_per_file: ${{ matrix.timeout_per_file }} + warmup_deep_gemm_models: ${{ matrix.warmup_deep_gemm_models }} + warmup_server_models: ${{ matrix.warmup_server_models }} + # Rows that do not set it keep _pr-test-stage.yml's own default. + warmup_timeout_minutes: ${{ matrix.warmup_timeout || '25' }} rust_ext_artifact: ${{ needs.rust-ext-build.outputs.artifact_name }} secrets: inherit - base-c-test-8-gpu-h20: - needs: [check-changes, call-gate, wait-for-base-b, sgl-kernel-build-wheels, rust-ext-build] + # Separate only because `needs` cannot vary per matrix row: this stage takes + # the aarch64 build, and neither table should wait on the other's. + base-c-test-aarch64: + name: base-c-test-${{ matrix.runner_config }} + needs: [check-changes, call-gate, wait-for-base-b, sgl-kernel-build-wheels, rust-ext-build-aarch64] if: ${{ !failure() && !cancelled() }} + strategy: + fail-fast: false + matrix: + include: + - { runner_config: 4-gpu-gb300, run_timeout: '30', timeout_per_file: '1800' } uses: ./.github/workflows/_pr-test-stage.yml with: - self_name: base-c-test-8-gpu-h20 - runner_config: 8-gpu-h20 + self_name: base-c-test-${{ matrix.runner_config }} + runner_config: ${{ matrix.runner_config }} check_changes: ${{ toJson(needs.check-changes.outputs) }} caller_inputs: ${{ toJson(inputs) }} partitions: ${{ needs.check-changes.outputs.partitions }} - run_timeout_minutes: '30' - rust_ext_artifact: ${{ needs.rust-ext-build.outputs.artifact_name }} - secrets: inherit - - base-c-test-4-gpu-b200: - needs: [check-changes, call-gate, wait-for-base-b, sgl-kernel-build-wheels, rust-ext-build] - if: ${{ !failure() && !cancelled() }} - uses: ./.github/workflows/_pr-test-stage.yml - with: - self_name: base-c-test-4-gpu-b200 - runner_config: 4-gpu-b200 - check_changes: ${{ toJson(needs.check-changes.outputs) }} - caller_inputs: ${{ toJson(inputs) }} - partitions: ${{ needs.check-changes.outputs.partitions }} - run_timeout_minutes: '30' - timeout_per_file: '1800' - rust_ext_artifact: ${{ needs.rust-ext-build.outputs.artifact_name }} - secrets: inherit - - base-c-test-4-gpu-gb300: - needs: [check-changes, call-gate, wait-for-base-b, sgl-kernel-build-wheels, rust-ext-build] - if: ${{ !failure() && !cancelled() }} - uses: ./.github/workflows/_pr-test-stage.yml - with: - self_name: base-c-test-4-gpu-gb300 - runner_config: 4-gpu-gb300 - check_changes: ${{ toJson(needs.check-changes.outputs) }} - caller_inputs: ${{ toJson(inputs) }} - partitions: ${{ needs.check-changes.outputs.partitions }} - run_timeout_minutes: '30' - timeout_per_file: '1800' - # aarch64 has no prebuild, so this stage compiles during install - which also - # makes it the only stage still covering that path. Keep it that way. - skip_prebuilt_rust_ext: true - secrets: inherit - - base-c-test-8-gpu-b300: - needs: [check-changes, call-gate, wait-for-base-b, sgl-kernel-build-wheels, rust-ext-build] - if: ${{ !failure() && !cancelled() }} - uses: ./.github/workflows/_pr-test-stage.yml - with: - self_name: base-c-test-8-gpu-b300 - runner_config: 8-gpu-b300 - check_changes: ${{ toJson(needs.check-changes.outputs) }} - caller_inputs: ${{ toJson(inputs) }} - partitions: ${{ needs.check-changes.outputs.partitions }} - run_timeout_minutes: '60' - timeout_per_file: '3600' - rust_ext_artifact: ${{ needs.rust-ext-build.outputs.artifact_name }} + run_timeout_minutes: ${{ matrix.run_timeout }} + timeout_per_file: ${{ matrix.timeout_per_file }} + rust_ext_artifact: ${{ needs.rust-ext-build-aarch64.outputs.artifact_name }} secrets: inherit # List every build and test job: `skipped` passes here, so an omission turns that @@ -499,6 +463,7 @@ jobs: call-sgl-kernel-tests, rust-ext-build, + rust-ext-build-aarch64, wait-for-base-a, wait-for-base-b, @@ -507,18 +472,11 @@ jobs: call-multimodal-gen-tests, - base-a-test-1-gpu-small, + base-a-test, base-a-test-cpu, - base-b-test-1-gpu-small, - base-b-test-1-gpu-large, - base-b-test-2-gpu-large, - base-b-test-4-gpu-b200, - base-c-test-4-gpu-h100, - base-c-test-8-gpu-h20, - base-c-test-8-gpu-h200, - base-c-test-4-gpu-b200, - base-c-test-4-gpu-gb300, - base-c-test-8-gpu-b300, + base-b-test, + base-c-test, + base-c-test-aarch64, ] if: always() runs-on: ubuntu-latest diff --git a/.github/workflows/release-branch-cut.yml b/.github/workflows/release-branch-cut.yml index 8c733c8dc1bb..1edeec727ea5 100644 --- a/.github/workflows/release-branch-cut.yml +++ b/.github/workflows/release-branch-cut.yml @@ -174,10 +174,10 @@ jobs: run-pr-tests-amd: needs: cut-release-branch - uses: ./.github/workflows/pr-test-amd-rocm720.yml + uses: ./.github/workflows/pr-test-amd.yml with: ref: ${{ needs.cut-release-branch.outputs.branch_name }} - rocm_version: rocm724 + rocm_version: rocm10 run_all_tests: true secrets: inherit @@ -214,10 +214,10 @@ jobs: run-nightly-tests-amd: needs: cut-release-branch - uses: ./.github/workflows/nightly-test-amd-rocm720.yml + uses: ./.github/workflows/nightly-test-amd.yml with: ref: ${{ needs.cut-release-branch.outputs.branch_name }} - rocm_version: rocm724 + rocm_version: rocm10 secrets: inherit run-nightly-tests-npu: diff --git a/.github/workflows/release-docker-amd-gfx1151-nightly.yml b/.github/workflows/release-docker-amd-gfx1151-nightly.yml new file mode 100644 index 000000000000..a026cb76a7a2 --- /dev/null +++ b/.github/workflows/release-docker-amd-gfx1151-nightly.yml @@ -0,0 +1,63 @@ +name: Release Docker Images Nightly gfx1151 (AMD) + +on: + workflow_dispatch: + schedule: + # Stagger this build from the CDNA ROCm nightlies at 12:00 UTC. + - cron: "0 14 * * *" + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +permissions: + contents: read + +jobs: + publish: + if: github.repository == 'sgl-project/sglang' + runs-on: amd-docker-scale + environment: prod + timeout-minutes: 120 + steps: + - name: Checkout repository + uses: actions/checkout@v4 + with: + fetch-depth: 0 + + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: "3.10" + + - name: Set date + run: echo "DATE=$(date +%Y%m%d)" >> "$GITHUB_ENV" + + - name: Get version from latest tag + id: version + run: | + VERSION=$(python3 scripts/release/get_version_tag.py --tag-only | sed 's/^v//') + if [ -z "$VERSION" ]; then + echo "::error::Could not determine version from git tags" + exit 1 + fi + echo "version=${VERSION}" >> "$GITHUB_OUTPUT" + + - name: Log in to Docker Hub + uses: docker/login-action@v3 + with: + username: ${{ secrets.DOCKERHUB_AMD_USERNAME }} + password: ${{ secrets.DOCKERHUB_AMD_TOKEN }} + + - name: Build and push to rocm/sgl-dev + env: + VERSION: ${{ steps.version.outputs.version }} + run: | + IMAGE_TAG="v${VERSION}-rocm724-gfx1151-${DATE}" + docker build . \ + -f docker/rocm-gfx1151.Dockerfile \ + --no-cache \ + --progress=plain \ + -t "rocm/sgl-dev:${IMAGE_TAG}" + docker push "rocm/sgl-dev:${IMAGE_TAG}" + echo "Published rocm/sgl-dev:${IMAGE_TAG}" >> "$GITHUB_STEP_SUMMARY" diff --git a/.github/workflows/release-docker-amd-miles-rocm10-nightly.yml b/.github/workflows/release-docker-amd-miles-rocm10-nightly.yml new file mode 100644 index 000000000000..166d110cac1e --- /dev/null +++ b/.github/workflows/release-docker-amd-miles-rocm10-nightly.yml @@ -0,0 +1,60 @@ +name: Release Docker Images Nightly Miles ROCm10 (AMD) + +on: + workflow_dispatch: + schedule: + - cron: "0 12 * * *" + +concurrency: + group: ${{ github.workflow }}-${{ github.event.number || github.sha }} + cancel-in-progress: true + +jobs: + publish: + if: github.repository == 'sgl-project/sglang' + runs-on: amd-docker-scale + environment: "prod" + permissions: + contents: read + strategy: + fail-fast: false + matrix: + variant: + - rocm10-mi35x + steps: + - name: Checkout miles repository + uses: actions/checkout@v4 + with: + repository: radixark/miles + ref: main + fetch-depth: 0 + persist-credentials: false + + - name: Record miles commit + run: echo "Building from miles main @ $(git rev-parse HEAD)" + + - name: Set date + run: echo "DATE=$(date +%Y%m%d)" >> $GITHUB_ENV + + - name: Install docker/build.py dependencies + run: | + sudo apt-get update + sudo apt-get install -y python3-pip + python3 -m pip install --break-system-packages --quiet typer + + - name: Build image via miles docker/build.py and tag with date + run: | + python3 docker/build.py --variant ${{ matrix.variant }} --image-tag custom --custom-tag miles + docker tag rocm/sgl-dev:miles-${{ matrix.variant }} rocm/sgl-dev:miles-${{ matrix.variant }}-${DATE} + + - name: Login to Docker Hub (AMD) + uses: docker/login-action@v2 + with: + username: ${{ secrets.DOCKERHUB_AMD_USERNAME }} + password: ${{ secrets.DOCKERHUB_AMD_TOKEN }} + + - name: Push dated tag + run: docker push rocm/sgl-dev:miles-${{ matrix.variant }}-${DATE} + + - name: Push the undated tag + run: docker push rocm/sgl-dev:miles-${{ matrix.variant }} diff --git a/.github/workflows/release-docker-amd-miles-rocm700-nightly.yml b/.github/workflows/release-docker-amd-miles-rocm700-nightly.yml deleted file mode 100644 index 0ea0e8b5e52e..000000000000 --- a/.github/workflows/release-docker-amd-miles-rocm700-nightly.yml +++ /dev/null @@ -1,61 +0,0 @@ -name: Release Docker Images Nightly Miles ROCm7.0 (AMD) - -on: - workflow_dispatch: - schedule: - - cron: "0 12 * * *" - -concurrency: - group: ${{ github.workflow }}-${{ github.event.number || github.sha }} - cancel-in-progress: true - -jobs: - publish: - if: github.repository == 'sgl-project/sglang' - runs-on: amd-docker-scale - environment: "prod" - permissions: - contents: read - strategy: - fail-fast: false - matrix: - variant: - - rocm700-mi35x - - rocm700-mi30x - steps: - - name: Checkout miles repository - uses: actions/checkout@v4 - with: - repository: radixark/miles - ref: main - fetch-depth: 0 - persist-credentials: false - - - name: Record miles commit - run: echo "Building from miles main @ $(git rev-parse HEAD)" - - - name: Set date - run: echo "DATE=$(date +%Y%m%d)" >> $GITHUB_ENV - - - name: Install docker/build.py dependencies - run: | - sudo apt-get update - sudo apt-get install -y python3-pip - python3 -m pip install --break-system-packages --quiet typer - - - name: Build image via miles docker/build.py and tag with date - run: | - python3 docker/build.py --variant ${{ matrix.variant }} --image-tag custom --custom-tag miles - docker tag rocm/sgl-dev:miles-${{ matrix.variant }} rocm/sgl-dev:miles-${{ matrix.variant }}-${DATE} - - - name: Login to Docker Hub (AMD) - uses: docker/login-action@v2 - with: - username: ${{ secrets.DOCKERHUB_AMD_USERNAME }} - password: ${{ secrets.DOCKERHUB_AMD_TOKEN }} - - - name: Push dated tag - run: docker push rocm/sgl-dev:miles-${{ matrix.variant }}-${DATE} - - - name: Push the undated tag - run: docker push rocm/sgl-dev:miles-${{ matrix.variant }} diff --git a/.github/workflows/release-docker-amd-miles-rocm720-nightly.yml b/.github/workflows/release-docker-amd-miles-rocm720-nightly.yml deleted file mode 100644 index e3296b12508e..000000000000 --- a/.github/workflows/release-docker-amd-miles-rocm720-nightly.yml +++ /dev/null @@ -1,60 +0,0 @@ -name: Release Docker Images Nightly Miles ROCm7.2 (AMD) - -on: - workflow_dispatch: - schedule: - - cron: "0 12 * * *" - -concurrency: - group: ${{ github.workflow }}-${{ github.event.number || github.sha }} - cancel-in-progress: true - -jobs: - publish: - if: github.repository == 'sgl-project/sglang' - runs-on: amd-docker-scale - environment: "prod" - permissions: - contents: read - strategy: - fail-fast: false - matrix: - variant: - - rocm720-mi35x - steps: - - name: Checkout miles repository - uses: actions/checkout@v4 - with: - repository: radixark/miles - ref: main - fetch-depth: 0 - persist-credentials: false - - - name: Record miles commit - run: echo "Building from miles main @ $(git rev-parse HEAD)" - - - name: Set date - run: echo "DATE=$(date +%Y%m%d)" >> $GITHUB_ENV - - - name: Install docker/build.py dependencies - run: | - sudo apt-get update - sudo apt-get install -y python3-pip - python3 -m pip install --break-system-packages --quiet typer - - - name: Build image via miles docker/build.py and tag with date - run: | - python3 docker/build.py --variant ${{ matrix.variant }} --image-tag custom --custom-tag miles - docker tag rocm/sgl-dev:miles-${{ matrix.variant }} rocm/sgl-dev:miles-${{ matrix.variant }}-${DATE} - - - name: Login to Docker Hub (AMD) - uses: docker/login-action@v2 - with: - username: ${{ secrets.DOCKERHUB_AMD_USERNAME }} - password: ${{ secrets.DOCKERHUB_AMD_TOKEN }} - - - name: Push dated tag - run: docker push rocm/sgl-dev:miles-${{ matrix.variant }}-${DATE} - - - name: Push the undated tag - run: docker push rocm/sgl-dev:miles-${{ matrix.variant }} diff --git a/.github/workflows/release-docker-amd-miles-rocm724-nightly.yml b/.github/workflows/release-docker-amd-miles-rocm724-nightly.yml new file mode 100644 index 000000000000..7851946d894f --- /dev/null +++ b/.github/workflows/release-docker-amd-miles-rocm724-nightly.yml @@ -0,0 +1,60 @@ +name: Release Docker Images Nightly Miles ROCm7.2.4 (AMD) + +on: + workflow_dispatch: + schedule: + - cron: "0 12 * * *" + +concurrency: + group: ${{ github.workflow }}-${{ github.event.number || github.sha }} + cancel-in-progress: true + +jobs: + publish: + if: github.repository == 'sgl-project/sglang' + runs-on: amd-docker-scale + environment: "prod" + permissions: + contents: read + strategy: + fail-fast: false + matrix: + variant: + - rocm724-mi35x + steps: + - name: Checkout miles repository + uses: actions/checkout@v4 + with: + repository: radixark/miles + ref: main + fetch-depth: 0 + persist-credentials: false + + - name: Record miles commit + run: echo "Building from miles main @ $(git rev-parse HEAD)" + + - name: Set date + run: echo "DATE=$(date +%Y%m%d)" >> $GITHUB_ENV + + - name: Install docker/build.py dependencies + run: | + sudo apt-get update + sudo apt-get install -y python3-pip + python3 -m pip install --break-system-packages --quiet typer + + - name: Build image via miles docker/build.py and tag with date + run: | + python3 docker/build.py --variant ${{ matrix.variant }} --image-tag custom --custom-tag miles + docker tag rocm/sgl-dev:miles-${{ matrix.variant }} rocm/sgl-dev:miles-${{ matrix.variant }}-${DATE} + + - name: Login to Docker Hub (AMD) + uses: docker/login-action@v2 + with: + username: ${{ secrets.DOCKERHUB_AMD_USERNAME }} + password: ${{ secrets.DOCKERHUB_AMD_TOKEN }} + + - name: Push dated tag + run: docker push rocm/sgl-dev:miles-${{ matrix.variant }}-${DATE} + + - name: Push the undated tag + run: docker push rocm/sgl-dev:miles-${{ matrix.variant }} diff --git a/.github/workflows/release-docker-amd-nightly.yml b/.github/workflows/release-docker-amd-nightly.yml deleted file mode 100644 index 98b958d9f8ed..000000000000 --- a/.github/workflows/release-docker-amd-nightly.yml +++ /dev/null @@ -1,252 +0,0 @@ -name: Release Docker Images Nightly ROCm7.0 (AMD) -on: - workflow_dispatch: - schedule: - - cron: '0 12 * * *' - -concurrency: - # A PR number if a pull request and otherwise the commit hash. This cancels - # queued and in-progress runs for the same PR (presubmit) or commit - # (postsubmit). The workflow name is prepended to avoid conflicts between - # different workflows. - group: ${{ github.workflow }}-${{ github.event.number || github.sha }} - cancel-in-progress: true - -jobs: - publish: - if: github.repository == 'sgl-project/sglang' - runs-on: amd-docker-scale - environment: 'prod' - strategy: - fail-fast: false - matrix: - gpu_arch: ['gfx942', 'gfx950'] - build_type: ['all'] - steps: - - name: Checkout repository - uses: actions/checkout@v4 - with: - fetch-depth: 0 # Required for git describe to find tags - - - name: Set up Python - uses: actions/setup-python@v5 - with: - python-version: "3.10" - - - name: "Set Date" - run: | - echo "DATE=$(date +%Y%m%d)" >> $GITHUB_ENV - - - name: Get version from latest tag - id: version - run: | - # Use the shared helper so stable/post releases sort above rc tags. - VERSION=$(python3 scripts/release/get_version_tag.py --tag-only | sed 's/^v//') - - if [ -z "$VERSION" ]; then - echo "::error::Could not determine version from git tags" - exit 1 - fi - - # Get short commit hash of current HEAD - COMMIT_HASH=$(git rev-parse --short HEAD) - - # Compose pretend version for setuptools_scm: e.g., 0.5.8.dev20260129+g1a2b3c4 - PRETEND_VERSION="${VERSION}.dev${{ env.DATE }}+g${COMMIT_HASH}" - - echo "version=${VERSION}" >> $GITHUB_OUTPUT - echo "pretend_version=${PRETEND_VERSION}" >> $GITHUB_OUTPUT - echo "Detected version: ${VERSION}" - echo "Pretend version for pip: ${PRETEND_VERSION}" - - - name: Login to Docker Hub (AMD) - uses: docker/login-action@v2 - with: - username: ${{ secrets.DOCKERHUB_AMD_USERNAME }} - password: ${{ secrets.DOCKERHUB_AMD_TOKEN }} - - - name: Build and Push to rocm/sgl-dev - run: | - version=${{ steps.version.outputs.version }} - pretend_version=${{ steps.version.outputs.pretend_version }} - echo "Version: ${version}" - echo "Pretend version: ${pretend_version}" - - if [ "${{ matrix.gpu_arch }}" = "gfx942" ]; then - rocm_tag="rocm700-mi30x" - elif [ "${{ matrix.gpu_arch }}" = "gfx950" ]; then - rocm_tag="rocm700-mi35x" - else - echo "Unsupported gfx arch" - exit 1 - fi - - tag=v${version}-${rocm_tag} - echo "IMAGE_TAG=${tag}-${{ env.DATE }}" >> $GITHUB_ENV - - # remove --build-arg NIC_BACKEND=ainic for auto detection nic support in mori - # UBUNTU_MIRROR forces apt over HTTPS to dodge port-80 reachability flakes - # to Canonical's archive.ubuntu.com mirror IPs from the amd-docker-scale runner. - docker build . -f docker/rocm.Dockerfile --build-arg SGL_BRANCH=${{ github.ref_name }} --build-arg BUILD_TYPE=${{ matrix.build_type }} --build-arg GPU_ARCH=${{ matrix.gpu_arch }} --build-arg ENABLE_MORI=1 --build-arg ENABLE_NIXL=1 --build-arg SETUPTOOLS_SCM_PRETEND_VERSION=${pretend_version} --build-arg UBUNTU_MIRROR=https://archive.ubuntu.com -t rocm/sgl-dev:${tag}-${{ env.DATE }} --no-cache - docker push rocm/sgl-dev:${tag}-${{ env.DATE }} - - # Persist the tag right after rocm/sgl-dev push succeeds so the local - # registry mirror can run even if a later step in this job (lmsys push) - # fails. By default this step only runs when the previous step succeeded, - # so the artifact only exists when an image actually landed on Docker Hub. - - name: Save published image tag - run: | - mkdir -p image-tag - echo "${{ env.IMAGE_TAG }}" > "image-tag/${{ matrix.gpu_arch }}.txt" - - - name: Upload image tag artifact - uses: actions/upload-artifact@v4 - with: - name: image-tag-${{ matrix.gpu_arch }} - path: image-tag/${{ matrix.gpu_arch }}.txt - retention-days: 1 - - - name: Login to Docker Hub (lmsys) - uses: docker/login-action@v2 - with: - username: ${{ secrets.DOCKERHUB_USERNAME }} - password: ${{ secrets.DOCKERHUB_TOKEN }} - - - name: Push to lmsysorg/sglang-rocm - run: | - docker tag rocm/sgl-dev:${{ env.IMAGE_TAG }} lmsysorg/sglang-rocm:${{ env.IMAGE_TAG }} - docker push lmsysorg/sglang-rocm:${{ env.IMAGE_TAG }} - - # Mirror the freshly published rocm/sgl-dev image to the in-network Docker - # registry so AMD CI runners can pull without hitting Docker Hub rate limits. - # The tag is read verbatim from the publish job's artifact so this job uses - # exactly the same tag that publish pushed (only the registry prefix differs). - # `!cancelled()` lets us still mirror successful matrix legs when other legs - # of publish failed; legs without an artifact will fail at download and be - # the only ones marked red. - push_local_registry: - if: ${{ false }} - runs-on: linux-mi300-1gpu-sglang - environment: 'prod' - needs: publish - strategy: - fail-fast: false - matrix: - gpu_arch: ['gfx942', 'gfx950'] - steps: - - name: Download image tag artifact - uses: actions/download-artifact@v4 - with: - name: image-tag-${{ matrix.gpu_arch }} - - - name: Read image tag - run: | - image_tag=$(tr -d '[:space:]' < "${{ matrix.gpu_arch }}.txt") - if [ -z "${image_tag}" ]; then - echo "::error::Image tag artifact is empty" - exit 1 - fi - echo "IMAGE_TAG=${image_tag}" >> $GITHUB_ENV - echo "Resolved IMAGE_TAG=${image_tag}" - - - name: Login to Docker Hub (AMD) - uses: docker/login-action@v2 - with: - username: ${{ secrets.DOCKERHUB_AMD_USERNAME }} - password: ${{ secrets.DOCKERHUB_AMD_TOKEN }} - - - name: Mirror rocm/sgl-dev to local registry - run: | - src="rocm/sgl-dev:${{ env.IMAGE_TAG }}" - dst="10.44.14.109:5000/rocm/sgl-dev:${{ env.IMAGE_TAG }}" - docker pull "${src}" - docker tag "${src}" "${dst}" - docker push "${dst}" - - # Temporarily disable docker cache seeding until performant storage is in place - # cache: - # if: false - # # if: always() && github.repository == 'sgl-project/sglang' - # runs-on: linux-mi300-gpu-1 - # environment: 'prod' - # needs: publish - # strategy: - # fail-fast: false - # matrix: - # gpu_arch: ['gfx942'] - # build_type: ['all'] - # steps: - # - name: Checkout repository - # uses: actions/checkout@v4 - # with: - # fetch-depth: 0 # Required for git describe to find tags - - # - name: "Set Date" - # run: | - # echo "DATE=$(date +%Y%m%d)" >> $GITHUB_ENV - - # - name: Get version from latest tag - # id: version - # run: | - # # Use the shared helper so stable/post releases sort above rc tags. - # VERSION=$(python3 scripts/release/get_version_tag.py --tag-only | sed 's/^v//') - - # if [ -z "$VERSION" ]; then - # echo "::error::Could not determine version from git tags" - # exit 1 - # fi - - # echo "version=${VERSION}" >> $GITHUB_OUTPUT - # echo "Detected version: ${VERSION}" - - # - name: Login to Docker Hub - # uses: docker/login-action@v2 - # with: - # username: ${{ secrets.DOCKERHUB_AMD_USERNAME }} - # password: ${{ secrets.DOCKERHUB_AMD_TOKEN }} - - # - name: Pull and Save Docker Image to Cache - # run: | - # set -euxo pipefail - - # version=${{ steps.version.outputs.version }} - # echo "Version: ${version}" - - # if [ "${{ matrix.gpu_arch }}" = "gfx942" ]; then - # rocm_tag="rocm700-mi30x" - # else - # echo "Unsupported gfx arch" - # exit 1 - # fi - - # tag=v${version}-${rocm_tag} - - # if [ "${{ matrix.build_type }}" = "all" ]; then - # tag_suffix="" - # else - # echo "Unsupported build type" - # exit 1 - # fi - - # image="rocm/sgl-dev:${tag}-${{ env.DATE }}${tag_suffix}" - - # # Determine target cache file name based on ROCm variant - # if [[ "${rocm_tag}" == rocm700* ]]; then - # final_path="/home/runner/sgl-data/docker/image-700.tar" - # else - # echo "Unexpected ROCm tag: ${rocm_tag}" - # exit 1 - # fi - - # tmp_path="${final_path}.tmp" - - # echo "Pulling image: ${image}" - # docker pull "${image}" - - # echo "Saving to temp file: ${tmp_path}" - # docker save "${image}" -o "${tmp_path}" - - # echo "Moving to final path: ${final_path}" - # mv -f "${tmp_path}" "${final_path}" - - # echo "Cache populated successfully at ${final_path}" diff --git a/.github/workflows/release-docker-amd-rocm10.yml b/.github/workflows/release-docker-amd-rocm10.yml index 4d537320a60c..663c7049320f 100644 --- a/.github/workflows/release-docker-amd-rocm10.yml +++ b/.github/workflows/release-docker-amd-rocm10.yml @@ -118,21 +118,17 @@ jobs: docker tag "rocm/sgl-dev:${IMAGE_TAG}" "lmsysorg/sglang-rocm:${IMAGE_TAG}" docker push "lmsysorg/sglang-rocm:${IMAGE_TAG}" - # gfx1250 (MI45x) is still being brought up on the amd_helios branch: main - # has no gfx1250 stage in docker/rocm.Dockerfile, so this job builds from - # that branch rather than from the ref the workflow was triggered on. It runs - # on the same schedule as the pair above but stays a separate job, because - # there is no MI450 runner pool consuming the image and a bring-up build + # gfx1250 (MI45x) runs on the same schedule as the pair above but stays a + # separate job: no MI450 runner pool consumes the image, so a bring-up build # failure must not gate the released gfx942/gfx950 images. publish-gfx1250: if: github.repository == 'sgl-project/sglang' && (github.event_name != 'workflow_dispatch' || inputs.job_select == 'all' || inputs.job_select == 'publish-gfx1250') runs-on: amd-docker-scale environment: 'prod' steps: - - name: Checkout amd_helios + - name: Checkout repository uses: actions/checkout@v4 with: - ref: amd_helios fetch-depth: 0 - name: Set up Python @@ -143,7 +139,7 @@ jobs: - name: Set date run: echo "DATE=$(date +%Y%m%d)" >> "$GITHUB_ENV" - - name: Get version and branch head + - name: Get version from latest tag id: version run: | VERSION=$(python3 scripts/release/get_version_tag.py --tag-only | sed 's/^v//') @@ -153,16 +149,11 @@ jobs: exit 1 fi - # The checkout above is amd_helios, so this is the branch head that - # both the build context and the cloned sglang source come from. - BRANCH_SHA=$(git rev-parse HEAD) COMMIT_HASH=$(git rev-parse --short HEAD) PRETEND_VERSION="${VERSION}.dev${DATE}+g${COMMIT_HASH}" echo "version=${VERSION}" >> "$GITHUB_OUTPUT" - echo "branch_sha=${BRANCH_SHA}" >> "$GITHUB_OUTPUT" echo "pretend_version=${PRETEND_VERSION}" >> "$GITHUB_OUTPUT" - echo "Building gfx1250 from amd_helios at ${BRANCH_SHA}" - name: Login to Docker Hub (AMD) uses: docker/login-action@v3 @@ -172,18 +163,15 @@ jobs: - name: Build and push to rocm/sgl-dev env: - BRANCH_SHA: ${{ steps.version.outputs.branch_sha }} PRETEND_VERSION: ${{ steps.version.outputs.pretend_version }} VERSION: ${{ steps.version.outputs.version }} run: | - # `dev` marks this as built from a bring-up branch rather than from - # the ref the release ran on, so it never reads as a released tag. - image_tag="v${VERSION}-rocm10-mi45x-dev-${DATE}" - echo "Publishing rocm/sgl-dev:${image_tag}" + image_tag="v${VERSION}-rocm10-mi45x-${DATE}" + echo "IMAGE_TAG=${image_tag}" >> "$GITHUB_ENV" docker build . \ -f docker/rocm.Dockerfile \ - --build-arg "SGL_BRANCH=${BRANCH_SHA}" \ + --build-arg "SGL_BRANCH=${GITHUB_SHA}" \ --build-arg BUILD_TYPE=all \ --build-arg GPU_ARCH=gfx1250-rocm1000 \ --build-arg ENABLE_MORI=1 \ @@ -193,3 +181,14 @@ jobs: -t "rocm/sgl-dev:${image_tag}" \ --no-cache docker push "rocm/sgl-dev:${image_tag}" + + - name: Login to Docker Hub (lmsys) + uses: docker/login-action@v3 + with: + username: ${{ secrets.DOCKERHUB_USERNAME }} + password: ${{ secrets.DOCKERHUB_TOKEN }} + + - name: Push to lmsysorg/sglang-rocm + run: | + docker tag "rocm/sgl-dev:${IMAGE_TAG}" "lmsysorg/sglang-rocm:${IMAGE_TAG}" + docker push "lmsysorg/sglang-rocm:${IMAGE_TAG}" diff --git a/.github/workflows/release-docker-amd-rocm7_15-nightly.yml b/.github/workflows/release-docker-amd-rocm7_15-nightly.yml deleted file mode 100644 index 7ba67d4017e7..000000000000 --- a/.github/workflows/release-docker-amd-rocm7_15-nightly.yml +++ /dev/null @@ -1,109 +0,0 @@ -name: Release Docker Images Nightly ROCm7.15 (AMD) -on: - workflow_dispatch: - inputs: - job_select: - description: 'Select which release job to run' - required: false - type: choice - default: 'all' - options: - - 'all' - - publish - # schedule: - # - cron: '0 12 * * *' - -concurrency: - # A PR number if a pull request and otherwise the commit hash. This cancels - # queued and in-progress runs for the same PR (presubmit) or commit - # (postsubmit). The workflow name is prepended to avoid conflicts between - # different workflows. - group: ${{ github.workflow }}-${{ github.event.number || github.sha }} - cancel-in-progress: false - -jobs: - publish: - if: github.repository == 'sgl-project/sglang' && (github.event_name != 'workflow_dispatch' || inputs.job_select == 'all' || inputs.job_select == 'publish') - runs-on: linux-mi300-1gpu-sglang - environment: 'prod' - strategy: - fail-fast: false - matrix: - gpu_arch: ['gfx1250-rocm7_15'] - build_type: ['all'] - steps: - - name: Checkout repository - uses: actions/checkout@v4 - with: - fetch-depth: 0 # Required for git describe to find tags - - - name: Set up Python - uses: actions/setup-python@v5 - with: - python-version: "3.10" - - - name: "Set Date" - run: | - echo "DATE=$(date +%Y%m%d)" >> $GITHUB_ENV - - - name: Get version from latest tag - id: version - run: | - # Use the shared helper so stable/post releases sort above rc tags. - VERSION=$(python3 scripts/release/get_version_tag.py --tag-only | sed 's/^v//') - - if [ -z "$VERSION" ]; then - echo "::error::Could not determine version from git tags" - exit 1 - fi - - # Get short commit hash of current HEAD - COMMIT_HASH=$(git rev-parse --short HEAD) - - # Compose pretend version for setuptools_scm: e.g., 0.5.8.post1.dev20260211+g1a2b3c4 - PRETEND_VERSION="${VERSION}.dev${{ env.DATE }}+g${COMMIT_HASH}" - - echo "version=${VERSION}" >> $GITHUB_OUTPUT - echo "pretend_version=${PRETEND_VERSION}" >> $GITHUB_OUTPUT - echo "Detected version: ${VERSION}" - echo "Pretend version for pip: ${PRETEND_VERSION}" - - # - name: Login to Docker Hub (AMD) - # uses: docker/login-action@v2 - # with: - # username: ${{ secrets.DOCKERHUB_AMD_USERNAME }} - # password: ${{ secrets.DOCKERHUB_AMD_TOKEN }} - - - name: Build and Push to rocm/sgl-dev - run: | - version=${{ steps.version.outputs.version }} - pretend_version=${{ steps.version.outputs.pretend_version }} - echo "Version: ${version}" - echo "Pretend version: ${pretend_version}" - - if [ "${{ matrix.gpu_arch }}" = "gfx1250-rocm7_15" ]; then - rocm_tag="rocm7.15-mi45x" - else - echo "Unsupported gfx arch" - exit 1 - fi - - tag=v${version}-${rocm_tag} - echo "IMAGE_TAG=${tag}-${{ env.DATE }}" >> $GITHUB_ENV - - # remove --build-arg NIC_BACKEND=ainic for auto detection nic support in mori - # UBUNTU_MIRROR forces apt over HTTPS to dodge port-80 reachability flakes - # to Canonical's archive.ubuntu.com mirror IPs from the build runner. - docker build . -f docker/rocm.Dockerfile --build-arg SGL_BRANCH=${{ github.sha }} --build-arg BUILD_TYPE=${{ matrix.build_type }} --build-arg GPU_ARCH=${{ matrix.gpu_arch }} --build-arg GPU_ARCH_LIST_ARG=gfx1250 --build-arg ENABLE_MORI=1 --build-arg SGLANG_VERSION=${pretend_version} --build-arg UBUNTU_MIRROR=https://archive.ubuntu.com -t rocm/sgl-dev:${tag}-${{ env.DATE }} --no-cache - # docker push rocm/sgl-dev:${tag}-${{ env.DATE }} - - - name: Login to Docker Hub (lmsys) - uses: docker/login-action@v2 - with: - username: ${{ secrets.DOCKERHUB_USERNAME }} - password: ${{ secrets.DOCKERHUB_TOKEN }} - - - name: Push to lmsysorg/sglang-rocm - run: | - docker tag rocm/sgl-dev:${{ env.IMAGE_TAG }} lmsysorg/sglang-rocm:${{ env.IMAGE_TAG }} - docker push lmsysorg/sglang-rocm:${{ env.IMAGE_TAG }} diff --git a/.github/workflows/release-docker-amd.yml b/.github/workflows/release-docker-amd.yml index e5cd66ca35b1..7973232a01f0 100644 --- a/.github/workflows/release-docker-amd.yml +++ b/.github/workflows/release-docker-amd.yml @@ -15,9 +15,9 @@ on: default: 'all' options: - 'all' + - rocm10 - rocm724 - rocm720 - - rocm700 jobs: publish: @@ -29,7 +29,7 @@ jobs: # others mid-push and leave the release tag with a partial image set. fail-fast: false matrix: - rocm_version: ${{ fromJson((github.event_name == 'workflow_dispatch' && inputs.rocm_version != 'all' && inputs.rocm_version != '') && format('["{0}"]', inputs.rocm_version) || '["rocm700", "rocm720", "rocm724"]') }} + rocm_version: ${{ fromJson((github.event_name == 'workflow_dispatch' && inputs.rocm_version != 'all' && inputs.rocm_version != '') && format('["{0}"]', inputs.rocm_version) || '["rocm10", "rocm720", "rocm724"]') }} gpu_arch: ['gfx942', 'gfx950'] build_type: ['all'] steps: @@ -70,21 +70,25 @@ jobs: echo "Version: ${version}" gpu_arch_suffix="" - if [ "${{ matrix.rocm_version }}" = "rocm700" ]; then + if [ "${{ matrix.rocm_version }}" = "rocm720" ] || [ "${{ matrix.rocm_version }}" = "rocm724" ]; then + gpu_arch_suffix="-${{ matrix.rocm_version }}" if [ "${{ matrix.gpu_arch }}" = "gfx942" ]; then - rocm_tag="rocm700-mi30x" + rocm_tag="${{ matrix.rocm_version }}-mi30x" elif [ "${{ matrix.gpu_arch }}" = "gfx950" ]; then - rocm_tag="rocm700-mi35x" + rocm_tag="${{ matrix.rocm_version }}-mi35x" else echo "Unsupported gfx arch" exit 1 fi - elif [ "${{ matrix.rocm_version }}" = "rocm720" ] || [ "${{ matrix.rocm_version }}" = "rocm724" ]; then - gpu_arch_suffix="-${{ matrix.rocm_version }}" + elif [ "${{ matrix.rocm_version }}" = "rocm10" ]; then + # The Dockerfile stages carry the full SDK version (gfx942-rocm1000) + # while the published tag keeps the rocm10 flavor name that CI and + # the nightly images already use. + gpu_arch_suffix="-rocm1000" if [ "${{ matrix.gpu_arch }}" = "gfx942" ]; then - rocm_tag="${{ matrix.rocm_version }}-mi30x" + rocm_tag="rocm10-mi30x" elif [ "${{ matrix.gpu_arch }}" = "gfx950" ]; then - rocm_tag="${{ matrix.rocm_version }}-mi35x" + rocm_tag="rocm10-mi35x" else echo "Unsupported gfx arch" exit 1 diff --git a/.github/workflows/release-docker-dev.yml b/.github/workflows/release-docker-dev.yml index d212f0ceffae..bff64e9620bf 100644 --- a/.github/workflows/release-docker-dev.yml +++ b/.github/workflows/release-docker-dev.yml @@ -25,7 +25,7 @@ on: required: false default: "" overlay_cudas: - description: "Comma-separated cuda variants to overlay onto (cu12, cu13). Default 'cu13'." + description: "Comma-separated cuda variants to overlay onto (cu13). Default 'cu13'." required: false default: "cu13" overlay_tag_suffix: @@ -86,9 +86,9 @@ jobs: # consumers pinned to the pre-flip names. if [ -z "${SUFFIX}" ]; then # Nightly: include dated tags - TAG_CONFIG='[{"cuda":"cu129","tags":["dev-cu12","nightly-dev-cu12-{date}-{short_sha}"]},{"cuda":"cu130","tags":["dev","dev-cu13","nightly-dev-{date}-{short_sha}","nightly-dev-cu13-{date}-{short_sha}"]}]' + TAG_CONFIG='[{"cuda":"cu130","tags":["dev","dev-cu13","nightly-dev-{date}-{short_sha}","nightly-dev-cu13-{date}-{short_sha}"]}]' else - TAG_CONFIG="[{\"cuda\":\"cu129\",\"tags\":[\"dev-cu12${SUFFIX}\"]},{\"cuda\":\"cu130\",\"tags\":[\"dev${SUFFIX}\",\"dev-cu13${SUFFIX}\"]}]" + TAG_CONFIG="[{\"cuda\":\"cu130\",\"tags\":[\"dev${SUFFIX}\",\"dev-cu13${SUFFIX}\"]}]" fi echo "tag_config=${TAG_CONFIG}" >> $GITHUB_OUTPUT @@ -97,7 +97,7 @@ jobs: # SUFFIX was computed above from inputs.tag / inputs.pr_number. if [ -n "${OVERLAY_DOCKERFILE_INPUT}" ]; then # Refuse to overlay without a tag/pr_number: SUFFIX would be empty, - # so base would point at the moving 'dev'/'dev-cu12'/'dev-cu13' + # so base would point at the moving 'dev'/'dev-cu13' # latest tags and overwrite mode would clobber them. if [ -z "${SUFFIX}" ]; then echo "overlay_dockerfile is set but neither 'tag' nor 'pr_number' was given." >&2 @@ -112,10 +112,8 @@ jobs: C="$(echo "${C}" | xargs)" if [ "${C}" = "cu13" ]; then BASES='["dev'"${SUFFIX}"'","dev-cu13'"${SUFFIX}"'"]' - elif [ "${C}" = "cu12" ]; then - BASES='["dev-cu12'"${SUFFIX}"'"]' else - echo "Unknown overlay cuda variant: ${C} (expected cu12 or cu13)" >&2 + echo "Unknown overlay cuda variant: ${C} (expected cu13)" >&2 exit 1 fi if [ -n "${OVERLAY_SUFFIX_INPUT}" ]; then @@ -213,7 +211,7 @@ jobs: if: ${{ !inputs.build_only && !inputs.tag && !inputs.pr_number }} uses: ./.github/workflows/_docker-cleanup-nightly.yml with: - tag_prefixes: '["nightly-dev", "nightly-dev-cu12", "nightly-dev-cu13"]' + tag_prefixes: '["nightly-dev", "nightly-dev-cu13"]' image_repo: ${{ inputs.image_repo || 'lmsysorg/sglang' }} secrets: inherit diff --git a/.github/workflows/release-docker-intel-xpu-nightly.yml b/.github/workflows/release-docker-intel-xpu-nightly.yml index 73cb4baac2a8..9e9fc1963ede 100644 --- a/.github/workflows/release-docker-intel-xpu-nightly.yml +++ b/.github/workflows/release-docker-intel-xpu-nightly.yml @@ -1,11 +1,16 @@ name: Release Docker Images Nightly (Intel XPU) on: workflow_dispatch: + inputs: + branch: + description: 'Branch or tag to build from, e.g. main, release/v0.5.20, v0.5.20.' + required: false + default: 'main' schedule: - cron: '0 12 * * *' concurrency: - group: ${{ github.workflow }}-${{ github.event.number || github.sha }} + group: ${{ github.workflow }}-${{ inputs.branch || 'main' }} cancel-in-progress: true jobs: @@ -17,11 +22,15 @@ jobs: - name: Checkout repository uses: actions/checkout@v4 with: + # Scheduled runs always build main; manual runs may pick a release branch or tag. + ref: ${{ inputs.branch || 'main' }} path: sglang-nightly-build fetch-depth: 0 - name: Set date and commit hash id: image_meta + env: + SG_LANG_BRANCH: ${{ inputs.branch || 'main' }} run: | cd sglang-nightly-build date_tag=$(date +%Y%m%d) @@ -30,10 +39,18 @@ jobs: echo "::error::Could not resolve commit hash" exit 1 fi - image_tag="nightly-dev-xpu-bmg-${date_tag}-${commit_hash}" + if [ "${SG_LANG_BRANCH}" = "main" ]; then + image_tag="nightly-dev-xpu-bmg-${date_tag}-${commit_hash}" + else + # Slugify the ref so it is usable in a Docker tag (e.g. release/v0.5.20 -> release-v0.5.20). + branch_slug=$(echo "${SG_LANG_BRANCH}" | sed 's#[^a-zA-Z0-9._-]#-#g') + image_tag="nightly-dev-xpu-bmg-${branch_slug}-${date_tag}-${commit_hash}" + fi echo "DATE=${date_tag}" >> $GITHUB_ENV echo "commit_hash=${commit_hash}" >> $GITHUB_OUTPUT + echo "SG_LANG_BRANCH=${SG_LANG_BRANCH}" >> $GITHUB_ENV echo "IMAGE_TAG=${image_tag}" >> $GITHUB_ENV + echo "Building ref ${SG_LANG_BRANCH} at ${commit_hash}" echo "Image tag: ${image_tag}" - name: Login to Docker Hub (Intel) @@ -44,14 +61,19 @@ jobs: - name: Build intel/sglang-dev run: | - echo "Building intel/sglang-dev:${{ env.IMAGE_TAG }}" + echo "Building intel/sglang-dev:${IMAGE_TAG} from ${SG_LANG_BRANCH}" + # Only main advances the rolling `latest` tag. + latest_tag=() + if [ "${SG_LANG_BRANCH}" = "main" ]; then + latest_tag=(-t "intel/sglang-dev:latest") + fi docker build sglang-nightly-build \ -f sglang-nightly-build/docker/xpu.Dockerfile \ --build-arg SG_LANG_REPO=https://github.com/sgl-project/sglang.git \ - --build-arg SG_LANG_BRANCH=main \ + --build-arg SG_LANG_BRANCH="${SG_LANG_BRANCH}" \ --no-cache --progress=plain \ - -t "intel/sglang-dev:${{ env.IMAGE_TAG }}" \ - -t "intel/sglang-dev:latest" + -t "intel/sglang-dev:${IMAGE_TAG}" \ + "${latest_tag[@]}" - name: Push intel/sglang-dev run: | @@ -63,5 +85,7 @@ jobs: done return 1 } - push_with_retry "intel/sglang-dev:${{ env.IMAGE_TAG }}" - push_with_retry "intel/sglang-dev:latest" + push_with_retry "intel/sglang-dev:${IMAGE_TAG}" + if [ "${SG_LANG_BRANCH}" = "main" ]; then + push_with_retry "intel/sglang-dev:latest" + fi diff --git a/.github/workflows/release-docker-intel-xpu.yml b/.github/workflows/release-docker-intel-xpu.yml new file mode 100644 index 000000000000..611962699a16 --- /dev/null +++ b/.github/workflows/release-docker-intel-xpu.yml @@ -0,0 +1,76 @@ +name: Release Docker Images (Intel XPU) +on: + push: + tags: + - 'v[0-9]+.*' + workflow_dispatch: + inputs: + version: + description: 'Version to build (without v prefix, e.g., 0.5.7)' + required: true + +jobs: + publish: + if: github.repository == 'sgl-project/sglang' + runs-on: intel-bmg + environment: 'prod' + steps: + - name: Checkout repository + uses: actions/checkout@v4 + + - name: Login to Docker Hub + uses: docker/login-action@v2 + with: + username: ${{ secrets.DOCKERHUB_USERNAME }} + password: ${{ secrets.DOCKERHUB_TOKEN }} + + - name: Get version from tag + id: version + run: | + if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then + VERSION="${{ github.event.inputs.version }}" + IS_WORKFLOW_DISPATCH=true + else + # Extract version from the tag that triggered this workflow (e.g., v0.5.7 -> 0.5.7) + VERSION="${GITHUB_REF_NAME#v}" + IS_WORKFLOW_DISPATCH=false + fi + + # Validate version format + if [ -z "$VERSION" ]; then + echo "::error::Version is empty" + exit 1 + fi + if ! echo "$VERSION" | grep -qE '^[0-9]+\.[0-9]+\.[0-9]+'; then + echo "::error::Invalid version format: $VERSION (expected: X.Y.Z)" + exit 1 + fi + + if [ "$IS_WORKFLOW_DISPATCH" = "true" ] && ! git ls-remote --exit-code --tags origin "refs/tags/v${VERSION}" >/dev/null 2>&1; then + echo "::error::Version tag v${VERSION} does not exist" + exit 1 + fi + + echo "version=${VERSION}" >> $GITHUB_OUTPUT + + - name: Build and Push + run: | + version=${{ steps.version.outputs.version }} + tag=v${version}-xpu + + docker build . -f docker/xpu.Dockerfile \ + --build-arg SG_LANG_REPO=https://github.com/sgl-project/sglang.git \ + --build-arg SG_LANG_BRANCH=v${version} \ + -t lmsysorg/sglang:${tag} \ + --no-cache --progress=plain + + push_with_retry() { + for i in 1 2 3 4 5; do + docker push "$1" && return 0 + echo "push failed (attempt $i), retrying in $((i*15))s..." + sleep $((i*15)) + done + return 1 + } + + push_with_retry lmsysorg/sglang:${tag} diff --git a/.github/workflows/release-docker-npu-nightly.yml b/.github/workflows/release-docker-npu-nightly.yml index 5a0dfe6ab009..4dc129f19782 100644 --- a/.github/workflows/release-docker-npu-nightly.yml +++ b/.github/workflows/release-docker-npu-nightly.yml @@ -1,4 +1,5 @@ name: Release Docker Images Nightly (NPU) + on: pull_request: branches: @@ -8,25 +9,41 @@ on: - 'docker/npu.Dockerfile' workflow_dispatch: schedule: - - cron: "0 14 * * *" # Execute at 10:00 p.m. Beijing Time every day - + - cron: "0 14 * * *" concurrency: group: ${{ github.workflow }}-${{ github.sha }} cancel-in-progress: true jobs: build: - runs-on: ubuntu-22.04-arm + name: build ${{ matrix.platform }}-${{ matrix.device_type }} + runs-on: ${{ matrix.runner }} strategy: + fail-fast: false matrix: - cann_version: ["9.0.0"] - device_type: ["910b", "a3"] + include: + - device_type: a3 + cann_version: "9.1.0" + platform: amd64 + runner: ubuntu-latest + - device_type: a3 + cann_version: "9.1.0" + platform: arm64 + runner: ubuntu-22.04-arm + - device_type: 950 + cann_version: "9.1.0" + platform: amd64 + runner: ubuntu-latest + - device_type: 950 + cann_version: "9.1.0" + platform: arm64 + runner: ubuntu-22.04-arm steps: - name: Checkout repository uses: actions/checkout@v4 - name: Free up disk space - uses: jlumbroso/free-disk-space@54081f138730dfa15788a46383842cd2f914a1be # v1.3.1 + uses: jlumbroso/free-disk-space@54081f138730dfa15788a46383842cd2f914a1be with: tool-cache: true docker-images: false @@ -38,48 +55,64 @@ jobs: id: meta uses: docker/metadata-action@v5 with: - images: | - lmsysorg/sglang - # push with schedule event - # push with workflow_dispatch event + images: lmsysorg/sglang tags: | type=ref,event=pr type=ref,event=branch type=schedule,pattern=main flavor: | latest=false - suffix=-cann${{ matrix.cann_version }}-${{ matrix.device_type }},onlatest=true - # Login against a Docker registry except on PR - # https://github.com/docker/login-action - - name: Log into docker hub + suffix=-cann${{ matrix.cann_version }}-${{ matrix.device_type }}-${{ matrix.platform }} + + - name: Log into Docker Hub uses: docker/login-action@v3 if: ${{ github.repository == 'sgl-project/sglang' && github.event_name != 'pull_request' }} with: username: ${{ secrets.DOCKERHUB_USERNAME }} password: ${{ secrets.DOCKERHUB_TOKEN }} - # Enable Docker multi-architecture build environment - # Emulate non-native architectures - - name: Set up QEMU - uses: docker/setup-qemu-action@v3 - # Required for building and pushing multi-arch Docker images - - name: Set up Docker Buildx - uses: docker/setup-buildx-action@v3 - - # Build and push Docker image with Buildx (don't push on PR) - # https://github.com/docker/build-push-action - name: Build and push Docker image - id: build-and-push uses: docker/build-push-action@v6 with: context: docker file: docker/npu.Dockerfile - platforms: linux/arm64,linux/amd64 + platforms: linux/${{ matrix.platform }} labels: ${{ steps.meta.outputs.labels }} tags: ${{ steps.meta.outputs.tags }} - push: ${{ github.repository == 'sgl-project/sglang' && github.event_name != 'pull_request' }} + outputs: ${{ github.repository == 'sgl-project/sglang' && github.event_name != 'pull_request' && 'type=registry,oci-mediatypes=false' || '' }} provenance: false + sbom: false build-args: | - SGLANG_KERNEL_NPU_TAG=20260826 + SGLANG_KERNEL_NPU_TAG=2026.9.0.post5 CANN_VERSION=${{ matrix.cann_version }} DEVICE_TYPE=${{ matrix.device_type }} + + merge: + name: merge multi-arch manifest + needs: build + if: ${{ github.repository == 'sgl-project/sglang' && github.event_name != 'pull_request' }} + runs-on: ubuntu-latest + strategy: + matrix: + cann_version: ["9.1.0"] + device_type: ["a3", "950"] + steps: + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v3 + + - name: Log into Docker Hub + uses: docker/login-action@v3 + with: + username: ${{ secrets.DOCKERHUB_USERNAME }} + password: ${{ secrets.DOCKERHUB_TOKEN }} + + - name: Compute base image + run: | + echo "IMAGE=lmsysorg/sglang:${GITHUB_REF_NAME}-cann${{ matrix.cann_version }}-${{ matrix.device_type }}" >> "$GITHUB_ENV" + + - name: Create multi-arch manifest + run: | + docker buildx imagetools create \ + -t "${IMAGE}" \ + "${IMAGE}-amd64" \ + "${IMAGE}-arm64" diff --git a/.github/workflows/release-docker-npu.yml b/.github/workflows/release-docker-npu.yml index f0b44409e4c7..eb8a2fbb64a6 100644 --- a/.github/workflows/release-docker-npu.yml +++ b/.github/workflows/release-docker-npu.yml @@ -14,8 +14,8 @@ jobs: runs-on: ubuntu-22.04-arm strategy: matrix: - cann_version: ["9.0.0"] - device_type: ["910b", "a3"] + cann_version: ["9.1.0"] + device_type: ["a3", "950"] steps: - name: Checkout repository uses: actions/checkout@v4 @@ -87,7 +87,7 @@ jobs: push: ${{ github.repository == 'sgl-project/sglang' && github.event_name != 'pull_request' }} provenance: false build-args: | - SGLANG_KERNEL_NPU_TAG=20260826 + SGLANG_KERNEL_NPU_TAG=2026.9.0.post5 CANN_VERSION=${{ matrix.cann_version }} DEVICE_TYPE=${{ matrix.device_type }} SGLANG_TAG=${{ steps.version.outputs.version }} diff --git a/.github/workflows/release-docker-renderer.yml b/.github/workflows/release-docker-renderer.yml new file mode 100644 index 000000000000..391e03b2df69 --- /dev/null +++ b/.github/workflows/release-docker-renderer.yml @@ -0,0 +1,137 @@ +name: Publish Renderer Docker Image + +on: + push: + branches: [main] + tags: ["v[0-9]+.*"] + paths: + - "rust/sglang-renderer/**" + - "rust/Cargo.toml" + - "rust/Cargo.lock" + - "rust/rust-toolchain.toml" + - "docker/renderer.Dockerfile" + - "LICENSE" + - ".github/workflows/release-docker-renderer.yml" + +permissions: + contents: read + +jobs: + prepare: + if: github.repository == 'sgl-project/sglang' + runs-on: ubuntu-latest + outputs: + publish: ${{ steps.image.outputs.publish }} + repo: ${{ steps.image.outputs.repo }} + tag: ${{ steps.image.outputs.tag }} + steps: + - name: Select image destination + id: image + run: | + if [[ "$GITHUB_REF_TYPE" == "tag" ]]; then + if [[ ! "$GITHUB_REF_NAME" =~ ^v[0-9]+\.[0-9]+\.[0-9]+(\.post[0-9]+)?$ ]]; then + echo "Skipping non-final release tag $GITHUB_REF_NAME" + echo "publish=false" >> "$GITHUB_OUTPUT" + exit 0 + fi + echo "repo=lmsysorg/sglang" >> "$GITHUB_OUTPUT" + echo "tag=${GITHUB_REF_NAME}-renderer" >> "$GITHUB_OUTPUT" + else + echo "repo=lmsysorg/sglang-staging" >> "$GITHUB_OUTPUT" + echo "tag=renderer-${GITHUB_SHA:0:8}" >> "$GITHUB_OUTPUT" + fi + echo "publish=true" >> "$GITHUB_OUTPUT" + + build: + needs: prepare + if: needs.prepare.outputs.publish == 'true' + environment: ${{ github.ref_type == 'tag' && 'prod' || null }} + strategy: + fail-fast: false + matrix: + include: + - platform: linux/amd64 + runner: ubuntu-24.04 + arch: amd64 + - platform: linux/arm64 + runner: ubuntu-24.04-arm + arch: arm64 + runs-on: ${{ matrix.runner }} + env: + IMAGE_REPO: ${{ needs.prepare.outputs.repo }} + IMAGE_TAG: ${{ needs.prepare.outputs.tag }} + steps: + - name: Checkout repository + uses: actions/checkout@v4 + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v3 + + - name: Run renderer tests + run: | + docker buildx build . -f docker/renderer.Dockerfile \ + --platform ${{ matrix.platform }} --target test --progress=plain + + - name: Login to Docker Hub + uses: docker/login-action@v3 + with: + username: ${{ secrets.DOCKERHUB_USERNAME }} + password: ${{ secrets.DOCKERHUB_TOKEN }} + + - name: Build and push by digest + run: | + BUILD_COMMIT=$(git rev-parse HEAD) + BUILD_URL="${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}" + docker buildx build . -f docker/renderer.Dockerfile \ + --platform ${{ matrix.platform }} --target runtime --progress=plain \ + --build-arg "SGLANG_BUILD_COMMIT=${BUILD_COMMIT}" \ + --build-arg "SGLANG_BUILD_URL=${BUILD_URL}" \ + --build-arg "SGLANG_IMAGE_TAG=${IMAGE_REPO}:${IMAGE_TAG}" \ + --output "type=image,name=${IMAGE_REPO},push-by-digest=true,name-canonical=true,push=true" \ + --metadata-file /tmp/renderer-metadata.json + DIGEST=$(python3 -c "import json; print(json.load(open('/tmp/renderer-metadata.json'))['containerimage.digest'])") + docker run --rm "${IMAGE_REPO}@${DIGEST}" --help + mkdir -p /tmp/renderer-digests + echo "$DIGEST" > "/tmp/renderer-digests/${{ matrix.arch }}" + + - name: Upload digest + uses: actions/upload-artifact@v4 + with: + name: renderer-digest-${{ matrix.arch }} + path: /tmp/renderer-digests/${{ matrix.arch }} + retention-days: 7 + + publish: + needs: [prepare, build] + if: needs.prepare.outputs.publish == 'true' + environment: ${{ github.ref_type == 'tag' && 'prod' || null }} + runs-on: ubuntu-24.04 + env: + IMAGE_REPO: ${{ needs.prepare.outputs.repo }} + IMAGE_TAG: ${{ needs.prepare.outputs.tag }} + steps: + - name: Download digests + uses: actions/download-artifact@v4 + with: + path: /tmp/renderer-digests + pattern: renderer-digest-* + merge-multiple: true + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v3 + + - name: Login to Docker Hub + uses: docker/login-action@v3 + with: + username: ${{ secrets.DOCKERHUB_USERNAME }} + password: ${{ secrets.DOCKERHUB_TOKEN }} + + - name: Publish multi-architecture tag + run: | + AMD64_DIGEST=$(cat /tmp/renderer-digests/amd64) + ARM64_DIGEST=$(cat /tmp/renderer-digests/arm64) + docker buildx imagetools create \ + -t "${IMAGE_REPO}:${IMAGE_TAG}" \ + "${IMAGE_REPO}@${AMD64_DIGEST}" \ + "${IMAGE_REPO}@${ARM64_DIGEST}" + docker buildx imagetools inspect "${IMAGE_REPO}:${IMAGE_TAG}" diff --git a/.github/workflows/release-docker-runtime.yml b/.github/workflows/release-docker-runtime.yml index 0e224bf91e33..eee7b1479a81 100644 --- a/.github/workflows/release-docker-runtime.yml +++ b/.github/workflows/release-docker-runtime.yml @@ -2,7 +2,7 @@ name: Release Docker Runtime Images # # Builds and publishes runtime Docker images (production-optimized, ~50% smaller): # - lmsysorg/sglang:v{version}-runtime, lmsysorg/sglang:latest-runtime -# - lmsysorg/sglang:v{version}-cu129-runtime, lmsysorg/sglang:latest-cu129-runtime +# - lmsysorg/sglang:v{version}-cu130-runtime, lmsysorg/sglang:latest-cu130-runtime # on: push: @@ -49,7 +49,6 @@ jobs: image_repo: ${{ inputs.image_repo || 'lmsysorg/sglang' }} tag_config: | [ - {"cuda": "cu130", "tags": ["v${{ needs.resolve-version.outputs.version }}-runtime", "latest-runtime", "v${{ needs.resolve-version.outputs.version }}-cu130-runtime", "latest-cu130-runtime"]}, - {"cuda": "cu129", "tags": ["v${{ needs.resolve-version.outputs.version }}-cu129-runtime", "latest-cu129-runtime"]} + {"cuda": "cu130", "tags": ["v${{ needs.resolve-version.outputs.version }}-runtime", "latest-runtime", "v${{ needs.resolve-version.outputs.version }}-cu130-runtime", "latest-cu130-runtime"]} ] secrets: inherit diff --git a/.github/workflows/release-docker.yml b/.github/workflows/release-docker.yml index edf21469e134..fcc36a6315df 100644 --- a/.github/workflows/release-docker.yml +++ b/.github/workflows/release-docker.yml @@ -2,7 +2,7 @@ name: Release Docker Images # # Builds and publishes framework Docker images (full development environment): # - lmsysorg/sglang:v{version}, lmsysorg/sglang:latest (cuda 13) -# - lmsysorg/sglang:v{version}-cu129, lmsysorg/sglang:latest-cu129 +# - lmsysorg/sglang:v{version}-cu130, lmsysorg/sglang:latest-cu130 # on: push: @@ -49,7 +49,6 @@ jobs: image_repo: ${{ inputs.image_repo || 'lmsysorg/sglang' }} tag_config: | [ - {"cuda": "cu130", "tags": ["v${{ needs.resolve-version.outputs.version }}", "latest", "v${{ needs.resolve-version.outputs.version }}-cu130", "latest-cu130"]}, - {"cuda": "cu129", "tags": ["v${{ needs.resolve-version.outputs.version }}-cu129", "latest-cu129"]} + {"cuda": "cu130", "tags": ["v${{ needs.resolve-version.outputs.version }}", "latest", "v${{ needs.resolve-version.outputs.version }}-cu130", "latest-cu130"]} ] secrets: inherit diff --git a/.github/workflows/release-pypi-nightly.yml b/.github/workflows/release-pypi-nightly.yml index 86d1f65d0efe..8a069e5de8ee 100644 --- a/.github/workflows/release-pypi-nightly.yml +++ b/.github/workflows/release-pypi-nightly.yml @@ -24,7 +24,8 @@ jobs: if: github.repository == 'sgl-project/sglang' runs-on: ubuntu-latest outputs: - nightly_version: ${{ steps.build.outputs.nightly_version }} + nightly_version: ${{ steps.wheel.outputs.wheel_version }} + wheel_filename: ${{ steps.wheel.outputs.wheel_filename }} commit_hash: ${{ steps.build.outputs.commit_hash }} build_date: ${{ steps.build.outputs.build_date }} steps: @@ -41,7 +42,9 @@ jobs: - name: Install build dependencies run: | - pip install build wheel setuptools setuptools-scm + pip install \ + auditwheel build patchelf wheel "setuptools>=61.0" \ + "setuptools-rust>=1.11" "setuptools-scm>=8.0" "torch==2.13.0" # Needed by setuptools-rust to build the bundled native gRPC extension # (rust/sglang-grpc) when `python -m build` builds the sglang wheel. @@ -66,7 +69,7 @@ jobs: MINOR=$(echo "$VERSION" | cut -d. -f2) PATCH_RAW=$(echo "$VERSION" | cut -d. -f3) # Strip pre-release suffixes (rc0, post1, etc.) to get numeric patch - PATCH=$(echo "$PATCH_RAW" | sed 's/[^0-9].*//') + PATCH=${PATCH_RAW%%[^0-9]*} NEXT_PATCH=$((PATCH + 1)) NEXT_VERSION="${MAJOR}.${MINOR}.${NEXT_PATCH}" @@ -77,24 +80,25 @@ jobs: export SETUPTOOLS_SCM_PRETEND_VERSION="$FORCE_VERSION" # Build wheel - python3 -m build --wheel - - # Extract version from built wheel filename - WHEEL_FILE=$(ls dist/*.whl) - NIGHTLY_VERSION=$(echo "$WHEEL_FILE" | sed 's/.*sglang-\(.*\)-py3.*/\1/') + python3 -m build --wheel --no-isolation # Get commit info COMMIT_HASH=$(git rev-parse --short HEAD) BUILD_DATE=$(date -u +%Y-%m-%d) - echo "Built wheel: $WHEEL_FILE" - echo "Nightly version: ${NIGHTLY_VERSION}" echo "Commit: ${COMMIT_HASH}" echo "Build date: ${BUILD_DATE}" - echo "nightly_version=${NIGHTLY_VERSION}" >> $GITHUB_OUTPUT - echo "commit_hash=${COMMIT_HASH}" >> $GITHUB_OUTPUT - echo "build_date=${BUILD_DATE}" >> $GITHUB_OUTPUT + { + echo "commit_hash=${COMMIT_HASH}" + echo "build_date=${BUILD_DATE}" + } >> "$GITHUB_OUTPUT" + + - name: Repair and smoke-test wheel + id: wheel + run: | + python3 scripts/release/prepare_sglang_wheel.py python/dist \ + --github-output "$GITHUB_OUTPUT" - name: Upload wheel artifact uses: actions/upload-artifact@v4 @@ -109,13 +113,13 @@ jobs: environment: 'prod' strategy: fail-fast: false - # The wheel is CUDA-agnostic and built once — we just register the same - # artifact under cu129/sglang/ and cu130/sglang/ wheel indexes so users - # can install via either --extra-index-url. Serialize because both matrix - # runs clone and push to the same sgl-whl branch. + # The wheel is CUDA-agnostic and built once — we just register it under + # the cu130/sglang/ wheel index so users can install via + # --extra-index-url. Serialize because matrix runs clone and push to the + # same sgl-whl branch. max-parallel: 1 matrix: - cuda_version: ['129', '130'] + cuda_version: ['130'] steps: - uses: actions/checkout@v4 @@ -139,7 +143,7 @@ jobs: token: ${{ secrets.GH_PAT_FOR_WHL_RELEASE }} prerelease: true body: | - Nightly build from commit ${{ github.sha }} + Nightly build from commit ${{ needs.build-nightly-wheel.outputs.commit_hash }} Build date: ${{ needs.build-nightly-wheel.outputs.build_date }} Version: ${{ needs.build-nightly-wheel.outputs.nightly_version }} files: | @@ -147,7 +151,7 @@ jobs: - name: Clone wheel index repository run: | - git clone https://oauth2:${WHL_TOKEN}@github.com/sgl-project/whl.git sgl-whl + git clone "https://oauth2:${WHL_TOKEN}@github.com/sgl-project/whl.git" sgl-whl cd sgl-whl git config --local user.name "sglang-bot" git config --local user.email "sglangbot@gmail.com" @@ -162,10 +166,10 @@ jobs: - name: Update wheel index run: | python3 scripts/update_nightly_whl_index.py \ - --commit-hash ${{ needs.build-nightly-wheel.outputs.commit_hash }} \ - --nightly-version ${{ needs.build-nightly-wheel.outputs.nightly_version }} \ - --cuda-version ${{ matrix.cuda_version }} \ - --build-date ${{ needs.build-nightly-wheel.outputs.build_date }} + --commit-hash "${{ needs.build-nightly-wheel.outputs.commit_hash }}" \ + --nightly-version "${{ needs.build-nightly-wheel.outputs.nightly_version }}" \ + --cuda-version "${{ matrix.cuda_version }}" \ + --build-date "${{ needs.build-nightly-wheel.outputs.build_date }}" - name: Push wheel index run: | diff --git a/.github/workflows/release-pypi-pr.yml b/.github/workflows/release-pypi-pr.yml index edcca5c06ae3..627f3750cbce 100644 --- a/.github/workflows/release-pypi-pr.yml +++ b/.github/workflows/release-pypi-pr.yml @@ -17,7 +17,8 @@ jobs: if: github.repository == 'sgl-project/sglang' runs-on: ubuntu-latest outputs: - wheel_version: ${{ steps.gen_version.outputs.wheel_version }} + wheel_version: ${{ steps.wheel.outputs.wheel_version }} + wheel_filename: ${{ steps.wheel.outputs.wheel_filename }} commit_hash: ${{ steps.gen_version.outputs.commit_hash }} build_date: ${{ steps.gen_version.outputs.build_date }} steps: @@ -34,7 +35,7 @@ jobs: - name: Generate PR wheel version id: gen_version run: | - LATEST_TAG=$(python3 scripts/release/get_version_tag.py) + LATEST_TAG=$(python3 scripts/release/get_version_tag.py --tag-only) BASE_VERSION=${LATEST_TAG#v} echo "Latest release tag: ${LATEST_TAG}" @@ -57,10 +58,12 @@ jobs: echo "Commit: ${COMMIT_HASH}" echo "Build date: ${BUILD_DATE}" - echo "wheel_version=${WHEEL_VERSION}" >> $GITHUB_OUTPUT - echo "commit_hash=${COMMIT_HASH}" >> $GITHUB_OUTPUT - echo "base_version=${BASE_VERSION}" >> $GITHUB_OUTPUT - echo "build_date=${BUILD_DATE}" >> $GITHUB_OUTPUT + { + echo "wheel_version=${WHEEL_VERSION}" + echo "commit_hash=${COMMIT_HASH}" + echo "base_version=${BASE_VERSION}" + echo "build_date=${BUILD_DATE}" + } >> "$GITHUB_OUTPUT" - name: Update pyproject.toml with PR wheel version run: | @@ -79,19 +82,26 @@ jobs: - name: Install build dependencies run: | - cd python - pip install build wheel setuptools + pip install \ + auditwheel build patchelf wheel "setuptools>=61.0" \ + "setuptools-rust>=1.11" "setuptools-scm>=8.0" "torch==2.13.0" - name: Build wheel run: | cd python cp ../README.md ../LICENSE . - python3 -m build --wheel + python3 -m build --wheel --no-isolation # List built wheels echo "Built wheel:" ls -lh dist/ + - name: Repair and smoke-test wheel + id: wheel + run: | + python3 scripts/release/prepare_sglang_wheel.py python/dist \ + --github-output "$GITHUB_OUTPUT" + - name: Upload wheel artifact uses: actions/upload-artifact@v4 with: @@ -127,7 +137,7 @@ jobs: prerelease: true body: | PR wheel build from PR #${{ inputs.pr_number }} - Commit: ${{ github.sha }} + Commit: ${{ needs.build-pr-wheel.outputs.commit_hash }} Build date: ${{ needs.build-pr-wheel.outputs.build_date }} Version: ${{ needs.build-pr-wheel.outputs.wheel_version }} @@ -143,14 +153,14 @@ jobs: **Direct installation:** ```bash - pip install https://github.com/sgl-project/whl/releases/download/pr-${{ inputs.pr_number }}-${{ needs.build-pr-wheel.outputs.build_date }}-${{ needs.build-pr-wheel.outputs.commit_hash }}/sglang-${{ needs.build-pr-wheel.outputs.wheel_version }}-py3-none-any.whl + pip install https://github.com/sgl-project/whl/releases/download/pr-${{ inputs.pr_number }}-${{ needs.build-pr-wheel.outputs.build_date }}-${{ needs.build-pr-wheel.outputs.commit_hash }}/${{ needs.build-pr-wheel.outputs.wheel_filename }} ``` files: | dist/*.whl - name: Clone wheel index repository run: | - git clone https://oauth2:${WHL_TOKEN}@github.com/sgl-project/whl.git sgl-whl + git clone "https://oauth2:${WHL_TOKEN}@github.com/sgl-project/whl.git" sgl-whl cd sgl-whl git config --local user.name "sglang-bot" git config --local user.email "sglangbot@gmail.com" @@ -165,10 +175,10 @@ jobs: - name: Update wheel index run: | python3 scripts/update_pr_whl_index.py \ - --pr-number ${{ inputs.pr_number }} \ - --commit-hash ${{ needs.build-pr-wheel.outputs.commit_hash }} \ - --wheel-version ${{ needs.build-pr-wheel.outputs.wheel_version }} \ - --build-date ${{ needs.build-pr-wheel.outputs.build_date }} + --pr-number "${{ inputs.pr_number }}" \ + --commit-hash "${{ needs.build-pr-wheel.outputs.commit_hash }}" \ + --wheel-version "${{ needs.build-pr-wheel.outputs.wheel_version }}" \ + --build-date "${{ needs.build-pr-wheel.outputs.build_date }}" - name: Push wheel index run: | diff --git a/.github/workflows/release-pypi.yml b/.github/workflows/release-pypi.yml index 6fffb2f6b243..591bcde68aa5 100644 --- a/.github/workflows/release-pypi.yml +++ b/.github/workflows/release-pypi.yml @@ -61,25 +61,23 @@ jobs: run: | cd python cp ../README.md ../LICENSE . - pip install build wheel setuptools setuptools-scm setuptools-rust + pip install \ + build wheel "setuptools>=61.0" "setuptools-rust>=1.11" \ + "setuptools-scm>=8.0" "torch==2.13.0" if [ -n "$RELEASE_VERSION" ]; then export SETUPTOOLS_SCM_PRETEND_VERSION="${RELEASE_VERSION#v}" echo "Pinning wheel version to $SETUPTOOLS_SCM_PRETEND_VERSION" fi - python3 -m build --wheel + python3 -m build --wheel --no-isolation # PyPI rejects plain `linux_x86_64` / `linux_aarch64` platform tags; # auditwheel rewrites the wheel's platform tag to a `manylinux_*` tag # and bundles any external native deps. The runner's glibc determines # the lowest acceptable manylinux policy. - - name: Repair wheel for manylinux + - name: Repair and smoke-test wheel run: | - cd python pip install auditwheel patchelf - mkdir -p dist-repaired - python3 -m auditwheel repair dist/*.whl -w dist-repaired/ - rm dist/*.whl - mv dist-repaired/*.whl dist/ + python3 scripts/release/prepare_sglang_wheel.py python/dist - name: Upload artifacts uses: actions/upload-artifact@v4 diff --git a/.github/workflows/release-whl-deepep.yml b/.github/workflows/release-whl-deepep.yml index 2a23e36d79c5..93b71f8aca70 100644 --- a/.github/workflows/release-whl-deepep.yml +++ b/.github/workflows/release-whl-deepep.yml @@ -7,15 +7,6 @@ on: description: "Public wheel version (for example, 0.1.0 or 0.1.1rc0)" type: string required: true - target: - description: "CUDA target" - type: choice - required: false - default: "all" - options: - - "all" - - "cu129" - - "cu130" packaging-ref: description: "DeepEP ref containing the shared sgl_deep_ep overlay" type: string @@ -30,75 +21,8 @@ env: TORCH_VERSION: "2.13.0" jobs: - build-cu129-matrix: - if: | - github.repository == 'sgl-project/sglang' && - (inputs.target == 'all' || inputs.target == 'cu129') - strategy: - fail-fast: false - matrix: - python-version: ["3.10", "3.12"] - arch: [x86_64, aarch64] - include: - - arch: x86_64 - runner: x64-kernel-build-node - source-branch: sgl-deepep-cu12-x86 - - arch: aarch64 - runner: arm-kernel-build-node - source-branch: sgl-deepep-cu12-arm - runs-on: ${{ matrix.runner }} - steps: - - name: Clean workspace - run: | - docker run --rm -v "${{ github.workspace }}:/workspace" alpine:3 \ - sh -c 'rm -rf /workspace/..?* /workspace/.[!.]* /workspace/*' || true - - - name: Checkout SGLang - uses: actions/checkout@v4 - - - name: Checkout DeepEP implementation - uses: actions/checkout@v4 - with: - repository: sgl-project/DeepEP - ref: ${{ matrix.source-branch }} - path: DeepEP-source - submodules: recursive - - - name: Checkout DeepEP packaging overlay - uses: actions/checkout@v4 - with: - repository: sgl-project/DeepEP - ref: ${{ inputs['packaging-ref'] || 'sgl-deepep-packaging' }} - path: DeepEP-packaging - sparse-checkout: sgl_deep_ep - - - name: Set wheel version - env: - RELEASE_VERSION: ${{ inputs.version }} - run: | - printf '%s' "${RELEASE_VERSION}" > DeepEP-packaging/sgl_deep_ep/VERSION - - - name: Build wheel - run: | - chmod +x scripts/build_sgl_deepep.sh DeepEP-packaging/sgl_deep_ep/build_sgl_deep_ep.sh - scripts/build_sgl_deepep.sh \ - "${{ matrix.python-version }}" \ - "12.9" \ - "${{ github.workspace }}/DeepEP-source" \ - "${{ github.workspace }}/DeepEP-packaging/sgl_deep_ep" \ - "${{ matrix.arch }}" - - - name: Upload wheel - uses: actions/upload-artifact@v4 - with: - name: deepep-wheel-cu129-py${{ matrix.python-version }}-${{ matrix.arch }} - path: DeepEP-source/dist/*.whl - if-no-files-found: error - build-cu130-matrix: - if: | - github.repository == 'sgl-project/sglang' && - (inputs.target == 'all' || inputs.target == 'cu130') + if: github.repository == 'sgl-project/sglang' strategy: fail-fast: false matrix: @@ -168,19 +92,10 @@ jobs: if-no-files-found: error release-whl: - needs: [build-cu129-matrix, build-cu130-matrix] + needs: build-cu130-matrix if: | - always() && github.repository == 'sgl-project/sglang' && - ( - (inputs.target == 'cu129' && needs.build-cu129-matrix.result == 'success') || - (inputs.target == 'cu130' && needs.build-cu130-matrix.result == 'success') || - ( - inputs.target == 'all' && - needs.build-cu129-matrix.result == 'success' && - needs.build-cu130-matrix.result == 'success' - ) - ) + needs.build-cu130-matrix.result == 'success' runs-on: ubuntu-latest steps: - name: Checkout SGLang @@ -209,14 +124,8 @@ jobs: git clone --branch gh-pages \ "https://oauth2:${WHL_TOKEN}@github.com/sgl-project/whl.git" sgl-whl - - name: Update wheel indexes - run: | - if [[ "${{ inputs.target }}" == "all" || "${{ inputs.target }}" == "cu129" ]]; then - python3 scripts/update_deepep_whl_index.py --cuda 129 - fi - if [[ "${{ inputs.target }}" == "all" || "${{ inputs.target }}" == "cu130" ]]; then - python3 scripts/update_deepep_whl_index.py --cuda 130 - fi + - name: Update wheel index + run: python3 scripts/update_deepep_whl_index.py --cuda 130 - name: Push wheel indexes env: @@ -234,15 +143,10 @@ jobs: fi release-pypi: - needs: [build-cu129-matrix, build-cu130-matrix] + needs: build-cu130-matrix if: | - always() && github.repository == 'sgl-project/sglang' && - needs.build-cu130-matrix.result == 'success' && - ( - inputs.target == 'cu130' || - (inputs.target == 'all' && needs.build-cu129-matrix.result == 'success') - ) + needs.build-cu130-matrix.result == 'success' runs-on: ubuntu-latest steps: - name: Download PyPI wheels diff --git a/.github/workflows/release-whl-deepgemm.yml b/.github/workflows/release-whl-deepgemm.yml index b321bc946e67..60edbed7568b 100644 --- a/.github/workflows/release-whl-deepgemm.yml +++ b/.github/workflows/release-whl-deepgemm.yml @@ -14,13 +14,17 @@ on: default: 'all' options: - 'all' - - 'cu129' - 'cu130' branch: description: "DeepGEMM branch to build from (default: dev)" type: string required: false default: 'dev' + validation_only: + description: "Build and test without publishing wheels" + type: boolean + required: false + default: false concurrency: group: release-sgl-deepgemm-${{ github.ref }} @@ -33,92 +37,6 @@ env: TORCH_VER: "2.13.0" jobs: - build-cu129-matrix: - if: | - github.repository == 'sgl-project/sglang' && - (github.event.inputs.target == 'all' || github.event.inputs.target == 'cu129') - strategy: - matrix: - python-version: ["3.12"] - cuda-version: ["12.9"] - arch: [x86_64, aarch64] - include: - - arch: x86_64 - runner: x64-kernel-build-node - - arch: aarch64 - runner: arm-kernel-build-node - runs-on: ${{ matrix.runner }} - steps: - - name: Clean workspace (remove root-owned files from prior runs) - run: | - docker run --rm -v "${{ github.workspace }}:/workspace" alpine:3 \ - sh -c 'rm -rf /workspace/..?* /workspace/.[!.]* /workspace/*' || true - - - uses: actions/checkout@v4 - - - name: Checkout DeepGEMM - uses: actions/checkout@v4 - with: - repository: sgl-project/DeepGEMM - ref: ${{ inputs.branch || 'dev' }} - path: DeepGEMM - submodules: recursive - - - name: Set wheel version - run: | - echo -n "${{ inputs.version }}" > DeepGEMM/sgl_deep_gemm/VERSION - cat DeepGEMM/sgl_deep_gemm/VERSION - - - name: Build wheel - run: | - chmod +x ./scripts/build_sgl_deep_gemm.sh ./scripts/rename_sgl_deep_gemm_whl.sh - ./scripts/build_sgl_deep_gemm.sh "${{ matrix.python-version }}" "${{ matrix.cuda-version }}" "${{ github.workspace }}/DeepGEMM" "${{ matrix.arch }}" - - - name: Upload artifacts - uses: actions/upload-artifact@v4 - with: - name: deepgemm-wheel-cuda${{ matrix.cuda-version }}-${{ matrix.arch }} - path: DeepGEMM/dist/*.whl - - release-cu129: - needs: build-cu129-matrix - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - - - name: Download artifacts - uses: actions/download-artifact@v4 - with: - path: dist/ - merge-multiple: true - pattern: deepgemm-wheel-cuda12.9-* - - - name: Release - uses: softprops/action-gh-release@v2 - with: - tag_name: v${{ inputs.version }} - repository: sgl-project/whl - token: ${{ secrets.GH_PAT_FOR_WHL_RELEASE }} - files: | - dist/* - - - name: Clone wheel index - run: git clone https://oauth2:${WHL_TOKEN}@github.com/sgl-project/whl.git sgl-whl - env: - WHL_TOKEN: ${{ secrets.GH_PAT_FOR_WHL_RELEASE }} - - - name: Update wheel index - run: python3 scripts/update_deepgemm_whl_index.py --cuda 129 - - - name: Push wheel index - run: | - cd sgl-whl - git config --local user.name "sglang-bot" - git config --local user.email "sglangbot@gmail.com" - git add -A - git commit -m "update sgl-deep-gemm whl index for v${{ inputs.version }}" - git push - build-cu130-matrix: if: | github.repository == 'sgl-project/sglang' && @@ -187,9 +105,11 @@ jobs: - arch_label: sm90 runner: 8-gpu-h200 wheel_arch: x86_64 + timeout: 60 - arch_label: sm100 runner: 8-gpu-b200 wheel_arch: x86_64 + timeout: 60 # The sm120 tests require a runner with RTX 6000. It causes OOM on 5090 runners. # - arch_label: sm120 # runner: 1-gpu-5090 @@ -197,8 +117,9 @@ jobs: - arch_label: sm100-aarch64 runner: 4-gpu-gb300 wheel_arch: aarch64 + timeout: 60 runs-on: ${{ matrix.runner }} - timeout-minutes: 120 + timeout-minutes: ${{ matrix.timeout }} steps: - uses: actions/checkout@v4 @@ -225,16 +146,73 @@ jobs: # deep_gemm imports torch before loading _C.so, so torch must preload the # libcudart the cu130 wheel links. Pin the CUDA-matched torch (see sgl-deep-gemm.Dockerfile). python3 -m pip install "torch==${TORCH_VER}" --index-url "https://download.pytorch.org/whl/${CU_TAG}" --force-reinstall - python3 -m pip install numpy + python3 -m pip install numpy "tilelang==0.1.9" "tile-kernels==1.0.0" "sgl-deep-ep==0.1.2" python3 -m pip install dist/*.whl + # Torch pins NCCL 2.29.7, but this DeepEP wheel requires 2.30.7. + # Override it after every dependency-resolving pip command. + python3 -m pip install "nvidia-nccl-cu13==2.30.7" --force-reinstall --no-deps + nccl_lib=$(python3 -c "from importlib.metadata import distribution; print(distribution('nvidia-nccl-cu13').locate_file('nvidia/nccl/lib'))") + export LD_LIBRARY_PATH="${nccl_lib}${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}" + echo "LD_LIBRARY_PATH=${LD_LIBRARY_PATH}" >> "${GITHUB_ENV}" python3 -c "import deep_gemm; print('deep_gemm:', deep_gemm.__file__)" + python3 - <<'PY' + import ctypes + import sys + import torch + import deep_ep + from tile_kernels.moe import top2_sum_gate + + # torch.cuda.nccl.version() reports Torch's compiled version, not the loaded library. + nccl_version = ctypes.c_int() + assert ctypes.CDLL("libnccl.so.2").ncclGetVersion(ctypes.byref(nccl_version)) == 0 + assert nccl_version.value == 23007, nccl_version.value + assert hasattr(deep_ep, "ElasticBuffer"), "DeepEP ElasticBuffer is required" + sys.path.insert(0, "DeepGEMM/third-party") + from tilelang_ops import ref_mhc + assert ref_mhc.has_baseline(), "Mega mHC reference operators are unavailable" + print("DeepGEMM test dependencies ready; NCCL:", nccl_version.value) + PY + + - name: Install Compute Sanitizer 2025.4.1 + env: + WHEEL_ARCH: ${{ matrix.wheel_arch }} + run: | + set -euo pipefail + # CUDA 13.1.1's standalone sanitizer fixes Python tensor retention in + # host backtraces. Keep the wheel and runtime on CUDA 13.0. + case "${WHEEL_ARCH}" in + x86_64) + sanitizer_arch=linux-x86_64 + sanitizer_sha256=b9637777a31cd0f0ffe1e965523e8b0d382019b8496855fd688c886988583cf4 + ;; + aarch64) + sanitizer_arch=linux-sbsa + sanitizer_sha256=1dc68ccb146032bcdc9d932569c865d68e3966ecf136940ca38fd7d5abfda4ac + ;; + *) echo "Unsupported sanitizer architecture: ${WHEEL_ARCH}" >&2; exit 1 ;; + esac + archive="cuda_sanitizer_api-${sanitizer_arch}-13.1.118-archive" + install_dir="${RUNNER_TEMP}/deepgemm-compute-sanitizer" + mkdir -p "${install_dir}" + curl -fsSL --retry 3 \ + "https://developer.download.nvidia.com/compute/cuda/redist/cuda_sanitizer_api/${sanitizer_arch}/${archive}.tar.xz" \ + -o "${install_dir}/${archive}.tar.xz" + echo "${sanitizer_sha256} ${install_dir}/${archive}.tar.xz" | sha256sum -c - + tar -xJf "${install_dir}/${archive}.tar.xz" -C "${install_dir}" + sanitizer="${install_dir}/${archive}/bin/compute-sanitizer" + "${sanitizer}" --version + echo "COMPUTE_SANITIZER=${sanitizer}" >> "${GITHUB_ENV}" - name: Run DeepGEMM test suite + env: + # Each runner is a single NVLink domain; DeepEP references do not need RDMA/GIN. + EP_DISABLE_GIN: "1" run: | chmod +x "${{ github.workspace }}/DeepGEMM/sgl_deep_gemm/run_tests.sh" - "${{ github.workspace }}/DeepGEMM/sgl_deep_gemm/run_tests.sh" + "${{ github.workspace }}/DeepGEMM/sgl_deep_gemm/run_tests.sh" --release release-cu130: + if: ${{ !inputs.validation_only }} needs: [build-cu130-matrix, test-cu130] runs-on: ubuntu-latest steps: diff --git a/.github/workflows/release-whl-kernel-xpu.yml b/.github/workflows/release-whl-kernel-xpu.yml index 4d3a7c5722f7..f47192dac267 100644 --- a/.github/workflows/release-whl-kernel-xpu.yml +++ b/.github/workflows/release-whl-kernel-xpu.yml @@ -126,7 +126,7 @@ jobs: - name: Release uses: softprops/action-gh-release@v2 with: - tag_name: ${{ inputs.tag_name }} + tag_name: ${{ inputs.tag_name }}+xpu repository: sgl-project/whl token: ${{ secrets.GH_PAT_FOR_WHL_RELEASE }} files: | diff --git a/.github/workflows/release-whl-kernel.yml b/.github/workflows/release-whl-kernel.yml index 50adb14f37f7..876aed4fbef7 100644 --- a/.github/workflows/release-whl-kernel.yml +++ b/.github/workflows/release-whl-kernel.yml @@ -15,10 +15,9 @@ on: default: 'all' options: - 'all' - - 'cu129' - 'cu130' - - 'rocm700' - 'rocm720' + - 'rocm1000' - 'musa43' tag_name: description: "Version number, must be in the form of vX.Y.Z (e.g. v0.4.0)" @@ -34,110 +33,6 @@ concurrency: cancel-in-progress: true jobs: - # cu130 is the PyPI-released variant; cu129 wheels are published only to the - # sgl-project/whl index (consumed via `pip install ...+cu129` for the legacy - # cuda 12.9 path), not to PyPI. - build-cu129-matrix: - if: | - github.repository == 'sgl-project/sglang' && - (github.event_name == 'push' || github.event.inputs.target == 'all' || github.event.inputs.target == 'cu129') - strategy: - matrix: - python-version: ["3.10"] - cuda-version: ["12.9"] - arch: [x86_64, aarch64] - include: - - arch: x86_64 - runner: x64-kernel-build-node - - arch: aarch64 - runner: arm-kernel-build-node - runs-on: ${{ matrix.runner }} - steps: - # Self-hosted build nodes retain the workspace across jobs. Prior builds - # leave root-owned artifacts under python/sglang/kernels/aot/build/ that actions/checkout - # cannot remove, causing EACCES on rmdir. Wipe them via a throwaway root - # container before checkout recreates the workspace. - - name: Clean workspace (remove root-owned files from prior runs) - run: | - docker run --rm -v "${{ github.workspace }}:/workspace" alpine:3 \ - sh -c 'rm -rf /workspace/..?* /workspace/.[!.]* /workspace/*' || true - - - uses: actions/checkout@v4 - with: - submodules: "recursive" - ref: ${{ inputs.pr_number && format('refs/pull/{0}/head', inputs.pr_number) || '' }} - - - name: Set up Python ${{ matrix.python-version }} - uses: actions/setup-python@v5 - with: - python-version: ${{ matrix.python-version }} - - - name: Build wheels - run: | - cd python/sglang/kernels/aot - chmod +x ./build.sh - ./build.sh "${{ matrix.python-version }}" "${{ matrix.cuda-version }}" ${{ matrix.arch == 'aarch64' && 'aarch64' || '' }} - env: - BUILD_JOBS: 64 - NVCC_THREADS: 8 - - - name: Upload artifacts - uses: actions/upload-artifact@v4 - with: - name: wheel-python${{ matrix.python-version }}-cuda${{ matrix.cuda-version }}${{ matrix.arch == 'aarch64' && '-aarch64' || '' }} - path: python/sglang/kernels/aot/dist/* - - release-cu129: - needs: build-cu129-matrix - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - with: - ref: ${{ inputs.pr_number && format('refs/pull/{0}/head', inputs.pr_number) || '' }} - - - name: Download artifacts - uses: actions/download-artifact@v4 - with: - path: python/sglang/kernels/aot/dist/ - merge-multiple: true - pattern: wheel-* - - - name: Set tag name - id: set_tag_name - run: | - if [ -z "${{ inputs.tag_name }}" ]; then - TAG_NAME="v$(cat python/sglang/kernels/aot/python/sgl_kernel/version.py | cut -d'"' -f2)" - echo "tag_name=$TAG_NAME" >> $GITHUB_OUTPUT - else - echo "tag_name=${{ inputs.tag_name }}" >> $GITHUB_OUTPUT - fi - - - name: Release - uses: softprops/action-gh-release@v2 - with: - tag_name: ${{ steps.set_tag_name.outputs.tag_name }} - repository: sgl-project/whl - token: ${{ secrets.GH_PAT_FOR_WHL_RELEASE }} - files: | - python/sglang/kernels/aot/dist/* - - - name: Clone wheel index - run: git clone https://oauth2:${WHL_TOKEN}@github.com/sgl-project/whl.git sgl-whl - env: - WHL_TOKEN: ${{ secrets.GH_PAT_FOR_WHL_RELEASE }} - - - name: Update wheel index - run: python3 scripts/update_kernel_whl_index.py --cuda 129 - - - name: Push wheel index - run: | - cd sgl-whl - git config --local user.name "sglang-bot" - git config --local user.email "sglangbot@gmail.com" - git add -A - git commit -m "update whl index" - git push - build-cu130-matrix: if: | github.repository == 'sgl-project/sglang' && @@ -273,12 +168,20 @@ jobs: build-rocm-matrix: if: | github.repository == 'sgl-project/sglang' && - (github.event_name == 'push' || github.event.inputs.target == 'all' || github.event.inputs.target == 'rocm700' || github.event.inputs.target == 'rocm720') + (github.event_name == 'push' || github.event.inputs.target == 'all' || github.event.inputs.target == 'rocm720' || github.event.inputs.target == 'rocm1000') runs-on: amd-docker-scale strategy: matrix: python-version: ["3.10"] - rocm-version: ["700", "720"] + # ROCm 10.0.0 builds inside a Python 3.12 container; the wheel is still + # tagged cp310-abi3 (wheel.py-api in pyproject.toml), so it stays under + # the same python-version leg as the 7.x wheels. + # + # A dispatch that names one flavor builds only that flavor. Every extra + # leg is another image build on the shared AMD node, and its release job + # appends the same wheel to the index a second time. `inputs` is empty + # on push, which falls through to the full set. + rocm-version: ${{ fromJson(inputs.target == 'rocm720' && '["720"]' || inputs.target == 'rocm1000' && '["1000"]' || '["720", "1000"]') }} steps: # Self-hosted build nodes retain the workspace across jobs. Prior builds # leave root-owned artifacts under python/sglang/kernels/aot/build/ that actions/checkout @@ -312,8 +215,9 @@ jobs: name: wheel-python${{ matrix.python-version }}-rocm${{ matrix.rocm-version }} path: python/sglang/kernels/aot/dist/* - release-rocm700: + release-rocm720: needs: build-rocm-matrix + if: ${{ github.event_name == 'push' || inputs.target == 'all' || inputs.target == 'rocm720' }} runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 @@ -325,7 +229,7 @@ jobs: with: path: python/sglang/kernels/aot/dist/ merge-multiple: true - pattern: wheel-*-rocm700 + pattern: wheel-*-rocm720 - name: Set tag name id: set_tag_name @@ -352,7 +256,7 @@ jobs: WHL_TOKEN: ${{ secrets.GH_PAT_FOR_WHL_RELEASE }} - name: Update wheel index - run: python3 scripts/update_kernel_whl_index.py --rocm 700 + run: python3 scripts/update_kernel_whl_index.py --rocm 720 - name: Push wheel index run: | @@ -363,8 +267,9 @@ jobs: git commit -m "update whl index" git push - release-rocm720: + release-rocm1000: needs: build-rocm-matrix + if: ${{ github.event_name == 'push' || inputs.target == 'all' || inputs.target == 'rocm1000' }} runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 @@ -376,7 +281,7 @@ jobs: with: path: python/sglang/kernels/aot/dist/ merge-multiple: true - pattern: wheel-*-rocm720 + pattern: wheel-*-rocm1000 - name: Set tag name id: set_tag_name @@ -403,7 +308,7 @@ jobs: WHL_TOKEN: ${{ secrets.GH_PAT_FOR_WHL_RELEASE }} - name: Update wheel index - run: python3 scripts/update_kernel_whl_index.py --rocm 720 + run: python3 scripts/update_kernel_whl_index.py --rocm 1000 - name: Push wheel index run: | diff --git a/.github/workflows/rerun-test.yml b/.github/workflows/rerun-test.yml index cb0fda3debb5..85fccc696c27 100644 --- a/.github/workflows/rerun-test.yml +++ b/.github/workflows/rerun-test.yml @@ -86,8 +86,42 @@ permissions: issues: read jobs: + prepare-partitions: + runs-on: ubuntu-latest + timeout-minutes: 10 + permissions: + contents: read + outputs: + matrix: ${{ steps.partition.outputs.matrix }} + steps: + - name: Checkout partition script + uses: actions/checkout@v4 + with: + ref: ${{ github.sha }} + persist-credentials: false + + - name: Checkout test registrations + uses: actions/checkout@v4 + with: + ref: ${{ inputs.pr_head_sha || github.sha }} + path: partition-source + persist-credentials: false + + - name: Partition tests by estimated duration + id: partition + env: + TEST_COMMAND: ${{ inputs.test_command }} + TEST_ROOT: partition-source + MODE: ${{ inputs.mode }} + run: python3 scripts/ci/partition_rerun_tests.py + rerun-test-cuda: + name: rerun-test-cuda (partition ${{ matrix.partition }}) + needs: prepare-partitions if: inputs.mode == 'cuda' + strategy: + fail-fast: false + matrix: ${{ fromJson(needs.prepare-partitions.outputs.matrix) }} runs-on: ${{ inputs.runs_on }} timeout-minutes: 120 permissions: @@ -127,6 +161,10 @@ jobs: - name: Install dependencies timeout-minutes: ${{ fromJson(inputs.install_timeout) }} + env: + # Public-read-only PAT: the checkout above leaves no job token for the + # install script's github.com clones. Unset -> anonymous. + GH_TOKEN: ${{ secrets.GH_PAT_FOR_CI_CLONE }} run: | if [[ "${{ inputs.runs_on }}" == "1-gpu-5090" ]]; then source /etc/profile.d/sglang-ci.sh @@ -146,7 +184,8 @@ jobs: - name: Run test timeout-minutes: 60 env: - TEST_COMMAND: ${{ inputs.test_command }} + TEST_COMMAND: ${{ matrix.test_command }} + REQUESTED_TEST_COMMAND: ${{ inputs.test_command }} SGLANG_PRECISION_HF_TOKEN: ${{ inputs.refresh_precision_baseline && inputs.mode == 'cuda' && inputs.runs_on == '8-gpu-h200' && inputs.test_command == 'registered/debug_utils/test_nightly_precision_regression.py' && inputs.pr_head_sha == '' && secrets.HF_TOKEN_PRECISION_STORE || '' }} run: | if [[ "${{ inputs.refresh_precision_baseline }}" == "true" ]]; then @@ -154,7 +193,7 @@ jobs: if [[ "${{ inputs.mode }}" != "cuda" \ || "${{ inputs.runs_on }}" != "8-gpu-h200" \ || -n "${{ inputs.pr_head_sha }}" \ - || "$TEST_COMMAND" != "$expected" ]]; then + || "$REQUESTED_TEST_COMMAND" != "$expected" ]]; then echo "::error::Precision baseline refresh only accepts $expected on 8-gpu-h200" exit 1 fi @@ -197,9 +236,16 @@ jobs: - uses: ./.github/actions/upload-cuda-coredumps if: failure() + with: + artifact-suffix: ${{ matrix.partition }} rerun-test-multimodal-gen: + name: rerun-test-multimodal-gen (partition ${{ matrix.partition }}) + needs: prepare-partitions if: inputs.mode == 'multimodal_gen' + strategy: + fail-fast: false + matrix: ${{ fromJson(needs.prepare-partitions.outputs.matrix) }} runs-on: ${{ inputs.runs_on }} timeout-minutes: 120 permissions: @@ -234,16 +280,20 @@ jobs: - name: Install dependencies (diffusion) timeout-minutes: 20 + env: + GH_TOKEN: ${{ secrets.GH_PAT_FOR_CI_CLONE }} run: bash scripts/ci/cuda/ci_install_dependency.sh diffusion - name: Run test timeout-minutes: 60 + env: + TEST_COMMAND: ${{ matrix.test_command }} run: | cmds=() while IFS= read -r cmd; do [ -z "$cmd" ] && continue cmds+=("$cmd") - done <<< "${{ inputs.test_command }}" + done <<< "$TEST_COMMAND" total=${#cmds[@]} suite_start=$SECONDS for idx in "${!cmds[@]}"; do @@ -266,9 +316,16 @@ jobs: - uses: ./.github/actions/upload-cuda-coredumps if: failure() + with: + artifact-suffix: ${{ matrix.partition }} rerun-test-cpu: + name: rerun-test-cpu (partition ${{ matrix.partition }}) + needs: prepare-partitions if: inputs.mode == 'cpu' + strategy: + fail-fast: false + matrix: ${{ fromJson(needs.prepare-partitions.outputs.matrix) }} runs-on: ubuntu-latest timeout-minutes: 120 permissions: @@ -337,13 +394,15 @@ jobs: - name: Run test timeout-minutes: 60 + env: + TEST_COMMAND: ${{ matrix.test_command }} run: | cd test/ cmds=() while IFS= read -r cmd; do [ -z "$cmd" ] && continue cmds+=("$cmd") - done <<< "${{ inputs.test_command }}" + done <<< "$TEST_COMMAND" total=${#cmds[@]} suite_start=$SECONDS for idx in "${!cmds[@]}"; do @@ -365,7 +424,7 @@ jobs: echo "All $total test(s) passed in ${total_elapsed}s" write-back-result: - needs: [rerun-test-cuda, rerun-test-multimodal-gen, rerun-test-cpu] + needs: [prepare-partitions, rerun-test-cuda, rerun-test-multimodal-gen, rerun-test-cpu] if: always() && inputs.reply_comment_id != '' && inputs.reply_marker != '' runs-on: ubuntu-latest permissions: diff --git a/.github/workflows/retag-docker.yml b/.github/workflows/retag-docker.yml index 633a275ed033..aa6ab03a7970 100644 --- a/.github/workflows/retag-docker.yml +++ b/.github/workflows/retag-docker.yml @@ -4,7 +4,7 @@ on: workflow_dispatch: inputs: source_tag: - description: "Existing image tag (e.g., v0.4.7-cu129-amd64)" + description: "Existing image tag (e.g., v0.4.7-cu130-amd64)" required: true target_tag: description: "New tag to apply (e.g., latest)" diff --git a/.github/workflows/runner-utilization.yml b/.github/workflows/runner-utilization.yml index d126ea40b719..15147e33ae79 100644 --- a/.github/workflows/runner-utilization.yml +++ b/.github/workflows/runner-utilization.yml @@ -68,4 +68,18 @@ jobs: python scripts/ci/utils/runner_utilization_report.py \ --repo ${{ github.repository }} \ --hours ${{ (github.event_name == 'pull_request' && '0.34') || inputs.hours || '24' }} \ - ${{ inputs.filter && format('--filter {0}', inputs.filter) || '' }} + ${{ inputs.filter && format('--filter {0}', inputs.filter) || '' }} \ + --queue-series-out queue-series.json + + # The card rides on the scan above rather than running its own: a 24h + # window is ~5000 API calls, the whole hourly budget of ci-lark-notify's PAT. + - name: Post queue timeline card + if: >- + github.repository == 'sgl-project/sglang' && + github.event_name != 'pull_request' + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + LARK_WEBHOOK: ${{ secrets.LARK_WEBHOOK }} + run: | + python scripts/ci_monitor/lark_notify.py queue-timeline \ + --series-file queue-series.json diff --git a/.github/workflows/seed-rust-ext-cache.yml b/.github/workflows/seed-rust-ext-cache.yml index 6ca4fb90db0b..6740b16df8d7 100644 --- a/.github/workflows/seed-rust-ext-cache.yml +++ b/.github/workflows/seed-rust-ext-cache.yml @@ -9,9 +9,17 @@ on: branches: [main] paths: - 'rust/**' + - 'proto/**' - 'python/setup.py' + - 'python/pyproject.toml' + - 'python/sglang/srt/rust_extensions/torch_build.py' # The key's prefix lives in this file, so a bump there also moves the key. - '.github/workflows/_pr-test-rust-ext-build.yml' + - 'scripts/ci/utils/stage_rust_ext_modules.sh' + # Not hashed into the key, but they carry the prefix each side uses: the one + # seeded per arch here, and the one consumers restore with there. + - '.github/workflows/seed-rust-ext-cache.yml' + - '.github/actions/download-rust-ext/action.yml' workflow_dispatch: # Only the newest merge needs to seed; earlier ones are already stale. @@ -30,5 +38,15 @@ jobs: seed: uses: ./.github/workflows/_pr-test-rust-ext-build.yml with: - runs_on: 1-gpu-5090 + runs_on: ubuntu-22.04 + secrets: inherit + + # One entry per arch, since the arch is part of the key. + seed-aarch64: + uses: ./.github/workflows/_pr-test-rust-ext-build.yml + with: + runs_on: ubuntu-22.04-arm + collect_runs_on: ubuntu-22.04-arm + artifact_name: rust-ext-aarch64 + cache_key_prefix: rust-ext-aarch64-cp310-cp312 secrets: inherit diff --git a/.github/workflows/slash-command-handler.yml b/.github/workflows/slash-command-handler.yml index 3c48d1aa7d48..99884117d387 100644 --- a/.github/workflows/slash-command-handler.yml +++ b/.github/workflows/slash-command-handler.yml @@ -14,13 +14,24 @@ jobs: slash_command: # Only run if it is a PR and the comment contains a recognized command # Use contains() since startsWith() can't handle leading whitespace/newlines + # + # The trailing startsWith() namespace clauses admit *unrecognized* commands + # so the script can answer them; they are startsWith() because + # contains('/run-') would fire on any comment quoting `scripts/run-tests.sh`. if: > github.event.issue.pull_request && (contains(github.event.comment.body, '/tag-run-ci-label') || contains(github.event.comment.body, '/rerun-failed-ci') || contains(github.event.comment.body, '/tag-and-rerun-ci') || + contains(github.event.comment.body, '/run-full-ci') || + contains(github.event.comment.body, '/run-extra-ci') || + contains(github.event.comment.body, '/rerun-full-ci') || + contains(github.event.comment.body, '/rerun-extra-ci') || contains(github.event.comment.body, '/rerun-group') || - contains(github.event.comment.body, '/rerun-test')) + contains(github.event.comment.body, '/rerun-test') || + startsWith(github.event.comment.body, '/tag-') || + startsWith(github.event.comment.body, '/rerun-') || + startsWith(github.event.comment.body, '/run-')) runs-on: ubuntu-latest steps: diff --git a/.github/workflows/trivy-scan-dev.yml b/.github/workflows/trivy-scan-dev.yml index f354765978f1..ab315cab6399 100644 --- a/.github/workflows/trivy-scan-dev.yml +++ b/.github/workflows/trivy-scan-dev.yml @@ -22,7 +22,7 @@ jobs: strategy: fail-fast: false matrix: - tag: ${{ inputs.tag && fromJSON(format('["{0}"]', inputs.tag)) || fromJSON('["dev", "dev-cu12"]') }} + tag: ${{ inputs.tag && fromJSON(format('["{0}"]', inputs.tag)) || fromJSON('["dev", "dev-cu13"]') }} steps: - name: Cleanup workspace (remove root-owned files from prior runs) run: sudo rm -rf "$GITHUB_WORKSPACE"/* || true diff --git a/.github/workflows/weekly-test-cpu.yml b/.github/workflows/weekly-test-cpu.yml new file mode 100644 index 000000000000..77e5cd6a5980 --- /dev/null +++ b/.github/workflows/weekly-test-cpu.yml @@ -0,0 +1,49 @@ +# Weekly CPU tests. Holds the CPU-only unit tests that are not worth a +# per-commit slot -- debug tooling (dump comparator, source patcher) whose +# breakage is fine to notice once a week. +# +# Separate from weekly-test-nvidia.yml because CPU goes through a different +# reusable workflow, and `uses:` takes no expressions -- so it cannot be another +# row in that file's matrix. Both run the same shared stage their per-commit +# counterparts do, so a test behaves identically in a PR and in the weekly run. +name: Weekly Test (CPU) + +on: + schedule: + - cron: '0 0 * * 0' # Sunday 00:00 UTC, alongside the Nvidia weekly run + workflow_dispatch: + +concurrency: + group: weekly-test-cpu-${{ github.ref }} + cancel-in-progress: true + +permissions: + actions: write + contents: read + issues: read + pull-requests: read + +jobs: + # run_all_tests skips the paths-filter, so every test runs regardless of what + # the last commit touched. + check-changes: + uses: ./.github/workflows/_pr-test-check-changes.yml + with: + pr_test_yml: '.github/workflows/weekly-test-cpu.yml' + run_all_tests: true + force_continue_on_error: true + secrets: inherit + + # No rust_ext_artifact: nothing builds one here, so the stage falls back to + # its cache and compiles on a miss. + weekly-test-cpu: + needs: check-changes + if: github.repository == 'sgl-project/sglang' + uses: ./.github/workflows/_pr-test-stage-cpu.yml + with: + self_name: weekly-test-cpu + check_changes: ${{ toJson(needs.check-changes.outputs) }} + caller_inputs: ${{ toJson(inputs) }} + partitions: ${{ needs.check-changes.outputs.partitions }} + run_timeout_minutes: '90' + secrets: inherit diff --git a/.github/workflows/weekly-test-nvidia.yml b/.github/workflows/weekly-test-nvidia.yml index 416a3a1d8fa4..6fc23124dcd0 100644 --- a/.github/workflows/weekly-test-nvidia.yml +++ b/.github/workflows/weekly-test-nvidia.yml @@ -8,13 +8,13 @@ on: workflow_dispatch: inputs: runner_filter: - description: 'Select which runner_config to run (leave empty or "all" to run all)' + # A free-text field rather than a choice: `on:` blocks take no + # expressions, so an options list would be a second copy of the matrix + # below to keep in sync. Matched in _pr-test-stage.yml. + description: 'runner_config to run alone (empty or "all" runs every one)' required: false - type: choice + type: string default: 'all' - options: - - 'all' - - '8-gpu-h200' full_parallel: description: 'Run all shards of a job at once (faster, but competes with per-commit CI for machines). Off by default: one shard at a time.' required: false @@ -43,17 +43,33 @@ jobs: force_continue_on_error: true secrets: inherit - weekly-test-8-gpu-h200: + # One row per machine type; timeouts mirror the same runner's nightly job. + # Adding a machine is one row here, and adding a test to an existing machine + # is none -- `stage="weekly"` + `runner_config=` on the test is enough. + weekly-test: + # Without this, the matrix default names the job after every value in the + # row -- "weekly-test (1-gpu-large, 120, 180)". + name: weekly-test-${{ matrix.runner_config }} needs: check-changes - if: github.repository == 'sgl-project/sglang' && (inputs.runner_filter == '' || inputs.runner_filter == 'all' || inputs.runner_filter == '8-gpu-h200') + if: github.repository == 'sgl-project/sglang' + strategy: + fail-fast: false + matrix: + include: + - { runner_config: 1-gpu-large, run_timeout: '120', job_timeout: '180' } + - { runner_config: 2-gpu-large, run_timeout: '240', job_timeout: '300' } + - { runner_config: 4-gpu-h100, run_timeout: '120', job_timeout: '180' } + - { runner_config: 4-gpu-b200, run_timeout: '150', job_timeout: '210' } + - { runner_config: 8-gpu-h200, run_timeout: '240', job_timeout: '300' } + - { runner_config: 8-gpu-b200, run_timeout: '360', job_timeout: '420' } uses: ./.github/workflows/_pr-test-stage.yml with: - self_name: weekly-test-8-gpu-h200 - runner_config: 8-gpu-h200 + self_name: weekly-test-${{ matrix.runner_config }} + runner_config: ${{ matrix.runner_config }} check_changes: ${{ toJson(needs.check-changes.outputs) }} caller_inputs: ${{ toJson(inputs) }} partitions: ${{ needs.check-changes.outputs.partitions }} - run_timeout_minutes: '240' - job_timeout_minutes: '300' + run_timeout_minutes: ${{ matrix.run_timeout }} + job_timeout_minutes: ${{ matrix.job_timeout }} scheduled: true secrets: inherit diff --git a/.github/workflows/xpu-ci-job-monitor.yml b/.github/workflows/xpu-ci-job-monitor.yml index 460a64e92afe..e51883ac85bf 100644 --- a/.github/workflows/xpu-ci-job-monitor.yml +++ b/.github/workflows/xpu-ci-job-monitor.yml @@ -127,9 +127,9 @@ jobs: id: parse run: | # Parse pr-test-xpu.yml and extract job names (exclude utility jobs) - # Excluded: check-changes, pr-gate, wait-for-stage-a, finish + # Excluded: check-changes, pr-gate, finish pr_jobs=$(yq -r '.jobs | keys | .[]' .github/workflows/pr-test-xpu.yml | \ - grep -v -E '^(check-changes|pr-gate|wait-for-stage-a|finish)$' | \ + grep -v -E '^(check-changes|pr-gate|finish)$' | \ jq -R -s -c 'split("\n") | map(select(length > 0))') echo "pr_jobs=$pr_jobs" >> $GITHUB_OUTPUT echo "PR jobs: $pr_jobs" @@ -254,3 +254,366 @@ jobs: --hours ${{ inputs.hours || '24' }} \ --input-data-file ci-data/actions-job-snapshot.json \ --summary + + # Colorful nightly XPU status dashboard, rendered into the workflow summary. + # Success-rate thresholds: >=90% healthy, >=70% degraded, else critical. + nightly-xpu-status-dashboard: + name: Nightly XPU Status Dashboard + if: ${{ !inputs.job_filter }} + needs: [fetch-actions-data, nightly-reports] + runs-on: ubuntu-latest + env: + HEALTHY_THRESHOLD: '90' + DEGRADED_THRESHOLD: '70' + WINDOW_HOURS: ${{ inputs.hours || '24' }} + steps: + - name: Download Actions data snapshot + uses: actions/download-artifact@v4 + with: + name: actions-job-snapshot + path: ci-data + + - name: Render colorful nightly status dashboard + run: | + python - <<'PY' + import json, os, sys + from datetime import datetime, timezone + + snap_path = "ci-data/actions-job-snapshot.json" + out = os.environ.get("GITHUB_STEP_SUMMARY", "/dev/stdout") + healthy = float(os.environ["HEALTHY_THRESHOLD"]) + degraded = float(os.environ["DEGRADED_THRESHOLD"]) + window_hours = os.environ.get("WINDOW_HOURS", "24") + + with open(snap_path) as f: + snapshot = json.load(f) + + # Snapshot writer emits `workflow` (filename), not `workflow_path`. + nightly = [ + j for j in snapshot.get("jobs", []) + if (j.get("workflow") or "").endswith("nightly-test-intel.yml") + ] + + # status_summary style counts, per job name + per_job = {} + for j in nightly: + name = j.get("job_name") or j.get("name", "unknown") + d = per_job.setdefault(name, {"success": 0, "failure": 0, "cancelled": 0, "skipped": 0, "in_progress": 0, "other": 0}) + status = j.get("status", "") + conclusion = j.get("conclusion", "") + if status == "completed": + if conclusion == "success": + d["success"] += 1 + elif conclusion == "failure": + d["failure"] += 1 + elif conclusion == "skipped": + d["skipped"] += 1 + elif conclusion == "cancelled": + d["cancelled"] += 1 + else: + d["other"] += 1 + elif status in ("in_progress", "queued", "waiting"): + d["in_progress"] += 1 + else: + d["other"] += 1 + + def badge(label, value, color): + # shields.io renders inline in GitHub markdown summaries. + label = label.replace("-", "--").replace(" ", "_") + value = str(value).replace("-", "--").replace(" ", "_") + return f"![{label}](https://img.shields.io/badge/{label}-{value}-{color})" + + def health(rate): + if rate is None: + return ("gray", "UNKNOWN", "⚪") # white circle + if rate >= healthy: + return ("brightgreen", "HEALTHY", "\U0001F7E2") # green circle + if rate >= degraded: + return ("yellow", "DEGRADED", "\U0001F7E1") # yellow circle + return ("red", "CRITICAL", "\U0001F534") # red circle + + totals = {"success": 0, "failure": 0, "cancelled": 0, "skipped": 0, "in_progress": 0, "other": 0} + for counts in per_job.values(): + for k, v in counts.items(): + totals[k] += v + finished = totals["success"] + totals["failure"] + overall_rate = (totals["success"] / finished * 100.0) if finished else None + fleet_color, fleet_label, fleet_dot = health(overall_rate) + + now = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M UTC") + lines = [] + lines.append("## Nightly XPU Status Dashboard") + lines.append("") + lines.append(f"_Window: last **{window_hours}h** · Snapshot rendered at **{now}**_") + lines.append("") + lines.append("### Fleet health") + lines.append("") + lines.append( + " ".join([ + badge("fleet", fleet_label, fleet_color), + badge("success_rate", f"{overall_rate:.1f}%25" if overall_rate is not None else "n%2Fa", fleet_color), + badge("healthy_threshold", f"{int(healthy)}%25", "brightgreen"), + badge("degraded_threshold", f"{int(degraded)}%25", "yellow"), + ]) + ) + lines.append("") + lines.append( + f"**{fleet_dot} Overall:** " + f"{totals['success']} passed · {totals['failure']} failed · " + f"{totals['cancelled']} cancelled · {totals['skipped']} skipped · " + f"{totals['in_progress']} in flight" + ) + lines.append("") + lines.append("### Per-model / per-job breakdown") + lines.append("") + lines.append("| Status | Job | Success rate | Passed | Failed | Skipped | Cancelled | In-flight |") + lines.append("|:------:|:----|-------------:|-------:|-------:|--------:|----------:|----------:|") + + def sort_key(item): + name, c = item + finished = c["success"] + c["failure"] + rate = (c["success"] / finished) if finished else 2.0 # untested last + return (rate, name) + + for name, c in sorted(per_job.items(), key=sort_key): + finished = c["success"] + c["failure"] + rate = (c["success"] / finished * 100.0) if finished else None + color, label, dot = health(rate) + rate_txt = f"{rate:.1f}%" if rate is not None else "n/a" + rate_badge = badge("rate", f"{rate:.1f}%25" if rate is not None else "n%2Fa", color) + lines.append( + f"| {dot} `{label}` | `{name}` | {rate_badge} {rate_txt} | " + f"{c['success']} | {c['failure']} | {c['skipped']} | {c['cancelled']} | {c['in_progress']} |" + ) + + lines.append("") + lines.append("Thresholds: " + f"\U0001F7E2 healthy ≥ {int(healthy)}% · " + f"\U0001F7E1 degraded ≥ {int(degraded)}% · " + f"\U0001F534 critical below") + lines.append("") + + with open(out, "a") as f: + f.write("\n".join(lines) + "\n") + + # Fail the step if fleet is CRITICAL so oncall sees a red X on the run. + if overall_rate is not None and overall_rate < degraded: + print(f"::error::Nightly XPU fleet is CRITICAL: success rate {overall_rate:.1f}% < {int(degraded)}%") + sys.exit(1) + PY + + # ------------------------------------------------------------------- + # Intel(XPU) Nightly report — per-model 1x/2x/4x/8x tables. + # Downloads the per-model metrics jsonl artifacts uploaded by + # nightly-test-intel.yml (SGLANG_TEST_METRICS_FILE side-writer) and renders + # one Markdown table per GPU-count suite. 8x is future-ready: it renders + # "No metrics recorded" until an 8-GPU suite is enabled upstream. + # ------------------------------------------------------------------- + nightly-xpu-per-model-report: + name: Intel(XPU) Nightly report + # Fork PRs don't get secrets and can't read cross-workflow artifacts; + # rest is same gating as the fetch-actions-data job above. + if: >- + github.event_name != 'pull_request' || + (github.event.pull_request.head.repo.full_name == github.repository && + contains(github.event.pull_request.labels.*.name, 'run-ci')) + runs-on: ubuntu-latest + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + steps: + - name: Resolve latest Nightly Test (Intel) run id + id: source-run + run: | + # Pick the most recent completed run of nightly-test-intel.yml on the + # default branch. workflow_dispatch reruns are included; skipped or + # in-progress runs are not. + RUN_ID=$(gh run list \ + --repo "${GITHUB_REPOSITORY}" \ + --workflow nightly-test-intel.yml \ + --branch main \ + --status completed \ + --limit 1 \ + --json databaseId --jq '.[0].databaseId // empty') + if [ -z "$RUN_ID" ]; then + echo "No completed Nightly Test (Intel) run found on main." + echo "run_id=" >> "$GITHUB_OUTPUT" + else + echo "Latest completed nightly run: $RUN_ID" + echo "run_id=$RUN_ID" >> "$GITHUB_OUTPUT" + fi + + - name: Download per-model metrics artifacts + if: steps.source-run.outputs.run_id != '' + uses: actions/download-artifact@v4 + with: + pattern: nightly-xpu-*-metrics + path: metrics/ + run-id: ${{ steps.source-run.outputs.run_id }} + github-token: ${{ secrets.GITHUB_TOKEN }} + repository: ${{ github.repository }} + # Keep artifact names as subdirs so we can split by GPU-count suite. + merge-multiple: false + continue-on-error: true + + - name: Render per-model 1x / 2x / 4x / 8x tables + env: + SOURCE_RUN_ID: ${{ steps.source-run.outputs.run_id }} + run: | + python - <<'PY' + import glob + import json + import os + + SUITES = [ + ("1x", "nightly-xpu-1-gpu-metrics"), + ("2x", "nightly-xpu-2-gpu-metrics"), + ("4x", "nightly-xpu-4-gpu-metrics"), + # 8x is here for future readiness. Until nightly-test-intel.yml + # adds an 8-GPU suite + upload step, its artifact directory is + # missing and the section renders "No metrics recorded". + ("8x", "nightly-xpu-8-gpu-metrics"), + ] + + def load(subdir: str): + path = os.path.join("metrics", subdir) + records = [] + for f in sorted(glob.glob(os.path.join(path, "*.jsonl"))): + with open(f) as fh: + for line in fh: + line = line.strip() + if not line: + continue + try: + records.append(json.loads(line)) + except json.JSONDecodeError: + # Skip malformed rows so one bad line doesn't + # blank the whole table. + continue + return records + + def fmt(value, precision: int = 2) -> str: + if value is None or value == "" or value == "N/A": + return "N/A" + try: + return f"{float(value):.{precision}f}" + except (TypeError, ValueError): + return str(value) + + out = os.environ.get("GITHUB_STEP_SUMMARY", "/dev/stdout") + source_run = os.environ.get("SOURCE_RUN_ID", "") + repo = os.environ.get("GITHUB_REPOSITORY", "") + + lines = ["", "## Intel(XPU) Nightly report", ""] + if source_run and repo: + lines.append( + f"Source: [Intel(XPU) Nightly report - run #{source_run}]" + f"(https://github.com/{repo}/actions/runs/{source_run})" + ) + lines.append("") + elif not source_run: + lines.append("_No completed Nightly Test (Intel) run found; nothing to render._") + lines.append("") + + def rows_for(records): + # Group records by test_file. Two record shapes: + # kind=="model": rich per-model row emitted by tests that call + # write_results_to_github_step_summary. + # kind=="file": file-level fallback emitted by run_unittest_files + # for every test file in the suite. + # If a test_file has any model rows, we show those and drop the + # matching file row so a test does not double-count. Test files + # that never emit a model row fall back to the file row - that + # is what makes newly-added tests appear automatically. + by_file_model = {} + by_file_file = {} + orphans_model = [] + for r in records: + kind = r.get("kind") + tf = r.get("test_file", "") or "" + if kind == "model": + if tf: + by_file_model.setdefault(tf, []).append(r) + else: + orphans_model.append(r) + else: + # "file" or legacy records without kind + if tf and "model" not in r: + by_file_file[tf] = r + else: + orphans_model.append(r) + + rendered = [] + for tf in sorted(set(by_file_model) | set(by_file_file)): + if tf in by_file_model: + for r in by_file_model[tf]: + rendered.append(("model", r)) + else: + rendered.append(("file", by_file_file[tf])) + for r in orphans_model: + rendered.append(("model", r)) + return rendered + + any_fail = 0 + for label, subdir in SUITES: + records = load(subdir) + lines.append(f"### {label} ({subdir[:-len('-metrics')]})") + lines.append("") + if not records: + lines.append("_No metrics recorded for this suite._") + lines.append("") + continue + + lines.append( + "| Model / Test file | Prompts | Ref accuracy | Actual accuracy " + "| Ref throughput | Actual throughput | Duration | Status |" + ) + lines.append( + "|-------------------|--------:|-------------:|----------------:" + "|---------------:|------------------:|---------:|:------:|" + ) + for row_kind, r in rows_for(records): + status = str(r.get("status", "")).lower() + is_pass = status == "pass" + if not is_pass: + any_fail += 1 + status_cell = "PASS" if is_pass else (status.upper() or "N/A") + if row_kind == "model": + duration = r.get("latency") + lines.append( + f"| `{r.get('model','?')}` " + f"| {r.get('num_prompts','N/A')} " + f"| {fmt(r.get('accuracy_threshold'), 3)} " + f"| {fmt(r.get('accuracy'), 3)} " + f"| {fmt(r.get('output_throughput_threshold'), 2)} " + f"| {fmt(r.get('output_throughput'), 2)} " + f"| {fmt(duration, 2)} " + f"| {status_cell} |" + ) + else: + tf = r.get("test_file", "?") + lines.append( + f"| {tf} " + f"| N/A | N/A | N/A | N/A | N/A " + f"| {fmt(r.get('duration'), 2)} " + f"| {status_cell} |" + ) + lines.append("") + + lines.append( + "Rows with a model name are per-model records emitted by " + "`write_results_to_github_step_summary`; rows with just a test " + "file name are auto-recorded by `run_unittest_files`. Ref columns " + "are the accuracy / throughput thresholds declared by each test; " + "Actual is what nightly measured. Any new test added to " + "run_suite.py shows up here on the next nightly." + ) + lines.append("") + + with open(out, "a") as f: + f.write("\n".join(lines) + "\n") + + # Surface — but don't hard-fail — the report step: nightly-test-intel + # already flags model failures. This job is a rendering step. + if any_fail: + print(f"::warning::{any_fail} nightly XPU model row(s) marked FAIL in the per-model tables") + PY diff --git a/.gitignore b/.gitignore index cd1c6325d355..6009240589a6 100644 --- a/.gitignore +++ b/.gitignore @@ -171,8 +171,10 @@ benchmark/mmlu/data.tar benchmark/llava_bench/images benchmark/llava_bench/mme_pack *.jsonl +!tools/sglang-simulator/examples/replay/trace.jsonl tmp*.txt /tmp/ +.tmp/ # Torch Compile logs tl_out/ @@ -231,6 +233,8 @@ work_dirs/ # Rust lib Cargo.lock !rust/Cargo.lock +!rust/sglang-radix-tree/Cargo.lock +!experimental/sgl-router/Cargo.lock # Generated vision test fixtures (regenerate with: python scripts/generate_vision_golden.py) sgl-model-gateway/tests/fixtures/golden/ diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index f7a14d03e84e..5584fd5dd295 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -48,10 +48,7 @@ repos: python/sglang/srt/grpc/.*_pb2\.pyi$| python/sglang/srt/grpc/.*_pb2_grpc\.pyi$| )$ - - repo: https://github.com/psf/black - rev: 26.1.0 - hooks: - - id: black-jupyter + - id: ruff-format exclude: '^python/sglang/srt/grpc/.*_pb2\.py$|^python/sglang/srt/grpc/.*_pb2_grpc\.py$|^python/sglang/srt/grpc/.*_pb2\.pyi$|^python/sglang/srt/grpc/.*_pb2_grpc\.pyi$' - repo: https://github.com/codespell-project/codespell rev: v2.4.1 @@ -114,12 +111,6 @@ repos: entry: python3 scripts/lint/check_no_bare_pytest_main.py language: system files: ^(python|test)/.*\.py$ - - id: check-lint-script-tests - name: unit tests for lint checkers - entry: python3 -m unittest discover -s scripts/lint -p 'test_check_*.py' - language: system - files: ^scripts/lint/(check_|test_check_).*\.py$ - pass_filenames: false - id: check-registered-tests name: validate registered test CI registries entry: python3 scripts/lint/check_registered_tests.py @@ -151,13 +142,13 @@ repos: # cover sglang-mm's PyO3 bindings + rayon fan-out — both sit behind # non-default features, so `--workspace` alone never compiles them. protoc # is not required: sglang-grpc's build.rs falls back to a vendored binary. - entry: bash -c 'cd rust && cargo clippy --workspace --fix --allow-dirty --allow-staged && cargo clippy --workspace -- -D warnings && cargo clippy -p sglang-mm --features python,parallel --lib -- -D warnings' + entry: bash -c 'cd rust && cargo clippy --workspace --fix --allow-dirty --allow-staged && cargo clippy --workspace -- -D warnings && cargo clippy -p sglang-mm --features python,parallel --lib -- -D warnings && cargo clippy --manifest-path sglang-radix-tree/Cargo.toml --all-targets --no-default-features --features tch/doc-only -- -D warnings' language: system files: ^rust/.*\.rs$ pass_filenames: false - id: rustfmt-rust-workspace name: rustfmt rust/ workspace - entry: bash -c 'cd rust && cargo fmt' + entry: bash -c 'cd rust && cargo fmt && cargo fmt --manifest-path sglang-radix-tree/Cargo.toml' language: system files: ^rust/.*\.rs$ pass_filenames: false diff --git a/3rdparty/amd/tuning/benchmark_moe_rocm.py b/3rdparty/amd/tuning/benchmark_moe_rocm.py index d7ea67c5fab1..71b4d17107ad 100644 --- a/3rdparty/amd/tuning/benchmark_moe_rocm.py +++ b/3rdparty/amd/tuning/benchmark_moe_rocm.py @@ -187,8 +187,10 @@ def run_grid(bs, model, method, tp_size, dtype: str): configs = union_of_list_of_dicts(prune_configs_1, prune_configs_2) - print(f"{bs=} || {len(full_configs)=} | {len(prune_configs_1)=} | \ - {len(prune_configs_2)=} | {len(configs)=}") + print( + f"{bs=} || {len(full_configs)=} | {len(prune_configs_1)=} | \ + {len(prune_configs_2)=} | {len(configs)=}" + ) best_config = None best_time_us = 1e20 diff --git a/3rdparty/amd/wheel/README.md b/3rdparty/amd/wheel/README.md index 7d14c704fe0b..6b2e13f771ce 100644 --- a/3rdparty/amd/wheel/README.md +++ b/3rdparty/amd/wheel/README.md @@ -4,7 +4,7 @@ Building and releasing `sglang-kernel` as a wheel is a part of the release workf # sglang -`3rdparty/amd/wheel/sglang/pyproject.toml` is the AMD-specific pyproject for building the `amd-sglang` wheel. It extends `python/pyproject_other.toml` with two ROCm-version extras (`rocm700`, `rocm720`) that pin the matching torch/triton/torchaudio/torchvision/`sglang-kernel` wheels, and renames the package to `amd-sglang`. +`3rdparty/amd/wheel/sglang/pyproject.toml` is the AMD-specific pyproject for building the `amd-sglang` wheel. It extends `python/pyproject_other.toml` with a `rocm720` extra that pins the matching torch/triton/torchaudio/torchvision/`sglang-kernel` wheels, and renames the package to `amd-sglang`. ## Operation to build sglang wheel @@ -18,12 +18,6 @@ $ cd python && python -m build ### v0.5.9 -ROCm 7.0.0: -``` -pip uninstall sglang-kernel sglang amd-sglang -pip install "amd-sglang[all-hip,rocm700]" -i https://pypi.amd.com/rocm-7.0.0/simple --extra-index-url https://pypi.org/simple -``` - ROCm 7.2.0: ``` pip uninstall sglang-kernel sglang amd-sglang @@ -39,13 +33,9 @@ Note: You must resolve the two dependencies, AITER and triton, below. Others ar [AITER](https://github.com/ROCm/aiter) is a fundamental dependency. Wheel-izing it is ongoing. Until we can pin it reliably, install it manually (typically following the [ROCm docker recipe](https://github.com/sgl-project/sglang/blob/main/docker/rocm.Dockerfile#L106). -### Revolving triton +### Resolving triton -To avoid known issues in triton 3.5.1 installed by default, we recommend upgrading triton after installation. In ROCm 7.0.0 environment, -``` -pip install triton==3.6.0 -``` -or ROCm 7.2.0, +To avoid known issues in triton 3.5.1 installed by default, we recommend upgrading triton after installation in a ROCm 7.2.0 environment: ``` pip install https://repo.radeon.com/rocm/manylinux/rocm-rel-7.2/triton-3.6.0%2Brocm7.2.0.gitba5c1517-cp310-cp310-linux_x86_64.whl ``` diff --git a/3rdparty/amd/wheel/sgl-kernel/build_rocm.sh b/3rdparty/amd/wheel/sgl-kernel/build_rocm.sh index 4347737fa534..906a5fb2af59 100755 --- a/3rdparty/amd/wheel/sgl-kernel/build_rocm.sh +++ b/3rdparty/amd/wheel/sgl-kernel/build_rocm.sh @@ -3,18 +3,59 @@ set -euo pipefail ROCM_VERSION=${1:-} -if [[ "${ROCM_VERSION}" == "700" ]]; then - IMAGE="lmsysorg/sglang:v0.5.8.post1-rocm700-mi35x" -elif [[ "${ROCM_VERSION}" == "720" ]]; then +if [[ "${ROCM_VERSION}" == "720" ]]; then IMAGE="rocm/pytorch:rocm7.2_ubuntu22.04_py3.10_pytorch_release_2.9.1" +elif [[ "${ROCM_VERSION}" == "1000" ]]; then + # Ubuntu 24.04 / Python 3.12 / torch 2.11 matches the stack the released + # ROCm 10 images carry, and this image is built for every device arch, so one + # wheel can still cover both gfx942 and gfx950. + IMAGE="rocm/pytorch:rocm10.0_ubuntu24.04_py3.12_pytorch_release_2.11.0" else - echo "ERROR: Unsupported ROCM_VERSION='${ROCM_VERSION}'. Only '700' and '720' are supported." >&2 + echo "ERROR: Unsupported ROCM_VERSION='${ROCM_VERSION}'. Only '720' and '1000' are supported." >&2 exit 1 fi PYTHON_ROOT_PATH="/opt/venv/bin" AMDGPU_TARGET="gfx942;gfx950" +# ROCm 10.0.0 is distributed as pip packages that unpack into site-packages, so +# the image has neither the /opt/rocm tree CMakeLists_rocm.txt looks for +# hip-lang under nor, being a runtime image, a devel tree to compile against. +# These are the same fixups docker/rocm.Dockerfile's rocm1000-base stage makes +# for the released images. +ROCM_SETUP="" +if [[ "${ROCM_VERSION}" == "1000" ]]; then + ROCM_SETUP=$(cat <<'ROCM1000_SETUP' + set -e + apt-get update && apt-get install -y --no-install-recommends build-essential ca-certificates wget + rocm_sdk_version=$(pip show rocm-sdk-core | awk '/^Version:/ {print $2}') + test -n "${rocm_sdk_version}" + pip install --no-cache-dir --index-url https://stable.repo.amd.com/rocm/whl-next/ "rocm-sdk-devel==${rocm_sdk_version}" + rocm-sdk init + export ROCM_HOME=$(python -c 'import sysconfig; print(sysconfig.get_paths()["purelib"])')/_rocm_sdk_devel + export ROCM_PATH="${ROCM_HOME}" + export CPATH="${ROCM_HOME}/include" + export LIBRARY_PATH="${ROCM_HOME}/lib" + export LD_LIBRARY_PATH="${ROCM_HOME}/lib" + export PATH="${ROCM_HOME}/llvm/bin:${ROCM_HOME}/bin:${PATH}" + ln -sfn "${ROCM_HOME}" /opt/rocm + # The SDK's hsakmtTargets.cmake hardcodes /usr/lib64/libc.so from its own + # build host; Ubuntu keeps libc in /lib/x86_64-linux-gnu. + mkdir -p /usr/lib64 && ln -sf /lib/x86_64-linux-gnu/libc.so /usr/lib64/libc.so +ROCM1000_SETUP +) +fi + +# ROCm 10 has no /opt/rocm- directory for rename_wheels_rocm.sh to read +# the local version tag from, so name it here instead. Keeping it in step with +# the `--rocm` argument release-whl-kernel.yml passes to +# scripts/update_kernel_whl_index.py is what puts the wheel in the index it +# gets published under. +WHEEL_ROCM_VERSION="" +if [[ "${ROCM_VERSION}" == "1000" ]]; then + WHEEL_ROCM_VERSION="1000" +fi + # Pull and run the latest image echo "Pulling Docker image: ${IMAGE}" docker pull "${IMAGE}" @@ -26,11 +67,10 @@ docker run --rm \ ${IMAGE} \ bash -c " # Install torch, triton, and friends, depending on the ROCm version - if [[ "${ROCM_VERSION}" == "700" ]]; then - ${PYTHON_ROOT_PATH}/pip install https://repo.radeon.com/rocm/manylinux/rocm-rel-7.0.2/torch-2.9.1.dev20251204%2Brocm7.0.2.lw.git351ff442-cp310-cp310-linux_x86_64.whl https://repo.radeon.com/rocm/manylinux/rocm-rel-7.0.2/triton-3.5.1%2Brocm7.0.2.gita272dfa8-cp310-cp310-linux_x86_64.whl https://repo.radeon.com/rocm/manylinux/rocm-rel-7.0.2/torchaudio-2.9.0%2Brocm7.0.2.gite3c6ee2b-cp310-cp310-linux_x86_64.whl https://repo.radeon.com/rocm/manylinux/rocm-rel-7.0.2/torchvision-0.24.0%2Brocm7.0.2.gitb919bd0c-cp310-cp310-linux_x86_64.whl - elif [[ "${ROCM_VERSION}" == "720" ]]; then + if [[ "${ROCM_VERSION}" == "720" ]]; then ${PYTHON_ROOT_PATH}/pip install https://repo.radeon.com/rocm/manylinux/rocm-rel-7.2/torch-2.9.1%2Brocm7.2.0.lw.git7e1940d4-cp310-cp310-linux_x86_64.whl https://repo.radeon.com/rocm/manylinux/rocm-rel-7.2/triton-3.5.1%2Brocm7.2.0.gita272dfa8-cp310-cp310-linux_x86_64.whl https://repo.radeon.com/rocm/manylinux/rocm-rel-7.2/torchaudio-2.9.0%2Brocm7.2.0.gite3c6ee2b-cp310-cp310-linux_x86_64.whl https://repo.radeon.com/rocm/manylinux/rocm-rel-7.2/torchvision-0.24.0%2Brocm7.2.0.gitb919bd0c-cp310-cp310-linux_x86_64.whl fi +${ROCM_SETUP} # Install CMake (version >= 3.26) - Robust Installation export CMAKE_VERSION_MAJOR=3.31 export CMAKE_VERSION_MINOR=1 @@ -46,5 +86,5 @@ docker run --rm \ rm -rf CMakeLists.txt && mv CMakeLists_rocm.txt CMakeLists.txt && \ ${PYTHON_ROOT_PATH}/python rocm_hipify.py && \ ${PYTHON_ROOT_PATH}/python -m uv build --wheel -Cbuild-dir=build . --color=always --no-build-isolation && \ - ./rename_wheels_rocm.sh + ./rename_wheels_rocm.sh ${WHEEL_ROCM_VERSION} " diff --git a/3rdparty/amd/wheel/sgl-kernel/rename_wheels_rocm.sh b/3rdparty/amd/wheel/sgl-kernel/rename_wheels_rocm.sh index 3c8fb2a63b6e..a73273614380 100755 --- a/3rdparty/amd/wheel/sgl-kernel/rename_wheels_rocm.sh +++ b/3rdparty/amd/wheel/sgl-kernel/rename_wheels_rocm.sh @@ -3,6 +3,11 @@ set -ex WHEEL_DIR="dist" +# ROCm version for the wheel's local version tag (e.g. 720 -> +rocm720). It is +# read from the /opt/rocm- tree when the caller does not name one; the +# pip-installed ROCm 10 SDK has no such tree, so that flavor passes it in. +ROCM_WHEEL_VERSION="${1:-}" + wheel_files=($WHEEL_DIR/*.whl) for wheel in "${wheel_files[@]}"; do intermediate_wheel="${wheel/linux/manylinux2014}" @@ -17,7 +22,10 @@ for wheel in "${wheel_files[@]}"; do fi # Detect ROCm version and add appropriate suffix - ver_abrv=$(realpath /opt/rocm-* | sed -e 's/.*-//' -e 's/\.//g') + ver_abrv="${ROCM_WHEEL_VERSION}" + if [[ -z "$ver_abrv" ]]; then + ver_abrv=$(realpath /opt/rocm-* | sed -e 's/.*-//' -e 's/\.//g') + fi new_wheel=${intermediate_wheel/-cp${cp_version}/+rocm${ver_abrv}-cp${cp_version}} if [[ "$wheel" != "$new_wheel" ]]; then diff --git a/3rdparty/amd/wheel/sgl-kernel/rocm_hipify.py b/3rdparty/amd/wheel/sgl-kernel/rocm_hipify.py index 7383408bed16..fdfdf6074ed5 100644 --- a/3rdparty/amd/wheel/sgl-kernel/rocm_hipify.py +++ b/3rdparty/amd/wheel/sgl-kernel/rocm_hipify.py @@ -18,7 +18,7 @@ "csrc/common_extension_rocm.cc", "csrc/elementwise/activation.cu", "csrc/elementwise/pos_enc.cu", - "csrc/elementwise/topk.cu", + # topk.hip is maintained as native HIP instead of being generated from topk.cu. "csrc/grammar/apply_token_bitmask_inplace_cuda.cu", "csrc/kvcacheio/transfer.cu", "csrc/memory/weak_ref_tensor.cpp", diff --git a/3rdparty/amd/wheel/sglang/pyproject.toml b/3rdparty/amd/wheel/sglang/pyproject.toml index 0df2b20ee282..cbccd1008e80 100644 --- a/3rdparty/amd/wheel/sglang/pyproject.toml +++ b/3rdparty/amd/wheel/sglang/pyproject.toml @@ -62,23 +62,11 @@ runtime_common = [ "transformers==5.12.1", "uvicorn", "uvloop", - "xgrammar==0.2.1", - "smg-grpc-servicer>=0.5.0", + "xgrammar==0.2.7", + "smg-grpc-servicer>=0.9.0", ] # ROCm specific packages (https://repo.radeon.com/rocm/manylinux/) -# Existing practice for daily rocm700 docker images relies on 700-rc -# versions of software that are not public available. Here we pin some -# from rocm702 as the closest set as daily rocm700 images. -rocm700 = [ - "torch @ https://repo.radeon.com/rocm/manylinux/rocm-rel-7.0.2/torch-2.9.1.dev20251204%2Brocm7.0.2.lw.git351ff442-cp310-cp310-linux_x86_64.whl", - "triton @ https://repo.radeon.com/rocm/manylinux/rocm-rel-7.0.2/triton-3.5.1%2Brocm7.0.2.gita272dfa8-cp310-cp310-linux_x86_64.whl", - "torchaudio @ https://repo.radeon.com/rocm/manylinux/rocm-rel-7.0.2/torchaudio-2.9.0%2Brocm7.0.2.gite3c6ee2b-cp310-cp310-linux_x86_64.whl", - "torchvision @ https://repo.radeon.com/rocm/manylinux/rocm-rel-7.0.2/torchvision-0.24.0%2Brocm7.0.2.gitb919bd0c-cp310-cp310-linux_x86_64.whl", - "mooncake-transfer-engine-non-cuda==0.3.8.post1", - "sglang-kernel @ https://github.com/sgl-project/whl/releases/download/v0.4.0/sglang_kernel-0.4.0+rocm700-cp310-abi3-manylinux2014_x86_64.whl", -] - rocm720 = [ "torch @ https://repo.radeon.com/rocm/manylinux/rocm-rel-7.2/torch-2.9.1%2Brocm7.2.0.lw.git7e1940d4-cp310-cp310-linux_x86_64.whl", "triton @ https://repo.radeon.com/rocm/manylinux/rocm-rel-7.2/triton-3.5.1%2Brocm7.2.0.gita272dfa8-cp310-cp310-linux_x86_64.whl", @@ -89,8 +77,7 @@ rocm720 = [ ] # HIP (Heterogeneous-computing Interface for Portability) for AMD -# Install with one of: -# pip install "amd-sglang[srt_hip,rocm700]" +# Install with: # pip install "amd-sglang[srt_hip,rocm720]" srt_hip = [ "amd-sglang[runtime_common]", diff --git a/README.md b/README.md index 37b81417513c..79bfc4fdf4b5 100644 --- a/README.md +++ b/README.md @@ -75,6 +75,7 @@ Its core features include: ## Getting Started - [Install SGLang](https://docs.sglang.io/get_started/install.html) - [Quick Start](https://docs.sglang.io/basic_usage/send_request.html) +- [Cookbook](https://docs.sglang.io/cookbook) — the deployment command we recommend for each supported model - [Backend Tutorial](https://docs.sglang.io/basic_usage/openai_api_completions.html) - [Frontend Tutorial](https://docs.sglang.io/references/frontend/frontend_tutorial.html) - [Contribution Guide](https://docs.sglang.io/developer_guide/contribution_guide.html) diff --git a/benchmark/agentic-rollout/README.md b/benchmark/agentic-rollout/README.md new file mode 100644 index 000000000000..be9155ce007d --- /dev/null +++ b/benchmark/agentic-rollout/README.md @@ -0,0 +1,102 @@ +# Rollout Simulator and Explorer + +Run independent conversations against an existing SGLang server, with simulated tool waits and growing history. The client runs on any machine with Python 3.11+ and network access to the server. `uv run` installs the client dependencies; it does not need SGLang, CUDA, a trainer, or a dataset locally. + +```text +prompt -> generate -> tool delay -> append tool result -> repeat +finish configured turns -> close +``` + +`--conversations N --turns T` runs exactly N conversations of T turns each. `--concurrency` limits in-flight generation requests; tool waits release those slots. + +Inputs are readable synthetic text, tokenized, repeated, and trimmed to exact lengths. A distinct conversation prefix limits cross-conversation reuse. Model outputs remain in the history as exact token IDs. The client inserts a tool step after every generation; it does not parse or execute model-produced tool calls. The seed controls tool delays and synthetic inputs, not bitwise reproducibility of GPU outputs. + +## Quick start + +Example server command (GPU required; validation status is in the evidence note): + +```bash +python -m sglang.launch_server \ + --model-path Qwen/Qwen3-0.6B --host 0.0.0.0 --port 30000 \ + --enable-session-radix-cache --enable-hierarchical-cache \ + --hicache-ratio 1.6 --hicache-write-policy write_through \ + --enable-metrics --stream-interval 1 +``` + +From the repository root (or use an absolute script path from another directory): + +```bash +uv run benchmark/agentic-rollout/simulate.py \ + --base-url http://SERVER:30000 --tokenizer Qwen/Qwen3-0.6B \ + --mode ordinary --conversations 8 --concurrency 8 \ + --initial-tokens 2048 --tool-tokens 256 --output-tokens 128 \ + --turns 8 --tool-delay 1 3 --seed 1 \ + --output-dir results/ordinary + +uv run benchmark/agentic-rollout/explore.py results/ordinary --output rollout.html + +uv run benchmark/agentic-rollout/plot.py results/ordinary \ + --output ordinary.png +``` + +The tokenizer must match the server; use a fixed local tokenizer directory for reproducible runs. `--trust-remote-code` is explicit. The defaults are a small functional workload, not a guarantee of cache pressure. For a smoke test, use `--turns 2 --initial-tokens 128 --tool-tokens 32 --output-tokens 16`. + +## Compare features + +| Client mode | History sent on each continuation | +| --- | --- | +| `full-history` | All previous input and output token IDs, plus the new result | +| `ordinary` | Session/request IDs and only the new result | +| `streaming` | Streaming-session/request IDs and only the new result | + +All modes use HTTP streaming for timing. The server must support the selected session mode. For servers that gate streaming sessions, launch with `--enable-streaming-session` for that mode. Unsupported requests fail; the client never substitutes a different mode. + +Restart the server between runs. Hold the model, workload, seed, rank assignment, and all unrelated launch settings fixed. Change one feature at a time: session mode, `--enable-session-radix-cache`, `--enable-hierarchical-cache`, `--hicache-ratio`, `--hicache-write-policy`, or `--page-size`. Supported combinations depend on the serving version and attention backend. Keep the exact server command and wheel/source revision with each result. The client records resolved `/server_info`, but cannot read a remote machine's wheel checksum or environment. + +By default, client-side DP sticky routing assigns conversation `i` to `i % dp_size`, using the server's reported size. Every turn stays on that rank. A missing response rank is accepted only when the server reports exactly one DP worker. Use `--disable-dp-sticky-routing` to let the server choose ranks. Choose conversation and concurrency counts divisible by DP size. Tool waits release concurrency slots. The client sends `capacity_of_str_len=1000` as an unused placeholder required by older servers; it does not change engine KV capacity. + +A larger pressure workload: + +```bash +uv run benchmark/agentic-rollout/simulate.py \ + --base-url http://SERVER:30000 --tokenizer /path/to/matching/tokenizer \ + --mode ordinary --conversations 256 --concurrency 256 \ + --initial-tokens 8192 --tool-tokens 2048 --output-tokens 128 \ + --turns 56 --tool-delay 1 3 --seed 1 \ + --output-dir results/page1-a +``` + +Check measured GPU/host capacities first. Final history per conversation is `initial + (turns - 1) * (tool + output) + output`. Increase conversations until the live histories exceed GPU capacity while fitting host capacity, then freeze the workload. Verify actual CPU restores; a large configured workload alone is not evidence of cache pressure. Use fresh output directories and servers for each comparison; repeat in reverse order to check repeatability. + +## Read results + +The client writes `manifest.json`, `requests.jsonl`, `sessions.jsonl`, and raw `metrics.jsonl`. It stops on request failures, records partial evidence, and attempts session cleanup. A close HTTP response confirms submission, not synchronous release of every cache allocation. Metrics scrape errors are recorded separately; missing metrics do not invalidate successful client timing. + +The plotter accepts multiple directories and writes elapsed-time and per-turn plots, per-run CSV/summary files, and a configuration sidecar for reviewing differences. Failed runs are labeled and must be excluded from clean performance comparisons. It retains their successful requests for diagnosis. + +TTFT starts when the client submits HTTP work and ends at the first generated-token event. Local semaphore waiting and tool delays are separate. Token timing uses observed stream events; several tokens can arrive together, so it is not a kernel timing or a distribution of individual token gaps. Output rate counts token increments in their arrival windows. Cache-hit fractions are token-weighted; missing GPU/host breakdowns stay unavailable. + +By default, scrape the generation endpoint's `/metrics` once per second. Repeat `--metrics-url http://NODE:PORT/metrics` for **distinct** distributed exporters. Do not list the same exporter twice through aliases. Plots keep exporters separate and show maximum reported occupancy across ranks; raw labels remain intact. Only exported ranks are observed. A flat total hit rate can conceal a shift from GPU hits to more expensive CPU restores. + +A candidate slowdown is three consecutive 30-second windows with p95 TTFT at least twice a preceding stable period while total cache hit stays within five percentage points. Inspect context lengths and queueing before attributing it to HiCache. Compare matching turns as well as elapsed time; faster runs advance through the workload sooner. The plotter exposes measurements rather than automatically declaring a cause. + +## Offline Explorer + +Open the generated rollout.html directly in a browser. Data, CSS and JavaScript are embedded; no server, CDN, Grafana or frontend build is needed. Keep sibling source files together when running explore.py. + +- **Timeline:** one conversation per lane, grouped by worker. Sampling spans HTTP submission to completion/failure; tool call spans the simulated delay **before that turn**; wait spans client queue entry to submission. Select a phase or use the request picker, including failed requests. Drag to zoom, or use Window/Position. +- **Engine Metrics:** 15 panels with client distributions in two-second windows and server samples at recorded scrape times. Select one exporter by endpoint; exporters are never combined. Click legends to hide series. +- **L1/L2:** GPU occupied = active + evictable; CPU occupied = host used. Each usage percentage divides by its own capacity; capacities are dashed. Hit fractions use deltas of prefill_effective_tokens_total: device, host and storage hits plus uncached input form the denominator; total hits include storage. +- Missing counters, changed label sets, resets and zero hit denominators are gaps. Explicit host-hit zero remains 0%. Occupancy aggregates matching label sets inside the selected exporter. Mamba usage shows the maximum reported rank; speculative acceptance length shows an unweighted rank mean. +- Older recordings without exact tool/wait timestamps mark those phases unavailable. Failed runs retain measured phases, token events and available TTFT. Interrupted runs are labeled incomplete. + +The output contains model identifiers, endpoint labels and recorded errors. Review it before sharing. Raw recordings and generated viewers should stay outside Git. + +## CPU tests + +```bash +uv run --no-project --with aiohttp --with transformers --with prometheus-client \ + python -m unittest discover -s benchmark/agentic-rollout/tests -v +``` + +See [the spec](spec/feature-00-session-hicache-latency.md) and [validation evidence](spec/evidence/feature-00-session-hicache-latency.md). diff --git a/benchmark/agentic-rollout/explore.py b/benchmark/agentic-rollout/explore.py new file mode 100644 index 000000000000..4ad7ce81c1a1 --- /dev/null +++ b/benchmark/agentic-rollout/explore.py @@ -0,0 +1,433 @@ +# /// script +# requires-python = ">=3.11" +# dependencies = ["prometheus-client"] +# /// +"""Build an offline Rollout Simulator and Explorer HTML from a recording.""" + +import argparse +import json +import math +import statistics +from pathlib import Path + +from metrics import ( + counter_rate, +) +from metrics import gauge as metric_gauge +from metrics import ( + hit_percentages, + metric_values, + occupancy, + values, +) + + +def read_rows(path): + if not path.exists(): + return [] + rows = [] + for line in path.read_text().splitlines(): + if line.strip(): + try: + rows.append(json.loads(line)) + except json.JSONDecodeError: + # Interrupted writes remain visible as recording errors. + rows.append({"error": "Incomplete JSONL record"}) + return rows + + +def panels_for(reqs, scrapes, origin, duration): + series = [] + parsed = [] + for row in sorted(scrapes, key=lambda r: r["timestamp"]): + try: + samples = metric_values(row.get("text", "")) if not row.get("error") else {} + except ValueError: + samples = {} + parsed.append((row["timestamp"] - origin, samples)) + + def gauge(name, mult=1): + return [ + [t, None if (v := metric_gauge(ss, name)) is None else v * mult] + for t, ss in parsed + ] + + def rate(name): + return [ + [t1, counter_rate(s0, s1, "sglang:" + name, t1 - t0)] + for (t0, s0), (t1, s1) in zip(parsed, parsed[1:]) + ] + + cache_hits = [[], [], []] + for (t0, s0), (t1, s1) in zip(parsed, parsed[1:]): + for points, value in zip( + cache_hits, hit_percentages(s0, s1) if t1 > t0 else [None] * 3 + ): + points.append([t1, value]) + l1_occupied, l1_capacity, l2_occupied, l2_capacity = [], [], [], [] + l1_usage, l2_usage = [], [] + for t, samples in parsed: + active, evictable, capacity, usage1 = occupancy( + samples, ("kv_used_tokens", "kv_evictable_tokens", "max_total_num_tokens") + ) + host, host_capacity, usage2 = occupancy( + samples, ("hicache_host_used_tokens", "hicache_host_total_tokens") + ) + l1_occupied.append([t, active + evictable if active is not None else None]) + l1_capacity.append([t, metric_gauge(samples, "max_total_num_tokens")]) + l2_occupied.append([t, metric_gauge(samples, "hicache_host_used_tokens")]) + l2_capacity.append([t, metric_gauge(samples, "hicache_host_total_tokens")]) + l1_usage.append([t, usage1]) + l2_usage.append([t, usage2]) + + bins = [(i, min(i + 2, duration)) for i in range(0, math.ceil(duration), 2)] + + def distribution(observations): + buckets = [[] for _ in bins] + for elapsed, value in observations: + if 0 <= elapsed < duration: + buckets[int(elapsed // 2)].append(value) + groups = [ + ((a + b) / 2, sorted(values)) for (a, b), values in zip(bins, buckets) + ] + return [ + { + "name": name, + "points": [ + [ + t, + ( + ( + statistics.mean(v) + if q is None + else v[math.ceil(q * len(v)) - 1] + ) + if v + else None + ), + ] + for t, v in groups + ], + } + for name, q in (("avg", None), ("p50", 0.5), ("p90", 0.9), ("p99", 0.99)) + ] + + def dist(key, scale=1): + return distribution( + [ + ( + r.get( + "completed_at", + r.get( + "failed_at", + r.get("first_token_at", r.get("submitted_at", origin)), + ), + ) + - origin, + r[key] * scale, + ) + for r in reqs + if r.get(key) is not None + ] + ) + + def add(title, unit, source, items): + series.append({"title": title, "unit": unit, "source": source, "series": items}) + + def one(name, points, **style): + return {"name": name, "points": points, **style} + + add( + "QPS", + "req/s", + "Successful client completions / 2-second window", + [ + one( + "requests", + [ + [ + (a + b) / 2, + sum( + a <= r.get("completed_at", float("inf")) - origin < b + and not r.get("error") + for r in reqs + ) + / (b - a), + ] + for a, b in bins + ], + ) + ], + ) + add( + "Running And Queued Requests", + "requests", + "Server num_running_reqs / num_queue_reqs gauges", + [ + one("running", gauge("num_running_reqs")), + one("queued", gauge("num_queue_reqs")), + ], + ) + add( + "Input Length", + "tokens", + "Client context, grouped by completion/failure time", + dist("context_tokens"), + ) + reqs = [ + {**r, "output_tokens": (r.get("meta_info") or {}).get("completion_tokens")} + for r in reqs + ] + add( + "Output Length", + "tokens", + "Observed output token counts; includes partial requests", + dist("output_tokens"), + ) + add( + "TTFT", + "ms", + "Client submit to first output; includes partial requests with measured TTFT", + dist("ttft_s", 1000), + ) + # Report the distribution of observed per-token stream intervals, not GPU timing. + itl = [] + for r in reqs: + for (t0, n0), (t1, n1) in zip(r.get("events", []), r.get("events", [])[1:]): + if n1 > n0: + itl.append( + (r["submitted_at"] + t1 - origin, (t1 - t0) / (n1 - n0) * 1000) + ) + add( + "ITL", + "ms", + "Client stream intervals / received token increments; not kernel timing", + distribution(itl), + ) + add( + "Prompt / Input Throughput", + "tokens/s", + "Server counter deltas / scrape interval; cached sums all tiers", + [ + one("prompt", rate("prompt_tokens_total")), + one("cached", rate("cached_tokens_total")), + ], + ) + add( + "Output Throughput", + "tokens/s", + "Server generation counter deltas / scrape interval", + [one("total", rate("generation_tokens_total"))], + ) + add( + "Cache Hit Rate", + "%", + "Counter deltas from prefill_effective_tokens_total; each tier / all prompt tokens. Total includes storage hits.", + [ + one("L1 GPU", cache_hits[0], color=0), + one("L2 CPU", cache_hits[1], color=1), + one("Total", cache_hits[2], color=2, dashed=True), + ], + ) + add( + "E2E Request Latency", + "ms", + "Client submit to completed stream", + dist("latency_s", 1000), + ) + add( + "KV Usage", + "%", + "L1: (active + evictable) / GPU capacity. L2: host used / host capacity.", + [one("L1 GPU", l1_usage, color=0), one("L2 CPU", l2_usage, color=1)], + ) + add( + "KV Tokens", + "tokens", + "L1 occupied = active + evictable GPU tokens. L2 occupied = host used tokens. Dashed lines show capacity.", + [ + one("L1 occupied", l1_occupied, color=0), + one("L1 capacity", l1_capacity, color=0, dashed=True), + one("L2 occupied", l2_occupied, color=1), + one("L2 capacity", l2_capacity, color=1, dashed=True), + ], + ) + add( + "Mamba Usage", + "%", + "Maximum server mamba_usage across reported ranks", + [ + one( + "usage", + [ + [ + t, + ( + None + if not (v := list(values(ss, "mamba_usage").values())) + else max(v) * 100 + ), + ] + for t, ss in parsed + ], + ) + ], + ) + add( + "Mamba Tokens", + "tokens", + "Server Mamba token gauges", + [ + one("used", gauge("mamba_used_tokens")), + one("available", gauge("mamba_available_tokens")), + one("evictable", gauge("mamba_evictable_tokens")), + ], + ) + add( + "Spec Acceptance Length", + "tokens", + "Unweighted mean spec_accept_length across reported ranks", + [ + one( + "accepted", + [ + [ + t, + ( + statistics.mean(v) + if (v := list(values(ss, "spec_accept_length").values())) + else None + ), + ] + for t, ss in parsed + ], + ) + ], + ) + return series + + +def load_run(directory): + manifest = json.loads((directory / "manifest.json").read_text()) + requests = read_rows(directory / "requests.jsonl") + scrapes = read_rows(directory / "metrics.jsonl") + sessions = read_rows(directory / "sessions.jsonl") + origin = manifest["started_at"] + stamps = [origin] + [ + r[k] + for r in requests + scrapes + sessions + for k in ("completed_at", "failed_at", "submitted_at", "timestamp") + if isinstance(r.get(k), (int, float)) + ] + stamps.extend( + r["submitted_at"] + event[0] + for r in requests + if "submitted_at" in r + for event in r.get("events", []) + ) + duration = max(0.001, manifest.get("finished_at", max(stamps)) - origin) + info = manifest.get("server_info", {}) + args = manifest.get("arguments", {}) + server = info.get("server_args", info) + rows = [] + for r in requests: + meta = r.get("meta_info") or {} + rank = meta.get("dp_rank") + if rank is None: + rank = r.get("rank") + if rank is None and info.get("dp_size") == 1: + rank = 0 + phases, unavailable = [], [] + for kind, first, last in ( + ("Tool call", "tool_started_at", "tool_completed_at"), + ("Wait", "client_wait_started_at", "submitted_at"), + ( + "Sampling", + "submitted_at", + "completed_at" if "completed_at" in r else "failed_at", + ), + ): + if kind == "Tool call" and r.get("turn") == 0: + continue + if ( + r.get(first) is not None + and r.get(last) is not None + and r[last] >= r[first] + ): + phases.append( + {"type": kind, "start": r[first] - origin, "end": r[last] - origin} + ) + else: + unavailable.append(kind) + rows.append( + { + "conversation": r.get("conversation", "unknown"), + "turn": r.get("turn", "?"), + "worker": rank, + "phases": phases, + "unavailable": unavailable, + "error": r.get("error"), + "ttft": r.get("ttft_s"), + "latency": r.get("latency_s"), + "wait": r.get("client_wait_s"), + "context": r.get("context_tokens"), + "output": meta.get("completion_tokens"), + "hits": meta.get("cached_tokens_details") or {}, + } + ) + exporters = {} + for scrape in scrapes: + if "timestamp" in scrape: + exporters.setdefault(scrape.get("url", "unknown"), []).append(scrape) + if not exporters: + exporters["No metrics recorded"] = [] + summary = { + "model": server.get("model_path", args.get("tokenizer", "Unknown model")), + "duration": duration, + "requests": len(requests), + "conversations": len({r["conversation"] for r in rows}), + "turns": args.get("turns", "?"), + "concurrency": args.get("concurrency", "?"), + "page_size": server.get("page_size", "?"), + "status": ( + manifest.get("status", "incomplete") + if manifest.get("finished_at") + else "incomplete" + ), + "errors": sum(bool(r.get("error")) for r in requests + sessions + scrapes), + "error": manifest.get("error"), + } + return { + "summary": summary, + "rows": rows, + "exporters": { + url: panels_for(requests, records, origin, duration) + for url, records in exporters.items() + }, + } + + +def build(directory, output): + data = load_run(directory) + assets = Path(__file__).parent + encoded = json.dumps(data, allow_nan=False).replace("<", r"\u003c") + html = (assets / "viewer.html").read_text() + html = html.replace("__CSS__", (assets / "viewer.css").read_text()) + html = html.replace("__JS__", (assets / "viewer.js").read_text()) + html = html.replace("__REAL_DATA__", encoded) + output.write_text(html) + return data + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("directory", type=Path, help="Recorded run directory") + parser.add_argument( + "--output", + type=Path, + default=Path("rollout.html"), + help="Self-contained HTML output", + ) + args = parser.parse_args() + build(args.directory, args.output) + print(args.output) diff --git a/benchmark/agentic-rollout/metrics.py b/benchmark/agentic-rollout/metrics.py new file mode 100644 index 000000000000..6a983cf2782b --- /dev/null +++ b/benchmark/agentic-rollout/metrics.py @@ -0,0 +1,96 @@ +"""Prometheus parsing and label-aware cache measurements.""" + +import math + +from prometheus_client.parser import text_string_to_metric_families + + +def metric_values(text): + return { + (sample.name, tuple(sorted(sample.labels.items()))): sample.value + for family in text_string_to_metric_families(text) + for sample in family.samples + if math.isfinite(sample.value) + } + + +def counter_rate(previous, current, name, seconds): + keys = {key for key in previous.keys() | current.keys() if key[0] == name} + if not keys or seconds <= 0: + return None + # Missing series and counter resets are gaps, not zero or negative rates. + if any( + k not in previous + or k not in current + or not math.isfinite(previous[k]) + or not math.isfinite(current[k]) + or current[k] < previous[k] + for k in keys + ): + return None + return sum(current[k] - previous[k] for k in keys) / seconds + + +def values(samples, name): + return { + labels: value + for (metric, labels), value in samples.items() + if metric == "sglang:" + name + } + + +def gauge(samples, name): + items = values(samples, name) + return ( + sum(items.values()) + if items and all(math.isfinite(v) for v in items.values()) + else None + ) + + +def percent(used, capacity): + return ( + 100 * used / capacity + if used is not None and capacity is not None and capacity > 0 + else None + ) + + +def occupancy(samples, names): + tiers = [values(samples, name) for name in names] + if not all(tiers) or any(t.keys() != tiers[0].keys() for t in tiers): + return [None] * (len(names) + 1) + totals = [sum(t.values()) for t in tiers] + if not all(math.isfinite(v) for v in totals): + return [None] * (len(names) + 1) + return [*totals, percent(sum(totals[:-1]), totals[-1])] + + +def hit_percentages(before, after): + old = values(before, "prefill_effective_tokens_total") + new = values(after, "prefill_effective_tokens_total") + modes = ("device_hit", "host_hit", "storage_hit", "input") + if not old or old.keys() != new.keys(): + return [None] * 3 + workers = {} + for key, value in new.items(): + labels = dict(key) + mode = labels.pop("mode", None) + if mode not in modes or not math.isfinite(value) or not math.isfinite(old[key]): + return [None] * 3 + delta = value - old[key] + if delta < 0: + return [None] * 3 + workers.setdefault(tuple(sorted(labels.items())), {})[mode] = delta + if any(set(v) != set(modes) for v in workers.values()): + return [None] * 3 + deltas = {mode: sum(v[mode] for v in workers.values()) for mode in modes} + denominator = sum(deltas.values()) + return [ + percent(v, denominator) + for v in ( + deltas["device_hit"], + deltas["host_hit"], + denominator - deltas["input"], + ) + ] diff --git a/benchmark/agentic-rollout/plot.py b/benchmark/agentic-rollout/plot.py new file mode 100644 index 000000000000..de6f64a18a65 --- /dev/null +++ b/benchmark/agentic-rollout/plot.py @@ -0,0 +1,277 @@ +# /// script +# requires-python = ">=3.11" +# dependencies = ["matplotlib", "prometheus-client"] +# /// +"""Plot one or more synthetic-session result directories; no running server needed.""" + +import argparse +import csv +import json +import math +import os +from collections import defaultdict +from pathlib import Path + +from metrics import counter_rate, metric_values + + +def read_rows(path): + with path.open() as file: + for line in file: + if line.strip(): + yield json.loads(line) + + +def percentile(values, q): + values = sorted(v for v in values if v is not None) + return values[max(0, math.ceil(q * len(values)) - 1)] if values else None + + +def cache_hit(rows, source): + values = [ + ( + r["meta_info"].get("cached_tokens") + if source == "total" + else (r["meta_info"].get("cached_tokens_details") or {}).get(source) + ) + for r in rows + ] + total = sum(r["meta_info"]["prompt_tokens"] for r in rows) + return sum(values) / total if total and all(v is not None for v in values) else None + + +def summarize(rows): + return { + "requests": len(rows), + "ttft_p50_s": percentile([r["ttft_s"] for r in rows], 0.5), + "ttft_p95_s": percentile([r["ttft_s"] for r in rows], 0.95), + "avg_token_time_s": ( + sum(r["avg_token_time_s"] for r in rows) / len(rows) + if rows and all(r["avg_token_time_s"] is not None for r in rows) + else None + ), + "context_tokens": ( + sum(r["context_tokens"] for r in rows) / len(rows) if rows else None + ), + **{ + f"{source}_hit": cache_hit(rows, source) + for source in ("total", "device", "host") + }, + } + + +def export_csv(path, rows): + if not rows: + return + with path.open("w") as file: + writer = csv.DictWriter(file, fieldnames=list(rows[0])) + writer.writeheader() + writer.writerows(rows) + + +def analyze(directory, window): + manifest = json.loads((directory / "manifest.json").read_text()) + all_rows = list(read_rows(directory / "requests.jsonl")) + rows = [r for r in all_rows if "error" not in r] + start, finish = manifest["started_at"], manifest["finished_at"] + windows, turns, rates = defaultdict(list), defaultdict(list), defaultdict(int) + for row in rows: + windows[int((row["first_token_at"] - start) // window)].append(row) + turns[row["turn"]].append(row) + # Attribute received token increments to their actual arrival windows. + previous_count = 0 + for offset, count in row["events"]: + rates[int((row["submitted_at"] + offset - start) // window)] += ( + count - previous_count + ) + previous_count = count + by_time = [] + for index in range(int((finish - start) // window) + 1): + duration = min(window, finish - start - index * window) + if duration <= 0: + continue + by_time.append( + { + "elapsed_s": index * window, + **summarize(windows[index]), + "output_tokens_s": rates[index] / duration, + } + ) + by_turn = [{"turn": t, **summarize(group)} for t, group in sorted(turns.items())] + export_csv(directory / "windows.csv", by_time) + export_csv(directory / "turns.csv", by_turn) + summary = { + "status": manifest["status"], + "failed_requests": len(all_rows) - len(rows), + **summarize(rows), + "output_tokens_s": sum(rates.values()) / (finish - start), + } + (directory / "summary.json").write_text(json.dumps(summary, indent=2) + "\n") + return manifest, by_time, by_turn + + +def plot(directories, output, window): + import matplotlib + + matplotlib.use("Agg") + import matplotlib.pyplot as plt + + figure, axes = plt.subplots(4, 2, figsize=(15, 15), constrained_layout=True) + axes = axes.ravel() + titles = [ + "TTFT (seconds)", + "Received output tokens / second", + "Average time per output token (s)", + "Token-weighted cache hit", + "Running / queued requests (per exporter)", + "Cache occupancy (per exporter)", + "Cache transfer rates (tokens/s)", + "Mean context tokens", + ] + turn_figure, turn_axes = plt.subplots( + 1, 2, figsize=(13, 5), constrained_layout=True + ) + configs = {} + common = Path(os.path.commonpath([p.resolve().parent for p in directories])) + colors = plt.rcParams["axes.prop_cycle"].by_key()["color"] + for run_index, directory in enumerate(directories): + color = colors[run_index % len(colors)] + manifest, by_time, by_turn = analyze(directory, window) + label = str(directory.resolve().relative_to(common)) + ( + " [FAILED]" if manifest["status"] != "completed" else "" + ) + configs[label] = { + "arguments": manifest["arguments"], + "server_info": manifest["server_info"], + } + x = [r["elapsed_s"] for r in by_time] + for axis, key, suffix in [ + (0, "ttft_p50_s", "p50"), + (0, "ttft_p95_s", "p95"), + (1, "output_tokens_s", ""), + (2, "avg_token_time_s", ""), + (3, "total_hit", "total"), + (3, "device_hit", "GPU"), + (3, "host_hit", "CPU"), + (7, "context_tokens", ""), + ]: + values = [r[key] for r in by_time] + if any(v is not None for v in values): + axes[axis].plot( + x, + values, + color=color, + linestyle={"p50": "--", "GPU": "--", "CPU": ":"}.get(suffix, "-"), + marker=".", + label=f"{label} {suffix}", + ) + for axis, key in [(0, "ttft_p95_s"), (1, "context_tokens")]: + turn_axes[axis].plot( + [r["turn"] for r in by_turn], + [r[key] for r in by_turn], + color=color, + marker=".", + label=label, + ) + series = defaultdict(list) + previous = {} + for row in read_rows(directory / "metrics.jsonl"): + url = row["url"] + if "error" in row: + previous.pop(url, None) + continue + current = metric_values(row["text"]) + elapsed = row["timestamp"] - manifest["started_at"] + for name in ( + "sglang:num_running_reqs", + "sglang:num_queue_reqs", + "sglang:token_usage", + ): + values = [ + value for (metric, _), value in current.items() if metric == name + ] + if values: + value = max(values) if "usage" in name else sum(values) + series[(5 if "usage" in name else 4, url, name)].append( + (elapsed, value) + ) + host = [] + for (name, labels), used in current.items(): + if name == "sglang:hicache_host_used_tokens": + total = current.get(("sglang:hicache_host_total_tokens", labels)) + if total is not None and total > 0: + host.append(used / total) + if host: + series[(5, url, "host_usage")].append((elapsed, max(host))) + if url in previous: + stamp, before = previous[url] + names = { + name + for name, _ in current + if name.endswith("_total") + and any(word in name for word in ("restore", "backup", "load")) + and "token" in name + } + for name in sorted(names): + value = counter_rate( + before, current, name, row["timestamp"] - stamp + ) + series[(6, url, name)].append((elapsed, value)) + previous[url] = row["timestamp"], current + urls = { + url: index for index, url in enumerate(sorted({key[1] for key in series})) + } + for (axis, url, name), points in series.items(): + if any(v is not None for _, v in points): + axes[axis].plot( + [t for t, _ in points], + [v for _, v in points], + color=color, + linestyle=( + "--" + if name in ("sglang:num_queue_reqs", "host_usage") + or "backup" in name + else "-" + ), + marker=".", + label=f"{label} exporter {urls[url]} {name.removeprefix('sglang:')}", + ) + for axis, title in zip(axes, titles): + axis.set(title=title, xlabel="Elapsed seconds") + axis.grid(alpha=0.2) + if axis.lines: + axis.legend(fontsize=6) + else: + axis.text(0.5, 0.5, "Unavailable", ha="center", transform=axis.transAxes) + for axis, title in zip(turn_axes, ("TTFT p95 (seconds)", "Mean context tokens")): + axis.set(title=title, xlabel="Turn") + axis.grid(alpha=0.2) + if axis.lines: + axis.legend(fontsize=7) + figure.savefig(output, dpi=160) + turn_figure.savefig(output.with_name(output.stem + "-turns.png"), dpi=160) + output.with_suffix(".configs.json").write_text(json.dumps(configs, indent=2) + "\n") + plt.close("all") + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "directories", nargs="+", type=Path, help="Run result directories to compare" + ) + parser.add_argument( + "--output", + type=Path, + default=Path("synthetic-sessions.png"), + help="PNG path; also writes a turn plot and configuration sidecar", + ) + parser.add_argument( + "--window", + type=float, + default=30, + help="Seconds per time window for request statistics", + ) + args = parser.parse_args() + if args.window <= 0: + parser.error("--window must be positive") + plot(args.directories, args.output, args.window) diff --git a/benchmark/agentic-rollout/simulate.py b/benchmark/agentic-rollout/simulate.py new file mode 100644 index 000000000000..6301cdf8558c --- /dev/null +++ b/benchmark/agentic-rollout/simulate.py @@ -0,0 +1,443 @@ +# /// script +# requires-python = ">=3.11" +# dependencies = ["aiohttp", "transformers"] +# /// +"""Synthetic multi-turn HTTP workload. See README.md for examples.""" + +import argparse +import asyncio +import hashlib +import importlib.metadata +import json +import random +import sys +import time +import uuid +from pathlib import Path + +import aiohttp + + +def write_json(path, value): + path.write_text(json.dumps(value, indent=2, sort_keys=True) + "\n") + + +def record(file, value): + file.write(json.dumps(value, sort_keys=True) + "\n") + file.flush() + + +def synthetic_tokens(tokenizer, seed, conversation, turn, length): + text = ( + f"Session {seed}/{conversation}. Tool result {turn}: " + "The local search returned a synthetic measurement. " + "Read the result and continue the investigation. " + ) + ids = tokenizer.encode(text, add_special_tokens=False) + if not ids: + raise ValueError("Tokenizer produced no input tokens") + return (ids * ((length + len(ids) - 1) // len(ids)))[:length] + + +async def sse_events(content): + # aiohttp iterates complete lines, even when TCP splits a UTF-8 character. + data = [] + async for raw in content: + line = raw.decode("utf-8").rstrip("\r\n") + if not line: + if data: + yield "\n".join(data) + data.clear() + elif line.startswith("data:"): + data.append(line[5:].lstrip(" ")) + if data: + raise ValueError("Truncated SSE frame") + + +async def check_response(response): + if response.status >= 400: + detail = (await response.text())[:2000] + raise ValueError(f"HTTP {response.status} {response.url}: {detail}") + + +async def json_request(http, method, url, **kwargs): + async with http.request(method, url, **kwargs) as response: + await check_response(response) + return await response.json() + + +async def generate(http, url, payload, row, incremental, dp_size=None): + start = time.perf_counter() + row.update(submitted_at=time.time(), events=[]) + count, ids, meta, done = 0, [], {}, False + async with http.post(url + "/generate", json=payload) as response: + await check_response(response) + async for event in sse_events(response.content): + if event == "[DONE]": + done = True + break + message = json.loads(event) + if "error" in message: + raise ValueError(str(message["error"])) + meta = message["meta_info"] + current = meta["completion_tokens"] + if current < count: + raise ValueError("Completion count moved backwards") + if current > count: + row["events"].append([time.perf_counter() - start, current]) + if not count: + row["first_token_at"] = time.time() + row["ttft_s"] = row["events"][-1][0] + count = current + if "output_ids" in message: + if incremental: + ids.extend(message["output_ids"]) + else: + ids = message["output_ids"] + row["meta_info"] = meta + row.update(completed_at=time.time(), latency_s=time.perf_counter() - start) + if ( + not done + or not count + or (meta.get("finish_reason") or {}).get("type") != "length" + ): + raise ValueError( + f"Incomplete or aborted generation: {meta.get('finish_reason')}" + ) + expected = payload["sampling_params"]["max_new_tokens"] + if count != expected or len(ids) != expected: + raise ValueError( + f"Expected {expected} tokens, got count={count}, IDs={len(ids)}" + ) + if meta["prompt_tokens"] != row["context_tokens"]: + raise ValueError(f"Wrong context length: {meta['prompt_tokens']}") + if ( + row["rank"] is not None + and meta.get("dp_rank") != row["rank"] + and not (dp_size == 1 and row["rank"] == 0 and meta.get("dp_rank") is None) + ): + raise ValueError(f"Wrong DP rank: {meta.get('dp_rank')}") + events = row["events"] + row["avg_token_time_s"] = ( + (events[-1][0] - events[0][0]) / (count - events[0][1]) + if count > events[0][1] + else None + ) + row["output_sha256"] = hashlib.sha256(json.dumps(ids).encode()).hexdigest() + return meta["id"], ids + + +async def scrape_metrics(http, url, file, stopped): + while not stopped.is_set(): + started = time.monotonic() + row = {"timestamp": time.time(), "url": url} + try: + async with http.get(url) as response: + response.raise_for_status() + row["text"] = await response.text() + except (aiohttp.ClientError, OSError, asyncio.TimeoutError) as exc: + row["error"] = str(exc) + row["duration_s"] = time.monotonic() - started + record(file, row) + try: + await asyncio.wait_for(stopped.wait(), max(0.01, 1 - row["duration_s"])) + except asyncio.TimeoutError: + pass + + +def max_context_tokens(args): + return ( + args.initial_tokens + + (args.turns - 1) * (args.tool_tokens + args.output_tokens) + + args.output_tokens + ) + + +async def conversation( + http, tokenizer, args, info, slot, semaphore, requests, sessions +): + rank = slot % info["dp_size"] if not args.disable_dp_sticky_routing else None + rng = random.Random(f"{args.seed}/{slot}") + await asyncio.sleep(args.start_spread * slot / args.conversations) + sid, rid, history = None, None, [] + identity = {"conversation": slot, "rank": rank} + try: + if args.mode != "full-history": + sid = uuid.uuid4().hex + opened = await json_request( + http, + "POST", + args.base_url + "/open_session", + json={ + "session_id": sid, + "streaming": args.mode == "streaming", + # Required by older servers, but unused by session execution. + "capacity_of_str_len": 1000, + }, + ) + if opened != sid: + raise ValueError(f"Unexpected open response: {opened}") + record( + sessions, + {**identity, "event": "open", "id": sid, "timestamp": time.time()}, + ) + for turn in range(args.turns): + delay = rng.uniform(*args.tool_delay) if turn else 0 + tool_started_at = time.time() + delay_start = time.perf_counter() + await asyncio.sleep(delay) + actual_delay = time.perf_counter() - delay_start + tool_completed_at = time.time() + length = args.tool_tokens if turn else args.initial_tokens + chunk = synthetic_tokens(tokenizer, args.seed, slot, turn, length) + history.extend(chunk) + row = { + **identity, + "turn": turn, + "context_tokens": len(history), + "tool_delay_s": delay, + "tool_started_at": tool_started_at if turn else None, + "tool_completed_at": tool_completed_at if turn else None, + "actual_tool_delay_s": actual_delay, + "input_tokens": length, + } + payload = { + "input_ids": history if args.mode == "full-history" else chunk, + "stream": True, + "sampling_params": { + "temperature": 0, + "ignore_eos": True, + "max_new_tokens": args.output_tokens, + }, + } + if sid is not None: + payload["session_params"] = {"id": sid, "rid": rid} + if rank is not None: + payload["routed_dp_rank"] = rank + row["client_wait_started_at"] = time.time() + waiting = time.perf_counter() + try: + async with semaphore: + row["client_wait_s"] = time.perf_counter() - waiting + rid, output = await generate( + http, + args.base_url, + payload, + row, + info.get("incremental_streaming_output", False), + dp_size=info.get("dp_size"), + ) + history.extend(output) + except BaseException as exc: + row["failed_at"] = time.time() + row["error"] = f"{type(exc).__name__}: {exc}" + raise + finally: + record(requests, row) + finally: + if sid is not None: + event = { + **identity, + "event": "close", + "id": sid, + "timestamp": time.time(), + } + try: + async with http.post( + args.base_url + "/close_session", + json={"session_id": sid}, + timeout=aiohttp.ClientTimeout(total=10), + ) as response: + response.raise_for_status() + except (aiohttp.ClientError, OSError, asyncio.TimeoutError) as exc: + event["error"] = str(exc) + raise + finally: + record(sessions, event) + + +async def run(args, tokenizer): + args.output_dir.mkdir(parents=True, exist_ok=False) + manifest = { + "arguments": { + k: str(v) if isinstance(v, Path) else v for k, v in vars(args).items() + }, + "started_at": time.time(), + "status": "running", + "python": sys.version, + "script_sha256": hashlib.sha256(Path(__file__).read_bytes()).hexdigest(), + "dependencies": { + n: importlib.metadata.version(n) for n in ("aiohttp", "transformers") + }, + } + stopped = asyncio.Event() + manifest_path = args.output_dir / "manifest.json" + write_json(manifest_path, manifest) + try: + async with ( + aiohttp.ClientSession( + timeout=aiohttp.ClientTimeout(total=args.timeout), + connector=aiohttp.TCPConnector(limit=0), + ) as http, + aiohttp.ClientSession( + timeout=aiohttp.ClientTimeout(total=10) + ) as metrics_http, + ): + info = await json_request(http, "GET", args.base_url + "/server_info") + manifest["server_info"] = info + context = max_context_tokens(args) + if info.get("context_length") and context > info["context_length"]: + raise ValueError( + f"Workload requires {context} tokens; server context is too small" + ) + if not args.disable_dp_sticky_routing and info.get("dp_size", 0) < 1: + raise ValueError("DP sticky routing requires server_info.dp_size") + write_json(manifest_path, manifest) + semaphore = asyncio.Semaphore(args.concurrency) + with ( + (args.output_dir / "requests.jsonl").open("w") as requests, + (args.output_dir / "sessions.jsonl").open("w") as sessions, + (args.output_dir / "metrics.jsonl").open("w") as metrics, + ): + scrapers = [ + asyncio.create_task( + scrape_metrics(metrics_http, url, metrics, stopped) + ) + for url in (args.metrics_url or [args.base_url + "/metrics"]) + ] + try: + async with asyncio.TaskGroup() as group: + for slot in range(args.conversations): + group.create_task( + conversation( + http, + tokenizer, + args, + info, + slot, + semaphore, + requests, + sessions, + ) + ) + finally: + stopped.set() + await asyncio.gather(*scrapers) + manifest["status"] = "completed" + except BaseException as exc: + manifest.update(status="failed", error=repr(exc)) + raise + finally: + manifest["finished_at"] = time.time() + write_json(manifest_path, manifest) + print(f"{manifest['status']}: {args.output_dir}", flush=True) + + +def parse_args(argv=None): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--base-url", default="http://127.0.0.1:30000", help="SGLang server URL" + ) + parser.add_argument( + "--tokenizer", + required=True, + help="Model ID or tokenizer path matching the server", + ) + parser.add_argument( + "--trust-remote-code", action="store_true", help="Allow custom tokenizer code" + ) + parser.add_argument( + "--mode", + choices=("full-history", "ordinary", "streaming"), + default="ordinary", + help="Send full history or use ordinary/streaming sessions", + ) + parser.add_argument( + "--conversations", + type=int, + default=8, + help="Number of independent conversations", + ) + parser.add_argument( + "--concurrency", + type=int, + default=8, + help="Maximum in-flight generation requests", + ) + parser.add_argument( + "--turns", + type=int, + default=8, + help="Generation turns per conversation", + ) + parser.add_argument( + "--initial-tokens", + type=int, + default=2048, + help="Tokens in each initial prompt", + ) + parser.add_argument( + "--tool-tokens", + type=int, + default=256, + help="Synthetic tool-result tokens appended each turn", + ) + parser.add_argument( + "--output-tokens", + type=int, + default=128, + help="Exact number of tokens generated per turn", + ) + parser.add_argument( + "--tool-delay", + type=float, + nargs=2, + default=[1, 3], + metavar=("MIN", "MAX"), + help="Range of simulated tool waits between turns, in seconds", + ) + parser.add_argument( + "--start-spread", + type=float, + default=5, + help="Seconds over which conversation starts are staggered", + ) + parser.add_argument( + "--seed", type=int, default=1, help="Seed for synthetic inputs and tool delays" + ) + parser.add_argument( + "--disable-dp-sticky-routing", + action="store_true", + help="Disable default routing of conversation i to data-parallel rank i %% dp_size", + ) + parser.add_argument( + "--timeout", + type=float, + default=600, + help="HTTP request timeout in seconds", + ) + parser.add_argument( + "--metrics-url", + action="append", + help="Metrics endpoint; repeat for distinct exporters (default: BASE_URL/metrics)", + ) + parser.add_argument( + "--output-dir", + type=Path, + required=True, + help="New directory for request records, metrics, and run settings", + ) + args = parser.parse_args(argv) + args.base_url = args.base_url.rstrip("/") + return args + + +if __name__ == "__main__": + from transformers import AutoTokenizer + + args = parse_args() + tokenizer = AutoTokenizer.from_pretrained( + args.tokenizer, + trust_remote_code=args.trust_remote_code, + ) + asyncio.run(run(args, tokenizer)) diff --git a/benchmark/agentic-rollout/spec/README.md b/benchmark/agentic-rollout/spec/README.md new file mode 100644 index 000000000000..6cd9c9700dfc --- /dev/null +++ b/benchmark/agentic-rollout/spec/README.md @@ -0,0 +1,12 @@ +Status: implemented + +# Rollout Simulator and Explorer + +**TL;DR:** A portable client makes growing-conversation cache experiments repeatable. + +- [feature-00-session-hicache-latency.md](feature-00-session-hicache-latency.md): workload, measurements, and validation contract. +- [evidence/feature-00-session-hicache-latency.md](evidence/feature-00-session-hicache-latency.md): exact validation commands and measured results. + +## Boundaries + +This spec covers the standalone benchmark, not the serving engine. diff --git a/benchmark/agentic-rollout/spec/evidence/feature-00-session-hicache-latency.md b/benchmark/agentic-rollout/spec/evidence/feature-00-session-hicache-latency.md new file mode 100644 index 000000000000..6b5d819ef731 --- /dev/null +++ b/benchmark/agentic-rollout/spec/evidence/feature-00-session-hicache-latency.md @@ -0,0 +1,52 @@ +Status: implemented + +# Validation evidence + +## Packaging checks — 2026-09-17 + +17 CPU tests pass: all three request modes and both stream formats, default DP routing, verified-single-worker omitted ranks, strict multiworker ranks, context/output validation, aborted/truncated streams, timeouts and cleanup. Measurement tests cover occupancy, per-label resets, missing/idle counters, token-weighted denominators, separate exporters, old/interrupted recordings, partial failed requests and safe self-contained HTML. + +~~~bash +uv run --no-project --with aiohttp --with transformers --with prometheus-client \ + python -m unittest discover -s benchmark/agentic-rollout/tests -v +~~~ + +Viewers were rebuilt from the two saved Qwen runs below, each with 16 conversations and 96 requests. Both tabs, all 16 lanes, zoom, request selection, legends and static plots were checked. Embedded-asset tests pass, and the browser renders with external resource loading blocked by the page policy. Direct file-URL navigation was blocked by the browser automation policy, so that path was not manually verified. Packaging does not rerun GPU inference. Raw recordings, generated HTML and machine-specific orchestration stay outside this PR. + +## Saved Qwen comparison — 2026-09-17 + +One H200, Qwen/Qwen3-1.7B, a downstream SGLang serving build at revision 41da06adca698c0032f75000445087f0922dbe43. These saved runs are not GPU validation against current upstream main. Ordinary sessions, session-aware cache, HiCache ratio 1.6, write-through, direct I/O, layer-first layout and resolved FA3 attention. One fresh-server run per page size. + +Launch command, with executable and port normalized for portability: + +~~~bash +python -m sglang.launch_server \ + --model-path Qwen/Qwen3-1.7B --host 127.0.0.1 --port 30000 \ + --enable-session-radix-cache --enable-hierarchical-cache \ + --hicache-ratio 1.6 --hicache-write-policy write_through \ + --hicache-io-backend direct --hicache-mem-layout layer_first \ + --mem-fraction-static 0.5 --max-total-tokens 32768 --context-length 8192 \ + --cuda-graph-max-bs-decode 8 --enable-metrics --stream-interval 1 \ + --page-size "$PAGE" + +uv run benchmark/agentic-rollout/simulate.py \ + --tokenizer Qwen/Qwen3-1.7B --mode ordinary \ + --conversations 16 --concurrency 16 --turns 6 \ + --initial-tokens 1024 --tool-tokens 256 --output-tokens 128 \ + --tool-delay 1 3 --start-spread 1 --seed 101 \ + --disable-dp-sticky-routing --output-dir "results/page$PAGE" +~~~ + +The recorded client disabled routing hints because this single-worker server omitted its response rank. The packaged client now accepts that omission with default sticky routing only for verified DP=1; this adjustment is CPU-tested, not GPU-rerun. + +| Measurement | Page 1 | Page 64 | +| --- | ---: | ---: | +| Successful requests | 96/96 | 96/96 | +| GPU / CPU capacity (tokens) | 32,768 / 52,429 | 32,768 / 52,480 | +| Duration | 18.54 s | 14.96 s | +| Output throughput | 662.8 tokens/s | 821.1 tokens/s | +| Turns 4–5 TTFT p95 | 2,502.49 ms | 20.73 ms | +| Turns 4–5 mean time per output token | 10.56 ms | 2.33 ms | +| Last observed eviction / restore counters | 92,200 / 76,088 | 66,112 / 49,984 | + +Limitations: one short run per setting, no reverse-order repeat. Client inputs, delays and lengths matched, but server-generated random seeds differed; 85/96 output hashes matched. This validates the client and exposes cache pressure; it does not establish a general page-size speedup or a kernel-level cause. Counters are last scraped values and may miss the tail. diff --git a/benchmark/agentic-rollout/spec/feature-00-session-hicache-latency.md b/benchmark/agentic-rollout/spec/feature-00-session-hicache-latency.md new file mode 100644 index 000000000000..bec3dac8a8ff --- /dev/null +++ b/benchmark/agentic-rollout/spec/feature-00-session-hicache-latency.md @@ -0,0 +1,27 @@ +Status: implemented + +# Synthetic session workload + +**TL;DR:** A portable HTTP client compares full-history requests, ordinary sessions, and streaming sessions using independent synthetic conversations. Server launch settings control feature ablations. + +```text +generate -> seeded tool delay -> append synthetic result -> repeat +finish configured turns -> close +``` + +The client runs exactly `--conversations` conversations, each with `--turns` sequential generations. Each conversation uses fixed input/output lengths and a distinct prefix. Actual output token IDs remain in its history. Conversations run concurrently, each with sequential turns; tool waits release concurrency slots. Client-side DP sticky routing is enabled by default: every turn of conversation `i` routes to rank `i % dp_size`. `--disable-dp-sticky-routing` omits the routing hint. All modes stream responses for timing. + +The client records resolved server settings, its arguments/version, exact request timings, streamed token-count increments, output hashes, session events, and one-second raw metrics. Request errors stop the run and trigger bounded cleanup; evidence survives. Metrics errors remain visible. Missing measurements remain unavailable. + +Separate plots show time and turn number, TTFT p50/p95, observed output throughput, average token timing, token-weighted GPU/CPU/total cache hit, running/queued requests, occupancy, and available transfer rates. Independent metrics exporters remain separate. Several tokens arriving together are not individual token-gap samples. + +Cache-pressure claims require observed restores and measured capacity. Public smoke success does not establish slowdown reproduction. Exact commands and measured results belong in [evidence](evidence/feature-00-session-hicache-latency.md). + +## Boundaries + +Synthetic text and simulated tool delays only. No trainer, training-stack dependency, real tool execution, engine changes, server orchestration, or private deployment details in the shared client. Features are tested only when supported by the chosen server; no silent compatibility fallback. + + +## Explorer packaging (2026-09-17) + +Keep the client, plotter, shared Prometheus parser, viewer assets and tests in benchmark/agentic-rollout/. The explorer embeds all assets/data without a frontend dependency. Timeline contains only sampling, tool call and client wait using exact timestamps; missing old phases remain unavailable. Engine Metrics retains 15 panels with L1/L2 in KV Usage, KV Tokens and Cache Hit Rate. Exporters remain separately selectable. Missing/reset/idle counter intervals are gaps. Preserve failed-run status and partial measurements. Verify with CPU tests and both saved 16-conversation Qwen recordings; no new GPU run is required. diff --git a/benchmark/agentic-rollout/tests/test_explore.py b/benchmark/agentic-rollout/tests/test_explore.py new file mode 100644 index 000000000000..d38ba2a765a8 --- /dev/null +++ b/benchmark/agentic-rollout/tests/test_explore.py @@ -0,0 +1,173 @@ +"""Explorer checks use small recordings, without GPUs or external services.""" + +import json +import sys +import tempfile +import unittest +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) +import explore +from metrics import counter_rate, hit_percentages, metric_values, occupancy + + +def counters(device=0, host=0, storage=0, uncached=0, rank="0"): + return metric_values( + "\n".join( + f'sglang:prefill_effective_tokens_total{{mode="{mode}",dp_rank="{rank}"}} {value}' + for mode, value in zip( + ("device_hit", "host_hit", "storage_hit", "input"), + (device, host, storage, uncached), + ) + ) + ) + + +class MetricTests(unittest.TestCase): + def test_occupancy_includes_evictable_and_matches_labels(self): + samples = metric_values(""" +sglang:kv_used_tokens{rank="0"} 20 +sglang:kv_evictable_tokens{rank="0"} 30 +sglang:max_total_num_tokens{rank="0"} 100 +sglang:hicache_host_used_tokens{rank="0"} 80 +sglang:hicache_host_total_tokens{rank="0"} 200 +""") + names = ("kv_used_tokens", "kv_evictable_tokens", "max_total_num_tokens") + self.assertEqual(occupancy(samples, names), [20, 30, 100, 50]) + self.assertEqual( + occupancy( + samples, ("hicache_host_used_tokens", "hicache_host_total_tokens") + ), + [80, 200, 40], + ) + samples.update(metric_values('sglang:kv_used_tokens{rank="1"} 1')) + self.assertEqual(occupancy(samples, names), [None] * 4) + + def test_hits_include_storage_and_uncached_denominator(self): + self.assertEqual( + hit_percentages(counters(), counters(40, 20, 10, 30)), [40, 20, 70] + ) + self.assertEqual( + hit_percentages(counters(), counters(80, 0, 0, 20)), [80, 0, 80] + ) + self.assertEqual(hit_percentages(counters(), counters()), [None] * 3) + missing = counters(80, 0, 0, 20) + missing.pop(next(k for k in missing if dict(k[1])["mode"] == "host_hit")) + self.assertEqual(hit_percentages(counters(), missing), [None] * 3) + + def test_per_label_reset_cannot_be_hidden_by_another_worker(self): + before = counters(20) | counters(20, rank="1") + after = counters(19) | counters(100, rank="1") + self.assertEqual(hit_percentages(before, after), [None] * 3) + self.assertIsNone( + counter_rate(before, after, "sglang:prefill_effective_tokens_total", 1) + ) + self.assertIsNone( + counter_rate({}, after, "sglang:prefill_effective_tokens_total", 1) + ) + + def test_scrape_error_breaks_rates(self): + records = [ + {"timestamp": 0, "text": "sglang:generation_tokens_total 1"}, + {"timestamp": 1, "error": "timeout"}, + {"timestamp": 2, "text": "sglang:generation_tokens_total 9"}, + ] + panels = explore.panels_for([], records, 0, 3) + rate = next(p for p in panels if p["title"] == "Output Throughput") + self.assertEqual(rate["series"][0]["points"], [[1, None], [2, None]]) + + def test_nonfinite_missing_and_zero_capacity(self): + self.assertEqual(metric_values("sglang:kv_used_tokens NaN"), {}) + self.assertEqual(occupancy({}, ("a", "b")), [None] * 3) + self.assertEqual( + occupancy(metric_values("sglang:a 1\nsglang:b 0"), ("a", "b")), [1, 0, None] + ) + + +class RecordingTests(unittest.TestCase): + def test_old_failed_recording_and_offline_assets(self): + with tempfile.TemporaryDirectory() as tmp: + path = Path(tmp) + (path / "manifest.json").write_text( + json.dumps( + { + "started_at": 10, + "status": "failed", + "finished_at": 15, + "server_info": {"dp_size": 1}, + "arguments": { + "tokenizer": "" + }, + } + ) + ) + (path / "requests.jsonl").write_text( + json.dumps( + { + "conversation": 7, + "turn": 2, + "submitted_at": 11, + "failed_at": 14, + "ttft_s": 0.5, + "first_token_at": 11.5, + "events": [[0.5, 1], [1, 2]], + "error": "disconnected", + "meta_info": {"completion_tokens": 2}, + } + ) + + "\n" + ) + (path / "metrics.jsonl").write_text( + "\n".join( + json.dumps( + { + "timestamp": 12, + "url": url, + "text": f"sglang:num_running_reqs {n}", + } + ) + for url, n in (("one", 2), ("two", 3)) + ) + ) + output = path / "view.html" + data = explore.build(path, output) + row = data["rows"][0] + self.assertEqual(row["worker"], 0) + self.assertEqual(row["unavailable"], ["Tool call", "Wait"]) + self.assertEqual( + row["phases"], [{"type": "Sampling", "start": 1, "end": 4}] + ) + self.assertEqual(data["summary"]["errors"], 1) + self.assertEqual(set(data["exporters"]), {"one", "two"}) + for url, expected in (("one", 2), ("two", 3)): + panels = data["exporters"][url] + self.assertEqual(len(panels), 15) + running = next( + p for p in panels if p["title"] == "Running And Queued Requests" + ) + self.assertEqual(running["series"][0]["points"], [[2, expected]]) + ttft = next(p for p in panels if p["title"] == "TTFT") + self.assertTrue(any(v == 500 for _, v in ttft["series"][0]["points"])) + html = output.read_text() + self.assertNotIn(" + + diff --git a/benchmark/agentic-rollout/viewer.js b/benchmark/agentic-rollout/viewer.js new file mode 100644 index 000000000000..ac20f0304477 --- /dev/null +++ b/benchmark/agentic-rollout/viewer.js @@ -0,0 +1,483 @@ +const DATA = __REAL_DATA__; +const $ = (s) => document.querySelector(s), + NS = "http://www.w3.org/2000/svg", + COLORS = ["var(--sampling)", "var(--tool)", "var(--s3)", "var(--s4)"]; +let start = 0, + span = DATA.summary.duration, + selected = DATA.rows[0], + drag = null; +const total = span; +const lanes = [ + ...new Map( + DATA.rows.map((r) => [ + JSON.stringify([r.worker, r.conversation]), + { worker: r.worker, conversation: r.conversation }, + ]), + ).values(), +].sort( + (a, b) => + (a.worker ?? Infinity) - (b.worker ?? Infinity) || + String(a.conversation).localeCompare(String(b.conversation), undefined, { + numeric: true, + }), +); + +let metrics = Object.values(DATA.exporters)[0]; +$("#subtitle").textContent = + `${DATA.summary.model} · ${DATA.summary.requests} requests · ${DATA.summary.conversations} conversations × ${DATA.summary.turns} turns · concurrency ${DATA.summary.concurrency} · page ${DATA.summary.page_size} · ${total.toFixed(1)}s · ${DATA.summary.errors} errors`; +function elem(tag, attrs, parent) { + const e = document.createElementNS(NS, tag); + Object.entries(attrs).forEach(([k, v]) => e.setAttribute(k, v)); + parent.append(e); + return e; +} +function text(parent, x, y, s, extra = {}) { + const e = elem( + "text", + { x, y, fill: "var(--muted)", "font-size": 12, ...extra }, + parent, + ); + e.textContent = s; + return e; +} +function fmt(v) { + if (v === null || v === undefined) return "Unavailable"; + return Math.abs(v) >= 1000 + ? (v / 1000).toFixed(1) + "k" + : Math.abs(v) >= 10 + ? v.toFixed(1) + : v.toFixed(2); +} +function details() { + if (!selected) { + $("#selected-title").textContent = "No request records"; + return; + } + $("#request").value = String(DATA.rows.indexOf(selected)); + $("#selected-title").textContent = + `Conversation ${selected.conversation} · Turn ${selected.turn} · Worker ${selected.worker ?? "unknown"}`; + $("#availability").textContent = [ + selected.error ? "Failed: " + selected.error : "", + selected.unavailable.length + ? "Phase timestamps unavailable: " + selected.unavailable.join(", ") + : "", + ] + .filter(Boolean) + .join(" · "); + const fields = [ + ["Context", fmt(selected.context) + " tokens"], + ["Output", fmt(selected.output) + " tokens"], + [ + "Sampling", + fmt(selected.latency == null ? null : selected.latency * 1000) + " ms", + ], + ["TTFT", fmt(selected.ttft == null ? null : selected.ttft * 1000) + " ms"], + [ + "Client wait", + fmt(selected.wait == null ? null : selected.wait * 1000) + " ms", + ], + [ + "GPU / CPU hits", + `${selected.hits.device ?? "N/A"} / ${selected.hits.host ?? "N/A"} tokens`, + ], + ]; + $("#details").replaceChildren(); + for (const [label, value] of fields) { + const field = document.createElement("div"); + const name = document.createElement("span"); + const content = document.createElement("strong"); + name.textContent = label; + content.textContent = value; + field.append(name, content); + $("#details").append(field); + } +} +function drawTimeline() { + const svg = $("#timeline"); + if ($("#timeline-panel").hidden) return; + let w = svg.parentElement.clientWidth, + L = w < 500 ? 70 : 112, + R = 12, + h = + 102 + + lanes.length * 33 + + new Set(lanes.map((lane) => lane.worker)).size * 24; + const x = (t) => L + ((t - start) / span) * (w - L - R); + svg.replaceChildren(); + svg.setAttribute("viewBox", `0 0 ${w} ${h}`); + svg.style.width = "100%"; + svg.style.height = h + "px"; + let ticks = w < 500 ? 3 : 6; + for (let i = 0; i <= ticks; i++) { + let t = start + (span * i) / ticks, + px = x(t); + elem( + "line", + { x1: px, x2: px, y1: 25, y2: h - 30, stroke: "var(--line)" }, + svg, + ); + text(svg, px, 15, t.toFixed(span < 3 ? 2 : 1) + "s", { + "text-anchor": i === 0 ? "start" : i === ticks ? "end" : "middle", + }); + } + let groupIndex = -1; + for (let lane = 0; lane < lanes.length; lane++) { + let { conversation: c, worker } = lanes[lane]; + const startsGroup = lane === 0 || lanes[lane - 1].worker !== worker; + if (startsGroup) groupIndex++; + let y = 66 + lane * 33 + groupIndex * 24; + if (startsGroup) + text( + svg, + 0, + y - 5, + worker === null ? "Worker unknown" : "Worker " + worker, + { "font-size": 10 }, + ); + text(svg, 4, y + 14, "Convo " + c); + for (const r of DATA.rows.filter( + (v) => v.conversation === c && v.worker === worker, + )) { + for (const p of r.phases) { + let a = Math.max(start, p.start), + b = Math.min(start + span, p.end); + if (b <= a) continue; + let fill = + p.type === "Sampling" + ? "var(--sampling)" + : p.type === "Tool call" + ? "var(--tool)" + : "var(--wait)"; + const g = elem( + "g", + { + role: "button", + "aria-label": `Conversation ${c} turn ${r.turn} ${p.type}, ${((p.end - p.start) * 1000).toFixed(2)} ms`, + tabindex: 0, + }, + svg, + ); + const rect = elem( + "rect", + { + x: x(a), + y, + width: Math.max(0.4, x(b) - x(a)), + height: 22, + fill, + opacity: selected === r ? 1 : 0.66, + rx: 2, + }, + g, + ); + g.style.cursor = "pointer"; + const choose = () => { + selected = r; + details(); + drawTimeline(); + }; + g.addEventListener("click", () => { + if (!drag?.moved) choose(); + }); + g.addEventListener("keydown", (e) => { + if (e.key === "Enter") choose(); + }); + if (p.type === "Sampling" && x(b) - x(a) > 24) + text(g, (x(a) + x(b)) / 2, y + 15, "T" + r.turn, { + "text-anchor": "middle", + fill: "var(--bg)", + "pointer-events": "none", + }); + g.addEventListener("pointermove", (e) => { + let tip = $("#tooltip"), + bounds = svg.getBoundingClientRect(); + tip.hidden = false; + tip.textContent = `Convo ${c} · Turn ${r.turn}\n${p.type}: ${((p.end - p.start) * 1000).toFixed(2)} ms\n${p.start.toFixed(3)}–${p.end.toFixed(3)}s`; + tip.style.left = + Math.max(0, Math.min(w - 270, e.clientX - bounds.left + 12)) + "px"; + tip.style.top = e.clientY - bounds.top + 24 + "px"; + }); + g.addEventListener("pointerleave", () => ($("#tooltip").hidden = true)); + } + } + } + text(svg, (L + w - R) / 2, h - 7, "Elapsed time (seconds)", { + "text-anchor": "middle", + }); + svg.onpointerdown = (e) => { + let px = e.clientX - svg.getBoundingClientRect().left; + if (px < L) return; + drag = { px, t: start + ((px - L) / (w - L - R)) * span, moved: false }; + }; + svg.onpointermove = (e) => { + if ( + drag && + Math.abs(e.clientX - svg.getBoundingClientRect().left - drag.px) > 8 + ) + drag.moved = true; + }; + svg.onpointerup = (e) => { + if (drag?.moved) { + let px = Math.max( + L, + Math.min(w - R, e.clientX - svg.getBoundingClientRect().left), + ), + end = start + ((px - L) / (w - L - R)) * span; + start = Math.min(drag.t, end); + span = Math.max(0.05, Math.abs(end - drag.t)); + $("#tooltip").hidden = true; + let o = $("#window option[data-custom]"); + if (!o) { + o = document.createElement("option"); + o.dataset.custom = "1"; + $("#window").append(o); + } + o.value = span; + o.textContent = span.toFixed(2) + " seconds"; + $("#window").value = String(span); + render(); + } + setTimeout(() => (drag = null), 0); + }; +} +function drawMetrics() { + if ($("#metrics-panel").hidden) return; + const grid = $("#metric-grid"); + grid.replaceChildren(); + let group = null; + for (const m of metrics) { + if ((m.group || "Topline Metrics") !== group) { + group = m.group || "Topline Metrics"; + let heading = document.createElement("h2"); + heading.textContent = group; + heading.style.cssText = "grid-column:1/-1;font-size:16px;margin:6px 0 0"; + grid.append(heading); + } + drawMetricPanel(m, grid); + } +} + +function drawMetricPanel(m, grid) { + const panel = document.createElement("div"); + panel.className = "panel"; + const heading = document.createElement("h3"); + heading.textContent = m.title; + const svg = document.createElementNS(NS, "svg"); + svg.setAttribute("role", "img"); + svg.setAttribute("aria-label", m.title); + const legend = document.createElement("div"); + legend.className = "metric-legend"; + const source = document.createElement("p"); + source.className = "source"; + source.textContent = m.source; + panel.append(heading, svg, legend, source); + grid.append(panel); + let w = panel.clientWidth, + h = 160, + L = 64, + R = 12, + T = 20, + B = 35; + svg.setAttribute("viewBox", `0 0 ${w} ${h}`); + svg.style.width = "100%"; + svg.style.height = h + "px"; + let vs = m.series.flatMap((s) => + s.points.map((p) => p[1]).filter((v) => v !== null), + ); + let max = + m.unit === "%" + ? 100 + : vs.length + ? vs.reduce((maximum, value) => Math.max(maximum, value), 0) * 1.08 + : 1; + if (max === 0) max = 1; + const x = (t) => L + ((t - start) / span) * (w - L - R), + y = (v) => T + (1 - v / max) * (h - T - B); + elem( + "rect", + { + x: L, + y: T, + width: w - L - R, + height: h - T - B, + fill: "none", + stroke: "var(--line)", + }, + svg, + ); + for (let f of [0, 0.5, 1]) { + let yy = y(max * f); + text(svg, L - 6, yy + 4, fmt(max * f), { "text-anchor": "end" }); + if (f === 0.5) + elem( + "line", + { x1: L, x2: w - R, y1: yy, y2: yy, stroke: "var(--line)" }, + svg, + ); + } + for (let f of [0, 0.5, 1]) + text(svg, x(start + span * f), h - 18, (start + span * f).toFixed(1), { + "text-anchor": f === 0 ? "start" : f === 1 ? "end" : "middle", + }); + text(svg, L, T - 6, m.unit); + text(svg, (L + w - R) / 2, h - 2, "Elapsed time (s)", { + "text-anchor": "middle", + "font-size": 11, + }); + if (!vs.length) + text(svg, (L + w - R) / 2, 76, "Unavailable", { + "text-anchor": "middle", + }); + const paths = []; + m.series.forEach((s, j) => { + let path = "", + pen = false; + let points = s.points.filter((p) => p[0] >= start && p[0] <= start + span); + for (let [t, v] of points) { + if (v === null) { + pen = false; + continue; + } + path += (pen ? "L" : "M") + x(t) + "," + y(v); + pen = true; + } + let g = elem("g", {}, svg); + elem( + "path", + { + d: path, + fill: "none", + stroke: COLORS[s.color ?? j % 4], + "stroke-width": 1.6, + "stroke-dasharray": s.dashed ? "5 4" : "", + }, + g, + ); + for (let [t, v] of points) + if (v !== null) + elem( + "circle", + { cx: x(t), cy: y(v), r: 2, fill: COLORS[s.color ?? j % 4] }, + g, + ); + paths.push(g); + let b = document.createElement("button"); + b.type = "button"; + b.setAttribute("aria-pressed", "true"); + let sw = document.createElement("i"); + sw.style.background = COLORS[s.color ?? j % 4]; + if (s.dashed) { + sw.style.background = "none"; + sw.style.borderTop = "2px dashed " + COLORS[s.color ?? j % 4]; + } + b.append(sw, document.createTextNode(s.name)); + b.onclick = () => { + let on = b.getAttribute("aria-pressed") !== "true"; + b.setAttribute("aria-pressed", on); + g.style.display = on ? "" : "none"; + }; + legend.append(b); + }); + let guide = elem( + "line", + { y1: T, y2: h - B, stroke: "var(--muted)", visibility: "hidden" }, + svg, + ); + svg.addEventListener("pointermove", (e) => { + if (!vs.length) return; + let px = Math.max( + L, + Math.min(w - R, e.clientX - svg.getBoundingClientRect().left), + ), + t = start + ((px - L) / (w - L - R)) * span; + guide.setAttribute("x1", px); + guide.setAttribute("x2", px); + guide.setAttribute("visibility", "visible"); + const vals = m.series + .map((s, j) => { + if (paths[j].style.display === "none") return null; + let pt = s.points.reduce( + (best, p) => + !best || Math.abs(p[0] - t) < Math.abs(best[0] - t) ? p : best, + null, + ); + return ( + s.name + + ": " + + (pt ? fmt(pt[1]) : "Unavailable") + + (pt ? " @ " + pt[0].toFixed(1) + "s" : "") + ); + }) + .filter(Boolean); + source.textContent = vals.join(" · "); + }); + svg.addEventListener("pointerleave", () => { + guide.setAttribute("visibility", "hidden"); + source.textContent = m.source; + }); +} + +function render() { + $("#range").textContent = `${start.toFixed(2)}–${(start + span).toFixed(2)}s`; + $("#position").max = Math.max(0, total - span); + $("#position").value = start; + drawTimeline(); + drawMetrics(); +} +for (let b of document.querySelectorAll("[role=tab]")) + b.onclick = () => { + for (let tab of document.querySelectorAll("[role=tab]")) { + const on = b === tab; + tab.setAttribute("aria-selected", on); + $("#" + tab.getAttribute("aria-controls")).hidden = !on; + } + render(); + }; +$("#window").onchange = () => { + span = $("#window").value === "all" ? total : Number($("#window").value); + span = Math.min(span, total); + start = Math.min(start, total - span); + render(); +}; +$("#position").oninput = () => { + start = Number($("#position").value); + render(); +}; +$("#reset").onclick = () => { + start = 0; + span = total; + $("#window").value = "all"; + render(); +}; +const endpoint = $("#exporter"); +for (const url of Object.keys(DATA.exporters)) { + const option = document.createElement("option"); + option.value = url; + option.textContent = url; + endpoint.append(option); +} +endpoint.onchange = () => { + metrics = DATA.exporters[endpoint.value]; + render(); +}; +$(".badge").textContent = + DATA.summary.status.toUpperCase() + + " · " + + DATA.summary.errors + + " recorded errors"; +if (DATA.summary.error) $(".badge").title = DATA.summary.error; +const requestPicker = $("#request"); +DATA.rows.forEach((row, index) => { + const option = document.createElement("option"); + option.value = index; + option.textContent = `Conversation ${row.conversation}, turn ${row.turn}${row.error ? " (failed)" : ""}`; + requestPicker.append(option); +}); +requestPicker.onchange = () => { + selected = DATA.rows[Number(requestPicker.value)]; + details(); + drawTimeline(); +}; +details(); +new ResizeObserver(render).observe(document.querySelector("main")); +render(); diff --git a/benchmark/asr/bench_sglang.py b/benchmark/asr/bench_sglang.py index 875ed952bf60..c60685d62c04 100644 --- a/benchmark/asr/bench_sglang.py +++ b/benchmark/asr/bench_sglang.py @@ -343,7 +343,7 @@ def add_duration(sample): print("\n" + "=" * 20 + " Sample Predictions " + "=" * 20) num_to_show = min(args.print_n, len(results)) for i in range(num_to_show): - print(f"Sample {i+1}:") + print(f"Sample {i + 1}:") print(f" REF: {references[i]}") print(f" PRED: {predictions[i]}") print("-" * 40) diff --git a/benchmark/bench_linear_attention/bench_int8_checkpoint_reuse.py b/benchmark/bench_linear_attention/bench_int8_checkpoint_reuse.py index 12d5f26e6b90..9e30f177ade2 100644 --- a/benchmark/bench_linear_attention/bench_int8_checkpoint_reuse.py +++ b/benchmark/bench_linear_attention/bench_int8_checkpoint_reuse.py @@ -20,7 +20,7 @@ Run the server twice and compare (same flags, toggle int8): python -m sglang.launch_server --model-path --tp 4 \ - --trust-remote-code --mamba-scheduler-strategy extra_buffer \ + --trust-remote-code --mamba-radix-cache-strategy extra_buffer \ --max-mamba-cache-size 256 [--enable-int8-mamba-checkpoint] --port 30000 python benchmark/bench_linear_attention/bench_int8_checkpoint_reuse.py \ @@ -28,7 +28,7 @@ NOTE: prefix-tokens must cross the mamba cache chunk granularity (typically ~512), otherwise nothing is cacheable and reuse is 0 by construction (not a regression). -Use ``--mamba-scheduler-strategy extra_buffer`` on the server: ``no_buffer`` only +Use ``--mamba-radix-cache-strategy extra_buffer`` on the server: ``no_buffer`` only snapshots state at the full-sequence leaf, so a divergent-suffix probe never reuses. """ diff --git a/benchmark/bench_linear_attention/bench_kda_flashinfer_mtp.py b/benchmark/bench_linear_attention/bench_kda_flashinfer_mtp.py index 5e988392e01f..903841d57510 100644 --- a/benchmark/bench_linear_attention/bench_kda_flashinfer_mtp.py +++ b/benchmark/bench_linear_attention/bench_kda_flashinfer_mtp.py @@ -243,9 +243,7 @@ def run(task, fi, tri, device, dtype, args): ) # noqa: E731 else: inp = make_decode_inputs(B, H, HV, K, V, pool, device, dtype) - corr = lambda kern: call_decode( - kern, inp, inp["ssm"].clone() - ) # noqa: E731 + corr = lambda kern: call_decode(kern, inp, inp["ssm"].clone()) # noqa: E731 ssm_t = inp["ssm"].clone() timed = lambda kern: call_decode(kern, inp, ssm_t) # noqa: E731 diff --git a/benchmark/disaggregation/bench_cached_prefix.py b/benchmark/disaggregation/bench_cached_prefix.py new file mode 100644 index 000000000000..92062c5de31b --- /dev/null +++ b/benchmark/disaggregation/bench_cached_prefix.py @@ -0,0 +1,113 @@ +"""Measure PD TTFT with a shared cached prefix and unique request suffixes. + +Start a PD router and workers before running this client. Both worker caches are +flushed after warmup; the shared prefix is then warmed on prefill. Decode radix +caching should be disabled to exercise transfer of the entire missing KV range. +Saves per-request responses and client-observed TTFT to the requested JSON file. +""" + +import argparse +import asyncio +import json +import random +import time +from pathlib import Path + +import aiohttp +import numpy as np + + +async def main(args): + rng = random.Random(35762) + prefix = [rng.randrange(1000, 30000) for _ in range(args.prefix)] + prompts = [ + prefix + [rng.randrange(1000, 30000) for _ in range(args.unique)] + for _ in range(args.requests) + ] + timeout = aiohttp.ClientTimeout(total=1800) + async with aiohttp.ClientSession(timeout=timeout) as session: + + async def generate(ids, stream=True): + start = time.perf_counter() + first = None + result = None + body = { + "input_ids": ids, + "sampling_params": { + "temperature": 0, + "max_new_tokens": args.output, + "ignore_eos": True, + }, + "stream": stream, + } + async with session.post(args.url + "/generate", json=body) as response: + response.raise_for_status() + if stream: + async for line in response.content: + if not line.startswith(b"data:"): + continue + payload = line[5:].strip() + if payload == b"[DONE]": + continue + result = json.loads(payload) + if first is None: + first = time.perf_counter() - start + else: + result = await response.json() + elapsed = time.perf_counter() - start + if result is None or "error" in result: + raise RuntimeError(result) + return {"ttft_s": first, "elapsed_s": elapsed, "response": result} + + # Warm both cold-prefill and cached-prefix/concurrent shapes. Clear both + # sides afterwards so measured requests never reuse a unique suffix. + await generate(prompts[0]) + await asyncio.gather(*(generate(p) for p in prompts[1 : args.concurrency + 1])) + for url in args.workers: + async with session.post(url + "/flush_cache", params={"timeout": 30}) as r: + r.raise_for_status() + if prefix: + await generate(prefix) + sem = asyncio.Semaphore(args.concurrency) + + async def run(ids): + async with sem: + return await generate(ids) + + start = time.perf_counter() + records = await asyncio.gather(*(run(ids) for ids in prompts)) + elapsed = time.perf_counter() - start + ttfts = [r["ttft_s"] for r in records] + summary = { + "args": vars(args), + "elapsed_s": elapsed, + "requests_per_s": len(records) / elapsed, + "ttft_mean_ms": float(np.mean(ttfts) * 1000), + "ttft_p50_ms": float(np.percentile(ttfts, 50) * 1000), + "ttft_p99_ms": float(np.percentile(ttfts, 99) * 1000), + "records": records, + } + Path(args.result).write_text(json.dumps(summary, indent=2)) + print(json.dumps({k: v for k, v in summary.items() if k != "records"}), flush=True) + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument("--url", default="http://127.0.0.1:30000") + parser.add_argument( + "--workers", + nargs="+", + default=["http://127.0.0.1:30001", "http://127.0.0.1:30002"], + ) + parser.add_argument("--prefix", type=int, default=83264) + parser.add_argument("--unique", type=int, default=6720) + parser.add_argument("--output", type=int, default=1) + parser.add_argument("--requests", type=int, default=16) + parser.add_argument("--concurrency", type=int, default=1) + parser.add_argument("--result", required=True) + args = parser.parse_args() + if args.prefix < 0 or args.unique <= 0: + parser.error("--prefix must be nonnegative and --unique must be positive") + if min(args.output, args.requests, args.concurrency) <= 0: + parser.error("--output, --requests and --concurrency must be positive") + asyncio.run(main(args)) diff --git a/benchmark/dllm/README.md b/benchmark/dllm/README.md new file mode 100644 index 000000000000..0615465ca5d9 --- /dev/null +++ b/benchmark/dllm/README.md @@ -0,0 +1,27 @@ +# DiffusionGemma serving measurements + +`bench_diffusion_gemma.py` exercises a running server through its completions API using pretokenized chat prompts. For completed cases it records trial timings, returned text, prompt hashes, and usage counts. It rejects empty responses and unexpected token counts. + +Start the server with the checkpoint and supported serving defaults: + +```sh +python -m sglang.launch_server --model-path "$MODEL" \ + --dllm-algorithm Gemma4Renoise +``` + +Point the client at that server and the same tokenizer: + +```sh +python benchmark/dllm/bench_diffusion_gemma.py \ + --url http://127.0.0.1:30000 --tokenizer "$MODEL" \ + --input-lengths 128 2048 --concurrencies 1 4 8 --output-length 256 \ + --run-id serving-check --result result.json +``` + +## Interpretation + +The client disables EOS stopping to request a fixed output length. Canvas positions after EOS count toward the reported token rate, which is not useful-text throughput. `--denoising-steps` records metadata and does not configure or verify the server's convergence settings. Its default is unset. Set it only when the actual server work has been verified. If a run fails, keep both server and client logs, since the result file contains only completed cases. + +For comparisons, freeze the protocol before collecting results. Match checkpoint identity, physical hardware, precision, prompts, resource limits, generation settings, and stopping policy. Use the same run ID to generate identical prompts. Give each runtime equivalent access to its normal compilation and graph features, preserve every trial and failed request, and disclose dependency and implementation differences. Validate normal text, images, mixed requests, and multi-block generation separately. + +Performance for the merged revision remains unmeasured. Earlier measurements and test counts do not validate this revision. A fixed-work experiment does not establish adaptive-generation performance, model accuracy, or a serving performance lead. diff --git a/benchmark/dllm/bench_diffusion_gemma.py b/benchmark/dllm/bench_diffusion_gemma.py new file mode 100644 index 000000000000..588165249182 --- /dev/null +++ b/benchmark/dllm/bench_diffusion_gemma.py @@ -0,0 +1,166 @@ +"""Compare fixed-work DiffusionGemma serving through the completions API. + +Both servers must use the same checkpoint, canvas length, denoising limit and +stopping settings. Set confidence_threshold=0.0 and stability_threshold=1 on +both servers so early convergence cannot shorten work. This keeps the normal +history size instead of adding history work to enforce the step count. Run them +sequentially on the same physical GPUs, using the same run ID and arguments. +Each request has a unique prompt prefix to avoid prefix-cache reuse. + +This measures complete canvas computation with EOS stopping disabled. Canvas +tokens per second include tokens after EOS and are not useful-text throughput. +Validate output quality separately with the model's default generation settings. +The denoising-steps argument records the server configuration; it does not change it. +""" + +import argparse +import asyncio +import hashlib +import json +import statistics +import time +from pathlib import Path + +import aiohttp +from transformers import AutoTokenizer + + +async def benchmark(args): + tokenizer = AutoTokenizer.from_pretrained(args.tokenizer) + body = tokenizer.encode( + "The city library has books about science, history, music and art. " * 256, + add_special_tokens=False, + ) + template = tokenizer.apply_chat_template( + [ + { + "role": "user", + "content": ( + "Request __REQUEST_ID__.\n__BENCHMARK_BODY__\n" + "Write a detailed explanation of why public libraries are useful." + ), + } + ], + tokenize=False, + add_generation_prompt=True, + enable_thinking=False, + ) + prefix_template, suffix_text = template.split("__BENCHMARK_BODY__") + suffix = tokenizer.encode(suffix_text, add_special_tokens=False) + results = [] + request_number = 0 + timeout = aiohttp.ClientTimeout(total=600) + async with aiohttp.ClientSession(timeout=timeout) as session: + async with session.get(args.url + "/v1/models") as response: + response.raise_for_status() + model = (await response.json())["data"][0]["id"] + + async def request(prompt): + start = time.perf_counter() + async with session.post( + args.url + "/v1/completions", + json={ + "model": model, + "prompt": prompt, + "max_tokens": args.output_length, + "ignore_eos": True, + "stream": False, + }, + ) as response: + payload = await response.json() + if response.status != 200: + raise RuntimeError(f"HTTP {response.status}: {payload}") + elapsed = time.perf_counter() - start + if not payload["choices"][0]["text"].strip(): + raise RuntimeError( + "Empty completion; check the chat template and model" + ) + usage = payload["usage"] + if usage["completion_tokens"] != args.output_length: + raise RuntimeError(f"Unexpected output length: {usage}") + if usage["prompt_tokens"] != len(prompt): + raise RuntimeError(f"Unexpected prompt length: {usage}") + return { + "latency_s": elapsed, + "usage": usage, + "text": payload["choices"][0]["text"], + "prompt_sha256": hashlib.sha256( + json.dumps(prompt).encode() + ).hexdigest(), + } + + for input_length in args.input_lengths: + for concurrency in args.concurrencies: + trials = [] + for trial in range(args.warmups + args.trials): + prompts = [] + for _ in range(concurrency): + # Vary the beginning of every prompt to avoid measuring + # cached prompts. Keep run_id identical across runtimes. + prefix = tokenizer.encode( + prefix_template.replace( + "__REQUEST_ID__", f"{args.run_id}-{request_number}" + ), + add_special_tokens=False, + ) + request_number += 1 + n = input_length - len(prefix) - len(suffix) + if n < 0: + raise ValueError("Input length is too short") + prompts.append(prefix + body[:n] + suffix) + start = time.perf_counter() + requests = await asyncio.gather(*(request(p) for p in prompts)) + trials.append( + { + "warmup": trial < args.warmups, + "wall_s": time.perf_counter() - start, + "requests": requests, + } + ) + measured = trials[args.warmups :] + wall_times = [t["wall_s"] for t in measured] + latencies = [r["latency_s"] for t in measured for r in t["requests"]] + result = { + "input_length": input_length, + "output_length": args.output_length, + "concurrency": concurrency, + "median_wall_s": statistics.median(wall_times), + "median_request_s": statistics.median(latencies), + "canvas_tokens_per_s": concurrency + * args.output_length + / statistics.median(wall_times), + "trials": trials, + } + results.append(result) + print( + json.dumps({k: v for k, v in result.items() if k != "trials"}), + flush=True, + ) + args.result.write_text( + json.dumps( + { + "settings": vars(args) | {"result": str(args.result)}, + "results": results, + }, + indent=2, + ) + ) + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--url", required=True) + parser.add_argument("--tokenizer", required=True) + parser.add_argument("--input-lengths", type=int, nargs="+", default=[128, 512]) + parser.add_argument("--output-length", type=int, default=256) + parser.add_argument( + "--denoising-steps", + type=int, + help="Verified fixed denoising step count, recorded as metadata only", + ) + parser.add_argument("--concurrencies", type=int, nargs="+", default=[1, 4, 8]) + parser.add_argument("--warmups", type=int, default=3) + parser.add_argument("--trials", type=int, default=15) + parser.add_argument("--run-id", required=True) + parser.add_argument("--result", type=Path, required=True) + asyncio.run(benchmark(parser.parse_args())) diff --git a/benchmark/gsm8k/bench_sglang.py b/benchmark/gsm8k/bench_sglang.py index a0e09a39c3c5..b3989673a495 100644 --- a/benchmark/gsm8k/bench_sglang.py +++ b/benchmark/gsm8k/bench_sglang.py @@ -53,9 +53,9 @@ def main(args): if args.enable_thinking: from transformers import AutoTokenizer - assert ( - args.tokenizer_path is not None - ), "--tokenizer-path is required when --enable-thinking is set" + assert args.tokenizer_path is not None, ( + "--tokenizer-path is required when --enable-thinking is set" + ) tokenizer = AutoTokenizer.from_pretrained( args.tokenizer_path, trust_remote_code=True ) diff --git a/benchmark/hf3fs/bench_client.py b/benchmark/hf3fs/bench_client.py index ef4a967c4300..7b18a8b2c6fb 100644 --- a/benchmark/hf3fs/bench_client.py +++ b/benchmark/hf3fs/bench_client.py @@ -14,11 +14,11 @@ def print_stats(x: List[int]): x = sorted(x) lenx = len(x) print( - f"mean = {sum(x)/len(x):.2f}, " + f"mean = {sum(x) / len(x):.2f}, " f"min = {min(x):.2f}, " - f"p25 = {x[int(lenx*0.25)]:.2f}, " - f"p50 = {x[int(lenx*0.5)]:.2f}, " - f"p75 = {x[int(lenx*0.75)]:.2f}, " + f"p25 = {x[int(lenx * 0.25)]:.2f}, " + f"p50 = {x[int(lenx * 0.5)]:.2f}, " + f"p75 = {x[int(lenx * 0.75)]:.2f}, " f"max = {max(x):.2f}" ) diff --git a/benchmark/hf3fs/bench_storage.py b/benchmark/hf3fs/bench_storage.py index f0ce171bf675..71122a41d1e6 100644 --- a/benchmark/hf3fs/bench_storage.py +++ b/benchmark/hf3fs/bench_storage.py @@ -18,11 +18,11 @@ def print_stats(x: List[int]): x = sorted(x) lenx = len(x) print( - f"mean = {sum(x)/len(x):.2f}, " + f"mean = {sum(x) / len(x):.2f}, " f"min = {min(x):.2f}, " - f"p25 = {x[int(lenx*0.25)]:.2f}, " - f"p50 = {x[int(lenx*0.5)]:.2f}, " - f"p75 = {x[int(lenx*0.75)]:.2f}, " + f"p25 = {x[int(lenx * 0.25)]:.2f}, " + f"p50 = {x[int(lenx * 0.5)]:.2f}, " + f"p75 = {x[int(lenx * 0.75)]:.2f}, " f"max = {max(x):.2f}" ) diff --git a/benchmark/hf3fs/bench_zerocopy.py b/benchmark/hf3fs/bench_zerocopy.py index 0f9c5e150f79..8559e45b7558 100644 --- a/benchmark/hf3fs/bench_zerocopy.py +++ b/benchmark/hf3fs/bench_zerocopy.py @@ -5,10 +5,10 @@ from tqdm import tqdm from sglang.srt.distributed import ( - get_world_group, init_distributed_environment, initialize_model_parallel, ) +from sglang.srt.distributed.parallel_state import get_world_group from sglang.srt.managers.cache_controller import ( HiCacheController, PrefetchOperation, @@ -17,6 +17,7 @@ from sglang.srt.mem_cache.allocator import TokenToKVPoolAllocator from sglang.srt.mem_cache.memory_pool import MHATokenToKVPool from sglang.srt.mem_cache.pool_host.mha import MHATokenToKVPoolHost +from sglang.test.test_utils import publish_build_topology init_distributed_environment( world_size=1, @@ -26,10 +27,8 @@ backend="gloo", ) -initialize_model_parallel( - tensor_model_parallel_size=1, - pipeline_model_parallel_size=1, -) +publish_build_topology() +initialize_model_parallel() group = get_world_group().cpu_group @@ -109,7 +108,7 @@ for operation in operations: cache_controller.generic_page_backup(operation, batch_size=128) tok = time.monotonic() -print(f"{tok-tik:.6f} s") +print(f"{tok - tik:.6f} s") operations = [ PrefetchOperation( @@ -137,4 +136,4 @@ for operation in operations: cache_controller.generic_page_transfer(operation, batch_size=128) tok = time.monotonic() -print(f"{tok-tik:.6f} s") +print(f"{tok - tik:.6f} s") diff --git a/benchmark/hicache/bench_mix.py b/benchmark/hicache/bench_mix.py index 2a65574ea882..5d26151869f0 100644 --- a/benchmark/hicache/bench_mix.py +++ b/benchmark/hicache/bench_mix.py @@ -457,7 +457,7 @@ def response_handler(self): try: user_data, response = self.response_queue.get(timeout=10) logger.info( - f"{((time.perf_counter()-self.start_time)/self.duration*100):.2f}%" + f"{((time.perf_counter() - self.start_time) / self.duration * 100):.2f}%" ) if not response.success: raise ValueError(f"Request failed with error: {response.error}") @@ -540,10 +540,10 @@ def run(self): output_stats = self.user_generator.output_stats print(f"round_ratios: {user_stats}") print( - f"mean_new_tokens_per_round: {[int(a/b) if b > 0 else 0 for a, b in input_stats]}" + f"mean_new_tokens_per_round: {[int(a / b) if b > 0 else 0 for a, b in input_stats]}" ) print( - f"mean_return_tokens_per_round: {[int(a/b) if b > 0 else 0 for a, b in output_stats]}" + f"mean_return_tokens_per_round: {[int(a / b) if b > 0 else 0 for a, b in output_stats]}" ) return performance_data diff --git a/benchmark/hicache/bench_multiturn.py b/benchmark/hicache/bench_multiturn.py index 49483fe849db..c1f1c3d03f7a 100644 --- a/benchmark/hicache/bench_multiturn.py +++ b/benchmark/hicache/bench_multiturn.py @@ -17,6 +17,7 @@ from sglang.test.kits.cache_hit_kit import ( async_request_openai_chat_completions, async_request_sglang_generate, + calculate_tpot_statistics, gen_payload, gen_payload_openai, ) @@ -362,6 +363,7 @@ def __init__(self, args): self.pbar = tqdm(total=self.total_requests) self.performance_metrics = { "ttft": [], + "tpot": [], "itl": [], "latency": [], "prompt_len": [], @@ -458,6 +460,8 @@ def response_handler(self): current_round = self.client_records[client_id]["round"] self.client_records[client_id]["round"] += 1 self.performance_metrics["ttft"].append(response.ttft) + if response.tpot is not None: + self.performance_metrics["tpot"].append(response.tpot) self.performance_metrics["itl"].extend(response.itl) self.performance_metrics["latency"].append(response.latency) self.performance_metrics["prompt_len"].append(response.prompt_len) @@ -582,6 +586,8 @@ def percentile(sorted_vals, q): def max_or_zero(sorted_vals): return sorted_vals[-1] if sorted_vals else 0.0 + tpot_statistics = calculate_tpot_statistics(self.performance_metrics["tpot"]) + performance_data = { "summary": { "total_requests": len(self.performance_metrics["ttft"]), @@ -608,6 +614,7 @@ def max_or_zero(sorted_vals): "p99_ttft": percentile(sorted_ttft, 0.99), "median_ttft": percentile(sorted_ttft, 0.5), "max_ttft": max_or_zero(sorted_ttft), + **tpot_statistics, "average_itl": ( sum(self.performance_metrics["itl"]) / len(self.performance_metrics["itl"]) @@ -686,6 +693,11 @@ def max_or_zero(sorted_vals): print(f" P99 TTFT: {performance_data['summary']['p99_ttft']:.2f}") print(f" Median TTFT: {performance_data['summary']['median_ttft']:.2f}") print(f" Max TTFT: {performance_data['summary']['max_ttft']:.2f}") + print(f" Average TPOT: {performance_data['summary']['average_tpot']:.4f}") + print(f" P90 TPOT: {performance_data['summary']['p90_tpot']:.4f}") + print(f" P99 TPOT: {performance_data['summary']['p99_tpot']:.4f}") + print(f" Median TPOT: {performance_data['summary']['median_tpot']:.4f}") + print(f" Max TPOT: {performance_data['summary']['max_tpot']:.4f}") print(f" Average ITL: {performance_data['summary']['average_itl']:.4f}") print(f" P90 ITL: {performance_data['summary']['p90_itl']:.4f}") print(f" P99 ITL: {performance_data['summary']['p99_itl']:.4f}") diff --git a/benchmark/hicache/bench_serving.py b/benchmark/hicache/bench_serving.py index a80c059e538a..66dec6176c7d 100644 --- a/benchmark/hicache/bench_serving.py +++ b/benchmark/hicache/bench_serving.py @@ -75,9 +75,9 @@ async def async_request_openai_completions( pbar: Optional[tqdm] = None, ) -> RequestFuncOutput: api_url = request_func_input.api_url - assert api_url.endswith( - "completions" - ), "OpenAI Completions API URL must end with 'completions'." + assert api_url.endswith("completions"), ( + "OpenAI Completions API URL must end with 'completions'." + ) async with aiohttp.ClientSession(timeout=AIOHTTP_TIMEOUT) as session: payload = { @@ -619,14 +619,6 @@ async def limited_request_func(request_func_input, queue, tokenizer, pbar): "std_itl_ms": metrics.std_itl_ms, "p99_itl_ms": metrics.p99_itl_ms, "concurrency": metrics.concurrency, - "input_throughput": metrics.input_throughput, - "output_throughput": metrics.output_throughput, - "fixed_output_len": args.fixed_output_len, - "random_input_len": args.random_input_len, - "random_output_len": args.random_output_len, - "random_range_ratio": args.random_range_ratio, - "duration": benchmark_duration, - "completed": metrics.completed, } else: print(f"Error running benchmark for request rate: {request_rate}") diff --git a/benchmark/hicache/nextqa.py b/benchmark/hicache/nextqa.py index 4db6caa1cd49..1c74dac713f4 100644 --- a/benchmark/hicache/nextqa.py +++ b/benchmark/hicache/nextqa.py @@ -120,7 +120,7 @@ def get_video_prompt(self, entry, max_frames) -> VideoPrompt: video = Video(video_path, num_frames) prompt = entry["question"] + "?" if self.task == "MC": # add choices - prompt += f' a0: {entry["a0"]}, a1: {entry["a1"]}, a2: {entry["a2"]}, a3: {entry["a3"]}' + prompt += f" a0: {entry['a0']}, a1: {entry['a1']}, a2: {entry['a2']}, a3: {entry['a3']}" return VideoPrompt(video_path, num_frames, prompt) def __iter__(self): diff --git a/benchmark/io/bench_mm_token_padding.py b/benchmark/io/bench_mm_token_padding.py new file mode 100644 index 000000000000..23b5991fe3ec --- /dev/null +++ b/benchmark/io/bench_mm_token_padding.py @@ -0,0 +1,198 @@ +"""CPU comparison of baseline and optimized offset-based scheduler padding. + +Run from the repository root: + PYTHONPATH=python python benchmark/io/bench_mm_token_padding.py --baseline-ref BASE_SHA + +Inputs are synthetic. Both paths include the scheduler's final ownership copy. +This measures padding CPU time, not model forward time or end-to-end TTFT. +""" + +import argparse +import ast +import gc +import json +import platform +import statistics +import subprocess +import time +from array import array +from pathlib import Path + +import torch + +from sglang.srt.managers import mm_utils +from sglang.srt.managers.mm_utils import ( + MultiModalityDataPaddingPatternMultimodalTokens, + duplicate_pad_mm_input_ids, +) +from sglang.srt.managers.schedule_batch import ( + Modality, + MultimodalDataItem, + MultimodalInputs, +) + + +def load_baseline(ref): + """Load the exact baseline helper classes without replacing installed SGLang.""" + root = Path(__file__).resolve().parents[2] + commit = subprocess.check_output( + ["git", "rev-parse", "--verify", f"{ref}^{{commit}}"], cwd=root, text=True + ).strip() + path = "python/sglang/srt/managers/mm_utils.py" + source = subprocess.check_output( + ["git", "show", f"{commit}:{path}"], cwd=root, text=True + ) + names = { + "MultiModalityDataPaddingPattern", + "MultiModalityDataPaddingPatternMultimodalTokens", + } + classes = [ + node + for node in ast.parse(source).body + if isinstance(node, ast.ClassDef) and node.name in names + ] + if {node.name for node in classes} != names: + raise ValueError("Baseline does not contain the expected padding helpers") + namespace = dict(vars(mm_utils)) + # Reuse dependency imports, but compile the original method bodies verbatim. + exec( + compile(ast.Module(body=classes, type_ignores=[]), f"{commit}:{path}", "exec"), + namespace, + ) + return commit, namespace["MultiModalityDataPaddingPatternMultimodalTokens"]() + + +def make_case(num_tokens, num_items, span_tokens): + if num_items and num_tokens < 3 * num_items: + raise ValueError( + "Each item needs three tokens for its span and surrounding text" + ) + input_ids = array("q", range(257, 257 + num_tokens)) + items = [] + stride = num_tokens // max(num_items, 1) + for index in range(num_items): + start = index * stride + 1 + end = start + min(span_tokens, stride - 2) - 1 + items.append( + MultimodalDataItem( + modality=Modality.IMAGE, + pad_value=2**40 + index, + offsets=[(start, end)], + ) + ) + return input_ids, MultimodalInputs(mm_items=items, im_token_id=1) + + +def benchmark_case(num_tokens, num_items, args, baseline): + pattern = MultiModalityDataPaddingPatternMultimodalTokens() + input_ids, mm_inputs = make_case(num_tokens, num_items, args.span_tokens) + original = input_ids.tobytes() + + def baseline_padding(): + return array("q", baseline.pad_input_tokens(input_ids, mm_inputs)) + + def optimized_padding(): + return duplicate_pad_mm_input_ids( + input_ids, mm_inputs, pattern.pad_input_tokens + ) + + expected = baseline_padding() + actual = optimized_padding() + assert actual == expected, "Token values differ" + assert input_ids.tobytes() == original, "Original input was mutated" + assert actual is not input_ids, "Scheduler result aliases the original input" + + functions = {"baseline": baseline_padding, "optimized": optimized_padding} + timings = {name: [] for name in functions} + for _ in range(args.warmups): + for function in functions.values(): + function() + gc_was_enabled = gc.isenabled() + gc.disable() + try: + for trial in range(args.trials): + order = ( + ("baseline", "optimized") if trial % 2 else ("optimized", "baseline") + ) + for name in order: + start = time.perf_counter_ns() + result = functions[name]() + elapsed = time.perf_counter_ns() - start + timings[name].append(elapsed / 1e6) + del result + finally: + if gc_was_enabled: + gc.enable() + assert input_ids.tobytes() == original + medians = {name: statistics.median(samples) for name, samples in timings.items()} + return { + "tokens": num_tokens, + "items": num_items, + "baseline_ms": medians["baseline"], + "optimized_ms": medians["optimized"], + "speedup": medians["baseline"] / medians["optimized"], + } + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--baseline-ref", required=True, help="Local pre-change git ref" + ) + parser.add_argument( + "--tokens", nargs="+", type=int, default=[128, 4096, 16384, 131072] + ) + parser.add_argument("--items", nargs="+", type=int, default=[0, 1, 10]) + parser.add_argument("--span-tokens", type=int, default=32) + parser.add_argument("--trials", type=int, default=41) + parser.add_argument("--warmups", type=int, default=5) + parser.add_argument("--threads", type=int, default=1) + parser.add_argument("--json", action="store_true") + args = parser.parse_args() + if ( + min(args.tokens + args.items) < 0 + or min(args.span_tokens, args.trials, args.threads) < 1 + or args.warmups < 0 + ): + parser.error("Lengths/items/warmups must be nonnegative; other counts positive") + if any(n < 3 * items for n in args.tokens for items in args.items): + parser.error("Each token length must accommodate at least 3 tokens per item") + torch.set_num_threads(args.threads) + baseline_commit, baseline = load_baseline(args.baseline_ref) + rows = [ + benchmark_case(tokens, items, args, baseline) + for tokens in args.tokens + for items in args.items + ] + if args.json: + print( + json.dumps( + { + "baseline_commit": baseline_commit, + "python": platform.python_version(), + "torch": torch.__version__, + "architecture": platform.machine(), + "threads": args.threads, + "trials": args.trials, + "warmups": args.warmups, + "span_tokens": args.span_tokens, + "results": rows, + }, + indent=2, + ) + ) + return + print("Offset-based CPU padding; median milliseconds; synthetic inputs") + print( + f"{'Tokens':>8} {'Items':>6} {'Baseline (ms)':>14} {'Optimized (ms)':>14} {'Speedup':>9}" + ) + for row in rows: + print( + f"{row['tokens']:>8} {row['items']:>6} " + f"{row['baseline_ms']:>14.4f} {row['optimized_ms']:>14.4f} " + f"{row['speedup']:>8.2f}x" + ) + + +if __name__ == "__main__": + main() diff --git a/benchmark/kernels/all_gather/benchmark_mscclpp.py b/benchmark/kernels/all_gather/benchmark_mscclpp.py new file mode 100644 index 000000000000..1e902ca0d9c8 --- /dev/null +++ b/benchmark/kernels/all_gather/benchmark_mscclpp.py @@ -0,0 +1,284 @@ +"""Benchmark multi-node MSCCL++ AllGather against NCCL.""" + +import argparse +import os + +import torch +import torch.distributed as dist + +from sglang.srt.distributed import init_distributed_environment +from sglang.srt.distributed.parallel_state import ( + cleanup_dist_env_and_memory, + get_tensor_model_parallel_group, + initialize_model_parallel, + set_mscclpp, +) +from sglang.test.test_utils import publish_build_topology + + +def _max_across_ranks(value: float, group) -> float: + tensor = torch.tensor([value], dtype=torch.float64) + dist.all_reduce(tensor, op=dist.ReduceOp.MAX, group=group) + return tensor.item() + + +def _time_eager(func, group, warmup: int = 2, repeat: int = 10) -> float: + for _ in range(warmup): + func() + torch.cuda.synchronize() + dist.barrier(group=group) + + start = torch.cuda.Event(enable_timing=True) + end = torch.cuda.Event(enable_timing=True) + start.record() + for _ in range(repeat): + func() + end.record() + end.synchronize() + return _max_across_ranks(start.elapsed_time(end) * 1000 / repeat, group) + + +def _time_cuda_graph( + func, + group, + warmup: int = 100, + graph_loop: int = 10, + repeat: int = 100, +) -> float: + current_stream = torch.cuda.current_stream() + capture_stream = torch.cuda.Stream() + capture_stream.wait_stream(current_stream) + with torch.cuda.stream(capture_stream): + func(capture_stream) + capture_stream.synchronize() + + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph, stream=capture_stream): + for _ in range(graph_loop): + func(capture_stream) + + with torch.cuda.stream(capture_stream): + for _ in range(warmup): + graph.replay() + capture_stream.synchronize() + + dist.barrier(group=group) + start = torch.cuda.Event(enable_timing=True) + end = torch.cuda.Event(enable_timing=True) + start.record(capture_stream) + with torch.cuda.stream(capture_stream): + for _ in range(repeat): + graph.replay() + end.record(capture_stream) + end.synchronize() + torch.cuda.current_stream().wait_stream(capture_stream) + return _max_across_ranks( + start.elapsed_time(end) * 1000 / (repeat * graph_loop), + group, + ) + + +def _validate_cuda_graph_updates( + func, + reference_func, + input_tensor: torch.Tensor, + actual: torch.Tensor, + expected: torch.Tensor, + group, +) -> None: + current_stream = torch.cuda.current_stream() + capture_stream = torch.cuda.Stream() + capture_stream.wait_stream(current_stream) + with torch.cuda.stream(capture_stream): + func(capture_stream) + capture_stream.synchronize() + + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph, stream=capture_stream): + func(capture_stream) + + input_tensor.neg_() + reference_func() + current_stream.synchronize() + capture_stream.wait_stream(current_stream) + with torch.cuda.stream(capture_stream): + graph.replay() + capture_stream.synchronize() + + if not torch.equal(actual, expected): + raise AssertionError("MSCCL++ CUDA graph replay used stale AllGather data") + dist.barrier(group=group) + + +def _message_sizes(min_bytes: int, max_bytes: int): + size = 1 << (min_bytes - 1).bit_length() + while size <= max_bytes: + yield size + size <<= 1 + + +def human_readable_size(size, decimal_places=1): + for unit in ["B", "KiB", "MiB", "GiB", "TiB", "PiB"]: + if size < 1024.0 or unit == "PiB": + break + size /= 1024.0 + return f"{size:.{decimal_places}f} {unit}" + + +try: + from tabulate import tabulate +except ImportError: + print("tabulate not installed, skipping table printing") + tabulate = None + + +def print_markdown_table(data): + if tabulate is not None: + print(tabulate(data, headers="keys", tablefmt="github")) + return + headers = data[0].keys() + header_row = "| " + " | ".join(headers) + " |" + separator = "| " + " | ".join(["---"] * len(headers)) + " |" + rows = [] + for item in data: + row = "| " + " | ".join(str(item[key]) for key in headers) + " |" + rows.append(row) + markdown_table = "\n".join([header_row, separator] + rows) + print(markdown_table) + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--min-bytes", type=int, default=1 << 10) + parser.add_argument("--max-bytes", type=int, default=8 << 20) + args = parser.parse_args() + if not (1 << 10) <= args.min_bytes <= args.max_bytes <= (8 << 20): + parser.error("Require 1 KiB <= min-bytes <= max-bytes <= 8 MiB") + + local_rank = int(os.environ.get("LOCAL_RANK", 0)) + torch.cuda.set_device(local_rank) + if not dist.is_initialized(): + dist.init_process_group( + backend="nccl", + device_id=torch.device("cuda", local_rank), + ) + world_size = dist.get_world_size() + rank = dist.get_rank() + device = torch.cuda.current_device() + set_mscclpp(True) + init_distributed_environment( + world_size=world_size, + rank=rank, + local_rank=local_rank, + ) + publish_build_topology(world_rank=rank, tp_size=world_size) + initialize_model_parallel() + tp_group = get_tensor_model_parallel_group() + gpu_group = tp_group.device_group + cpu_group = tp_group.cpu_group + pynccl_comm = tp_group.pynccl_comm + mscclpp_comm = tp_group.pymscclpp_comm + assert pynccl_comm is not None + assert mscclpp_comm is not None + if args.min_bytes % world_size != 0 or args.max_bytes % world_size != 0: + parser.error("Total message sizes must be divisible by WORLD_SIZE") + + if not mscclpp_comm._best_configs["allgather"]: + raise RuntimeError("No tuned MSCCL++ AllGather configuration is available") + + result = [] + + for total_bytes in _message_sizes(args.min_bytes, args.max_bytes): + input_bytes = total_bytes // world_size + if input_bytes % torch.bfloat16.itemsize != 0: + continue + elements = input_bytes // torch.bfloat16.itemsize + input_tensor = ( + torch.arange( + elements, + dtype=torch.long, + device=device, + ) + % 193 + + rank + ).to(torch.bfloat16) + nccl_output = torch.empty( + elements * world_size, + dtype=input_tensor.dtype, + device=device, + ) + mscclpp_output = torch.empty_like(nccl_output) + + def torch_nccl(): + dist.all_gather_into_tensor( + nccl_output, + input_tensor, + group=gpu_group, + ) + + def py_nccl(): + pynccl_comm.all_gather(nccl_output, input_tensor) + + def mscclpp(): + mscclpp_comm.all_gather(mscclpp_output, input_tensor) + + def mscclpp_on_stream(stream): + mscclpp_comm.all_gather( + mscclpp_output, + input_tensor, + stream=stream, + ) + + def py_nccl_on_stream(_stream): + with pynccl_comm.change_state(enable=True): + pynccl_comm.all_gather(nccl_output, input_tensor) + + torch_nccl() + mscclpp() + py_nccl() + torch.cuda.synchronize() + if not torch.equal(mscclpp_output, nccl_output): + raise AssertionError( + f"MSCCL++ AllGather mismatch at {input_bytes} input bytes" + ) + _validate_cuda_graph_updates( + mscclpp_on_stream, + torch_nccl, + input_tensor, + mscclpp_output, + nccl_output, + cpu_group, + ) + + nccl_eager_us = _time_eager(torch_nccl, cpu_group) + mscclpp_eager_us = _time_eager(mscclpp, cpu_group) + mscclpp_graph_us = _time_cuda_graph(mscclpp_on_stream, cpu_group) + pynccl_graph_us = _time_cuda_graph(py_nccl_on_stream, cpu_group) + + result.append( + { + "msg_size": human_readable_size(total_bytes), + "torch eager time": nccl_eager_us, + "msccl eager time": mscclpp_eager_us, + "msccl graph time": mscclpp_graph_us, + "pynccl graph time": pynccl_graph_us, + } + ) + if rank == 0: + print( + f"sz={total_bytes // input_tensor.element_size()}, " + f"dtype={input_tensor.dtype}: correctness check PASS!" + ) + + torch.cuda.empty_cache() + + if rank == 0: + print_markdown_table(result) + + mscclpp_comm.destroy() + dist.barrier() + cleanup_dist_env_and_memory() + + +if __name__ == "__main__": + main() diff --git a/benchmark/kernels/all_reduce/benchmark_all_reduce.py b/benchmark/kernels/all_reduce/benchmark_all_reduce.py index 30a7733f3dde..41dda1930058 100644 --- a/benchmark/kernels/all_reduce/benchmark_all_reduce.py +++ b/benchmark/kernels/all_reduce/benchmark_all_reduce.py @@ -21,6 +21,7 @@ init_distributed_environment, initialize_model_parallel, ) +from sglang.test.test_utils import publish_build_topology def parse_args(): @@ -85,7 +86,8 @@ def init_dist(backend: str): distributed_init_method=distributed_init_method, local_rank=rank, ) - initialize_model_parallel(tensor_model_parallel_size=world_size) + publish_build_topology(world_rank=rank, tp_size=world_size) + initialize_model_parallel() return dist.group.WORLD diff --git a/benchmark/kernels/all_reduce/benchmark_fused_ar_rms_amd.py b/benchmark/kernels/all_reduce/benchmark_fused_ar_rms_amd.py index 1fa3819cc290..107533fb12d0 100644 --- a/benchmark/kernels/all_reduce/benchmark_fused_ar_rms_amd.py +++ b/benchmark/kernels/all_reduce/benchmark_fused_ar_rms_amd.py @@ -41,6 +41,7 @@ initialize_model_parallel, set_custom_all_reduce, ) +from sglang.test.test_utils import publish_build_topology Shape = Tuple[int, int] @@ -381,7 +382,8 @@ def main(): distributed_init_method="env://", backend="nccl", ) - initialize_model_parallel(tensor_model_parallel_size=world_size) + publish_build_topology(world_rank=rank, tp_size=world_size) + initialize_model_parallel() prefill_shapes = parse_shapes(args.prefill_shapes) decode_shapes = parse_shapes(args.decode_shapes) diff --git a/benchmark/kernels/all_reduce/benchmark_fused_ar_rms_quant_amd.py b/benchmark/kernels/all_reduce/benchmark_fused_ar_rms_quant_amd.py index 85cabe94c9be..4c02a5df48bb 100644 --- a/benchmark/kernels/all_reduce/benchmark_fused_ar_rms_quant_amd.py +++ b/benchmark/kernels/all_reduce/benchmark_fused_ar_rms_quant_amd.py @@ -47,6 +47,7 @@ initialize_model_parallel, set_custom_all_reduce, ) +from sglang.test.test_utils import publish_build_topology Shape = Tuple[int, int] FP8_DTYPE = torch.float8_e4m3fnuz @@ -400,7 +401,8 @@ def main() -> None: distributed_init_method="env://", backend="nccl", ) - initialize_model_parallel(tensor_model_parallel_size=world_size) + publish_build_topology(world_rank=rank, tp_size=world_size) + initialize_model_parallel() if rank == 0: print( diff --git a/benchmark/kernels/all_reduce/benchmark_mscclpp.py b/benchmark/kernels/all_reduce/benchmark_mscclpp.py index 5b72d2f460d1..5c131619a221 100644 --- a/benchmark/kernels/all_reduce/benchmark_mscclpp.py +++ b/benchmark/kernels/all_reduce/benchmark_mscclpp.py @@ -12,9 +12,9 @@ --master_port $MASTER_PORT benchmark/kernels/all_reduce/benchmark_mscclpp.py """ +import argparse import os from contextlib import nullcontext -from typing import List import torch import torch.distributed as dist @@ -23,13 +23,17 @@ from sglang.srt.distributed import init_distributed_environment from sglang.srt.distributed.device_communicators.pymscclpp import PyMscclppCommunicator from sglang.srt.distributed.device_communicators.pynccl import PyNcclCommunicator +from sglang.srt.distributed.device_communicators.pynccl_allocator import ( + use_symmetric_memory, +) from sglang.srt.distributed.parallel_state import ( cleanup_dist_env_and_memory, - get_tensor_model_parallel_group, graph_capture, initialize_model_parallel, - set_mscclpp_all_reduce, + set_mscclpp, ) +from sglang.srt.runtime_context import get_parallel +from sglang.test.test_utils import publish_build_topology def torch_allreduce(torch_input: torch.Tensor, group: ProcessGroup) -> torch.Tensor: @@ -50,14 +54,21 @@ def pynccl_allreduce( return msccl_input -def _bench_graph_time(func, inp_randn, warmup_loop=2, graph_loop=10, test_loop=10): - graph_input = inp_randn.clone() - graph_input_snapshot = inp_randn.clone() +def _bench_graph_time( + func, + inp_randn, + tp_group, + warmup_loop=2, + graph_loop=10, + test_loop=10, +): + with use_symmetric_memory(tp_group): + graph_input = inp_randn.clone() with graph_capture() as graph_capture_context: graph = torch.cuda.CUDAGraph() with torch.cuda.graph(graph, stream=graph_capture_context.stream): for _ in range(graph_loop): - graph_input.copy_(graph_input_snapshot) + graph_input.copy_(inp_randn) graph_out = func(graph_input) graph.replay() @@ -70,7 +81,7 @@ def _bench_graph_time(func, inp_randn, warmup_loop=2, graph_loop=10, test_loop=1 start_event = torch.cuda.Event(enable_timing=True) end_event = torch.cuda.Event(enable_timing=True) - latencies: List[float] = [] + latencies = [] for _ in range(test_loop): torch.cuda.synchronize() dist.barrier() @@ -84,8 +95,9 @@ def _bench_graph_time(func, inp_randn, warmup_loop=2, graph_loop=10, test_loop=1 return func_output, func_cost_us -def _bench_eager_time(func, inp_randn, warmup_loop=2, test_loop=10): - eager_input = inp_randn.clone() +def _bench_eager_time(func, inp_randn, tp_group, warmup_loop=2, test_loop=10): + with use_symmetric_memory(tp_group): + eager_input = inp_randn.clone() eager_output = func(eager_input) func_output = eager_output.clone() @@ -155,56 +167,77 @@ def print_markdown_table(data): if __name__ == "__main__": import logging + parser = argparse.ArgumentParser() + parser.add_argument( + "--enable-symm-mem", + action="store_true", + help="Allocate collective inputs from NCCL symmetric memory.", + ) + args = parser.parse_args() + logging.basicConfig( level=logging.INFO, format="%(asctime)s - %(levelname)s - %(message)s", datefmt="%Y-%m-%d %H:%M:%S", force=True, ) + local_rank = int(os.environ["LOCAL_RANK"]) + torch.cuda.set_device(local_rank) + device = torch.cuda.current_device() if not dist.is_initialized(): - dist.init_process_group(backend="nccl") + dist.init_process_group( + backend="nccl", device_id=torch.device("cuda", local_rank) + ) world, world_size = dist.group.WORLD, dist.get_world_size() rank = dist.get_rank() - torch.cuda.set_device(rank % 8) - device = torch.cuda.current_device() - set_mscclpp_all_reduce(True) + set_mscclpp(True) init_distributed_environment( world_size=world_size, rank=rank, - local_rank=rank % 8, + local_rank=local_rank, + ) + publish_build_topology( + world_rank=rank, + tp_size=world_size, + enable_symm_mem=args.enable_symm_mem, ) - initialize_model_parallel(tensor_model_parallel_size=world_size) - group = get_tensor_model_parallel_group().device_group - cpu_group = get_tensor_model_parallel_group().cpu_group - pynccl_comm = get_tensor_model_parallel_group().pynccl_comm - pymscclpp_comm = get_tensor_model_parallel_group().pymscclpp_comm + initialize_model_parallel(enable_symm_mem=args.enable_symm_mem) + tp_group = get_parallel().tp_group + group = tp_group.device_group + pynccl_comm = tp_group.pynccl_comm + pymscclpp_comm = tp_group.pymscclpp_comm dist.barrier() profile = False dtype = torch.bfloat16 ctx = get_torch_prof_ctx(profile) result = [] + max_message_bytes = 1 << 20 with ctx: for i in range(10, 20): sz = 2**i - if sz * dtype.itemsize > 2**20: + if sz * dtype.itemsize > max_message_bytes: break - inp_randn = torch.randint(1, 16, (sz,), dtype=dtype, device=device) + inp_randn = torch.randint(0, 2, (sz,), dtype=dtype, device=device) - memory = torch.empty_like(inp_randn) - memory_out = torch.empty_like(memory) torch_eager_output, torch_eager_time = _bench_eager_time( - lambda inp: torch_allreduce(inp, group), inp_randn + lambda inp: torch_allreduce(inp, group), inp_randn, tp_group ) msccl_eager_output, msccl_eager_time = _bench_eager_time( - lambda inp: msccl_allreduce(inp, pymscclpp_comm), inp_randn + lambda inp: msccl_allreduce(inp, pymscclpp_comm), + inp_randn, + tp_group, ) msccl_graph_output, msccl_graph_time = _bench_graph_time( - lambda inp: msccl_allreduce(inp, pymscclpp_comm), inp_randn + lambda inp: msccl_allreduce(inp, pymscclpp_comm), + inp_randn, + tp_group, ) # since pynccl is inplace op, this return result is not correct if graph loop > 1 _, pynccl_graph_time = _bench_graph_time( - lambda inp: pynccl_allreduce(inp, pynccl_comm), inp_randn + lambda inp: pynccl_allreduce(inp, pynccl_comm), + inp_randn, + tp_group, ) torch.testing.assert_close(torch_eager_output, msccl_graph_output) torch.testing.assert_close(torch_eager_output, msccl_eager_output) diff --git a/benchmark/kernels/all_reduce/benchmark_torch_symm_mem.py b/benchmark/kernels/all_reduce/benchmark_torch_symm_mem.py index 5bdb7f5d687d..b13d859ff3e6 100644 --- a/benchmark/kernels/all_reduce/benchmark_torch_symm_mem.py +++ b/benchmark/kernels/all_reduce/benchmark_torch_symm_mem.py @@ -39,11 +39,12 @@ TorchSymmMemCommunicator, ) from sglang.srt.distributed.parallel_state import ( - get_tensor_model_parallel_group, graph_capture, initialize_model_parallel, set_torch_symm_mem_all_reduce, ) +from sglang.srt.runtime_context import get_parallel +from sglang.test.test_utils import publish_build_topology from sglang.utils import is_in_ci IS_CI = is_in_ci() @@ -188,11 +189,12 @@ def print_markdown_table(data): rank=rank, local_rank=rank % 8, ) - initialize_model_parallel(tensor_model_parallel_size=world_size) - group = get_tensor_model_parallel_group().device_group - cpu_group = get_tensor_model_parallel_group().cpu_group - pynccl_comm = get_tensor_model_parallel_group().pynccl_comm - torch_symm_mem_comm = get_tensor_model_parallel_group().torch_symm_mem_comm + publish_build_topology(world_rank=rank, tp_size=world_size) + initialize_model_parallel() + group = get_parallel().tp_group.device_group + cpu_group = get_parallel().tp_group.cpu_group + pynccl_comm = get_parallel().tp_group.pynccl_comm + torch_symm_mem_comm = get_parallel().tp_group.torch_symm_mem_comm dist.barrier() profile = False dtype = torch.bfloat16 diff --git a/benchmark/kernels/attention/bench_gdn_replayssm_decode.py b/benchmark/kernels/attention/bench_gdn_replayssm_decode.py index c27ef70c4af2..b4659095a2f0 100644 --- a/benchmark/kernels/attention/bench_gdn_replayssm_decode.py +++ b/benchmark/kernels/attention/bench_gdn_replayssm_decode.py @@ -13,6 +13,13 @@ small ring append (d:[HV,V], k:[H,K], g:[HV] per step). The amortized state traffic ratio is reported per L. +Because the write happens only 1-in-L steps, the latency has to be amortized the +same way. The kernel branches on the caller's `write_pos` cursor and folds the +ring into the checkpoint only at `write_pos == L-1`, so both phases are timed and +combined: a cycle is L-1 non-flush steps and one flush step, and the flush step +costs roughly 2.3x a non-flush one. Timing a single pinned cursor position would +report whichever phase was pinned as if it were the per-step cost. + Run:: python benchmark/kernels/attention/bench_gdn_replayssm_decode.py @@ -62,7 +69,24 @@ def _state_bytes_per_step(B, HV, K, V, L, dtype): return packed, replay -def _bench_cfg(B, H, HV, K, V, Ls, dtype, device, num_slots=None, warmup=25, rep=100): +def _amortize(t_nonflush, t_flush, L): + """Mean per-step latency over one full L-phase cycle. + + `write_pos` walks 0, 1, ... L-1 and wraps, so a steady-state cycle is L-1 + non-flush steps and one flush step. The non-flush steps are not identical -- + each replays one more ring entry than the last -- so the midpoint phase is + sampled as their representative rather than phase 0, which replays nothing + and is the cheapest step in the cycle. Against a full per-phase sweep this + is accurate to <1%, where using phase 0 understates the mean by ~3%. + """ + if L == 1: + return t_flush + return ((L - 1) * t_nonflush + t_flush) / L + + +def _bench_cfg( + B, H, HV, K, V, Ls, dtype, device, nk=2, num_slots=None, warmup=25, rep=100 +): num_slots = num_slots or B mixed_qkv, a, b, A_log, dt_bias = _make_static(B, H, HV, K, V, dtype, device) scale = K**-0.5 @@ -94,9 +118,8 @@ def run_packed(): d_cache = torch.zeros(num_slots, HV, L, V, device=device, dtype=dtype) k_cache = torch.zeros(num_slots, H, L, K, device=device, dtype=dtype) g_cache = torch.zeros(num_slots, HV, L, device=device, dtype=torch.float32) - write_pos = torch.zeros(B, device=device, dtype=torch.int32) + write_pos = torch.empty(B, device=device, dtype=torch.int32) rout = mixed_qkv.new_empty(B, 1, HV, V) - nk = 1 if L == 1 else 2 def run_replay(): fused_recurrent_gdn_replayssm_decode( @@ -117,9 +140,29 @@ def run_replay(): nk=nk, ) - t_replay = triton.testing.do_bench(run_replay, warmup=warmup, rep=rep) + def time_at(pos): + write_pos.fill_(pos) + return triton.testing.do_bench(run_replay, warmup=warmup, rep=rep) + + # The kernel branches on write_pos: it appends to the ring on every step + # but folds the ring into the checkpoint only when write_pos == L-1. Both + # phases have to be timed, or the reported cost is the cheap one repeated + # (see `_amortize`). L == 1 is all flush, so it has no non-flush phase. + t_flush = time_at(L - 1) + t_nonflush = time_at((L - 1) // 2) if L > 1 else None + t_replay = _amortize(t_nonflush, t_flush, L) + packed_bytes, replay_bytes = _state_bytes_per_step(B, HV, K, V, L, dtype) - rows.append((L, t_replay, t_packed / t_replay, replay_bytes / packed_bytes)) + rows.append( + ( + L, + t_replay, + t_nonflush, + t_flush, + t_packed / t_replay, + replay_bytes / packed_bytes, + ) + ) return t_packed, rows @@ -132,6 +175,9 @@ def main(): parser.add_argument("--batch-sizes", type=int, nargs="+", default=[1, 16, 64, 256]) parser.add_argument("--ls", type=int, nargs="+", default=[1, 8, 16]) parser.add_argument("--dtype", choices=["bf16", "fp16", "fp32"], default="bf16") + parser.add_argument( + "--nk", type=int, default=2, help="K-chunks the reconstruction walks" + ) args = parser.parse_args() if not torch.cuda.is_available(): @@ -145,20 +191,26 @@ def main(): print( f"GDN ReplaySSM decode microbench HV={args.hv} H={args.h} " - f"K={args.k} V={args.v} dtype={args.dtype}\n" - "per-step latency (ms); speedup = packed/replay; " - "state-traffic = replay/packed (lower is better)" + f"K={args.k} V={args.v} dtype={args.dtype} nk={args.nk}\n" + "replay = mean per-step latency over one L-phase cycle (ms), split into " + "its\nnon-flush and flush steps; speedup = packed/replay; " + "state-traffic = replay/packed\n(lower is better)" ) for B in args.batch_sizes: t_packed, rows = _bench_cfg( - B, args.h, args.hv, args.k, args.v, args.ls, dtype, device + B, args.h, args.hv, args.k, args.v, args.ls, dtype, device, nk=args.nk ) print(f"\nB={B:<4d} packed={t_packed:.4f} ms") - for L, t_replay, speedup, traffic_ratio in rows: + for L, t_replay, t_nonflush, t_flush, speedup, traffic_ratio in rows: + split = ( + f"(flush only)" + if t_nonflush is None + else f"(non-flush {t_nonflush:.4f}, flush {t_flush:.4f})" + ) print( f" L={L:<3d} replay={t_replay:.4f} ms " f"speedup={speedup:5.2f}x " - f"state-traffic={traffic_ratio:5.2f}x" + f"state-traffic={traffic_ratio:5.2f}x {split}" ) diff --git a/benchmark/kernels/attention/fa4_benchmark_utils.py b/benchmark/kernels/attention/fa4_benchmark_utils.py index 8f5884777207..749ca50a76be 100644 --- a/benchmark/kernels/attention/fa4_benchmark_utils.py +++ b/benchmark/kernels/attention/fa4_benchmark_utils.py @@ -32,250 +32,3 @@ def amp_wrapper(*inputs, **kwinputs): if verbose: print(m) return t, m - - -def benchmark_backward( - fn, - *inputs, - grad=None, - repeats=10, - desc="", - verbose=True, - amp=False, - amp_dtype=torch.float16, - **kwinputs, -): - """Use Pytorch Benchmark on the backward pass of an arbitrary function.""" - if verbose: - print(desc, "- Backward pass") - with torch.autocast(device_type="cuda", dtype=amp_dtype, enabled=amp): - y = fn(*inputs, **kwinputs) - if type(y) is tuple: - y = y[0] - if grad is None: - grad = torch.randn_like(y) - else: - if grad.shape != y.shape: - raise RuntimeError("Grad shape does not match output shape") - - def f(*inputs, y, grad): - # Set .grad to None to avoid extra operation of gradient accumulation - for x in inputs: - if isinstance(x, torch.Tensor): - x.grad = None - y.backward(grad, retain_graph=True) - - t = benchmark.Timer( - stmt="f(*inputs, y=y, grad=grad)", - globals={"f": f, "inputs": inputs, "y": y, "grad": grad}, - num_threads=torch.get_num_threads(), - ) - m = t.timeit(repeats) - if verbose: - print(m) - return t, m - - -def benchmark_combined( - fn, - *inputs, - grad=None, - repeats=10, - desc="", - verbose=True, - amp=False, - amp_dtype=torch.float16, - **kwinputs, -): - """Use Pytorch Benchmark on the forward+backward pass of an arbitrary function.""" - if verbose: - print(desc, "- Forward + Backward pass") - with torch.autocast(device_type="cuda", dtype=amp_dtype, enabled=amp): - y = fn(*inputs, **kwinputs) - if type(y) is tuple: - y = y[0] - if grad is None: - grad = torch.randn_like(y) - else: - if grad.shape != y.shape: - raise RuntimeError("Grad shape does not match output shape") - - def f(grad, *inputs, **kwinputs): - for x in inputs: - if isinstance(x, torch.Tensor): - x.grad = None - with torch.autocast(device_type="cuda", dtype=amp_dtype, enabled=amp): - y = fn(*inputs, **kwinputs) - if type(y) is tuple: - y = y[0] - y.backward(grad, retain_graph=True) - - t = benchmark.Timer( - stmt="f(grad, *inputs, **kwinputs)", - globals={ - "f": f, - "fn": fn, - "inputs": inputs, - "grad": grad, - "kwinputs": kwinputs, - }, - num_threads=torch.get_num_threads(), - ) - m = t.timeit(repeats) - if verbose: - print(m) - return t, m - - -def benchmark_fwd_bwd( - fn, - *inputs, - grad=None, - repeats=10, - desc="", - verbose=True, - amp=False, - amp_dtype=torch.float16, - **kwinputs, -): - """Use Pytorch Benchmark on the forward+backward pass of an arbitrary function.""" - return ( - benchmark_forward( - fn, - *inputs, - repeats=repeats, - desc=desc, - verbose=verbose, - amp=amp, - amp_dtype=amp_dtype, - **kwinputs, - ), - benchmark_backward( - fn, - *inputs, - grad=grad, - repeats=repeats, - desc=desc, - verbose=verbose, - amp=amp, - amp_dtype=amp_dtype, - **kwinputs, - ), - ) - - -def benchmark_all( - fn, - *inputs, - grad=None, - repeats=10, - desc="", - verbose=True, - amp=False, - amp_dtype=torch.float16, - **kwinputs, -): - """Use Pytorch Benchmark on the forward+backward pass of an arbitrary function.""" - return ( - benchmark_forward( - fn, - *inputs, - repeats=repeats, - desc=desc, - verbose=verbose, - amp=amp, - amp_dtype=amp_dtype, - **kwinputs, - ), - benchmark_backward( - fn, - *inputs, - grad=grad, - repeats=repeats, - desc=desc, - verbose=verbose, - amp=amp, - amp_dtype=amp_dtype, - **kwinputs, - ), - benchmark_combined( - fn, - *inputs, - grad=grad, - repeats=repeats, - desc=desc, - verbose=verbose, - amp=amp, - amp_dtype=amp_dtype, - **kwinputs, - ), - ) - - -def pytorch_profiler( - fn, - *inputs, - trace_filename=None, - backward=False, - amp=False, - amp_dtype=torch.float16, - cpu=False, - verbose=True, - **kwinputs, -): - """Wrap benchmark functions in Pytorch profiler to see CUDA information.""" - if backward: - with torch.autocast(device_type="cuda", dtype=amp_dtype, enabled=amp): - out = fn(*inputs, **kwinputs) - if type(out) is tuple: - out = out[0] - g = torch.randn_like(out) - for _ in range(30): # Warm up - if backward: - for x in inputs: - if isinstance(x, torch.Tensor): - x.grad = None - with torch.autocast(device_type="cuda", dtype=amp_dtype, enabled=amp): - out = fn(*inputs, **kwinputs) - if type(out) is tuple: - out = out[0] - # Backward should be done outside autocast - if backward: - out.backward(g, retain_graph=True) - activities = ([torch.profiler.ProfilerActivity.CPU] if cpu else []) + [ - torch.profiler.ProfilerActivity.CUDA - ] - with torch.profiler.profile( - activities=activities, - record_shapes=True, - # profile_memory=True, - with_stack=True, - ) as prof: - if backward: - for x in inputs: - if isinstance(x, torch.Tensor): - x.grad = None - with torch.autocast(device_type="cuda", dtype=amp_dtype, enabled=amp): - out = fn(*inputs, **kwinputs) - if type(out) is tuple: - out = out[0] - if backward: - out.backward(g, retain_graph=True) - if verbose: - # print(prof.key_averages().table(sort_by="self_cuda_time_total", row_limit=50)) - print(prof.key_averages().table(row_limit=50)) - if trace_filename is not None: - prof.export_chrome_trace(trace_filename) - - -def benchmark_memory(fn, *inputs, desc="", verbose=True, **kwinputs): - torch.cuda.empty_cache() - torch.cuda.reset_peak_memory_stats() - torch.cuda.synchronize() - fn(*inputs, **kwinputs) - torch.cuda.synchronize() - mem = torch.cuda.max_memory_allocated() / ((2**20) * 1000) - if verbose: - print(f"{desc} max memory: {mem}GB") - torch.cuda.empty_cache() - return mem diff --git a/benchmark/kernels/bench_fused_rmsnorm_fp8_quant.py b/benchmark/kernels/bench_fused_rmsnorm_fp8_quant.py index 53e0b8c757e1..f3230134fb7d 100644 --- a/benchmark/kernels/bench_fused_rmsnorm_fp8_quant.py +++ b/benchmark/kernels/bench_fused_rmsnorm_fp8_quant.py @@ -149,9 +149,9 @@ def _check_correctness(): cos = torch.nn.functional.cosine_similarity( (q.float() * scale).flatten(), ref_deq.flatten(), dim=0 ).item() - assert ( - cos > 0.99 - ), f"{provider} h={hidden_size} residual={add_residual} cos={cos:.4f}" + assert cos > 0.99, ( + f"{provider} h={hidden_size} residual={add_residual} cos={cos:.4f}" + ) print("correctness check passed (all fused providers vs unfused within FP8)") diff --git a/benchmark/kernels/bench_kda_verify_sweep.py b/benchmark/kernels/bench_kda_verify_sweep.py new file mode 100644 index 000000000000..2990acd50549 --- /dev/null +++ b/benchmark/kernels/bench_kda_verify_sweep.py @@ -0,0 +1,285 @@ +"""Sweep benchmark for the KDA chain-verify kernels (one layer, in-graph). + +Compares the four target-verify variants the KDA backend can dispatch: + + unfused causal_conv1d_update + recurrence, per-step ssm snapshots + unfused+ring causal_conv1d_update + recurrence, ReplaySSM CACHE_RING + fused fused_kda_conv_gating_verify, per-step ssm snapshots + fused+ring fused_kda_conv_gating_verify, ReplaySSM CACHE_RING + +Timing replays a CUDA graph capturing GRAPH_BATCH calls, matching how the +production verify runs (in-graph; bare launches would drown these ~10us +kernels in launch overhead). Imports only sglang.kernels.*, so it runs on +boxes where the sglang.srt/test import chain is broken. + + PYTHONPATH=python python3 benchmark/kernels/bench_kda_verify_sweep.py + ... --batch-sizes 1 4 16 64 --modes fused fused+ring + ... --sweep-bv # re-tune KDA_VERIFY_BLOCK_V per mode/batch + ... --hv-heads 16 # GQA shape (HV != H) +""" + +import argparse + +import torch + +import sglang.kernels.ops.attention.fla.fused_kda_conv_recurrent_verify as fused_mod +from sglang.kernels.ops.attention.fla.fused_kda_conv_recurrent_verify import ( + fused_kda_conv_gating_verify, +) +from sglang.kernels.ops.attention.fla.fused_sigmoid_gating_recurrent import ( + fused_sigmoid_gating_delta_rule_update, +) +from sglang.kernels.ops.mamba.causal_conv1d_triton import causal_conv1d_update + +_DEVICE = "cuda" +_DTYPE = torch.bfloat16 +_W = 4 +# Ring length: power of two >= 2 * draft tokens (memory_pool.py invariant). +_RING_LEN = 16 +_MODES = ("unfused", "unfused+ring", "fused", "fused+ring") +GRAPH_BATCH = 10 + + +def make_inputs(B, T, H, HV, K, V, seed=0): + torch.manual_seed(seed) + dim = 2 * H * K + HV * V + seq_len = B * T + lines = slots = B + 1 + rnd = lambda *s, dt=_DTYPE: torch.randn(*s, device=_DEVICE, dtype=dt) + return { + "mixed": rnd(seq_len, dim) * 0.5, + "w": rnd(dim, _W) * 0.3, + "bias": rnd(dim) * 0.1, + "a": rnd(seq_len, HV * K) * 0.5, + "b": rnd(seq_len, HV), + "A_log": rnd(HV, dt=torch.float32) * 0.5, + "dt_bias": rnd(HV * K, dt=torch.float32) * 0.5, + "conv_pool": rnd(lines, _W - 1, dim), + "ssm": rnd(slots, HV, V, K, dt=torch.float32) * 0.2, + "win_pool": torch.zeros(lines, T, _W - 1, dim, device=_DEVICE, dtype=_DTYPE), + "inter_ssm": torch.zeros( + lines, T, HV, V, K, device=_DEVICE, dtype=torch.float32 + ), + "rawv": rnd(slots, HV, _RING_LEN, V), + "rawk": rnd(slots, H, _RING_LEN, K), + "g": rnd(slots, HV, _RING_LEN, K, dt=torch.float32), + "beta": rnd(slots, HV, _RING_LEN, dt=torch.float32), + "cache_indices": torch.arange(B, device=_DEVICE, dtype=torch.int32), + "inter_indices": torch.arange(B, device=_DEVICE, dtype=torch.int32), + "cu": torch.arange(0, B + 1, device=_DEVICE, dtype=torch.int32) * T, + } + + +def _ring_kwargs(inp, on): + return dict( + cache_ring=on, + replayssm_rawv=inp["rawv"] if on else None, + replayssm_rawk=inp["rawk"] if on else None, + replayssm_g=inp["g"] if on else None, + replayssm_beta=inp["beta"] if on else None, + ) + + +def make_runner(mode, inp, B, T, H, HV, K, V, lower_bound=None): + dim = 2 * H * K + HV * V + seq_len = B * T + ring = mode.endswith("+ring") + scale = K**-0.5 + + if mode.startswith("fused"): + + def fn(): + fused_kda_conv_gating_verify( + mixed_qkv=inp["mixed"], + conv_weight=inp["w"], + conv_bias=inp["bias"], + conv_state=inp["conv_pool"].transpose(-1, -2), + conv_state_indices=inp["cache_indices"], + intermediate_conv_window=inp["win_pool"].transpose(-1, -2), + intermediate_state_indices=inp["inter_indices"], + a=inp["a"], + b=inp["b"], + A_log=inp["A_log"], + dt_bias=inp["dt_bias"], + ssm_states=inp["ssm"], + cache_indices=inp["cache_indices"], + intermediate_states_buffer=None if ring else inp["inter_ssm"], + scale=scale, + T=T, + num_q_heads=H, + num_v_heads=HV, + head_k_dim=K, + head_v_dim=V, + lower_bound=lower_bound, + **_ring_kwargs(inp, ring), + ) + + return fn + + def fn(): + x3 = inp["mixed"].reshape(B, T, dim).transpose(1, 2) + out3 = causal_conv1d_update( + x3, + inp["conv_pool"].transpose(-1, -2), + inp["w"], + inp["bias"], + activation="silu", + conv_state_indices=inp["cache_indices"], + intermediate_conv_window=inp["win_pool"].transpose(-1, -2), + intermediate_state_indices=inp["inter_indices"], + ) + mixed_out = out3.transpose(1, 2).reshape(seq_len, dim) + q, k, v = mixed_out.split([H * K, H * K, HV * V], dim=-1) + fused_sigmoid_gating_delta_rule_update( + A_log=inp["A_log"], + a=inp["a"], + dt_bias=inp["dt_bias"], + softplus_beta=1.0, + softplus_threshold=20.0, + q=q.unflatten(-1, (H, K)).unsqueeze(0), + k=k.unflatten(-1, (H, K)).unsqueeze(0), + v=v.unflatten(-1, (HV, V)).unsqueeze(0), + b=inp["b"], + initial_state_source=inp["ssm"], + initial_state_indices=inp["cache_indices"], + use_qk_l2norm_in_kernel=True, + cu_seqlens=inp["cu"], + is_kda=True, + disable_state_update=True, + intermediate_states_buffer=None if ring else inp["inter_ssm"], + intermediate_state_indices=None if ring else inp["inter_indices"], + cache_steps=T, + retrieve_parent_token=None, + lower_bound=lower_bound, + **_ring_kwargs(inp, ring), + ) + + return fn + + +def bench_graph(fn, iters=200): + """us per call, timed as CUDA-graph replays of GRAPH_BATCH captured calls.""" + for _ in range(3): # compile outside capture + fn() + torch.cuda.synchronize() + graph = torch.cuda.CUDAGraph() + with torch.cuda.graph(graph): + for _ in range(GRAPH_BATCH): + fn() + for _ in range(5): + graph.replay() + torch.cuda.synchronize() + start = torch.cuda.Event(enable_timing=True) + end = torch.cuda.Event(enable_timing=True) + start.record() + for _ in range(iters): + graph.replay() + end.record() + torch.cuda.synchronize() + graph.reset() + return start.elapsed_time(end) * 1e3 / (iters * GRAPH_BATCH) + + +def check_ring_bitwise(B, T, H, HV, K, V, lower_bound=None): + """One-shot guard: fused+ring must fill the same ring bytes as unfused+ring.""" + ref, fus = (make_inputs(B, T, H, HV, K, V, seed=7) for _ in range(2)) + make_runner("unfused+ring", ref, B, T, H, HV, K, V, lower_bound)() + make_runner("fused+ring", fus, B, T, H, HV, K, V, lower_bound)() + torch.cuda.synchronize() + for name in ("rawv", "rawk", "g", "beta"): + assert torch.equal(ref[name], fus[name]), f"ring mismatch: {name}" + + +def run_modes(args, label_extra=""): + print( + f"H={args.heads} HV={args.hv_heads} K={args.head_k_dim} V={args.head_v_dim} " + f"T={args.draft_tokens} gate={'safe' if args.lower_bound is not None else 'std'} " + f"BV={fused_mod.KDA_VERIFY_BLOCK_V}{label_extra}" + ) + header = f"{'B':>4} " + "".join(f"{m:>14}" for m in args.modes) + print(header) + for B in args.batch_sizes: + times = [] + for mode in args.modes: + inp = make_inputs( + B, + args.draft_tokens, + args.heads, + args.hv_heads, + args.head_k_dim, + args.head_v_dim, + ) + fn = make_runner( + mode, + inp, + B, + args.draft_tokens, + args.heads, + args.hv_heads, + args.head_k_dim, + args.head_v_dim, + args.lower_bound, + ) + times.append(bench_graph(fn, iters=args.iters)) + row = f"{B:>4} " + "".join(f"{t:>11.2f} us" for t in times) + print(row) + print() + + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument("--heads", type=int, default=8) # ling-v3 TP4 KDA shape + parser.add_argument("--hv-heads", type=int, default=None) + parser.add_argument("--head-k-dim", type=int, default=128) + parser.add_argument("--head-v-dim", type=int, default=128) + parser.add_argument("--draft-tokens", type=int, default=4) + # ling-v3 runs the safe gate: --lower-bound -5.0 (kda_lower_bound). + parser.add_argument("--lower-bound", type=float, default=None) + parser.add_argument( + "--batch-sizes", type=int, nargs="+", default=[1, 2, 4, 8, 16, 32, 64] + ) + parser.add_argument("--modes", nargs="+", default=list(_MODES), choices=_MODES) + parser.add_argument("--iters", type=int, default=200) + parser.add_argument( + "--sweep-bv", + action="store_true", + help="re-run the fused modes across KDA_VERIFY_BLOCK_V candidates; " + "BLOCK_V was tuned with snapshot writes on, so ring mode may move it", + ) + parser.add_argument("--skip-check", action="store_true") + args = parser.parse_args() + if args.hv_heads is None: + args.hv_heads = args.heads + if args.draft_tokens * 2 > _RING_LEN: + raise ValueError(f"--draft-tokens > {_RING_LEN // 2} exceeds the bench ring") + + if not args.skip_check: + check_ring_bitwise( + 4, + args.draft_tokens, + args.heads, + args.hv_heads, + args.head_k_dim, + args.head_v_dim, + args.lower_bound, + ) + print("ring bitwise check: OK\n") + + run_modes(args) + + if args.sweep_bv: + args.modes = [m for m in args.modes if m.startswith("fused")] or [ + "fused", + "fused+ring", + ] + default_bv = fused_mod.KDA_VERIFY_BLOCK_V + try: + for bv in (2, 4, 8, 16, 32): + fused_mod.KDA_VERIFY_BLOCK_V = bv + run_modes(args, label_extra=" (BV sweep)") + finally: + fused_mod.KDA_VERIFY_BLOCK_V = default_bv + + +if __name__ == "__main__": + main() diff --git a/benchmark/kernels/deepep/deepep_utils.py b/benchmark/kernels/deepep/deepep_utils.py deleted file mode 100644 index 169529ef6cca..000000000000 --- a/benchmark/kernels/deepep/deepep_utils.py +++ /dev/null @@ -1,218 +0,0 @@ -# ADAPTED FROM https://github.com/deepseek-ai/DeepEP/blob/main/tests/utils.py - -import os -import sys -from typing import Optional - -import numpy as np -import torch -import torch.distributed as dist - - -def init_dist(local_rank: int, num_local_ranks: int, args): - ip = args.master_addr - port = args.master_port - num_nodes = args.nnodes - node_rank = args.node_rank - assert (num_local_ranks < 8 and num_nodes == 1) or num_local_ranks == 8 - - dist.init_process_group( - backend="nccl", - init_method=f"tcp://{ip}:{port}", - world_size=num_nodes * num_local_ranks, - rank=node_rank * num_local_ranks + local_rank, - ) - torch.set_default_dtype(torch.bfloat16) - torch.set_default_device("cuda") - torch.cuda.set_device(local_rank) - - return ( - dist.get_rank(), - dist.get_world_size(), - dist.new_group(list(range(num_local_ranks * num_nodes))), - ) - - -def calc_diff(x: torch.Tensor, y: torch.Tensor): - x, y = x.double() + 1, y.double() + 1 - denominator = (x * x + y * y).sum() - sim = 2 * (x * y).sum() / denominator - return (1 - sim).item() - - -def per_token_cast_to_fp8(x: torch.Tensor): - assert x.dim() == 2 and x.size(1) % 128 == 0 - m, n = x.shape - x_view = x.view(m, -1, 128) - x_amax = x_view.abs().float().amax(dim=2).view(m, -1).clamp(1e-4) - return (x_view * (448.0 / x_amax.unsqueeze(2))).to(torch.float8_e4m3fn).view( - m, n - ), (x_amax / 448.0).view(m, -1) - - -def per_token_cast_back(x_fp8: torch.Tensor, x_scales: torch.Tensor): - x_fp32 = x_fp8.to(torch.float32).view(x_fp8.size(0), -1, 128) - x_scales = x_scales.view(x_fp8.size(0), -1, 1) - return (x_fp32 * x_scales).view(x_fp8.shape).to(torch.bfloat16) - - -def inplace_unique(x: torch.Tensor, num_slots: int): - assert x.dim() == 2 - mask = x < 0 - x_padded = x.masked_fill(mask, num_slots) - bin_count = torch.zeros((x.size(0), num_slots + 1), dtype=x.dtype, device=x.device) - bin_count.scatter_add_(1, x_padded, torch.ones_like(x_padded)) - bin_count = bin_count[:, :num_slots] - sorted_bin_count, sorted_bin_idx = torch.sort(bin_count, dim=-1, descending=True) - sorted_bin_idx.masked_fill_(sorted_bin_count == 0, -1) - sorted_bin_idx = torch.sort(sorted_bin_idx, descending=True, dim=-1).values - x[:, :].fill_(-1) - valid_len = min(num_slots, x.size(1)) - x[:, :valid_len] = sorted_bin_idx[:, :valid_len] - - -def create_grouped_scores( - scores: torch.Tensor, group_idx: torch.Tensor, num_groups: int -): - num_tokens, num_experts = scores.shape - scores = scores.view(num_tokens, num_groups, -1) - mask = torch.zeros((num_tokens, num_groups), dtype=torch.bool, device=scores.device) - mask = mask.scatter_(1, group_idx, True).unsqueeze(-1).expand_as(scores) - return (scores * mask).view(num_tokens, num_experts) - - -def bench(fn, num_warmups: int = 20, num_tests: int = 30, post_fn=None): - # Flush L2 cache with 256 MB data - torch.cuda.synchronize() - cache = torch.empty(int(256e6 // 4), dtype=torch.int, device="cuda") - - # Warmup - for _ in range(num_warmups): - fn() - - # Flush L2 - cache.zero_() - - # Testing - start_events = [torch.cuda.Event(enable_timing=True) for _ in range(num_tests)] - end_events = [torch.cuda.Event(enable_timing=True) for _ in range(num_tests)] - for i in range(num_tests): - # Record - start_events[i].record() - fn() - end_events[i].record() - if post_fn is not None: - post_fn() - torch.cuda.synchronize() - - times = np.array( - [s.elapsed_time(e) / 1e3 for s, e in zip(start_events, end_events)] - )[1:] - return np.average(times), np.min(times), np.max(times) - - -class empty_suppress: - def __enter__(self): - return self - - def __exit__(self, *_): - pass - - -class suppress_stdout_stderr: - def __enter__(self): - self.outnull_file = open(os.devnull, "w") - self.errnull_file = open(os.devnull, "w") - - self.old_stdout_fileno_undup = sys.stdout.fileno() - self.old_stderr_fileno_undup = sys.stderr.fileno() - - self.old_stdout_fileno = os.dup(sys.stdout.fileno()) - self.old_stderr_fileno = os.dup(sys.stderr.fileno()) - - self.old_stdout = sys.stdout - self.old_stderr = sys.stderr - - os.dup2(self.outnull_file.fileno(), self.old_stdout_fileno_undup) - os.dup2(self.errnull_file.fileno(), self.old_stderr_fileno_undup) - - sys.stdout = self.outnull_file - sys.stderr = self.errnull_file - return self - - def __exit__(self, *_): - sys.stdout = self.old_stdout - sys.stderr = self.old_stderr - - os.dup2(self.old_stdout_fileno, self.old_stdout_fileno_undup) - os.dup2(self.old_stderr_fileno, self.old_stderr_fileno_undup) - - os.close(self.old_stdout_fileno) - os.close(self.old_stderr_fileno) - - self.outnull_file.close() - self.errnull_file.close() - - -def bench_kineto( - fn, - kernel_names, - num_tests: int = 30, - suppress_kineto_output: bool = False, - trace_path: Optional[str] = None, - barrier_comm_profiling: bool = False, -): - # Profile - suppress = suppress_stdout_stderr if suppress_kineto_output else empty_suppress - with suppress(): - schedule = torch.profiler.schedule(wait=0, warmup=1, active=1, repeat=1) - with torch.profiler.profile( - activities=[torch.profiler.ProfilerActivity.CUDA], schedule=schedule - ) as prof: - for i in range(2): - # NOTES: use a large kernel and a barrier to eliminate the unbalanced CPU launch overhead - if barrier_comm_profiling: - lhs = torch.randn((8192, 8192), dtype=torch.float, device="cuda") - rhs = torch.randn((8192, 8192), dtype=torch.float, device="cuda") - lhs @ rhs - dist.all_reduce(torch.ones(1, dtype=torch.float, device="cuda")) - for _ in range(num_tests): - fn() - prof.step() - - # Parse the profiling table - assert isinstance(kernel_names, str) or isinstance(kernel_names, tuple) - is_tupled = isinstance(kernel_names, tuple) - prof_lines = ( - prof.key_averages() - .table(sort_by="cuda_time_total", max_name_column_width=100) - .split("\n") - ) - kernel_names = (kernel_names,) if isinstance(kernel_names, str) else kernel_names - assert all([isinstance(name, str) for name in kernel_names]) - for name in kernel_names: - assert ( - sum([name in line for line in prof_lines]) == 1 - ), f"Errors of the kernel {name} in the profiling table" - - # Save chrome traces - if trace_path is not None: - prof.export_chrome_trace(trace_path) - - # Return average kernel times - units = {"ms": 1e3, "us": 1e6} - kernel_times = [] - for name in kernel_names: - for line in prof_lines: - if name in line: - time_str = line.split()[-2] - for unit, scale in units.items(): - if unit in time_str: - kernel_times.append(float(time_str.replace(unit, "")) / scale) - break - break - return tuple(kernel_times) if is_tupled else kernel_times[0] - - -def hash_tensor(t: torch.Tensor): - return t.view(torch.int64).sum().item() diff --git a/benchmark/kernels/deepep/tuning_deepep.py b/benchmark/kernels/deepep/tuning_deepep.py index 191819d2cf30..97dd14504dbb 100644 --- a/benchmark/kernels/deepep/tuning_deepep.py +++ b/benchmark/kernels/deepep/tuning_deepep.py @@ -16,17 +16,41 @@ import deep_ep import torch import torch.distributed as dist -from deepep_utils import ( + +from sglang.test.test_deepep_utils import ( bench, calc_diff, create_grouped_scores, - init_dist, inplace_unique, per_token_cast_back, per_token_cast_to_fp8, ) +def init_dist(local_rank: int, num_local_ranks: int, args): + ip = args.master_addr + port = args.master_port + num_nodes = args.nnodes + node_rank = args.node_rank + assert (num_local_ranks < 8 and num_nodes == 1) or num_local_ranks == 8 + + dist.init_process_group( + backend="nccl", + init_method=f"tcp://{ip}:{port}", + world_size=num_nodes * num_local_ranks, + rank=node_rank * num_local_ranks + local_rank, + ) + torch.set_default_dtype(torch.bfloat16) + torch.set_default_device("cuda") + torch.cuda.set_device(local_rank) + + return ( + dist.get_rank(), + dist.get_world_size(), + dist.new_group(list(range(num_local_ranks * num_nodes))), + ) + + def test_main( num_sms: int, local_rank: int, @@ -155,7 +179,7 @@ def check_data(check_x, recv_gbl_rank_prefix_sum): for with_topk in (False, True): if local_rank == 0: print( - f'[testing] Running with {"FP8" if isinstance(current_x, tuple) else "BF16"}, {"with" if with_topk else "without"} top-k (async={async_mode}, previous={previous_mode}) ...', + f"[testing] Running with {'FP8' if isinstance(current_x, tuple) else 'BF16'}, {'with' if with_topk else 'without'} top-k (async={async_mode}, previous={previous_mode}) ...", flush=True, end="", ) @@ -198,9 +222,9 @@ def check_data(check_x, recv_gbl_rank_prefix_sum): # Checks recv_gbl_rank_prefix_sum = handle[-4] - assert gbl_num_tokens_per_rank[rank].item() == recv_x.size( - 0 - ), f"{gbl_num_tokens_per_rank[rank].item()} != {recv_x.size(0)}" + assert gbl_num_tokens_per_rank[rank].item() == recv_x.size(0), ( + f"{gbl_num_tokens_per_rank[rank].item()} != {recv_x.size(0)}" + ) assert ( gbl_num_tokens_per_expert.view(num_ranks, -1)[rank].tolist() == recv_num_tokens_per_expert_list @@ -325,11 +349,14 @@ def check_data(check_x, recv_gbl_rank_prefix_sum): tune_args = {"x": current_x, "handle": handle, "config": config} t = bench(lambda: buffer.dispatch(**tune_args))[0] if t < best_time: - best_time, best_results = t, ( - num_sms, - nvl_chunk_size, - rdma_chunk_size, - config_kwargs, + best_time, best_results = ( + t, + ( + num_sms, + nvl_chunk_size, + rdma_chunk_size, + config_kwargs, + ), ) if local_rank == 0: print( @@ -338,7 +365,7 @@ def check_data(check_x, recv_gbl_rank_prefix_sum): ) if local_rank == 0: print( - f'[tuning] Best dispatch ({"FP8" if isinstance(current_x, tuple) else "BF16"}): SMs {best_results[0]}, NVL chunk {best_results[1]}, RDMA chunk {best_results[2]}: {rdma_send_bytes / 1e9 / best_time:.2f} GB/s (RDMA), {nvl_recv_bytes / 1e9 / best_time:.2f} GB/s (NVL)', + f"[tuning] Best dispatch ({'FP8' if isinstance(current_x, tuple) else 'BF16'}): SMs {best_results[0]}, NVL chunk {best_results[1]}, RDMA chunk {best_results[2]}: {rdma_send_bytes / 1e9 / best_time:.2f} GB/s (RDMA), {nvl_recv_bytes / 1e9 / best_time:.2f} GB/s (NVL)", flush=True, ) print("", flush=True) @@ -399,11 +426,14 @@ def check_data(check_x, recv_gbl_rank_prefix_sum): flush=True, ) if t < best_time: - best_time, best_results = t, ( - num_sms, - nvl_chunk_size, - rdma_chunk_size, - config_kwargs, + best_time, best_results = ( + t, + ( + num_sms, + nvl_chunk_size, + rdma_chunk_size, + config_kwargs, + ), ) if local_rank == 0: diff --git a/benchmark/kernels/deepseek/benchmark_deepgemm_fp8_gemm.py b/benchmark/kernels/deepseek/benchmark_deepgemm_fp8_gemm.py index 0b958de68074..e994ea5df07f 100644 --- a/benchmark/kernels/deepseek/benchmark_deepgemm_fp8_gemm.py +++ b/benchmark/kernels/deepseek/benchmark_deepgemm_fp8_gemm.py @@ -1,17 +1,16 @@ -from typing import Tuple - import deep_gemm import tilelang import tilelang.language as T import torch import triton -from deep_gemm import ceil_div from deep_gemm.utils.layout import get_mn_major_tma_aligned_tensor -from vllm.model_executor.layers.quantization.utils.fp8_utils import ( - w8a8_block_fp8_matmul as vllm_w8a8_block_fp8_matmul, -) from sglang.benchmark.bench_utils import run_bench +from sglang.benchmark.deepseek_utils import ( + get_weight_shapes, + per_block_cast_to_fp8, + per_token_cast_to_fp8, +) from sglang.kernels.ops.quantization.fp8_kernel import ( w8a8_block_fp8_matmul_deepgemm as w8a8_block_fp8_matmul, ) @@ -59,7 +58,6 @@ def main( bx, by, ): - A_shared = T.alloc_shared(A_shared_shape, in_dtype) B_shared = T.alloc_shared(B_shared_shape, in_dtype) C_shared = T.alloc_shared(C_shared_shape, out_dtype) @@ -95,31 +93,6 @@ def main( return main -def per_token_cast_to_fp8(x: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]: - assert x.dim() == 2 and x.size(1) % 128 == 0 - m, n = x.shape - x_view = x.view(m, -1, 128) - x_amax = x_view.abs().float().amax(dim=2).view(m, -1).clamp(1e-4) - return (x_view * (448.0 / x_amax.unsqueeze(2))).to(torch.float8_e4m3fn).view( - m, n - ), (x_amax / 448.0).view(m, -1) - - -def per_block_cast_to_fp8(x: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]: - assert x.dim() == 2 - m, n = x.shape - x_padded = torch.zeros( - (ceil_div(m, 128) * 128, ceil_div(n, 128) * 128), dtype=x.dtype, device=x.device - ) - x_padded[:m, :n] = x - x_view = x_padded.view(-1, 128, x_padded.size(1) // 128, 128) - x_amax = x_view.abs().float().amax(dim=(1, 3), keepdim=True).clamp(1e-4) - x_scaled = (x_view * (448.0 / x_amax)).to(torch.float8_e4m3fn) - return x_scaled.view_as(x_padded)[:m, :n].contiguous(), (x_amax / 448.0).view( - x_view.size(0), x_view.size(2) - ) - - def fp8_gemm_deepgemm( x_fp8: torch.Tensor, x_scale: torch.Tensor, @@ -156,25 +129,6 @@ def fp8_gemm_sglang( return out -def fp8_gemm_vllm( - x_fp8: torch.Tensor, - x_scale: torch.Tensor, - y_fp8: torch.Tensor, - y_scale: torch.Tensor, - m: int, - n: int, - k: int, -): - """vLLM implementation of FP8 GEMM""" - block_size = [128, 128] # Matches the block size in per_block_cast_to_fp8 - - # Run vLLM kernel - out = vllm_w8a8_block_fp8_matmul( - x_fp8, y_fp8, x_scale, y_scale, block_size, torch.bfloat16 - ) - return out - - def calculate_diff(m: int, n: int, k: int): x = torch.randn((m, k), device="cuda", dtype=torch.bfloat16) y = torch.randn((n, k), device="cuda", dtype=torch.bfloat16) @@ -233,39 +187,6 @@ def calculate_diff(m: int, n: int, k: int): print(f" - TileLang vs SGLang: {'✅' if tilelang_sglang_match else '❌'}\n") -def get_weight_shapes(tp_size): - # cannot TP - total = [ - (512 + 64, 7168), - ((128 + 64) * 128, 7168), - (128 * (128 + 128), 512), - (7168, 16384), - (7168, 18432), - ] - # N can TP - n_tp = [ - (18432 * 2, 7168), - ((128 + 64) * 128, 7168), - (128 * (128 + 128), 512), - (24576, 1536), - (4096, 7168), - ] - # K can TP - k_tp = [(7168, 18432), (7168, 16384), (7168, 2048)] - - weight_shapes = [] - for t in total: - weight_shapes.append(t) - for n_t in n_tp: - new_t = (n_t[0] // tp_size, n_t[1]) - weight_shapes.append(new_t) - for k_t in k_tp: - new_t = (k_t[0], k_t[1] // tp_size) - weight_shapes.append(new_t) - - return weight_shapes - - def create_benchmark_configs(tp_size): configs = [] weight_shapes = get_weight_shapes(tp_size) @@ -350,7 +271,7 @@ def benchmark(m, n, k, tp_size, provider): tflops = flops / (ms * 1e-3) / 1e12 # Print shape-specific results with TFLOPS - print(f"Time: {ms*1000:.2f} ms, TFLOPS: {tflops:.2f}") + print(f"Time: {ms * 1000:.2f} ms, TFLOPS: {tflops:.2f}") return ms * 1000, max_ms * 1000, min_ms * 1000 # convert to ms return benchmark diff --git a/benchmark/kernels/deepseek/benchmark_deepgemm_fp8_gemm_blackwell.py b/benchmark/kernels/deepseek/benchmark_deepgemm_fp8_gemm_blackwell.py index 5670f3c33314..8fae14b1913e 100644 --- a/benchmark/kernels/deepseek/benchmark_deepgemm_fp8_gemm_blackwell.py +++ b/benchmark/kernels/deepseek/benchmark_deepgemm_fp8_gemm_blackwell.py @@ -1,12 +1,14 @@ import argparse -from typing import Tuple import torch import triton -from deep_gemm import ceil_div from flashinfer.gemm import gemm_fp8_nt_groupwise from sglang.benchmark.bench_utils import run_bench +from sglang.benchmark.deepseek_utils import ( + get_weight_shapes, + per_block_cast_to_fp8, +) from sglang.kernels.ops.quantization.fp8_kernel import ( sglang_per_token_group_quant_fp8, w8a8_block_fp8_matmul_deepgemm, @@ -16,55 +18,6 @@ BLOCK_SIZE = 128 -def per_block_cast_to_fp8(x: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]: - assert x.dim() == 2 - assert BLOCK_SIZE == 128 - m, n = x.shape - x_padded = torch.zeros( - (ceil_div(m, 128) * 128, ceil_div(n, 128) * 128), dtype=x.dtype, device=x.device - ) - x_padded[:m, :n] = x - x_view = x_padded.view(-1, 128, x_padded.size(1) // 128, 128) - x_amax = x_view.abs().float().amax(dim=(1, 3), keepdim=True).clamp(1e-4) - x_scaled = (x_view * (448.0 / x_amax)).to(torch.float8_e4m3fn) - return x_scaled.view_as(x_padded)[:m, :n].contiguous(), (x_amax / 448.0).view( - x_view.size(0), x_view.size(2) - ) - - -def get_weight_shapes(tp_size): - # cannot TP - total = [ - (512 + 64, 7168), - ((128 + 64) * 128, 7168), - (128 * (128 + 128), 512), - (7168, 16384), - (7168, 18432), - ] - # N can TP - n_tp = [ - (18432 * 2, 7168), - ((128 + 64) * 128, 7168), - (128 * (128 + 128), 512), - (24576, 1536), - (4096, 7168), - ] - # K can TP - k_tp = [(7168, 18432), (7168, 16384), (7168, 2048)] - - weight_shapes = [] - for t in total: - weight_shapes.append(t) - for n_t in n_tp: - new_t = (n_t[0] // tp_size, n_t[1]) - weight_shapes.append(new_t) - for k_t in k_tp: - new_t = (k_t[0], k_t[1] // tp_size) - weight_shapes.append(new_t) - - return weight_shapes - - def create_benchmark_configs(tp_size): configs = [] weight_shapes = get_weight_shapes(tp_size) @@ -224,7 +177,7 @@ def _benchmark(m, n, k, tp_size, provider): tflops = flops / (ms * 1e-3) / 1e12 # Print shape-specific results with TFLOPS - print(f"Time: {ms*1000:.2f} us, TFLOPS: {tflops:.2f}") + print(f"Time: {ms * 1000:.2f} us, TFLOPS: {tflops:.2f}") return ms, max_ms, min_ms diff --git a/benchmark/kernels/deepseek/benchmark_deepgemm_fp8_group_gemm.py b/benchmark/kernels/deepseek/benchmark_deepgemm_fp8_group_gemm.py index 8b1be7b888f9..4e2eaa864b91 100644 --- a/benchmark/kernels/deepseek/benchmark_deepgemm_fp8_group_gemm.py +++ b/benchmark/kernels/deepseek/benchmark_deepgemm_fp8_group_gemm.py @@ -4,12 +4,12 @@ import torch import triton import triton.language as tl -from deep_gemm import calc_diff +from deep_gemm.testing import calc_diff from deep_gemm.utils.layout import get_mn_major_tma_aligned_tensor -# Import shared functionality from the regular GEMM benchmark from sglang.benchmark.bench_utils import run_bench -from sglang.benchmark.kernels.deepseek.benchmark_deepgemm_fp8_gemm import ( +from sglang.benchmark.deepseek_utils import ( + get_weight_shapes, per_block_cast_to_fp8, per_token_cast_to_fp8, ) @@ -318,39 +318,6 @@ def calculate_diff(m: int, n: int, k: int, num_groups: int): ) -def get_weight_shapes(tp_size): - # cannot TP - total = [ - (512 + 64, 7168), - ((128 + 64) * 128, 7168), - (128 * (128 + 128), 512), - (7168, 16384), - (7168, 18432), - ] - # N can TP - n_tp = [ - (18432 * 2, 7168), - ((128 + 64) * 128, 7168), - (128 * (128 + 128), 512), - (24576, 1536), - (4096, 7168), - ] - # K can TP - k_tp = [(7168, 18432), (7168, 16384), (7168, 2048)] - - weight_shapes = [] - for t in total: - weight_shapes.append(t) - for n_t in n_tp: - new_t = (n_t[0] // tp_size, n_t[1]) - weight_shapes.append(new_t) - for k_t in k_tp: - new_t = (k_t[0], k_t[1] // tp_size) - weight_shapes.append(new_t) - - return weight_shapes - - def create_benchmark_configs(tp_size): configs = [] weight_shapes = get_weight_shapes(tp_size) @@ -435,7 +402,7 @@ def benchmark(m, n, k, num_groups, tp_size, provider): flops = 2 * m * n * k # multiply-adds tflops = flops / (ms * 1e-3) / 1e12 - print(f"Time: {ms*1000:.2f} ms, TFLOPS: {tflops:.2f}") + print(f"Time: {ms * 1000:.2f} ms, TFLOPS: {tflops:.2f}") return ms * 1000, max_ms * 1000, min_ms * 1000 # convert to ms return benchmark diff --git a/benchmark/kernels/deepseek/benchmark_q8kv8_kv_gather.py b/benchmark/kernels/deepseek/benchmark_q8kv8_kv_gather.py index 2f885b0ee6d8..12a73faf87dc 100755 --- a/benchmark/kernels/deepseek/benchmark_q8kv8_kv_gather.py +++ b/benchmark/kernels/deepseek/benchmark_q8kv8_kv_gather.py @@ -243,8 +243,7 @@ def main(): else: speedup = f"{legacy_us / us:.2f}x" print( - f"{s_q:>5} {kv_len:>7} {name:>10} {us:>9.1f} " - f"{tbps:>9.3f} {speedup:>8}" + f"{s_q:>5} {kv_len:>7} {name:>10} {us:>9.1f} {tbps:>9.3f} {speedup:>8}" ) print() diff --git a/benchmark/kernels/elementwise/benchmark_concat_mla.py b/benchmark/kernels/elementwise/benchmark_concat_mla.py index 7bc51d3da4be..8ae687aadc69 100644 --- a/benchmark/kernels/elementwise/benchmark_concat_mla.py +++ b/benchmark/kernels/elementwise/benchmark_concat_mla.py @@ -143,9 +143,7 @@ def execute_and_get_output(f, data): if not torch.all(output_ref == output_exp): abs_delta = torch.abs(output_ref - output_exp) raise AssertionError( - f"{output_ref=} {output_exp=} " - f"{abs_delta=} " - f"{torch.argwhere(abs_delta != 0.0)=} " + f"{output_ref=} {output_exp=} {abs_delta=} {torch.argwhere(abs_delta != 0.0)=} " ) diff --git a/benchmark/kernels/flashinfer_allreduce_fusion/benchmark_fused_collective.py b/benchmark/kernels/flashinfer_allreduce_fusion/benchmark_fused_collective.py index 2905a62a19b9..9af15ac87798 100644 --- a/benchmark/kernels/flashinfer_allreduce_fusion/benchmark_fused_collective.py +++ b/benchmark/kernels/flashinfer_allreduce_fusion/benchmark_fused_collective.py @@ -30,14 +30,16 @@ from sglang.kernels.ops.quantization.fp8_kernel import fp8_dtype as SGLANG_FP8_DTYPE from sglang.kernels.ops.quantization.fp8_kernel import static_quant_fp8 -from sglang.srt.distributed import get_tp_group, tensor_model_parallel_all_reduce +from sglang.srt.distributed import tensor_model_parallel_all_reduce from sglang.srt.distributed.parallel_state import ( cleanup_dist_env_and_memory, + get_tp_group, graph_capture, init_distributed_environment, initialize_model_parallel, ) from sglang.srt.layers.layernorm import RMSNorm # noqa +from sglang.test.test_utils import publish_build_topology try: from sgl_kernel import fused_add_rmsnorm as SGL_FUSED_ADD_RMS_NORM @@ -1178,7 +1180,8 @@ def main(): local_rank=rank, backend="nccl", ) - initialize_model_parallel(tensor_model_parallel_size=world_size) + publish_build_topology(world_rank=rank, tp_size=world_size) + initialize_model_parallel() # Validate world size (must be > 1 for collective operations) if world_size <= 1: diff --git a/benchmark/kernels/fused_moe_triton/benchmark_sglang_fused_moe_triton.py b/benchmark/kernels/fused_moe_triton/benchmark_sglang_fused_moe_triton.py index 4515ff53b34c..6934650adb92 100644 --- a/benchmark/kernels/fused_moe_triton/benchmark_sglang_fused_moe_triton.py +++ b/benchmark/kernels/fused_moe_triton/benchmark_sglang_fused_moe_triton.py @@ -26,6 +26,7 @@ select_experts, ) from sglang.srt.server_args import ServerArgs, set_global_server_args_for_scheduler +from sglang.test.test_utils import publish_build_topology def fused_moe_triton_api( @@ -227,10 +228,8 @@ def main(): backend="nccl" if torch.cuda.is_available() else "gloo", ) - initialize_model_parallel( - tensor_model_parallel_size=1, - expert_model_parallel_size=1, - ) + publish_build_topology() + initialize_model_parallel() model_config = get_model_config(args.model, args.tp_size, args.ep_size) benchmark.run( diff --git a/benchmark/kernels/fused_moe_triton/benchmark_vllm_vs_sglang_fused_moe_triton.py b/benchmark/kernels/fused_moe_triton/benchmark_vllm_vs_sglang_fused_moe_triton.py index fc100ce50804..1adecbeae6f6 100644 --- a/benchmark/kernels/fused_moe_triton/benchmark_vllm_vs_sglang_fused_moe_triton.py +++ b/benchmark/kernels/fused_moe_triton/benchmark_vllm_vs_sglang_fused_moe_triton.py @@ -15,6 +15,7 @@ from sglang.srt.layers.moe.moe_runner.triton_utils.fused_moe import ( fused_moe as fused_moe_sglang, ) +from sglang.test.test_utils import publish_build_topology from .common_utils import get_model_config @@ -243,10 +244,8 @@ def main(): backend="nccl" if torch.cuda.is_available() else "gloo", ) - initialize_model_parallel( - tensor_model_parallel_size=1, - pipeline_model_parallel_size=1, - ) + publish_build_topology() + initialize_model_parallel() shape_configs = get_model_config(args.model, args.tp_size, args.ep_size) benchmark.run( diff --git a/benchmark/kernels/fused_moe_triton/common_utils.py b/benchmark/kernels/fused_moe_triton/common_utils.py index 4c8b4625af22..354ec9a32b2f 100644 --- a/benchmark/kernels/fused_moe_triton/common_utils.py +++ b/benchmark/kernels/fused_moe_triton/common_utils.py @@ -84,19 +84,30 @@ def get_model_config( "Qwen3VLMoeForConditionalGeneration", "Qwen3_5MoeForCausalLM", "Qwen3_5MoeForConditionalGeneration", + "Qwen4ExpForConditionalGeneration", "InternS2PreviewForConditionalGeneration", "MellumForCausalLM", ]: E = config.num_experts // ep_size topk = config.num_experts_per_tok intermediate_size = config.moe_intermediate_size + elif architecture in [ + "Qwen4ExpForCausalLM", + "Qwen4ExpForConditionalGeneration", + ]: + E = config.num_experts // ep_size + topk = config.num_experts_per_tok + intermediate_size = config.moe_intermediate_size elif architecture in [ "DeepseekV2ForCausalLM", "DeepseekV3ForCausalLM", "DeepseekV32ForCausalLM", "DeepseekV4ForCausalLM", + "DeepseekOCRForCausalLM", "Glm4MoeForCausalLM", + "Glm4MoeLiteForCausalLM", "GlmMoeDsaForCausalLM", + "Glm5NextForConditionalGeneration", "KimiVLForConditionalGeneration", "MistralLarge3ForCausalLM", ]: @@ -108,6 +119,7 @@ def get_model_config( "DeepseekV3ForCausalLM", "DeepseekV32ForCausalLM", "Glm4MoeForCausalLM", + "Glm4MoeLiteForCausalLM", "GlmMoeDsaForCausalLM", "MistralLarge3ForCausalLM", ] @@ -196,6 +208,9 @@ def get_model_config( # text_config may not carry torch_dtype; fall back to bf16. torch_dtype = getattr(config, "torch_dtype", None) or torch.bfloat16 + num_layers = getattr(config, "num_hidden_layers", 0) + # Only the DeepSeek family replaces leading MoE layers with dense ones. + dense_layers = getattr(config, "first_k_dense_replace", 0) return { "num_experts": E, @@ -205,6 +220,8 @@ def get_model_config( "dtype": torch_dtype, "block_shape": block_shape, "architecture": architecture, + "num_layers": num_layers, + "dense_layers": dense_layers, } @@ -304,8 +321,6 @@ def get_config_filename( # NOTE(woosuk): The current naming convention uses w2.shape[2], which # is the intermediate size after silu_and_mul. N = shard_intermediate_size // 2 - if use_int4_w4a16: - N = N // 2 filename = get_config_file_name( num_experts, diff --git a/benchmark/kernels/fused_moe_triton/tuning_fused_moe_triton.py b/benchmark/kernels/fused_moe_triton/tuning_fused_moe_triton.py index 8cfc19dbdebc..e1cbe9f6a7fd 100644 --- a/benchmark/kernels/fused_moe_triton/tuning_fused_moe_triton.py +++ b/benchmark/kernels/fused_moe_triton/tuning_fused_moe_triton.py @@ -281,8 +281,6 @@ def benchmark( block_n = block_shape[0] if block_shape else 0 block_k = block_shape[1] if block_shape else 0 N = shard_intermediate_size // 2 - if use_int4_w4a16: - N = N // 2 op_config = get_moe_configs( num_experts, N, diff --git a/benchmark/kernels/fused_moe_triton/tuning_fused_moe_triton_sep.py b/benchmark/kernels/fused_moe_triton/tuning_fused_moe_triton_sep.py index a74972c33b07..3e2f717d804f 100644 --- a/benchmark/kernels/fused_moe_triton/tuning_fused_moe_triton_sep.py +++ b/benchmark/kernels/fused_moe_triton/tuning_fused_moe_triton_sep.py @@ -24,6 +24,7 @@ ) from ray.experimental.tqdm_ray import tqdm +from sglang.kernels.ops.moe.fused_moe_triton_kernels import clear_b_tma_desc_cache from sglang.srt.layers.moe.moe_runner import MoeRunnerConfig from sglang.srt.layers.moe.moe_runner.triton_utils.fused_moe import ( get_config_dtype_str, @@ -34,13 +35,18 @@ get_config_file_name, ) from sglang.srt.layers.moe.topk import TopKConfig, select_experts -from sglang.srt.server_args import ( - ServerArgs, - set_global_server_args_for_scheduler, +from sglang.srt.runtime_context import get_model, get_parallel +from sglang.srt.server_args import ServerArgs, set_global_server_args_for_scheduler +from sglang.srt.utils import ( + get_device, + get_device_module, + is_hip, + is_xpu, ) -from sglang.srt.utils import is_hip _is_hip = is_hip() +_is_xpu = is_xpu() +device_module = get_device_module() @dataclasses.dataclass @@ -72,11 +78,12 @@ def cuda_graph_wrapper(self): expert_ids=moe_input.expert_ids, num_tokens_post_padded=moe_input.num_tokens_post_padded, ) - torch.cuda.synchronize() + device_module.synchronize() - # Capture 10 invocations with CUDA graph - graph = torch.cuda.CUDAGraph() - with torch.cuda.graph(graph): + # Capture inner_iter invocations into one replayable graph. + graph_cls = torch.xpu.XPUGraph if _is_xpu else torch.cuda.CUDAGraph + graph = graph_cls() + with device_module.graph(graph): for k in range(self.inner_iter): moe_input = self.moe_inputs[k] self.func( @@ -86,19 +93,19 @@ def cuda_graph_wrapper(self): expert_ids=moe_input.expert_ids, num_tokens_post_padded=moe_input.num_tokens_post_padded, ) - torch.cuda.synchronize() + device_module.synchronize() # Warmup for _ in range(5): graph.replay() - torch.cuda.synchronize() + device_module.synchronize() return graph def forward_cost(self, try_cnt=2): time_cost = float("inf") for _ in range(try_cnt): - start_event = torch.cuda.Event(enable_timing=True) - end_event = torch.cuda.Event(enable_timing=True) + start_event = device_module.Event(enable_timing=True) + end_event = device_module.Event(enable_timing=True) start_event.record() if self.use_cuda_graph: self.graph.replay() @@ -113,14 +120,19 @@ def forward_cost(self, try_cnt=2): num_tokens_post_padded=moe_input.num_tokens_post_padded, ) end_event.record() - torch.cuda.synchronize() + device_module.synchronize() time_cost = min(time_cost, start_event.elapsed_time(end_event)) return time_cost def load_topk_ids(topk_ids_dir, i: int): - num_layers = 61 - dense_layers = 3 + model_config = get_model_config( + get_model().model_path, + tp_size=get_parallel().tp_size, + ep_size=get_parallel().ep_size, + ) + num_layers = model_config["num_layers"] + dense_layers = model_config["dense_layers"] moe_layers = num_layers - dense_layers return torch.load( f"{topk_ids_dir}/topk_ids_layer{i % moe_layers + dense_layers}_idx{i // moe_layers}.pt" @@ -150,6 +162,8 @@ def benchmark_config( ncu_enable = os.getenv("NCU_ENABLE", "0") == "1" if ncu_enable: num_iters = 1 + # Weights here are per-call, so a cached descriptor only pins a dead w1/w2. + clear_b_tma_desc_cache() init_dtype = torch.float16 if use_fp8_w8a8 else dtype hidden_states = torch.randn(num_tokens, hidden_size, dtype=dtype) if use_int8_w8a16 or use_int8_w8a8: @@ -470,7 +484,7 @@ def prepare(i: int, inner_iter): # update inputs according to topk_ids if build_down: ts1.append(kernel1.forward_cost()) # down no-tma ts_tma1.append(kernel1_tma.forward_cost()) # down tma - torch.cuda.synchronize() + device_module.synchronize() avg = sum(ts0) / (num_iters) * 1000 if ts0 else float("inf") avg1 = sum(ts1) / (num_iters) * 1000 if ts1 else float("inf") @@ -535,14 +549,13 @@ def config_dict(self, block_m): class BenchmarkWorker: - def __init__(self, seed: int, server_args: ServerArgs) -> None: - torch.set_default_device("cuda") - torch.cuda.manual_seed_all(0) + torch.set_default_device(get_device()) + device_module.manual_seed_all(0) self.seed = seed # Get the device ID to allocate tensors and kernels # on the respective GPU. - self.device_id = 0 # int(ray.get_gpu_ids()[0]) + self.device_id = 0 if not ray.is_initialized() else int(ray.get_gpu_ids()[0]) set_global_server_args_for_scheduler(server_args) def benchmark( @@ -563,9 +576,13 @@ def benchmark( ep_size: int = 1, enable_up_tma: bool = False, ) -> Tuple[Dict[str, int], float]: - torch.cuda.manual_seed_all(0) + device_module.manual_seed_all(0) topk_ids_list = [load_topk_ids(topk_ids_dir, i) for i in range(100)] - with torch.cuda.device(self.device_id) if is_hip() else nullcontext(): + with ( + device_module.device(self.device_id) + if _is_xpu or _is_hip + else nullcontext() + ): if enable_up_tma: # Two-step: first measure down to determine c_sorted, # then measure up with the correct c_sorted. @@ -655,7 +672,11 @@ def tune( trace0 = BestConfigTrace("kernel0", down_moe=False, enable_up_tma=False) trace1 = BestConfigTrace("kernel1", down_moe=True) - with torch.cuda.device(self.device_id) if is_hip() else nullcontext(): + with ( + device_module.device(self.device_id) + if _is_xpu or _is_hip + else nullcontext() + ): for config in tqdm(search_space): try: kt0_no_tma, kt0_tma, kt1_no_tma, kt1_tma = benchmark_config( @@ -694,7 +715,11 @@ def tune( trace1 = BestConfigTrace("kernel1", down_moe=True) # === Round 1: Down-only === - with torch.cuda.device(self.device_id) if is_hip() else nullcontext(): + with ( + device_module.device(self.device_id) + if _is_xpu or _is_hip + else nullcontext() + ): for config in tqdm(search_space, desc="Round 1 (down)"): try: _, _, kt1_no_tma, kt1_tma = benchmark_config( @@ -729,12 +754,15 @@ def tune( down_use_tma_map[block_m] = time_cost_all[2] > time_cost_all[3] print( - f"Round 1 done. Down TMA decisions per BLOCK_SIZE_M: " - f"{down_use_tma_map}" + f"Round 1 done. Down TMA decisions per BLOCK_SIZE_M: {down_use_tma_map}" ) # === Round 2: Up with c_sorted from round 1 === - with torch.cuda.device(self.device_id) if is_hip() else nullcontext(): + with ( + device_module.device(self.device_id) + if _is_xpu or _is_hip + else nullcontext() + ): for config in tqdm(search_space, desc="Round 2 (up)"): try: kt0_no_tma, kt0_tma, _, _ = benchmark_config( @@ -806,8 +834,12 @@ def cmp_configs( print(f"config {i}: {file}") topk_ids_list = [load_topk_ids(topk_ids_dir, i) for i in range(100)] - torch.cuda.manual_seed_all(0) - with torch.cuda.device(self.device_id) if is_hip() else nullcontext(): + device_module.manual_seed_all(0) + with ( + device_module.device(self.device_id) + if _is_xpu or _is_hip + else nullcontext() + ): for bs in num_tokens: kernel_times = [] cfgs = [] diff --git a/benchmark/kernels/lora_csgmv/tune_lora_csgmv.py b/benchmark/kernels/lora_csgmv/tune_lora_csgmv.py index aa684dfa84ea..84a2d8f5a549 100755 --- a/benchmark/kernels/lora_csgmv/tune_lora_csgmv.py +++ b/benchmark/kernels/lora_csgmv/tune_lora_csgmv.py @@ -470,9 +470,9 @@ def _tune_shrink( device: torch.device, ) -> tuple: """Tune shrink kernel for one layer type. Returns (best_configs, results).""" - print(f"\n{'='*80}") + print(f"\n{'=' * 80}") print(f"Tuning SHRINK — {label} (K={K}, N={N}, slices={num_slices})") - print(f"{'='*80}") + print(f"{'=' * 80}") search = get_shrink_search_space() print(f"Search space: {len(search)} configs") @@ -508,7 +508,7 @@ def _tune_shrink( best_config = config if (i + 1) % 20 == 0: print( - f" chunk={chunk_size}: {i+1}/{len(search)} tested, best={best_time:.3f}ms" + f" chunk={chunk_size}: {i + 1}/{len(search)} tested, best={best_time:.3f}ms" ) best_configs[chunk_size] = sort_config(best_config) @@ -533,9 +533,9 @@ def _tune_expand( device: torch.device, ) -> tuple: """Tune expand kernel for one layer type. Returns (best_configs, results).""" - print(f"\n{'='*80}") + print(f"\n{'=' * 80}") print(f"Tuning EXPAND — {label} (output_dim={output_dim}, slices={num_slices})") - print(f"{'='*80}") + print(f"{'=' * 80}") search = get_expand_search_space() print(f"Search space: {len(search)} configs") @@ -584,7 +584,7 @@ def _tune_expand( best_config = config if (i + 1) % 50 == 0: print( - f" chunk={chunk_size}: {i+1}/{len(search)} tested, best={best_time:.3f}ms" + f" chunk={chunk_size}: {i + 1}/{len(search)} tested, best={best_time:.3f}ms" ) best_configs[chunk_size] = sort_config(best_config) @@ -673,9 +673,9 @@ def main(args: argparse.Namespace): ) # --- Summary --- - print(f"\n{'='*80}") + print(f"\n{'=' * 80}") print(f"SUMMARY") - print(f"{'='*80}") + print(f"{'=' * 80}") print( f"\n{'layer':<10} {'kernel':<8} {'K/dim':>6} {'chunk':>6}" f" {'baseline':>10} {'tuned':>10} {'speedup':>8} config" diff --git a/benchmark/kernels/quantization/tuning_block_wise_kernel.py b/benchmark/kernels/quantization/tuning_block_wise_kernel.py index 1c232fa3894a..f2c4cca0ab5a 100644 --- a/benchmark/kernels/quantization/tuning_block_wise_kernel.py +++ b/benchmark/kernels/quantization/tuning_block_wise_kernel.py @@ -27,6 +27,7 @@ mp.set_start_method("spawn", force=True) +from sglang.benchmark.deepseek_utils import get_weight_shapes from sglang.kernels.ops.quantization.fp8_kernel import ( _w8a8_block_fp8_matmul, _w8a8_block_fp8_matmul_unrolledx4, @@ -192,39 +193,6 @@ def get_configs_compute_bound(): return configs -def get_weight_shapes(tp_size): - # NOTE(HandH1998): The weight shapes only works for DeepSeek-V3. Modify them, if you tune for another different model. - # cannot TP - total = [ - (512 + 64, 7168), - ((128 + 64) * 128, 7168), - (128 * (128 + 128), 512), - (7168, 16384), - (7168, 18432), - ] - # N can TP - n_tp = [ - (18432 * 2, 7168), - ((128 + 64) * 128, 7168), - (128 * (128 + 128), 512), - (24576, 1536), - (4096, 7168), - ] - # K can TP - k_tp = [(7168, 18432), (7168, 16384), (7168, 2048)] - - weight_shapes = [] - for t in total: - weight_shapes.append(t) - for n_t in n_tp: - new_t = (n_t[0] // tp_size, n_t[1]) - weight_shapes.append(new_t) - for k_t in k_tp: - new_t = (k_t[0], k_t[1] // tp_size) - weight_shapes.append(new_t) - return weight_shapes - - def benchmark_config( A, B, As, Bs, block_size, config, out_dtype=torch.float16, num_iters=10 ): diff --git a/benchmark/lean_kernel_sweep.py b/benchmark/lean_kernel_sweep.py index f6a6679ac321..6e5ce6a8fc09 100755 --- a/benchmark/lean_kernel_sweep.py +++ b/benchmark/lean_kernel_sweep.py @@ -137,19 +137,21 @@ def main(): # b32 x 128K on the 8-KV-head config exceeds the microbench's single # contiguous KV tensor (faults the GPU); real serving uses a paged pool. if H_KV == 8 and B == 32 and S == 131072: - print(f"{B:>5} {S//1024:>5}K {'skipped (contiguous-KV limit)':>30}") + print( + f"{B:>5} {S // 1024:>5}K {'skipped (contiguous-KV limit)':>30}" + ) rows.append(f"{name},{H_Q},{H_KV},{B},{S},,,,skip,") continue try: std, lean, cos, gate = run(H_Q, H_KV, B, S) except torch.cuda.OutOfMemoryError: torch.cuda.empty_cache() - print(f"{B:>5} {S//1024:>5}K {'OOM':>9}") + print(f"{B:>5} {S // 1024:>5}K {'OOM':>9}") rows.append(f"{name},{H_Q},{H_KV},{B},{S},,,,OOM,") continue sp = std / lean print( - f"{B:>5} {S//1024:>5}K {std:>9.3f} {lean:>9.3f} {sp:>7.2f}x {('ON' if gate else 'OFF'):>5} {cos:>7.4f}" + f"{B:>5} {S // 1024:>5}K {std:>9.3f} {lean:>9.3f} {sp:>7.2f}x {('ON' if gate else 'OFF'):>5} {cos:>7.4f}" ) rows.append( f"{name},{H_Q},{H_KV},{B},{S},{std:.4f},{lean:.4f},{sp:.4f},{cos:.4f},{int(gate)}" diff --git a/benchmark/mmlu/bench_sglang.py b/benchmark/mmlu/bench_sglang.py index 9a2006e3d2c1..b86aec46c5c2 100644 --- a/benchmark/mmlu/bench_sglang.py +++ b/benchmark/mmlu/bench_sglang.py @@ -163,7 +163,7 @@ def few_shot_mmlu(s, examples, question): pt = 0 for subject, num_qs in zip(subjects[: args.nsub], num_questions): print( - f"subject: {subject}, #q:{num_qs}, acc: {np.mean(cors[pt: pt + num_qs]):.3f}" + f"subject: {subject}, #q:{num_qs}, acc: {np.mean(cors[pt : pt + num_qs]):.3f}" ) pt += num_qs assert pt == len(cors) diff --git a/benchmark/mmmu/data_utils.py b/benchmark/mmmu/data_utils.py index 8c36768d0a4b..aca2029692df 100644 --- a/benchmark/mmmu/data_utils.py +++ b/benchmark/mmmu/data_utils.py @@ -75,22 +75,6 @@ } -def get_multi_choice_info(options): - """ - Given the list of options for multiple choice question - Return the index2ans and all_choices - """ - - start_chr = "A" - all_choices = [] - index2ans = {} - for i, option in enumerate(options): - index2ans[chr(ord(start_chr) + i)] = option - all_choices.append(chr(ord(start_chr) + i)) - - return index2ans, all_choices - - def load_yaml(file_path): with open(file_path, "r") as stream: try: @@ -142,33 +126,6 @@ def save_json(filename, ds): json.dump(ds, f, indent=4) -def save_jsonl(filename, data): - """ - Save a dictionary of data to a JSON Lines file with the filename as key and caption as value. - - Args: - filename (str): The path to the file where the data should be saved. - data (dict): The dictionary containing the data to save where key is the image path and value is the caption. - """ - with open(filename, "w", encoding="utf-8") as f: - for img_path, caption in data.items(): - # Extract the base filename without the extension - base_filename = os.path.basename(img_path) - # Create a JSON object with the filename as the key and caption as the value - json_record = json.dumps({base_filename: caption}, ensure_ascii=False) - # Write the JSON object to the file, one per line - f.write(json_record + "\n") - - -def save_args(args, path_dir): - argsDict = args.__dict__ - with open(path_dir + "setting.txt", "w") as f: - f.writelines("------------------ start ------------------" + "\n") - for eachArg, value in argsDict.items(): - f.writelines(eachArg + " : " + str(value) + "\n") - f.writelines("------------------- end -------------------") - - # DATA PROCESSING def construct_prompt(sample, config): question = sample["question"] diff --git a/benchmark/ocr/bench_sglang.py b/benchmark/ocr/bench_sglang.py index 651f61417011..f210c6232cea 100644 --- a/benchmark/ocr/bench_sglang.py +++ b/benchmark/ocr/bench_sglang.py @@ -502,7 +502,7 @@ async def process_sample( } ) print( - f"[INPUT ] [{i+1}] type={ttype!r:12s} expected: {expected[:120]}", + f"[INPUT ] [{i + 1}] type={ttype!r:12s} expected: {expected[:120]}", flush=True, ) # Print OCR output (truncate long outputs) diff --git a/benchmark/ocr/generate_report.py b/benchmark/ocr/generate_report.py index 770c827eeca9..95bba28d6ca2 100644 --- a/benchmark/ocr/generate_report.py +++ b/benchmark/ocr/generate_report.py @@ -159,12 +159,12 @@ def _render_sample(sample: dict, failures_only: bool) -> str: if failures_only and passed == total and not error: return "" - pct = f"{100*passed//total}%" if total else "—" + pct = f"{100 * passed // total}%" if total else "—" header_cls = "fail" if (error or passed < total) else "pass" parts = [f'
'] parts.append( - f'
' + f"
" f'' f"📄 {html.escape(pdf)}  ·  page {page}" f"" @@ -218,7 +218,7 @@ def _render_sample(sample: dict, failures_only: bool) -> str: f'
{_latex_to_display(latex)}
' ) elif ttype in ("present", "absent", "text_presence", "text_absence"): - parts.append(f'
{html.escape(ti.get("text", ""))}
') + parts.append(f"
{html.escape(ti.get('text', ''))}
") elif ttype in ("order", "natural_reading_order"): before = ti.get("before", "") after = ti.get("after", "") @@ -236,7 +236,7 @@ def _render_sample(sample: dict, failures_only: bool) -> str: parts.append(f'
{m}
') if len(matches) > 6: parts.append( - f'

… and {len(matches)-6} more

' + f'

… and {len(matches) - 6} more

' ) else: parts.append( diff --git a/benchmark/prefill_only/bench_setwise_score.py b/benchmark/prefill_only/bench_setwise_score.py new file mode 100644 index 000000000000..735c5b436511 --- /dev/null +++ b/benchmark/prefill_only/bench_setwise_score.py @@ -0,0 +1,271 @@ +"""Engine benchmark for setwise (multi-position) scoring. + +Benchmarks `Engine.async_score(..., score_extraction_token_id=...)` — the setwise +path that pools the classification head at each candidate's anchor token and +returns an ``[N, num_labels]`` matrix in a single forward pass. + +Unlike the general ``bench_score.py`` (label-token / MIS scoring), this is a +focused, self-contained harness: it sweeps the **set size** (number of +candidates packed into one item) and reports latency percentiles and throughput +(requests/s and candidates/s) so you can see how a single-forward setwise pass +scales with N. + +It talks to the Engine directly (no HTTP) and requires the setwise deployment +constraints (``--disable-radix-cache``, ``--chunked-prefill-size -1``), which it +sets automatically. + +Usage: + # Default sweep over set sizes on the target model + python bench_setwise_score.py --model-path /path/to/seqcls_model + + # Custom sweep + concurrency + per-candidate length + python bench_setwise_score.py --model-path /path/to/model \ + --set-sizes 8 16 32 64 --concurrency 16 --num-requests 100 \ + --item-tokens 180 --query-tokens 64 + + # Include pooled hidden states in the response (per-anchor pre-head vectors) + python bench_setwise_score.py --model-path /path/to/model --return-pooled-hidden-states +""" + +import argparse +import asyncio +import csv +import random +import statistics +import time +from typing import List, Optional + +from transformers import AutoTokenizer + + +def parse_args(): + parser = argparse.ArgumentParser( + description="Benchmark setwise scoring via Engine.async_score", + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + parser.add_argument( + "--model-path", + type=str, + required=True, + help="Path to a SequenceClassification model (e.g. Qwen3ForSequenceClassification).", + ) + parser.add_argument( + "--score-extraction-token", + type=str, + default="<|object_ref_start|>", + help="Anchor token pooled at each candidate (default: <|object_ref_start|>).", + ) + parser.add_argument( + "--set-sizes", + type=int, + nargs="+", + default=[8, 16, 32, 64], + help="Candidates per set (each is one anchor in one item). Swept in order.", + ) + parser.add_argument( + "--item-tokens", + type=int, + default=180, + help="Content tokens per candidate before its anchor (default: 180).", + ) + parser.add_argument( + "--query-tokens", + type=int, + default=0, + help="Shared query-prefix tokens (default: 0).", + ) + parser.add_argument( + "--num-requests", + type=int, + default=50, + help="Timed requests per set size (default: 50).", + ) + parser.add_argument( + "--concurrency", + type=int, + default=8, + help="Max in-flight requests (default: 8).", + ) + parser.add_argument( + "--warmup", + type=int, + default=3, + help="Warmup requests per set size, not timed (default: 3).", + ) + parser.add_argument("--apply-softmax", action="store_true") + parser.add_argument("--return-pooled-hidden-states", action="store_true") + + # Engine configuration + parser.add_argument("--dtype", type=str, default="float16") + parser.add_argument("--mem-fraction-static", type=float, default=0.5) + parser.add_argument("--max-prefill-tokens", type=int, default=60000) + parser.add_argument("--attention-backend", type=str, default="flashinfer") + parser.add_argument("--seed", type=int, default=0) + parser.add_argument( + "--output", "-o", type=str, default=None, help="Optional CSV output path." + ) + return parser.parse_args() + + +def _random_content(num_tokens: int, anchor_id: int, lo: int = 10, hi: int = 30000): + """Random content token ids that never collide with the anchor id.""" + ids = [random.randint(lo, hi) for _ in range(num_tokens)] + return [t + 1 if t == anchor_id else t for t in ids] + + +def _build_setwise_request(args, anchor_id: int, n_candidates: int): + """One setwise request: shared query prefix + ONE item with N anchors.""" + query_ids = ( + _random_content(args.query_tokens, anchor_id) if args.query_tokens else [] + ) + item_ids: List[int] = [] + for _ in range(n_candidates): + item_ids.extend(_random_content(args.item_tokens, anchor_id)) + item_ids.append(anchor_id) + return query_ids, [item_ids] + + +def _percentile(sorted_vals: List[float], pct: float) -> float: + if not sorted_vals: + return 0.0 + k = max(0, min(len(sorted_vals) - 1, round((pct / 100.0) * (len(sorted_vals) - 1)))) + return sorted_vals[k] + + +async def _run_set_size(engine, args, anchor_id: int, n: int) -> Optional[dict]: + # Pre-build requests so generation cost is outside the timed section. + requests = [ + _build_setwise_request(args, anchor_id, n) + for _ in range(args.num_requests + args.warmup) + ] + prompt_tokens = len(requests[0][0]) + len(requests[0][1][0]) + + async def _one(query_ids, items): + t0 = time.perf_counter() + res = await engine.async_score( + query=query_ids, + items=items, + score_extraction_token_id=anchor_id, + apply_softmax=args.apply_softmax, + return_pooled_hidden_states=args.return_pooled_hidden_states, + ) + # Setwise scores are nested (one [Ni x num_labels] matrix per item) and + # this harness sends exactly one item, so the per-candidate row count is + # len(res.scores[0]), not len(res.scores) (which is always 1 = num items). + return time.perf_counter() - t0, len(res.scores[0]) if res.scores else 0 + + # Warmup (also validates the row count == N). + for q, it in requests[: args.warmup]: + _, rows = await _one(q, it) + if rows != n: + print(f"[warn] set_size={n}: expected {n} score rows, got {rows}") + + sem = asyncio.Semaphore(args.concurrency) + + async def _guarded(q, it): + async with sem: + return await _one(q, it) + + timed = requests[args.warmup :] + wall_start = time.perf_counter() + results = await asyncio.gather(*(_guarded(q, it) for q, it in timed)) + wall = time.perf_counter() - wall_start + + latencies_ms = sorted(1000.0 * lat for lat, _ in results) + n_req = len(results) + return { + "set_size": n, + "prompt_tokens": prompt_tokens, + "requests": n_req, + "concurrency": args.concurrency, + "avg_ms": statistics.mean(latencies_ms), + "p50_ms": _percentile(latencies_ms, 50), + "p95_ms": _percentile(latencies_ms, 95), + "p99_ms": _percentile(latencies_ms, 99), + "req_per_s": n_req / wall if wall > 0 else 0.0, + "cand_per_s": (n_req * n) / wall if wall > 0 else 0.0, + } + + +def _print_table(rows: List[dict]): + headers = [ + "set_size", + "prompt_tokens", + "requests", + "avg_ms", + "p50_ms", + "p95_ms", + "p99_ms", + "req_per_s", + "cand_per_s", + ] + print("\n" + " | ".join(f"{h:>13}" for h in headers)) + print("-" * (16 * len(headers))) + for r in rows: + cells = [] + for h in headers: + v = r[h] + cells.append(f"{v:>13.2f}" if isinstance(v, float) else f"{v:>13}") + print(" | ".join(cells)) + + +async def main(): + args = parse_args() + random.seed(args.seed) + + tokenizer = AutoTokenizer.from_pretrained(args.model_path, trust_remote_code=True) + anchor_id = tokenizer.convert_tokens_to_ids(args.score_extraction_token) + unk_id = tokenizer.unk_token_id + if anchor_id is None or (unk_id is not None and anchor_id == unk_id): + raise ValueError( + f"score_extraction_token {args.score_extraction_token!r} did not resolve " + f"to a dedicated token id." + ) + + from sglang.srt.entrypoints.engine import Engine + + # Setwise deployment constraints: positions are full-prompt coordinates, so + # radix reuse / chunked prefill must be off (the engine also validates this). + engine = Engine( + model_path=args.model_path, + dtype=args.dtype, + mem_fraction_static=args.mem_fraction_static, + max_prefill_tokens=args.max_prefill_tokens, + attention_backend=args.attention_backend, + disable_radix_cache=True, + chunked_prefill_size=-1, + ) + + print( + f"Setwise scoring benchmark | model={args.model_path}\n" + f"anchor={args.score_extraction_token!r} (id={anchor_id}) | " + f"item_tokens={args.item_tokens} query_tokens={args.query_tokens} | " + f"concurrency={args.concurrency} num_requests={args.num_requests}" + ) + + rows = [] + try: + for n in args.set_sizes: + row = await _run_set_size(engine, args, anchor_id, n) + if row is not None: + rows.append(row) + print( + f"set_size={n:>4}: avg={row['avg_ms']:.1f}ms " + f"p95={row['p95_ms']:.1f}ms " + f"{row['req_per_s']:.1f} req/s {row['cand_per_s']:.0f} cand/s" + ) + finally: + engine.shutdown() + + _print_table(rows) + + if args.output and rows: + with open(args.output, "w", newline="") as f: + writer = csv.DictWriter(f, fieldnames=list(rows[0].keys())) + writer.writeheader() + writer.writerows(rows) + print(f"\nWrote {len(rows)} rows to {args.output}") + + +if __name__ == "__main__": + asyncio.run(main()) diff --git a/benchmark/prefill_only/util.py b/benchmark/prefill_only/util.py index 2451239d61ab..ad21dc1b34ec 100644 --- a/benchmark/prefill_only/util.py +++ b/benchmark/prefill_only/util.py @@ -383,14 +383,14 @@ async def send_warmup_requests( http_url, data=request_json, headers=headers ) as resp: if resp.status == 200: - print(f"Warmup request {i+1}/{num_warmup} completed successfully") + print(f"Warmup request {i + 1}/{num_warmup} completed successfully") else: print( - f"Warmup request {i+1}/{num_warmup} failed with status {resp.status}" + f"Warmup request {i + 1}/{num_warmup} failed with status {resp.status}" ) except Exception as e: - print(f"Warmup request {i+1}/{num_warmup} failed with error: {e}") + print(f"Warmup request {i + 1}/{num_warmup} failed with error: {e}") print("HTTP warmup requests completed") @@ -745,7 +745,6 @@ async def run_generic_benchmark( async with aiohttp.ClientSession( timeout=aiohttp.ClientTimeout(total=300) ) as session: - # Send START_PROFILE if profiling is enabled if config.profile: await send_profile_request("START_PROFILE", http_url, session=session) diff --git a/benchmark/scheduler/bench_token_storage.py b/benchmark/scheduler/bench_token_storage.py index ea8ef7418c4c..79ee6aa6d118 100644 --- a/benchmark/scheduler/bench_token_storage.py +++ b/benchmark/scheduler/bench_token_storage.py @@ -254,7 +254,7 @@ def print_breakdown(title: str, results: dict[str, dict[str, float]]) -> None: def microbench_torch_tensor_paths( - sizes: tuple[int, ...] = (1_000, 10_000, 100_000) + sizes: tuple[int, ...] = (1_000, 10_000, 100_000), ) -> None: """Compare three CPU-buffer -> pinned cuda tensor paths. @@ -293,9 +293,11 @@ def t(fn, iterations: int) -> float: ), ( "(C) from_numpy(frombuf(array('q'))).pin() -> cuda", - lambda x: torch.from_numpy(np.frombuffer(x, dtype=np.int64)) - .pin_memory() - .to("cuda", non_blocking=True), + lambda x: ( + torch.from_numpy(np.frombuffer(x, dtype=np.int64)) + .pin_memory() + .to("cuda", non_blocking=True) + ), ), ]: cells = [] diff --git a/benchmark/simulator/bench_runner.py b/benchmark/simulator/bench_runner.py new file mode 100644 index 000000000000..6ae952969727 --- /dev/null +++ b/benchmark/simulator/bench_runner.py @@ -0,0 +1,168 @@ +"""In-process benchmark runner for SGLang Simulator.""" + +import asyncio +import atexit +import json +import os +from typing import Iterator + +import numpy as np +from msgspec.structs import asdict +from sglang_simulator.compat import apply_simulator_server_args +from sglang_simulator.dataset import BaseDataset, GenericRequest +from sglang_simulator.simulation.benchmark import BaseBenchmarkRunner, BenchmarkConfig +from sglang_simulator.utils.logger import get_logger + +SGLANG_SIMULATOR_OUTPUT_DIR = os.getenv( + "SGLANG_SIMULATOR_OUTPUT_DIR", "/tmp/sglang_simulator/output" +) +SIMULATION_METRICS_PATH = f"{SGLANG_SIMULATOR_OUTPUT_DIR}/metrics.json" +os.environ["SGLANG_SIMULATOR_OUTPUT_DIR"] = SGLANG_SIMULATOR_OUTPUT_DIR + +if os.getenv("SGLANG_SIMULATOR_OUTPUT_MODE") is None: + os.environ["SGLANG_SIMULATOR_OUTPUT_MODE"] = "OFFLINE" + +# Import the simulator engine only after configuring its worker environment. +from sglang_simulator.simulation.sglang.engine import ( # noqa: E402 + SGLangSimulationEngine, +) + +# SGLang must be imported after the simulator hooks are installed by engine.py. +from sglang.srt.server_args import ServerArgs # noqa: E402 + +logger = get_logger("sglang_simulator") + + +class SGLangBenchmarkRunner(BaseBenchmarkRunner): + """Run a simulator workload directly through SGLang's in-process Engine.""" + + def __init__(self, server_args: ServerArgs): + # Disable features that are unnecessary for simulation. + server_args_kwargs = asdict(server_args) + apply_simulator_server_args(server_args_kwargs) + self.engine = SGLangSimulationEngine(**server_args_kwargs) + self.server_args = self.engine.server_args + self._shutdown = False + + def flush_cache(self): + self.engine.flush_cache() + + def clear_hicache_storage(self): + self.engine.loop.run_until_complete( + self.engine.tokenizer_manager.clear_hicache_storage() + ) + + def get_request( + self, + dataset: BaseDataset, + ignore_timestamp: bool = False, + request_rate: float = float("inf"), + ) -> Iterator[tuple[GenericRequest, dict]]: + yield_delay = 0 + for req in dataset: + if ignore_timestamp: + created_time = yield_delay + yield_delay += np.random.exponential(1.0 / request_rate) + else: + created_time = req.custom_params.get("created_time", 0) + + simulation_params = { + "total_request": len(dataset), # Include the warmup requests. + "created_time": created_time, + } + + yield (req, simulation_params) + + async def async_benchmark( + self, + benchmark_config: BenchmarkConfig, + dataset: BaseDataset, + ): + await self.engine.tokenizer_manager.start_profile() + + if os.path.exists(SIMULATION_METRICS_PATH): + with open(SIMULATION_METRICS_PATH, "w") as metrics_file: + # Clear data from a previous benchmark in the same process. + pass + + tasks = [] + logger.info(f"Created {len(dataset)} request tasks.") + for req, simulation_params in self.get_request( + dataset, + ignore_timestamp=benchmark_config.ignore_request_timestamp, + request_rate=benchmark_config.request_rate, + ): + task = asyncio.create_task( + self.engine.async_generate( + prompt=req.prompt, + input_ids=req.token_ids, + sampling_params={ + "ignore_eos": True, + "max_new_tokens": req.output_length, + "custom_params": { + # Transfer simulation arguments through sampling params. + "simulation": simulation_params + }, + }, + ) + ) + tasks.append(task) + + _ = await asyncio.gather(*tasks) + + # Trigger the simulator's profile handler to flush final metrics. + await self.engine.tokenizer_manager.start_profile() + + if os.path.exists(SIMULATION_METRICS_PATH): + with open(SIMULATION_METRICS_PATH) as metrics_file: + metrics = json.load(metrics_file) + else: + logger.error( + f"Failed to load metrics from serving backend. The metrics file " + f"should be loaded from {SIMULATION_METRICS_PATH}." + ) + return None + + return metrics + + def benchmark(self, benchmark_config: BenchmarkConfig, dataset: BaseDataset): + return self.engine.loop.run_until_complete( + self.async_benchmark(benchmark_config, dataset) + ) + + def get_iteration_stats(self) -> list[dict]: + data = [] + file_path = f"{SGLANG_SIMULATOR_OUTPUT_DIR}/iteration.jsonl" + if os.path.exists(file_path): + with open(file_path) as stats_file: + line = stats_file.readline() + while line: + data.append(json.loads(line)) + line = stats_file.readline() + else: + logger.error(f"The iteration statistics data({file_path}) does not exist.") + return data + + def get_request_stats(self) -> list[dict]: + data = [] + file_path = f"{SGLANG_SIMULATOR_OUTPUT_DIR}/request.jsonl" + if os.path.exists(file_path): + with open(file_path) as stats_file: + line = stats_file.readline() + while line: + data.append(json.loads(line)) + line = stats_file.readline() + else: + logger.error(f"The request statistics data({file_path}) does not exist.") + return data + + def shutdown(self): + if self._shutdown: + return None + + logger.info("Attempting to shut down the SGLang backend engine.") + try: + return self.engine.shutdown() + finally: + self._shutdown = True + atexit.unregister(self.engine.shutdown) diff --git a/benchmark/simulator/bench_serving.py b/benchmark/simulator/bench_serving.py new file mode 100644 index 000000000000..cd5fd05257cd --- /dev/null +++ b/benchmark/simulator/bench_serving.py @@ -0,0 +1,280 @@ +"""SGLang serving benchmark adapter for simulator traffic. + +This script deliberately reuses SGLang's benchmark implementation and dataset +loaders. It only owns the simulator-specific parts of the protocol: + +* convert request-rate or trace timestamps into logical arrival timestamps; +* inject the internal ``sampling_params.custom_params.simulation`` metadata; +* avoid client-side pacing in OFFLINE mode; and +* display backend-produced simulator metrics when they are locally available. + +User datasets must not contain simulator metadata. +""" + +import argparse +import contextlib +import json +import os +import re +import sys +from dataclasses import fields +from pathlib import Path +from typing import AsyncGenerator, List, Optional + +import aiohttp +import numpy as np +from sglang_simulator.compat import validate_benchmark_runtime +from sglang_simulator.dataset.autobench import register_autobench_dataset + +register_autobench_dataset() + +from sglang.benchmark import serving +from sglang.benchmark.datasets.common import DatasetRow + +_ORIGINAL_AIOHTTP_REQUEST = None +_ORIGINAL_CALCULATE_METRICS = serving.calculate_metrics +_ORIGINAL_GET_REQUEST = serving.get_request +_ORIGINAL_RUN_BENCHMARK = serving.run_benchmark +_SIMULATOR_MODE = "offline" +_USE_TRACE_TIMESTAMPS = False + + +def _metrics_path() -> Path: + output_dir = Path( + os.getenv("SGLANG_SIMULATOR_OUTPUT_DIR", "/tmp/sglang_simulator/output") + ) + return output_dir / "metrics.json" + + +def _load_backend_metrics() -> Optional[dict]: + metrics_path = _metrics_path() + if not metrics_path.is_file(): + return None + return json.loads(metrics_path.read_text(encoding="utf-8")) + + +class _DurationReplacingStream: + """Keep SGLang's output format but print the simulated duration.""" + + def __init__(self, target): + self.target = target + + def write(self, text): + if "Benchmark duration (s):" in text: + metrics = _load_backend_metrics() + if metrics is not None and "duration" in metrics: + text = "{:<40} {:<10.2f}".format( + "Benchmark duration (s):", metrics["duration"] + ) + return self.target.write(text) + + def flush(self): + return self.target.flush() + + +def _set_simulation_metadata( + request: DatasetRow, *, created_time_ms: float, total_request: int +) -> None: + """Attach transient metadata without replacing dataset-specific parameters.""" + extra_request_body = dict(request.extra_request_body or {}) + extra_request_body["simulation"] = { + "created_time_ms": created_time_ms, + "total_request": total_request, + } + request.extra_request_body = extra_request_body + + +async def simulator_get_request( + input_requests: List[DatasetRow], + request_rate: float, + use_trace_timestamps: bool = False, + slowdown_factor: float = 1.0, +) -> AsyncGenerator[DatasetRow, None]: + """Generate simulator traffic while retaining official BLOCKING pacing.""" + # The benchmark may not forward --use-trace-timestamps to get_request(), + # so preserve the parsed value in this adapter. + use_trace_timestamps = use_trace_timestamps or _USE_TRACE_TIMESTAMPS + if _SIMULATOR_MODE == "blocking": + async for request in _ORIGINAL_GET_REQUEST( + input_requests, + request_rate, + use_trace_timestamps=use_trace_timestamps, + slowdown_factor=slowdown_factor, + ): + yield request + return + + total_request = len(input_requests) + if use_trace_timestamps: + if any(request.timestamp is None for request in input_requests): + raise ValueError( + "--use-trace-timestamps requires every request to have timestamp" + ) + input_requests.sort(key=lambda request: request.timestamp) + trace_start_time_ms = input_requests[0].timestamp if input_requests else 0.0 + for request in input_requests: + created_time_ms = ( + float(request.timestamp) - float(trace_start_time_ms) + ) * slowdown_factor + _set_simulation_metadata( + request, + created_time_ms=created_time_ms, + total_request=total_request, + ) + yield request + return + + created_time_ms = 0.0 + for request in input_requests: + _set_simulation_metadata( + request, + created_time_ms=created_time_ms, + total_request=total_request, + ) + yield request + if request_rate != float("inf"): + created_time_ms += np.random.exponential(1.0 / request_rate) * 1000.0 + + +def install_aiohttp_json_hijack( + *, hijack_url_regex: Optional[str] = r"/generate(?:\?.*)?$" +) -> None: + """Move transient metadata into the already-built sampling parameters.""" + global _ORIGINAL_AIOHTTP_REQUEST + if _ORIGINAL_AIOHTTP_REQUEST is not None: + return + + pattern = re.compile(hijack_url_regex) if hijack_url_regex else None + _ORIGINAL_AIOHTTP_REQUEST = aiohttp.ClientSession._request + + async def patched_request(self, method, url, **kwargs): + if pattern is None or pattern.search(str(url)): + payload = kwargs.get("json") + if isinstance(payload, dict) and "simulation" in payload: + simulation = payload.pop("simulation") + sampling_params = payload.setdefault("sampling_params", {}) + custom_params = sampling_params.setdefault("custom_params", {}) + custom_params["simulation"] = simulation + kwargs["json"] = payload + return await _ORIGINAL_AIOHTTP_REQUEST(self, method, url, **kwargs) + + aiohttp.ClientSession._request = patched_request + + +def simulator_calculate_metrics(*args, **kwargs): + """Use simulator metrics; mark unsupported client-only fields with -1.""" + client_metrics, output_lens = _ORIGINAL_CALCULATE_METRICS(*args, **kwargs) + backend_metrics = _load_backend_metrics() + if backend_metrics is None: + print( + f"Simulator metrics are not available at {_metrics_path()}; " + "showing client-side benchmark metrics." + ) + return client_metrics, output_lens + + metric_names = {field.name for field in fields(serving.BenchmarkMetrics)} + values = {name: backend_metrics.get(name, -1) for name in metric_names} + return serving.BenchmarkMetrics(**values), output_lens + + +def _replace_output_file_duration( + args: argparse.Namespace, simulated_duration: float +) -> None: + output_file = getattr(args, "output_file", None) + if not output_file: + return + path = Path(output_file) + if not path.is_file(): + return + lines = path.read_text(encoding="utf-8").splitlines() + if not lines: + return + last_result = json.loads(lines[-1]) + last_result["duration"] = simulated_duration + lines[-1] = json.dumps(last_result) + path.write_text("\n".join(lines) + "\n", encoding="utf-8") + + +def simulator_run_benchmark(args: argparse.Namespace): + global _USE_TRACE_TIMESTAMPS + if args.backend != "sglang": + raise ValueError( + "benchmark/simulator/bench_serving.py requires --backend sglang" + ) + if args.dataset_name == "mooncake": + raise ValueError( + "Mooncake's multi-round scheduler is not supported by the simulator " + "benchmark adapter" + ) + _USE_TRACE_TIMESTAMPS = getattr(args, "use_trace_timestamps", False) + args.profile = True + with contextlib.redirect_stdout(_DurationReplacingStream(sys.stdout)): + result = _ORIGINAL_RUN_BENCHMARK(args) + + backend_metrics = _load_backend_metrics() + if backend_metrics is not None and "duration" in backend_metrics: + simulated_duration = backend_metrics["duration"] + if isinstance(result, dict): + result["duration"] = simulated_duration + _replace_output_file_duration(args, simulated_duration) + return result + + +def _extract_simulator_args(argv: list[str]) -> tuple[str, list[str]]: + parser = argparse.ArgumentParser(add_help=False) + parser.add_argument( + "--simulator-mode", + choices=("offline", "blocking"), + default="offline", + help=argparse.SUPPRESS, + ) + args, remaining = parser.parse_known_args(argv) + return args.simulator_mode, remaining + + +def _simulator_argument_parser(base_parser): + """Include simulator-owned datasets in SGLang's hard-coded CLI choices.""" + + class SimulatorArgumentParser(base_parser): + def add_argument(self, *name_or_flags, **kwargs): + choices = kwargs.get("choices") + if ( + "--dataset-name" in name_or_flags + and choices is not None + and "autobench" not in choices + ): + kwargs["choices"] = [*choices, "autobench"] + if "--warmup-requests" in name_or_flags: + kwargs["default"] = 0 + return super().add_argument(*name_or_flags, **kwargs) + + return SimulatorArgumentParser + + +def cli_main() -> None: + global _SIMULATOR_MODE + validate_benchmark_runtime() + if any(argument in ("-h", "--help") for argument in sys.argv[1:]): + print( + "SGLang Simulator option: " + "--simulator-mode {offline,blocking} (default: offline)\n" + ) + _SIMULATOR_MODE, remaining = _extract_simulator_args(sys.argv[1:]) + sys.argv = [sys.argv[0], *remaining] + + serving.get_request = simulator_get_request + serving.calculate_metrics = simulator_calculate_metrics + serving.run_benchmark = simulator_run_benchmark + install_aiohttp_json_hijack() + + print(f"SGLang Simulator benchmark mode: {_SIMULATOR_MODE.upper()}") + original_parser = serving.ArgumentParser + serving.ArgumentParser = _simulator_argument_parser(original_parser) + try: + serving.cli_main() + finally: + serving.ArgumentParser = original_parser + + +if __name__ == "__main__": + cli_main() diff --git a/benchmark/uno/README.md b/benchmark/uno/README.md new file mode 100644 index 000000000000..2036235c8076 --- /dev/null +++ b/benchmark/uno/README.md @@ -0,0 +1,148 @@ +# UNO full-dataset math evaluation + +`run_math_eval.py` evaluates AR, UNO, DFLASH, EAGLE, or EAGLE3 with identical +datasets, prompts, sampling parameters, and grading. It creates an in-process +`sgl.Engine`; there is no separate server process. Engine startup is excluded +from the timed interval, and no additional request warmup is run. + +The runner downloads pinned revisions of GSM8K, MATH-500, AIME 2024, AIME +2025, and AIME 2026. It applies the same boxed-answer instruction and Qwen +reasoning chat template to every engine, then grades with `math_verify`. + +Install SGLang with its evaluation dependencies: + +```bash +pip install -e "python[test]" +``` + +## Reproduce the H200 table + +Run from the SGLang repository root. Each invocation below produces one row of +the PR table. GSM8K and MATH-500 use one sample per problem; AIME 2025 uses ten +samples per problem, or 300 completions. + +```bash +export MODEL_PATH=Qwen/Qwen3-8B +export TOKENIZER_PATH=Qwen/Qwen3-8B +export UNO_LORA_PATH=s-sahoo/uno-qwen3-8B +export DATA_ROOT=/path/to/math-eval-data +export RESULT_ROOT=/path/to/math-eval-results + +COMMON_ARGS=( + --model-path "$MODEL_PATH" + --tokenizer-path "$TOKENIZER_PATH" + --data-root "$DATA_ROOT" + --context-length 40960 + --max-tokens 32768 + --temperature 1 + --top-k 50 + --top-p 0.95 + --random-seed 42 +) + +run_ar() { + local benchmark=$1 samples=$2 requests=$3 output_name=$4 + PYTHONPATH=python python -m benchmark.uno.run_math_eval \ + "${COMMON_ARGS[@]}" \ + --benchmark "$benchmark" \ + --num-samples "$samples" \ + --max-running-requests "$requests" \ + --output-dir "$RESULT_ROOT/$output_name" +} + +run_linear_uno() { + local benchmark=$1 samples=$2 requests=$3 output_name=$4 + PYTHONPATH=python python -m benchmark.uno.run_math_eval \ + "${COMMON_ARGS[@]}" \ + --benchmark "$benchmark" \ + --num-samples "$samples" \ + --max-running-requests "$requests" \ + --output-dir "$RESULT_ROOT/$output_name" \ + --speculative-algorithm UNO \ + --uno-lora-path "$UNO_LORA_PATH" \ + --speculative-num-steps 1 \ + --speculative-eagle-topk 1 \ + --speculative-num-draft-tokens 8 +} + +run_tree_uno() { + local benchmark=$1 samples=$2 requests=$3 output_name=$4 + PYTHONPATH=python python -m benchmark.uno.run_math_eval \ + "${COMMON_ARGS[@]}" \ + --benchmark "$benchmark" \ + --num-samples "$samples" \ + --max-running-requests "$requests" \ + --output-dir "$RESULT_ROOT/$output_name" \ + --speculative-algorithm UNO \ + --uno-lora-path "$UNO_LORA_PATH" \ + --speculative-num-steps 15 \ + --speculative-eagle-topk 32 \ + --speculative-num-draft-tokens 32 +} +``` + +Run the six batch-64 AR and linear `B/K/V = 8/1/8` rows: + +```bash +run_ar gsm8k 1 64 ar-gsm8k-c64 +run_linear_uno gsm8k 1 64 uno-linear-b8-k1-v8-gsm8k-c64 +run_ar math500 1 64 ar-math500-c64 +run_linear_uno math500 1 64 uno-linear-b8-k1-v8-math500-c64 +run_ar aime25 10 64 ar-aime25-c64 +run_linear_uno aime25 10 64 uno-linear-b8-k1-v8-aime25-c64 +``` + +Run the six batch-1 AR and tree `B/K/V = 16/32/32` rows: + +```bash +run_ar gsm8k 1 1 ar-gsm8k-c1 +run_tree_uno gsm8k 1 1 uno-tree-b16-k32-v32-gsm8k-c1 +run_ar math500 1 1 ar-math500-c1 +run_tree_uno math500 1 1 uno-tree-b16-k32-v32-math500-c1 +run_ar aime25 10 1 ar-aime25-c1 +run_tree_uno aime25 10 1 uno-tree-b16-k32-v32-aime25-c1 +``` + +Each output directory contains raw generations, per-answer grades, and +`summary.json` and `summary.md`. AR TPF is one. UNO TPF counts both full +target-model forwards in each cycle: the diffusion-pathway draft and +AR-pathway verification forwards. + +## Other speculative decoders + +The runner uses the same public option names as `sglang serve`. For example, +DFLASH can be evaluated with: + +```bash +PYTHONPATH=python python -m benchmark.uno.run_math_eval \ + "${COMMON_ARGS[@]}" \ + --benchmark math500 \ + --num-samples 1 \ + --output-dir "$RESULT_ROOT/dflash-b8-math500-c64" \ + --max-running-requests 64 \ + --speculative-algorithm DFLASH \ + --speculative-draft-model-path z-lab/Qwen3-8B-DFlash-b16 \ + --speculative-dflash-block-size 8 \ + --speculative-draft-attention-backend fa3 +``` + +EAGLE or EAGLE3 can be evaluated with the corresponding draft model: + +```bash +export EAGLE_DRAFT_MODEL=/path/to/compatible-eagle-draft-model + +PYTHONPATH=python python -m benchmark.uno.run_math_eval \ + "${COMMON_ARGS[@]}" \ + --benchmark math500 \ + --num-samples 1 \ + --output-dir "$RESULT_ROOT/eagle3-b8-math500-c64" \ + --max-running-requests 64 \ + --speculative-algorithm EAGLE3 \ + --speculative-draft-model-path "$EAGLE_DRAFT_MODEL" \ + --speculative-num-steps 7 \ + --speculative-eagle-topk 1 \ + --speculative-num-draft-tokens 8 +``` + +For EAGLE and DFLASH, TPF follows SGLang's acceptance-length convention and +counts generated tokens per target verification forward. diff --git a/benchmark/uno/__init__.py b/benchmark/uno/__init__.py new file mode 100644 index 000000000000..f67261e49707 --- /dev/null +++ b/benchmark/uno/__init__.py @@ -0,0 +1 @@ +"""Speculative-decoding benchmark utilities.""" diff --git a/benchmark/uno/math_data.py b/benchmark/uno/math_data.py new file mode 100644 index 000000000000..a3eef51fcd92 --- /dev/null +++ b/benchmark/uno/math_data.py @@ -0,0 +1,159 @@ +"""Pinned dataset preparation for speculative-decoding math evaluation.""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Any, NamedTuple + +MATH_INSTRUCTION = "Please reason step by step and put your final answer in \\boxed{}." + + +class BenchmarkConfig(NamedTuple): + name: str + expected_rows: int + instruction: str + chat_template_kwargs: dict[str, object] + + +BENCHMARKS = { + name: BenchmarkConfig( + name=name, + expected_rows=expected_rows, + instruction=MATH_INSTRUCTION, + chat_template_kwargs={"reasoning_effort": "high"}, + ) + for name, expected_rows in ( + ("gsm8k", 1319), + ("math500", 500), + ("aime24", 30), + ("aime25", 30), + ("aime26", 30), + ) +} + +DATASET_REVISIONS = { + "openai/gsm8k": "740312add88f781978c0658806c59bc2815b9866", + "HuggingFaceH4/MATH-500": "6e4ed1a2a79af7d8630a6b768ec859cb5af4d3be", + "hypaai/Hypa_AIME2024": "11ab79f0eed5f4fdf3d469b466663ab86bbd77c8", + "math-ai/aime25": "563bb8404243c5f09de6ec262f2db674fe5bce9b", + "math-ai/aime26": "79037aebdb6580008fb960d17cb21fd3099083e3", +} + + +def get_benchmark(name: str) -> BenchmarkConfig: + try: + return BENCHMARKS[name] + except KeyError as exc: + available = ", ".join(BENCHMARKS) + raise KeyError(f"Unknown benchmark {name!r}. Available: {available}") from exc + + +def _load_dataset( + repo_id: str, + config_name: str | None = None, + *, + split: str, +) -> list[dict[str, Any]]: + from datasets import load_dataset + + dataset = load_dataset( + repo_id, + config_name, + split=split, + revision=DATASET_REVISIONS[repo_id], + ) + return [dict(row) for row in dataset] + + +def _prepare_gsm8k() -> list[dict[str, Any]]: + rows = _load_dataset("openai/gsm8k", "main", split="test") + records = [] + for index, row in enumerate(rows): + prompt = f"Q: {row['question']}\nA: Let's think step by step." + records.append( + { + "row": index, + "ground_truth": row["answer"], + "chat_input": [{"role": "user", "content": prompt}], + } + ) + return records + + +def _prepare_math500() -> list[dict[str, Any]]: + rows = _load_dataset("HuggingFaceH4/MATH-500", split="test") + return [ + { + "row": index, + "ground_truth": row["answer"], + "chat_input": [{"role": "user", "content": row["problem"]}], + } + for index, row in enumerate(rows) + ] + + +def _prepare_aime(rows: list[dict[str, Any]]) -> list[dict[str, Any]]: + records = [] + for index, row in enumerate(rows): + prompt = f"Question: {row['problem']}\nAnswer:" + records.append( + { + "row": index, + "ground_truth": str(row.get("answer", row.get("solution"))).strip(), + "chat_input": [{"role": "user", "content": prompt}], + } + ) + return records + + +def _prepare_aime24() -> list[dict[str, Any]]: + return _prepare_aime(_load_dataset("hypaai/Hypa_AIME2024", split="english")) + + +def _prepare_aime25() -> list[dict[str, Any]]: + return _prepare_aime(_load_dataset("math-ai/aime25", split="test")) + + +def _prepare_aime26() -> list[dict[str, Any]]: + return _prepare_aime(_load_dataset("math-ai/aime26", split="test")) + + +BUILDERS = { + "gsm8k": _prepare_gsm8k, + "math500": _prepare_math500, + "aime24": _prepare_aime24, + "aime25": _prepare_aime25, + "aime26": _prepare_aime26, +} + + +def _row_count(path: Path) -> int: + with path.open("rb") as handle: + return sum(bool(line.strip()) for line in handle) + + +def prepare_benchmark_data(name: str, *, output_dir: Path) -> Path: + benchmark = get_benchmark(name) + output = output_dir / f"{name}.jsonl" + if output.is_file(): + count = _row_count(output) + if count == benchmark.expected_rows: + return output + raise ValueError( + f"{output} has {count} rows; expected {benchmark.expected_rows}" + ) + + records = BUILDERS[name]() + if len(records) != benchmark.expected_rows: + raise ValueError( + f"Expected {benchmark.expected_rows} rows for {name}, got {len(records)}" + ) + + output.parent.mkdir(parents=True, exist_ok=True) + temporary = output.with_suffix(".jsonl.tmp") + with temporary.open("w", encoding="utf-8") as handle: + for record in records: + handle.write(json.dumps(record, ensure_ascii=False) + "\n") + temporary.replace(output) + return output diff --git a/benchmark/uno/math_grader.py b/benchmark/uno/math_grader.py new file mode 100644 index 000000000000..2c1247d045d0 --- /dev/null +++ b/benchmark/uno/math_grader.py @@ -0,0 +1,310 @@ +"""Minimal math scorer adapted from Nano-vLLM-UNO's Eval360 grader.""" + +from __future__ import annotations + +import re +from typing import Any + +try: + from math_verify import parse, verify +except ModuleNotFoundError: # Allow CLI help without optional eval dependencies. + parse = None + verify = None + + +def _require_math_verify() -> None: + if parse is None or verify is None: + raise RuntimeError("math scoring requires the 'math_verify' package") + + +def extract_last_boxed_content(text: str | None) -> str | None: + if not text: + return None + matches = list(re.finditer(r"\\(?:boxed|fbox)\s*\{", text, re.IGNORECASE)) + for match in reversed(matches): + start = match.end() + depth = 1 + index = start + while index < len(text) and depth: + depth += (text[index] == "{") - (text[index] == "}") + index += 1 + if depth == 0: + return text[start : index - 1].strip().strip("$").strip() + return None + + +def _generation_list(row: dict[str, Any]) -> list[str]: + if isinstance(row.get("generations"), list): + return [str(value or "") for value in row["generations"]] + return [str(row.get("generation") or "")] + + +def _get_accuracy(correct: list[bool]) -> float: + return sum(correct) / len(correct) if correct else float("nan") + + +def _source_key(row: dict[str, Any], fallback: int) -> str: + return str(row.get("source_row", row.get("row", fallback))) + + +def normalize_answer_text(text: str | None) -> str: + if text is None: + return "" + text = str(text).strip().strip("$").strip() + gsm8k_answer = re.search(r"####\s*([^\n]+)", text) + if gsm8k_answer: + text = gsm8k_answer.group(1).strip() + text = re.sub(r"\\(?:boxed|fbox)\s*\{(.+)\}\s*$", r"\1", text) + text = text.replace("\\dfrac", "\\frac").replace("\\tfrac", "\\frac") + text = re.sub(r"\\(?:left|right|bigl|bigr|Bigl|Bigr|big|Big)", "", text) + text = text.replace("\\displaystyle", "") + text = re.sub(r"\\mathbf\s*\{([^{}]+)\}", r"\1", text) + text = re.sub(r"\\mathbf\s+([A-Za-z])", r"\1", text) + text = re.sub(r"\\\\\s*\[[^\]]+\]", r"\\\\", text) + text = re.sub(r"\\frac\s*\{([^{}]+)\}\s*\{([^{}]+)\}", r"\1/\2", text) + text = re.sub(r"\\frac\s*([+-]?\d+)\s*([+-]?\d+)", r"\1/\2", text) + text = text.replace("{,}", "") + text = re.sub(r"\\(?:,|;|!|:)", "", text) + text = text.replace("\\%", "%") + text = text.replace("\\$", "").replace("$", "") + text = re.sub(r"\\(?:text|mathrm)\s*\{([^{}]*)\}", r"\1", text) + text = re.sub(r"\^\{([^{}])\}", r"^\1", text) + text = re.sub(r"_\{([^{}])\}", r"_\1", text) + text = re.sub(r"\s+", "", text) + text = re.sub(r"(?<=\d),(?=\d{3}(?:\D|$))", "", text) + return re.sub(r"^[A-Za-z]+=(?=\\begin\{pmatrix\})", "", text) + + +def _has_plain_variable(text: str) -> bool: + text = re.sub(r"\\(?:text|mathrm)\s*\{[^{}]*\}", "", text) + text = re.sub(r"\\[A-Za-z]+", "", text) + return bool(re.search(r"[A-Za-z]", text)) + + +def _parse_boxed_content(boxed: str) -> Any: + _require_math_verify() + boxed = normalize_answer_text(boxed) + leading_number = re.match(r"\s*([-+]?[0-9][0-9,]*(?:\.[0-9]+)?)", boxed) + if leading_number and re.fullmatch(r"[A-Za-z]+", boxed[leading_number.end() :]): + answer = leading_number.group(1).replace(",", "") + try: + return parse(answer) + except Exception: + return answer + if _has_plain_variable(boxed): + return boxed + try: + parsed = parse(f"${boxed}$") or parse(boxed) + if parsed: + return parsed + except Exception: + pass + if leading_number: + answer = leading_number.group(1).replace(",", "") + try: + return parse(answer) + except Exception: + return answer + return boxed + + +def _parse_unboxed_answer(text: str) -> Any: + _require_math_verify() + try: + parsed = parse(f"${text}$") or parse(text) + if parsed: + return parsed + except Exception: + pass + + patterns = [ + r"The answer is:?\s*\$?([\-0-9\.,]+)", + r"#### ?\$?([\-0-9\.,]+)", + r"Therefore,? the answer is:?\s*\$?([\-0-9\.,]+)", + r"So,? the answer is:?\s*\$?([\-0-9\.,]+)", + r"Thus,? the answer is:?\s*\$?([\-0-9\.,]+)", + r"Hence,? the answer is:?\s*\$?([\-0-9\.,]+)", + r"Final answer:?\s*\$?([\-0-9\.,]+)", + r"The final answer is:?\s*\$?([\-0-9\.,]+)", + r"The answer is:?\s*\$?([\-0-9\.,]+)\s*(?:miles?|minutes?|hours?|dollars?|GB)?", + ( + r"=\s*\$?([\-0-9\.,]+)" + r"\s*(?:miles?|minutes?|hours?|dollars?|GB)?" + r"\.?\s*(?:The answer|$)" + ), + ] + for pattern in patterns: + matches = re.findall(pattern, text, re.IGNORECASE) + if matches: + answer = matches[-1].replace(",", "").strip().rstrip(".") + try: + return parse(answer) + except Exception: + return answer + + sentence_end = ( + r"(?:is|are|equals?|makes?|has|have|gets?|arrives?|covers?|travels?)" + r"\s+\$?([\-0-9\.,]+)(?:\s*(?:miles?|minutes?|hours?|dollars?|GB))?" + r"\.?\s*$" + ) + match = re.search(sentence_end, text, re.MULTILINE | re.IGNORECASE) + if match: + answer = match.group(1).replace(",", "").strip().rstrip(".") + try: + return parse(answer) + except Exception: + return answer + + for sentence in reversed(text.split(".")): + if "Human:" in sentence or "Assistant:" in sentence: + continue + numbers = re.findall(r"[-+]?[0-9]*\.?[0-9]+", sentence) + if numbers: + answer = numbers[-1].lstrip("0") or "0" + try: + return parse(answer) + except Exception: + return answer + return None + + +def _parse_answer(text: str) -> Any: + boxed = extract_last_boxed_content(text) + return _parse_boxed_content(boxed) if boxed else _parse_unboxed_answer(text) + + +def _answer_candidates(text: str) -> list[Any]: + boxed = extract_last_boxed_content(text) + candidates = ( + [_parse_boxed_content(boxed), normalize_answer_text(boxed), boxed] + if boxed + else [_parse_unboxed_answer(text)] + ) + unique = [] + seen = set() + for candidate in candidates: + if candidate is not None and repr(candidate) not in seen: + seen.add(repr(candidate)) + unique.append(candidate) + return unique + + +def _vector_components(text: str) -> tuple[str, ...] | None: + matrix = re.fullmatch(r"\\begin\{pmatrix\}(.+)\\end\{pmatrix\}", text) + if matrix: + return tuple(part for part in matrix.group(1).split(r"\\") if part) + tuple_match = re.fullmatch(r"\(([^()]+)\)", text) + if tuple_match and "," in tuple_match.group(1): + return tuple(part for part in tuple_match.group(1).split(",") if part) + return None + + +def _text_answers_match(answer: str, gold: str) -> bool: + answer_norm = normalize_answer_text(answer) + gold_norm = normalize_answer_text(gold) + if answer_norm == gold_norm: + return True + try: + if float(answer_norm.rstrip("%")) == float(gold_norm.rstrip("%")): + return True + except ValueError: + pass + if re.fullmatch(r"\([A-Za-z]\)", gold_norm) and answer_norm == gold_norm[1:-1]: + return True + answer_parts = _vector_components(answer_norm) + gold_parts = _vector_components(gold_norm) + return bool(answer_parts and answer_parts == gold_parts) + + +def _compare_answers(answer: Any, gold: str | None) -> bool: + _require_math_verify() + if not answer or gold is None: + return False + if isinstance(answer, str) and _text_answers_match(answer, gold): + return True + try: + if verify(answer, gold): + return True + gold_answer = _parse_answer(gold) + if gold_answer: + return bool(verify(gold_answer, answer)) + except Exception: + return isinstance(answer, str) and _text_answers_match(answer, gold) + + +def grade_math_row(row: dict[str, Any]) -> dict[str, Any]: + expected = row.get("ground_truth") + expected_values = expected if isinstance(expected, list) else [expected] + correct = [] + parsed_generations = [] + for generation in _generation_list(row): + candidates = _answer_candidates(generation) + parsed_generations.append([str(candidate) for candidate in candidates]) + correct.append( + any( + _compare_answers(candidate, str(gold)) + for candidate in candidates + for gold in expected_values + if gold is not None + ) + ) + return { + **row, + "parsed_generations": parsed_generations, + "correct": correct, + "accuracy": _get_accuracy(correct), + "grader": "math", + } + + +def score_math( + rows: list[dict[str, Any]], +) -> tuple[list[dict[str, Any]], dict[str, Any]]: + graded = [grade_math_row(row) for row in rows] + correct = [bool(value) for row in graded for value in row["correct"]] + by_source: dict[str, list[tuple[int, bool]]] = {} + for index, row in enumerate(graded): + try: + sample_index = int(row.get("sample_index", 0)) + except (TypeError, ValueError): + sample_index = 0 + samples = by_source.setdefault(_source_key(row, index), []) + samples.extend( + (sample_index + offset, bool(value)) + for offset, value in enumerate(row["correct"]) + ) + + per_problem = [ + [value for _, value in sorted(samples)] + for samples in by_source.values() + if samples + ] + samples_per_problem = max((len(samples) for samples in per_problem), default=0) + sample0 = [samples[0] for samples in per_problem] + summary = { + "grader": "math", + "num_rows": len(graded), + "num_problems": len(per_problem), + "samples_per_problem": samples_per_problem, + "num_correct": sum(correct), + "accuracy": _get_accuracy(correct), + } + if per_problem: + summary.update( + avg_at_1=_get_accuracy(sample0), + pass_at_1=_get_accuracy(sample0), + num_correct_at_1=sum(sample0), + ) + if samples_per_problem > 1: + summary.update( + { + f"avg_at_{samples_per_problem}": _get_accuracy(correct), + f"pass_at_{samples_per_problem}": _get_accuracy( + [any(samples) for samples in per_problem] + ), + f"num_pass_at_{samples_per_problem}": sum( + any(samples) for samples in per_problem + ), + } + ) + return graded, summary diff --git a/benchmark/uno/run_math_eval.py b/benchmark/uno/run_math_eval.py new file mode 100644 index 000000000000..1827b5c2fee9 --- /dev/null +++ b/benchmark/uno/run_math_eval.py @@ -0,0 +1,517 @@ +"""Run full-dataset AR or speculative-decoding math evaluation offline.""" + +from __future__ import annotations + +import argparse +import json +from collections import defaultdict +from pathlib import Path +from time import perf_counter +from typing import Any + +from benchmark.uno.math_data import ( + BENCHMARKS, + BenchmarkConfig, + get_benchmark, + prepare_benchmark_data, +) +from benchmark.uno.math_grader import score_math + +CONTEXT_LENGTH = 40960 +MAX_TOKENS = 2**15 +SPECULATIVE_ALGORITHMS = ("EAGLE", "EAGLE3", "DFLASH", "UNO") +SPECULATIVE_OPTION_NAMES = ( + "speculative_algorithm", + "speculative_draft_model_path", + "speculative_draft_model_revision", + "speculative_num_steps", + "speculative_eagle_topk", + "speculative_num_draft_tokens", + "speculative_dflash_block_size", + "speculative_draft_attention_backend", + "uno_lora_path", +) + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--model-path", default="Qwen/Qwen3-8B") + parser.add_argument("--tokenizer-path") + parser.add_argument("--revision") + parser.add_argument("--dtype", default="bfloat16") + parser.add_argument("--attention-backend", default="fa3") + parser.add_argument("--data-root", type=Path, required=True) + parser.add_argument("--output-dir", type=Path, required=True) + parser.add_argument( + "--speculative-algorithm", + type=str.upper, + choices=SPECULATIVE_ALGORITHMS, + ) + parser.add_argument("--speculative-draft-model-path") + parser.add_argument("--speculative-draft-model-revision") + parser.add_argument("--speculative-num-steps", type=int) + parser.add_argument("--speculative-eagle-topk", type=int) + parser.add_argument("--speculative-num-draft-tokens", type=int) + parser.add_argument("--speculative-dflash-block-size", type=int) + parser.add_argument("--speculative-draft-attention-backend") + parser.add_argument("--uno-lora-path") + parser.add_argument( + "--benchmark", + action="append", + choices=tuple(BENCHMARKS), + help="Benchmark to run; repeat to select multiple (default: all).", + ) + parser.add_argument("--limit", type=int, help="Maximum problems per benchmark.") + parser.add_argument("--num-samples", type=int, default=1) + parser.add_argument("--max-running-requests", type=int, default=4) + parser.add_argument("--context-length", type=int, default=CONTEXT_LENGTH) + parser.add_argument("--max-tokens", type=int, default=MAX_TOKENS) + parser.add_argument("--temperature", type=float, default=1.0) + parser.add_argument("--top-k", type=int, default=50) + parser.add_argument("--top-p", type=float, default=0.95) + parser.add_argument("--random-seed", type=int, default=42) + args = parser.parse_args() + _validate_args(parser=parser, args=args) + return args + + +def _validate_args( + *, parser: argparse.ArgumentParser, args: argparse.Namespace +) -> None: + for name in ( + "num_samples", + "max_running_requests", + "context_length", + "max_tokens", + "top_k", + ): + if getattr(args, name) <= 0: + parser.error(f"--{name.replace('_', '-')} must be positive") + if args.limit is not None and args.limit <= 0: + parser.error("--limit must be positive") + if not 0 < args.top_p <= 1: + parser.error("--top-p must be in (0, 1]") + if args.temperature < 0: + parser.error("--temperature must be non-negative") + + +def _context_reserve(args: argparse.Namespace) -> int: + width = args.speculative_num_draft_tokens + if width is None: + width = args.speculative_dflash_block_size or 1 + if args.speculative_algorithm == "DFLASH": + return 2 * width + if args.speculative_algorithm == "UNO" and (args.speculative_eagle_topk or 1) > 1: + draft_width = (args.speculative_num_steps or 0) + 1 + return max(width, draft_width) + 1 + return width + + +def _model_ref(value: str) -> str: + path = Path(value).expanduser() + return str(path.resolve()) if path.exists() else value + + +def _read_jsonl(path: Path) -> list[dict[str, Any]]: + with path.open(encoding="utf-8") as handle: + return [json.loads(line) for line in handle if line.strip()] + + +def _first_value(row: dict[str, Any], *names: str) -> Any | None: + return next((row[name] for name in names if row.get(name) is not None), None) + + +def _format_prompt( + *, + tokenizer: Any, + messages: list[dict[str, Any]], + benchmark: BenchmarkConfig, +) -> tuple[list[int], str]: + messages = [dict(message) for message in messages] + system_message = next( + (message for message in messages if message.get("role") == "system"), + None, + ) + if system_message is None: + messages.insert(0, {"role": "system", "content": benchmark.instruction}) + elif benchmark.instruction not in system_message["content"]: + existing = system_message["content"].strip() + system_message["content"] = f"{benchmark.instruction}\n\n{existing}" + + rendered = tokenizer.apply_chat_template( + messages, + tokenize=False, + add_generation_prompt=True, + **benchmark.chat_template_kwargs, + ) + token_ids = tokenizer.apply_chat_template( + messages, + tokenize=True, + add_generation_prompt=True, + return_dict=False, + **benchmark.chat_template_kwargs, + ) + return list(token_ids), str(rendered) + + +def _prepare_prompts( + *, + args: argparse.Namespace, + tokenizer: Any, + context_reserve: int, +) -> list[dict[str, Any]]: + prompts = [] + for benchmark_name in args.benchmark or BENCHMARKS: + benchmark = get_benchmark(benchmark_name) + data_path = prepare_benchmark_data( + benchmark.name, + output_dir=args.data_root, + ) + rows = _read_jsonl(data_path) + rows = rows[: args.limit] if args.limit is not None else rows + for row_index, row in enumerate(rows): + input_ids, rendered = _format_prompt( + tokenizer=tokenizer, + messages=row["chat_input"], + benchmark=benchmark, + ) + available = args.context_length - len(input_ids) - context_reserve + if available < args.max_tokens: + source = _first_value(row, "id", "problem_id", "index", "row") + source = row_index if source is None else source + raise ValueError( + f"{benchmark.name}:{source} has only {max(0, available)} " + "completion tokens available; increase --context-length" + ) + source = _first_value(row, "id", "problem_id", "index", "row") + source = row_index if source is None else source + for sample_index in range(args.num_samples): + prompt_id = f"{source}:sample{sample_index}" + prompts.append( + { + "id": prompt_id, + "benchmark": benchmark.name, + "input_ids": input_ids, + "row": { + "id": prompt_id, + "source_row": source, + "sample_index": sample_index, + "problem": rendered, + "ground_truth": row["ground_truth"], + "prompt_token_count": len(input_ids), + "resolved_max_tokens": args.max_tokens, + }, + } + ) + if not prompts: + raise ValueError("No prompts were prepared") + return prompts + + +def _engine_options( + *, + args: argparse.Namespace, +) -> dict[str, Any]: + tokenizer_path = _model_ref(args.tokenizer_path or args.model_path) + options: dict[str, Any] = { + "model_path": _model_ref(args.model_path), + "tokenizer_path": tokenizer_path, + "skip_tokenizer_init": True, + "context_length": args.context_length, + "dtype": args.dtype, + "random_seed": args.random_seed, + "max_running_requests": args.max_running_requests, + "attention_backend": args.attention_backend, + "log_level": "info", + } + if args.revision is not None: + options["revision"] = args.revision + for name in SPECULATIVE_OPTION_NAMES: + value = getattr(args, name) + if value is not None: + if name in ( + "speculative_draft_model_path", + "uno_lora_path", + ): + value = _model_ref(value) + options[name] = value + return options + + +def _generate( + *, + args: argparse.Namespace, + prompts: list[dict[str, Any]], + engine_options: dict[str, Any], +) -> tuple[list[dict[str, Any]], float]: + import sglang as sgl + + engine = sgl.Engine(**engine_options) + sampling = [ + { + "max_new_tokens": args.max_tokens, + "temperature": args.temperature, + "top_k": args.top_k, + "top_p": args.top_p, + } + for _ in prompts + ] + try: + start = perf_counter() + outputs = engine.generate( + input_ids=[prompt["input_ids"] for prompt in prompts], + sampling_params=sampling, + rid=[f"{prompt['benchmark']}:{prompt['id']}" for prompt in prompts], + ) + elapsed = perf_counter() - start + finally: + engine.shutdown() + return outputs, elapsed + + +def _build_rows( + *, + prompts: list[dict[str, Any]], + outputs: list[dict[str, Any]], + tokenizer: Any, + speculative_algorithm: str | None, +) -> list[dict[str, Any]]: + rows = [] + for prompt, output in zip(prompts, outputs, strict=True): + token_ids = list(output["output_ids"]) + metadata = output["meta_info"] + verify_forwards = int(metadata.get("spec_verify_ct", 0)) + if speculative_algorithm == "UNO": + # Both UNO pathways are full target-model forward passes. + num_forwards = 2 * verify_forwards + elif speculative_algorithm is not None: + # EAGLE and DFLASH TPF conventionally count target verification. + num_forwards = verify_forwards + else: + num_forwards = len(token_ids) + rows.append( + { + "benchmark": prompt["benchmark"], + **prompt["row"], + "output_ids": token_ids, + "generation": tokenizer.decode(token_ids, skip_special_tokens=True), + "num_tokens": len(token_ids), + "num_forwards": num_forwards, + "tokens_per_forward": _divide(len(token_ids), num_forwards), + "sglang_meta_info": metadata, + } + ) + return rows + + +def _divide(numerator: int | float, denominator: int | float) -> float | None: + return numerator / denominator if denominator else None + + +def _write_json(path: Path, value: dict[str, Any]) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text( + json.dumps(value, indent=2, ensure_ascii=False) + "\n", + encoding="utf-8", + ) + + +def _write_jsonl(path: Path, rows: list[dict[str, Any]]) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + with path.open("w", encoding="utf-8") as handle: + for row in rows: + handle.write(json.dumps(row, ensure_ascii=False) + "\n") + + +def _benchmark_summary(rows: list[dict[str, Any]]) -> dict[str, Any]: + tokens = sum(row["num_tokens"] for row in rows) + forwards = sum(row["num_forwards"] for row in rows) + tpfs = [ + row["tokens_per_forward"] + for row in rows + if row["tokens_per_forward"] is not None + ] + return { + "num_tokens": tokens, + "num_forwards": forwards, + "tokens_per_forward": _divide(tokens, forwards), + "unweighted_mean_tokens_per_forward": (sum(tpfs) / len(tpfs) if tpfs else None), + } + + +def _write_markdown(path: Path, summary: dict[str, Any]) -> None: + lines = [ + "| Dataset | Accuracy | TPF | tok/s | tok/s/request |", + "| --- | ---: | ---: | ---: | ---: |", + ] + for name, metrics in summary["by_benchmark"].items(): + tokens_per_second = metrics.get("tokens_per_second") + per_request = metrics.get("tokens_per_second_per_request") + tokens_per_forward = metrics.get("tokens_per_forward") + tps = f"{tokens_per_second:.2f}" if tokens_per_second is not None else "—" + request_tps = f"{per_request:.2f}" if per_request is not None else "—" + tpf = f"{tokens_per_forward:.3f}" if tokens_per_forward is not None else "—" + lines.append( + f"| {name} | {metrics['accuracy']:.2%} | {tpf} | {tps} | {request_tps} |" + ) + lines.extend( + [ + "| **Average** | " + f"**{summary['macro_average_accuracy']:.2%}** | " + f"**{summary['macro_average_tokens_per_forward']:.3f}** | " + f"**{summary['tokens_per_second']:.2f}** | " + f"**{summary['tokens_per_second_per_request']:.2f}** |", + "", + "Accuracy and TPF in the Average row are unweighted means across " + "datasets. Throughput is aggregate output tokens divided by timed " + "generation seconds; tok/s/request divides it by max running requests.", + ] + ) + path.write_text("\n".join(lines) + "\n", encoding="utf-8") + + +def _write_results( + *, + args: argparse.Namespace, + rows: list[dict[str, Any]], + generation_seconds: float, + engine_options: dict[str, Any], +) -> None: + rows_by_benchmark: dict[str, list[dict[str, Any]]] = defaultdict(list) + for row in rows: + output_row = {key: value for key, value in row.items() if key != "benchmark"} + rows_by_benchmark[row["benchmark"]].append(output_row) + + benchmark_metrics = {} + totals = {"prompts": 0, "completions": 0, "correct": 0, "tokens": 0, "forwards": 0} + request_tpfs = [] + for name, benchmark_rows in rows_by_benchmark.items(): + output_dir = args.output_dir / name + _write_jsonl(output_dir / "generations.jsonl", benchmark_rows) + graded, scores = score_math(benchmark_rows) + _write_jsonl(output_dir / "grades.jsonl", graded) + _write_json(output_dir / "scores.json", scores) + + metrics = { + "num_prompts": scores["num_problems"], + "num_completions": scores["num_rows"], + "num_correct": scores["num_correct"], + "accuracy": scores["accuracy"], + **_benchmark_summary(benchmark_rows), + } + benchmark_metrics[name] = metrics + totals["prompts"] += scores["num_problems"] + totals["completions"] += scores["num_rows"] + totals["correct"] += scores["num_correct"] + totals["tokens"] += metrics["num_tokens"] + totals["forwards"] += metrics["num_forwards"] + request_tpfs.extend( + row["tokens_per_forward"] + for row in benchmark_rows + if row["tokens_per_forward"] is not None + ) + + tokens_per_second = totals["tokens"] / generation_seconds + tokens_per_second_per_request = tokens_per_second / args.max_running_requests + if len(benchmark_metrics) == 1: + only_metrics = next(iter(benchmark_metrics.values())) + only_metrics["tokens_per_second"] = tokens_per_second + only_metrics["tokens_per_second_per_request"] = tokens_per_second_per_request + + dataset_accuracies = [value["accuracy"] for value in benchmark_metrics.values()] + dataset_tpfs = [ + value["tokens_per_forward"] + for value in benchmark_metrics.values() + if value["tokens_per_forward"] is not None + ] + summary = { + "engine": "sglang-offline", + "mode": _mode_name(args), + "model_path": engine_options["model_path"], + "tokenizer_path": engine_options["tokenizer_path"], + "revision": args.revision, + "dtype": args.dtype, + "attention_backend": args.attention_backend, + "speculative_parameters": { + name: engine_options[name] + for name in SPECULATIVE_OPTION_NAMES + if name in engine_options + }, + "max_running_requests": args.max_running_requests, + "benchmarks": list(rows_by_benchmark), + "context_length": args.context_length, + "max_tokens": args.max_tokens, + "num_samples": args.num_samples, + "random_seed": args.random_seed, + "sampling": { + "temperature": args.temperature, + "top_k": args.top_k, + "top_p": args.top_p, + }, + "num_prompts": totals["prompts"], + "num_completions": totals["completions"], + "num_correct": totals["correct"], + "accuracy": _divide(totals["correct"], totals["completions"]), + "macro_average_accuracy": sum(dataset_accuracies) / len(dataset_accuracies), + "generation_seconds": generation_seconds, + "num_tokens": totals["tokens"], + "tokens_per_second": tokens_per_second, + "tokens_per_second_per_request": tokens_per_second_per_request, + "num_forwards": totals["forwards"], + "tokens_per_forward": _divide(totals["tokens"], totals["forwards"]), + "macro_average_tokens_per_forward": sum(dataset_tpfs) / len(dataset_tpfs), + "unweighted_mean_tokens_per_forward": ( + sum(request_tpfs) / len(request_tpfs) if request_tpfs else None + ), + "by_benchmark": benchmark_metrics, + } + _write_json(args.output_dir / "summary.json", summary) + _write_markdown(args.output_dir / "summary.md", summary) + print(json.dumps(summary, indent=2)) + + +def _mode_name(args: argparse.Namespace) -> str: + algorithm = args.speculative_algorithm + if algorithm is None: + return "ar" + if algorithm == "UNO": + return "uno-tree" if (args.speculative_eagle_topk or 1) > 1 else "uno-linear" + return algorithm.lower() + + +def main() -> None: + args = parse_args() + from transformers import AutoTokenizer + + engine_options = _engine_options(args=args) + tokenizer = AutoTokenizer.from_pretrained( + engine_options["tokenizer_path"], + use_fast=True, + trust_remote_code=True, + ) + prompts = _prepare_prompts( + args=args, + tokenizer=tokenizer, + context_reserve=_context_reserve(args), + ) + outputs, elapsed = _generate( + args=args, + prompts=prompts, + engine_options=engine_options, + ) + rows = _build_rows( + prompts=prompts, + outputs=outputs, + tokenizer=tokenizer, + speculative_algorithm=args.speculative_algorithm, + ) + _write_results( + args=args, + rows=rows, + generation_seconds=elapsed, + engine_options=engine_options, + ) + + +if __name__ == "__main__": + main() diff --git a/docker/Dockerfile b/docker/Dockerfile index aa8c71896caa..8d6ed1b7b1fc 100644 --- a/docker/Dockerfile +++ b/docker/Dockerfile @@ -4,9 +4,8 @@ FROM nvidia/cuda:${CUDA_VERSION}-cudnn-devel-ubuntu24.04 AS base ARG TARGETARCH ARG BUILD_TYPE=all ARG BRANCH_TYPE=remote -ARG SGL_KERNEL_VERSION=0.4.6.post1 +ARG SGL_KERNEL_VERSION=0.4.7 ARG SGL_VERSION -ARG SGL_DEEP_GEMM_VERSION=0.1.5.post2 ARG USE_LATEST_SGLANG=0 ARG GDRCOPY_VERSION=2.5.1 ARG SGL_NCCL_VERSION=2.30.7 @@ -14,9 +13,9 @@ ARG PIP_DEFAULT_INDEX ARG UBUNTU_MIRROR ARG GITHUB_ARTIFACTORY=github.com ARG INSTALL_FLASHINFER_JIT_CACHE=0 -ARG FLASHINFER_VERSION=0.6.17 +ARG FLASHINFER_VERSION=0.6.18 ARG MOONCAKE_VERSION=0.3.13 -ARG MSCCLPP_VERSION=sglang-v0.9.1 +ARG MSCCLPP_VERSION=sglang-v0.10.1 ENV DEBIAN_FRONTEND=noninteractive \ CUDA_HOME=/usr/local/cuda \ @@ -185,25 +184,11 @@ RUN curl --proto '=https' --tlsv1.2 --retry 3 --retry-delay 2 -sSf https://sh.ru RUN --mount=type=cache,target=/root/.cache/pip \ python3 -m pip install --upgrade pip setuptools wheel html5lib six \ && case "$CUDA_VERSION" in \ - 12.6.3) CUINDEX=126 ;; \ - 12.9.2) CUINDEX=129 ;; \ - 13.0.3) CUINDEX=130 ;; \ + 13.0.3) ;; \ *) echo "Unsupported CUDA version: $CUDA_VERSION" && exit 1 ;; \ esac \ - && if [ "$CUDA_VERSION" = "12.6.3" ]; then \ - python3 -m pip install https://${GITHUB_ARTIFACTORY}/sgl-project/whl/releases/download/v${SGL_KERNEL_VERSION}/sglang_kernel-${SGL_KERNEL_VERSION}+cu124-cp310-abi3-manylinux2014_$(uname -m).whl --force-reinstall --no-deps \ - ; \ - elif [ "$CUDA_VERSION" = "12.9.2" ]; then \ - python3 -m pip install https://${GITHUB_ARTIFACTORY}/sgl-project/whl/releases/download/v${SGL_KERNEL_VERSION}/sglang_kernel-${SGL_KERNEL_VERSION}+cu129-cp310-abi3-manylinux2014_$(uname -m).whl --force-reinstall --no-deps \ - ; \ - elif [ "$CUDA_VERSION" = "13.0.3" ]; then \ - # --no-deps prevents pip from pulling torch from default PyPI - python3 -m pip install sglang-kernel==${SGL_KERNEL_VERSION} --force-reinstall --no-deps \ - ; \ - else \ - echo "Unsupported CUDA version: $CUDA_VERSION" && exit 1 \ - ; \ - fi + # --no-deps prevents pip from pulling torch from default PyPI + && python3 -m pip install sglang-kernel==${SGL_KERNEL_VERSION} --force-reinstall --no-deps # Copy dep spec + Rust crate source + proto files. setuptools-rust compiles the # Rust extension during the stub wheel build; the crate's build.rs references @@ -214,15 +199,11 @@ COPY rust/sglang-grpc /tmp/sglang_deps/rust/sglang-grpc COPY rust/sglang-mm /tmp/sglang_deps/rust/sglang-mm COPY proto /tmp/sglang_deps/proto -# Install sglang dependencies (torch, transformers, etc.). CUDA 12 DeepEP -# wheels live only on the SGLang index, so preinstall the local-version wheel; -# it satisfies the public-version pyproject pin during the full dependency solve. +# Install sglang dependencies (torch, transformers, etc.). # Generate constraints.txt to prevent reinstalling these deps in later stages. RUN --mount=type=cache,target=/root/.cache/pip \ --mount=type=cache,target=/root/.cargo/registry \ case "$CUDA_VERSION" in \ - 12.6.3) CUINDEX=126 ;; \ - 12.9.2) CUINDEX=129 ;; \ 13.0.3) CUINDEX=130 ;; \ *) echo "Unsupported CUDA version: $CUDA_VERSION" && exit 1 ;; \ esac \ @@ -232,27 +213,7 @@ RUN --mount=type=cache,target=/root/.cache/pip \ && echo '__version__ = "0.0.0"' > sglang/version.py \ && touch README.md \ && touch LICENSE \ - && SGL_DEEP_EP_VERSION="$(sed -n 's/^[[:space:]]*"sgl-deep-ep==\([^"]*\)",/\1/p' pyproject.toml)" \ - && test -n "${SGL_DEEP_EP_VERSION}" \ - && if [ "${CUDA_VERSION%%.*}" = "12" ]; then \ - python3 -m pip install \ - "sgl-deep-ep==${SGL_DEEP_EP_VERSION}+cu129" \ - --index-url "https://docs.sglang.ai/whl/cu129/" \ - --no-deps; \ - fi \ - && if [ "${CUDA_VERSION%%.*}" = "12" ]; then \ - sed -i 's/cuda-python>=13\.0/cuda-python>=12,<13/' pyproject.toml && \ - sed -i 's/flashinfer_python\[cu13\]/flashinfer_python[cu12]/' pyproject.toml && \ - sed -i 's/nvidia-cutlass-dsl\[cu13\]/nvidia-cutlass-dsl/' pyproject.toml; \ - fi \ && python3 -m pip install --extra-index-url https://download.pytorch.org/whl/cu${CUINDEX} ".[${BUILD_TYPE}]" \ - && if [ "${CUDA_VERSION%%.*}" = "12" ]; then \ - pip list --format=freeze | awk -F'==' '/-cu13(==|$)/ {print $1}' \ - | xargs -r python3 -m pip uninstall -y && \ - python3 -m pip install --index-url https://download.pytorch.org/whl/cu${CUINDEX} \ - torch==2.13.0 torchvision==0.28.0 torchaudio==2.11.0 --force-reinstall; \ - python3 -m pip install https://github.com/sgl-project/whl/releases/download/v${SGL_DEEP_GEMM_VERSION}/sgl_deep_gemm-${SGL_DEEP_GEMM_VERSION}+cu129-py3-none-manylinux2014_$(uname -m).whl --force-reinstall; \ - fi \ && cd /sgl-workspace \ && rm -rf /tmp/sglang_deps \ && pip freeze | grep -v "^sglang==" > /sgl-workspace/constraints.txt @@ -315,8 +276,6 @@ ARG FLASHINFER_VERSION # Stage jit-cache/cubin artifacts into /flashinfer_jit_output for clean COPY later RUN --mount=type=cache,target=/root/.cache/pip \ case "$CUDA_VERSION" in \ - 12.6.3) CUINDEX=126 ;; \ - 12.9.2) CUINDEX=129 ;; \ 13.0.3) CUINDEX=130 ;; \ *) echo "Unsupported CUDA version: $CUDA_VERSION" && exit 1 ;; \ esac \ @@ -503,21 +462,12 @@ RUN --mount=type=cache,target=/var/cache/apt,id=framework-apt \ # Install Mooncake RUN --mount=type=cache,target=/root/.cache/pip \ - CUDA_MAJOR="${CUDA_VERSION%%.*}" && \ - if [ "$CUDA_MAJOR" -ge 13 ]; then \ - python3 -m pip install mooncake-transfer-engine-cuda13==${MOONCAKE_VERSION}; \ - else \ - python3 -m pip install mooncake-transfer-engine==${MOONCAKE_VERSION}; \ - fi + python3 -m pip install mooncake-transfer-engine-cuda13==${MOONCAKE_VERSION} # Install MSCCL++ Python dependencies and package (builds extension via CMake through pip) RUN --mount=type=cache,target=/root/.cache/pip \ git clone --depth=1 --branch ${MSCCLPP_VERSION} https://${GITHUB_ARTIFACTORY}/microsoft/mscclpp.git /tmp/mscclpp \ && case "${CUDA_VERSION}" in \ - 12.*) \ - CMAKE_ARGS="-DMSCCLPP_BYPASS_GPU_CHECK=ON -DMSCCLPP_USE_CUDA=ON -DMSCCLPP_GPU_ARCHS=80,90,100,100a,103,103a" \ - python3 -m pip install "/tmp/mscclpp[cuda12]"; \ - ;; \ 13.*) \ CMAKE_ARGS="-DMSCCLPP_BYPASS_GPU_CHECK=ON -DMSCCLPP_USE_CUDA=ON -DMSCCLPP_GPU_ARCHS=80,90,100,100a,103,103a" \ python3 -m pip install "/tmp/mscclpp[cuda13]"; \ @@ -553,22 +503,19 @@ RUN --mount=type=cache,target=/root/.cache/pip \ # resolving dependencies and generating constraints.txt. DeepEP needs 2.30.7 # at runtime; apply the override only after the constrained dependency solve. RUN --mount=type=cache,target=/root/.cache/pip \ - if [ "${CUDA_VERSION%%.*}" = "13" ]; then \ - python3 -m pip install --force-reinstall --no-deps \ - "nvidia-nccl-cu13==${SGL_NCCL_VERSION}"; \ - fi + python3 -m pip install --force-reinstall --no-deps \ + "nvidia-nccl-cu13==${SGL_NCCL_VERSION}" -# Per-CUDA-major package installs. The `nixl` stub package is needed (it owns -# the `nixl` import path) but unconditionally requires nixl-cu12, so we install -# it with --no-deps and pair it with the matching nixl-cu12 / nixl-cu13 binary -# to avoid shipping wrong-CUDA libs on cu13 images. -RUN --mount=type=cache,target=/root/.cache/pip if [ "${CUDA_VERSION%%.*}" = "12" ]; then \ - python3 -m pip install nixl nixl-cu12 --no-deps ; \ - python3 -m pip install "cuda-python>=12,<13" ; \ -elif [ "${CUDA_VERSION%%.*}" = "13" ]; then \ - python3 -m pip install nixl nixl-cu13 --no-deps ; \ - python3 -m pip install "cuda-python>=13,<14" ; \ -fi +# The `nixl` stub package is needed (it owns the `nixl` import path) but +# unconditionally requires nixl-cu12, so install it with --no-deps and pair it +# with the nixl-cu13 binary to avoid shipping wrong-CUDA libs. +RUN --mount=type=cache,target=/root/.cache/pip \ + python3 -m pip install nixl nixl-cu13 --no-deps \ + && python3 -m pip install "cuda-python>=13,<14" + +# Install the latest available AI Dynamo prerelease from NVIDIA's package index. +RUN --mount=type=cache,target=/root/.cache/pip \ + python3 -m pip install --pre --extra-index-url https://pypi.nvidia.com/ ai-dynamo # Add yank script COPY --chown=root:root --chmod=755 docker/configs/yank /usr/local/bin/yank @@ -634,11 +581,6 @@ RUN if [ "$BRANCH_TYPE" = "local" ]; then \ # Clean up __pycache__/tests/pyc in same RUN to avoid writing ~28k files to layer RUN --mount=type=cache,target=/root/.cache/pip \ cd /sgl-workspace/sglang \ - && if [ "${CUDA_VERSION%%.*}" = "12" ]; then \ - sed -i 's/cuda-python>=13\.0/cuda-python>=12,<13/' python/pyproject.toml && \ - sed -i 's/flashinfer_python\[cu13\]/flashinfer_python[cu12]/' python/pyproject.toml && \ - sed -i 's/nvidia-cutlass-dsl\[cu13\]/nvidia-cutlass-dsl/' python/pyproject.toml; \ - fi \ && python3 -m pip install --no-deps -e "python[${BUILD_TYPE}]" \ && kernels lock python \ && ( success=0; \ @@ -655,8 +597,8 @@ RUN --mount=type=cache,target=/root/.cache/pip \ echo "sgl-kernel cubin download failed, retrying in 30s..." && sleep 30; \ done; \ # x86: if no prebuilt sgl-flash-attn3 variant matches this torch+CUDA \ - # combo (e.g. cu129 publishes no torch>=2.10 cubin), fall back to \ - # runtime JIT instead of failing the build, mirroring the aarch64 branch. \ + # combo, fall back to runtime JIT instead of failing the build, \ + # mirroring the aarch64 branch. \ if [ "$success" != "1" ]; then \ echo "WARNING: no matching sgl-flash-attn3 cubin variant for this torch+CUDA; kernels will be JIT-compiled at runtime"; \ success=1; \ @@ -667,6 +609,8 @@ RUN --mount=type=cache,target=/root/.cache/pip \ && ( if [ -f python/kernels.lock ]; then mv python/kernels.lock /root/.cache/sglang/; fi ) \ && ( find /opt/sglang/lib/python3.12/site-packages -type d -name "__pycache__" -exec rm -rf {} + 2>/dev/null || true ) +ENV SGLANG_RUST_BUILD_MODE=never + # Install pre-built gateway artifacts from parallel builder COPY --from=gateway_builder /build/sgl-model-gateway-bin /opt/sglang/bin/sgl-model-gateway @@ -719,7 +663,8 @@ ARG GDRCOPY_VERSION=2.5.1 ENV DEBIAN_FRONTEND=noninteractive \ CUDA_HOME=/usr/local/cuda \ - GDRCOPY_HOME=/usr/src/gdrdrv-${GDRCOPY_VERSION}/ + GDRCOPY_HOME=/usr/src/gdrdrv-${GDRCOPY_VERSION}/ \ + SGLANG_RUST_BUILD_MODE=never # Add GKE default lib and bin locations + CUDA compiler paths for FlashInfer JIT ENV PATH="${PATH}:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/cuda/nvvm/bin" \ @@ -737,6 +682,7 @@ RUN --mount=type=cache,target=/var/cache/apt,id=runtime-apt \ ca-certificates \ netcat-openbsd \ curl \ + ffmpeg \ git \ # Runtime libraries libopenmpi3 \ @@ -821,12 +767,13 @@ COPY --from=framework_final /root/.cache/huggingface /root/.cache/huggingface COPY --from=framework_final /root/.cache/sglang /root/.cache/sglang # Copy GDRCopy runtime libraries (but not the build artifacts) -COPY --from=framework_final /usr/lib/libgdrapi.so* /usr/lib/ +COPY --from=framework_final /usr/lib/*-linux-gnu/libgdrapi.so* /usr/lib/ COPY --from=framework_final /usr/bin/gdrcopy_* /usr/bin/ COPY --from=framework_final /usr/src/gdrdrv-2.5.1 /usr/src/gdrdrv-2.5.1 # Fix DeepEP IBGDA symlink in runtime -RUN ln -sf /usr/lib/$(uname -m)-linux-gnu/libmlx5.so.1 /usr/lib/$(uname -m)-linux-gnu/libmlx5.so +RUN ln -sf /usr/lib/$(uname -m)-linux-gnu/libmlx5.so.1 /usr/lib/$(uname -m)-linux-gnu/libmlx5.so \ + && ldconfig WORKDIR /sgl-workspace/sglang diff --git a/docker/Dockerfile.cu134 b/docker/Dockerfile.cu134 index b6f85442d9d0..86e58bfe5867 100644 --- a/docker/Dockerfile.cu134 +++ b/docker/Dockerfile.cu134 @@ -99,7 +99,7 @@ ARG PIP_DEFAULT_INDEX ARG UBUNTU_MIRROR ARG GITHUB_ARTIFACTORY=github.com ARG INSTALL_FLASHINFER_JIT_CACHE=1 -ARG FLASHINFER_VERSION=0.6.18rc10 +ARG FLASHINFER_VERSION=0.6.18 ARG CUTLASS_DSL_VERSION=4.8.0.dev0 ARG MOONCAKE_VERSION=0.3.12.post1 ARG MSCCLPP_VERSION=sglang-v0.9.1 @@ -354,7 +354,9 @@ ARG DEEPGEMM_REF ARG SGL_DEEP_GEMM_VERSION=0.1.5.post2 ARG TVM_FFI_VERSION=0.1.11 +# DeepJIT's exception handling requires elfutils/libdwfl.h and libelf headers. RUN yum install -y --nogpgcheck git wget tar gcc gcc-c++ make \ + elfutils-devel elfutils-libelf-devel \ && yum clean all && rm -rf /var/cache/yum RUN --mount=type=cache,id=cu134-wheels-pip,target=/root/.cache/pip \ @@ -493,19 +495,11 @@ RUN curl --proto '=https' --tlsv1.2 --retry 3 --retry-delay 2 -sSf https://sh.ru RUN --mount=type=cache,target=/root/.cache/pip \ python3 -m pip install --upgrade pip setuptools wheel html5lib six \ && case "$CUDA_VERSION" in \ - 12.6.3) CUINDEX=126 ;; \ - 12.9.2) CUINDEX=129 ;; \ 13.0.3) CUINDEX=130 ;; \ 13.4.0) CUINDEX=134 ;; \ *) echo "Unsupported CUDA version: $CUDA_VERSION" && exit 1 ;; \ esac \ - && if [ "$CUDA_VERSION" = "12.6.3" ]; then \ - python3 -m pip install https://${GITHUB_ARTIFACTORY}/sgl-project/whl/releases/download/v${SGL_KERNEL_VERSION}/sglang_kernel-${SGL_KERNEL_VERSION}+cu124-cp310-abi3-manylinux2014_$(uname -m).whl --force-reinstall --no-deps \ - ; \ - elif [ "$CUDA_VERSION" = "12.9.2" ]; then \ - python3 -m pip install https://${GITHUB_ARTIFACTORY}/sgl-project/whl/releases/download/v${SGL_KERNEL_VERSION}/sglang_kernel-${SGL_KERNEL_VERSION}+cu129-cp310-abi3-manylinux2014_$(uname -m).whl --force-reinstall --no-deps \ - ; \ - elif [ "$CUDA_VERSION" = "13.0.3" ]; then \ + && if [ "$CUDA_VERSION" = "13.0.3" ]; then \ # --no-deps prevents pip from pulling torch from default PyPI python3 -m pip install sglang-kernel==${SGL_KERNEL_VERSION} --force-reinstall --no-deps \ ; \ @@ -527,15 +521,11 @@ COPY rust/sglang-grpc /tmp/sglang_deps/rust/sglang-grpc COPY rust/sglang-mm /tmp/sglang_deps/rust/sglang-mm COPY proto /tmp/sglang_deps/proto -# Install sglang dependencies (torch, transformers, etc.). CUDA 12 DeepEP -# wheels live only on the SGLang index, so preinstall the local-version wheel; -# it satisfies the public-version pyproject pin during the full dependency solve. +# Install sglang dependencies (torch, transformers, etc.). # Generate constraints.txt to prevent reinstalling these deps in later stages. RUN --mount=type=cache,target=/root/.cache/pip \ --mount=type=cache,target=/root/.cargo/registry \ case "$CUDA_VERSION" in \ - 12.6.3) CUINDEX=126 ;; \ - 12.9.2) CUINDEX=129 ;; \ 13.0.3) CUINDEX=130 ;; \ 13.4.0) CUINDEX=134 ;; \ *) echo "Unsupported CUDA version: $CUDA_VERSION" && exit 1 ;; \ @@ -546,19 +536,6 @@ RUN --mount=type=cache,target=/root/.cache/pip \ && echo '__version__ = "0.0.0"' > sglang/version.py \ && touch README.md \ && touch LICENSE \ - && SGL_DEEP_EP_VERSION="$(sed -n 's/^[[:space:]]*"sgl-deep-ep==\([^"]*\)",/\1/p' pyproject.toml)" \ - && test -n "${SGL_DEEP_EP_VERSION}" \ - && if [ "${CUDA_VERSION%%.*}" = "12" ]; then \ - python3 -m pip install \ - "sgl-deep-ep==${SGL_DEEP_EP_VERSION}+cu129" \ - --index-url "https://docs.sglang.ai/whl/cu129/" \ - --no-deps; \ - fi \ - && if [ "${CUDA_VERSION%%.*}" = "12" ]; then \ - sed -i 's/cuda-python>=13\.0/cuda-python>=12,<13/' pyproject.toml && \ - sed -i 's/flashinfer_python\[cu13\]/flashinfer_python[cu12]/' pyproject.toml && \ - sed -i 's/nvidia-cutlass-dsl\[cu13\]/nvidia-cutlass-dsl/' pyproject.toml; \ - fi \ && if [ "$CUDA_VERSION" = "13.4.0" ]; then \ sed -i "s|^ \"flashinfer_python\[cu13\]==.*| \"flashinfer_python[cu13]==${FLASHINFER_VERSION}\",|" pyproject.toml; \ fi \ @@ -579,17 +556,8 @@ RUN --mount=type=cache,target=/root/.cache/pip \ TORCH_INDEX="https://download.pytorch.org/whl/cu${CUINDEX}"; \ fi \ && python3 -m pip install --extra-index-url "${TORCH_INDEX}" ".[${BUILD_TYPE}]" \ - && if [ "${CUDA_VERSION%%.*}" = "13" ]; then \ - python3 -m pip install --force-reinstall --no-deps \ - "nvidia-nccl-cu13==${NCCL_VERSION}"; \ - fi \ - && if [ "${CUDA_VERSION%%.*}" = "12" ]; then \ - pip list --format=freeze | awk -F'==' '/-cu13(==|$)/ {print $1}' \ - | xargs -r python3 -m pip uninstall -y && \ - python3 -m pip install --index-url https://download.pytorch.org/whl/cu${CUINDEX} \ - torch==2.13.0 torchvision==0.28.0 torchaudio==2.11.0 --force-reinstall; \ - python3 -m pip install https://github.com/sgl-project/whl/releases/download/v${SGL_DEEP_GEMM_VERSION}/sgl_deep_gemm-${SGL_DEEP_GEMM_VERSION}+cu129-py3-none-manylinux2014_$(uname -m).whl --force-reinstall; \ - fi \ + && python3 -m pip install --force-reinstall --no-deps \ + "nvidia-nccl-cu13==${NCCL_VERSION}" \ && if [ "$CUDA_VERSION" = "13.4.0" ]; then \ python3 -m pip install --force-reinstall --no-deps /tmp/cu134_wheels/*.whl; \ fi \ @@ -643,8 +611,6 @@ ARG FLASHINFER_VERSION # Stage jit-cache/cubin artifacts into /flashinfer_jit_output for clean COPY later RUN --mount=type=cache,target=/root/.cache/pip \ case "$CUDA_VERSION" in \ - 12.6.3) CUINDEX=126 ;; \ - 12.9.2) CUINDEX=129 ;; \ 13.0.3) CUINDEX=130 ;; \ 13.4.0) CUINDEX=134 ;; \ *) echo "Unsupported CUDA version: $CUDA_VERSION" && exit 1 ;; \ @@ -830,21 +796,12 @@ RUN --mount=type=cache,target=/var/cache/apt,id=framework-apt \ # Install Mooncake RUN --mount=type=cache,target=/root/.cache/pip \ - CUDA_MAJOR="${CUDA_VERSION%%.*}" && \ - if [ "$CUDA_MAJOR" -ge 13 ]; then \ - python3 -m pip install mooncake-transfer-engine-cuda13==${MOONCAKE_VERSION}; \ - else \ - python3 -m pip install mooncake-transfer-engine==${MOONCAKE_VERSION}; \ - fi + python3 -m pip install mooncake-transfer-engine-cuda13==${MOONCAKE_VERSION} # Install MSCCL++ Python dependencies and package (builds extension via CMake through pip) RUN --mount=type=cache,target=/root/.cache/pip \ git clone --depth=1 --branch ${MSCCLPP_VERSION} https://${GITHUB_ARTIFACTORY}/microsoft/mscclpp.git /tmp/mscclpp \ && case "${CUDA_VERSION}" in \ - 12.*) \ - CMAKE_ARGS="-DMSCCLPP_BYPASS_GPU_CHECK=ON -DMSCCLPP_USE_CUDA=ON -DMSCCLPP_GPU_ARCHS=80,90,100,100a,103,103a" \ - python3 -m pip install "/tmp/mscclpp[cuda12]"; \ - ;; \ 13.*) \ CMAKE_ARGS="-DMSCCLPP_BYPASS_GPU_CHECK=ON -DMSCCLPP_USE_CUDA=ON -DMSCCLPP_GPU_ARCHS=80,90,100,100a,103,103a" \ python3 -m pip install "/tmp/mscclpp[cuda13]"; \ @@ -882,17 +839,16 @@ RUN --mount=type=cache,target=/root/.cache/pip \ termplotlib \ "runai-model-streamer[s3,gcs,azure]>=0.15.7" -# Per-CUDA-major package installs. The `nixl` stub package is needed (it owns -# the `nixl` import path) but unconditionally requires nixl-cu12, so we install -# it with --no-deps and pair it with the matching nixl-cu12 / nixl-cu13 binary -# to avoid shipping wrong-CUDA libs on cu13 images. -RUN --mount=type=cache,target=/root/.cache/pip if [ "${CUDA_VERSION%%.*}" = "12" ]; then \ - python3 -m pip install nixl nixl-cu12 --no-deps ; \ - python3 -m pip install "cuda-python>=12,<13" ; \ -elif [ "${CUDA_VERSION%%.*}" = "13" ]; then \ - python3 -m pip install nixl nixl-cu13 --no-deps ; \ - python3 -m pip install "cuda-python>=13,<14" ; \ -fi +# The `nixl` stub package is needed (it owns the `nixl` import path) but +# unconditionally requires nixl-cu12, so install it with --no-deps and pair it +# with the nixl-cu13 binary to avoid shipping wrong-CUDA libs. +RUN --mount=type=cache,target=/root/.cache/pip \ + python3 -m pip install nixl nixl-cu13 --no-deps \ + && python3 -m pip install "cuda-python>=13,<14" + +# Install the latest available AI Dynamo prerelease from NVIDIA's package index. +RUN --mount=type=cache,target=/root/.cache/pip \ + python3 -m pip install --pre --extra-index-url https://pypi.nvidia.com/ ai-dynamo # Add yank script COPY --chown=root:root --chmod=755 docker/configs/yank /usr/local/bin/yank @@ -940,6 +896,9 @@ ARG CUDA_VERSION ARG SGL_VERSION ARG USE_LATEST_SGLANG +# TODO(Jialin): Set SGLANG_RUST_BUILD_MODE=never after Rust TreeCore supports +# the PyTorch 2.15 nightly used by this preview image. + WORKDIR /sgl-workspace COPY --from=local_src /src /tmp/local_src @@ -958,11 +917,6 @@ RUN if [ "$BRANCH_TYPE" = "local" ]; then \ # Clean up __pycache__/tests/pyc in same RUN to avoid writing ~28k files to layer RUN --mount=type=cache,target=/root/.cache/pip \ cd /sgl-workspace/sglang \ - && if [ "${CUDA_VERSION%%.*}" = "12" ]; then \ - sed -i 's/cuda-python>=13\.0/cuda-python>=12,<13/' python/pyproject.toml && \ - sed -i 's/flashinfer_python\[cu13\]/flashinfer_python[cu12]/' python/pyproject.toml && \ - sed -i 's/nvidia-cutlass-dsl\[cu13\]/nvidia-cutlass-dsl/' python/pyproject.toml; \ - fi \ && if [ "$CUDA_VERSION" = "13.4.0" ]; then \ sed -i "s|^ \"flashinfer_python\[cu13\]==.*| \"flashinfer_python[cu13]==${FLASHINFER_VERSION}\",|" python/pyproject.toml; \ fi \ @@ -982,8 +936,8 @@ RUN --mount=type=cache,target=/root/.cache/pip \ echo "sgl-kernel cubin download failed, retrying in 30s..." && sleep 30; \ done; \ # x86: if no prebuilt sgl-flash-attn3 variant matches this torch+CUDA \ - # combo (e.g. cu129 publishes no torch>=2.10 cubin), fall back to \ - # runtime JIT instead of failing the build, mirroring the aarch64 branch. \ + # combo, fall back to runtime JIT instead of failing the build, \ + # mirroring the aarch64 branch. \ if [ "$success" != "1" ]; then \ echo "WARNING: no matching sgl-flash-attn3 cubin variant for this torch+CUDA; kernels will be JIT-compiled at runtime"; \ success=1; \ diff --git a/docker/kimi_k3/apply_deepep_k3_patch.sh b/docker/kimi_k3/apply_deepep_k3_patch.sh deleted file mode 100755 index 6f4506196653..000000000000 --- a/docker/kimi_k3/apply_deepep_k3_patch.sh +++ /dev/null @@ -1,169 +0,0 @@ -#!/bin/bash -# Kimi-K3 DeepEP patch + rebuild -- IMAGE-BUILD variant of the runtime patcher. -# -# Stock DeepEP (deepseek-ai@d28bd67, shipped in the base image at -# /sgl-workspace/DeepEP) does not serve Kimi-K3: -# - LL topk capped at 11; K3 routes top-16 -> assert at internode_ll.cu -# - SWITCH_HIDDEN lacks 3584 (K3 latent-MoE dispatch dim) -# - normal-internode issues an unaligned 64-bit SourceMeta access for packed -# K3 FP8 scales when dispatch crosses the 8-rank NVL domain (EP > 8) -# - CUDA 13 relocated the cccl headers -> the stock build cannot find them -# -# Differences vs the devbox runtime script (why a separate copy): -# 1. arch comes from $TORCH_CUDA_ARCH_LIST (no GPU at image-build time; the -# runtime script probes torch.cuda.get_device_capability which needs a GPU) -# 2. adds the setup.py cccl include-dir fix (required to COMPILE on CUDA 13; -# the devbox had it applied manually) -# 3. adds the configs.cuh CPU/cycle timeout bump (cross-node init headroom) -# -# Idempotent (grep/count-guarded). The default wheel contains native cubins for -# Hopper sm_90, B200 sm_100a, and GB300 sm_103a. -set -euo pipefail -: "${TORCH_CUDA_ARCH_LIST:=9.0;10.0a;10.3a}" -DEEPEP_DIR="${DEEPEP_DIR:-/sgl-workspace/DeepEP}" -DEEPEP_COMMIT="${DEEPEP_COMMIT:-d28bd676c2120573c9f1425f0c16c39faa4117e6}" - -# PyTorch accepts CUDA architecture lists separated by spaces or semicolons. -read -r -a CUDA_ARCHES <<< "${TORCH_CUDA_ARCH_LIST//;/ }" -[ "${#CUDA_ARCHES[@]}" -gt 0 ] || { echo "ERROR: TORCH_CUDA_ARCH_LIST is empty"; exit 1; } -for CUDA_ARCH in "${CUDA_ARCHES[@]}"; do - [[ "$CUDA_ARCH" =~ ^[0-9]+\.[0-9]+a?$ ]] || { - echo "ERROR: unsupported CUDA architecture '$CUDA_ARCH' in TORCH_CUDA_ARCH_LIST='$TORCH_CUDA_ARCH_LIST'" - exit 1 - } -done - -# The base image is expected to ship the DeepEP source here; clone-pin if not. -if [ ! -d "$DEEPEP_DIR/csrc" ]; then - echo "== $DEEPEP_DIR missing — cloning deepseek-ai/DeepEP @ $DEEPEP_COMMIT" - git clone --recursive https://github.com/deepseek-ai/DeepEP.git "$DEEPEP_DIR" - git -C "$DEEPEP_DIR" checkout "$DEEPEP_COMMIT" - git -C "$DEEPEP_DIR" submodule update --init --recursive -fi -cd "$DEEPEP_DIR" - -echo "== [1/6] internode_ll.cu: LL topk caps 9/11 -> 16" -TOPK_CAPS_EXPECTED=$(grep -Eci "kNumMaxTop[Kk] = (9|11|16)" csrc/kernels/internode_ll.cu || true) -[ "$TOPK_CAPS_EXPECTED" -ge 2 ] || { - echo "ERROR: expected >=2 recognized topk caps, found $TOPK_CAPS_EXPECTED" - exit 1 -} -sed -i 's/constexpr int kNumMaxTopK = 11;/constexpr int kNumMaxTopK = 16;/' csrc/kernels/internode_ll.cu -sed -i 's/constexpr int kNumMaxTopK = 9;/constexpr int kNumMaxTopK = 16;/' csrc/kernels/internode_ll.cu -sed -i 's/constexpr int kNumMaxTopk = 9;/constexpr int kNumMaxTopk = 16;/' csrc/kernels/internode_ll.cu -sed -i 's/constexpr int kNumMaxTopk = 11;/constexpr int kNumMaxTopk = 16;/' csrc/kernels/internode_ll.cu -TOPK_CAPS_PATCHED=$(grep -ci "kNumMaxTop[Kk] = 16" csrc/kernels/internode_ll.cu || true) -TOPK_CAPS_UNPATCHED=$(grep -Eci "kNumMaxTop[Kk] = (9|11)" csrc/kernels/internode_ll.cu || true) -if [ "$TOPK_CAPS_PATCHED" -ne "$TOPK_CAPS_EXPECTED" ] || [ "$TOPK_CAPS_UNPATCHED" -ne 0 ]; then - echo "ERROR: expected $TOPK_CAPS_EXPECTED topk caps patched, found $TOPK_CAPS_PATCHED patched and $TOPK_CAPS_UNPATCHED unpatched" - exit 1 -fi - -echo "== [2/6] launch.cuh: SWITCH_HIDDEN += case 3584 (K3 latent MoE)" -grep -q "case_macro(3584)" csrc/kernels/launch.cuh || \ - sed -i 's/case 4096: case_macro(4096);/case 3584: case_macro(3584); case 4096: case_macro(4096);/' csrc/kernels/launch.cuh -grep -q "case_macro(3584)" csrc/kernels/launch.cuh || { echo "ERROR: hidden 3584 case not applied"; exit 1; } - -echo "== [3/6] configs.cuh: raise CPU/cycle timeouts 100s -> 1000s (cross-node init headroom)" -# '@' delimiter (pattern contains '#'); '$' anchor avoids the ENABLE_FAST_DEBUG '...10' line. -sed -i 's@^#define NUM_CPU_TIMEOUT_SECS 100$@#define NUM_CPU_TIMEOUT_SECS 1000@' csrc/kernels/configs.cuh -sed -i 's@^#define NUM_TIMEOUT_CYCLES 200000000000ull@#define NUM_TIMEOUT_CYCLES 2000000000000ull@' csrc/kernels/configs.cuh -grep -q "NUM_CPU_TIMEOUT_SECS 1000$" csrc/kernels/configs.cuh || echo " WARN: CPU timeout not raised (layout changed?) — non-fatal" - -echo "== [4/6] tests/test_low_latency.py: nvfp4 tolerance 0.007 -> 0.008 (topk16 noise)" -python3 - <<'EOF' -p = "tests/test_low_latency.py" -s = open(p).read() -old = "elif dispatch_use_nvfp4:\n diff_threshold = 0.007" -new = "elif dispatch_use_nvfp4:\n diff_threshold = 0.008" -if old in s: - open(p, "w").write(s.replace(old, new)); print(" patched") -else: - print(" already patched or layout changed (skipped)") -EOF - -echo "== [5/6] internode.cu: 4-byte-aligned SourceMeta scalar access (EP>8 normal internode)" -python3 - <<'EOF' -from pathlib import Path -path = Path("csrc/kernels/internode.cu") -source = path.read_text() -changed = False -abi_old = 'EP_STATIC_ASSERT(sizeof(SourceMeta) % sizeof(int) == 0, "Invalid size of `SourceMeta`");' -abi_new = 'EP_STATIC_ASSERT(sizeof(SourceMeta) == 2 * sizeof(int), "SourceMeta scalar access requires exactly two int fields");' -if source.count(abi_old) == 2 and source.count(abi_new) == 0: - source = source.replace(abi_old, abi_new); changed = True; print(" tightened both SourceMeta ABI assertions") -elif source.count(abi_old) == 0 and source.count(abi_new) == 2: - print(" ABI assertions already tightened") -else: - raise SystemExit(f"ERROR: unexpected SourceMeta ABI layout: old={source.count(abi_old)}, new={source.count(abi_new)}") -replacements = ( - ("sender store", - """ // Copy source metadata into symmetric send buffer - if (lane_id < num_topk_ranks) - st_na_global(reinterpret_cast(dst_send_buffers[lane_id]), src_meta); -""", - """ // SourceMeta may be only 4-byte aligned after packed scales. - // Store its two int fields separately to avoid an unaligned 64-bit store. - if (lane_id < num_topk_ranks) { - auto meta_values = reinterpret_cast(dst_send_buffers[lane_id]); - st_na_global(meta_values, src_meta.src_rdma_rank); - st_na_global(meta_values + 1, src_meta.is_token_in_nvl_rank_bits); - } -"""), - ("forwarder load", - """ auto src_meta = ld_nc_global(reinterpret_cast(shifted + hidden_bytes + scale_bytes)); -""", - """ auto src_meta_values = reinterpret_cast(shifted + hidden_bytes + scale_bytes); - SourceMeta src_meta; - src_meta.src_rdma_rank = ld_nc_global(src_meta_values); - src_meta.is_token_in_nvl_rank_bits = ld_nc_global(src_meta_values + 1); -"""), - ("receiver load", - """ auto meta = ld_nc_global(reinterpret_cast(shifted + hidden_bytes + scale_bytes)); -""", - """ auto meta_values = reinterpret_cast(shifted + hidden_bytes + scale_bytes); - SourceMeta meta; - meta.src_rdma_rank = ld_nc_global(meta_values); - meta.is_token_in_nvl_rank_bits = ld_nc_global(meta_values + 1); -"""), -) -for label, old, new in replacements: - oc, nc = source.count(old), source.count(new) - if oc == 1 and nc == 0: - source = source.replace(old, new); changed = True; print(f" patched {label}") - elif oc == 0 and nc == 1: - print(f" {label} already patched") - else: - raise SystemExit(f"ERROR: unexpected {label} layout: old={oc}, new={nc}") -for unsafe in ("st_na_global(reinterpret_cast", "ld_nc_global(reinterpret_cast"): - if unsafe in source: - raise SystemExit(f"ERROR: unsafe SourceMeta access remains: {unsafe}") -if changed: - path.write_text(source) -EOF - -echo "== [5b] setup.py: add CUDA 13 cccl include dir (compile fix)" -grep -q "/usr/local/cuda/include/cccl" setup.py || \ - sed -i "s#\( include_dirs = \['csrc/'\]\)#\1\n include_dirs.append('/usr/local/cuda/include/cccl')#" setup.py -grep -q "/usr/local/cuda/include/cccl" setup.py || { echo "ERROR: cccl include not added to setup.py"; exit 1; } - -echo "== [6/6] rebuild + reinstall for TORCH_CUDA_ARCH_LIST=$TORCH_CUDA_ARCH_LIST (no GPU needed)" -rm -rf build dist -TORCH_CUDA_ARCH_LIST="$TORCH_CUDA_ARCH_LIST" python3 setup.py bdist_wheel -pip install dist/*.whl --force-reinstall --no-deps - -echo "== verify installed cubin arches" -SO=$(find /usr/local/lib/python3*/dist-packages -maxdepth 1 -name "deep_ep_cpp*.so" | head -1) -[ -n "$SO" ] || { echo "ERROR: installed deep_ep_cpp shared object not found"; exit 1; } -CUBIN_LIST=$(cuobjdump --list-elf "$SO") -VERIFIED_SMS=() -for CUDA_ARCH in "${CUDA_ARCHES[@]}"; do - SM="sm_${CUDA_ARCH/./}" - grep -Eq "(^|[^[:alnum:]_])${SM}([^[:alnum:]_]|$)" <<< "$CUBIN_LIST" || { - echo "ERROR: cubin arch $SM not found in $SO" - printf '%s\n' "$CUBIN_LIST" | head - exit 1 - } - VERIFIED_SMS+=("$SM") -done -echo "== OK: DeepEP rebuilt (topk16 + hidden3584 + SourceMeta align + cccl) for ${VERIFIED_SMS[*]}" diff --git a/docker/kimi_k3/kimi_k3_cu12.Dockerfile b/docker/kimi_k3/kimi_k3_cu12.Dockerfile deleted file mode 100644 index b9eff4f379cf..000000000000 --- a/docker/kimi_k3/kimi_k3_cu12.Dockerfile +++ /dev/null @@ -1,113 +0,0 @@ -# Kimi-K3 serving image (x86_64 / CUDA 12.9 / sm_90 + sm_100a). -# -# Base ships stock SGLang (editable at /sgl-workspace/sglang), DeepEP source -# (deepseek-ai@d28bd67 at /sgl-workspace/DeepEP), the deep_gemm pip package, -# and the CUDA 12.9 toolchain. -# -# This image adds the three Kimi-K3-specific pieces that stock lacks: -# 1. the Kimi-K3 SGLang code (this repo), editable-installed -# 2. DeepEP patch + rebuild: -# topk 11->16, SWITCH_HIDDEN += 3584, EP>8 SourceMeta alignment, -# and cross-node timeout headroom; rebuilt for sm_90 and sm_100a only -# 3. DeepGEMM upgrade to 0.1.5.post2: -# official MegaMoE runtime-JIT header with Kimi-K3 SiTU support -# -# Build (on/for x86_64; nvcc cross-compiles the DeepEP cubin, no GPU needed): -# docker build -f docker/kimi_k3/kimi_k3_cu12.Dockerfile \ -# --build-arg 'TORCH_CUDA_ARCH_LIST=9.0;10.0a' -t kimi-k3-cu129 . -# -# The FlashInfer MXFP4 MoE runner cubins are installed in the image below. -# The runner is auto-selected on SM100; the remaining kernel sources -# JIT-compile from the installed FlashInfer wheel on first launch and are -# cached. - -FROM lmsysorg/sglang:v0.5.16-cu129 AS base - -ARG SGL_DEEP_GEMM_VERSION="0.1.5.post2" -ARG NVIMGCODEC_VERSION="0.9.0.20" - -# Current Kimi-K3 source auto-discovers and builds its PyO3 extensions. -ARG RUST_VERSION="1.90.0" -ENV RUSTUP_HOME="/usr/local/rustup" \ - CARGO_HOME="/usr/local/cargo" \ - PATH="/usr/local/cargo/bin:${PATH}" -RUN apt-get update && \ - apt-get install -y --no-install-recommends \ - ca-certificates \ - curl && \ - curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | \ - sh -s -- -y --no-modify-path --profile minimal \ - --default-toolchain "${RUST_VERSION}" && \ - cargo --version && \ - rustc --version && \ - rm -rf /var/lib/apt/lists/* - -# Build one DeepEP wheel with native cubins for Hopper and B200. This CUDA 12.9 -# recipe intentionally excludes GB300 (sm_103/sm_103a). -ARG TORCH_CUDA_ARCH_LIST="9.0;10.0a" -RUN set -eu; \ - for arch in $(printf '%s' "${TORCH_CUDA_ARCH_LIST}" | tr ';' ' '); do \ - case "${arch}" in \ - 10.3|10.3a) \ - echo "ERROR: CUDA 12.9 image does not support SM103 (${arch})" >&2; \ - exit 1 \ - ;; \ - esac; \ - done - -# --- 1. Kimi-K3 SGLang code (replaces the base's stock sglang, editable) --- -# Keep the installed extension modules, but discard Rust and pip build -# artifacts that are not used at runtime. -RUN rm -rf /sgl-workspace/sglang && \ - git clone --branch main \ - https://github.com/sgl-project/sglang.git /sgl-workspace/sglang && \ - cd /sgl-workspace/sglang && \ - rm -rf .git && \ - test ! -e .git && \ - pip install -e python --no-deps && \ - rm -rf \ - rust/target \ - rust/sglang-grpc/target \ - rust/sglang-mm/target \ - rust/sglang-server/target \ - /usr/local/cargo/registry \ - /root/.cache/pip - -# --- 2. DeepEP: patch (topk16 / hidden3584 / SourceMeta) + multi-arch rebuild --- -RUN TORCH_CUDA_ARCH_LIST="${TORCH_CUDA_ARCH_LIST}" \ - bash /sgl-workspace/sglang/docker/kimi_k3/apply_deepep_k3_patch.sh && \ - rm -rf /sgl-workspace/DeepEP/build /sgl-workspace/DeepEP/dist - -# --- 3. DeepGEMM: upgrade to the first release with Kimi-K3 SiTU --- -# The v0.5.16 base contains DeepGEMM 0.1.4.post1. PyPI publishes the CUDA 13 -# build, so CUDA 12.9 uses the matching official release asset. -RUN python3 -m pip install --no-deps --force-reinstall \ - "https://github.com/sgl-project/whl/releases/download/v${SGL_DEEP_GEMM_VERSION}/sgl_deep_gemm-${SGL_DEEP_GEMM_VERSION}+cu129-py3-none-manylinux2014_x86_64.whl" - -# High-fidelity GPU JPEG decode. The K3 processor enables nvJPEG interpolated -# chroma upsampling through nvImageCodec and zero-copy DLPack handoff to Torch. -RUN python3 -m pip install \ - "nvidia-nvimgcodec-cu12[all]==${NVIMGCODEC_VERSION}" && \ - rm -rf /root/.cache/pip - -# Install the matching official FlashInfer package trio. A mixed -# Python/cubin/JIT-cache installation fails at import time. -# flashinfer-python and flashinfer-cubin are CUDA-independent packages; the -# JIT-cache wheel is selected from the official CUDA 12.9 index. -RUN python3 -m pip uninstall -y \ - flashinfer-python flashinfer-cubin flashinfer-jit-cache && \ - rm -rf /root/.cache/flashinfer /root/.cache/pip && \ - python3 -m pip install --no-deps \ - "flashinfer-python==0.6.17" && \ - python3 -m pip install --no-deps \ - "flashinfer-cubin==0.6.17" \ - --index-url https://flashinfer.ai/whl && \ - python3 -m pip install --no-deps \ - "flashinfer-jit-cache==0.6.17" \ - --index-url https://flashinfer.ai/whl/cu129 && \ - python3 -c 'from importlib.metadata import version; expected = "0.6.17"; assert version("flashinfer-python").split("+", 1)[0] == expected; assert version("flashinfer-cubin").split("+", 1)[0] == expected; assert version("flashinfer-jit-cache").startswith(expected + "+cu129"), version("flashinfer-jit-cache")' && \ - rm -rf /root/.cache/pip - -ENV FLASHINFER_VERSION="0.6.17" - -WORKDIR /sgl-workspace/sglang diff --git a/docker/kimi_k3/kimi_k3_cu13.Dockerfile b/docker/kimi_k3/kimi_k3_cu13.Dockerfile deleted file mode 100644 index ed738bab4464..000000000000 --- a/docker/kimi_k3/kimi_k3_cu13.Dockerfile +++ /dev/null @@ -1,101 +0,0 @@ -# Kimi-K3 serving image (aarch64 / sm_90 + sm_100a + sm_103a). -# -# Base ships stock SGLang (editable at /sgl-workspace/sglang), DeepEP source -# (deepseek-ai@d28bd67 at /sgl-workspace/DeepEP), the deep_gemm pip package, -# and the CUDA 13 toolchain (nvcc + /usr/local/cuda/include/cccl). -# -# This image adds the three Kimi-K3-specific pieces that stock lacks: -# 1. the Kimi-K3 SGLang code (this repo), editable-installed -# 2. DeepEP patch + rebuild: -# topk 11->16, SWITCH_HIDDEN += 3584, EP>8 SourceMeta alignment, -# cross-node timeout headroom, CUDA-13 cccl include; rebuilt for -# sm_90, sm_100a, and sm_103a -# 3. DeepGEMM upgrade to 0.1.5.post2: -# official MegaMoE runtime-JIT header with Kimi-K3 SiTU support -# -# Build (on/for aarch64; nvcc cross-compiles the DeepEP cubin, no GPU needed): -# docker build -f docker/kimi_k3/kimi_k3_cu13.Dockerfile \ -# --build-arg 'TORCH_CUDA_ARCH_LIST=9.0;10.0a;10.3a' -t kimi-k3 . -# -# The FlashInfer MXFP4 MoE runner cubins are installed in the image below. -# The runner is auto-selected on SM100/103; the remaining kernel sources -# JIT-compile from the installed FlashInfer wheel on first launch and are -# cached. - -FROM lmsysorg/sglang:v0.5.16 AS base - -ARG SGL_DEEP_GEMM_VERSION="0.1.5.post2" -ARG NVIMGCODEC_VERSION="0.9.0.20" - -# Current Kimi-K3 source auto-discovers and builds its PyO3 extensions. -ARG RUST_VERSION="1.90.0" -ENV RUSTUP_HOME="/usr/local/rustup" \ - CARGO_HOME="/usr/local/cargo" \ - PATH="/usr/local/cargo/bin:${PATH}" -RUN apt-get update && \ - apt-get install -y --no-install-recommends \ - ca-certificates \ - curl && \ - curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | \ - sh -s -- -y --no-modify-path --profile minimal \ - --default-toolchain "${RUST_VERSION}" && \ - cargo --version && \ - rustc --version && \ - rm -rf /var/lib/apt/lists/* - -# Build one DeepEP wheel with native cubins for Hopper, B200, and GB300. -ARG TORCH_CUDA_ARCH_LIST="9.0;10.0a;10.3a" - -# --- 1. Kimi-K3 SGLang code (replaces the base's stock sglang, editable) --- -# Keep the installed extension modules, but discard Rust and pip build -# artifacts that are not used at runtime. -RUN rm -rf /sgl-workspace/sglang && \ - git clone --branch main \ - https://github.com/sgl-project/sglang.git /sgl-workspace/sglang && \ - cd /sgl-workspace/sglang && \ - rm -rf .git && \ - test ! -e .git && \ - pip install -e python --no-deps && \ - rm -rf \ - rust/target \ - rust/sglang-grpc/target \ - rust/sglang-mm/target \ - rust/sglang-server/target \ - /usr/local/cargo/registry \ - /root/.cache/pip - -# --- 2. DeepEP: patch (topk16 / hidden3584 / SourceMeta / cccl) + multi-arch rebuild --- -RUN TORCH_CUDA_ARCH_LIST="${TORCH_CUDA_ARCH_LIST}" \ - bash /sgl-workspace/sglang/docker/kimi_k3/apply_deepep_k3_patch.sh && \ - rm -rf /sgl-workspace/DeepEP/build /sgl-workspace/DeepEP/dist - -# --- 3. DeepGEMM: upgrade to the first release with Kimi-K3 SiTU --- -# The v0.5.16 base contains DeepGEMM 0.1.4.post1. -RUN python3 -m pip install --no-deps --force-reinstall \ - "sgl-deep-gemm==${SGL_DEEP_GEMM_VERSION}" - -# High-fidelity GPU JPEG decode. The K3 processor enables nvJPEG interpolated -# chroma upsampling through nvImageCodec and zero-copy DLPack handoff to Torch. -RUN python3 -m pip install \ - "nvidia-nvimgcodec-cu13[all]==${NVIMGCODEC_VERSION}" && \ - rm -rf /root/.cache/pip - -# Install the matching official FlashInfer package trio. A mixed -# Python/cubin/JIT-cache installation fails at import time. -RUN python3 -m pip uninstall -y \ - flashinfer-python flashinfer-cubin flashinfer-jit-cache && \ - rm -rf /root/.cache/flashinfer /root/.cache/pip && \ - python3 -m pip install --no-deps \ - "flashinfer-python==0.6.17" && \ - python3 -m pip install --no-deps \ - "flashinfer-cubin==0.6.17" \ - --index-url https://flashinfer.ai/whl && \ - python3 -m pip install --no-deps \ - "flashinfer-jit-cache==0.6.17" \ - --index-url https://flashinfer.ai/whl/cu130 && \ - python3 -c 'from importlib.metadata import version; expected = "0.6.17"; packages = ("flashinfer-python", "flashinfer-cubin", "flashinfer-jit-cache"); actual = {package: version(package).split("+", 1)[0] for package in packages}; assert all(value == expected for value in actual.values()), actual' && \ - rm -rf /root/.cache/pip - -ENV FLASHINFER_VERSION="0.6.17" - -WORKDIR /sgl-workspace/sglang diff --git a/docker/musa.Dockerfile b/docker/musa.Dockerfile new file mode 100644 index 000000000000..528abc6c5c63 --- /dev/null +++ b/docker/musa.Dockerfile @@ -0,0 +1,126 @@ +# syntax=docker/dockerfile:1.7 + +# Usage: +# docker build -f docker/musa.Dockerfile -t sglang:main-musa . +# docker run --rm -it --network=host \ +# --env MTHREADS_VISIBLE_DEVICES=0 \ +# --env MTHREADS_DRIVER_CAPABILITIES=all \ +# --shm-size=32g \ +# sglang:main-musa + +ARG BASE_IMAGE=ubuntu:22.04 + +FROM ${BASE_IMAGE} + +SHELL ["/bin/bash", "-o", "pipefail", "-c"] + +ARG DEBIAN_FRONTEND=noninteractive +ARG MUSA_APT_SOURCE=https://dl.mthreads.com/repo/repository/ubuntu2204/ +ARG MUSA_PIP_INDEX_URL=https://dl.mthreads.com/repo/api/pypi/pypi/simple +ARG PYPI_INDEX_URL=https://pypi.org/simple + +ENV MUSA_HOME=/usr/local/musa \ + MATE_MUSA_ARCH_LIST=3.1 \ + PATH=/usr/local/musa/bin:/usr/local/musa/mudnn/bin:${PATH} \ + LD_LIBRARY_PATH=/usr/local/musa/lib:/usr/local/musa/mudnn/lib:/usr/local/mtshmem/lib:/usr/local/lib \ + TORCH_EXTENSIONS_DIR=/root/.cache/torch_extensions \ + TRITON_CACHE_DIR=/root/.triton/cache + +RUN apt-get update \ + && apt-get install -y --no-install-recommends \ + bash \ + build-essential \ + ca-certificates \ + cmake \ + curl \ + ffmpeg \ + g++ \ + gcc \ + git \ + libmkl-core \ + libmkl-def \ + libmkl-gnu-thread \ + libmkl-intel-lp64 \ + libmkl-vml-def \ + libomp-dev \ + libopenmpi3 \ + libsndfile1 \ + ninja-build \ + pkg-config \ + python-is-python3 \ + python3 \ + python3-dev \ + python3-pip \ + sox \ + && true + +RUN echo "deb [trusted=true] ${MUSA_APT_SOURCE} jammy main" \ + > /etc/apt/sources.list.d/mthreads-musa.list \ + && apt-get update \ + && apt-get install -y --no-install-recommends \ + libmthreads-compute \ + libmthreads-mtml \ + libmublas-5-2 \ + libmublaslt-5-2 \ + libmudnn3-dev-musa-5-2 \ + libmudnn3-musa-5-2 \ + libmufft-5-2 \ + libmupp-5-2 \ + libmurand-5-2 \ + libmusolver-5-2 \ + libmusparse-5-2 \ + mccl-s5000 \ + mccl-s5000-dev \ + mtcc-5-2 \ + musa-mualg-5-2 \ + musa-mupti-5-2 \ + musa-musart-5-2 \ + musa-muthrust-5-2 \ + musa-toolkit-5-2 \ + musa-toolkit-5-2-config-common \ + && printf '%s\n' \ + "${MUSA_HOME}/lib" \ + "${MUSA_HOME}/mudnn/lib" \ + "/usr/local/mtshmem/lib" \ + "/usr/lib/x86_64-linux-gnu" \ + > /etc/ld.so.conf.d/musa-runtime.conf \ + && ln -sf /usr/lib/x86_64-linux-gnu/libmkl_intel_lp64.so /usr/lib/x86_64-linux-gnu/libmkl_intel_lp64.so.2 \ + && ln -sf /usr/lib/x86_64-linux-gnu/libmkl_core.so /usr/lib/x86_64-linux-gnu/libmkl_core.so.2 \ + && ln -sf /usr/lib/x86_64-linux-gnu/libmkl_gnu_thread.so /usr/lib/x86_64-linux-gnu/libmkl_gnu_thread.so.2 \ + && ldconfig + +RUN python -m pip install --upgrade pip "setuptools<82" wheel + +WORKDIR /workspace/sglang +COPY . . + +# Pip does not prioritize --index-url over --extra-index-url for equal-version +# candidates. Reinstall Triton from the MUSA index in this same layer. +RUN cp python/pyproject_other.toml python/pyproject.toml \ + && python -m pip install -e "python[all_musa]" \ + --index-url "${MUSA_PIP_INDEX_URL}" \ + --extra-index-url "${PYPI_INDEX_URL}" \ + --trusted-host dl.mthreads.com \ + --no-build-isolation \ + && python -m pip install --no-cache-dir --force-reinstall --no-deps \ + --index-url "${MUSA_PIP_INDEX_URL}" \ + --trusted-host dl.mthreads.com \ + "triton==3.2.0" \ + && python -c "import triton.backends.mtgpu" \ + && ! python -m pip freeze | grep -E '^(nvidia-|cuda-)' + +RUN cd python/sglang/kernels/aot \ + && cp pyproject_musa.toml pyproject.toml \ + && MTGPU_TARGET=mp_31 python setup_musa.py install + +# Keep this check in a single shell command: the legacy Docker builder does not +# reliably preserve Dockerfile heredocs and can turn the check into a silent +# `python -` EOF success. +RUN python -c \ + "import torch; \ +assert getattr(torch.version, 'musa', None), torch.__version__; \ +assert hasattr(torch, 'musa'); \ +import torchada, triton, tilelang, sglang; \ +import triton.backends.mtgpu" + +CMD ["/bin/bash"] diff --git a/docker/npu.Dockerfile b/docker/npu.Dockerfile index 39ebaf0425e7..3a5f71fd3d6f 100644 --- a/docker/npu.Dockerfile +++ b/docker/npu.Dockerfile @@ -1,37 +1,44 @@ -ARG CANN_VERSION=9.0.0 -ARG DEVICE_TYPE=a3 +ARG CANN_VERSION=9.1.0 +ARG DEVICE_TYPE=950 ARG OS=ubuntu22.04 -ARG PYTHON_VERSION=py3.11 +ARG PYTHON_VERSION=py3.12 +ARG arch FROM quay.io/ascend/cann:$CANN_VERSION-$DEVICE_TYPE-$OS-$PYTHON_VERSION -# Update pip & apt sources ARG TARGETARCH ARG CANN_VERSION ARG DEVICE_TYPE +ARG arch ARG PIP_INDEX_URL="https://pypi.org/simple/" ARG APTMIRROR="" +# torch_npu 2.10.0.post6 requires torch==2.10.0, so PYTORCH_VERSION stays at 2.10.0 ARG PYTORCH_VERSION="2.10.0" ARG TORCHVISION_VERSION="0.25.0" ARG TORCHAUDIO_VERSION="2.10.0" -ARG PTA_URL_ARM64="https://gitcode.com/Ascend/pytorch/releases/download/v26.0.0-pytorch2.10.0/torch_npu-2.10.0-cp311-cp311-manylinux_2_28_aarch64.whl" -ARG PTA_URL_AMD64="https://gitcode.com/Ascend/pytorch/releases/download/v26.0.0-pytorch2.10.0/torch_npu-2.10.0-cp311-cp311-manylinux_2_28_x86_64.whl" +ARG TORCH_NPU_VERSION="2.10.0.post6" +ARG TORCH_NPU_INDEX_URL="https://ascend.devcloud.huaweicloud.com/pypi/simple/" ARG SGLANG_TAG=main ARG ASCEND_CANN_PATH=/usr/local/Ascend/ascend-toolkit -ARG SGLANG_KERNEL_NPU_TAG=main - +ARG SGLANG_KERNEL_NPU_TAG=2026.9.0.post5 ARG PIP_INSTALL="python3 -m pip install --no-cache-dir" ARG DEVICE_TYPE +ARG MODELSCOPE_VERSION="" +ARG EVALSCOPE_VERSION="" + +ARG MF_VERSION="1.2.1" +ARG MF_WHEEL_URL_AARCH64="https://obs-memfabric-hybrid.obs.cn-north-4.myhuaweicloud.com/mf/v1.2.1/20260923.4/memfabric_hybrid-1.2.1-cp312-cp312-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl" +ARG MF_WHEEL_URL_X86_64="https://obs-memfabric-hybrid.obs.cn-north-4.myhuaweicloud.com/mf/v1.2.1/20260923.4/memfabric_hybrid-1.2.1-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl" +ARG MC_WHEEL_URL_AARCH64="https://obs-memfabric-hybrid.obs.cn-north-4.myhuaweicloud.com/memcache/v1.2.1/20260923.4/memcache_hybrid-1.2.1-cp312-cp312-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl" +ARG MC_WHEEL_URL_X86_64="https://obs-memfabric-hybrid.obs.cn-north-4.myhuaweicloud.com/memcache/v1.2.1/20260923.4/memcache_hybrid-1.2.1-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl" + +# memfabric-zbal: 950 与 a3 使用不同版本 +ARG ZBAL_VERSION_950="1.2.21004.post1" +ARG ZBAL_VERSION_A3="1.1.3" -RUN if [ "$TARGETARCH" = "amd64" ]; then \ - echo "Using x86_64 dependencies"; \ - echo "PTA_URL=$PTA_URL_AMD64" >> /etc/environment_new; \ - elif [ "$TARGETARCH" = "arm64" ]; then \ - echo "Using aarch64 dependencies"; \ - echo "PTA_URL=$PTA_URL_ARM64" >> /etc/environment_new; \ - else \ - echo "Unsupported TARGETARCH: $TARGETARCH"; exit 1; \ - fi + +# Later RUN steps source /etc/environment_new, so make sure it exists +RUN touch /etc/environment_new WORKDIR /workspace @@ -55,7 +62,6 @@ RUN apt-get update -y && apt upgrade -y && apt-get install -y \ clang \ locales \ ccache \ - ffmpeg \ openssl \ libssl-dev \ pkg-config \ @@ -71,13 +77,30 @@ ENV LANG=en_US.UTF-8 ENV LANGUAGE=en_US:en ENV LC_ALL=en_US.UTF-8 - -### Install MemFabric -RUN ${PIP_INSTALL} memfabric-hybrid==1.0.8 - -### Install zbal -RUN ${PIP_INSTALL} memfabric-zbal==1.2.0 - +### Install MemFabric and MemCache (按 TARGETARCH 直接取对应架构的 OBS wheel) +RUN set -eux; \ + case "$TARGETARCH" in \ + arm64) \ + MF_URL="$MF_WHEEL_URL_AARCH64"; \ + MC_URL="$MC_WHEEL_URL_AARCH64"; \ + ;; \ + amd64) \ + MF_URL="$MF_WHEEL_URL_X86_64"; \ + MC_URL="$MC_WHEEL_URL_X86_64"; \ + ;; \ + *) \ + echo "Unsupported architecture: $TARGETARCH" >&2; \ + exit 1; \ + ;; \ + esac; \ + ${PIP_INSTALL} "$MF_URL" --force-reinstall; \ + mfcli kernel install; \ + ${PIP_INSTALL} "$MC_URL" --force-reinstall --no-deps + +### Install memfabric-zbal +RUN if [ "$DEVICE_TYPE" = "950" ]; then ZBAL_PKG="memfabric-zbal==${ZBAL_VERSION_950}"; \ + else ZBAL_PKG="memfabric-zbal==${ZBAL_VERSION_A3}"; fi; \ + ${PIP_INSTALL} "$ZBAL_PKG" -i https://pypi.org/simple/ ### Install SGLang Model Gateway RUN ${PIP_INSTALL} sglang-router @@ -85,20 +108,46 @@ RUN ${PIP_INSTALL} sglang-router ### Install PyTorch and PTA RUN . /etc/environment_new && \ (${PIP_INSTALL} torch==${PYTORCH_VERSION} torchvision==${TORCHVISION_VERSION} torchaudio==${TORCHAUDIO_VERSION} --index-url https://download.pytorch.org/whl/cpu) \ - && (${PIP_INSTALL} ${PTA_URL}) + && (${PIP_INSTALL} torch-npu==${TORCH_NPU_VERSION} --extra-index-url ${TORCH_NPU_INDEX_URL}) + + +### Install ModelScope & EvalScope +# Installed right after torch/torch-npu so their dependencies resolve against the pinned torch. +# MODELSCOPE_VERSION / EVALSCOPE_VERSION are empty by default -> latest release. +RUN . /etc/environment_new && \ + MS_PKG="modelscope" && \ + ES_PKG="evalscope" && \ + if [ -n "${MODELSCOPE_VERSION}" ]; then MS_PKG="modelscope==${MODELSCOPE_VERSION}"; fi && \ + if [ -n "${EVALSCOPE_VERSION}" ]; then ES_PKG="evalscope==${EVALSCOPE_VERSION}"; fi && \ + ${PIP_INSTALL} "${MS_PKG}" "${ES_PKG}" ## Install triton-ascend -RUN (${PIP_INSTALL} pybind11) && \ - (${PIP_INSTALL} triton-ascend==3.2.1.dev20260530 --extra-index-url=https://mirrors.huaweicloud.com/ascend/repos/pypi/nightly --trusted-host triton-ascend.osinfra.cn) +RUN . /etc/environment_new && \ + ${PIP_INSTALL} pybind11 && \ + if [ "$TARGETARCH" = "arm64" ]; then \ + ${PIP_INSTALL} https://sglang-ascend.obs.cn-east-3.myhuaweicloud.com/ta/triton_ascend-3.2.2-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl; \ + elif [ "$TARGETARCH" = "amd64" ]; then \ + ${PIP_INSTALL} https://github.com/triton-lang/triton-ascend/releases/download/v3.2.2/triton_ascend-3.2.2-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl; \ + else \ + echo "Unsupported architecture: $TARGETARCH"; \ + exit 1; \ + fi -# Install SGLang (editable mode to preserve source and git history) -RUN git clone https://github.com/sgl-project/sglang --branch $SGLANG_TAG /sgl-workspace/sglang && \ +# Install SGLang +RUN git clone https://github.com/sgl-project/sglang --branch ${SGLANG_TAG} /sgl-workspace/sglang && \ cd /sgl-workspace/sglang/python && rm -rf pyproject.toml && mv pyproject_npu.toml pyproject.toml && \ + sed -i '/"memfabric-hybrid==1.1.4"/d; /"memfabric-zbal==1.1.2"/d' pyproject.toml && \ ${PIP_INSTALL} -v -e .[all_npu] +ENV ASCEND_HOME_PATH=/usr/local/Ascend/cann-${CANN_VERSION} + +ENV LD_LIBRARY_PATH=/usr/local/Ascend/cann-${CANN_VERSION}/lib64:/usr/local/Ascend/cann-${CANN_VERSION}/lib:/usr/local/Ascend/cann-${CANN_VERSION}/x86_64-linux/devlib/device:/usr/local/Ascend/driver/lib64:/usr/local/lib:${LD_LIBRARY_PATH} + + RUN mkdir cann-custom-ops && \ cd cann-custom-ops && \ + source /usr/local/Ascend/cann-${CANN_VERSION}/set_env.sh && \ wget https://github.com/sgl-project/sgl-kernel-npu/releases/download/${SGLANG_KERNEL_NPU_TAG}/custom-ops-${SGLANG_KERNEL_NPU_TAG}-torch2.10.0-cann${CANN_VERSION}-${DEVICE_TYPE}-$(arch).zip && \ wget https://github.com/sgl-project/sgl-kernel-npu/releases/download/${SGLANG_KERNEL_NPU_TAG}/ops-transformer-${SGLANG_KERNEL_NPU_TAG}-torch2.10.0-cann${CANN_VERSION}-${DEVICE_TYPE}-$(arch).zip && \ unzip custom-ops-${SGLANG_KERNEL_NPU_TAG}-torch2.10.0-cann${CANN_VERSION}-${DEVICE_TYPE}-$(arch).zip && \ @@ -106,7 +155,11 @@ RUN mkdir cann-custom-ops && \ chmod +x *.run && \ ./CANN-custom_ops-none-linux.$(arch).run --install-path=/usr/local/Ascend/cann-${CANN_VERSION}/opp && \ ./cann-ops-transformer-custom_linux-$(arch).run --install-path=/usr/local/Ascend/cann-${CANN_VERSION}/opp && \ - ${PIP_INSTALL} custom_ops-1.0-cp311-cp311-linux_$(arch).whl && \ + source /usr/local/Ascend/cann-${CANN_VERSION}/opp/vendors/customize/bin/set_env.bash && \ + source /usr/local/Ascend/cann-${CANN_VERSION}/opp/vendors/custom_transformer/bin/set_env.bash && \ + source /usr/local/Ascend/ascend-toolkit/latest/set_env.sh && \ + source /usr/local/Ascend/nnal/atb/set_env.sh && \ + ${PIP_INSTALL} custom_ops-1.0-cp312-cp312-linux_$(arch).whl && \ cd .. && rm -rf cann-custom-ops # Install Deep-ep @@ -114,9 +167,9 @@ RUN mkdir cann-custom-ops && \ RUN ${PIP_INSTALL} wheel==0.45.1 pybind11 pyyaml decorator scipy attrs psutil \ && mkdir sgl-kernel-npu \ && cd sgl-kernel-npu \ - && wget https://github.com/sgl-project/sgl-kernel-npu/releases/download/${SGLANG_KERNEL_NPU_TAG}/sgl-kernel-npu-${SGLANG_KERNEL_NPU_TAG}-torch2.10.0-py311-cann${CANN_VERSION}-${DEVICE_TYPE}-$(arch).zip \ - && unzip sgl-kernel-npu-${SGLANG_KERNEL_NPU_TAG}-torch2.10.0-py311-cann${CANN_VERSION}-${DEVICE_TYPE}-$(arch).zip \ - && ${PIP_INSTALL} deep_ep*.whl sgl_kernel_npu*.whl attentions*.whl \ + && wget https://github.com/sgl-project/sgl-kernel-npu/releases/download/${SGLANG_KERNEL_NPU_TAG}/sgl-kernel-npu-${SGLANG_KERNEL_NPU_TAG}-torch2.10.0-py312-cann${CANN_VERSION}-${DEVICE_TYPE}-$(arch).zip \ + && unzip sgl-kernel-npu-${SGLANG_KERNEL_NPU_TAG}-torch2.10.0-py312-cann${CANN_VERSION}-${DEVICE_TYPE}-$(arch).zip \ + && ${PIP_INSTALL} deep_ep*.whl sgl_kernel_npu*.whl torch_memory_saver*.whl \ && cd .. && rm -rf sgl-kernel-npu \ && cd "$(python3 -m pip show deep-ep | awk '/^Location:/ {print $2}')" && ln -sf deep_ep/deep_ep_cpp*.so diff --git a/docker/patches/sgl-kernel-gfx1151.sh b/docker/patches/sgl-kernel-gfx1151.sh new file mode 100755 index 000000000000..72cb73d9035e --- /dev/null +++ b/docker/patches/sgl-kernel-gfx1151.sh @@ -0,0 +1,90 @@ +#!/bin/sh +# Teach python/sglang/kernels/aot/setup_rocm.py to build for gfx1151 (Strix Halo). +# Applied at image-build time by docker/rocm-gfx1151.Dockerfile; the repo files are +# left untouched because gfx1151 is not a supported SGLang target. +# +# Two changes: +# 1. Lift the {gfx942, gfx950, gfx1250} allowlist, which otherwise sys.exit(1)s. +# Current main gives non-gfx942 targets a 40KB TopK dynamic-LDS budget, +# which already fits gfx1151's 64KB workgroup limit. +# 2. Force a single WARP_SIZE across the host and device compiler passes. +# include/utils.h resolves WARP_SIZE to 64 whenever __HIP_DEVICE_COMPILE__ +# is undefined -- i.e. on the host pass -- and to 32 on a non-__GFX9__ +# device pass. On CDNA both come out 64 and nothing is wrong, which is why +# upstream never sees this. On gfx1151 the two passes disagree, and the MoE +# TopK kernels use WARP_SIZE on both sides of the launch: +# moe_topk_softmax_kernels.cu __launch_bounds__(WARPS_PER_CTA * WARP_SIZE) -> device, 4*32 = 128 +# moe_topk_softmax_kernels.cu dim3 block_dim(WARP_SIZE, WARPS_PER_TB) -> host, 64*4 = 256 +# Launching 256 threads into a 128-thread bound fails with +# hipErrorLaunchFailure, poisons the queue, and typically surfaces as a +# page fault in whatever kernel runs next (moe_align_block_size_kernel), +# which makes it easy to misattribute. The same pattern is in +# moe_topk_sigmoid_kernels.cu. It also desynchronizes the launcher's +# TopkConstants math (ROWS_PER_WARP, VECs_PER_THREAD) from the kernel's. +# Only MoE models hit this; dense models never call these kernels. +# 32 is simply correct here -- gfx1151 is a wave32 part -- so the override +# pins both passes to 32 rather than renaming the symbol per call site. +# +# Each edit is guarded: if the upstream line has changed, fail rather than +# silently produce an image whose kernels were built with the wrong limits. +set -e + +FILE="${1:?usage: sgl-kernel-gfx1151.sh }" +UTILS="$(dirname "${FILE}")/include/utils.h" + +GATE_OLD='if amdgpu_target not in ["gfx942", "gfx950", "gfx1250"]:' +GATE_NEW='if amdgpu_target not in ["gfx942", "gfx950", "gfx1250", "gfx1151"]:' + +FLAGS_OLD=' f"-DSGL_TOPK_DYNAMIC_SMEM_BYTES={topk_dynamic_smem_bytes}",' +FLAGS_NEW=' f"-DSGL_TOPK_DYNAMIC_SMEM_BYTES={topk_dynamic_smem_bytes}", + # gfx1151 is wave32; pin both compiler passes to it (see utils.h below). + *(["-DSGL_ROCM_WARP_SIZE=32"] if amdgpu_target == "gfx1151" else []),' + +WARP_OLD='#if defined(__GFX9__) || !defined(__HIP_DEVICE_COMPILE__) +#define WARP_SIZE 64' +WARP_NEW='#if defined(SGL_ROCM_WARP_SIZE) +#define WARP_SIZE SGL_ROCM_WARP_SIZE +#elif defined(__GFX9__) || !defined(__HIP_DEVICE_COMPILE__) +#define WARP_SIZE 64' + +for pattern in "${GATE_OLD}" "${FLAGS_OLD}"; do + if ! grep -qF "${pattern}" "${FILE}"; then + echo "ERROR: expected line not found in ${FILE}:" >&2 + echo " ${pattern}" >&2 + echo "setup_rocm.py changed upstream; re-check this patch before building." >&2 + exit 1 + fi +done + +if ! grep -qF "${WARP_OLD}" "${UTILS}"; then + echo "ERROR: expected WARP_SIZE block not found in ${UTILS}." >&2 + echo "utils.h changed upstream; re-check the wave32 fix before building." >&2 + exit 1 +fi + +python3 - "${UTILS}" "${WARP_OLD}" "${WARP_NEW}" <<'PY' +import sys + +path, old, new = sys.argv[1:4] +with open(path) as f: + src = f.read() +with open(path, "w") as f: + f.write(src.replace(old, new, 1)) +PY + +python3 - "${FILE}" "${GATE_OLD}" "${GATE_NEW}" "${FLAGS_OLD}" "${FLAGS_NEW}" <<'PY' +import sys + +path, gate_old, gate_new, flags_old, flags_new = sys.argv[1:6] +with open(path) as f: + src = f.read() +src = src.replace(gate_old, gate_new) +src = src.replace(flags_old, flags_new, 1) +with open(path, "w") as f: + f.write(src) +PY + +echo "Patched ${FILE} for gfx1151:" +grep -nF -e "${GATE_NEW}" -e "SGL_ROCM_WARP_SIZE" "${FILE}" +echo "Patched ${UTILS} for wave32:" +grep -nF "SGL_ROCM_WARP_SIZE" "${UTILS}" diff --git a/docker/renderer.Dockerfile b/docker/renderer.Dockerfile new file mode 100644 index 000000000000..804095194f3b --- /dev/null +++ b/docker/renderer.Dockerfile @@ -0,0 +1,89 @@ +# syntax=docker/dockerfile:1 + +# Keep the compiler aligned with rust/rust-toolchain.toml. Pin image indexes +# rather than individual architecture manifests so both platforms use this file. +FROM rust:1.92.0-slim-bookworm@sha256:f1f73538ebe623fd3673a35aff3df358ae1084c64c55646516e5b17b321b6c9b AS build + +ARG TARGETARCH +ARG CARGO_BUILD_JOBS=4 +ENV RUSTUP_TOOLCHAIN=1.92.0 \ + CARGO_BUILD_JOBS=${CARGO_BUILD_JOBS} \ + PCRE2_SYS_STATIC=1 + +RUN apt-get update \ + && apt-get install -y --no-install-recommends build-essential pkg-config \ + && rm -rf /var/lib/apt/lists/* + +WORKDIR /build +COPY rust/Cargo.toml rust/Cargo.lock rust/rust-toolchain.toml rust/ +# Cargo loads every workspace member even when building only the renderer. +COPY rust/sglang-grpc/Cargo.toml rust/sglang-grpc/ +COPY rust/sglang-grpc/src/ rust/sglang-grpc/src/ +COPY rust/sglang-mm/Cargo.toml rust/sglang-mm/ +COPY rust/sglang-mm/src/ rust/sglang-mm/src/ +COPY rust/sglang-server/Cargo.toml rust/sglang-server/ +COPY rust/sglang-server/src/ rust/sglang-server/src/ +COPY rust/sglang-renderer/Cargo.toml rust/sglang-renderer/ +COPY rust/sglang-renderer/src/ rust/sglang-renderer/src/ + +# Avoid rustup downloading development components from rust-toolchain.toml, +# but fail if the image's compiler and the workspace toolchain drift apart. +RUN channel=$(sed -n 's/^channel = "\([^"]*\)"/\1/p' rust/rust-toolchain.toml) \ + && case "${RUSTUP_TOOLCHAIN}" in "$channel"|"$channel".*) ;; *) exit 1 ;; esac + +RUN --mount=type=cache,id=renderer-registry-${TARGETARCH},target=/usr/local/cargo/registry,sharing=locked \ + --mount=type=cache,id=renderer-git-${TARGETARCH},target=/usr/local/cargo/git,sharing=locked \ + --mount=type=cache,id=renderer-target-${TARGETARCH},target=/build/rust/target,sharing=locked \ + cargo build --manifest-path rust/Cargo.toml -p sglang-renderer \ + --bin sglang-renderer --release --features http --locked \ + && install -D rust/target/release/sglang-renderer /out/sglang-renderer + +# Run the existing unit suite in the same Linux toolchain used for the image. +# This sibling stage is selected by CI and is not a dependency of the runtime. +FROM build AS test +COPY rust/sglang-renderer/tests/ rust/sglang-renderer/tests/ +COPY experimental/sgl-router/tests/fixtures/tiny_tokenizer.json experimental/sgl-router/tests/fixtures/tiny_tokenizer.json +RUN --mount=type=cache,id=renderer-registry-${TARGETARCH},target=/usr/local/cargo/registry,sharing=locked \ + --mount=type=cache,id=renderer-git-${TARGETARCH},target=/usr/local/cargo/git,sharing=locked \ + --mount=type=cache,id=renderer-target-${TARGETARCH},target=/build/rust/target,sharing=locked \ + cargo test --manifest-path rust/Cargo.toml -p sglang-renderer --features http --locked + +FROM debian:bookworm-slim@sha256:88200866dfff7ea7f5cbcb6ec7c8a701889efe6fe859fe64d6990e4b07ea4171 AS runtime + +RUN apt-get update \ + && apt-get install -y --no-install-recommends ca-certificates libgcc-s1 \ + && rm -rf /var/lib/apt/lists/* \ + && groupadd --gid 65532 sglang \ + && useradd --uid 65532 --gid 65532 --no-log-init --create-home \ + --home-dir /home/sglang --shell /usr/sbin/nologin sglang \ + && mkdir -p /home/sglang/.cache/huggingface \ + && chown -R 65532:65532 /home/sglang + +COPY --from=build /out/sglang-renderer /usr/local/bin/sglang-renderer +COPY LICENSE /usr/share/licenses/sglang-renderer/LICENSE + +# Metadata changes must not invalidate compilation. +ARG SGLANG_BUILD_COMMIT=unknown +ARG SGLANG_BUILD_URL= +ARG SGLANG_IMAGE_TAG=local/sglang-renderer:dev +ENV HOME=/home/sglang \ + HF_HOME=/home/sglang/.cache/huggingface \ + SGLANG_BUILD_COMMIT=${SGLANG_BUILD_COMMIT} \ + SGLANG_BUILD_URL=${SGLANG_BUILD_URL} \ + SGLANG_IMAGE_TAG=${SGLANG_IMAGE_TAG} +LABEL org.opencontainers.image.source="https://github.com/sgl-project/sglang" \ + org.opencontainers.image.licenses="Apache-2.0" \ + org.opencontainers.image.revision="${SGLANG_BUILD_COMMIT}" \ + org.opencontainers.image.version="${SGLANG_IMAGE_TAG}" \ + org.opencontainers.image.url="${SGLANG_BUILD_URL}" \ + ai.sglang.build.commit="${SGLANG_BUILD_COMMIT}" \ + ai.sglang.build.url="${SGLANG_BUILD_URL}" \ + ai.sglang.image.tag="${SGLANG_IMAGE_TAG}" + +USER 65532:65532 +WORKDIR /home/sglang +EXPOSE 30000 +# The renderer's existing graceful shutdown handler listens for Ctrl-C. +STOPSIGNAL SIGINT +ENTRYPOINT ["/usr/local/bin/sglang-renderer"] +CMD ["--help"] diff --git a/docker/rocm-gfx1151.Dockerfile b/docker/rocm-gfx1151.Dockerfile new file mode 100644 index 000000000000..cc80e785be90 --- /dev/null +++ b/docker/rocm-gfx1151.Dockerfile @@ -0,0 +1,167 @@ +# SGLang for AMD Strix Halo / Ryzen AI MAX+ (gfx1151, RDNA3.5 iGPU). +# +# This is NOT a variant of docker/rocm.Dockerfile. That file targets CDNA +# (gfx942/gfx950) and includes components which do not support gfx1151. This +# image starts from AMD's stable ROCm/PyTorch image with native gfx1151 support. +# +# Build: +# docker build -f docker/rocm-gfx1151.Dockerfile -t sglang-rocm:gfx1151 . +# +# Run (Strix Halo has no discrete VRAM; the GPU carves out of system RAM): +# docker run -it --rm \ +# --device=/dev/kfd --device=/dev/dri \ +# --group-add video --group-add render \ +# --security-opt seccomp=unconfined \ +# --ipc=host --shm-size 16g \ +# -p 30000:30000 \ +# -v ~/.cache/huggingface:/root/.cache/huggingface \ +# sglang-rocm:gfx1151 \ +# python3 -m sglang.launch_server --model-path \ +# --attention-backend triton --host 0.0.0.0 + +# ROCm 7.2.4 / PyTorch 2.9.1 is AMD's stable gfx1151-supported combination. +# Pin the image digest so rebuilding cannot silently change the toolchain. +ARG BASE_IMAGE="rocm/pytorch@sha256:7fe531fa185af260352fe7fbb3fa64ad749abe72adf0600a648c4692801b125a" + +# ============================================================================= +# Stage 1: stable ROCm + PyTorch for gfx1151. +# Pullable and testable on its own: +# docker build --target rocm-torch -f docker/rocm-gfx1151.Dockerfile -t rocm-torch:gfx1151 . +# ============================================================================= +FROM ${BASE_IMAGE} AS rocm-torch + +ARG GPU_ARCH=gfx1151 + +ENV DEBIAN_FRONTEND=noninteractive +ENV PYTHONUNBUFFERED=1 + +RUN apt-get update && apt-get install -y --no-install-recommends \ + cmake \ + libnuma-dev \ + && rm -rf /var/lib/apt/lists/* + +ENV PYTORCH_ROCM_ARCH=${GPU_ARCH} +# ROCDXG requires this under WSL. It is inert when /dev/dxg is absent. +ENV HSA_ENABLE_DXG_DETECTION=1 + +# Fail loudly if the base image or its expected development toolchain changes. +RUN python3 -c "import torch; print('torch', torch.__version__); assert torch.version.hip" \ + && test -x /opt/rocm/bin/hipcc + +# ============================================================================= +# Stage 2: SGLang on top of the gfx1151 ROCm stack. +# ============================================================================= +FROM rocm-torch AS sglang + +ARG GPU_ARCH=gfx1151 +# sgl-kernel's ROCm build (python/sglang/kernels/aot/setup_rocm.py) only accepts +# gfx942/gfx950/gfx1250 and hard-exits on anything else; the patch below lifts +# that gate. Set to 0 to skip the AOT kernels entirely and run Triton-only. +ARG BUILD_SGL_KERNEL=1 +ARG MAX_JOBS=12 + +WORKDIR /sgl-workspace + +# setuptools-rust builds the sglang-mm extension during the pip install below. +ENV PATH="/root/.cargo/bin:${PATH}" +RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --profile minimal \ + && rustc --version +ENV CARGO_BUILD_JOBS=8 + +COPY . /sgl-workspace/sglang + +# pyproject.toml pins the CUDA stack (torch, flashinfer[cu13], cuda-python, +# ...). pyproject_other.toml carries the srt_hip extra, which is +# torch-version-agnostic -- same swap docker/rocm.Dockerfile performs. +RUN cd /sgl-workspace/sglang \ + && rm -f python/pyproject.toml \ + && mv python/pyproject_other.toml python/pyproject.toml + +# One problem in setup_rocm.py for this target, plus one in include/utils.h: +# the arch gate sys.exit(1)s outside {gfx942, gfx950, gfx1250}, and WARP_SIZE +# resolves to 64 on the host pass but 32 on the device pass for a wave32 part, +# which mismatches the MoE TopK launch bounds. Current main already limits +# non-gfx942 TopK dynamic LDS to 40KB, which fits gfx1151's 64KB limit. The +# remaining two problems are fixed here rather than upstream: +# gfx1151 is not a supported SGLang target, and the sources themselves compile +# clean for it. Each edit greps for the expected text first, so a rewrite +# upstream breaks the build loudly instead of silently misconfiguring kernels. +COPY docker/patches/sgl-kernel-gfx1151.sh /tmp/sgl-kernel-gfx1151.sh + +RUN cd /sgl-workspace/sglang/python/sglang/kernels/aot \ + && if [ "${BUILD_SGL_KERNEL}" = "1" ]; then \ + rm -f pyproject.toml \ + && mv pyproject_rocm.toml pyproject.toml \ + && sh /tmp/sgl-kernel-gfx1151.sh setup_rocm.py \ + && AMDGPU_TARGET=${GPU_ARCH} MAX_JOBS=${MAX_JOBS} python3 setup_rocm.py install; \ + else \ + echo "Skipping sgl-kernel build (BUILD_SGL_KERNEL=0)"; \ + fi + +# Current main composes extras through self-references +# (srt_hip -> sglang[runtime_common] -> sglang[runtime_base]). pip's resolver +# recursively walks that cycle from an editable source checkout. Flatten those +# three groups before installing the package itself without dependency solving. +# Keep compressed-tensors at its last torch-2.9-compatible release. +RUN cd /sgl-workspace/sglang \ + && python3 - <<'PY' +import subprocess +import sys +import tomllib +from pathlib import Path + +project = tomllib.loads(Path("python/pyproject.toml").read_text())["project"] +extras = project["optional-dependencies"] +requirements = list(project["dependencies"]) +for group in ("runtime_base", "runtime_common", "srt_hip", "diffusion_common"): + requirements.extend( + "compressed-tensors==0.15.0" + if requirement == "compressed-tensors" + else requirement + for requirement in extras[group] + if not requirement.startswith("sglang[") and requirement != "torch" + ) +# diffusion_hip's pure-Python packages, needed by sglang.multimodal_gen +# (Wan2.2/video-gen) at import time. st_attn/vsa are deliberately left out: +# they're prebuilt CUDA-oriented wheels of unverified gfx1151 compatibility, +# same posture as skipping aiter's compiled CDNA kernels below. +requirements.extend( + requirement + for requirement in extras["diffusion_hip"] + if not requirement.startswith("sglang[") + and not requirement.startswith("st_attn") + and not requirement.startswith("vsa") +) +requirements = list(dict.fromkeys(requirements)) +subprocess.check_call( + [sys.executable, "-m", "pip", "install", "--no-cache-dir", *requirements] +) +PY +RUN cd /sgl-workspace/sglang \ + && pip install --no-cache-dir --no-deps -e python + +# aiter is not optional on ROCm despite being CDNA-oriented: +# sglang/srt/layers/quantization/__init__.py imports quark, which imports +# aiter.ops.triton at module scope, so `import sglang.srt.layers.activation` +# fails outright without it. Installed WITHOUT PREBUILD_KERNELS -- that step +# AOT-compiles the CDNA assembly kernels and is what actually fails on gfx1151. +# In JIT mode aiter builds module_aiter_core for gfx1151 on demand instead. +ARG AITER_REPO="https://github.com/ROCm/aiter.git" +ARG AITER_COMMIT="c16d44b93a528b2a4bfd6d8d3409116d465872a9" + +RUN git clone --recursive ${AITER_REPO} /sgl-workspace/aiter \ + && cd /sgl-workspace/aiter \ + && git checkout ${AITER_COMMIT} \ + && git submodule update --init --recursive \ + && GPU_ARCHS=${GPU_ARCH} pip install --no-cache-dir --no-build-isolation \ + --config-settings editable_mode=compat -e . + +# aiter's compiled attention/MoE kernels are CDNA-only; keep sglang on the +# Triton paths. ServerArgs defaults to aiter on ROCm, so callers must pass +# `--attention-backend triton` until the RDNA default is fixed upstream. +# This is load-bearing beyond attention: aiter's RMSNorm uses v_pk_mul_f32, +# a CDNA-only instruction, and its CK attention templates assume wave64. +ENV SGLANG_USE_AITER=0 + +WORKDIR /sgl-workspace/sglang +CMD ["/bin/bash"] diff --git a/docker/rocm.Dockerfile b/docker/rocm.Dockerfile index a580d107751b..50cd8b0429ff 100644 --- a/docker/rocm.Dockerfile +++ b/docker/rocm.Dockerfile @@ -1,12 +1,13 @@ # Usage (to build SGLang ROCm docker image): -# docker build --build-arg SGL_BRANCH=v0.5.10.post1 --build-arg GPU_ARCH=gfx942 -t v0.5.10.post1-rocm700-mi30x -f rocm.Dockerfile . -# docker build --build-arg SGL_BRANCH=v0.5.10.post1 --build-arg GPU_ARCH=gfx942-rocm720 -t v0.5.10.post1-rocm720-mi30x -f rocm.Dockerfile . -# docker build --build-arg SGL_BRANCH=v0.5.10.post1 --build-arg GPU_ARCH=gfx942-rocm724 -t v0.5.10.post1-rocm724-mi30x -f rocm.Dockerfile . -# docker build --build-arg SGL_BRANCH=v0.5.10.post1 --build-arg GPU_ARCH=gfx950 -t v0.5.10.post1-rocm700-mi35x -f rocm.Dockerfile . -# docker build --build-arg SGL_BRANCH=v0.5.10.post1 --build-arg GPU_ARCH=gfx950-rocm720 -t v0.5.10.post1-rocm720-mi35x -f rocm.Dockerfile . -# docker build --build-arg SGL_BRANCH=v0.5.10.post1 --build-arg GPU_ARCH=gfx950-rocm724 -t v0.5.10.post1-rocm724-mi35x -f rocm.Dockerfile . -# docker build --build-arg SGL_BRANCH=v0.5.10.post1 --build-arg GPU_ARCH=gfx942-rocm1000 -t v0.5.10.post1-rocm1000-mi30x -f rocm.Dockerfile . -# docker build --build-arg SGL_BRANCH=v0.5.10.post1 --build-arg GPU_ARCH=gfx950-rocm1000 -t v0.5.10.post1-rocm1000-mi35x -f rocm.Dockerfile . +# docker build --build-arg SGL_BRANCH=v0.5.17 --build-arg GPU_ARCH=gfx942 -t v0.5.17-rocm700-mi30x -f rocm.Dockerfile . +# docker build --build-arg SGL_BRANCH=v0.5.17 --build-arg GPU_ARCH=gfx942-rocm720 -t v0.5.17-rocm720-mi30x -f rocm.Dockerfile . +# docker build --build-arg SGL_BRANCH=v0.5.17 --build-arg GPU_ARCH=gfx942-rocm724 -t v0.5.17-rocm724-mi30x -f rocm.Dockerfile . +# docker build --build-arg SGL_BRANCH=v0.5.17 --build-arg GPU_ARCH=gfx950 -t v0.5.17-rocm700-mi35x -f rocm.Dockerfile . +# docker build --build-arg SGL_BRANCH=v0.5.17 --build-arg GPU_ARCH=gfx950-rocm720 -t v0.5.17-rocm720-mi35x -f rocm.Dockerfile . +# docker build --build-arg SGL_BRANCH=v0.5.17 --build-arg GPU_ARCH=gfx950-rocm724 -t v0.5.17-rocm724-mi35x -f rocm.Dockerfile . +# docker build --build-arg SGL_BRANCH=v0.5.17 --build-arg GPU_ARCH=gfx942-rocm1000 -t v0.5.17-rocm1000-mi30x -f rocm.Dockerfile . +# docker build --build-arg SGL_BRANCH=v0.5.17 --build-arg GPU_ARCH=gfx950-rocm1000 -t v0.5.17-rocm1000-mi35x -f rocm.Dockerfile . +# docker build --build-arg SGL_BRANCH=v0.5.17 --build-arg GPU_ARCH=gfx1250-rocm1000 -t v0.5.17-rocm1000-mi45x -f rocm.Dockerfile . # # Flavor notes: # GPU_ARCH=*-rocm724 is built on a Python 3.12 base and upgrades the stack to @@ -15,22 +16,25 @@ # GPU_ARCH=*-rocm1000 is Python 3.12 + torch 2.11, and takes the # whole ROCm stack from AMD's stable wheel channel rather than an apt # ROCm base image; see the rocm1000-base stage for what that changes. +# GPU_ARCH=gfx1250-rocm1000 carries the gfx1250 bring-up onto that same GA +# wheel channel. The gfx1250 workarounds key off GPU_ARCH_LIST=gfx1250 rather +# than the flavor name. # Usage (to build SGLang ROCm + Mori docker image): # remove --build-arg NIC_BACKEND=ainic since new MoRI JIT will do NIC auto detection on target # Keep the build-arg for user to select the desired nic support, current choice: [ainic, bxnt] # if no set this arg, it will support nic auto detection. On a target with more than 1 type of # RDMA NICs installed (rare), overwrite w. runtime env MORI_DEVICE_NIC = "bnxt"|"ionic"|"mlx5" -# docker build --build-arg SGL_BRANCH=v0.5.10.post1 --build-arg GPU_ARCH=gfx942 --build-arg ENABLE_MORI=1 -t v0.5.10.post1-rocm700-mi30x -f rocm.Dockerfile . -# docker build --build-arg SGL_BRANCH=v0.5.10.post1 --build-arg GPU_ARCH=gfx942-rocm720 --build-arg ENABLE_MORI=1 -t v0.5.10.post1-rocm720-mi30x -f rocm.Dockerfile . -# docker build --build-arg SGL_BRANCH=v0.5.10.post1 --build-arg GPU_ARCH=gfx950 --build-arg ENABLE_MORI=1 -t v0.5.10.post1-rocm700-mi35x -f rocm.Dockerfile . -# docker build --build-arg SGL_BRANCH=v0.5.10.post1 --build-arg GPU_ARCH=gfx950-rocm720 --build-arg ENABLE_MORI=1 -t v0.5.10.post1-rocm720-mi35x -f rocm.Dockerfile . +# docker build --build-arg SGL_BRANCH=v0.5.17 --build-arg GPU_ARCH=gfx942 --build-arg ENABLE_MORI=1 -t v0.5.17-rocm700-mi30x -f rocm.Dockerfile . +# docker build --build-arg SGL_BRANCH=v0.5.17 --build-arg GPU_ARCH=gfx942-rocm720 --build-arg ENABLE_MORI=1 -t v0.5.17-rocm720-mi30x -f rocm.Dockerfile . +# docker build --build-arg SGL_BRANCH=v0.5.17 --build-arg GPU_ARCH=gfx950 --build-arg ENABLE_MORI=1 -t v0.5.17-rocm700-mi35x -f rocm.Dockerfile . +# docker build --build-arg SGL_BRANCH=v0.5.17 --build-arg GPU_ARCH=gfx950-rocm720 --build-arg ENABLE_MORI=1 -t v0.5.17-rocm720-mi35x -f rocm.Dockerfile . # Usage (to build SGLang ROCm + NIXL docker image, for prefill/decode disaggregation): # Builds UCX (--with-rocm) and upstream ai-dynamo/nixl from source by default. # Set ENABLE_NIXL=0 to skip NIXL. # At runtime use --disaggregation-transfer-backend nixl (env is wired via /etc/bash.bashrc). -# docker build --build-arg SGL_BRANCH=v0.5.10.post1 --build-arg GPU_ARCH=gfx950-rocm720 -t v0.5.10.post1-rocm720-mi35x -f rocm.Dockerfile . +# docker build --build-arg SGL_BRANCH=v0.5.17 --build-arg GPU_ARCH=gfx950-rocm720 -t v0.5.17-rocm720-mi35x -f rocm.Dockerfile . # Default base images ARG BASE_IMAGE_942="rocm/sgl-dev:rocm7-vllm-20250904" @@ -43,6 +47,7 @@ ARG BASE_IMAGE_950_ROCM724="rocm/pytorch:rocm7.2.4_ubuntu24.04_py3.12_pytorch_re # than a published image; point these at one to build on a prebuilt base. ARG BASE_IMAGE_942_ROCM1000="rocm1000-base" ARG BASE_IMAGE_950_ROCM1000="rocm1000-base" +ARG BASE_IMAGE_1250_ROCM1000="rocm1000-base" ARG BASE_IMAGE_ROCM1000="ubuntu:24.04" # This is necessary for scope purpose @@ -56,7 +61,7 @@ ENV BUILD_TRITON="0" ENV BUILD_LLVM="0" ENV BUILD_AITER_ALL="1" ENV BUILD_MOONCAKE="1" -ENV AITER_COMMIT_DEFAULT="c16d44b93a528b2a4bfd6d8d3409116d465872a9" +ENV AITER_COMMIT_DEFAULT="acf8fdf9307431ece8ee275971c41cb3d1a7020b" # =============================== # Base image 942 with rocm720 and args @@ -66,7 +71,8 @@ ENV BUILD_TRITON="1" ENV BUILD_LLVM="0" ENV BUILD_AITER_ALL="1" ENV BUILD_MOONCAKE="1" -ENV AITER_COMMIT_DEFAULT="c16d44b93a528b2a4bfd6d8d3409116d465872a9" +ENV AITER_COMMIT_DEFAULT="acf8fdf9307431ece8ee275971c41cb3d1a7020b" +ENV TRITON_COMMIT_DEFAULT="42270451990532c67e69d753fbd026f28fcc4840" # =============================== # Base image 942 with rocm724 and args (Python 3.12 + torch 2.11) @@ -76,7 +82,7 @@ ENV BUILD_TRITON="1" ENV BUILD_LLVM="0" ENV BUILD_AITER_ALL="1" ENV BUILD_MOONCAKE="1" -ENV AITER_COMMIT_DEFAULT="c16d44b93a528b2a4bfd6d8d3409116d465872a9" +ENV AITER_COMMIT_DEFAULT="acf8fdf9307431ece8ee275971c41cb3d1a7020b" # Pin the ROCm torch stack for every pip invocation in this flavor. The file is # filled in after the torch 2.11 upgrade below; it must already exist (empty is # valid) because pip reads PIP_CONSTRAINT from the first pip call onwards. @@ -100,7 +106,7 @@ ENV BUILD_TRITON="0" ENV BUILD_LLVM="0" ENV BUILD_AITER_ALL="1" ENV BUILD_MOONCAKE="1" -ENV AITER_COMMIT_DEFAULT="c16d44b93a528b2a4bfd6d8d3409116d465872a9" +ENV AITER_COMMIT_DEFAULT="acf8fdf9307431ece8ee275971c41cb3d1a7020b" # =============================== # Base image 950 with rocm720 and args @@ -110,7 +116,8 @@ ENV BUILD_TRITON="1" ENV BUILD_LLVM="0" ENV BUILD_AITER_ALL="1" ENV BUILD_MOONCAKE="1" -ENV AITER_COMMIT_DEFAULT="c16d44b93a528b2a4bfd6d8d3409116d465872a9" +ENV AITER_COMMIT_DEFAULT="acf8fdf9307431ece8ee275971c41cb3d1a7020b" +ENV TRITON_COMMIT_DEFAULT="42270451990532c67e69d753fbd026f28fcc4840" # =============================== # Base image 950 with rocm724 and args (Python 3.12 + torch 2.11) @@ -120,7 +127,7 @@ ENV BUILD_TRITON="1" ENV BUILD_LLVM="0" ENV BUILD_AITER_ALL="1" ENV BUILD_MOONCAKE="1" -ENV AITER_COMMIT_DEFAULT="c16d44b93a528b2a4bfd6d8d3409116d465872a9" +ENV AITER_COMMIT_DEFAULT="acf8fdf9307431ece8ee275971c41cb3d1a7020b" # Pin the ROCm torch stack for every pip invocation in this flavor. The file is # filled in after the torch 2.11 upgrade below; it must already exist (empty is # valid) because pip reads PIP_CONSTRAINT from the first pip call onwards. @@ -279,7 +286,7 @@ ENV BUILD_TRITON="0" ENV BUILD_LLVM="0" ENV BUILD_AITER_ALL="1" ENV BUILD_MOONCAKE="1" -ENV AITER_COMMIT_DEFAULT="c16d44b93a528b2a4bfd6d8d3409116d465872a9" +ENV AITER_COMMIT_DEFAULT="acf8fdf9307431ece8ee275971c41cb3d1a7020b" # Same reasoning as the rocm724 stages: keep pip from resolving the image's # ROCm torch away to a PyPI CUDA build. Populated after the stack is in place. ENV PIP_CONSTRAINT="/etc/sglang/constraints/torch-rocm.txt" @@ -293,7 +300,30 @@ ENV BUILD_TRITON="0" ENV BUILD_LLVM="0" ENV BUILD_AITER_ALL="1" ENV BUILD_MOONCAKE="1" -ENV AITER_COMMIT_DEFAULT="c16d44b93a528b2a4bfd6d8d3409116d465872a9" +ENV AITER_COMMIT_DEFAULT="acf8fdf9307431ece8ee275971c41cb3d1a7020b" +ENV PIP_CONSTRAINT="/etc/sglang/constraints/torch-rocm.txt" +RUN mkdir -p /etc/sglang/constraints && : > /etc/sglang/constraints/torch-rocm.txt + +# =============================== +# Base image 1250 with ROCm 10.0.0 GA and args (Python 3.12 + torch 2.11) +# The gfx1250 build paths are keyed on GPU_ARCH_LIST=gfx1250 rather than the +# flavor name, so they apply here unchanged. +FROM $BASE_IMAGE_1250_ROCM1000 AS gfx1250-rocm1000 +ENV BUILD_VLLM="0" +# Unlike the gfx942/gfx950 images, this one replaces the SDK's Triton: the +# revision below is what the MI45x bring-up ran on, and it carries a fix the +# SDK build does not have yet. +ENV BUILD_TRITON="1" +ENV BUILD_LLVM="0" +ENV BUILD_AITER_ALL="1" +ENV BUILD_MOONCAKE="1" +# gfx1250 tracks a different AITER than the gfx942/gfx950 baseline: this commit +# plus the four reverts applied at clone time are what the gfx1250 kernels were +# brought up against. +ENV AITER_COMMIT_DEFAULT="a6d2b564fd671724a3720b8edf70e8d674e4d694" +# The upstream Triton the gfx1250 bring-up was validated against, carried over +# from the ROCm 7.14 flavor this image replaced. Built from source below. +ENV TRITON_COMMIT_DEFAULT="76940ad348795521b3dc9f6c79acd7309ff924e3" ENV PIP_CONSTRAINT="/etc/sglang/constraints/torch-rocm.txt" RUN mkdir -p /etc/sglang/constraints && : > /etc/sglang/constraints/torch-rocm.txt @@ -308,13 +338,16 @@ FROM ${GPU_ARCH} # This is necessary for scope purpose, again ARG GPU_ARCH=gfx950 +RUN echo GPU_ARCH="${GPU_ARCH}" \ + && echo "GPU_ARCH_LIST=${GPU_ARCH%%-*}" >> /etc/environment \ + && echo "export GPU_ARCH_LIST=${GPU_ARCH%%-*}" >> /etc/bash.bashrc # ARG is build-time only. Stamp the stage name (gfx950-rocm724, gfx942, ...) # so CI can read which AITER_COMMIT_DEFAULT block to use instead of guessing # from torch or HIP — 720 may also ship torch 2.11 later, and both 7.2 flavors # report HIP 7.2*. ENV GPU_ARCH=${GPU_ARCH} ENV GPU_ARCH_LIST=${GPU_ARCH%-*} -ENV PYTORCH_ROCM_ARCH=gfx942;gfx950 +ENV PYTORCH_ROCM_ARCH="gfx942;gfx950;gfx1250" ARG SGL_REPO="https://github.com/sgl-project/sglang.git" ARG SGL_DEFAULT="main" @@ -324,6 +357,9 @@ ARG BRANCH_TYPE=remote # Version override for setuptools_scm (used in nightly builds) ARG SETUPTOOLS_SCM_PRETEND_VERSION="" +ARG TRITON_REPO="https://github.com/triton-lang/triton.git" +ENV TRITON_COMMIT="${TRITON_COMMIT:-${TRITON_COMMIT_DEFAULT}}" + # ROCm 7.2 Triton (BUILD_TRITON=1 stages only). Both wheels are the same # upstream revision, triton-lang/triton@89002410. AITER only requires # triton>=3.6.0 and treats the base image as the owner of the version, so the @@ -362,7 +398,7 @@ ARG ENABLE_MORI=0 ARG NIC_BACKEND=none ARG MORI_REPO="https://github.com/ROCm/mori.git" -ARG MORI_COMMIT="12d1bc32d0c93dcd5062e74f4e0f772e36e1aac4" +ARG MORI_COMMIT="879983bdbd8c65c52e9f79ad836a61cbffef98b6" # NIXL (upstream ai-dynamo/nixl) — KV transfer backend for prefill/decode disaggregation. # Built from source for ROCm; needs UCX built --with-rocm (built here from openucx). @@ -519,9 +555,8 @@ RUN if [ "$BUILD_LLVM" = "1" ]; then \ # ----------------------- # AITER -# Unset setuptools_scm override so AITER gets its own version (AITER_COMMIT), not SGLang's -# (SETUPTOOLS_SCM_PRETEND_VERSION is set later for SGLang nightly builds and would otherwise -# leak into AITER's version when AITER uses setuptools_scm) +# Clear any inherited setuptools_scm override so AITER gets its own version +# from AITER_COMMIT rather than SGLang's nightly version. ENV SETUPTOOLS_SCM_PRETEND_VERSION= # Compile AITER against the base image's Triton; the Triton step at the end of @@ -533,16 +568,56 @@ RUN pip uninstall -y aiter # block AITER_COMMIT overrides that predate that rule. The working tree was just # produced by a fresh `git clone` above, so there are no real user changes to # preserve. +# cherry pick ROCm/aiter#5283 and #5279 gfx950 dsv4 a8w8 blockscale bpreshuffle configs +# apply fix for v4 fp4 indexer, may be removed in next aiter upgrade RUN git clone ${AITER_REPO} \ && cd aiter \ && git checkout -f ${AITER_COMMIT} \ + && git cherry-pick --no-commit 7b481fbcaf834ce98b66004ea329c2ca2bba87d7 \ + && git cherry-pick --no-commit 24a62b1c122f23645a19b9d8b0abd4750c59359b \ + && sed -i 's/from functools import lru_cache/from functools import cache/' aiter/ops/flydsl/kernels/mqa_logits/pa_mqa_logits_fp4_prefill.py \ + && sed -i 's/@lru_cache(maxsize=32)/@cache/' aiter/ops/flydsl/kernels/mqa_logits/pa_mqa_logits_fp4_prefill.py \ && git submodule update --init --recursive \ - && pip install -r requirements.txt + && pip install -r requirements.txt \ + && if [ "${GPU_ARCH_LIST}" = "gfx1250" ]; then \ + git revert --no-edit --no-commit 8d581370b; \ + git revert --no-edit --no-commit 79c11b677; \ + git revert --no-edit --no-commit 1ecb760a5; \ + git revert --no-edit --no-commit e708f6c15; \ + fi + +# ROCm/aiter#5195's .co with a mask assembled in between the q rows sharing a +# tile, which the grouped dsv4 decode needs; bit-identical at max_seqlen_q=1. +COPY --from=local_src \ + /src/python/sglang/kernels/ops/attention/dsv4/asm/gfx950/mla_v4/mla_a8w8_qh64_qseqlen1_gqaratio64_nm.co \ + /sgl-workspace/aiter/hsa/gfx950/mla_v4/ + +# The pinned AITER revision uses std::optional in topk_per_row_kernels.cu without +# including : ROCm/aiter#4702 removed the torch headers that previously +# supplied it transitively. ROCm 7.0 therefore fails module_top_k_per_row, while +# ROCm 7.2 toolchains still obtain from another header. +# GPU_ARCH keeps the full selected stage name; only unsuffixed gfx942/gfx950 +# denote ROCm 7.0, while newer flavors carry a -rocm... suffix. Remove this +# backport once AITER_COMMIT contains the upstream fix from ROCm/aiter#4853. +RUN python3 <<'PY' +import os +from pathlib import Path + +p = Path("/sgl-workspace/aiter/csrc/kernels/topk_per_row_kernels.cu") +anchor = "#include \n" +if os.environ["GPU_ARCH"] in {"gfx942", "gfx950"} and p.exists(): + s = p.read_text() + if "std::optional" in s and "#include " not in s and anchor in s: + p.write_text(s.replace(anchor, "#include \n" + anchor, 1)) +PY RUN cd aiter \ && echo "[AITER] GPU_ARCH=${GPU_ARCH}" \ && echo "[AITER] AITER_USE_SYSTEM_TRITON=${AITER_USE_SYSTEM_TRITON}" \ - && if [ "$BUILD_AITER_ALL" = "1" ] && [ "$BUILD_LLVM" = "1" ]; then \ + && if [ "${GPU_ARCH_LIST}" = "gfx1250" ]; then \ + PATH=$PATH:$ROCM_HOME/llvm/bin ENABLE_CK=0 GPU_ARCHS="${GPU_ARCH_LIST}" python setup.py build_ext --inplace \ + && PATH=$PATH:$ROCM_HOME/llvm/bin ENABLE_CK=0 GPU_ARCHS="${GPU_ARCH_LIST}" pip install --no-build-isolation -e .; \ + elif [ "$BUILD_AITER_ALL" = "1" ] && [ "$BUILD_LLVM" = "1" ]; then \ sh -c "HIP_CLANG_PATH=/sgl-workspace/llvm-project/build/bin/ PREBUILD_KERNELS=1 GPU_ARCHS=$GPU_ARCH_LIST python setup.py build_ext --inplace" \ && sh -c "HIP_CLANG_PATH=/sgl-workspace/llvm-project/build/bin/ GPU_ARCHS=$GPU_ARCH_LIST pip install --config-settings editable_mode=compat -e ."; \ elif [ "$BUILD_AITER_ALL" = "1" ]; then \ @@ -643,21 +718,40 @@ RUN if [ "$BRANCH_TYPE" = "local" ]; then \ && cd python/sglang/kernels/aot \ && rm -f pyproject.toml \ && mv pyproject_rocm.toml pyproject.toml \ - && AMDGPU_TARGET=$GPU_ARCH_LIST python setup_rocm.py install \ - && cd ../../../.. \ - && rm -rf python/pyproject.toml && mv python/pyproject_other.toml python/pyproject.toml \ - # srt_hip pins compressed-tensors==0.15.0, which requires torch<2.11. Use - # the torch-2.11 ROCm 7.2.4 extra for both upgraded stacks; all other - # flavors keep the extras they used before. + && AMDGPU_TARGET=$GPU_ARCH_LIST python setup_rocm.py install +RUN pip list --format=freeze | grep -E '^(torch|triton)' > /tmp/constraints.txt + +# srt_hip pins compressed-tensors==0.15.0, which requires torch<2.11 and so +# cannot be satisfied on the ROCm 7.2.4/1000 torch 2.11 stack. The *_rocm724 +# extras carry a 0.16.0 pin instead; all other flavors keep the extras they used before. +RUN cd sglang \ + && cp python/pyproject_other.toml python/pyproject.toml \ && case "${GPU_ARCH}" in \ - *-rocm724) srt_extras="srt_hip_rocm724,diffusion_hip"; all_extras="all_hip_rocm724" ;; \ - *-rocm1000) srt_extras="srt_hip_rocm724,diffusion_hip"; all_extras="all_hip_rocm724" ;; \ - *) srt_extras="srt_hip,diffusion_hip"; all_extras="all_hip" ;; \ + *-rocm1000) srt_extras="srt_hip_rocm724,diffusion_hip"; \ + all_extras="all_hip_rocm724" ; \ + CONS="-c /tmp/constraints.txt" ; \ + ;; \ + *-rocm724) srt_extras="srt_hip_rocm724,diffusion_hip"; \ + all_extras="all_hip_rocm724" ; \ + CONS="-c /tmp/constraints.txt" ; \ + ;; \ + *-rocm720) srt_extras="srt_hip,diffusion_hip"; \ + all_extras="all_hip" ; \ + CONS="-c /tmp/constraints.txt" ; \ + ;; \ + *) srt_extras="srt_hip,diffusion_hip"; \ + all_extras="all_hip" ; \ + CONS="-c /tmp/constraints.txt" ; \ + echo 'diffusers==0.37.0' >> /tmp/constraints.txt ; \ + echo 'transformers==5.12.1' >> /tmp/constraints.txt ; \ + echo 'tokenizers==0.22.2' >> /tmp/constraints.txt ; \ + echo 'huggingface_hub==1.27.0' >> /tmp/constraints.txt ; \ + ;; \ esac \ && if [ "$BUILD_TYPE" = "srt" ]; then \ - export SETUPTOOLS_SCM_PRETEND_VERSION="${SETUPTOOLS_SCM_PRETEND_VERSION}" && python -m pip --no-cache-dir install -e "python[${srt_extras}]"; \ + export SETUPTOOLS_SCM_PRETEND_VERSION="${SETUPTOOLS_SCM_PRETEND_VERSION}" && python -m pip --no-cache-dir install $CONS -e "python[${srt_extras}]"; \ else \ - export SETUPTOOLS_SCM_PRETEND_VERSION="${SETUPTOOLS_SCM_PRETEND_VERSION}" && python -m pip --no-cache-dir install -e "python[${all_extras}]"; \ + export SETUPTOOLS_SCM_PRETEND_VERSION="${SETUPTOOLS_SCM_PRETEND_VERSION}" && python -m pip --no-cache-dir install $CONS -e "python[${all_extras}]"; \ fi RUN python -m pip cache purge @@ -762,7 +856,12 @@ RUN /bin/bash -lc 'set -euo pipefail; \ git fetch --depth=1 origin "${TILELANG_COMMIT}" || true && \ git checkout -f "${TILELANG_COMMIT}" && \ git submodule update --init --recursive && \ - export CMAKE_ARGS="-DUSE_CUDA=OFF -DUSE_ROCM=ON -DROCM_PATH=/opt/rocm -DLLVM_CONFIG=${LLVM_CONFIG} -DSKBUILD_SABI_VERSION= ${CMAKE_ARGS:-}" && \ + if [ "${GPU_ARCH_LIST}" = "gfx1250" ]; then \ + export ROCM_PATH=${ROCM_HOME}; \ + else \ + export ROCM_PATH=/opt/rocm; \ + fi; \ + export CMAKE_ARGS="-DUSE_CUDA=OFF -DUSE_ROCM=ON -DROCM_PATH=${ROCM_PATH} -DLLVM_CONFIG=${LLVM_CONFIG} -DSKBUILD_SABI_VERSION= ${CMAKE_ARGS:-}" && \ "$VENV_PIP" install -e . -v --no-build-isolation --no-deps; \ if [ -f pyproject.toml ]; then sed -i "/^[[:space:]]*\"torch/d" pyproject.toml || true; fi; \ "$VENV_PIP" cache purge || true; \ @@ -774,7 +873,7 @@ RUN /bin/bash -lc 'set -euo pipefail; \ git clone --branch "${FHT_BRANCH}" "${FHT_REPO}" fast-hadamard-transform; \ cd fast-hadamard-transform; \ git checkout -f "${FHT_COMMIT}"; \ - python setup.py install' + PYTORCH_ROCM_ARCH=${GPU_ARCH_LIST} python setup.py install' # ----------------------- # Python tools @@ -802,6 +901,19 @@ RUN /bin/bash -lc 'set -euo pipefail; \ initramfs-tools \ && rm -rf /var/lib/apt/lists/*; \ \ + # gfx1250 brought its own MORI build up against a wider set of RDMA, MPI and + # gRPC packages. Scoped to that arch so the CDNA images keep the package set + # they were released with. + if [ "${GPU_ARCH_LIST}" = "gfx1250" ]; then \ + apt-get update && apt-get install -y --no-install-recommends \ + libdrm-dev \ + librdmacm-dev rdmacm-utils infiniband-diags ibverbs-utils perftest ethtool \ + libibverbs-dev rdma-core \ + openssh-server openmpi-bin openmpi-common libopenmpi-dev \ + libgrpc++-dev protobuf-compiler-grpc \ + && rm -rf /var/lib/apt/lists/*; \ + fi; \ + \ # NIC backend deps — mori auto-detects NIC at runtime (MORI_DEVICE_NIC env var override). # Only vendor packages are installed here for dlopen (e.g. libionic.so); no compile-time flags needed. case "${NIC_BACKEND}" in \ @@ -873,7 +985,18 @@ RUN /bin/bash -lc 'set -euo pipefail; \ # so it shadows nothing system-wide. Scope this explicitly to ROCm 10 so the # ROCm 7.2 and 7.2.4 MORI build paths remain byte-for-byte equivalent here. ROCM_SYSDEPS="${ROCM_HOME:-/opt/rocm}/lib/rocm_sysdeps"; \ - if [ "${GPU_ARCH##*-}" = "rocm1000" ] && [ -d "${ROCM_SYSDEPS}" ]; then \ + if [ "${GPU_ARCH_LIST}" = "gfx1250" ] && [ -d "${ROCM_SYSDEPS}" ]; then \ + # gfx1250 was brought up needing the SDK's own cmake trees on the prefix + # path as well: that is what lets hsakmt-config.cmake resolve + # find_dependency(NUMA) without patching MORI's CMakeLists. + export PATH="${ROCM_HOME}/bin:${PATH}"; \ + export CMAKE_PREFIX_PATH="${ROCM_SYSDEPS}/lib/cmake:${ROCM_SYSDEPS}:${ROCM_HOME}/lib/cmake:${ROCM_HOME}${CMAKE_PREFIX_PATH:+:${CMAKE_PREFIX_PATH}}"; \ + export CPATH="${ROCM_SYSDEPS}/include${CPATH:+:${CPATH}}"; \ + export LIBRARY_PATH="${ROCM_SYSDEPS}/lib${LIBRARY_PATH:+:${LIBRARY_PATH}}"; \ + echo "${ROCM_SYSDEPS}/lib" > /etc/ld.so.conf.d/rocm-sysdeps.conf; \ + ldconfig; \ + echo "[MORI] rocm_sysdeps prefix: ${ROCM_SYSDEPS}"; \ + elif [ "${GPU_ARCH##*-}" = "rocm1000" ] && [ -d "${ROCM_SYSDEPS}" ]; then \ export CMAKE_PREFIX_PATH="${ROCM_SYSDEPS}${CMAKE_PREFIX_PATH:+:${CMAKE_PREFIX_PATH}}"; \ export CPATH="${ROCM_SYSDEPS}/include${CPATH:+:${CPATH}}"; \ export LIBRARY_PATH="${ROCM_SYSDEPS}/lib${LIBRARY_PATH:+:${LIBRARY_PATH}}"; \ @@ -901,13 +1024,20 @@ RUN /bin/bash -lc 'set -euo pipefail; \ build-essential autoconf automake libtool pkg-config git \ libibverbs-dev librdmacm-dev rdma-core && rm -rf /var/lib/apt/lists/*; \ # Mooncake's dependencies.sh apt-installs Ubuntu's libabsl-dev (20220623 on - # the noble base used by rocm724 and rocm1000). NIXL's meson then finds - # absl_base but no absl_log and refuses to fall back to its bundled Abseil -- - # "that would result in a mix of Abseil versions at runtime" -- so nixl fails - # at metadata generation. Drop just the -dev package (headers and pkg-config - # files); the runtime library that already-built components link against - # stays in place. - case "${GPU_ARCH}" in *-rocm724|*-rocm1000) apt-get remove -y libabsl-dev ;; esac; \ + # the noble base used by rocm724 and rocm1000). NIXL's meson then + # finds absl_base but no absl_log and refuses to fall back to its bundled + # Abseil -- "that would result in a mix of Abseil versions at runtime" -- so + # nixl fails at metadata generation. Drop just the -dev package (headers and + # pkg-config files); the runtime library that already-built components link + # against stays in place. + # + # gfx1250 was brought up dropping the runtime package as well, so keep that + # for gfx1250 only rather than changing what the CDNA images were released with. + if [ "${GPU_ARCH_LIST}" = "gfx1250" ]; then \ + apt-get remove -y libabsl-dev libabsl20220623 || true; \ + else \ + case "${GPU_ARCH}" in *-rocm724|*-rocm1000) apt-get remove -y libabsl-dev ;; esac; \ + fi; \ pip install --no-cache-dir meson ninja pybind11 meson-python patchelf pyyaml; \ git clone --depth=1 -b "${UCX_BRANCH}" "${UCX_REPO}" /sgl-workspace/ucx; \ cd /sgl-workspace/ucx && ./autogen.sh && mkdir build && cd build && \ @@ -1011,9 +1141,25 @@ RUN python3 -c "from pathlib import Path; import transformers.dynamic_module_uti # install pull CUDA torch instead. The hip check below is the tripwire. # torch 2.11 names this `triton-rocm`; uninstall it so the pin is the only Triton. RUN if [ "$BUILD_TRITON" = "1" ]; then \ - pip uninstall -y triton-rocm || true \ - && PIP_NO_CACHE_DIR=1 pip install --extra-index-url ${TRITON_INDEX_URL} \ - "triton==${TRITON_VERSION}" "triton-kernels==${TRITON_KERNELS_VERSION}" \ + case "${GPU_ARCH}" in \ + gfx1250-rocm1000) \ + echo "[Triton] gfx1250: building ${TRITON_COMMIT} from source"; \ + pip uninstall -y triton triton-rocm || true \ + && apt-get update && apt-get install -y --no-install-recommends cmake && rm -rf /var/lib/apt/lists/* \ + && git clone ${TRITON_REPO} triton-custom \ + && cd triton-custom \ + && git checkout ${TRITON_COMMIT} \ + && pip install -r python/requirements.txt \ + && pip install -e . \ + && if [ -d python/triton_kernels ]; then pip install -e python/triton_kernels --no-deps; fi; \ + ;; \ + *rocm72*) \ + echo "[Triton] ROCm 7.2: installing pinned wheels from ${TRITON_INDEX_URL}"; \ + pip uninstall -y triton-rocm || true && \ + PIP_NO_CACHE_DIR=1 pip install --extra-index-url ${TRITON_INDEX_URL} \ + "triton==${TRITON_VERSION}" "triton-kernels==${TRITON_KERNELS_VERSION}"; \ + ;; \ + esac \ && python3 -c "import torch; from importlib.metadata import version; v = version('triton'); k = version('triton-kernels'); assert torch.version.hip is not None, torch.__version__; print(f'[Triton] ROCm Torch {torch.__version__}, Triton {v}, triton-kernels {k}')"; \ fi diff --git a/docker/sgl-deep-ep.Dockerfile b/docker/sgl-deep-ep.Dockerfile index a3743073b895..4ccf930f322e 100644 --- a/docker/sgl-deep-ep.Dockerfile +++ b/docker/sgl-deep-ep.Dockerfile @@ -6,7 +6,6 @@ FROM ${BASE_IMAGE}:cuda${CUDA_VERSION} ARG ARCHITECTURE=x86_64 ARG CUDA_TAG=cu130 ARG CUDA_VERSION=13.0 -ARG GDRCOPY_VERSION=2.5.1 ARG NCCL_VERSION=2.30.7 ARG PYTHON_TAG=cp312-cp312 ARG TORCH_VERSION=2.13.0 @@ -49,19 +48,6 @@ RUN set -eux; \ ln -sf "${cuda_stub}" /usr/lib64/libcuda.so; \ ln -sf "${cuda_stub}" "/usr/lib/${ARCHITECTURE}-linux-gnu/libcuda.so" -# DeepEP v2 uses NCCL Gin on CUDA 13. Keep GDRCopy only for the CUDA 12 -# legacy NVSHMEM/IBGDA build. -RUN set -eux; \ - if [ "${CUDA_TAG}" = cu129 ]; then \ - git clone --depth 1 --branch "v${GDRCOPY_VERSION}" \ - https://github.com/NVIDIA/gdrcopy.git /opt/gdrcopy; \ - make -C /opt/gdrcopy CUDA="${CUDA_HOME}" prefix=/usr/local lib_install; \ - printf '%s\n' /usr/local/lib > /etc/ld.so.conf.d/gdrcopy.conf; \ - ldconfig; \ - test -f /usr/local/include/gdrapi.h; \ - ldconfig -p | grep -q libgdrapi; \ - fi - RUN --mount=type=cache,id=sgl-deep-ep-pip-${CUDA_TAG}-${PYTHON_TAG}-${ARCHITECTURE},target=/root/.cache/pip \ set -eux; \ "${PYTHON_BIN}" -m pip uninstall -y deep-ep sgl-deep-ep || true; \ @@ -69,10 +55,8 @@ RUN --mount=type=cache,id=sgl-deep-ep-pip-${CUDA_TAG}-${PYTHON_TAG}-${ARCHITECTU "${PYTHON_BIN}" -m pip install --force-reinstall \ "torch==${TORCH_VERSION}" \ --index-url "https://download.pytorch.org/whl/${CUDA_TAG}"; \ - if [ "${CUDA_TAG}" = cu130 ]; then \ - "${PYTHON_BIN}" -m pip install --force-reinstall --no-deps \ - "nvidia-nccl-cu13==${NCCL_VERSION}"; \ - fi; \ + "${PYTHON_BIN}" -m pip install --force-reinstall --no-deps \ + "nvidia-nccl-cu13==${NCCL_VERSION}"; \ "${PYTHON_BIN}" -m pip install \ "auditwheel>=6.0" \ build \ diff --git a/docker/sgl-deep-gemm.Dockerfile b/docker/sgl-deep-gemm.Dockerfile index 9fc653014958..6d89bf884e2d 100644 --- a/docker/sgl-deep-gemm.Dockerfile +++ b/docker/sgl-deep-gemm.Dockerfile @@ -15,7 +15,9 @@ ARG PYTORCH_MIRROR=download.pytorch.org ENV PYTHON_ROOT_PATH=/opt/python/${PYTHON_TAG} ENV PATH=${PYTHON_ROOT_PATH}/bin:${PATH} +# DeepJIT's exception handling requires elfutils/libdwfl.h and libelf headers. RUN yum install -y --nogpgcheck git wget tar gcc gcc-c++ make \ + elfutils-devel elfutils-libelf-devel \ && yum clean all && rm -rf /var/cache/yum RUN set -eux; \ @@ -27,8 +29,7 @@ RUN --mount=type=cache,id=sgl-deep-gemm-pip,target=/root/.cache/pip \ set -eux; \ case "${CUDA_VERSION}" in \ 13.0) CU_TAG=cu130 ;; \ - 12.9) CU_TAG=cu129 ;; \ - *) CU_TAG=cu130 ;; \ + *) echo "Unsupported CUDA version: ${CUDA_VERSION}" && exit 1 ;; \ esac; \ ${PYTHON_ROOT_PATH}/bin/pip install torch==${TORCH_VER} --index-url https://${PYTORCH_MIRROR}/whl/${CU_TAG}; \ ${PYTHON_ROOT_PATH}/bin/pip install --index-url ${PIP_DEFAULT_INDEX} \ diff --git a/docker/sgl-router.Dockerfile b/docker/sgl-router.Dockerfile index 67471a52cbb9..9301e77c2166 100644 --- a/docker/sgl-router.Dockerfile +++ b/docker/sgl-router.Dockerfile @@ -12,11 +12,9 @@ # manifests → cargo fetch → copy src" approach caches only the fetched # registry; every source change still recompiles every dep. # -# `Cargo.lock` is gitignored repo-wide (root .gitignore "# Rust lib" -# block), so we generate it inside the chef stage with `cargo -# generate-lockfile` and propagate that lockfile to the builder via -# `COPY --from=chef`. Both stages thus build against the same lockfile, -# preserving --locked semantics within a single Docker build. +# `experimental/sgl-router/Cargo.lock` is committed, and every cargo step +# below runs with `--locked`, so the image ships exactly the dependency +# graph CI checked, tested, and license-audited. # # Build (from the repo root): # docker build -f docker/sgl-router.Dockerfile -t sgl-router:dev . @@ -28,24 +26,23 @@ # Image budget: < 100 MB stripped (M6 acceptance). Verify with # `docker image inspect sgl-router:dev --format '{{.Size}}'`. -ARG RUST_VERSION=1.90 +ARG RUST_VERSION=1.92 ARG DEBIAN_VERSION=bookworm ######################## STAGE 1 — chef recipe ########################## FROM rust:${RUST_VERSION}-${DEBIAN_VERSION} AS chef RUN cargo install cargo-chef --locked --version ^0.1 WORKDIR /work/sgl-router -COPY experimental/sgl-router/Cargo.toml ./ +COPY experimental/sgl-router/Cargo.toml experimental/sgl-router/Cargo.lock ./ COPY experimental/sgl-router/sgl-kv-indexer/Cargo.toml sgl-kv-indexer/Cargo.toml -# Stub a minimal src tree so cargo can resolve the workspace, generate -# the lockfile (gitignored upstream), then prepare the chef recipe. +# Stub a minimal src tree so cargo can see the workspace targets, then +# prepare the chef recipe. RUN mkdir -p src sgl-kv-indexer/src/bin \ && echo "fn main() {}" > src/main.rs \ && echo "" > src/lib.rs \ && echo "" > sgl-kv-indexer/src/lib.rs \ && echo "fn main() {}" > sgl-kv-indexer/src/bin/kv-indexer-server.rs \ && echo "fn main() {}" > sgl-kv-indexer/src/bin/kv-indexer-bridge.rs \ - && cargo generate-lockfile \ && cargo chef prepare --recipe-path recipe.json \ && rm -rf src sgl-kv-indexer/src @@ -66,26 +63,23 @@ WORKDIR /work/sgl-router ENV PCRE2_SYS_STATIC=1 COPY --from=chef /work/sgl-router/recipe.json ./recipe.json -COPY --from=chef /work/sgl-router/Cargo.lock ./Cargo.lock COPY experimental/sgl-router/sgl-kv-indexer/Cargo.toml sgl-kv-indexer/Cargo.toml # Cook (compile + cache) the dep graph from the recipe. The recipe carries every -# workspace member's manifest, so chef recreates the Indexer's source stubs itself. -RUN cargo chef cook --release --recipe-path recipe.json +# workspace member's manifest and the lockfile, so chef recreates the Indexer's +# source stubs itself. +RUN cargo chef cook --locked --release --recipe-path recipe.json -# Now bring in the real sources and the manifest they need. -COPY experimental/sgl-router/Cargo.toml ./ +# Now bring in the real sources and the manifests they need. +COPY experimental/sgl-router/Cargo.toml experimental/sgl-router/Cargo.lock ./ COPY experimental/sgl-router/src ./src COPY experimental/sgl-router/sgl-kv-indexer/Cargo.toml sgl-kv-indexer/Cargo.toml COPY experimental/sgl-router/sgl-kv-indexer/build.rs sgl-kv-indexer/build.rs COPY experimental/sgl-router/sgl-kv-indexer/proto sgl-kv-indexer/proto COPY experimental/sgl-router/sgl-kv-indexer/src sgl-kv-indexer/src -# --locked is intentionally omitted: the lockfile is generated in-container -# (gitignored upstream) and `cargo chef cook` may have mutated it during the -# dep-cook step, so a strict --locked check would spuriously fail. RUN touch sgl-kv-indexer/build.rs \ - && cargo build --release --bin sgl-router \ + && cargo build --locked --release --bin sgl-router \ && strip target/release/sgl-router ######################## STAGE 3 — runtime ############################## diff --git a/docker/xeon.Dockerfile b/docker/xeon.Dockerfile index 2a45c17c5f12..a55a530201a5 100644 --- a/docker/xeon.Dockerfile +++ b/docker/xeon.Dockerfile @@ -1,4 +1,4 @@ -FROM ubuntu:24.04 +FROM ubuntu:26.04 SHELL ["/bin/bash", "-c"] ARG SGLANG_REPO=https://github.com/sgl-project/sglang.git @@ -8,6 +8,7 @@ RUN apt-get update && \ apt-get full-upgrade -y && \ DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends -y \ ca-certificates \ + ffmpeg \ git \ curl \ wget \ @@ -15,8 +16,9 @@ RUN apt-get update && \ gcc \ g++ \ make \ + libssl-dev \ libsqlite3-dev \ - google-perftools \ + libgoogle-perftools-dev \ libtbb-dev \ libnuma-dev \ numactl @@ -43,8 +45,7 @@ RUN source /opt/.venv/bin/activate && \ cd sglang/kernels/aot && \ cp pyproject_cpu.toml pyproject.toml && \ uv pip install . && \ - source /sgl-workspace/sglang/scripts/ci/utils/sgl_eval_ref.sh && \ - uv pip install "$SGL_EVAL_SPEC" + uv pip install "sgl-eval==0.1.0" ENV SGLANG_USE_CPU_ENGINE=1 ENV LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libtcmalloc.so.4:/usr/lib/x86_64-linux-gnu/libtbbmalloc.so:/opt/.venv/lib/libiomp5.so diff --git a/docker/xpu.Dockerfile b/docker/xpu.Dockerfile index c249f0d06778..f4e49f1941bd 100644 --- a/docker/xpu.Dockerfile +++ b/docker/xpu.Dockerfile @@ -58,10 +58,14 @@ RUN apt-get update && apt-get install -y software-properties-common curl && \ RUN apt-get update && apt-get install -y \ python3-dev \ build-essential \ + libssl-dev \ + protobuf-compiler \ && rm -rf /var/lib/apt/lists/* RUN curl -LsSf https://astral.sh/uv/install.sh | sh -ENV PATH="/root/.local/bin:$PATH" +ENV PATH="/root/.local/bin:/root/.cargo/bin:$PATH" +RUN curl --proto '=https' --retry 3 --retry-delay 2 --tlsv1.2 -sSf https://sh.rustup.rs \ +| sh -s -- -y --no-modify-path --profile minimal && rustc --version && cargo --version ENV VIRTUAL_ENV="/opt/venv" ENV UV_PYTHON_INSTALL_DIR=/opt/uv/python RUN uv venv --python ${PYTHON_VERSION} --seed ${VIRTUAL_ENV} @@ -80,4 +84,17 @@ RUN echo "Cloning ${SG_LANG_BRANCH} from ${SG_LANG_REPO}" && \ pip install --no-cache-dir ".[dev,diffusion]" --extra-index-url https://download.pytorch.org/whl/xpu && \ pip install --no-cache-dir --no-deps xgrammar==0.1.33 -CMD ["bash", "-c", "source /opt/intel/oneapi/setvars.sh --force && exec bash"] +# Install torch_memory_saver for release/resume_memory_occupation ("memory saver"). +# XPU ships no prebuilt wheel: it is built from source against the local oneAPI + +# torch-XPU runtime (the .so links libsycl.so., which must match the installed +# intel-sycl-rt). TMS_PLATFORM=xpu forces the XPU backend; --no-build-isolation +# lets the build import the installed torch (above) so it can match the libsycl +# major to it -- under build isolation torch is absent and the match is skipped. +# Pinned (v0.0.10b2) so image builds are reproducible; bump via --build-arg. +ARG TORCH_MEMORY_SAVER_REF=a5c99f11b18ebb8e9fda71a68812e476ae49e417 +# Base image already applies setvars.sh in its own layers (SETVARS_COMPLETED=1, +# icpx on PATH, LIBRARY_PATH/CPATH populated), so re-sourcing here is redundant. +RUN TMS_PLATFORM=xpu pip install --no-cache-dir --no-build-isolation \ + git+https://github.com/fzyzcjy/torch_memory_saver.git@${TORCH_MEMORY_SAVER_REF} + +CMD ["bash"] diff --git a/docs/cards/logos/ifm.png b/docs/cards/logos/ifm.png new file mode 100644 index 000000000000..1b76c749cc6b Binary files /dev/null and b/docs/cards/logos/ifm.png differ diff --git a/docs/cookbook/autoregressive/DeepSeek/DeepSeek-OCR-2.mdx b/docs/cookbook/autoregressive/DeepSeek/DeepSeek-OCR-2.mdx index cfe54990bd07..3aa56939e200 100644 --- a/docs/cookbook/autoregressive/DeepSeek/DeepSeek-OCR-2.mdx +++ b/docs/cookbook/autoregressive/DeepSeek/DeepSeek-OCR-2.mdx @@ -30,15 +30,15 @@ For more details, please refer to the [official DeepSeek-OCR-2 repository](https Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. -For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation). - ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration -**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and deployment strategy. SGLang supports serving DeepSeek-OCR-2 on NVIDIA H200 and B200, AMD MI300X, MI355X, and MI325X GPUs, as well as Intel Xeon CPUs. +The DeepSeek-OCR-2 series offers models in various sizes and architectures, optimized for different hardware platforms including NVIDIA GPUs, AMD GPUs, Intel Arc Pro B-Series GPUs(codename: BMG (Battlemage)), and Intel Xeon CPUs. The recommended launch configurations vary by hardware and model size. + +**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, quantization method, and deployment strategy. SGLang supports serving DeepSeek-OCR-2 on NVIDIA H200 and B200, AMD MI300X, MI355X, and MI325X GPUs, and Intel Arc Pro B-Series GPUs, as well as Intel Xeon CPUs. diff --git a/docs/cookbook/autoregressive/DeepSeek/DeepSeek-R1.mdx b/docs/cookbook/autoregressive/DeepSeek/DeepSeek-R1.mdx index 72880aeb6334..19c4cac2935b 100644 --- a/docs/cookbook/autoregressive/DeepSeek/DeepSeek-R1.mdx +++ b/docs/cookbook/autoregressive/DeepSeek/DeepSeek-R1.mdx @@ -79,7 +79,7 @@ DeepSeek-R1 shares the same MoE architecture as DeepSeek-V3, so the same hardwar INT8 - 16× A100/A800, 32× L40S, Xeon 6980P CPU, 4× Atlas 800I A3 + 16× A100/A800, 32× L40S, Xeon 6980P CPU, 4× Ascend A3 Series W4A8 / AWQ / MXFP4 / NVFP4 diff --git a/docs/cookbook/autoregressive/DeepSeek/DeepSeek-V3.mdx b/docs/cookbook/autoregressive/DeepSeek/DeepSeek-V3.mdx index f61150d1ab3d..ec65a5866866 100644 --- a/docs/cookbook/autoregressive/DeepSeek/DeepSeek-V3.mdx +++ b/docs/cookbook/autoregressive/DeepSeek/DeepSeek-V3.mdx @@ -58,7 +58,7 @@ import { DeepSeekV3Deployment } from "/src/snippets/autoregressive/deepseek-v3-d INT8 - 16× A100/A800, 32× L40S, Xeon 6980P CPU, 4× Atlas 800I A3 + 16× A100/A800, 32× L40S, Xeon 6980P CPU, 4× Ascend A3 Series W4A8 / AWQ / MXFP4 / NVFP4 @@ -392,7 +392,7 @@ python3 -m sglang.launch_server \ The default configuration is `--speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`. Find the best values for your workload with [bench_speculative.py](https://github.com/sgl-project/sglang/blob/main/scripts/playground/bench_speculative.py). The minimum viable config is `--speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2`. -For large batch sizes (>48), increase `--max-running-requests` beyond the default of 48 for MTP. Also set `--cuda-graph-bs` to include your target batch sizes (default captured sizes for speculative decoding: 48). +For large batch sizes (>48), increase `--max-running-requests` beyond the default of 48 for MTP. Also set `--cuda-graph-bs-decode` to include your target batch sizes (default captured sizes for speculative decoding: 48). @@ -404,7 +404,7 @@ The spec-v2 overlap scheduler is enabled by default. It improves performance by DeepSeek V3 uses [Multi-head Latent Attention (MLA)](https://arxiv.org/pdf/2405.04434), an attention mechanism that improves inference efficiency. SGLang implements several optimizations: - **Weight Absorption:** Reorders matrix multiplications to improve decoding phase efficiency. -- **MLA Attention Backends:** FA3, Flashinfer, FlashMLA, CutlassMLA, TRTLLM MLA (Blackwell), and Triton. FA3 is the default. +- **MLA Attention Backends:** FA3, Flashinfer, FlashMLA, TRTLLM MLA (Blackwell), and Triton. FA3 is the default. - **FP8 Quantization:** W8A8 FP8 and KV Cache FP8, with BMM operators for weight-absorbed MLA in FP8. - **CUDA Graph & Torch.compile:** Both MLA and MoE support CUDA Graph and Torch.compile for reduced decoding latency. - **Chunked Prefix Cache:** Increases throughput for long-sequence chunked prefill (FlashAttention3 backend only). @@ -425,7 +425,7 @@ For multi-node serving and hardware-specific examples: - [16× A100 INT8](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-16-a100a800-with-int8-quantization) - [32× L40S INT8](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-32-l40s-with-int8-quantization) - [Xeon 6980P CPU](../../../docs/hardware-platforms/cpu_server#example-running-deepseek-v3-1-terminus) -- [4× Atlas 800I A3 (int8)](../../../docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_r1#multi-node-pd-disaggregation-deployment) +- [4× Ascend A3 Series (int8)](../../../docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_r1#multi-node-pd-disaggregation-deployment) **Blog references for large-scale deployment:** diff --git a/docs/cookbook/autoregressive/DeepSeek/DeepSeek-V3_1.mdx b/docs/cookbook/autoregressive/DeepSeek/DeepSeek-V3_1.mdx index 3c721c8d5f7c..883ac1077dc8 100644 --- a/docs/cookbook/autoregressive/DeepSeek/DeepSeek-V3_1.mdx +++ b/docs/cookbook/autoregressive/DeepSeek/DeepSeek-V3_1.mdx @@ -56,7 +56,7 @@ DeepSeek-V3.1 shares the same model architecture as DeepSeek-V3, so the same har INT8 - 16× A100/A800, 32× L40S, Xeon 6980P CPU, 4× Atlas 800I A3 + 16× A100/A800, 32× L40S, Xeon 6980P CPU, 4× Ascend A3 Series W4A8 / AWQ / MXFP4 / NVFP4 diff --git a/docs/cookbook/autoregressive/DeepSeek/DeepSeek-V3_2.mdx b/docs/cookbook/autoregressive/DeepSeek/DeepSeek-V3_2.mdx index 14166f925a93..f0c4a7df4101 100644 --- a/docs/cookbook/autoregressive/DeepSeek/DeepSeek-V3_2.mdx +++ b/docs/cookbook/autoregressive/DeepSeek/DeepSeek-V3_2.mdx @@ -40,7 +40,7 @@ docker pull lmsysorg/sglang:v0.5.8-rocm700-mi35x # Prefer the newest MI30x ROCm image tag from Docker Hub when available, or build from source. docker pull lmsysorg/sglang:v0.5.8-rocm700-mi30x -# Ascend NPU (Atlas 800I A2 / A3) +# Ascend NPU (A2/A3 Series) docker pull lmsysorg/sglang:dsv32-a2 docker pull lmsysorg/sglang:dsv32-a3 ``` @@ -432,25 +432,19 @@ For production deployments (RBG / LWS-based, DeepEP EP parallelism), see [multi_ #### 4.2.5 DSA Long-Sequence Context Parallel and PP/CP -SGLang provides two context parallel (CP) modes for long-sequence workloads, controlled with `--dsa-prefill-cp-mode`. +Enable prefill context parallelism (CP) on CUDA with `--enable-prefill-cp --cp-strategy interleave` for long-sequence workloads. -**In-sequence splitting** (`--dsa-prefill-cp-mode in-seq-split`): Each CP rank handles a uniform shard of the sequence; KV cache is gathered via all-gather. Batch size is restricted to 1 during prefill. See [PR #12065](https://github.com/sgl-project/sglang/pull/12065). - -```bash Command -# In-seq splitting mode — EP + DP, batch size 1 -python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.2-Exp \ - --tp 8 --ep 8 --dp 2 --enable-dp-attention \ - --enable-dsa-prefill-context-parallel --attn-cp-size 4 \ - --dsa-prefill-cp-mode in-seq-split --max-running-requests 32 -``` + +Zigzag prefill CP (`--cp-strategy zigzag`) is temporarily unavailable for DeepSeek V3.2, GLM-5, GLM-5.1, GLM-5.2, and GLM-5.3. Use `interleave` for these models and keep `--dp 1`; interleave DSA CP does not support `--dp` greater than 1. + -**Round-robin splitting** (`--dsa-prefill-cp-mode round-robin-split`, default): Distributes tokens by `token_idx % cp_size`. Supports fused MoE, FP8 KV cache, and multi-batch prefill. Cannot be combined with DP attention. See [PR #13959](https://github.com/sgl-project/sglang/pull/13959). +**Interleave** (`--cp-strategy interleave`): Distributes tokens by `token_idx % cp_size`. It supports fused MoE, FP8 KV cache, and multi-batch prefill. ```bash Command -# Round-robin splitting — FusedMoE + CP8 +# Interleave — FusedMoE + CP8 python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.2-Exp \ - --tp 8 --enable-dsa-prefill-context-parallel --attn-cp-size 8 \ - --dsa-prefill-cp-mode round-robin-split --max-running-requests 32 + --tp 8 --enable-prefill-cp --attn-cp-size 8 \ + --cp-strategy interleave --max-running-requests 32 ``` **PP + CP (multi-node):** Combines Pipeline Parallelism and Context Parallelism for cross-node scaling. The production-optimized configurations below have been verified on Hopper: @@ -499,8 +493,8 @@ sglang_args=$(echo serve \ --attention-backend dsa \ --dsa-prefill-backend flashmla_sparse \ --dsa-decode-backend flashmla_sparse \ - --enable-dsa-prefill-context-parallel \ - --dsa-prefill-cp-mode round-robin-split \ + --enable-prefill-cp \ + --cp-strategy interleave \ --cuda-graph-max-bs-decode 128 \ --max-running-requests 128 \ --trust-remote-code --host "0.0.0.0" --port 30000 \ @@ -518,34 +512,19 @@ sglang "${sglang_args[@]}" 2>&1 | tee $LOG_DIR/$RANK.log **fp8 KV + CP + PP** -With FP8 KV, we can have less memory footprint. This can be combined with various parallel schemes: +FP8 KV reduces the memory footprint. For DeepSeek V3.2, combine it with interleave CP and PP while keeping `--dp 1`: ```shell Command # verified in Hopper platform -dp=1 - dp_config=" \ --dp 1 --enable-dp-attention \ " cp_config=" \ - --enable-dsa-prefill-context-parallel \ -" - -if [ "$dp" -eq 1 ]; then - -cp_config=" \ - $cp_config \ - --dsa-prefill-cp-mode round-robin-split \ + --enable-prefill-cp \ + --cp-strategy interleave \ " -else -cp_config=" \ - $cp_config \ - --dsa-prefill-cp-mode in-seq-split \ -" -fi - # see discussion : https://github.com/sgl-project/sglang/pull/12065 sglang_args=$(echo serve \ --model-path $MAPPED_MODEL_PATH \ @@ -813,7 +792,7 @@ python3 -m sglang.test.run_eval --port 30000 --eval-name gpqa --num-examples 198 #### 5.2.4 AIME 2025 Benchmark -Results on AIME 2025 (8×B200), evaluated with [NeMo-Skills](https://github.com/NVIDIA/NeMo-Skills.git): +Results on AIME 2025 (8×B200), evaluated with [sgl-eval](https://github.com/sgl-project/sgl-eval): @@ -846,25 +825,19 @@ Results on AIME 2025 (8×B200), evaluated with [NeMo-Skills](https://github.com/
-**Reproduction.** Install [NeMo-Skills](https://github.com/NVIDIA/NeMo-Skills), launch the server with the tool-call and reasoning parsers, then run `ns eval`: +**Reproduction.** Install [sgl-eval](https://github.com/sgl-project/sgl-eval), launch the server with the tool-call and reasoning parsers, then run `sgl-eval run`. The AIME 2025 dataset ships with sgl-eval, and thinking is on by default for it: ```bash Command -pip install git+https://github.com/NVIDIA/NeMo-Skills.git --ignore-installed blinker - -export NEMO_SKILLS_DISABLE_UNCOMMITTED_CHANGES_CHECK=1 -ns prepare_data aime25 - -ns eval \ - --benchmarks=aime25:4 \ - --server_type=sglang \ - --model=deepseek-ai/DeepSeek-V3.2-Exp \ - --server_address=http://localhost:30000/v1 \ - --output_dir=nemo_skills_aime25_output \ - ++chat_template_kwargs.thinking=true \ - ++inference.temperature=1.0 \ - ++inference.top_p=0.95 \ - ++inference.tokens_to_generate=64000 - # Use ++inference.tokens_to_generate=120000 for the DeepSeek-V3.2-Speciale model +pip install sgl-eval + +sgl-eval run aime25 \ + --base-url http://localhost:30000/v1 \ + --model deepseek-ai/DeepSeek-V3.2-Exp \ + --n-repeats 4 \ + --temperature 1.0 \ + --top-p 0.95 \ + --max-tokens 64000 + # Use --max-tokens 120000 for the DeepSeek-V3.2-Speciale model ``` ### 5.3 Speed Benchmark on Hopper diff --git a/docs/cookbook/autoregressive/DeepSeek/DeepSeek-V4.mdx b/docs/cookbook/autoregressive/DeepSeek/DeepSeek-V4.mdx index 7e507211bd3e..a4b8e2166944 100644 --- a/docs/cookbook/autoregressive/DeepSeek/DeepSeek-V4.mdx +++ b/docs/cookbook/autoregressive/DeepSeek/DeepSeek-V4.mdx @@ -1,7 +1,6 @@ --- title: DeepSeek-V4 -description: "Deploy DeepSeek-V4 with SGLang — verified launch commands, benchmarks, and tuning for Flash Official (0731), Flash, Pro, and Pro Official (0813)." -tag: NEW +description: "Deploy DeepSeek-V4 with SGLang — verified launch commands, benchmarks, and tuning for Flash Official (0731), Flash, Flash Vision (Exp), Pro, and Pro Official (0813)." --- ## Deployment @@ -32,7 +31,7 @@ For how to launch the image, see [Install → Method 3: Using Docker](../../../d **NVIDIA GPUs** -A single image — `lmsysorg/sglang:latest` — covers the **datacenter GPUs** in this cookbook (B200 / B300 / GB200 / GB300 / H100 / H200 / RTX PRO 6000). +A single image — `lmsysorg/sglang:latest` — covers the **datacenter GPUs** in this cookbook (B200 / B300 / GB200 / GB300 / H100 / H200 / RTX PRO 6000). The one exception is **Flash Vision (Exp)**, whose support has not shipped in a release yet: its cells use the preview image `lmsysorg/sglang:dev-dsv4-flash-vision` (the command panel picks it automatically — see the [Flash Vision notes](#vision-note)). **DGX Spark** is the other exception: its three cells (Flash Official FP4, Flash Official NVFP4, Flash Vision FP4) use the DGX Spark–only preview image `lmsysorg/sglang:dev-v4f-2dgx-v2` (the command panel picks it automatically — see the [DGX Spark notes](#spark-note)); do not use that image on any other hardware. ```bash Command docker pull lmsysorg/sglang:latest @@ -52,11 +51,11 @@ docker run --gpus all \ AMD uses the daily-updated `lmsysorg/sglang-rocm` images. You can find the latest images on [Docker Hub](https://hub.docker.com/r/lmsysorg/sglang-rocm/tags). We recommend the ROCm 7.2 version. For example: -- **MI355X** → `lmsysorg/sglang-rocm:v0.5.18-rocm720-mi35x-20260828` -- **MI300X** → `lmsysorg/sglang-rocm:v0.5.18-rocm720-mi30x-20260828` +- **MI355X** → `lmsysorg/sglang-rocm:v0.5.19-rocm720-mi35x-20260914` +- **MI300X** → `lmsysorg/sglang-rocm:v0.5.19-rocm720-mi30x-20260914` ```bash Command -docker pull lmsysorg/sglang-rocm:v0.5.18-rocm720-{mi35x,mi30x}-20260828 +docker pull lmsysorg/sglang-rocm:v0.5.19-rocm720-{mi35x,mi30x}-20260914 docker run \ --device=/dev/kfd --device=/dev/dri \ @@ -66,7 +65,7 @@ docker run \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=" \ - lmsysorg/sglang-rocm:v0.5.18-rocm720-{mi35x,mi30x}-20260828 \ + lmsysorg/sglang-rocm:v0.5.19-rocm720-{mi35x,mi30x}-20260914 \ sglang serve ``` @@ -86,6 +85,10 @@ import { Deployment } from "/src/snippets/_deployment.jsx"; import { config } from "/src/snippets/configs/deepseek-ai/deepseek-v4.jsx"; import { benchmarks } from "/src/snippets/configs/deepseek-ai/deepseek-v4-benchmarks.jsx"; + + The **Flash Vision (Exp)** variant runs on a dedicated preview Docker image, `lmsysorg/sglang:dev-dsv4-flash-vision` — its support ([sgl-project/sglang#37253](https://github.com/sgl-project/sglang/pull/37253)) has not shipped in a release yet. The command panel's Docker mode emits that image automatically for Flash Vision cells; see the [Flash Vision notes](#vision-note). + + @@ -127,7 +130,7 @@ import { Playground } from "/src/snippets/_playground.jsx"; ## 1. Model Introduction -**DeepSeek-V4** is the next-generation Mixture-of-Experts model from DeepSeek, released 2026-04-24 under an **MIT License**. The 0731 Flash and 0813 Pro refreshes add checkpoints with a bundled DSpark draft head: +**DeepSeek-V4** is the next-generation Mixture-of-Experts model from DeepSeek, released 2026-04-24 under an **MIT License**. The 0731 Flash and 0813 Pro refreshes add checkpoints with a bundled DSpark draft head, and the experimental Flash Vision checkpoint builds image understanding on top of the 0731 Flash base: @@ -157,6 +160,12 @@ import { Playground } from "/src/snippets/_playground.jsx"; + + + + + + @@ -178,7 +187,7 @@ The Instruct checkpoints ship as **FP4 MoE experts + FP8 attention / dense** (on **Recommended generation:** `temperature=1.0`, `top_p=1.0`. -**Resources:** HuggingFace · [Flash Official (0731)](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731) · [Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash) · [Pro](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro) · [Pro Official (0813)](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro-0813) · ModelScope · [Flash](https://modelscope.cn/models/deepseek-ai/DeepSeek-V4-Flash) · [Pro](https://modelscope.cn/models/deepseek-ai/DeepSeek-V4-Pro). +**Resources:** HuggingFace · [Flash Official (0731)](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731) · [Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash) · [Flash Vision (Exp)](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp) · [Pro](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro) · [Pro Official (0813)](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro-0813) · ModelScope · [Flash](https://modelscope.cn/models/deepseek-ai/DeepSeek-V4-Flash) · [Pro](https://modelscope.cn/models/deepseek-ai/DeepSeek-V4-Pro). ## 2. Configuration Tips @@ -205,6 +214,30 @@ For the original Flash and Pro checkpoints: - `high-throughput`: MTP disabled — at saturation the verify step costs more than it saves. - MTP runs on the v2 speculative path. + + +**DGX Spark (2x GB10): Flash Official FP4 / NVFP4, Flash Vision FP4** + +The **DGX Spark** row has three recipes, all **Balanced · Multi-Nodes**: **Flash Official (0731) · FP4**, **Flash Official (0731) · NVFP4**, and **Flash Vision (Exp) · FP4**. None of these checkpoints fits one 128GB GB10, so every recipe runs TP=2 across two DGX Sparks connected over ConnectX-7 (RoCE). Every other DGX Spark combination is greyed out on purpose. + +- **Docker image** — all three cells use `lmsysorg/sglang:dev-v4f-2dgx-v2`, a preview build made **only for DGX Spark** (branch `b12x-vision` @ `452239a74f`): it bakes in the SM12x `b12x` MoE (W4A8) and compressed-MLA attention kernels ([#34878](https://github.com/sgl-project/sglang/pull/34878), [#35899](https://github.com/sgl-project/sglang/pull/35899), [#34018](https://github.com/sgl-project/sglang/pull/34018)), the Flash Vision model support ([#37253](https://github.com/sgl-project/sglang/pull/37253)), the b12x image-prefill fix that lets Flash Vision serve images on SM12x, the NVFP4 MTP-layer dispatch fix, and the CuTeDSL and NCCL pins the GB10 pair needs. Do not use it on other hardware, and use the panel's Docker mode — the bare Python command needs the `b12x` kernel package this image ships. +- **Run the same command on both Sparks** with `--node-rank 0` / `--node-rank 1` and `--dist-init-addr` pointing at node 0 over the ConnectX-7 link. The `docker run` flags the panel emits (`--network host --ulimit memlock=-1:-1 --cap-add IPC_LOCK --device /dev/infiniband`) are what let NCCL use RDMA; without them NCCL silently falls back to TCP and decode slows by roughly 40%. +- **Env knobs** in the cells are part of the recipe: `SGLANG_SM120_FLASHMLA_BACKEND=b12x` selects the b12x attention path, `SGLANG_B12X_MAX_TOKENS` must equal `--chunked-prefill-size`, and `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` avoids unified-memory fragmentation OOMs on GB10. +- **NVFP4 (`nvidia/DeepSeek-V4-Flash-0731-NVFP4`)** — only the routed experts are NVFP4; attention, shared experts and the DSpark MTP layer stay in the checkpoint's native formats. On SM12x that means three extra flags: `--moe-runner-backend flashinfer_cutlass` (b12x's MoE is MXFP4-only and trtllm-gen kernels are sm100-only), `--speculative-moe-runner-backend b12x` (the DSpark draft's MTP experts are MXFP4 and run on b12x), and `--disable-shared-experts-fusion` (HashTopK rejects fused shared experts under the cutlass runner). Throughput and DSpark acceptance match the FP4 cell within noise. +- **Flash Vision** — images are served natively on the b12x recipe with the same flags as Flash Official (send `image_url` content on `/v1/chat/completions`, see [Vision](#3-5-vision-image-inputs)); text-only requests work unchanged. Expect roughly 15–20% lower text throughput than Flash Official on this checkpoint — its bundled DSpark head accepts fewer drafts (~3.2 vs ~3.9) — with text accuracy intact. + + + +**DeepSeek-V4-Flash-Vision-Exp (Experimental)** + +[`deepseek-ai/DeepSeek-V4-Flash-Vision-Exp`](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp) is DeepSeek's first experimental multimodal V4 checkpoint: the 0731 Flash base plus a vision encoder and aligner, served through the same `sglang serve` flow with OpenAI-style `image_url` inputs (see [Vision](#3-5-vision-image-inputs) below). Select the **Flash Vision** variant in the Deploy panel for its recipes. + +- **Preview build required** — support lands via [sgl-project/sglang#37253](https://github.com/sgl-project/sglang/pull/37253) and has not shipped in a release. Docker mode on the Flash Vision cells already emits the preview image `lmsysorg/sglang:dev-dsv4-flash-vision`; for a Python environment, install SGLang from that PR's branch. The DGX Spark Flash Vision cell is the exception: it uses the DGX Spark image `lmsysorg/sglang:dev-v4f-2dgx-v2` (see the [DGX Spark notes](#spark-note)). +- **Verified matrix** — MMMU-Pro via sgl-eval at `temperature 1.0`, `top-p 0.95`, `--reasoning-effort max`. +- **Engine auto-configuration** — the engine picks the `flashinfer_mxfp4` MoE runner and auto-disables shared-experts fusion for this checkpoint (its HashTopK routing rejects fused shared experts); don't pass `--enforce-shared-experts-fusion`. +- **Chunked prefill & radix cache stay enabled** — the scheduler keeps image spans consistent automatically: chunked-prefill truncation points are span-aligned (an image span always prefills within a single extend, overshooting the chunk budget by at most one span), and a radix-cache prefix match ending deep inside an image span is re-issued from the span start. +- **Speculative decoding** — the checkpoint bundles a DSpark head, and the low-latency recipes enable it with `--speculative-algorithm DSPARK` (verified with image batches on B200 via the MMMU-Pro round; the other hardware rows are pending verification). The balanced and high-throughput recipes run target-only: they use DP Attention, which DSpark is incompatible with on current releases. As on the 0731/0813 checkpoints, do not pass the EAGLE flags. + **Shared experts fusion (Blackwell, flashinfer_mxfp4)** On the Blackwell fp4 recipes (`--moe-runner-backend flashinfer_mxfp4`), the shared expert runs as a separate FP8 MLP on an alternate stream by default. Adding: @@ -215,7 +248,7 @@ On the Blackwell fp4 recipes (`--moe-runner-backend flashinfer_mxfp4`), the shar routes it as one extra MXFP4 expert through the same trtllm-gen MoE kernel, so the whole MoE runs on a single stream (~4 fewer kernel launches and 2 fewer stream syncs per MoE layer). The shared expert is requantized from FP8 to MXFP4 at load time. Measured on GB200 tp4: gsm8k and AIME25 accuracy on par with the unfused baseline; Mean TTFT -13% to -21% and P99 ITL -15% to -53% at QPS 1-8 with neutral throughput. -Only for deployments without expert parallelism (e.g. the single-node low-latency recipes): with `moe_ep_size > 1` the flag is rejected at startup, unless the DeepEP/MegaMOE per-rank shared-slot path is in use. +Only for deployments without expert parallelism (e.g. the single-node low-latency recipes): with `moe_ep_size > 1` the flag is rejected at startup, unless the DeepEP/MegaMOE per-rank shared-slot path is in use. Not applicable to [Flash Vision (Exp)](#vision-note) — the engine auto-disables the fusion on that checkpoint. **Compressed attention state dtype** @@ -273,7 +306,9 @@ fused shared expert slot. **FP4 Indexer (Experimental)** -DeepSeek-V4 uses the default indexer path unless `--enable-deepseek-v4-fp4-indexer` is set. Enable this flag to use the experimental FP4 C4 indexer on SM100 GPUs with DeepGEMM FP4 indexer support. This path is intended for decode-heavy long-context workloads where reducing indexer cache bandwidth is beneficial. +DeepSeek-V4 uses the default indexer path unless `--enable-deepseek-v4-fp4-indexer` is set. Enable this flag to use the experimental FP4 C4 indexer. This path is intended for decode-heavy long-context workloads where reducing indexer cache bandwidth is beneficial. + +On **NVIDIA (SM100 / SM120)**, pair the flag with DeepGEMM FP4 indexer support and the FlashInfer MXFP4 MoE runner: ```bash Command # Please use the latest main branch for this feature. @@ -284,12 +319,27 @@ sglang serve \ --enable-deepseek-v4-fp4-indexer ``` +On **AMD MI355X (gfx95)**, the AITER FP4 indexer kernels are available on ROCm. Keep the standard MI355 recipe and add only the indexer flag. Do not need to pass `--moe-runner-backend flashinfer_mxfp4`: + +```bash Command +# Please use the latest ROCm image for this feature. +sglang serve \ + --model-path deepseek-ai/DeepSeek-V4-Pro \ + --tp 8 \ + --attention-backend dsv4 \ + --enable-deepseek-v4-fp4-indexer \ + +``` + **NVFP4 Hybrid Checkpoints** The [`nvidia/DeepSeek-V4-Pro-NVFP4`](https://huggingface.co/nvidia/DeepSeek-V4-Pro-NVFP4) and [`nvidia/DeepSeek-V4-Flash-NVFP4`](https://huggingface.co/nvidia/DeepSeek-V4-Flash-NVFP4) checkpoints quantize MoE experts to **NVFP4** while keeping attention and dense layers in -**FP8**. It requires `--moe-runner-backend flashinfer_trtllm_routed` which will be automatically selected if not provided. +**FP8**. The official releases have matching NVFP4 checkpoints at +[`nvidia/DeepSeek-V4-Flash-0731-NVFP4`](https://huggingface.co/nvidia/DeepSeek-V4-Flash-0731-NVFP4) and +[`nvidia/DeepSeek-V4-Pro-0813-NVFP4`](https://huggingface.co/nvidia/DeepSeek-V4-Pro-0813-NVFP4). +All of them require `--moe-runner-backend flashinfer_trtllm_routed` which will be automatically selected if not provided. ```bash Command sglang serve \ @@ -309,13 +359,18 @@ Requires Blackwell (SM100+). The MTP layer in this checkpoint stays MXFP4-packed and is routed through the `Mxfp4FlashinferTrtllmMoEMethod` path automatically. +The official (0731 / 0813) NVFP4 checkpoints preserve the bundled DSpark draft +head, so their low-latency recipes use `--speculative-algorithm DSPARK` instead +of the EAGLE/MTP shape flags — same as the corresponding original-precision +official checkpoints. + **Hopper (H100 / H200) note** Two options are available for running DeepSeek-V4 on Hopper: -- **Original FP4 checkpoints** — apply the W4A16 MoE kernels (Marlin) as the command generator picks for Hopper cells. This path works on both H100 and H200 and is the only option for H100 (no FP8 path). It is TP-only; on H200 the Pro variant fits on a single 8-GPU node, while H100 Pro needs 2 nodes (TP=16). +- **Original FP4 checkpoints** — run the MoE experts with W4A16 kernels (Marlin or the FlashInfer SM90 CUTLASS runner) as the command generator picks for Hopper cells. With FlashInfer >= 0.6.18 you can instead select the **W4A8** path — MXFP4 weights with FP8 activations via FlashInfer's Humming kernels — by adding `--flashinfer-mxfp4-moe-precision fp8` to `--moe-runner-backend flashinfer_mxfp4`; the low-latency Hopper cells now generate this form. Both work on H100 and H200; FP4 is the only option for H100 (no FP8 path). It is TP-only; on H200 the Pro variant fits on a single 8-GPU node, while H100 Pro needs 2 nodes (TP=16). - **Converted FP8 checkpoints** (H100 and H200 only) — pre-repackaged FP8 weights at [`sgl-project/DeepSeek-V4-Flash-FP8`](https://huggingface.co/sgl-project/DeepSeek-V4-Flash-FP8) and [`sgl-project/DeepSeek-V4-Pro-FP8`](https://huggingface.co/sgl-project/DeepSeek-V4-Pro-FP8) unlock DP-attention + DeepEP and richer parallelism (e.g. Pro TP=16 across 2 nodes). On these FP8 checkpoints you can additionally enable the all-FP8 **MegaMoE** path on SM90 for higher long-context / large-decode throughput — see the **SM90 (Hopper) FP8 MegaMoE** note in Configuration Tips below. @@ -335,9 +390,11 @@ HiCache and MegaMoE are **not** supported on RTX PRO 6000. - **Model checkpoints** — for correct accuracy, the FP4 model uses the stock `deepseek-ai/DeepSeek-V4-{Flash,Pro}`, and the FP8 model uses the repackaged `sgl-project/DeepSeek-V4-{Flash,Pro}-FP8`. - **Supported models** — **MI300X** supports DeepSeek-V4-Flash in FP8; **MI355X** supports DeepSeek-V4-Flash / Pro in both FP4 and FP8. All recipes run single-node. -- **TP / DP setting** — both TP=4 and TP=8 are supported. At low concurrency we recommend **TP-only**; at high concurrency use **TP + DP**, which additionally needs `--dp 8 --enable-dp-attention --enable-prefill-delayer --prefill-delayer-max-delay-ms 5000`. -- **MTP** — speculative decoding is supported; add `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`. +- **TP / DP setting (MI355X)** — both TP=4 and TP=8 are supported. At low concurrency we recommend **TP-only**; at high concurrency use **TP + DP** (balanced / high-throughput recipes). The verified MI355X DP recipes additionally set `--dp 8 --enable-dp-attention --enable-dp-attention-local-control-broadcast --tokenizer-worker-num 8 --stream-interval 20 --prefill-decode-interval 10`. +- **MTP** — speculative decoding on the original Flash / Pro checkpoints; add `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`. The 0813 Pro Official checkpoint keeps its MTP head too, so the same flags work there as a fallback to DSpark. +- **DSpark (MI355X Pro Official 0813)** — the 0813 checkpoint bundles the DSpark draft head. Prefer it over EAGLE on 0813 for ordinary serving: enable `--speculative-algorithm DSPARK` (low-latency) and see the [MI355X agentic recipe](#3-7-agentic-long-context-with-hicache-dram-offload-mi355x-fp4-dspark) for the long-context path. DSpark also runs under **PD disaggregation** — see the Pro Official pairs in [§3.8](#3-8-pd-disaggregation-on-mi355x-mori-io). - **Kernels** — uses the Unified KV attention and the flydsl MoE. +- **FP4 indexer (MI355X)** — FP4 C4 indexer is supported via `--enable-deepseek-v4-fp4-indexer` on top of the standard ROCm recipe. **MoRI EP (AMD expert parallelism)** @@ -597,12 +654,14 @@ For AMD devices, The Write policy knob defaults to `write_through` (the upstream default); switch to `write_back` / `write_through_selective` to trade durability for write speed when the storage tier is slow. -For more details, see the [HiCache documentation](../../../docs/advanced_features/hicache). +For more details, see the [HiCache documentation](../../../docs/advanced_features/hicache). For the alternative that skips the host tier entirely, see [§3.9 UMBP](#3-9-umbp-direct-external-kv-store) — it is a sibling of HiCache, not a storage backend for it, and the two cannot be enabled together. ### 3.4 DSpark (Speculative Decoding) Flash Official (0731) and Pro Official (0813) bundle a DSpark draft head in `deepseek-ai/DeepSeek-V4-Flash-0731` and `deepseek-ai/DeepSeek-V4-Pro-0813`. The target and draft weights therefore come from the same checkpoint: enable DSpark with `--speculative-algorithm DSPARK` and do not set a separate `--speculative-draft-model-path`. +The experimental [Flash Vision checkpoint](#vision-note) also bundles a DSpark head, enabled the same way: the Flash Vision low-latency recipes ship with `--speculative-algorithm DSPARK` (verified with image batches on B200 via the MMMU-Pro round; other hardware rows pending). The balanced and high-throughput Flash Vision recipes stay target-only because they run DP Attention. + Unlike the EAGLE recipes for the original Flash and Pro checkpoints, this recipe omits `--speculative-num-steps`, `--speculative-eagle-topk`, and `--speculative-num-draft-tokens`. SGLang reads the DSpark shape from the checkpoint. @@ -633,4 +692,370 @@ Larger blocks can improve decode latency when acceptance stays high, but they al For every candidate, compare with the same recipe without `--speculative-algorithm DSPARK`. Restart the server between the DSpark and non-speculative legs, keep the request corpus, sampling, concurrency, and warmup identical, and give each `bench_serving` leg its own `--flush-cache`. Leave `--speculative-draft-attention-backend` unset unless a separate profiling run justifies an override. -DSpark currently requires CUDA, `pp_size == 1`, and DP Attention disabled. It is not compatible with PD disaggregation on current SGLang releases; selecting a prefill or decode role in the Playground automatically removes the inherited DSpark flags. The DP-Attention and MI355X Flash Official recipes therefore run target-only. If a larger draft block or concurrency causes graph-capture OOM, lower `--mem-fraction-static`, the draft block size, or the configured maximum running requests, then rerun both performance and accuracy gates. +DSpark requires `pp_size == 1`. Selecting a prefill or decode role in the Playground keeps the inherited DSpark flags on both roles; for PD-disaggregated configurations that run DSpark, see the [B200 agentic recipe](#3-6-agentic-long-context-with-hicache-dram-offload-b200-fp4-dspark) and the [MI355X Pro Official PD pairs](#3-8-pd-disaggregation-on-mi355x-mori-io). The MI355X Flash Official recipes run target-only, and so do the DP-Attention recipes in the Deploy panel — for DP-Attention configurations that do run DSpark, see the [B200 agentic recipe](#3-6-agentic-long-context-with-hicache-dram-offload-b200-fp4-dspark) and the [MI355X agentic recipe](#3-7-agentic-long-context-with-hicache-dram-offload-mi355x-fp4-dspark). If a larger draft block or concurrency causes graph-capture OOM, lower `--mem-fraction-static`, the draft block size, or the configured maximum running requests, then rerun both performance and accuracy gates. + +### 3.5 Vision (Image Inputs) + +The experimental [`DeepSeek-V4-Flash-Vision-Exp`](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp) checkpoint (the **Flash Vision** variant in the Deploy panel — see the [configuration notes](#vision-note)) takes images via the OpenAI-compatible `image_url` content type, as public URLs or base64 `data:` URIs; text and images mix freely in one message. Vision input works with the same server the Deploy panel produces — no extra model-specific flags needed. + + + +```python Example +from openai import OpenAI + +client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") + +response = client.chat.completions.create( + model="deepseek-ai/DeepSeek-V4-Flash-Vision-Exp", + messages=[{ + "role": "user", + "content": [ + {"type": "image_url", + "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"}}, + {"type": "text", "text": "Describe this image in a few sentences."}, + ], + }], +) + +message = response.choices[0].message +if getattr(message, "reasoning_content", None): + print("=============== Thinking =================") + print(message.reasoning_content) +print("=============== Content =================") +print(message.content) +``` + + + + + +```text Output +Pending update... +``` + + + +### 3.6 Agentic Long-Context with HiCache DRAM Offload (B200 FP4, DSpark) + +**TP8, concurrency 1–8:** +```bash Command +python3 -m sglang.launch_server \ + --model-path deepseek-ai/DeepSeek-V4-Pro-0813 \ + --trust-remote-code \ + --tp 8 \ + --moe-runner-backend flashinfer_mxfp4 \ + --enable-deepseek-v4-fp4-indexer \ + --disable-flashinfer-autotune \ + --mem-fraction-static 0.90 \ + --swa-full-tokens-ratio 0.1 \ + --chunked-prefill-size 8192 \ + --tool-call-parser deepseekv4 \ + --reasoning-parser deepseek-v4 \ + --speculative-algorithm DSPARK \ + --speculative-dspark-block-size 6 \ + --enable-hierarchical-cache \ + --hicache-ratio 2.75 \ + --hicache-write-policy write_through \ + --hicache-io-backend direct \ + --hicache-mem-layout page_first_direct +``` + +DSv4 HiCache sizes the host tier with `--hicache-ratio` (host/device token ratio), not `--hicache-size`. + +**DEP8 (DP Attention), concurrency 64–160:** +```bash Command +SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=8320 \ +python3 -m sglang.launch_server \ + --model-path deepseek-ai/DeepSeek-V4-Pro-0813 \ + --trust-remote-code \ + --tp 8 \ + --dp 8 \ + --enable-dp-attention \ + --enable-dp-lm-head \ + --ep-size 8 \ + --moe-a2a-backend megamoe \ + --enable-w4a4-mxfp4-megamoe \ + --enable-deepseek-v4-fp4-indexer \ + --disable-shared-experts-fusion \ + --disable-flashinfer-autotune \ + --mem-fraction-static 0.88 \ + --swa-full-tokens-ratio 0.02 \ + --chunked-prefill-size 49152 \ + --tool-call-parser deepseekv4 \ + --reasoning-parser deepseek-v4 \ + --speculative-algorithm DSPARK \ + --speculative-dspark-block-size 6 \ + --enable-hierarchical-cache \ + --hicache-ratio 8 \ + --hicache-write-policy write_through \ + --hicache-io-backend direct \ + --hicache-mem-layout page_first_direct +``` + +`--chunked-prefill-size` is a global budget divided by `--dp`, so this keeps 6144 tokens per rank. + +**Disaggregated 1P1D (DEP8 prefill / DEP8 decode), concurrency 64–128.** Prefill, on one 8-GPU node: +```bash Command +SGLANG_DSV4_MHC_PREWARM=1 \ +SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=9216 \ +NCCL_MNNVL_ENABLE=1 \ +NCCL_CUMEM_ENABLE=1 \ +python3 -m sglang.launch_server \ + --model-path deepseek-ai/DeepSeek-V4-Pro-0813 \ + --trust-remote-code \ + --tp 8 \ + --dp 8 \ + --enable-dp-attention \ + --enable-dp-lm-head \ + --ep-size 8 \ + --moe-dense-tp-size 1 \ + --moe-a2a-backend megamoe \ + --enable-w4a4-mxfp4-megamoe \ + --enable-deepseek-v4-fp4-indexer \ + --disable-flashinfer-autotune \ + --mem-fraction-static 0.85 \ + --page-size 256 \ + --swa-full-tokens-ratio 0.01 \ + --chunked-prefill-size 65536 \ + --tool-call-parser deepseekv4 \ + --reasoning-parser deepseek-v4 \ + --speculative-algorithm DSPARK \ + --speculative-dspark-block-size 6 \ + --enable-hierarchical-cache \ + --hicache-ratio 8 \ + --hicache-write-policy write_back \ + --hicache-io-backend direct \ + --hicache-mem-layout page_first_direct \ + --disaggregation-mode prefill \ + --disaggregation-transfer-backend mooncake \ + --disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_10,mlx5_11 +``` + +Decode, on a second 8-GPU node: +```bash Command +SGLANG_DSV4_MHC_PREWARM=1 \ +SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=4096 \ +NCCL_MNNVL_ENABLE=1 \ +NCCL_CUMEM_ENABLE=1 \ +python3 -m sglang.launch_server \ + --model-path deepseek-ai/DeepSeek-V4-Pro-0813 \ + --trust-remote-code \ + --tp 8 \ + --dp 8 \ + --enable-dp-attention \ + --enable-dp-lm-head \ + --ep-size 8 \ + --moe-dense-tp-size 1 \ + --moe-a2a-backend megamoe \ + --enable-w4a4-mxfp4-megamoe \ + --enable-deepseek-v4-fp4-indexer \ + --disable-flashinfer-autotune \ + --mem-fraction-static 0.90 \ + --page-size 256 \ + --swa-full-tokens-ratio 0.02 \ + --tool-call-parser deepseekv4 \ + --reasoning-parser deepseek-v4 \ + --speculative-algorithm DSPARK \ + --speculative-dspark-block-size 6 \ + --disaggregation-mode decode \ + --disaggregation-transfer-backend mooncake \ + --disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_10,mlx5_11 +``` + +Router: +```bash Command +python3 -m sglang_router.launch_router \ + --pd-disaggregation \ + --prefill http://:30000 8998 \ + --decode http://:30000 \ + --host 0.0.0.0 --port 8000 +``` + +HiCache runs on the prefill role only, with `write_back` rather than the aggregate recipe's `write_through`. Set `--disaggregation-ib-device` to the node's own external HCAs. + +Concurrency 256 uses 2P1D — two prefill workers, one node each, added with a second `--prefill` on the router — with `--swa-full-tokens-ratio 0.02` on prefill and `--mem-fraction-static 0.91 --swa-full-tokens-ratio 0.005` on decode. + +### 3.7 Agentic Long-Context with HiCache DRAM Offload (MI355X FP4, DSpark) + +DeepSeek-V4-Pro-0813 bundles the DSpark head, so `--speculative-draft-model-path` is not needed. `--speculative-dspark-block-size 6` is the AL-optimal draft length on the committed golden curve (verify window 7). + +**TP8, concurrency 1–16:** +```bash Command +SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS=1 \ +SGLANG_USE_ROCM700A=0 \ +TORCH_BLAS_PREFER_HIPBLASLT=1 \ +SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton \ +AITER_BF16_FP8_MOE_BOUND=0 \ +SGLANG_OPT_USE_AITER_BATCHED_GEMM=1 \ +python3 -m sglang.launch_server \ + --model-path deepseek-ai/DeepSeek-V4-Pro-0813 \ + --trust-remote-code \ + --tp 8 \ + --prefill-decode-interval 20 \ + --attention-backend dsv4 \ + --enable-deepseek-v4-fp4-indexer \ + --page-size 256 \ + --swa-full-tokens-ratio 0.1 \ + --kv-cache-dtype fp8_e4m3 \ + --enforce-shared-experts-fusion \ + --mem-fraction-static 0.86 \ + --chunked-prefill-size 16384 \ + --tool-call-parser deepseekv4 \ + --reasoning-parser deepseek-v4 \ + --speculative-algorithm DSPARK \ + --speculative-dspark-block-size 6 \ + --watchdog-timeout 3600 +``` + +Concurrency 32–48 keeps the same TP-only command and adds HiCache DRAM offload. DSv4 HiCache sizes the host tier with `--hicache-ratio` (host/device token ratio). Ratio `1.5` is the value that stays under a ~2.7 TB host DRAM budget at TP8: + +```bash Command + --enable-hierarchical-cache \ + --hicache-ratio 1.5 \ + --hicache-write-policy write_through \ + --hicache-io-backend direct \ + --hicache-mem-layout page_first_direct +``` + +**DP8 (DP Attention), concurrency 128–256:** +```bash Command +SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS=1 \ +SGLANG_USE_ROCM700A=0 \ +TORCH_BLAS_PREFER_HIPBLASLT=1 \ +SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton \ +AITER_BF16_FP8_MOE_BOUND=0 \ +SGLANG_OPT_USE_AITER_BATCHED_GEMM=1 \ +SGLANG_SHARED_EXPERT_TP1=1 \ +SGLANG_DP_SHARED_EXPERT_LOCAL=1 \ +SGLANG_DP_USE_GATHERV=1 \ +SGLANG_DP_USE_REDUCE_SCATTER=1 \ +python3 -m sglang.launch_server \ + --model-path deepseek-ai/DeepSeek-V4-Pro-0813 \ + --trust-remote-code \ + --tp 8 \ + --dp 8 \ + --enable-dp-attention \ + --enable-dp-lm-head \ + --enable-prefill-delayer \ + --enable-dp-attention-local-control-broadcast \ + --tokenizer-worker-num 8 \ + --stream-interval 20 \ + --prefill-decode-interval 20 \ + --prefill-delayer-token-usage-low-watermark 0.7 \ + --attention-backend dsv4 \ + --enable-deepseek-v4-fp4-indexer \ + --page-size 256 \ + --swa-full-tokens-ratio 0.1 \ + --kv-cache-dtype fp8_e4m3 \ + --enforce-shared-experts-fusion \ + --mem-fraction-static 0.92 \ + --chunked-prefill-size 65536 \ + --tool-call-parser deepseekv4 \ + --reasoning-parser deepseek-v4 \ + --speculative-algorithm DSPARK \ + --speculative-dspark-block-size 6 \ + --enable-hierarchical-cache \ + --hicache-ratio 1.5 \ + --hicache-write-policy write_through \ + --hicache-io-backend direct \ + --hicache-mem-layout page_first_direct \ + --watchdog-timeout 3600 +``` + +`--chunked-prefill-size` is a global budget divided by `--dp`, so this keeps 8192 tokens per rank. `--enable-dp-lm-head` is required for DSpark under DP attention. Front the DP ranks with `sglang_router --policy cache_aware` so multi-turn sessions land on the rank holding the longest radix/HiCache prefix; above concurrency 160 add `--balance-abs-threshold 32`. + +### 3.8 PD Disaggregation on MI355X (MORI-IO) + +On ROCm the prefill and decode roles talk over **MORI-IO** rather than Mooncake or NiXL, so the **PD Disagg** card in the [Playground above](#playground) hides the CUDA-only transports once MI300X / MI355X is selected and defaults the IB device list to the node's `rdmaN` NICs instead of ConnectX `mlx5_*` names. + +Both roles need the RDMA NICs passed into the container, so the **Docker** output adds the fabric flags below on top of the usual ROCm device access from the [Install panel](#install). `/dev/infiniband` covers `rdma_cm` and the per-NIC `uverbs0`…`uverbs7` nodes, and the memlock/`IPC_LOCK` pair is what lets MORI-IO pin the buffers it registers with the NICs: + +```bash Command +docker run \ + --device=/dev/kfd --device=/dev/dri --device /dev/infiniband \ + --group-add video --cap-add IPC_LOCK \ + --cap-add=SYS_PTRACE --security-opt seccomp=unconfined \ + --ulimit memlock=-1 --ulimit stack=67108864 \ + --ulimit nofile=1048576:1048576 \ + --network host --ipc=host --shm-size 32g \ + -v ~/.cache/huggingface:/root/.cache/huggingface \ + lmsysorg/sglang-rocm:v0.5.20-rocm720-mi35x-20260923 \ + python3 -m sglang.launch_server +``` + +The Pro Official (0813) PD pairs ran end-to-end on this `v0.5.20` build, and the Playground's Docker output uses it for that variant. `--network host` replaces the single-node `-p` mapping because the two roles rendezvous across hosts. Long agentic runs on the 1.6T checkpoint may need more than `--shm-size 32g`; raise it if the workers die during weight load. + +Selecting either role with **MORI** on MI355X emits the shared MORI-IO transport env — `SGLANG_MORI_COMBINE_DTYPE` plus the RDMA send-queue tuning `MORI_IO_SQ_BACKOFF_TIMEOUT_US` / `MORI_IO_QP_MAX_SEND_WR` — together with that role's sizing. `--max-running-requests` is deliberately small on both: the pair streams KV between workers rather than holding a large running batch. + +The roles are sized per **Strategy**, so the Deploy panel's operating point carries through to the PD command: + +- **Low-Latency** — TP-only, `--mem-fraction-static 0.86`, and a running-request ceiling of 8. Decode captures graphs for batches 1–8. No offload tier: the KV pool is all there is. + - **Pro Official (0813)** has its own low-latency pair: a **TP4 prefill** in front of a **TP8 decode**, both keeping the bundled **DSpark** head with `--speculative-dspark-block-size 6` (SGLang derives steps 1 / topk 1 / 7 draft tokens from it). The ceiling is `--max-running-requests 32` on both roles and decode ladders 1–32. Prefill adds a 16384 chunked-prefill budget, `--optimistic-prefill-attempts 2`, `--enable-cache-report`, and a **UMBP** tier (see [§3.9](#3-9-umbp-direct-external-kv-store)) that the Playground turns on by default. Both roles drop the base cell's `--prefill-decode-interval`, which only applies to aggregated serving. +- **Balanced** — still TP-only, with the ceiling at `--max-running-requests 96` and a 16384 chunked-prefill budget. Decode ladders all the way to 96, since without DP the server-wide ceiling *is* the per-rank batch. Prefill runs a **HiCache** host tier (`--hicache-ratio 2.5`, `page_first`, write-through, best-effort prefetch); decode does not, because HiCache is not recommended on the decode role with MORI. + - **Pro Official (0813)** runs balanced as the same asymmetric pair as its low-latency one, re-sized: a **TP4 prefill** in front of a **TP8 decode**, both with **DSpark** at `--speculative-dspark-block-size 3` (steps 1 / topk 1 / 4 draft tokens), a ceiling of `--max-running-requests 96` on both roles, and a decode ladder of 1–96. The balanced base cell is target-only DP, so the roles add `--speculative-algorithm DSPARK` back. Prefill keeps the 16384 chunked-prefill budget and adds `--optimistic-prefill-attempts 2` and `--enable-cache-report`. Its offload tier is **UMBP** rather than HiCache, on by default in the Playground; switch the UMBP card off to fall back to the HiCache tier above. +- **High-Throughput** — inherits TP8 / DP8 and the 65536 chunked-prefill budget from the base cell, raises the ceiling to `--max-running-requests 256` and `--mem-fraction-static 0.92`, and ladders decode to 32 — the per-rank batch 256 leaves across DP 8. It adds `--enable-cache-report`, which surfaces the prefix hit rate you need to judge whether an offload tier is paying for itself at that concurrency. + - **Pro Official (0813)** keeps TP8 / DP8 on both roles and adds **DSpark** at `--speculative-dspark-block-size 3`, the same draft shape as its balanced pair, plus `--enable-dp-lm-head`. The ceiling is `--max-running-requests 512` on both roles at `--mem-fraction-static 0.92`, which is 64 per DP rank, so decode ladders 1–64. Prefill keeps the base cell's 65536 chunked-prefill budget and `--swa-full-tokens-ratio 0.15`, adds `--optimistic-prefill-attempts 2`, and runs **UMBP** on by default in the Playground against the same standalone tier server as the other two Pro Official pairs. Decode drops `--chunked-prefill-size`, and both roles drop `--prefill-decode-interval`. + +The offload tier is what actually separates balanced from high-throughput: balanced pairs prefill with HiCache's tiered GPU → host cache, while high-throughput pairs it with UMBP, which links the radix tree straight to MORI with no host tier. Pro Official is the exception, running UMBP under all three of its pairs, including the two TP-only ones. HiCache and UMBP cannot both be on, and the Playground enforces that. + + +The low-latency and balanced roles are TP-only, and that is exactly why their decode ladders run to the full ceiling. Selecting either role forces DP-Attention off and grays the control: `--max-running-requests` is server-wide and floor-divided by `attn_dp_size`, so switching DP on would cut the per-rank batch below the captured graphs without changing either flag in the command. With PD off, the balanced cell stays a DP recipe for aggregated serving. + + + +**Speculative decoding changes how you size the ceilings.** For a target concurrency of N, set `--max-running-requests N*2` on **both** roles. Sizing them at N caps the served batch below the concurrency you are aiming for. This applies to DSpark and EAGLE/MTP alike: the Pro Official low-latency pair's ceiling of 32 is sized for a concurrency of 16, its balanced pair's 96 for 48, and its high-throughput pair's 512 for 256. + +`--max-running-requests` is server-wide and floor-divided by `attn_dp_size`, so the decode graph ladder should cover the **per-rank** batch that leaves — `N*2 / dp_size`. That is why the two points above ladder to different places from ceilings that differ by less: balanced is TP-only, so 96 slots are 96 per rank, while high-throughput spreads 256 over DP 8 and only ever sees 32 at a time (Pro Official's 512 over DP 8 is 64). + + +The high-throughput prefill role is where [UMBP](#3-9-umbp-direct-external-kv-store) belongs — DP attention is already on, which UMBP otherwise requires, and the concurrency is high enough for the pool to matter. On Pro Official, all three prefill roles enable UMBP by default, and the low-latency and balanced ones do so without DP as the TP-only exceptions. Elsewhere, turn on the **UMBP** card alongside the role; it is a separate card rather than part of the role because the linker flags are its to own, and leaving them implicit would hide the dependency. + +Whichever point you pick, the two roles diverge in two places: + +- **Cuda graphs.** Prefill runs eager (`--disable-cuda-graph`) because the FP4 indexer's prefill path is not captured. Decode captures the small batch ladder that matches its running-request ceiling (`--cuda-graph-bs-decode 1 2 3 4 5 6 7 8`). +- **MORI dispatch budget.** `SGLANG_MORI_NUM_MAX_DISPATCH_TOKENS_PER_RANK` is `16384` on prefill, sized for a whole chunked-prefill batch, and `128` on decode, where a step never dispatches more than a token per running request. + +Picking a role keeps an inherited DSpark head on both roles (see [§3.4](#3-4-dspark-speculative-decoding)); keep `--speculative-dspark-block-size` the same on both roles. For the originals, which have no DSpark head, a speculative PD recipe uses the bundled MTP head through the EAGLE path instead: + +```bash Command +--speculative-algorithm EAGLE --speculative-eagle-topk 1 \ +--speculative-num-steps 3 --speculative-num-draft-tokens 4 +``` + +Run the two roles on separate nodes with the same transport and IB device selection, then front them with the router the card prints. On MI355X that router is cache-aware rather than round-robin: `--policy consistent_hashing` keeps a conversation on the worker that already holds its prefix, which is what makes the agentic reuse pay off, and `--balance-abs-threshold 2` / `--balance-rel-threshold 1.1` keep that affinity from starving the peer given how small the running-request ceiling is. Decode holds no reusable prefix, so `--decode-policy round_robin` spreads the decode side evenly. Health checks stay on with a long timeout so a wedged worker is still caught without a multi-minute prefill being mistaken for a failure. + +`--dp-aware` only does something when the workers run DP attention — turn on the Playground's **DP-Attention** card if you want it to take effect, otherwise it is inert on the TP-only recipe above. + +### 3.9 UMBP (Direct External KV Store) + +UMBP is the second way to spill KV off the GPU, and it is an **alternative to [HiCache](#3-3-hicache-hierarchical-kv-caching), not a tier inside it**. HiCache is a hierarchy — GPU, then pinned host memory, then optionally a storage backend. UMBP links the unified radix tree straight to MORI's buffer pool with no host cache tier in between, so pages load and offload against the external store directly. SGLang rejects the two together, so enabling the **UMBP** card in the [Playground above](#playground) strips the whole HiCache flag family from the command: + +```bash Command +--enable-unified-cache-external-linker \ +--unified-cache-external-linker-backend mori +``` + +The **Store** knob picks the linker backend. `mori` is the UMBP pool; `mooncake` drives the same direct-linker path against a Mooncake store instead. + +**DP Attention is required**, and the card's Enable chip stays greyed out until you turn it on. This is not a soft preference. The linker keys its objects by rank, and MLA KV is replicated across TP, so a TP8 prefill worker under pure TP opens eight separate keyspaces holding eight copies of the same tokens — the pool's effective distinct-token capacity would be an eighth of what its byte budget suggests, which is easy to mistake for a real measurement. DP attention collapses the keys onto one shared keyspace. + +The exception is the MI355X Pro Official prefill role in the low-latency and balanced pairs of [§3.8](#3-8-pd-disaggregation-on-mi355x-mori-io). At TP4 it holds four copies rather than eight, and that TP-only shape has run end-to-end, so the card turns UMBP on for those roles without DP. Every Pro Official prefill role (low-latency, balanced, and the DP8 high-throughput one) uses a **standalone** UMBP tier: a separate `umbp_standalone_server` process on the prefill node owns the DRAM pool, and the prefill server attaches to it over a Unix socket. Start the tier server first; the 1500 GB DRAM budget below is what the verified runs used: + +```bash Command +mkdir -p /tmp/umbp_sa +UMBP_DRAM_CAPACITY=1500000000000 \ +UMBP_DRAM_USE_HUGEPAGES=1 \ +UMBP_SSD_ENABLED=0 \ +MORI_UMBP_LOG_LEVEL=info \ +/umbp_standalone_server unix:///tmp/umbp_sa/sa.grpc.sock +``` + +Then launch the prefill role with the socket in its environment. The Playground adds both lines: the address has to come from the environment, and the extra-config gives the tier server up to two minutes to bind the socket before prefill gives up. + +```bash Command +UMBP_STANDALONE_ADDRESS=unix:///tmp/umbp_sa/sa.grpc.sock \ +sglang serve ... \ + --hicache-storage-backend-extra-config '{"standalone_startup_timeout_ms":120000}' +``` + +Two further limits are worth knowing before you size a run. The pool is a per-node process, so a prefill worker that spans nodes would shard its keyspace by node. And only the prefill worker offloads KV — the decode side is untouched. diff --git a/docs/cookbook/autoregressive/DeepSeek/DeepSeek-V4_1.mdx b/docs/cookbook/autoregressive/DeepSeek/DeepSeek-V4_1.mdx new file mode 100644 index 000000000000..26b1a53dccdf --- /dev/null +++ b/docs/cookbook/autoregressive/DeepSeek/DeepSeek-V4_1.mdx @@ -0,0 +1,206 @@ +--- +title: DeepSeek-V4.1 +description: "Deploy DeepSeek-V4.1 Flash with SGLang — launch recipes, feature compatibility, and tuning notes for GB300, H200, B200, B300 and MI350X." +tag: NEW +--- + +## Deployment + + + + + +For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel. + + + + + +DeepSeek-V4.1 Flash support has **not shipped in an SGLang release yet**, so a stock `pip install sglang` cannot serve it. Use the preview Docker image below. + + + + + +For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). + +**NVIDIA GPUs** — every NVIDIA cell on this page uses the preview build `lmsysorg/sglang:dev-dsv41` (H200 / B200 / B300 / GB300); the command panel picks it automatically. + +```bash Command +docker pull lmsysorg/sglang:dev-dsv41 + +docker run --gpus all \ + --shm-size 32g \ + -p 30000:30000 \ + -v ~/.cache/huggingface:/root/.cache/huggingface \ + --env "HF_TOKEN=" \ + --ipc=host \ + lmsysorg/sglang:dev-dsv41 \ + sglang serve +``` + +**AMD GPUs (ROCm)** — ROCm support for this model is likewise a preview build, `lmsysorg/sglang:dev-dsv41-mi35x`; the command panel picks it automatically for the MI350X cell. + +```bash Command +docker pull lmsysorg/sglang:dev-dsv41-mi35x + +docker run \ + --device=/dev/kfd --device=/dev/dri \ + --group-add video \ + --cap-add=SYS_PTRACE --security-opt seccomp=unconfined \ + --shm-size 32g --ipc=host \ + -p 30000:30000 \ + -v ~/.cache/huggingface:/root/.cache/huggingface \ + --env "HF_TOKEN=" \ + lmsysorg/sglang:dev-dsv41-mi35x \ + sglang serve +``` + + + + + + + +Pick your hardware + recipe to generate the launch command: + +- **Low-Latency** — fastest reply for a single user. Pick for chat. +- **High-Throughput** — most tokens per second across many users. Best for batch jobs. + +import { Deployment } from "/src/snippets/_deployment.jsx"; +import { config } from "/src/snippets/configs/deepseek-ai/deepseek-v4_1.jsx"; + + + + +## Playground + +The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits combinations the SGLang team has signed off on; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. + +import { Playground } from "/src/snippets/_playground.jsx"; + + + + +## 1. Model Introduction + +**DeepSeek-V4.1 Flash** is a sparse-attention Mixture-of-Experts model (`model_type: deepseek_v4.1`) served through SGLang's `dsv4` backend: 40 decoder layers, 384 routed experts at top-6 plus one shared expert, fp8 dense weights at a 32-wide `ue8m0` block scale with fp4 routed experts. It also carries **DSpark**, its own three-stage speculative draft, which the Low-Latency recipe turns on. + +**Architecture.** Every layer keeps one 512-wide KV latent per position and uses it as both key and value for all 64 query heads, with no separate value projection. Each layer reads two stores that behave oppositely: **compressed latents**, produced only at source layers and shared forward, collapsing 2 positions into 1 in layers 2–19 and 1-to-1 from layer 20 on; and a **128-position sliding window**, recomputed per layer from that layer's own activations, so it can never be shared but also never grows with context. Consumers address a source's cache by plain `loc // ratio` arithmetic with no mapping table, so compressed state lives and dies with the full prefix for free. The residual stream is four parallel copies mixed by a per-token doubly stochastic matrix; each sublayer's mixing coefficients are consumed by the *next* sublayer, which lets that projection overlap the main GEMMs. + +**Sparse retrieval.** KV source layers and index source layers are different lists — four of the former, eight of the latter. The four extra index layers produce no keys at all; they re-score layer 20's keys with their own query, so retrieval decisions are made twice as often as keys are stored. Each retrieving layer picks a top-512 candidate set. + +**Engram.** An additive n-gram hash memory at two layers. Token ids are normalized *before* hashing, so `" The"`, `"the"` and `"THE"` cannot fork into separate rows. Its two fp8 tables are the largest single block of weight in the checkpoint, and by default they load row-sharded across the TP group, which costs an all-reduce per engram layer. `SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE=1` (**opt-in**) moves them to one shared host copy instead: both all-reduces disappear, the freed HBM goes to the KV pool, and output is bitwise unchanged — at the cost of host RAM, a longer load, and needing huge-page backing to keep the gather cheap. + +**SWA bounded replay.** Because the window store is per-layer and cheap to rebuild, it need not all be recomputed. `--enable-decoder-swa-bounded-replay` (**opt-in**) runs the early layers over the whole extend and the late layers over only each request's last 128 tokens, sharing late-layer KV from source layer 20. Prefill gets materially faster. It is validated on the decode path only, refuses prompt logprobs by design, is not numerically equivalent to full prefill, and is excluded at launch with the prefill CUDA graph and with DP attention. + +**Kernels.** At decode token counts these kernels are launch-bound rather than bandwidth-bound, so the work is fusion and overlap: the compressor projection, ratio-2 pooling, RoPE with fp4 quantization and indexer packing, paged fp4 indexer scoring with top-k resolved straight to cache addresses, and the mHC statistics and Sinkhorn — with compression, indexing and mHC overlapped against attention and FFN on dedicated streams. All of it is selected automatically; the environment switches it once sat behind were removed, so there is nothing to turn on. One caveat outranks the fusions: **output is not bitwise stable across batch composition today** — two default kernels are shape-guarded to a single token — and `--enable-deterministic-inference` is refused on this backend. + +**Recommended generation:** reasoning evaluations were run at `temperature=1.0`, `top_p=0.95` with reasoning effort `max` (informational — do not hardcode these in application code). + +**Resources:** [HuggingFace](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash). + +## 2. Configuration Tips + +### Do not override the backends + +`--attention-backend`, `--moe-runner-backend` and `--fp8-gemm-backend` are selected automatically from the model, hardware, forward mode and shape. On GB300 they resolve to `dsv4` / `flashinfer_mxfp4` / `flashinfer_cutedsl`. Confirm them in the startup log rather than passing them. + +Overriding them is the most common cause of a disappointing measurement: it leaves the 32-wide ue8m0 blocks on the Triton `_w8a8_block_fp8_matmul` fallback, which dominates the decode step and costs most of the model's bs=1 throughput. If your decode rate looks like a small fraction of what you expected, check the resolved backends first. + +## 3. Advanced Usage + +### 3.1 Reasoning + +Enable the reasoning parser — `--reasoning-parser auto` resolves to `deepseek-v41` (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) to separate thinking from the final answer. The parser puts the thinking block in `reasoning_content` and the answer in `content`; without it both arrive concatenated in `content`. + +The parser can only split a thinking block the model actually produced, and **thinking is off by default** (`SGLANG_DEFAULT_THINKING=false`). Sending `reasoning_effort` turns it on: any value other than `none` also switches `chat_template_kwargs.thinking` on for you. A request that carries neither comes back with an empty `reasoning_content` however the parser is configured. + + + +```python Example +from openai import OpenAI + +client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") +resp = client.chat.completions.create( + model="deepseek-ai/DeepSeek-V4.1-Flash", + messages=[{"role": "user", "content": "What is 15% of 240?"}], + reasoning_effort="high", +) +msg = resp.choices[0].message +print("Reasoning:", getattr(msg, "reasoning_content", None)) +print("Answer:", msg.content) +``` + + + +On the request, `reasoning_effort` accepts the tiers `low` (50), `high` (75), `xhigh` (75), and `max` (100), or a float in `[0.0, 0.99]` that maps onto the model's 1–100 budget. An integer budget is reachable only through `chat_template_kwargs.reasoning_effort`, and that route does not switch thinking on by itself — pair it with `chat_template_kwargs.thinking`. `none` turns thinking off. The remaining OpenAI tiers, which have no V4.1 counterpart (`minimal`, `medium`), log a warning and fall back to the server-side default rather than erroring; that default is `high` (75), and `SGLANG_DSV41_REASONING_EFFORT` overrides it. + +### 3.2 Tool Calling + +Enable the tool-call parser — `--tool-call-parser auto` resolves to `deepseekv41` (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`. DeepSeek-V4.1 uses spaced DSML tool tags, which the stock DeepSeek-V4 detector does not parse — the `deepseekv41` detector is required. + + + +```python Example +from openai import OpenAI + +client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") +tools = [{ + "type": "function", + "function": { + "name": "get_weather", + "description": "Get the current weather in a city", + "parameters": { + "type": "object", + "properties": {"location": {"type": "string"}}, + "required": ["location"], + }, + }, +}] + +resp = client.chat.completions.create( + model="deepseek-ai/DeepSeek-V4.1-Flash", + messages=[{"role": "user", "content": "What's the weather in Beijing?"}], + tools=tools, + reasoning_effort="high", +) +choice = resp.choices[0] +print("finish_reason:", choice.finish_reason) +print("tool_calls:", choice.message.tool_calls) +print("reasoning:", getattr(choice.message, "reasoning_content", None)) +``` + + + +A successful call returns `finish_reason: "tool_calls"` with a single `get_weather` entry whose arguments carry the requested location. If you see the raw DSML markup in `content` instead, the parser flag is missing. + +### 3.3 Speculative Decoding (DSpark) + +DSpark is DeepSeek-V4.1 Flash's own bundled draft — there is no EAGLE or MTP path for this model, and no `--speculative-num-steps` knob (the draft-token count is resolved from the checkpoint). Turn it on with `--speculative-algorithm DSPARK --speculative-dspark-block-size 5`, which is what the Low-Latency cells do. + +What to expect: + +- A DSpark step costs meaningfully more than a plain decode step, so the win at bs=1 is roughly the accept length divided by that fixed step cost. It is a real speed-up on interactive workloads, but not proportional to the number of draft tokens. +- Acceptance is strongly workload-dependent — highest on maths and code, lower on open-ended chat, lower still on agentic traces. Size your expectations from the workload you actually run, and measure it rather than assuming. +- The step cost does not move with accept length or with the draft block size, so tuning `--speculative-dspark-block-size` will not buy throughput. This is also why the High-Throughput recipe turns speculation off: at large batch the fixed step cost stops paying for itself. + +### 3.4 PD Disaggregation + +Prefill/decode disaggregation is validated token-identical against a single server across 32 greedy prompts, with GSM8K matching through the router. Use the **PD Disaggregation** card in the Playground to generate the prefill role, the decode role and the router command. + +One deployment note: Mooncake needs the RDMA fabric visible inside the container, so launch with `--device /dev/infiniband:/dev/infiniband --cap-add IPC_LOCK --ulimit memlock=-1`. Without it Mooncake selects its NVLink transport, which only serves buffers from its own allocator and fails with `Requested address ... not found`. If you hit that, force TCP with `MOONCAKE_PROTOCOL=tcp` and `MC_FORCE_TCP=1` — the Playground's Mooncake option sets both. + +PD and speculative decoding cannot be combined. + +### 3.5 HiCache (Hierarchical KV Caching) + +HiCache extends RadixAttention with a hierarchy of KV cache tiers, significantly expanding effective context capacity for long-context and multi-turn scenarios. + +To enable HiCache, open the **HiCache** card in the [Playground above](#playground) and flip **Enable**: the Playground emits `--enable-hierarchical-cache` on top of the recipe's flags, and cold KV pages spill to CPU pinned memory (L2, GPU + CPU) instead of being dropped from the GPU pool. The host pool is sized by ratio (`--hicache-ratio 2`), not by a fixed `--hicache-size`. + +The Write policy knob controls the GPU → CPU write and defaults to `write_through` (the upstream default): every page is mirrored to the CPU tier as it is written. `write_through_selective` backs up only hot data and `write_back` defers the copy to eviction, trading cache freshness for host-side I/O. + +The card is not offered on MI350X: the ROCm recipes run `--disable-radix-cache`, and the server rejects that alongside `--enable-hierarchical-cache`. + +Only the L2 tier is exposed here. For the storage (L3) tier and the canonical flag set, see the [HiCache best-practices recipe](../../../docs/advanced_features/hicache_best_practices) and the [HiCache documentation](../../../docs/advanced_features/hicache). diff --git a/docs/cookbook/autoregressive/GLM/GLM-4.5V.mdx b/docs/cookbook/autoregressive/GLM/GLM-4.5V.mdx index 25560c1f073d..83a4b6c38020 100644 --- a/docs/cookbook/autoregressive/GLM/GLM-4.5V.mdx +++ b/docs/cookbook/autoregressive/GLM/GLM-4.5V.mdx @@ -48,8 +48,7 @@ import { GLM45VDeployment } from "/src/snippets/autoregressive/glm-45v-deploymen - **H200 / B200:** Runs out of the box, supporting full context length plus concurrent image + video processing. - **Additional Multimodal Parameters:** - `--mm-attention-backend fa3`: Specify multimodal attention backend (Flash Attention 3). - - `--keep-mm-feature-on-device`: Retain multimodal feature tensors on GPU after processing to avoid D2H memory copies. - - `SGLANG_USE_CUDA_IPC_TRANSPORT=1`: Use CUDA IPC shared memory for multimodal data transport to significantly improve E2E latency. + - `--mm-feature-transport=cuda_ipc`: Use CUDA IPC shared memory for multimodal data transport to avoid D2H memory copies and significantly improve E2E latency. **Example with full multimodal optimizations:** ```bash Command diff --git a/docs/cookbook/autoregressive/GLM/GLM-4.6V.mdx b/docs/cookbook/autoregressive/GLM/GLM-4.6V.mdx index a8e89d631faa..583685976408 100644 --- a/docs/cookbook/autoregressive/GLM/GLM-4.6V.mdx +++ b/docs/cookbook/autoregressive/GLM/GLM-4.6V.mdx @@ -76,8 +76,7 @@ import { GLM46VDeployment } from "/src/snippets/autoregressive/glm-46v-deploymen - **H200 / B200:** Runs out of the box, supporting full context length plus concurrent image + video processing. - **Additional Multimodal Parameters:** - `--mm-attention-backend fa3`: Specify multimodal attention backend (Flash Attention 3). - - `--keep-mm-feature-on-device`: Retain multimodal feature tensors on GPU after processing to avoid D2H memory copies. - - `SGLANG_USE_CUDA_IPC_TRANSPORT=1`: Use CUDA IPC shared memory for multimodal data transport to significantly improve E2E latency. + - `--mm-feature-transport=cuda_ipc`: Use CUDA IPC shared memory for multimodal data transport to avoid D2H memory copies and significantly improve E2E latency. **Example with full multimodal optimizations:** ```bash Command diff --git a/docs/cookbook/autoregressive/GLM/GLM-5.1.mdx b/docs/cookbook/autoregressive/GLM/GLM-5.1.mdx index 39445e63eed8..bfa362e97eda 100644 --- a/docs/cookbook/autoregressive/GLM/GLM-5.1.mdx +++ b/docs/cookbook/autoregressive/GLM/GLM-5.1.mdx @@ -100,6 +100,9 @@ import { GLM51Deployment } from '/src/snippets/autoregressive/glm-51-deployment. - **B300 and GB300**: NVFP4 is the recommended deployment path. Use `nvidia/GLM-5.1-NVFP4` with `--quantization modelopt_fp4`. Use `tp=8` on B300 and `tp=4` on GB300. The CUDA 13 image variant is required for B300 and GB300. - **AMD GPUs**: BF16 and FP8 checkpoints run on MI300X/MI325X/MI355X at tp=8. On MI355X (gfx950), the MXFP4 checkpoint `amd/GLM-5.1-MXFP4` is also supported at tp=4 with `--kv-cache-dtype fp8_e4m3`. All AMD paths pass `--dsa-prefill-backend tilelang --dsa-decode-backend tilelang`, `--chunked-prefill-size 131072`, and `--watchdog-timeout 1200` (20 minutes for weight loading). FP8 uses approximately half the memory of BF16 (~89 GB/GPU vs ~175 GB/GPU). EAGLE speculative decoding is supported on AMD GPUs: MI300X/MI325X (gfx942) and MI355X (gfx950), but it **requires `--disable-custom-all-reduce`** — the aiter custom all-reduce kernel deadlocks during EAGLE verify at high concurrency, so without this flag the server will hang. - For other configuration tips (MTP, DSA kernel, Context Parallel, HiSparse, NVFP4, Index Cache), see the [DeepSeek-V3.2 cookbook page](../DeepSeek/DeepSeek-V3_2). GLM-5.1 and DeepSeek-V3.2 share the same model structure, so the optimization techniques are common. + +- **Prefill CP on CUDA**: Zigzag (`--cp-strategy zigzag`) is temporarily unavailable for GLM-5.1. Use `--enable-prefill-cp --cp-strategy interleave` with `--dp 1`. + - Use `--json-model-override-args '{"index_topk_pattern": "FFSFSSSFSSFFFSSSFFFSFSSSSSSFFSFFSFFSSFFFFFFSFFFFFSFFSSSSSSFSFFFSFSSSFSFFSFFSSS"}'` to enable the [IndexCache](https://github.com/THUDM/IndexCache) method for GLM-5.1. This can improve serving efficiency with only a small accuracy loss. If you are running rigorous accuracy evaluations, do not enable this feature. ## 4. Model Invocation diff --git a/docs/cookbook/autoregressive/GLM/GLM-5.2.mdx b/docs/cookbook/autoregressive/GLM/GLM-5.2.mdx index 47e9ec0c33e9..7953cc30fb26 100644 --- a/docs/cookbook/autoregressive/GLM/GLM-5.2.mdx +++ b/docs/cookbook/autoregressive/GLM/GLM-5.2.mdx @@ -118,13 +118,13 @@ import { Playground } from "/src/snippets/_playground.jsx"; - **Chunked-prefill size is regime-dependent.** At long input (8K+) the default `--chunked-prefill-size 2048` is too small and leaves the balanced point prefill-bound (queueing dominates TTFT). Raising it to `--chunked-prefill-size 32768` on the balanced recipe gave roughly **+34–78% output throughput and −39–59% TTFT** on 8×H200 and 8×B200 (8K-in / 1K-out) in our testing. It is **neutral for high-throughput** (decode-bound there) — keep the default. `--max-running-requests` tracks KV capacity, not a tuning free-for-all: ~60–90 concurrent 8K+1K FP8 requests fit on a single 8-GPU node, so pin balanced near `--max-running-requests 80` and let high-throughput run wider. - **AMD GPUs (MI300X / MI325X / MI355X).** FP8 (`zai-org/GLM-5.2-FP8`) runs single-node at `tp=8` on all three. BF16 (`zai-org/GLM-5.2`, ~1.51 TB) only fits single-node on **MI325X** (2 TB HBM) and **MI355X** (2.3 TB); **MI300X** (1.5 TB) cannot hold the BF16 weights plus KV cache on one node, so use FP8 there (or a multi-node BF16 layout once validated). Use the DSA tilelang backend (`--dsa-prefill-backend tilelang --dsa-decode-backend tilelang`) and add `--chunked-prefill-size 131072` plus `--watchdog-timeout 1200` (20 min for weight loading). FP8 uses about half the memory of BF16 (~89 GB/GPU vs ~175 GB/GPU). GLM-5.2 and DeepSeek-V3.2 share the same model structure; for other DSA / HiSparse tips see the [DeepSeek-V3.2 cookbook](../DeepSeek/DeepSeek-V3_2). -- **MI355X MXFP4 (gfx950-only).** AMD publishes a Quark-quantized **`amd/GLM-5.2-MXFP4`** build for MI355X. It needs `--trust-remote-code` (Quark's custom quant config) and runs at `tp=4` (the 4-bit MoE weights fit a 4-GPU slice) with `--kv-cache-dtype fp8_e4m3`, the same DSA tilelang backends, `--chunked-prefill-size`, and `--watchdog-timeout` as the FP8/BF16 recipes above. This recipe is carried over from the validated `amd/GLM-5.1-MXFP4` MI355X deployment (same DSA architecture family) and has not yet been benchmarked on GLM-5.2, so the Deploy panel marks it unverified. +- **MI355X MXFP4 (gfx950-only).** AMD publishes a Quark-quantized **`amd/GLM-5.2-MXFP4`** build for MI355X. It needs `--trust-remote-code` (Quark's custom quant config) and runs at `tp=4` (the 4-bit MoE weights fit a 4-GPU slice) with `--kv-cache-dtype fp8_e4m3`, the DSA triton backends (`--dsa-prefill-backend triton --dsa-decode-backend triton`, also SGLang's ROCm default), and the same `--chunked-prefill-size` and `--watchdog-timeout` as the FP8/BF16 recipes above. At `tp=4` this shape — 16 query heads per rank on an FP8 KV cache — is the one the gfx950 sparse-MLA kernels are tuned for; the `tp=8` recipes above stay on tilelang, which is what their published numbers were measured with. This recipe is carried over from the validated `amd/GLM-5.1-MXFP4` MI355X deployment (same DSA architecture family) and has not yet been benchmarked on GLM-5.2, so the Deploy panel marks it unverified. **gfx950 block-FP8 accuracy: fixed as of the pinned MI355X image (`v0.5.13.post1-rocm720-mi35x-20260618`).** Earlier SGLang ROCm images miscompiled AMD aiter's `gemm_a8w8_blockscale_bpreshuffle` GEMM on gfx950 (ROCm 7.2): the error was small per layer but compounded across all 78 layers and silently corrupted output — in-context reasoning broke (GSM8K ≈ 0) while short factual prompts still looked fine. The root cause was a gfx950/ROCm-7.2 miscompile of the CK kernel (a packed illegal-type FMA that relied on an LLVM coercion pass removed in ROCm 7.2; non-deterministic wrong rows near tile boundaries). This is resolved in the pinned image and newer: GLM-5.2-FP8 on MI350X/MI355X (gfx950) was re-validated at TP4 and TP8 — **GSM8K ≈ 0.96 (0% invalid)** and **15/15 needle-in-haystack retrieval to ~118K tokens**. **MI300X / MI325X (gfx942) were never affected.** If you must run an older image, treat gfx950 FP8 output as unverified. Background: [sgl-project/sglang#28685](https://github.com/sgl-project/sglang/issues/28685) (analysis) and the upstream CK fix [ROCm/rocm-libraries#8639](https://github.com/ROCm/rocm-libraries/pull/8639) (scalar FMA + accumulator anchor; restores correctness and determinism at -O3). -- **MTP / EAGLE speculative decoding** is disabled for AMD in the Deploy panel. The block-FP8 accuracy bug that previously degraded it is now fixed (see note above), but MTP on gfx950 still depends on the spec-decode draft kernel, which is not yet validated on this hardware (and at `--speculative-num-steps > 3` hits a separate build issue). Until MTP is validated on gfx950, omit the `--speculative-*` flags and serve without MTP. +- **MTP / EAGLE speculative decoding on AMD.** Five-step MTP is validated for `amd/GLM-5.2-MXFP4` on MI355X with `lmsysorg/sglang-rocm:v0.5.20-rocm720-mi35x-20260920`. Choose **Low-Latency** for TP8/EP1, or **High-Throughput** for TP4/EP4 with HiCache (`--enable-hierarchical-cache --hicache-ratio 1.0`). HiCache uses the default `kernel`, `page_first`, and `write_through` settings. MTP remains unvalidated for GLM-5.2 on MI300X/MI325X and for the other MI355X checkpoint precisions. ## 3. Advanced Usage @@ -254,6 +254,10 @@ For the full setup (streaming, tool-use, count_tokens, persisting env in `~/.cla ### 3.5 Context Parallelism + +Zigzag prefill CP (`--cp-strategy zigzag`) is temporarily unavailable for GLM-5.2. For prefill CP on CUDA, use `interleave` with `--dp 1` as shown below. + + Prefill context parallelism can help with reduction of TTFT under long context. To enable prefill context parallelism for GLM 5.2, please append the following arguments: ```bash --attn-cp-size 8 \ @@ -269,3 +273,33 @@ When deploying with PD Disaggregation, the prefill node can choose to enable [La --cp-strategy interleave \ ``` With LayerSplit, the kv cache on each rank can be sharded over the CP attention group, and prefetched when necessary. This can reduce kv cache memory by up to 75%, thus increasing the throughput on prefill side. + +### 3.6 Agentic Long-Context with HiCache DRAM Offload (NVFP4, MTP) + +**B300 (TP8):** +```bash Command +python3 -m sglang.launch_server \ + --model-path nvidia/GLM-5.2-NVFP4 \ + --trust-remote-code \ + --tp 8 \ + --ep-size 1 \ + --quantization modelopt_fp4 \ + --kv-cache-dtype fp8_e4m3 \ + --bf16-gemm-backend cutedsl \ + --max-prefill-tokens 8192 \ + --chunked-prefill-size 8192 \ + --mem-fraction-static 0.85 \ + --tool-call-parser glm47 \ + --reasoning-parser glm45 \ + --speculative-algorithm EAGLE \ + --speculative-num-steps 3 \ + --speculative-eagle-topk 1 \ + --speculative-num-draft-tokens 4 \ + --enable-hierarchical-cache \ + --hicache-size 270 \ + --hicache-write-policy write_back \ + --hicache-io-backend direct \ + --hicache-mem-layout page_first_direct +``` + +**B200 (TP8):** same command, with `--mem-fraction-static 0.83` and `--hicache-size 169`. Lower concurrency (c1/c4/c8) uses `--hicache-ratio 0.75` instead. diff --git a/docs/cookbook/autoregressive/GLM/GLM-5.3-Flash.mdx b/docs/cookbook/autoregressive/GLM/GLM-5.3-Flash.mdx index 9124e1b5c129..abd56dc27599 100644 --- a/docs/cookbook/autoregressive/GLM/GLM-5.3-Flash.mdx +++ b/docs/cookbook/autoregressive/GLM/GLM-5.3-Flash.mdx @@ -1,6 +1,6 @@ --- title: GLM-5.3-Flash -description: "Deploy GLM-5.3-Flash with SGLang using NVIDIA CUDA and AMD ROCm recipes, with MTP and multimodal serving where validated." +description: "Deploy GLM-5.3-Flash with SGLang using recipes for H100, H200, B200, B300, GB200, and GB300, with MTP and multimodal serving." tag: NEW --- @@ -10,10 +10,10 @@ tag: NEW -Use an SGLang build that includes GLM-5.3-Flash support. The AMD ROCm recipes also require [the ROCm engine changes in PR #36607](https://github.com/sgl-project/sglang/pull/36607) until they are available in a published SGLang image. +Use an SGLang build that includes GLM-5.3-Flash support (v0.5.20 or later). ```bash Command -docker pull lmsysorg/sglang:glm-5.3-flash +docker pull lmsysorg/sglang:latest ``` The deployment panel can render a complete `docker run` command for the selected hardware and options. See [Install SGLang with Docker](/docs/get-started/install#method-3-using-docker) for host setup. @@ -22,12 +22,14 @@ The deployment panel can render a complete `docker run` command for the selected Choose your hardware, then choose the operating point that matches your workload: -- **Low Latency** starts with adaptive MTP 5/1/6 speculative decoding and tensor parallelism to shorten interactive responses. +- **Low Latency** starts with MTP 5/1/6 speculative decoding and tensor parallelism to shorten interactive responses. - **High Throughput** starts with speculative decoding off, which avoids draft-and-verify overhead under sustained batches. -NVIDIA platforms expose both strategies. AMD ROCm currently exposes only the non-speculative High Throughput recipe because MTP has not been validated there. A **Verified** badge means that exact hardware and command were tested. **Final Verification In Progress** means the recipe runs and is queued for measurement on the final weights. **Not Verified** means the command is a supported starting point that still needs workload validation. A choice is disabled only when the underlying runtime combination is known to be unsupported. +Every listed hardware platform exposes both strategies. A **Verified** badge means that exact hardware and command were tested. **Final Verification In Progress** means the recipe runs and is queued for measurement on the final weights. **Not Verified** means the command is a supported starting point that still needs workload validation. A choice is disabled only when the underlying runtime combination is known to be unsupported. -The recommended selection is only a starting point. The same panel also lets you override the KV/DSA pairing, multimodal feature transport, and HiCache tiers. Changing an option that was not part of the measured command changes the badge to **Not Verified** without hiding the option. +The recommended selection is only a starting point. The same panel also lets you override the KV/DSA pairing, multimodal feature transport, Breakable Cuda Graph, and HiCache tiers. Changing an option that was not part of the measured command changes the badge to **Not Verified** without hiding the option. + +**Breakable Cuda Graph** defaults to **Off**. Select **On** to add `--cuda-graph-backend-prefill breakable` to the generated command. This requires a build that includes [PR #38522](https://github.com/sgl-project/sglang/pull/38522). import { Deployment } from "/src/snippets/_deployment.jsx"; import { config } from "/src/snippets/configs/zai-org/glm-5.3-flash.jsx"; @@ -69,7 +71,7 @@ GLM-5.3-Flash is a natively multimodal Mixture-of-Experts model built around a h - + @@ -92,19 +94,19 @@ The deployment recipes use the checkpoint's generation configuration. Override s ### Choose a strategy -Start with **Low Latency** for chat and agent workloads. Adaptive MTP changes the draft depth as acceptance changes, reducing unnecessary draft work when the server is busy. Measure **High Throughput** for heavily batched traffic where disabling speculative decoding can be more efficient. SGLang serves MTP through `--speculative-algorithm EAGLE` (upstream folds the older NEXTN spelling into EAGLE), so generated commands use that flag value. +Start with **Low Latency** for chat and agent workloads. It drafts from the checkpoint's MTP head at a fixed depth (5 steps, top-k 1, 6 draft tokens) with natural acceptance. Measure **High Throughput** for heavily batched traffic where disabling speculative decoding can be more efficient. SGLang serves MTP through `--speculative-algorithm EAGLE` (upstream folds the older NEXTN spelling into EAGLE), so generated commands use that flag value. -Strategy labels describe the workload goal. Both strategies stay available on NVIDIA GPUs; the AMD ROCm recipes expose only High Throughput until MTP speculative decoding is validated there. +Strategy labels describe the workload goal, not a hardware restriction. Both strategies stay available when you switch hardware; only the verification badge changes. ### Change the speculative algorithm The **Speculative** card in the Playground changes the algorithm without leaving the selected strategy: -- **EAGLE / Adaptive MTP 5-1-6** is exactly what Low Latency serves, so a Low Latency base starts on this chip. Pick it from a High Throughput base to keep that recipe's other settings and add the MTP head. +- **EAGLE / MTP 5-1-6** is exactly what Low Latency serves, so a Low Latency base starts on this chip. Pick it from a High Throughput base to keep that recipe's other settings and add the MTP head. - **Off (greedy)** strips the whole `--speculative-*` family, which is what High Throughput already starts from. -- **DFlash2** swaps the in-checkpoint MTP head for the trained block-diffusion draft in [`incoai/GLM-5.3-Flash-DFlash2`](https://huggingface.co/incoai/GLM-5.3-Flash-DFlash2). The draft proposes a whole block per step and the target verifies it in one forward pass, so output quality stays the target's. Its block size comes from the draft checkpoint, and the draft runs on `fa4` rather than the target's DSA backends. It needs a build that carries the GLM-5.3-Flash hidden-state capture from [PR #36708](https://github.com/sgl-project/sglang/pull/36708), which is merged into the [PR #36507](https://github.com/sgl-project/sglang/pull/36507) support branch (`xinyuan/glm-5.3-flash-support`) rather than into `main`, so the image pinned above is not enough on its own — pull that branch at its current head, or add #36708's commit on top of an older checkout. The draft repository is also access-gated: request access on its model page, then download it alongside the target before serving. This combination is not yet measured on the cookbook hardware, so treat it as a starting point. +- **DFlash2** swaps the in-checkpoint MTP head for the trained block-diffusion draft in [`incoai/GLM-5.3-Flash-DFlash2`](https://huggingface.co/incoai/GLM-5.3-Flash-DFlash2). The draft proposes a whole block per step and the target verifies it in one forward pass, so output quality stays the target's. Its block size comes from the draft checkpoint, and the draft runs on `fa4` rather than the target's DSA backends. The hidden-state capture it needs ([PR #36708](https://github.com/sgl-project/sglang/pull/36708)) shipped with the GLM-5.3-Flash support in v0.5.20, so the image pinned above is enough. The draft repository is access-gated: request access on its model page, then download it alongside the target before serving. This combination is not yet measured on the cookbook hardware, so treat it as a starting point. -Neither algorithm runs with DP-Attention, and neither is available on the AMD ROCm recipes; the card disables the affected chips and names the reason. +Neither algorithm runs with DP-Attention; the card disables the affected chips and names the reason. ### Size both memory pools @@ -118,12 +120,6 @@ Keep the checkpoint's KDA lower-bound setting unchanged. In particular, do not o On Blackwell, the recipes default to an FP8 KV cache with TRT-LLM DSA: on GB300 this pairing measured 2.9–5.7% higher throughput and about 1.8x the KV token capacity at identical pool bytes, with GSM8K accuracy within noise of BF16. BF16 KV with TileLang DSA remains selectable in the deployment panel and is the default on H100 and H200, where FP8 KV with TRT-LLM DSA is disabled. Switch the dtype and both DSA backends together; TileLang DSA with FP8 KV is not a valid CUDA combination. -On AMD ROCm, use BF16 KV cache with TileLang DSA, set `SGLANG_USE_AITER=1`, keep the MoE runner on Triton, and disable CUDA graphs. The ROCm recipe uses TP8 on a single eight-GPU node. MI300X and MI325X both use gfx942, but the MI325X entry remains explicitly unverified because it is inferred from MI300X rather than measured directly. AMD validation covers text generation and GSM8K only; multimodal serving remains unverified. - -### Decode context parallelism - -Decode context parallelism (DCP) shards the KV across GPUs during decode to cut long-context latency. The **Context Parallelism** row offers DCP4, validated on 4x GB300 (TP4/EP4) with both KV pairings and adaptive MTP 5/1/6; it requires the current release image, which carries the TileLang LSE fix. Other platforms and attention backends are unvalidated, and draft-extend v2 is unsupported under DCP. - ### Extend the cache hierarchy Keep **HiCache** off when GPU memory is sufficient. Select **L1 + L2** to spill reusable cache entries into host memory. Select **+ L3** only after configuring Mooncake on every serving node; the generated command exposes the required configuration path. These options remain selectable but are marked **Not Verified** until the resulting command is validated on the chosen hardware. @@ -138,13 +134,13 @@ The default multimodal feature transport is automatic, and on a single CUDA node ### 3.1 Reasoning -Thinking is enabled by the checkpoint's generation configuration, and generated commands enable `--reasoning-parser glm45` by default. The OpenAI-compatible API then places thinking in `message.reasoning_content` and the final answer in `message.content`. You can disable **Reasoning Parser** in the Playground when an integration needs the raw response format. +Thinking is enabled by the checkpoint's generation configuration, and generated commands enable `--reasoning-parser auto` (which resolves to `glm45` for GLM-5.3-Flash) by default. The OpenAI-compatible API then places thinking in `message.reasoning_content` and the final answer in `message.content`. You can disable **Reasoning Parser** in the Playground when an integration needs the raw response format. To disable thinking for a request, pass `chat_template_kwargs: {"thinking": false}` in the request body. ### 3.2 Tool calling -Generated commands enable `--tool-call-parser glm47` by default, so structured calls are returned in `message.tool_calls`. You can disable **Tool Call Parser** in the Playground when tool calling is not needed. On follow-up turns, read both `reasoning_content` and `content` because a thinking model can use either field around tool execution. +Generated commands enable `--tool-call-parser auto` (which resolves to `glm47` for GLM-5.3-Flash) by default, so structured calls are returned in `message.tool_calls`. You can disable **Tool Call Parser** in the Playground when tool calling is not needed. On follow-up turns, read both `reasoning_content` and `content` because a thinking model can use either field around tool execution. ### 3.3 Multimodal serving @@ -152,7 +148,7 @@ The base recipes accept image and video content through the OpenAI-compatible ch ### 3.4 Encoder disaggregation -Encoder disaggregation separates vision preprocessing from language inference. The verified topology uses one 4x GB300 node shared by an encoder-only TP4 process on port 30001 and a language-only TP4/EP4 process on port 30000. Start the encoder first. +Encoder disaggregation separates vision preprocessing from language inference. The verified topology uses one 4x GB300 node shared by an encoder-only TP4 process on port 30001 and a language-only TP4 process on port 30000. Start the encoder first. @@ -167,30 +163,28 @@ sglang serve \ +{/* TODO: re-validate encoder disaggregation with this language-server command; the 238,080-visual-token run used the earlier TP4/EP4 deep_gemm command with adaptive MTP. */} ```bash Command sglang serve \ --model-path zai-org/GLM-5.3-Flash \ --tp-size 4 \ - --ep-size 4 \ --attention-backend dsa \ --dsa-prefill-backend tilelang \ --dsa-decode-backend tilelang \ --linear-attn-backend triton \ --kv-cache-dtype bfloat16 \ --quantization fp8 \ - --moe-runner-backend deep_gemm \ + --moe-runner-backend flashinfer_trtllm \ --max-running-requests 64 \ --chunked-prefill-size 8192 \ --max-prefill-tokens 8192 \ - --disable-shared-experts-fusion \ --disable-prefill-cuda-graph \ --speculative-algorithm EAGLE \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ - --speculative-adaptive \ --language-only \ --encoder-urls http://localhost:30001 \ --mem-fraction-static 0.78 \ @@ -262,7 +256,7 @@ Operational notes: - Give each role a distinct `--nccl-port` when both share one node. - Single-node NIXL needs `UCX_NET_DEVICES=lo` and `UCX_TLS=tcp,cuda_copy,cuda_ipc,self,sm` in both server environments. -- The validated arm used the triton MoE runner; deep_gemm under PD is untested. +- The validated arm used the triton MoE runner; the flashinfer_trtllm runner from the deployment recipes is untested under PD. Known limitations: diff --git a/docs/cookbook/autoregressive/GLM/GLM-5.3.mdx b/docs/cookbook/autoregressive/GLM/GLM-5.3.mdx index 640414a2f4dd..bd6b6760b912 100644 --- a/docs/cookbook/autoregressive/GLM/GLM-5.3.mdx +++ b/docs/cookbook/autoregressive/GLM/GLM-5.3.mdx @@ -103,6 +103,7 @@ import { Playground } from "/src/snippets/_playground.jsx"; - **DeepSeek Sparse Attention (DSA).** GLM-5.3 uses the `glm_moe_dsa` architecture; SGLang auto-selects the DSA attention backends (`flashmla_sparse` prefill, `fa3` decode, `sgl-kernel` indexer topk). No attention-backend flag is needed on the supported hardware. SGLang also auto-selects the KV-cache dtype for DSA models — `fp8_e4m3` on Blackwell (B200/GB300/B300, which then routes DSA through the TensorRT-LLM backend) and `bf16` on Hopper (H200) — so no `--kv-cache-dtype` flag is required. On Hopper, pairing `--kv-cache-dtype fp8_e4m3` with `--dsa-prefill-backend flashmla_sparse_q8 --dsa-decode-backend flashmla_kv` selects the native FP8 sparse prefill kernel (computes directly on the fp8 KV cache with no fp8→bf16 dequantization round-trip; GLM-5.3's 64 query heads match the kernel's native tile) — see the [DeepSeek-V3.2 page](../DeepSeek/DeepSeek-V3_2) for kernel details; the optional `SGLANG_ENABLE_DSA_Q8KV8_*` performance env vars are documented in `python/sglang/srt/environ.py`. - **MTP / speculative decoding.** The checkpoint ships one nextn layer. Enable EAGLE MTP for lower latency (`--speculative-algorithm EAGLE --speculative-num-steps 5 --speculative-eagle-topk 1 --speculative-num-draft-tokens 6` for low-latency; `1-1-2` for balanced). The config's `index_share_for_mtp_iteration` reuses the DSA indexer's topk across draft steps (effective only at `--speculative-eagle-topk 1`). Watch the server's reported **accept length** and adjust `--speculative-num-steps` / `--speculative-num-draft-tokens`: lower the draft length when rejected draft tokens create excess verification work. +- **DFlash2 (block-diffusion draft).** The **Speculative** card in the [Playground above](#playground) also offers **DFlash2**, which replaces the in-checkpoint MTP layer with the separately trained block-diffusion drafter [`incoai/GLM-5.3-DFlash2`](https://huggingface.co/incoai/GLM-5.3-DFlash2). It proposes a whole block per step and the target verifies the block in one forward pass, so output quality stays the target's. The block size — 8, i.e. 7 draft tokens per verification step — comes from the draft checkpoint's own `dflash_config`, so no `--speculative-num-draft-tokens` is passed; the draft is a small dense model and runs on `fa4` instead of the target's DSA backends. Note that DFLASH runs on **CUDA/NPU only** and rejects **DP-Attention**, so turn DP-Attention off in the **Attention** card before selecting it on a high-throughput base. The draft repository is public but licensed CC BY-NC-ND 4.0 for research and evaluation. - **Memory.** The FP8 weights are large (MoE total, not active params). Start around `--mem-fraction-static 0.8` on H200 (TP8) and tune up; raise it for the 4-GPU GB300 single-node layout (TP4). - **DP-Attention + DeepEP** for the balanced/high-throughput strategies spreads attention across data-parallel ranks and routes MoE through DeepEP. - **BF16 weights need more GPUs.** The full-precision build (`zai-org/GLM-5.3-BF16`, ~1.5 TB) does not fit a single 8×H200 / 8×B200 / 4×GB300 node. It fits single-node on **8×B300** (TP8, ~2.1 TB HBM); on the smaller GPUs it needs a **multi-node** layout (e.g. 2×8×H200 or 2×8×B200 at TP16, 2×4×GB300 at TP8). FP8 is the recommended deployment. Use the same DSA / MTP / chunked-prefill guidance as FP8. @@ -116,7 +117,7 @@ import { Playground } from "/src/snippets/_playground.jsx"; ### 3.1 Reasoning -GLM-5.3 is a reasoning model. Enable the `glm45` reasoning parser (toggle **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground)) to separate thinking from the final answer — thinking lands in `message.reasoning_content`, the answer in `message.content`. The chat template defaults `clear_thinking` to `false`; for multi-turn chat, pass `chat_template_kwargs: {"clear_thinking": True}` so previous reasoning is cleared before the next response. +GLM-5.3 is a reasoning model, and generated commands enable `--reasoning-parser auto` (which resolves to `glm45` for GLM-5.3) by default so thinking is separated from the final answer — thinking lands in `message.reasoning_content`, the answer in `message.content`. Without the parser the server returns the thinking and the answer as one `content` string with a stray `` between them, because the chat template opens `` in the generation prompt. You can disable **Reasoning Parser** in the **Parsers** card of the [Playground above](#playground) when an integration needs that raw format. The chat template defaults `clear_thinking` to `false`; for multi-turn chat, pass `chat_template_kwargs: {"clear_thinking": True}` so previous reasoning is cleared before the next response. **Reasoning effort.** Pass `chat_template_kwargs: {"reasoning_effort": ...}` to select `low`, `high`, or `max`. If you omit it or pass another value, the template uses `max`. @@ -163,7 +164,7 @@ Here is how you can calculate it: ### 3.2 Tool Calling -Enable the `glm47` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`. GLM-5.3 emits the newer `………` format, so it needs the **`glm47`** parser — the older `glm45` parser does not parse it (the call would be left as raw text in `content`). On thinking mode the turn also fills `reasoning_content`, so print both fields. +Generated commands enable `--tool-call-parser auto` by default, so structured calls are returned in `message.tool_calls` with `finish_reason: "tool_calls"`. `auto` resolves to **`glm47`** for GLM-5.3: the model emits the newer `………` format, which the older `glm45` parser does not parse (the call would be left as raw text in `content`). Running with no tool-call parser fails the same way, and `finish_reason` stays `"stop"`, so an agent loop never sees the call. You can disable **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground) when tool calling is not needed. On thinking mode the turn also fills `reasoning_content`, so print both fields. @@ -217,7 +218,7 @@ For long-context, prefix-heavy workloads, enable hierarchical KV caching to spil ### 3.4 Claude Code Integration -GLM-5.3's strong reasoning + tool-calling makes it a good backend for [Claude Code](https://code.claude.com/docs/en/overview), Anthropic's agentic CLI. SGLang exposes the Anthropic-compatible `/v1/messages` endpoint on every server, so Claude Code can talk to a GLM-5.3 server with only environment variables — no code change. Launch the server with `--reasoning-parser glm45 --tool-call-parser glm47` (any recipe from the Deployment panel above works), then: +GLM-5.3's strong reasoning + tool-calling makes it a good backend for [Claude Code](https://code.claude.com/docs/en/overview), Anthropic's agentic CLI. SGLang exposes the Anthropic-compatible `/v1/messages` endpoint on every server, so Claude Code can talk to a GLM-5.3 server with only environment variables — no code change. Launch the server with `--reasoning-parser auto --tool-call-parser auto` (any recipe from the Deployment panel above works), then: ```bash Command export ANTHROPIC_BASE_URL="http://127.0.0.1:30000" @@ -241,6 +242,10 @@ For the full setup (streaming, tool-use, count_tokens, persisting env in `~/.cla ### 3.5 Context Parallelism + +Zigzag prefill CP (`--cp-strategy zigzag`) is temporarily unavailable for GLM-5.3. For prefill CP on CUDA, use `interleave` with `--dp 1` as shown below. + + Prefill context parallelism can help with reduction of TTFT under long context. To enable prefill context parallelism for GLM 5.3, please append the following arguments: ```bash --attn-cp-size 8 \ diff --git a/docs/cookbook/autoregressive/GLM/GLM-5.mdx b/docs/cookbook/autoregressive/GLM/GLM-5.mdx index 3f89e593d976..dfd9b6aa9b02 100644 --- a/docs/cookbook/autoregressive/GLM/GLM-5.mdx +++ b/docs/cookbook/autoregressive/GLM/GLM-5.mdx @@ -93,11 +93,10 @@ import { GLM5Deployment } from '/src/snippets/autoregressive/glm-5-deployment.js - **AMD GPUs**: Use `--dsa-prefill-backend tilelang --dsa-decode-backend tilelang` for the DSA attention backend. Add `--chunked-prefill-size 131072` and `--watchdog-timeout 1200` (20 minutes for weight loading). EAGLE speculative decoding is not currently supported on AMD for GLM-5. - For other configuration tips (MTP, DSA kernel, Context Parallel, HiSparse, NVFP4, Index Cache), see the [DeepSeek-V3.2 cookbook page](../DeepSeek/DeepSeek-V3_2). GLM-5 and DeepSeek-V3.2 share the same model structure, so the optimization techniques are common. -- Use `--json-model-override-args '{"index_topk_pattern": "FFSFSSSFSSFFFSSSFFFSFSSSSSSFFSFFSFFSSFFFFFFSFFFFFSFFSSSSSSFSFFFSFSSSFSFFSFFSSS"}'` for GLM-5-FP8 if you want to enable the [IndexCache](https://github.com/THUDM/IndexCache) method. This feature is supported through [this PR](https://github.com/sgl-project/sglang/pull/21405) and introduces only a small accuracy loss. However, if you are running rigorous accuracy evaluations, it is not recommended to enable this feature. - -**FP8 KV Cache**: `--kv-cache-dtype fp8_e4m3` quantizes the KV cache to FP8 at runtime. Since these FP8 model checkpoints do not include pre-calibrated KV cache scaling factors, SGLang defaults to a scale of 1.0, which may cause noticeable accuracy degradation on reasoning-heavy tasks. It is not included in the generated commands above; add it manually only if memory constraints require the trade-off. - +- **Prefill CP on CUDA**: Zigzag (`--cp-strategy zigzag`) is temporarily unavailable for GLM-5. Use `--enable-prefill-cp --cp-strategy interleave` with `--dp 1`. + +- Use `--json-model-override-args '{"index_topk_pattern": "FFSFSSSFSSFFFSSSFFFSFSSSSSSFFSFFSFFSSFFFFFFSFFFFFSFFSSSSSSFSFFFSFSSSFSFFSFFSSS"}'` for GLM-5-FP8 if you want to enable the [IndexCache](https://github.com/THUDM/IndexCache) method. This feature is supported through [this PR](https://github.com/sgl-project/sglang/pull/21405) and introduces only a small accuracy loss. However, if you are running rigorous accuracy evaluations, it is not recommended to enable this feature. ## 4. Model Invocation diff --git a/docs/cookbook/autoregressive/Google/Gemma4.mdx b/docs/cookbook/autoregressive/Google/Gemma4.mdx index 946fd5dc027a..01ddccfc20ab 100644 --- a/docs/cookbook/autoregressive/Google/Gemma4.mdx +++ b/docs/cookbook/autoregressive/Google/Gemma4.mdx @@ -97,6 +97,8 @@ For other installation methods, please refer to the [official SGLang installatio ### 3.1 Basic Configuration +The Gemma 4 series offers models in various sizes and architectures, optimized for different hardware platforms including NVIDIA GPUs, AMD GPUs, and Intel Arc Pro B-Series GPUs(codename: BMG (Battlemage)). The recommended launch configurations vary by hardware and model size. + **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and model variant. @@ -1455,7 +1457,7 @@ Median ITL (ms): 15.11 Reproduce against a running server (`--base-url` points at your endpoint): ```bash Command -pip install git+https://github.com/sgl-project/sgl-eval +pip install sgl-eval # Sanity-check the endpoint sgl-eval ping --base-url http://localhost:30000/v1 @@ -1641,7 +1643,7 @@ sgl-eval run mmlu --base-url http://localhost:30000/v1 --num-examples 2000 **gemma-4-E2B-it** ```shell Command -$ python -m sglang.bench_one_batch --correct --model google/gemma-4-E2B-it .... +$ python -m sglang.benchmark.one_batch --correct --model google/gemma-4-E2B-it .... prefill logits (final): tensor([[-25.3063, -2.5718, -10.3674, ..., -25.3779, -25.5181, -25.2337]], device='cuda:0') .... @@ -1656,7 +1658,7 @@ prefill logits (final) tensor([-25.3281, -2.1367, -10.2266, ..., -25.4375, -25 **gemma-4-E4B-it** ```shell Command -$ python -m sglang.bench_one_batch --correct --model google/gemma-4-E4B-it .... +$ python -m sglang.benchmark.one_batch --correct --model google/gemma-4-E4B-it .... prefill logits (final): tensor([[-17.6478, 7.9901, -5.6505, ..., -17.5658, -17.6478, -17.7293]], device='cuda:0') .... @@ -1670,7 +1672,7 @@ prefill logits (final) tensor([-17.5625, 8.0469, -5.5742, ..., -17.4688, -17 **gemma-4-31B-it** ```shell Command -$ python -m sglang.bench_one_batch --correct --model google/gemma-4-31B-it .... +$ python -m sglang.benchmark.one_batch --correct --model google/gemma-4-31B-it .... prefill logits (final): tensor([[-2.0748, 1.1245, -7.4356, ..., -2.1059, -2.1525, -2.2303]], device='cuda:0') .... diff --git a/docs/cookbook/autoregressive/IFM/K2-Horizon.mdx b/docs/cookbook/autoregressive/IFM/K2-Horizon.mdx new file mode 100644 index 000000000000..9d0b5de045de --- /dev/null +++ b/docs/cookbook/autoregressive/IFM/K2-Horizon.mdx @@ -0,0 +1,254 @@ +--- +title: K2 Horizon +description: "Deploy the IFM K2 Horizon family with SGLang — BF16 recipes for six dense and Mixture-of-Experts checkpoints on NVIDIA H200 GPUs." +tag: NEW +--- + +## Deployment + + + + + +Install from a source checkout that includes the K2 Horizon runtime support in [PR #37654](https://github.com/sgl-project/sglang/pull/37654) before using the generated commands: + +```bash Command +pip install --upgrade pip +pip install -e "python" +``` + +For general source-install guidance, see [Install → Method 2: From source](/docs/get-started/install#method-2-from-source). + + + +Choose a model size to generate its validated single-node H200 launch command. The serving settings in all six base recipes completed two independent H200 benchmark launches at the pinned model revisions using the K2 Horizon runtime support in [PR #37654](https://github.com/sgl-project/sglang/pull/37654). + +import { Deployment } from "/src/snippets/_deployment.jsx"; +import { config } from "/src/snippets/configs/IFM/k2-horizon.jsx"; +import { benchmarks } from "/src/snippets/configs/IFM/k2-horizon-benchmarks.jsx"; + + + +Each speed and accuracy value in the benchmark cards is the arithmetic mean of two independent server launches. + +## Playground + +The base recipes enable the `k2_horizon` reasoning parser. Use the Playground to configure the `k2_horizon` tool-call parser, TP and MoE expert-parallel overrides, NGRAM speculative decoding, PD disaggregation, and HiCache. A separate validation pass functionally exercised the tool-call parser on all six checkpoints; every other Playground override remains unverified. + +import { Playground } from "/src/snippets/_playground.jsx"; + + + +## 1. Model introduction + +**K2 Horizon** is an IFM model family spanning six checkpoints from **0.9B** to **375B** parameters. The family includes dense models, Mixture-of-Experts models, and the MoVA-36B-A4B checkpoint's routed Mixture-of-Value-Attention path. This page provides native SGLang serving recipes for the family. + +
13B Flash Official (0731), with a bundled DSpark draft head; verified on 8×B200, 4×GB300, and 4×H200
DeepSeek-V4-Flash-Vision-Exp305B13BFlash Vision (Exp) — experimental multimodal (image-text-to-text): the 0731 Flash base + vision encoder & aligner; verified on 4×B200 (TP=4), requires the preview build
DeepSeek-V4-Pro 1.6T
PrecisionFP8 weights; FP8 KV cache by default on Blackwell, BF16 KV cache on H100, H200, and the AMD ROCm recipesFP8 weights; FP8 KV cache by default on Blackwell, BF16 KV cache on H100 and H200
Context
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
CheckpointArchitectureContext lengthH200 recipe
0.9BDense131,072BF16 · TP1
3.7BDense524,288BF16 · TP1
7BDense524,288BF16 · TP1
32BDense524,288BF16 · TP2
MoVA-36B-A4BMoE + MoVA524,288BF16 · TP2
375B-A23BMoE524,288BF16 · TP8
+ + + All six checkpoints use the Apache License 2.0. + + +**Resources:** [K2 Horizon collection](https://huggingface.co/collections/IFM/k2-horizon) · [K2 Horizon 375B-A23B model card](https://huggingface.co/IFM/K2-Horizon-375B-A23B). + +## 2. Configuration tips + +- **Hardware and precision:** The current matrix covers NVIDIA H200 GPUs and BF16 checkpoints only. +- **Parallelism:** Use TP1 for 0.9B, 3.7B, and 7B; TP2 for 32B and MoVA-36B-A4B; and TP8 for 375B-A23B. Each recipe stays within one node. +- **Attention:** The verified base recipes select FlashAttention-3 with `--attention-backend fa3`. +- **Revisions:** The recipes pin revisions `9b9ec1f7e17f62ed218df542687a144116219d84` (0.9B), `c177771836a4c460743c00002c22483f6f18d1eb` (3.7B), `69ada542b68fe13d767479db2ab9421baff88681` (7B), `e1fd0277713e4eefcd3416348fd6fedacf7f2392` (32B), `16d20c739c687c08423422d1a2fbba6c529014cd` (MoVA-36B-A4B), and `12812264242a14dce44aa7ae27f931ff4584bcbf` (375B-A23B). +- **Parsers:** The base recipes enable reasoning with `--reasoning-parser k2_horizon`. Enable `--tool-call-parser k2_horizon` from the **Parsers** card in the [Playground](#playground) when you need tool calling. +- **MoVA-36B-A4B router provenance:** The MoVA-36B-A4B checkpoint's source xLLM router used two GEMM partitions. `--json-model-override-args '{"xllm_source_router_gemm_partitions":2}'` preserves its BF16 partial-GEMM rounding and FP32 reduction order; SGLang does not infer this source topology from runtime TP. +- **375B-A23B loading:** The 375B-A23B recipe disables multithreaded weight loading with `--model-loader-extra-config '{"enable_multithread_load":false}'`. +- **Playground features:** TP, EP, NGRAM, PD disaggregation, and HiCache are optional, unverified command overrides. The tool-call parser was functionally exercised separately. +- **Deliberate omissions:** Context parallelism is unsupported or unvalidated for xLLM. DP-attention and alternate MoE backend selectors are not exposed until they are validated. +- **Validation:** All six recipes completed native speed and full GSM8K evaluation on H200 using the K2 Horizon runtime support in [PR #37654](https://github.com/sgl-project/sglang/pull/37654). +- **375B-A23B provenance:** The 375B-A23B benchmark launched a directly mounted checkpoint and passed revision `12812264242a14dce44aa7ae27f931ff4584bcbf` to the server. The other five launches used materialized Hugging Face snapshots at their pinned revisions. + +## 3. Benchmark results + +Speed was measured with native `sglang.benchmark.serving` using fixed random token IDs: 8,192 input tokens, 1,024 output tokens, 64 warmup requests, a cache flush, and seed `20260901`. The latency run used 32 prompts at concurrency 1. The throughput run used 256 prompts at concurrency 64 for TP1, 32 for TP2, and 8 for TP8. TTFT and TPOT are medians within each launch; the benchmark cards show the arithmetic mean across the two launches. Throughput per GPU is `(total input tokens + total output tokens) / duration / TP`. + +GSM8K used the full 1,319-example test split with `sgl-eval`, 32 client threads, a 32,768-token output limit, temperature 0, top-p 0.95, seed 0, and high reasoning effort. The mean and both independent server-launch scores are shown below. All requests completed without errors. Responses that reached the output limit remain in the score: truncation was 6.90% in both 0.9B launches, 4.09% and 4.70% for 3.7B, and at most 1.14% for the other checkpoints. + +| Checkpoint | Launch 1 | Launch 2 | Mean | +| --- | ---: | ---: | ---: | +| 0.9B | 85.22% | 85.29% | 85.25% | +| 3.7B | 92.12% | 91.89% | 92.00% | +| 7B | 95.30% | 94.47% | 94.88% | +| 32B | 95.75% | 96.21% | 95.98% | +| MoVA-36B-A4B | 95.45% | 94.84% | 95.15% | +| 375B-A23B | 95.53% | 95.60% | 95.56% | + +## 4. Advanced usage + +### 4.1 Reasoning + +The base recipes include `--reasoning-parser k2_horizon`. The parser returns thinking in `reasoning_content` and the final answer in `content`. + +The example output below was captured with the 3.7B checkpoint and both parser flags enabled. + + + +```python Example +from openai import OpenAI + +client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") +response = client.chat.completions.create( + model="IFM/K2-Horizon-3.7B", + messages=[{"role": "user", "content": "Hello"}], +) + +message = response.choices[0].message +reasoning = getattr(message, "reasoning_content", None) +if reasoning is None: + reasoning = (message.model_extra or {}).get("reasoning_content") +print("Reasoning:", reasoning) +print("Answer:", message.content) +``` + + + + + +```text Output +Reasoning: We have a user who just said "Hello". We need to respond. As an AI, we should be friendly, perhaps ask how can we help. If says "Hello" alone, respond with greeting and offer assistance. Potentially ask what they need. There's no special instruction; just typical. + +But maybe we must think about the context: The user gave just a greeting; we should respond with a greeting and ask how we can help. Could also ask if they want anything specific. + +We should be concise but friendly: "Hello! How can I assist you today?" Provide some content. + +But maybe the user expects something else. But in this scenario, we just reply with a greeting. + +Thus answer: "Hello! What can I do for you?" Etc. + +Check any style guidelines: We want to keep it short or perhaps ask follow-up. The user didn't ask any question, just saying hello. Could also respond: "Hello! ..." + +Thus arguably the answer: "Hi there! How can I help you today?" Let's do that. + +But also maybe we could ask if they need help with something specific. That's appropriate. + +Thus the final answer: "Hello! How can I assist you today?" Possibly add friendly banter. + +We'll respond. + +Answer: +Hello! 👋 How can I help you today? +``` + + + +### 4.2 Tool calling + +Enable **Tool Call Parser** in the **Parsers** card of the [Playground](#playground) to add `--tool-call-parser k2_horizon`. The parser exposes K2 tool output as OpenAI-compatible `message.tool_calls`. + +The example output below was captured with the 32B checkpoint and both parser flags enabled. + + + +```python Example +from openai import OpenAI + +client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") +tools = [ + { + "type": "function", + "function": { + "name": "get_weather", + "description": "Return weather for a city.", + "parameters": { + "type": "object", + "properties": {"city": {"type": "string"}}, + "required": ["city"], + }, + }, + } +] + +response = client.chat.completions.create( + model="IFM/K2-Horizon-32B", + messages=[{"role": "user", "content": "Call get_weather for Paris."}], + tools=tools, + tool_choice={"type": "function", "function": {"name": "get_weather"}}, + max_tokens=1024, +) + +message = response.choices[0].message +reasoning = getattr(message, "reasoning_content", None) +if reasoning is None: + reasoning = (message.model_extra or {}).get("reasoning_content") +print("Reasoning:", reasoning) +print("Content:", message.content or "") +for tool_call in message.tool_calls or []: + print(f"Tool call: {tool_call.function.name}({tool_call.function.arguments})") +``` + + + + + +```text Output +Reasoning: Okay, the user wants me to call the get_weather function for Paris. Let me check the tools available. There's a function called get_weather that takes a city parameter. The required parameter is city, and it's a string. So I need to make sure to pass "Paris" as the city. I don't see any other parameters needed. The user didn't mention any other details, so I'll just use the city name. Let me structure the tool call correctly. The function name is get_weather, and the arguments should be a JSON object with "city": "Paris". I need to make sure the JSON is properly formatted. Alright, that should do it. + +Content: +Tool call: get_weather({"city": "Paris"}) +``` + + + +### 4.3 HiCache + +For large checkpoints or repeated long-context prefixes, enable **HiCache** in the [Playground](#playground), then choose a storage backend and write policy. Treat the generated override as unverified until it is exercised on your storage stack. diff --git a/docs/cookbook/autoregressive/InclusionAI/Ling-3.0-flash-VL.mdx b/docs/cookbook/autoregressive/InclusionAI/Ling-3.0-flash-VL.mdx new file mode 100644 index 000000000000..53008fe10ed7 --- /dev/null +++ b/docs/cookbook/autoregressive/InclusionAI/Ling-3.0-flash-VL.mdx @@ -0,0 +1,324 @@ +--- +title: Ling-3.0-flash-VL +description: "Deploy Ling-3.0-flash-VL with SGLang — a ~125B hybrid KDA + MLA MoE vision-language model (5.1B active) with image and video input, thinking mode, and 128K context on Blackwell and Hopper GPUs." +tag: NEW +--- + +## Deployment + + + + + +For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel. + + + + + +```bash Command +pip install --upgrade pip +pip install uv +uv pip install --prerelease=allow sglang +``` + +Then run the **Python** output of the command panel below in that environment. Ling-3.0-flash-VL support requires sglang with sgl-project/sglang#38526 (or newer); until it merges, install directly from the PR head: + +```bash Command +pip install --upgrade pip +pip install uv +uv pip install "sglang @ git+https://github.com/sgl-project/sglang.git@refs/pull/38526/head#subdirectory=python" +uv pip install torchcodec==0.16.0 # optional, preferred video-decode backend +``` + +Video input needs `decord` or `torchcodec` in the environment; audio is not supported. Downloading the checkpoint requires `HF_TOKEN`. + + + + + +```bash Command +docker pull lmsysorg/sglang:dev-Ling-3.0-flash-VL +``` + +For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. + + + + + + + +Pick your hardware and quantization to generate the launch command. The checkpoint ships in BF16 with FP8, INT4, and FP4 variants, and each combination carries one **Balanced** recipe: BF16 uses `--tp 4` on 288 GB-class (GB300, B300) and 141 GB-class (B200, H200) GPUs and `--tp 8` on H100 (sized so the ~250 GB of BF16 weights fit with KV-cache headroom); FP8 uses `--tp 1` on GB300/B300/B200 and `--tp 2` on H200/H100; INT4 and FP4 use `--tp 1` everywhere. DGX Spark offers INT4 and FP4 only. + +import { Deployment } from "/src/snippets/_deployment.jsx"; +import { config } from "/src/snippets/configs/inclusionAI/ling-3.0-flash-vl.jsx"; +import { benchmarks } from "/src/snippets/configs/inclusionAI/ling-3.0-flash-vl-benchmarks.jsx"; + + + +## Playground + +The Playground is where you experiment with **SGLang features beyond the verified matrix**. The Deploy panel above only emits the recipes on this page; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. + +import { Playground } from "/src/snippets/_playground.jsx"; + + + +## 1. Model Introduction + +Ling-3.0-flash-VL is the vision-language member of the BailingMoeV3 family. Its language backbone is the [Ling-3.0-flash](/cookbook/autoregressive/InclusionAI/Ling-3.0-flash) hybrid-attention Mixture-of-Experts (MoE) model: 42 layers that interleave Kimi Delta Attention (KDA) linear-attention layers with gated Multi-head Latent Attention (MLA) full-attention layers, on top of a fine-grained MoE feed-forward network with 512 routed experts (8 active per token) plus a shared expert. A 27-layer vision encoder (0.4B parameters, 16-pixel patches with 2×2 spatial merging and paired frames for video) feeds image and video tokens into that backbone through a small MLP projector. In total the checkpoint holds **~125B parameters with ~5.1B active per token**, so per-token inference cost stays close to a small model. + +It is a thinking model: the chat template turns chain-of-thought on by default and exposes an `enable_thinking` switch. It supports structured tool calling. Inputs are text, images, and video; audio is not supported. Native context length is 128K tokens. Unlike Ling-3.0-flash, the VL checkpoint ships no built-in MTP draft layer, so there is no speculative-decoding recipe. + +**Available Models:** + +- **BF16**: [inclusionAI/Ling-3.0-flash-VL](https://huggingface.co/inclusionAI/Ling-3.0-flash-VL) — ~125B total / ~5.1B active +- **FP8**: [inclusionAI/Ling-3.0-flash-VL-FP8](https://huggingface.co/inclusionAI/Ling-3.0-flash-VL-FP8) — 128×128 block quantization +- **INT4 (GPTQ)**: [inclusionAI/Ling-3.0-flash-VL-int4](https://huggingface.co/inclusionAI/Ling-3.0-flash-VL-int4) — compressed-tensors, group size 32 +- **FP4 (MXFP4)**: [inclusionAI/Ling-3.0-flash-VL-fp4](https://huggingface.co/inclusionAI/Ling-3.0-flash-VL-fp4) — e2m1 experts + FP8 e4m3 activations, Blackwell only + +**License:** MIT + +**Recommended generation:** `temperature=1.0`, `top_p=0.95`, `top_k=20` (from the checkpoint's `generation_config.json`; SGLang applies these defaults, so the samples below do not set them). + +**Resources:** [HuggingFace](https://huggingface.co/inclusionAI/Ling-3.0-flash-VL). + +## 2. Configuration Tips + +- `--trust-remote-code` is required. The checkpoint declares its config and multimodal processor through `auto_map`, and SGLang's native implementation loads the image/video preprocessing and the chat template from those files. Serving from the Hub repo or from a local snapshot needs no extra environment setup. +- BF16 weights take ~250 GB. The recipes use `--tp 4` on 288 GB-class (GB300, B300) and 141 GB-class (B200, H200) GPUs and `--tp 8` on H100, with the default memory pool. The 4×GB300 and 4×H200 cells are validated end to end; the rest are sized by the same rule and carry the unverified badge. Adjust TP in the **Attention Parallelism** card of the [Playground](#playground). +- HiCache is supported: `--enable-hierarchical-cache` attaches the hierarchical cache to the hybrid SSM state (verified with CPU offload; L3 storage backends such as Mooncake are untested). On shared hosts, set an explicit `--hicache-size ` — the default ratio can request more host RAM than is free. +- Vision token budget: the encoder emits one token per 32×32-pixel block after merging, so a 1280×720 image costs about 880 tokens and the processor's `max_pixels` of 4,194,304 caps a single image at roughly 4,100 tokens. Video frames are paired before patching; the sample clip in §3.2 costs 4,224 tokens. Size `--chunked-prefill-size` and `--max-running-requests` with these counts in mind for image-heavy traffic. +- Parsers: the generated recipes include `--reasoning-parser auto --tool-call-parser auto` by default. Both resolve to `ling3` from the chat template (the server logs `Auto-detected --reasoning-parser as ling3 from chat template` and the same for the tool-call parser), so no model-specific parser name is needed; the explicit form `--reasoning-parser ling3 --tool-call-parser ling3` is equivalent. With the parsers on, reasoning is returned in `message.reasoning_content` and structured tool calls in `message.tool_calls`; without them, `message.content` carries the reasoning followed by `` and the answer (the opening `` tag sits in the prompt and is not echoed). +- Thinking is on by default (`enable_thinking` defaults to true in the chat template, and the `ling3` reasoning parser follows that default). Turn it off per request with `"chat_template_kwargs": {"enable_thinking": false}` (§3.3). +- Native context is 128K, and the generated recipes default to the 256K YaRN variant (`--context-length 262144` plus the `rope_scaling` override and `SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1`), matching the model card. For the native 128K window, drop all three from the command. +- Audio content parts are rejected before preprocessing; send images as `image_url` and clips as `video_url` content parts (§3). +- The FP8 variant (`inclusionAI/Ling-3.0-flash-VL-FP8`) uses 128×128 block quantization, so plain TP must keep `768 ÷ tp` divisible by 128 (TP 1, 2, 3, or 6 — TP=4 fails to load). FP8 recipes default to a single GPU where the ~126 GB weights fit (`--tp 1` on 288 GB-class and B200, verified on GB300) or `--tp 2` on tighter cards. As a manual alternative on four GPUs, `--tp 4 --ep 4` keeps experts whole per rank (measured on 4×GB300). The BF16 checkpoint can also be quantized online with `--quantization fp8`; measured accuracy and speed for both paths are in the benchmark cards below. +- The INT4 variant (`inclusionAI/Ling-3.0-flash-VL-int4`) is a GPTQ 4-bit export (compressed-tensors, group size 32). It runs on all listed hardware with single-GPU (`--tp 1`) recipes; GB300, H200, and DGX Spark cells are verified, the rest carry the unverified badge. Shared-expert fusion is auto-disabled for the INT4 and FP4 checkpoints by the current dev image and by sglang main; older images need `--disable-shared-experts-fusion` explicitly. +- The FP4 variant (`inclusionAI/Ling-3.0-flash-VL-fp4`) is an MXFP4 export (e2m1 routed-expert weights with e8m0 scales, group 32, plus FP8 e4m3 dynamic activations) and runs on Hopper (SM90 CUTLASS W4A16) and Blackwell (SM100/SM120): H200, GB300, B300, B200, and DGX Spark. The `flashinfer_mxfp4` MoE backend is selected automatically on the `lmsysorg/sglang:dev-Ling-3.0-flash-VL` image and on sglang main; on builds that already include the VL integration but predate the auto-selection, add `--moe-runner-backend flashinfer_mxfp4` explicitly. GB300, H200, and DGX Spark cells are verified, the rest carry the unverified badge. + +## 3. Advanced Usage + +The examples below were run against a server launched with the recipe above plus `--reasoning-parser auto --tool-call-parser auto`, which resolves both parsers to `ling3` (§2). Reasoning therefore arrives in `message.reasoning_content` and the answer in `message.content`. + +### 3.1 Image Understanding + +Send images as OpenAI-style `image_url` content parts: + + + +```python Example +from openai import OpenAI + +client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") +resp = client.chat.completions.create( + model="inclusionAI/Ling-3.0-flash-VL", + messages=[{ + "role": "user", + "content": [ + {"type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"}}, + {"type": "text", "text": "Describe this image in one sentence."}, + ], + }], + max_tokens=2048, +) +msg = resp.choices[0].message +print("Reasoning:", getattr(msg, "reasoning_content", None)) +print("Content:", msg.content) +``` + + + + + +```text Output +Reasoning: The user wants a one-sentence description of this image. Let me look at the image carefully. + +The image shows a man wearing a yellow shirt (with some kind of text on it, probably "Sudsy" or similar) standing behind a yellow taxi SUV, ironing clothes on a portable ironing board set up at the rear of the vehicle. On the left side, another yellow taxi (sedan) is passing by on the street. The background is an urban streetscape with buildings lined up. + +I need to write this in one sentence. For example: "A man wearing a yellow shirt irons clothes on a portable ironing board attached to the back of a yellow taxi on a city street, while another taxi passes by." + +Let me make it concise yet descriptive. One sentence. + +"On a city street, a man in a yellow sweatshirt irons clothes on a portable ironing board mounted on the back of a yellow taxi SUV, while another yellow taxi drives past." + +This works. I'll return only the answer. +Content: A man in a yellow shirt irons clothes on a portable ironing board set up at the back of a yellow taxi on a city street, while another yellow cab drives past. +``` + + + +### 3.2 Video Understanding + +Clips go in as `video_url` content parts. The processor samples and pairs frames itself; no client-side frame extraction is needed: + + + +```python Example +from openai import OpenAI + +client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY", timeout=900) +resp = client.chat.completions.create( + model="inclusionAI/Ling-3.0-flash-VL", + messages=[{ + "role": "user", + "content": [ + {"type": "video_url", "video_url": {"url": "https://github.com/sgl-project/sgl-test-files/raw/refs/heads/main/videos/jobs_presenting_ipod.mp4"}}, + {"type": "text", "text": "What is happening in this video? Answer in two sentences."}, + ], + }], + max_tokens=2048, +) +msg = resp.choices[0].message +print("Reasoning:", getattr(msg, "reasoning_content", None)) +print("Content:", msg.content) +``` + + + + + +```text Output +Reasoning: The user wants me to describe what is happening in this video in two sentences. + +Looking at the video, I can see: +- A man on stage (appears to be Steve Jobs) presenting something +- He's wearing a black shirt and jeans +- There's a large screen behind him showing a close-up of a hand reaching toward a pocket on blue jeans +- Text appears on screen asking "Anyone know what this pocket is for?" +- He pulls out a small white device (iPod Nano) from his pocket +- The text says "I've always wondered that" and "well now we know" and "is the new iPod Nano" +- He holds up the white iPod Nano device + +This is clearly Steve Jobs presenting the new iPod Nano at an Apple event. The small white device is the iPod Nano. + +I need to summarize this in exactly two sentences. +Content: A presenter on stage pulls a small white device from his pocket while a large screen behind him displays a close-up image of a hand reaching toward a jeans pocket. He holds the device up to reveal it is the new iPod Nano, answering the on-screen text that questioned what the small pocket was designed for. +``` + + + +### 3.3 Thinking Mode + +Thinking is on by default. Disable it for a single request through the chat template's `enable_thinking` kwarg; the model then answers directly and `reasoning_content` stays empty: + + + +```python Example +from openai import OpenAI + +client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") +resp = client.chat.completions.create( + model="inclusionAI/Ling-3.0-flash-VL", + messages=[{ + "role": "user", + "content": [ + {"type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"}}, + {"type": "text", "text": "Describe this image in one sentence."}, + ], + }], + max_tokens=512, + extra_body={"chat_template_kwargs": {"enable_thinking": False}}, +) +msg = resp.choices[0].message +print("Reasoning:", getattr(msg, "reasoning_content", None)) +print("Content:", msg.content) +``` + + + + + +```text Output +Reasoning: None +Content: A man in a yellow sweatshirt irons a blue shirt on a folding board attached to the back of a parked yellow taxi SUV, while another yellow cab drives past on a city street. +``` + + + +### 3.4 Tool Calling + +With the tool-call parser on, structured calls are parsed into `message.tool_calls` and `finish_reason` is `tool_calls`. Send the tool result back as a `tool` message to get the final answer; on this thinking model the follow-up turn may put text in `reasoning_content` as well as `content`, so print both: + + + +```python Example +import json +from openai import OpenAI + +client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") +tools = [{ + "type": "function", + "function": { + "name": "get_weather", + "description": "Get the current weather for a city", + "parameters": { + "type": "object", + "properties": {"city": {"type": "string", "description": "City name"}}, + "required": ["city"], + }, + }, +}] +messages = [{"role": "user", "content": "What's the weather like in Paris today?"}] + +resp = client.chat.completions.create( + model="inclusionAI/Ling-3.0-flash-VL", + messages=messages, + tools=tools, + tool_choice="auto", + max_tokens=2048, +) +msg = resp.choices[0].message +print("finish_reason:", resp.choices[0].finish_reason) +print("Reasoning:", getattr(msg, "reasoning_content", None)) +print("Content:", msg.content) +for call in msg.tool_calls: + print("Tool call:", call.function.name, call.function.arguments) + +messages.append({ + "role": "assistant", + "content": msg.content or "", + "tool_calls": [{"id": c.id, "type": "function", + "function": {"name": c.function.name, "arguments": c.function.arguments}} + for c in msg.tool_calls], +}) +messages.append({ + "role": "tool", + "tool_call_id": msg.tool_calls[0].id, + "content": json.dumps({"city": "Paris", "temperature_c": 18, "condition": "light rain"}), +}) +resp = client.chat.completions.create( + model="inclusionAI/Ling-3.0-flash-VL", + messages=messages, + tools=tools, + max_tokens=2048, +) +msg = resp.choices[0].message +print("finish_reason:", resp.choices[0].finish_reason) +print("Reasoning:", getattr(msg, "reasoning_content", None)) +print("Content:", msg.content) +``` + + + + + +```text Output +finish_reason: tool_calls +Reasoning: The user wants to know the current weather in Paris. I'll call the get_weather function with the city "Paris". +Content: Let me check the weather in Paris for you right away! +Tool call: get_weather {"city": "Paris"} +finish_reason: stop +Reasoning: None +Content: Here's the current weather in Paris: + +- 🌡️ **Temperature:** 18°C +- 🌧️ **Condition:** Light rain + +It looks like a mild day with some light rain. You might want to bring an umbrella if you're heading out! ☂️ +``` + + + +For more API examples, see the [SGLang OpenAI Vision API Guide](../../../docs/basic_usage/openai_api_vision) and the [Tool Parser guide](../../../docs/advanced_features/tool_parser). diff --git a/docs/cookbook/autoregressive/InclusionAI/Ling-3.0-flash.mdx b/docs/cookbook/autoregressive/InclusionAI/Ling-3.0-flash.mdx index e9f571c7a69a..4c794666d978 100644 --- a/docs/cookbook/autoregressive/InclusionAI/Ling-3.0-flash.mdx +++ b/docs/cookbook/autoregressive/InclusionAI/Ling-3.0-flash.mdx @@ -1,7 +1,6 @@ --- title: Ling-3.0-flash description: "Deploy Ling-3.0-flash with SGLang — a 124B hybrid KDA + MLA MoE in BF16, FP8, INT4, or MXFP4 on Hopper and Blackwell GPUs." -tag: NEW --- ## Deployment diff --git a/docs/cookbook/autoregressive/InclusionAI/Ling-3.0-tiny.mdx b/docs/cookbook/autoregressive/InclusionAI/Ling-3.0-tiny.mdx index c117f27b7f88..2a6f4e46afe3 100644 --- a/docs/cookbook/autoregressive/InclusionAI/Ling-3.0-tiny.mdx +++ b/docs/cookbook/autoregressive/InclusionAI/Ling-3.0-tiny.mdx @@ -1,7 +1,6 @@ --- title: Ling-3.0-tiny description: "Deploy Ling-3.0-tiny with SGLang — a compact ~7.9B total / ~1.2B active hybrid KDA + MLA MoE in BF16, FP8, or INT4, with thinking mode and tool calling." -tag: NEW --- ## Deployment diff --git a/docs/cookbook/autoregressive/Meituan/LongCat-2.0.mdx b/docs/cookbook/autoregressive/Meituan/LongCat-2.0.mdx index b3db0f884307..e50cf10330b7 100644 --- a/docs/cookbook/autoregressive/Meituan/LongCat-2.0.mdx +++ b/docs/cookbook/autoregressive/Meituan/LongCat-2.0.mdx @@ -102,7 +102,7 @@ import { Playground } from "/src/snippets/_playground.jsx"; - **Remote code.** Use `--trust-remote-code` for the Hugging Face checkpoint. - **Topology.** The 8x B300 recipe uses TP=8 and EP=8. H200, B200, and H20 use a 2-node 16 GPU layout with TP=16 and EP=16; the command panel injects the multi-node rank flags for you. -- **LongCat sparse attention.** Keep `--nsa-prefill-backend fa3` with `--chunked-prefill-size 2048` for the model-card-aligned prefill path. +- **LongCat sparse attention.** Keep `--dsa-prefill-backend fa3` with `--chunked-prefill-size 2048` for the model-card-aligned prefill path. - **Memory.** The recipe uses `--kv-cache-dtype bfloat16` and starts at `--mem-fraction-static 0.92`. Tune memory only after the generated command launches cleanly on your cluster. - **Weight loading.** `--model-loader-extra-config '{"enable_multithread_load":true,"num_threads":12}'` loads checkpoint shards in parallel and reduces startup time. - **FP8 backend selection.** Do not pass `--fp8-gemm-runner-backend` manually. SGLang selects the correct backend for the LongCat FP8 scale layout. diff --git a/docs/cookbook/autoregressive/Meta/Llama3.1.mdx b/docs/cookbook/autoregressive/Meta/Llama3.1.mdx index 8827abf673ff..2bccc224eb5f 100644 --- a/docs/cookbook/autoregressive/Meta/Llama3.1.mdx +++ b/docs/cookbook/autoregressive/Meta/Llama3.1.mdx @@ -22,11 +22,9 @@ SGLang offers multiple installation methods. You can choose the most suitable in Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. -For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation). - ## 3. Model Deployment -This section provides deployment configurations optimized for different hardware platforms and use cases. +This section provides deployment configurations optimized for different hardware platforms including NVIDIA GPUs, AMD GPUs, Intel Arc Pro B-Series GPUs(codename: BMG (Battlemage)), and Intel Xeon CPUs. ### 3.1 Basic Configuration diff --git a/docs/cookbook/autoregressive/Meta/Llama3.3-70B.mdx b/docs/cookbook/autoregressive/Meta/Llama3.3-70B.mdx index 549ca1a8dffa..ebceaa55f0cd 100644 --- a/docs/cookbook/autoregressive/Meta/Llama3.3-70B.mdx +++ b/docs/cookbook/autoregressive/Meta/Llama3.3-70B.mdx @@ -24,15 +24,13 @@ For more details, please refer to the [official Llama models repository](https:/ Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. -For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation). - ## 3. Model Deployment -This section provides deployment configurations optimized for AMD GPUs (MI300X, MI325X, MI355X) and Intel Xeon CPUs. +This section provides deployment configurations optimized for AMD GPUs (MI300X, MI325X, MI355X), Intel Arc Pro B-Series GPUs(codename: BMG (Battlemage)) and Intel Xeon CPUs. ### 3.1 Interactive Configuration -**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your AMD GPU setup. +**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your AMD GPU, Intel Arc Pro B-Series GPUs or Intel Xeon CPUs setup. import { Llama33Deployment } from "/src/snippets/autoregressive/llama33-70b-deployment.jsx"; diff --git a/docs/cookbook/autoregressive/MiniMax/MiniMax-M2.7.mdx b/docs/cookbook/autoregressive/MiniMax/MiniMax-M2.7.mdx index aa357327a4c8..97296c7cfe52 100644 --- a/docs/cookbook/autoregressive/MiniMax/MiniMax-M2.7.mdx +++ b/docs/cookbook/autoregressive/MiniMax/MiniMax-M2.7.mdx @@ -487,30 +487,25 @@ This section uses **industry-standard configurations** for comparable benchmark ### 5.1 Accuracy Benchmark -**Evaluation Tool**: [NVIDIA NeMo-Skills](https://github.com/NVIDIA-NeMo/Skills) +**Evaluation Tool**: [sgl-eval](https://github.com/sgl-project/sgl-eval) -**Evaluation Settings**: temperature=0.6, top_p=0.95, 8 seeds, max_tokens=120,000, `parse_reasoning=True` +**Evaluation Settings**: temperature=0.6, top_p=0.95, 8 repeats, max_tokens=120,000 (GPQA and AIME; MMLU-Pro is greedy and single-pass, see below) #### 5.1.1 GPQA Diamond - Dataset: [GPQA Diamond](https://huggingface.co/datasets/Idavidrein/gpqa) (198 questions) -- Prompt: `eval/aai/mcq-4choices` (4-choice multiple choice, matching [Artificial Analysis methodology](https://artificialanalysis.ai/methodology/intelligence-benchmarking)) +- Prompt: `eval/aai/mcq-4choices` (4-choice multiple choice, matching [Artificial Analysis methodology](https://artificialanalysis.ai/methodology/intelligence-benchmarking)) -- sgl-eval's default for this benchmark - Evaluation command: ```bash Command -ns prepare_data gpqa - -ns eval \ - --cluster=local \ - --server_type=openai \ - --model=MiniMaxAI/MiniMax-M2.7 \ - --server_address=http://localhost:30000/v1 \ - --output_dir=./m2.7-eval/ \ - --benchmarks=gpqa:8 \ - ++prompt_config=eval/aai/mcq-4choices \ - ++inference.tokens_to_generate=120000 \ - ++inference.temperature=0.6 \ - ++inference.top_p=0.95 \ - ++parse_reasoning=True +pip install sgl-eval + +sgl-eval run gpqa \ + --base-url http://localhost:30000/v1 \ + --model MiniMaxAI/MiniMax-M2.7 \ + --n-repeats 8 \ + --max-tokens 120000 \ + --temperature 0.6 \ + --top-p 0.95 ``` - Test Results: @@ -544,22 +539,16 @@ ns eval \ #### 5.1.2 AIME 2025 - Dataset: AIME 2025 (30 problems) -- Prompt: `generic/math` (boxed answer format) +- Prompt: `generic/math` (boxed answer format) -- sgl-eval's default for this benchmark - Evaluation command: ```bash Command -ns prepare_data aime25 - -ns eval \ - --cluster=local \ - --server_type=openai \ - --model=MiniMaxAI/MiniMax-M2.7 \ - --server_address=http://localhost:30000/v1 \ - --output_dir=./m2.7-eval/ \ - --benchmarks=aime25:8 \ - ++inference.tokens_to_generate=120000 \ - ++inference.temperature=0.6 \ - ++inference.top_p=0.95 \ - ++parse_reasoning=True +sgl-eval run aime25 \ + --base-url http://localhost:30000/v1 \ + --model MiniMaxAI/MiniMax-M2.7 \ + --n-repeats 8 \ + --max-tokens 120000 \ + --temperature 0.6 \ + --top-p 0.95 ``` - Test Results: @@ -593,22 +582,14 @@ ns eval \ #### 5.1.3 MMLU-Pro - Dataset: [MMLU-Pro](https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro) (12,032 questions, 10-choice) -- Prompt: `eval/aai/mcq-10choices` (10-choice multiple choice) +- Prompt: `eval/aai/mcq-10choices` (10-choice multiple choice) -- sgl-eval's default for this benchmark - Evaluation command: ```bash Command -ns prepare_data mmlu-pro - -ns eval \ - --cluster=local \ - --server_type=openai \ - --model=MiniMaxAI/MiniMax-M2.7 \ - --server_address=http://localhost:30000/v1 \ - --output_dir=./m2.7-eval/ \ - --benchmarks=mmlu-pro \ - ++prompt_config=eval/aai/mcq-10choices \ - ++inference.tokens_to_generate=32768 \ - ++inference.temperature=0.0 \ - ++parse_reasoning=True +sgl-eval run mmlu_pro \ + --base-url http://localhost:30000/v1 \ + --model MiniMaxAI/MiniMax-M2.7 \ + --max-tokens 32768 \ + --temperature 0.0 ``` - Test Results: diff --git a/docs/cookbook/autoregressive/MiniMax/MiniMax-M3.mdx b/docs/cookbook/autoregressive/MiniMax/MiniMax-M3.mdx index 8326f3dc83cf..bfdb5af6c5cf 100644 --- a/docs/cookbook/autoregressive/MiniMax/MiniMax-M3.mdx +++ b/docs/cookbook/autoregressive/MiniMax/MiniMax-M3.mdx @@ -39,7 +39,7 @@ Then run the **Python** output of the command panel below in that environment. T docker pull lmsysorg/sglang:dev-cu13-minimax-m3 ``` -The command panel below fills in the right tag per platform: `dev-cu13-minimax-m3` (CUDA 13 — B300, GB200, GB300), `dev-cu12-minimax-m3` (CUDA 12 — Hopper H200), or `dev-minimax-m3` (default). On AMD Instinct it uses the matching ROCm image (MI300X/MI325X → `aigmkt/minimax-m3-sglang-rocm700-mi30x`, MI350X/MI355X → `aigmkt/minimax-m3-sglang-rocm720-mi35x`). For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker), substituting the inner `sglang serve ...` with what the command generator produces. +The command panel below fills in the right tag per platform: `dev-cu13-minimax-m3` (CUDA 13 — B300, GB200, GB300), `dev-cu12-minimax-m3` (CUDA 12 — Hopper H200; final build, the CUDA 12 lane is retired), or `dev-minimax-m3` (default). On AMD Instinct it uses the matching ROCm image (MI300X/MI325X → `aigmkt/minimax-m3-sglang-rocm700-mi30x`, MI350X/MI355X → `aigmkt/minimax-m3-sglang-rocm720-mi35x`). For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker), substituting the inner `sglang serve ...` with what the command generator produces. These M3 dev images now **bundle MiniMax's MSA sparse-attention kernel** (`fmha_sm100`), so Blackwell users get the recommended fast path automatically — no manual install needed (see **§2.1**). On a custom image without it, the same recipe still serves on the built-in Triton sparse path. @@ -129,6 +129,7 @@ The NVIDIA Blackwell recipes are validated single-node: **B200 at `--tp 8`** and - **Memory**: `--mem-fraction-static` reserves GPU memory for weights + KV pool; the rest is prefill **activation headroom**. The value scales with *free* memory per GPU (card capacity minus per-GPU weight), so it tracks the card more than the TP degree: **`0.65` on B200** (180 GB — less headroom once weights are resident) and **`0.75` on the larger-memory B300 / GB300** (`0.80` on AMD). Lower TP packs more weight per GPU, so a tighter config needs a *lower* value — B200 needs `0.65` even at `--tp 4`. Raising it past the validated value is fine only for low-concurrency single-stream serving; it OOMs under high concurrency or long context. - **Long context (32K+)**: keep `--mem-fraction-static` at the platform default and raise `--chunked-prefill-size` to `16384`. Decode TPOT stays roughly flat in context length thanks to sparse attention; 1K–128K prompts are validated. +- **HiSparse for decode capacity**: on NVIDIA CUDA, HiSparse keeps the three dense layers on GPU, moves the 57 sparse-layer K/V caches to pinned host memory, and feeds selected block IDs directly to the swap-in kernel. For the released four-KV-head model, use `--tp 4` or greater, `--disable-radix-cache`, and `device_buffer_size >= 2048`. Enable it with `--enable-hisparse --hisparse-config='{"device_buffer_size":4096,"host_to_device_ratio":2}'` on the Triton launch command. - **Scaling TP**: B200 is documented at `--tp 8`; B300 / GB200 / GB300 at `--tp 4` (the single-node cross-family common denominator). On an 8-GPU B300 host you can also raise to `--tp 8` for more throughput / KV headroom. - **Expert parallelism**: to trade latency for throughput add `--ep` (see [Expert Parallelism Deployment](../../../docs/advanced_features/expert_parallelism)). On AMD, set `--ep` equal to `--tp`. Shared-experts fusion is automatically disabled when EP > 1; on AMD standard EP the server also disables `--enable-aiter-allreduce-fusion` automatically to preserve accuracy. - `--trust-remote-code` is required to load the MiniMax config / processor classes. diff --git a/docs/cookbook/autoregressive/Moonshotai/Kimi-K2.6.mdx b/docs/cookbook/autoregressive/Moonshotai/Kimi-K2.6.mdx index 178a7c1f755e..696859a10180 100644 --- a/docs/cookbook/autoregressive/Moonshotai/Kimi-K2.6.mdx +++ b/docs/cookbook/autoregressive/Moonshotai/Kimi-K2.6.mdx @@ -621,32 +621,25 @@ python tool_calls_eval.py tool-calls/samples.jsonl \ #### 5.1.2 AIME 2025 - Dataset: [AIME 2025](https://huggingface.co/datasets/nvidia/aime25) (30 problems) -- Evaluation Tool: [NVIDIA NeMo-Skills](https://github.com/NVIDIA/NeMo-Skills) +- Evaluation Tool: [sgl-eval](https://github.com/sgl-project/sgl-eval) - Prompt: `eval/matharena/aime` (MathArena format with `\boxed{}` answers) -- Settings: temperature=1.0, top_p=0.95, max_tokens=131,072, 32 seeds +- Settings: temperature=1.0, top_p=0.95, max_tokens=131,072, 32 repeats **Evaluation Command:** ```shell Command -# Prepare dataset -python3 nemo_skills/dataset/aime25/prepare.py - -# Run 32 seeds in parallel -for RS in $(seq 0 31); do - python3 nemo_skills/inference/generate.py \ - input_file=nemo_skills/dataset/aime25/test.jsonl \ - output_file=results/kimi-k26/aime25/output-rs${RS}.jsonl \ - prompt_config=eval/matharena/aime \ - prompt_format=openai \ - +server.server_type=openai \ - +server.model=moonshotai/Kimi-K2.6 \ - +server.base_url=http://localhost:30000/v1 \ - ++inference.temperature=1.0 \ - ++inference.top_p=0.95 \ - ++inference.tokens_to_generate=131072 \ - ++inference.random_seed=${RS} \ - max_concurrent_requests=512 & -done +pip install sgl-eval + +# The dataset ships with sgl-eval; --n-repeats runs all 32 samples per problem. +sgl-eval run aime25 \ + --base-url http://localhost:30000/v1 \ + --model moonshotai/Kimi-K2.6 \ + --n-repeats 32 \ + --temperature 1.0 \ + --top-p 0.95 \ + --max-tokens 131072 \ + --prompt matharena-aime \ + --num-threads 512 ``` **Results:** diff --git a/docs/cookbook/autoregressive/Moonshotai/Kimi-K3.mdx b/docs/cookbook/autoregressive/Moonshotai/Kimi-K3.mdx index 11fe7f8f1b37..b6dd00264aea 100644 --- a/docs/cookbook/autoregressive/Moonshotai/Kimi-K3.mdx +++ b/docs/cookbook/autoregressive/Moonshotai/Kimi-K3.mdx @@ -30,7 +30,7 @@ Then run the **Python** output of the command panel below in that environment. ```bash Command docker pull lmsysorg/sglang:latest # NVIDIA (CUDA) -docker pull lmsysorg/sglang-rocm:v0.5.17-rocm720-mi35x-20260817 # AMD MI350X / MI355X (ROCm) +docker pull lmsysorg/sglang-rocm:v0.5.19-rocm720-mi35x-20260916 # AMD MI350X / MI355X (ROCm) ``` For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. @@ -40,14 +40,16 @@ For how to launch the image, see [Install → Method 3: Using Docker](../../../d ```bash Command -docker pull quay.io/ascend/sglang:main-cann9.0.0-a3 +docker pull quay.io/ascend/sglang:main-cann9.0.0-a3 # Ascend A3 Series +docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann9.1.0-950-B070 # Ascend 950PR/DT Series ``` For host and platform setup, see the [NPU installation guide](../../../docs/hardware-platforms/ascend-npus/getting-started/installation) and the [quick start guide](../../../docs/hardware-platforms/ascend-npus/getting-started/quick_start). -**Weights (NPU):** [Kimi-K3-W4A8](https://www.modelscope.cn/models/sgl-npu/Kimi-K3-W4A8) (W4A8, 1.49 TB) · +**Weights (NPU):** [Kimi-K3](https://modelscope.cn/models/moonshotai/Kimi-K3) (950PR/DT Series, MXFP4) · +[Kimi-K3-W4A8](https://www.modelscope.cn/models/sgl-npu/Kimi-K3-W4A8) (A3 Series, W4A8, 1.49 TB) · [Kimi-K3-DSpark](https://www.modelscope.cn/models/RadixArk/Kimi-K3-DSpark) (DSPARK draft, 4.5 GB) @@ -56,22 +58,22 @@ For host and platform setup, see the -Pick your hardware, then the deployment shape and operating point. Node count follows the hardware recipe (B200 2×8, GB200 4×4, H100 4×8, B300 1×8, H200 2×8 — 4×8 on Unified High-Throughput, GB300 2×4, MI350X/MI355X 1×8, Atlas 800I A3 4×8 — 32 cards / 64 dies), so it is not a separate choice. If you serve the NVFP4 checkpoint (`nvidia/Kimi-K3-NVFP4`, the **Quantization** row in the panel below), use the `lmsysorg/sglang:dev-dev-kimi-k3-nvfp4` image. +Pick your hardware, then the deployment shape and operating point. Node count follows the hardware recipe (B200 2×8, GB200 4×4, H100 4×8, B300 1×8, H200 2×8 — 4×8 on Unified High-Throughput, GB300 2×4, MI350X/MI355X 1×8, Ascend A3 Series 4×8 — 32 cards / 64 dies, Ascend 950PR/DT Series 4×8), so it is not a separate choice. If you serve the NVFP4 checkpoint (`nvidia/Kimi-K3-NVFP4`, the **Quantization** row in the panel below), use the `lmsysorg/sglang:dev-dev-kimi-k3-nvfp4` image. **PD Mode** — `Unified` serves prefill and decode together. `Prefill` / `Decode` split them into dedicated pools (see [PD disaggregation](#3-4-pd-disaggregation)); `Prefill` ships two strategies, both chunked at 16k. On the 8-GPU platforms (B300 1×8, GB300 2×4), `Default` is TP8 and `Long-Context` is `--pp-size 8 --tp-size 1`. On the 16-GPU platforms (B200 2×8, GB200 4×4), both are `--pp-size 16 --tp-size 1` and differ only in `--mem-fraction-static` (0.85 vs 0.90) — deep PP is the throughput shape there, not just the long-context one (see [Deep PP](#deep-pp-for-prefill)). **Strategy** — the operating point within that shape: - **Low-Latency** — no DCP, so the MLA KV stays TP-replicated. For chat. B200 splits its two nodes into PP2 × TP8; every other platform is flat TP. -- **Balanced** — the accuracy-preserving default: PP2 × DCPEP8 on B200 (the two pipeline stages and DCP8 split KV and KDA state), TP16/DCP16 on GB200, TP8/DCP8 on B300/GB300, TP8 ROCm/AITER on MI35x. +- **Balanced** — the accuracy-preserving default: PP2 × DCPEP8 on B200 (the two pipeline stages and DCP8 split KV and KDA state), TP16/DCP16 on GB200, TP8/DCP8 on B300/GB300, TP8/DCP8 ROCm/AITER on MI35x. - **High-Throughput** — the large-scale lane: pick a **Cluster Size** and **Large-Scale Preset** in the Playground ([details](#large-scale-presets)). The cell itself is Balanced, except on H100 (plus `extra_buffer_lazy`) and H200 (widens to 4×8 TP32/EP32 at `--mem-fraction-static 0.90`). `Long-Context` appears only under the `Prefill` PD mode; for long-context unified serving on B200, start from High-Throughput and raise `--context-length`. -**Spec Decode** — layers onto the strategy without changing it, on every platform except B200. DSPARK proposes 7 draft tokens per step (tune in the Playground) and requires `pp_size == 1`, so on B200 it also drops the pipeline and re-lays the same 16 GPUs flat: PP2 × TP8 → TP16, PP2 × DCPEP8 → DCPEP16. DFLASH has no published draft checkpoint. The win is largest on short interactive traffic and fades as the prompt grows. +**Spec Decode** — layers onto the strategy without changing it, on every platform except B200. DSPARK proposes 7 draft tokens per step (tune in the Playground) and requires `pp_size == 1`, so on B200 it also drops the pipeline and re-lays the same 16 GPUs flat: PP2 × TP8 → TP16, PP2 × DCPEP8 → DCPEP16. DFLASH uses the [Kimi-K3-DFlash](https://huggingface.co/modal-labs/Kimi-K3-DFlash) draft at block size 8; it doesn't support pipeline parallelism or DP attention yet, so it is unavailable on the pipelined B200 Unified recipes. It has only been validated on Blackwell, so the Hopper and AMD recipes don't offer it yet. The win is largest on short interactive traffic and fades as the prompt grows. -`--mamba-full-memory-ratio` is the one sizing flag, computed live: set your average request length in the [Mamba ratio calculator](#mamba-ratio-calculator); everything else follows the panels, and the result is pinned into the command. (The Atlas 800I A3 uses `--max-mamba-cache-size` instead.) +`--mamba-full-memory-ratio` is the one sizing flag, computed live: set your average request length in the [Mamba ratio calculator](#mamba-ratio-calculator); everything else follows the panels, and the result is pinned into the command. (The Ascend NPU recipes don't use the calculator and set no ratio at all — the Ascend path sizes both pools itself: the A3 Series pins `--max-mamba-cache-size` instead, and the A5 recipe serves with the radix cache off.) import { Deployment } from "/src/snippets/_deployment.jsx"; @@ -81,6 +83,32 @@ import { KimiK3MambaRatioCalculator } from "/src/snippets/_kimi_k3_mamba_ratio_c + + B300 1×8 Unified speed numbers are measured on `v0.5.18 @ 71de97b2` with `--random-range-ratio 1.0`, + `--warmup-requests 64`, `--flush-cache`, at ISL 8192 / OSL 1024. DSPARK cells pin the acceptance + length via the serve env `SGLANG_SIMULATE_ACC_LEN=4.5` — they report what block size 7 delivers at + that acceptance, not a measured acceptance rate for this workload. Balanced DSPARK adds + `--max-running-requests 256`; without it speculation resets the cap to 48. The KDA state pool still + clamps admission below that (101 / 68 / 91 / 60 concurrent requests for MXFP4 NOSPEC / MXFP4 DSPARK / + NVFP4 NOSPEC / NVFP4 DSPARK), which is why no point past concurrency 64 is published for Balanced. + + +### AMD AITER with DCP8 + +The MI350X/MI355X unified Balanced recipe uses TP8/DCP8 with AITER prefill and +decode attention. DCP shards the target MLA KV cache; RadixArk DSPARK's draft KV +remains replicated. The pinned `v0.5.19-rocm720-mi35x-20260916` image records +SGLang revision `e7f7447333`, which includes +[AITER DCP support (#34432)](https://github.com/sgl-project/sglang/pull/34432) and +the [DCP KV-free fix (#38941)](https://github.com/sgl-project/sglang/pull/38941). +No source overlay is required for DCP. + +Keep `SGLANG_K3_KDA_FUSED_BACKEND` unset with this image. The separate fused-KDA +opt-in requires the [deferred-gate fix (#39066)](https://github.com/sgl-project/sglang/pull/39066), +which is not included in this image. This updated recipe remains **Final +Verification In Progress**; the recorded speed numbers use their original +configurations and do not validate the new image or DCP8 recipe. + ### Mamba ratio calculator @@ -124,9 +152,10 @@ are scheduled to release by July 27, 2026**. The recipes on this page were valid repository (`moonshotai/Kimi-K3`) and a public `lmsysorg/sglang` image with K3 support will be available at launch. -Every cell in the Deploy panel above is currently marked **Final Verification In Progress**: the -recipe runs, but its serving round on the final weights and current code is still open. Re-measure -throughput and accuracy before you rely on any of them. +The B300 1×8 `Unified` Low-Latency and Balanced cells are **Verified** — a speed round on the final +weights is published below. Every other cell is still marked **Final Verification In Progress**: the +recipe runs, but its serving round on the final weights and current code is still open. Accuracy has +not been re-measured on any cell — re-measure before you rely on one. **Recommended generation:** `temperature=1.0`, `top_p=0.95`, `presence_penalty=0`, `frequency_penalty=0` (fixed by the model; informational — do not hardcode in sample code). @@ -135,7 +164,7 @@ throughput and accuracy before you rely on any of them. ## 2. Configuration Tips -**Memory: two pools, one flag.** K3 splits static memory into a worst-case-reserved **KDA state pool** (it sets the concurrency ceiling) and a paged **MLA KV pool**, divided by `--mamba-full-memory-ratio`. The command panel pins that flag to the [calculator](#mamba-ratio-calculator)'s output — set your average request length there; every other calculator input follows the panels. (On the Atlas 800I A3: `--max-mamba-cache-size`, no calculator.) After boot, read back `max_total_num_tokens` (the KV side) and the admitted-request cap (the state side). +**Memory: two pools, one flag.** K3 splits static memory into a worst-case-reserved **KDA state pool** (it sets the concurrency ceiling) and a paged **MLA KV pool**, divided by `--mamba-full-memory-ratio`. The command panel pins that flag to the [calculator](#mamba-ratio-calculator)'s output — set your average request length there; every other calculator input follows the panels. (On the Ascend NPU recipes there is no calculator and no ratio flag: the Ascend path sizes both pools itself — the A3 Series pins `--max-mamba-cache-size`, and the A5 recipe runs the radix cache off.) After boot, read back `max_total_num_tokens` (the KV side) and the admitted-request cap (the state side). Capacity levers, all in the Playground. Each trades precision or cache behavior for capacity — re-verify accuracy on your workload: @@ -149,13 +178,13 @@ Capacity levers, all in the Playground. Each trades precision or cache behavior Speculation: DSPARK holds block size + 1 (= 8) intermediate states per request — the calculator folds this in — and an unset `--max-running-requests` resets to 48 under spec (the command panel reminds you; set it explicitly to raise). -**MoE runner.** Leave `--moe-runner-backend` unset on Blackwell: FlashInfer MXFP4 (W4A8, official trtllm-gen SiTU kernels) is selected with the pinned FlashInfer 0.6.17 dependency; H100/H200 pin Marlin. The B200 Balanced and High-Throughput cells pin `flashinfer_mxfp4` explicitly because that is the shape they were brought up on. The published Docker images install the matching official `flashinfer-python`, `flashinfer-cubin`, and `flashinfer-jit-cache` packages. +**MoE runner.** Leave `--moe-runner-backend` unset on Blackwell: FlashInfer MXFP4 (W4A8, official trtllm-gen SiTU kernels) is selected; H100/H200 pin Marlin. The B200 Balanced and High-Throughput cells pin `flashinfer_mxfp4` explicitly because that is the shape they were brought up on. **Attention backend.** Leave all three attention knobs unset on Blackwell: K3 resolves prefill, decode, and — under DSPARK — verification as a set (`trtllm_mla` across the board; `cutedsl_mla` takes decode and verification under DCP). On the non-DCP recipes, setting any one of the three cancels the auto-resolution for the others. The B200 Balanced and High-Throughput cells pin `--decode-attention-backend cutedsl_mla`, which is what auto-resolution picks for those DCP recipes anyway — it is written out because it is the shape they were brought up on, not because it changes the resolution. H100/H200 pin `flashmla` for decode. **Context length.** `--context-length` bounds the longest accepted request plus some context-scaled buffers; it does not size the KV pool. For long context the lever that adds capacity is `fp8_e4m3` KV. -**DSPARK.** Adds `--speculative-algorithm DSPARK` plus the draft checkpoint on top of the showing strategy. Leave `--speculative-draft-attention-backend` unset. No serving round on the final draft checkpoint has landed — measure against the same recipe running NOSPEC before adopting. +**DSPARK.** Adds `--speculative-algorithm DSPARK` plus the draft checkpoint on top of the showing strategy. Leave `--speculative-draft-attention-backend` unset. The published B300 DSPARK numbers pin the acceptance length with `SGLANG_SIMULATE_ACC_LEN`, so no measured acceptance rate exists for a real workload yet — measure against the same recipe running NOSPEC before adopting. **Per-platform notes:** @@ -167,10 +196,11 @@ Speculation: DSPARK holds block size + 1 (= 8) intermediate states per request | GB200 4×4 | TP16/DCP16 | MNNVL auto-detected | | H200 2×8 (4×8 on Unified High-Throughput) | TP16/EP16 + symm-mem, Marlin + FlashMLA; High-Throughput widens to TP32/EP32 over 4 nodes at mem-frac 0.90 with `extra_buffer_lazy` | same block on every node; export the cross-node NIC (`GLOO_SOCKET_IFNAME` / `NCCL_SOCKET_IFNAME`, `SGLANG_HOST_IP`); keep `NCCL_MNNVL_ENABLE=1 NCCL_CUMEM_ENABLE=1` | | H100 4×8 | TP32/EP32, Marlin + FlashMLA | SM90a build of the K3 image; pin NCCL/Gloo to the same NIC on all nodes; least post-weight headroom (80 GB) | -| MI350X/MI355X 1×8 | TP8 ROCm/AITER | AITER A8W4 FlyDSL MoE, Triton attention (`SGLANG_MLA_DECODE_TUNE=1` for gfx950 MLA decode geometry), graph bs up to 256, fp8 kvcache; DSPARK supported | -| Atlas 800I A3 4×8 (32 cards / 64 dies) | TP64/DP4 + DeepEP | PD-mixed `Unified` only; DSPARK baked in; pin `GLOO`/`HCCL_SOCKET_IFNAME` on every node | +| MI350X/MI355X 1×8 | TP8/DCP8 ROCm/AITER (Unified Balanced) | AITER A8W4 FlyDSL MoE, AITER prefill/decode attention with sharded target MLA KV, graph bs up to 256, fp8 kvcache; DSPARK supported. Activation-quant and fused-KDA-decode knobs: [AMD ROCm/AITER environment](#amd-env) | +| Ascend A3 Series 4×8 (32 cards / 64 dies) | TP64/DP4 + DeepEP | PD-mixed `Unified` only; DSPARK baked in; pin `GLOO`/`HCCL_SOCKET_IFNAME` on every node | +| Ascend 950PR/DT Series 4×8 | TP32/dp1 + DeepEP | PD-mixed `Unified` only; DSPARK baked in; shared experts / dense MLP shard over attention-TP (`--shared-experts-tp-size 4`); radix cache off; pin `GLOO`/`HCCL_SOCKET_IFNAME` on every node | -**DCP notes** — the DCP cells are Balanced and High-Throughput on every Blackwell platform, in both the `Unified` and `Decode` roles: +**Blackwell DCP notes** — the DCP cells are Balanced and High-Throughput on every Blackwell platform, in both the `Unified` and `Decode` roles: - DCP is the only axis that shards the TP-replicated MLA KV; Low-Latency skips it. - Leave `--dcp-comm-backend` unset (fabric-resolved: `fi_a2a` on GB200/GB300, `a2a` on B200/B300). @@ -179,7 +209,20 @@ Speculation: DSPARK holds block size + 1 (= 8) intermediate states per request - Calculator ratios run well above 1 here (`r > 1` is legal): `bfloat16` state buys admission, `fp8` KV buys context. - Don't use EP with an a2a backend: a2a buffers reclaim the KV that DCP buys. Compose only to measure. a2a backend is set when `--moe-a2a-backend` is set. -No cell has a serving round in this exact shape — treat them as starting points to verify. +Outside the two verified B300 1×8 `Unified` cells, no cell has a serving round in this exact shape — treat those as starting points to verify. + + + +**AMD ROCm/AITER environment (MI350X / MI355X).** The MI35x cell emits `SGLANG_USE_AITER=1 SGLANG_AITER_K3_OPT=1 AITER_FLYDSL_FORCE=1 AITER_SITUV2_A8W4=1` — the first three turn on the AITER ROCm path, its K3-specific fused kernels, and the FlyDSL MoE kernels; the rest of this table is what you can change on top. Everything here is gfx950/ROCm-only and inert elsewhere. These knobs first ship in the `20260903` daily ROCm image pinned above — on an older image they are simply unread — and the two SiTU rows also need an AITER revision at or past [ROCm/aiter#4534](https://github.com/ROCm/aiter/pull/4534) (FlyDSL 0.3.0). + +| Env var | Default | Effect | +|---|---|---| +| `AITER_SITUV2_A8W4=1` | unset | SiTU v2 MoE with A8W4 activation quantization, on AITER's GU-interleaved preshuffled weight layout. The performance default the cell ships. | +| `AITER_SITUV2_A4W4=1` | unset | A4W4 instead, on the generic separated shuffle layout. Numerically correct but slower than A8W4 (530.8 vs 537.3 tok/s median output on 8×MI35x). Setting **both** gives A8W4 precedence — SGLang follows AITER and keeps the GU-interleaved layout. | +| `SGLANG_K3_KDA_FUSED_BACKEND=aiter` | unset | Opt in to the fused ROCm KDA decode boundary: the `f_b` projection is deferred into the gfx950 FlyDSL kernel so decode fuses `f_b` + convolution + recurrent state update + gated RMSNorm. Any other value (or unset) keeps the unfused KDA path. | +| `SGLANG_K3_FLYDSL_SOURCE` | `auto` | Which FlyDSL implementation backs the fused decode: `auto` prefers SGLang's vendored kernels and falls back to the AITER module, `sglang` / `aiter` pin one. Leave it alone unless you are bisecting the two. | + +The fused KDA backend is fail-closed at two levels: it arms during model init only when the flag is exactly `aiter` **and** the gfx950 FlyDSL kernels are importable, and each decode step re-validates shapes, dtypes, strides, state indices, and output buffers before dispatch — anything unexpected falls back to the stock KDA implementation rather than erroring. Batch size 2 automatically picks a separately validated kernel schedule; every other batch keeps the original build options. Measured on a 69-layer graph the fused boundary is 9.20 → 8.38 µs/layer (−8.9%), with GSM8K 1319 at 0.950. ## 3. Advanced Usage @@ -215,7 +258,7 @@ Pending update... ### 3.2 Tool Calling -Enable the `kimi_k3` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`. Because K3 is a thinking model, the follow-up turn may put text in `reasoning_content` as well as `content` — print both. (Not yet supported on the Atlas 800I A3.) +Enable the `kimi_k3` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`. Because K3 is a thinking model, the follow-up turn may put text in `reasoning_content` as well as `content` — print both. (Not yet supported on the Ascend NPU recipes — A3 and A5.) @@ -259,7 +302,8 @@ Pending update... K3's hybrid HiCache tiers the paged MLA KV **and** the KDA/mamba state across L1 (GPU) / L2 (host) / L3 (Mooncake) — enable it from the **HiCache** card in the [Playground above](#playground) for long multi-turn workloads. -- On the DCP recipes (Blackwell Balanced / High-Throughput, in both the `Unified` and `Decode` roles), the host tiers are not fully DCP-aware yet: **L3 always, and L1+L2 with Spec Decode on, drop the DCP flags** (the command hints call it out — per-request KV capacity shrinks accordingly). L1+L2 with Spec Decode off keeps DCP. Only DCP goes: the MLA KV reverts to TP-replicated, but the cell's other parallelism stays, so B300/GB300/GB200 land on plain TP while B200 Unified keeps its `--pp-size 2` / `--ep-size`. +- On the DCP recipes (Blackwell Balanced / High-Throughput, `Unified` and `Decode` roles), **L1+L2 keeps DCP**, with Spec Decode off or on DSPARK. On B200 the DSPARK pipeline collapse scales it too (PP2 × DCPEP8 → DCPEP16). +- **L3 drops the DCP flags** on those recipes (storage keys are not dcp_rank-aware yet; the command hints call it out, and per-request KV capacity shrinks). Only DCP goes: the MLA KV reverts to TP-replicated while the cell's other parallelism stays, so B300/GB300/GB200 land on plain TP and B200 Unified keeps its `--pp-size 2` / `--ep-size`. - Low-Latency and the Hopper recipes take all tiers unchanged. diff --git a/docs/cookbook/autoregressive/NVIDIA/Nemotron3-Nano.mdx b/docs/cookbook/autoregressive/NVIDIA/Nemotron3-Nano.mdx index ee60b6c92c48..7ec09ef40204 100644 --- a/docs/cookbook/autoregressive/NVIDIA/Nemotron3-Nano.mdx +++ b/docs/cookbook/autoregressive/NVIDIA/Nemotron3-Nano.mdx @@ -31,6 +31,8 @@ This section provides a progressive guide from quick deployment to performance t ### 3.1 Basic Configuration +The Nemotron3-Nano series offers models in various sizes and architectures, optimized for different hardware platforms including NVIDIA GPUs and Intel Arc Pro B-Series GPUs(codename: BMG (Battlemage)). + **Interactive Command Generator**: select hardware, model variant, and common knobs to generate a launch command. diff --git a/docs/cookbook/autoregressive/OpenBMB/MiniCPM-V-4_6.mdx b/docs/cookbook/autoregressive/OpenBMB/MiniCPM-V-4_6.mdx index 5d68802ab318..55a788b4933f 100644 --- a/docs/cookbook/autoregressive/OpenBMB/MiniCPM-V-4_6.mdx +++ b/docs/cookbook/autoregressive/OpenBMB/MiniCPM-V-4_6.mdx @@ -2,7 +2,6 @@ title: MiniCPM-V 4.6 metatags: description: "Deploy OpenBMB MiniCPM-V 4.6 (Qwen3.5-style hybrid GDN backbone + NaViT vision encoder) on NVIDIA GPUs with SGLang — multimodal text + image + video, slicing for high-resolution images." -tag: NEW --- ## 1. Model Introduction @@ -33,7 +32,8 @@ Pull the nightly Docker image (rolling tag, tracks `main`): # CUDA 13 (Hopper / Blackwell, default) docker pull lmsysorg/sglang:dev -# CUDA 12 (Ampere or older drivers) +# CUDA 12 (Ampere or older drivers). Final build: the CUDA 12 lane is retired, +# so this tag no longer tracks main. docker pull lmsysorg/sglang:dev-cu12 ``` diff --git a/docs/cookbook/autoregressive/OpenBMB/MiniCPM5-2B.mdx b/docs/cookbook/autoregressive/OpenBMB/MiniCPM5-2B.mdx new file mode 100644 index 000000000000..da701717ba5d --- /dev/null +++ b/docs/cookbook/autoregressive/OpenBMB/MiniCPM5-2B.mdx @@ -0,0 +1,220 @@ +--- +title: MiniCPM5-2B +description: "Deploy OpenBMB MiniCPM5-2B with SGLang — a 2.5B dense Llama-architecture on-device model with 131K context, thinking mode, XML tool calling and DSpark speculative decoding on H200, RTX PRO 6000, RTX 5090 and DGX Spark." +tag: NEW +--- + +## Deployment + + + + + +For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel. + + + + + +```bash Command +pip install --upgrade pip +pip install uv +uv pip install --prerelease=allow "sglang>=0.5.12" +``` + +Then run the **Python** output of the command panel below in that environment. + + + + + +```bash Command +docker pull lmsysorg/sglang:dev +``` + +For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. + + + + + + + +Pick your card to generate the launch command. MiniCPM5-2B is a 2.5B dense model and runs single-GPU at TP=1 on every supported card, so the page ships one operating point per card plus an optional **DSpark** speculative-decoding row. + +import { Deployment } from "/src/snippets/_deployment.jsx"; +import { config } from "/src/snippets/configs/openbmb/minicpm5-2b.jsx"; +import { benchmarks } from "/src/snippets/configs/openbmb/minicpm5-2b-benchmarks.jsx"; + + + + + Speed numbers exist for the RTX 5090 and DGX Spark cells — a single card at + random 1024/1024, recorded as Mean. The H200 and RTX PRO 6000 cells are + pending measurement, and no accuracy numbers have been taken on any platform + yet. The DSpark overlay carries no speed numbers on any card. + + +## Playground + +The Playground is where you experiment with **SGLang features beyond the recipes above**. The Deploy panel emits this model's documented launch recipes; the Playground lets you turn on additional knobs on top of whichever cell the Deploy panel is currently showing. + +import { Playground } from "/src/snippets/_playground.jsx"; + + + +## 1. Model Introduction + +**MiniCPM5-2B** is the second model in OpenBMB's MiniCPM5 series, following MiniCPM5-1B. It is a dense 2B-class Transformer built for on-device assistants, local deployment, coding agents and tool-use workflows — scenarios where a compact model with a small deployment footprint is preferred. OpenBMB reports 2B-class open-source SOTA within its comparison set (average 53.9), with its clearest advantages in code reasoning, math reasoning, long-context understanding, tool use and agentic tasks. + +The checkpoint uses the standard `LlamaForCausalLM` architecture — no custom kernels and no model-code fork — so SGLang loads it through the stock Llama path. It is 2,516,756,480 parameters (1,981,982,720 non-embedding) over 42 layers with GQA (16 query heads, 2 KV heads), and a native context length of 131,072 tokens. Post-training runs SFT → RL → On-Policy Distillation, which merges 16 RL expert models into the single released checkpoint. Weights are released under the [Apache-2.0](https://github.com/OpenBMB/MiniCPM/blob/main/LICENSE) license. + + + + + + + + + + + + + + + + + + + + + + + + + + +
CheckpointPrecisionRole on this page
openbmb/MiniCPM5-2BBF16The served model in every cell above (final release, post-trained with RL + OPD).
openbmb/MiniCPM5-2B-DSparkBF16 draftDraft model loaded by the DSPARK row of the Deploy panel; not served on its own.
+ +OpenBMB also publishes SFT-only, mid-training and base checkpoints, plus GGUF, MLX and GPTQ-Int4 exports for llama.cpp / Ollama / LM Studio / Apple Silicon. Those target other runtimes and are not part of the SGLang matrix above. + +**Recommended generation:** `temperature=1.0`, `top_p=0.95` (informational — SGLang reads the checkpoint's `generation_config.json`; do not hardcode these in client code). + +**Resources:** [HuggingFace](https://huggingface.co/openbmb/MiniCPM5-2B) · [ModelScope](https://www.modelscope.cn/models/OpenBMB/MiniCPM5-2B) · [GitHub](https://github.com/OpenBMB/MiniCPM) · [Tech report](https://arxiv.org/pdf/2506.07900). + +## 2. Advanced Usage + +### 2.1 Thinking Mode + +Thinking is controlled by the chat template's `enable_thinking` flag, passed per request through `chat_template_kwargs`. The `qwen3` reasoning parser — already in every generated command, and toggleable from the **Parsers** card in the [Playground above](#playground) — splits the `` segment into `message.reasoning_content` and leaves the final answer in `message.content`. Drop the flag and the thinking text stays inline in `content`, closing tag and all. + + + +```python Example +from openai import OpenAI + +client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") + +# Thinking on (chat-template default). +resp = client.chat.completions.create( + model="openbmb/MiniCPM5-2B", + messages=[{"role": "user", "content": "What is 15% of 240?"}], + extra_body={"chat_template_kwargs": {"enable_thinking": True}}, +) +msg = resp.choices[0].message +print("Reasoning:", getattr(msg, "reasoning_content", None)) +print("Answer:", msg.content) + +# Thinking off — a direct answer, no deliberation segment. +resp = client.chat.completions.create( + model="openbmb/MiniCPM5-2B", + messages=[{"role": "user", "content": "What is 15% of 240?"}], + extra_body={"chat_template_kwargs": {"enable_thinking": False}}, +) +print("Without thinking:", resp.choices[0].message.content) +``` + + + + + +```text Output +Reasoning: We are asked: "What is 15% of 240?" This is a simple percentage calculation. To find 15% of 240, we multiply 240 by 15% (which is 0.15). So: 240 × 0.15 = 36. Alternatively, we can think of it as (15/100) × 240 = (15 × 240)/100 = 3600/100 = 36. So the answer is 36. + +We need to respond in a helpful way. The user might be testing or seeking quick answer. Let's provide a clear response with explanation if needed, but since it's straightforward, we can just give the answer directly or briefly explain. + +Answer: + +15% of 240 is **36**. + +To calculate: +\( 240 \times 0.15 = 36 \) +Reasoning: None +Answer: To find 15% of 240, multiply 240 by 15% (which is 0.15): + +\[ +240 \times 0.15 = 36 +\] + +So, 15% of 240 is **36**. +``` + + + +### 2.2 Tool Calling + +MiniCPM5-2B emits XML-style tool calls (`...`), and SGLang's built-in `minicpm5` detector converts them to OpenAI-compatible `tool_calls`, parallel calls included. The parser is already in every command the Deploy panel generates; the **Tool Call Parser** chip in the Playground's **Parsers** card is an opt-out, not an opt-in. + + + +```python Example +from openai import OpenAI + +client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY") + +tools = [{ + "type": "function", + "function": { + "name": "get_weather", + "description": "Get the current weather for a city.", + "parameters": { + "type": "object", + "properties": { + "city": {"type": "string", "description": "City name"}, + "date": {"type": "string", "description": "YYYY-MM-DD"}, + }, + "required": ["city"], + }, + }, +}] + +resp = client.chat.completions.create( + model="openbmb/MiniCPM5-2B", + messages=[{"role": "user", "content": "What is the weather in Beijing today?"}], + tools=tools, +) + +msg = resp.choices[0].message +# With the qwen3 reasoning parser on, the thinking segment lands in +# `reasoning_content` and `content` may be empty on a tool-call turn. +print("Reasoning:", getattr(msg, "reasoning_content", None)) +print("Content:", msg.content) +for call in (msg.tool_calls or []): + print("Tool:", call.function.name, call.function.arguments) +``` + + + + + +```text Output +Reasoning: +The user is asking about the weather in Beijing today. I have access to a get_weather function that can help with this. Let me call it to get the current weather for Beijing. + +Content: + + +Tool: get_weather {"city": "Beijing"} +``` + + diff --git a/docs/cookbook/autoregressive/Qwen/Qwen3-VL.mdx b/docs/cookbook/autoregressive/Qwen/Qwen3-VL.mdx index 376a484159b0..67cd6b11f5bf 100644 --- a/docs/cookbook/autoregressive/Qwen/Qwen3-VL.mdx +++ b/docs/cookbook/autoregressive/Qwen/Qwen3-VL.mdx @@ -56,7 +56,7 @@ import { Qwen3VLDeployment } from "/src/snippets/autoregressive/qwen3-vl-deploym - **H200 / B200:** Runs out of the box, supporting full context length plus concurrent image + video processing. **Additional multimodal server parameters:** -- `--keep-mm-feature-on-device`: Retain multimodal feature tensors on GPU after processing to avoid device-to-host memory copies, improving performance for high-frequency inference. +- `--mm-feature-transport=cuda_ipc`: Keep multimodal feature tensors in a bounded GPU pool shared over CUDA IPC to avoid device-to-host memory copies, improving performance for high-frequency inference. **Example with full multimodal optimizations:** ```bash Command diff --git a/docs/cookbook/autoregressive/Qwen/Qwen3.5.mdx b/docs/cookbook/autoregressive/Qwen/Qwen3.5.mdx index 2545de6ee4f6..78681d399d9b 100644 --- a/docs/cookbook/autoregressive/Qwen/Qwen3.5.mdx +++ b/docs/cookbook/autoregressive/Qwen/Qwen3.5.mdx @@ -42,7 +42,7 @@ Qwen3.5 features a Gated Delta Networks combined with sparse Mixture-of-Experts Qwen3.5-397B-A17B [Qwen/Qwen3.5-397B-A17B](https://huggingface.co/Qwen/Qwen3.5-397B-A17B) [Qwen/Qwen3.5-397B-A17B-FP8](https://huggingface.co/Qwen/Qwen3.5-397B-A17B-FP8) - NVIDIA NVFP4: [nvidia/Qwen3.5-397B-A17B-NVFP4-V2](https://huggingface.co/nvidia/Qwen3.5-397B-A17B-NVFP4-V2)
AMD MXFP4: [amd/Qwen3.5-397B-A17B-MXFP4](https://huggingface.co/amd/Qwen3.5-397B-A17B-MXFP4) + NVIDIA NVFP4: [nvidia/Qwen3.5-397B-A17B-NVFP4-V2](https://huggingface.co/nvidia/Qwen3.5-397B-A17B-NVFP4-V2)
AMD MXFP4: [amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2](https://huggingface.co/amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2) Qwen3.5-122B-A10B @@ -106,16 +106,14 @@ docker pull lmsysorg/sglang:latest docker pull lmsysorg/sglang-rocm:v0.5.15.post1-rocm720-mi30x-20260715 # Or use Docker (AMD MI355X) -docker pull lmsysorg/sglang-rocm:v0.5.15.post1-rocm720-mi35x-20260715 +docker pull lmsysorg/sglang-rocm:v0.5.19-rocm720-mi35x-20260911 ``` For the full Docker setup and other installation methods, please refer to the [official SGLang installation guide](../../../docs/get-started/install). -For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation). - ## 3. Model Deployment -This section provides deployment configurations optimized for different hardware platforms and use cases. +This section provides deployment configurations optimized for different hardware platforms including NVIDIA GPUs, AMD GPUs, Intel Arc Pro B-Series GPUs(codename: BMG (Battlemage)), and Intel Xeon CPUs. ### 3.1 Basic Configuration @@ -126,9 +124,9 @@ This section provides deployment configurations optimized for different hardware ### 3.2 Configuration Tips - Speculative decoding (MTP) can significantly reduce latency for interactive use cases. -- **H100 FP8:** Add `--enable-symm-mem` to enable NCCL symmetric memory for faster collectives and better performance under multi-GPU settings. -- **AMD GPUs (MI300X / MI325X / MI355X):** Use `SGLANG_USE_AITER=1` and `SGLANG_USE_AITER_UNIFIED_ATTN=1` with `--attention-backend aiter`, which requires `--page-size 16` and can also enable `--enable-aiter-allreduce-fusion`. Additionally set `AITER_FLYDSL_FORCE=1` to force the AITER FlyDSL MoE kernels and `SGLANG_MAMBA_SSM_DTYPE=bfloat16` to store the Mamba SSM state in bfloat16 (instead of the default float32). For the **MXFP4 checkpoint on MI355X**, set `ROCM_QUICK_REDUCE_QUANTIZATION=INT8` to route multi-GPU collectives through INT8-quantized ROCm quick all-reduce, and drop `--enable-aiter-allreduce-fusion` (the two are mutually exclusive; quick all-reduce is preferred for this recipe). -- **KV cache offloading (MXFP4 on MI355X):** Selecting **Host DRAM (HiCache)** adds a host-memory tier below the device KV cache, which lets long-context agentic workloads keep more prefix cached and reach higher concurrency than the device KV pool alone allows. It emits `--enable-hierarchical-cache --hicache-ratio 1.5 --hicache-write-policy write_through --hicache-io-backend direct --hicache-mem-layout page_first_direct`. HiCache extends the radix cache, so the generated command drops `--disable-radix-cache` — the two options are mutually exclusive and SGLang rejects them together at startup. `--hicache-ratio 1.5` sizes the host tier at 1.5x the device KV cache, so make sure the node has that much free CPU DRAM per server on top of the model weights. Leave this **Disabled** for fixed-length throughput serving, where the extra host traffic buys nothing. +- **H100 / B200 / B300 FP8:** Add `--enable-symm-mem` to enable NCCL symmetric memory for faster collectives and better performance under multi-GPU settings. It only helps when tp > 1, so on B200/B300 it applies to the 397B-A17B FP8 recipe (tp=4); the 122B-A10B and 35B-A3B FP8 recipes are single-GPU there. +- **AMD GPUs (MI300X / MI325X / MI355X):** Use `SGLANG_USE_AITER=1` and `SGLANG_USE_AITER_UNIFIED_ATTN=1` with `--attention-backend aiter`, which requires `--page-size 16` and can also enable `--enable-aiter-allreduce-fusion`. Additionally set `AITER_FLYDSL_FORCE=1` to force the AITER FlyDSL MoE kernels and `SGLANG_MAMBA_SSM_DTYPE=bfloat16` to store the Mamba SSM state in bfloat16 (instead of the default float32). For the **MXFP4 checkpoint on MI355X**, set `ROCM_QUICK_REDUCE_QUANTIZATION=INT4` to route multi-GPU collectives through INT4-quantized ROCm quick all-reduce, and drop `--enable-aiter-allreduce-fusion` (the two are mutually exclusive; quick all-reduce is preferred for this recipe). +- **KV cache offloading (MXFP4 on MI355X):** Selecting **Host DRAM (HiCache)** adds a host-memory tier below the device KV cache, which lets long-context agentic workloads keep more prefix cached and reach higher concurrency than the device KV pool alone allows. It emits `--enable-hierarchical-cache --hicache-ratio 1.5 --hicache-write-policy write_through --hicache-io-backend kernel --hicache-mem-layout page_first`. HiCache extends the radix cache, so the generated command drops `--disable-radix-cache` — the two options are mutually exclusive and SGLang rejects them together at startup. `--hicache-ratio 1.5` sizes the host tier at 1.5x the device KV cache, so make sure the node has that much free CPU DRAM per server on top of the model weights. Leave this **Disabled** for fixed-length throughput serving, where the extra host traffic buys nothing. - **Watchdog timeout:** Increase `--watchdog-timeout` to `1200` or higher for this large model, as weight loading can take significant time. - **Mamba Radix Cache**: Qwen3.5's hybrid Gated Delta Networks architecture supports two mamba scheduling strategies via `--mamba-radix-cache-strategy`: - **V1 (`no_buffer`)**: Default. No overlap scheduler, lower memory usage. Required for AMD MI GPUs. @@ -139,6 +137,8 @@ This section provides deployment configurations optimized for different hardware - **CUDA IPC Transport**: Add `SGLANG_USE_CUDA_IPC_TRANSPORT=1` as an environment variable to use CUDA IPC for transferring multimodal features, significantly improving TTFT (Time To First Token). Note: this consumes additional memory proportional to image size, so you may need to lower `--mem-fraction-static` or `--max-running-requests`. - **Multimodal Attention Backend**: Use `--mm-attention-backend fa3` on H100/H200 for better vision performance, or `--mm-attention-backend fa4` on B200/B300. - **B200 (FP8)**: Add `--enable-flashinfer-allreduce-fusion` for optimized throughput on Blackwell. +- **B200 / B300 (FP8 MoE)**: Add `--moe-runner-backend flashinfer_trtllm` to route the fused MoE through the trtllm-gen kernels on Blackwell. This is not picked up automatically — SGLang's auto-selection of `flashinfer_trtllm` is limited to the DeepSeek architecture family, so an FP8 Qwen3.5 MoE run otherwise falls back to the Triton MoE runner. It applies to the MoE sizes (397B-A17B / 122B-A10B / 35B-A3B); the dense sizes have no MoE layers. The NVFP4 recipes already set it. +- **B200 / B300 (FP8)**: Add `--linear-attn-prefill-backend flashinfer` to run GDN (linear attention) prefill on the FlashInfer CuTe-DSL kernel. SGLang auto-selects it only inside a narrow validated domain (a chunked prefill size of at most 8192 and a bfloat16 mamba state pool, among other conditions), which these FP8 recipes fall outside of, so the override has to be explicit. Requires CUDA 13+ and flashinfer >= 0.6.14 — SGLang raises at startup on SM100 with an older CUDA. Applies to every Qwen3.5 size, since all of them are hybrid GDN models. - For processing large images or videos, you may need to lower `--mem-fraction-static` to leave room for image feature tensors. - Hardware requirements: - **BF16**: ~397B parameters require ~800GB of GPU memory for weights. @@ -157,7 +157,7 @@ This section provides deployment configurations optimized for different hardware - **MI300X (192GB)** runs with tp=4. - **MI325X (256GB)** runs with tp=2. - **MI355X (288GB)** runs with tp=2. - - **FP4**: The FP4 quantized model requires ~250GB for weights, cutting memory by almost 4x. NVFP4 ([nvidia/Qwen3.5-397B-A17B-NVFP4-V2](https://huggingface.co/nvidia/Qwen3.5-397B-A17B-NVFP4-V2)) requires B200/B300 (Blackwell architecture); AMD provides an MXFP4 checkpoint ([amd/Qwen3.5-397B-A17B-MXFP4](https://huggingface.co/amd/Qwen3.5-397B-A17B-MXFP4)) for MI355X. + - **FP4**: The FP4 quantized model requires ~250GB for weights, cutting memory by almost 4x. NVFP4 ([nvidia/Qwen3.5-397B-A17B-NVFP4-V2](https://huggingface.co/nvidia/Qwen3.5-397B-A17B-NVFP4-V2)) requires B200/B300 (Blackwell architecture); AMD provides an MXFP4 checkpoint with FP8 attention ([amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2](https://huggingface.co/amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2)) for MI355X. - **B200 (183GB)** runs with tp=4 (tp=2 with expert parallelism 2 when MTP is enabled). (NVFP4) - **B300 (275GB)** runs with tp=2. (NVFP4) - **MI355X (288GB)** runs with tp=2 (use tp=4 for low concurrency). (MXFP4) @@ -777,6 +777,37 @@ Tool Call: None Arguments: } ``` +#### 4.3.3 Semantic Decisions + +Qwen3.5-35B-A3B can answer typed questions about an input with a probability for every option through `/v1/decisions`, without generating text. The server turns thinking off for these requests, so the answer is read after a closed think block, and the same server keeps serving chat traffic. No launch flag is needed. Until a release contains `/v1/decisions`, install a [nightly build](/docs/get-started/install#nightly-builds). With a server started by `sglang serve --model-path Qwen/Qwen3.5-35B-A3B --host 127.0.0.1 --port 30000`: + +```python Example +import requests + +response = requests.post( + "http://127.0.0.1:30000/v1/decisions", + json={ + "input": "I've been trying to connect my Stripe account for 3 days and the integration keeps failing.", + "questions": [ + { + "id": "team", + "type": "choice", + "question": "Which team should handle this ticket?", + "options": [{"name": "billing"}, {"name": "technical"}, {"name": "sales"}], + }, + {"id": "urgent", "type": "yes_no", "question": "The customer needs an answer today."}, + ], + }, + timeout=60, +) +response.raise_for_status() +answers = response.json()["answers"] +print(answers["team"]["choice"], answers["team"]["probabilities"]) +print(answers["urgent"]["probabilities"]["yes"]) +``` + +See [Decision models](/docs/supported-models/decision_models) for the request reference, how the probabilities are computed, and the limits. + ## 5. Benchmark ### 5.1 Accuracy Benchmark @@ -999,7 +1030,6 @@ Container image (pinned for reproducibility): Server Launch Command: ```bash Command -SGLANG_ENABLE_SPEC_V2=1 \ python3 -m sglang.launch_server \ --model-path Qwen/Qwen3.5-397B-A17B-FP8 \ --served-model-name Qwen/Qwen3.5-397B-A17B-FP8 \ diff --git a/docs/cookbook/autoregressive/Qwen/Qwen3.8-27B.mdx b/docs/cookbook/autoregressive/Qwen/Qwen3.8-27B.mdx index 2baa0a63ea0a..e99d6986f347 100644 --- a/docs/cookbook/autoregressive/Qwen/Qwen3.8-27B.mdx +++ b/docs/cookbook/autoregressive/Qwen/Qwen3.8-27B.mdx @@ -19,9 +19,7 @@ For all methods and hardware platforms, see the [official SGLang installation gu pip install --upgrade pip pip install uv -git clone https://github.com/sgl-project/sglang.git && cd sglang -git checkout 1cf2b8c54d81802abc15dcf23a29b9cc687bc01e -uv pip install --prerelease=allow -e "python[all]" +uv pip install --prerelease=allow sglang ``` Then run the **Python** output of the command panel below in that environment. @@ -31,7 +29,7 @@ Then run the **Python** output of the command panel below in that environment. ```bash Command -docker pull lmsysorg/sglang:dev-qwen38-27b-dflash2 +docker pull lmsysorg/sglang:latest ``` For how to launch the image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with what the command generator below produces. @@ -59,14 +57,12 @@ import { Qwen38MambaRatioCalculator } from "/src/snippets/_qwen38_mamba_ratio_ca - The RTX 5090 and RTX PRO 6000 cells above — including every Speculative - Decoding / Serving Strategy / SSM dtype combination — were validated at - ISL 8192 / OSL 1024, concurrency 1; for DFLASH2, to that full standard on - NVFP4 and to boot-and-serve on the RTX PRO 6000 BF16/FP8 cells. The DGX Spark - cells cover the full combination set — DFLASH2 included — re-measured end to - end on `1cf2b8c`, but to a weaker standard: each of the 48 was confirmed to - **boot and serve** at ISL 8192 / OSL 1024, concurrency 1, with no throughput - or acceptance-length numbers taken. + Every cell above — RTX 5090, RTX PRO 6000 and DGX Spark, across all five + checkpoints and every Speculative Decoding / Serving Strategy / SSM dtype + combination — is measured on **v0.5.19**. That is 202 cells, each one served + and scored on the full 1319-question GSM8K (93.18-95.15%). The serving + envelope behind the pins is ISL 8192 / OSL 1024 at concurrency 1; throughput + and acceptance-length numbers were not re-taken in that sweep. ### Mamba ratio calculator @@ -176,18 +172,49 @@ context from earlier messages. Same body, `lm_head` left dense in BF16 RadixArk/Qwen3.8-27B-NVFP4-BF16-LMHead + + Qwen3.8-27B-NVFP4 (NVIDIA) + NVIDIA's ModelOpt export of the same W4A4 body, `lm_head` packed to FP4 + nvidia/Qwen3.8-27B-NVFP4 + -The two NVFP4 exports differ only in the `lm_head`: one packs it to FP4, the -other leaves it dense in BF16. The dense head is ~1.7 GB larger on disk and -~3.2 GB larger at runtime, so it is the harder of the two to fit — every +The two RadixArk NVFP4 exports differ only in the `lm_head`: one packs it to +FP4, the other leaves it dense in BF16. The dense head is ~1.7 GB larger on disk +and ~3.2 GB larger at runtime, so it is the harder of the two to fit — every recipe on this page was measured against it, and the FP4-head cells reuse those pins unchanged. -Both NVFP4 checkpoints declare `kv_cache_quant_algo: FP8`; SGLang's default -`--kv-cache-dtype auto` honors it, so the KV pool runs in `fp8_e4m3` with the -checkpoint's calibration scales automatically. +NVIDIA's own export is that same W4A4 body with that same FP4 head: identical +quantized-layer map (FP8 attention and GDN projections, NVFP4 MLPs), identical +tensor set, identical 21.9 GB on disk. On GB300, RTX PRO 6000 and DGX Spark its +cells reuse the FP4-head pins unchanged, and both SM12x grids have been +re-measured against this export on v0.5.19: all 16 overlay combinations per +card serve and score 94.01-95.00% (RTX PRO 6000) and 94.16-95.07% (DGX Spark) +on the full 1319-question GSM8K. + +The RTX 5090 is measured too — all 15 overlay combinations it offers serve and +score 93.93-94.92% — and every winning launch command there is identical to the +FP4-head export's, which is the strongest form of the claim above. What the +32GB card does need is the draft-model rows pinning their own pools: those +recipes pin `--max-running-requests 1`, but nothing caps the pools to match, so +the KV pool sizes itself for 127,332 tokens against the 9,216 one +8192-in/1024-out request needs, and the engine's default split then leaves the +GDN state pool far short of the slots it needs once the draft model's weights +are counted against `--mem-fraction-static`. The DSPARK row therefore pins +`--max-total-tokens` and a measured `--mamba-full-memory-ratio`, as do DFLASH2 +and MTP on the dense-lm_head export at float32 state. Those pins override the +calculator's live value for the selections that carry them. The no-speculation +row needs none of it and runs at the pins shown. + +The two RadixArk checkpoints declare `kv_cache_quant_algo: FP8`, so SGLang's +default `--kv-cache-dtype auto` already puts their KV pool in `fp8_e4m3`. The +NVIDIA export ships no `kv_cache_scheme`, so `auto` would leave its pool in +BF16 instead. Every recipe on this page pins `--kv-cache-dtype fp8_e4m3` +explicitly, so all three run the same `fp8_e4m3` pool regardless; the +difference only shows up if you switch the Playground's **KV Cache Precision** +row back to Auto. ## 2. Configuration Tips @@ -204,25 +231,26 @@ checkpoint's calibration scales automatically. scheduler killed with `exit code -15` and no traceback (`journalctl -u earlyoom` shows the kill). At 0.85, 15 of the 48 cells were killed that way, and which cells is margin noise; at 0.80 every cell served on every attempt. - **Validated on SM121 / aarch64**: all 48 configurations (3 checkpoints x + **Validated on SM121 / aarch64**: all 80 configurations (5 checkpoints x Speculative Decoding x Serving Strategy x Mamba SSM Dtype, DFLASH2 included) - booted and served on GB10 at `1cf2b8c` at ISL 8192 / OSL 1024, concurrency 1. - That is boot-and-serve coverage only — no throughput or acceptance-length - numbers — and it includes the FlashInfer `plan` / `uniform_q_len` path above, - which raised no arity error on that build. Three host quirks when reproducing + served on GB10 on `v0.5.19` at ISL 8192 / OSL 1024, concurrency 1, and each + scored the full 1319-question GSM8K (93.18-95.15%); the float32 and bfloat16 + halves ran on two separate GB10 boxes. No throughput or acceptance-length + numbers were re-taken. The sweep exercises the FlashInfer `plan` / + `uniform_q_len` path above, which raised no arity error on that build. Three host quirks when reproducing on GB10: docker GPU access is CDI-only (`--device nvidia.com/gpu=all`, as no `nvidia` runtime is registered); `nvidia-smi` reports `Not Supported` for memory because it is unified with the CPU — gate a relaunch on `MemAvailable` in `/proc/meminfo` instead; and the BF16 checkpoint takes ~6.5 minutes just to load its 18 shards from NVMe, so budget ~10 minutes to READY before calling a boot hung. -- **H200 (SM90)**: BF16 and FP8 only — the card has no FP4 tensor cores, so the - NVFP4 checkpoint's MLP would fall back to the Marlin W4A16 weight-only path - and its cell is greyed out. The H200 recipes use 32768-token prefill chunks - (SM90 prefill is fast enough that a big chunk barely stalls decode, unlike - the SM120 guidance below), and the FlashInfer GDN prefill backend engages by - default under them. `--attention-backend fa3` is a valid alternative, - measured slightly faster at bs=1. +- **H200 (SM90)**: BF16 and FP8 only — the card has no FP4 tensor cores, so an + NVFP4 checkpoint's MLP would fall back to the Marlin W4A16 weight-only path, + and all three NVFP4 cells are greyed out. The H200 recipes use 32768-token + prefill chunks (SM90 prefill is fast enough that a big chunk barely stalls + decode, unlike the SM120 guidance below), and the FlashInfer GDN prefill + backend engages by default under them. `--attention-backend fa3` is a valid + alternative, measured slightly faster at bs=1. - **MTP**: `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4` uses the in-checkpoint MTP head. (This recipe was originally documented with `NEXTN`, @@ -247,24 +275,40 @@ checkpoint's calibration scales automatically. `--speculative-algorithm DFLASH --speculative-draft-model-path incoai/Qwen3.8-27B-DFlash2 --speculative-num-draft-tokens 8` (8 is the draft's block size, and it is the `D` term in the ratio, same value as - DSpark's). The selector projects candidates through the target `lm_head`, + DSpark's). Runs on Ascend NPUs as well + ([#35629](https://github.com/sgl-project/sglang/pull/35629)): the selector + verify falls back to argmax there, matching what EAGLE and first-generation + DFlash drafts (such as `z-lab/Qwen3-8B-DFlash-b16`) already do on NPU. + Therefore, NPU currently guarantees lossless verification only for greedy + requests; use `temperature=0` and `top_k=1`. Non-greedy requests log a warning; + both draft proposal and target verification fall back to greedy, so their + requested sampling distribution is not preserved. The selector projects + candidates through the target `lm_head`, including quantized heads, so it runs on the NVFP4 checkpoint (whose head is NVFP4-packed; the BF16 and FP8 checkpoints keep a dense head). - Validation: NVFP4 measured end-to-end on RTX PRO 6000 and RTX 5090; the - RTX PRO 6000 BF16/FP8 cells boot and serve; all 12 DGX Spark DFLASH2 cells - boot and serve on `1cf2b8c` with the selector folded into the draft CUDA - graph; on H200 and GB300 those cells carry the - **Final Verification In Progress** badge. The - RTX PRO 6000 recipe needs no changes. On the 32GB RTX 5090 every pin is - re-measured against that commit, and the panel applies them automatically: - DFlash2 at `--mem-fraction-static 0.91` with `--chunked-prefill-size 1024` — - the only cell on this page needing a smaller prefill chunk, because at 0.91 - the pools fit but a 2048-token chunk's activations do not — DSpark at 0.88, - EAGLE at 0.93 (bfloat16) and 0.94 (float32), and no-speculation at 0.90. + The Ascend comparison in #35629 used an A3 Series device with BF16 target weights, + `--tp-size 2 --attention-backend ascend --mamba-ssm-dtype bfloat16 + --mamba-radix-cache-strategy extra_buffer`, and disabled RadixCache for both + baseline and DFlash2 to exclude cache warm-up and prefix reuse. The DFlash2 + run added the three flags shown above. + That comparison's accuracy used zero-shot GSM8K with greedy sampling, + `max_new_tokens=2048`, 128 examples, and concurrency levels 1, 2, 4, 8 + and 16 — a different protocol from this page's own sweep below. + Validation: every SM12x cell on this page is measured end to end on + v0.5.19 — 202 cells over the five checkpoints, four speculative options, two + serving tiers and two GDN state dtypes, full 1319-question GSM8K on each, + 93.18-95.15%. The RTX PRO 6000 and DGX Spark recipes need no changes. On the + 32GB RTX 5090 the panel applies the measured pins automatically: DFlash2 at + `--mem-fraction-static 0.91` with `--chunked-prefill-size 1024` — at 0.91 the + pools fit but a 2048-token chunk's activations do not — DSpark at 0.88 + (bfloat16), 0.91 (float32) and 0.92 on the dense-lm_head export, all three + with their pools pinned and the last two also cutting the prefill chunk to + 1024 and 512, EAGLE at 0.93 (bfloat16) and 0.94 (float32), and + no-speculation at 0.90. Whether float32 is available with a draft model depends on the `lm_head`: on the BF16-head export it is greyed out for both DSpark and DFlash2, since the dense head's ~3.2 GB leave no fp32 state pool that also clears prefill graph - capture. The FP4-head export frees that headroom back — DSpark serves at 0.89 + capture. The FP4-head export frees that headroom back — DSpark serves at 0.91 and DFlash2 High-Throughput at 0.895 with `--mamba-full-memory-ratio 10` overriding the balanced value — and only DFlash2 Low-Latency stays out of reach, where five fp32 slots and a full request's KV never coexist. bfloat16 @@ -497,3 +541,34 @@ providers: ``` + +## 4. Semantic Decisions + +Qwen3.8-27B can answer typed questions about an input with a probability for every option through `/v1/decisions`, without generating text. The server turns thinking off for these requests, so the answer is read after a closed think block, and the same server keeps serving chat traffic. No launch flag is needed. Until a release contains `/v1/decisions`, install a [nightly build](/docs/get-started/install#nightly-builds). With a server started by `sglang serve --model-path Qwen/Qwen3.8-27B --host 127.0.0.1 --port 30000`: + +```python Example +import requests + +response = requests.post( + "http://127.0.0.1:30000/v1/decisions", + json={ + "input": "I've been trying to connect my Stripe account for 3 days and the integration keeps failing.", + "questions": [ + { + "id": "team", + "type": "choice", + "question": "Which team should handle this ticket?", + "options": [{"name": "billing"}, {"name": "technical"}, {"name": "sales"}], + }, + {"id": "urgent", "type": "yes_no", "question": "The customer needs an answer today."}, + ], + }, + timeout=60, +) +response.raise_for_status() +answers = response.json()["answers"] +print(answers["team"]["choice"], answers["team"]["probabilities"]) +print(answers["urgent"]["probabilities"]["yes"]) +``` + +See [Decision models](/docs/supported-models/decision_models) for the request reference, how the probabilities are computed, and the limits. diff --git a/docs/cookbook/autoregressive/Qwen/Qwen3.8-Flash-Next.mdx b/docs/cookbook/autoregressive/Qwen/Qwen3.8-Flash-Next.mdx index 23032e5ed104..28df6bee3e72 100644 --- a/docs/cookbook/autoregressive/Qwen/Qwen3.8-Flash-Next.mdx +++ b/docs/cookbook/autoregressive/Qwen/Qwen3.8-Flash-Next.mdx @@ -10,63 +10,56 @@ tag: NEW -For all methods and hardware platforms, see the [official SGLang installation guide](../../../docs/get-started/install). The two paths below match the **Python / Docker** toggle in the command panel. - - - - - -Qwen3.8-Flash-Next support is not in a tagged release yet, so build the model-support PR rather than installing from PyPI: +Use an SGLang build that includes Qwen3.8-Flash-Next support (v0.5.20 or later). ```bash Command -pip install -U uv -uv venv --python 3.12 && source .venv/bin/activate - -# Qwen3.8-Flash-Next model support: -# https://github.com/sgl-project/sglang/pull/36497 -git clone https://github.com/sgl-project/sglang.git -cd sglang -git fetch origin pull/36497/head && git checkout FETCH_HEAD -uv pip install -e python +docker pull lmsysorg/sglang:latest ``` - -Model support lands in [#36497](https://github.com/sgl-project/sglang/pull/36497). Once it is in a release, `uv pip install sglang` is enough and this whole step goes away. - +The deployment panel can render a complete `docker run` command for the selected hardware and options. See [Install SGLang with Docker](/docs/get-started/install#method-3-using-docker) for host setup. -Then run the **Python** output of the command panel below in that environment. + - +Pick your hardware + quantization to generate the launch command. - +import { Deployment } from "/src/snippets/_deployment.jsx"; +import { config } from "/src/snippets/configs/Qwen/qwen3.8-flash-next.jsx"; +import { benchmarks } from "/src/snippets/configs/Qwen/qwen3.8-flash-next-benchmarks.jsx"; -**NVIDIA GPUs** (H200 / B200 / B300 / GB300) — the launch image, since this is a day-0 model with no release cut yet: + -```bash Command -docker pull lmsysorg/sglang:qwen38flashnext -``` + -**AMD GPUs** (MI350X / MI355X) — the matching ROCm build. It targets CDNA4 (gfx950) and is **not** interchangeable with the CUDA image above: + -```bash Command -docker pull lmsysorg/sglang-rocm:qwen38flashnext -``` +The NVFP4 checkpoint is 126 GiB (78 GiB of experts and dense weights plus a 47.7 GiB FP8 N-gram table), so it does not fit one DGX Spark's 128 GB of unified memory, and `--ple-offload-embedding` does not help there: on GB10 the "offloaded" pinned-host table comes out of the same pool as the GPU weights. The two-node shape is TP=2 across two Sparks over the ConnectX-7 200GbE link, with `--no-ple-offload-embedding` keeping the table GPU-resident and sharded (~65 GB of weights per node). A single Spark works only with the table file-backed on NVMe (see the Single Spark bullet below). -For how to launch either image, see [Install → Method 3: Using Docker](../../../docs/get-started/install#method-3-using-docker). Substitute the inner `sglang serve ...` with whatever the command generator below produces. +- **Launch order.** Start rank 1 first, then rank 0 within a few seconds. When re-launching, stop both ranks and confirm nothing listens on the rendezvous port before starting again: a new rank 1 attaching to a stale rank 0 store fails with a gloo "Connection reset by peer". +- **NCCL.** The cross-node decode CUDA-graph deadlock seen on an earlier DGX Spark stack was on NCCL 2.28.x; both builds these images load have been verified here for TP=2 across two Sparks: `dev-qwen38-next-local` runs its pip NCCL 2.29.7 (a system 2.28.3 is also present but not loaded), `qwen38flashnext` its 2.30.7. Confirm with the startup log line `sglang is using nccl==…`, which reports the library actually loaded. If your Sparks also have a slower management NIC, pin `NCCL_SOCKET_IFNAME` / `GLOO_SOCKET_IFNAME` to the 200GbE interface; with `--dist-init-addr` on the 200GbE address the verified runs picked it without pinning. +- **Memory.** Both cells run `--mem-fraction-static 0.85`, which leaves ~8–12 GiB of host memory free per node under load (measured through GSM8K at full concurrency and a 100k-token prefill). Two precautions for long-context work beyond that: `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` stops variable-shape chunked-prefill buffers from fragmenting the caching allocator at 200k+ contexts, and a host-side watchdog that kills the server when `MemAvailable` drops below a few GiB, because a unified-memory exhaustion can take the whole box down and needs a power cycle to recover. +- **Concurrency.** The hybrid GDN/QSA model reserves mamba state slots per running request (5 with the default `extra_buffer` radix strategy, 4 with `extra_buffer_lazy`), and the scheduler caps `--max-running-requests` to what the mamba pool admits; read the effective value from the startup log, not from `/get_server_info`. The cells pin `--max-mamba-cache-size` to requests × slots (24 × 5 = 120 for low latency, 96 × 4 = 384 for high throughput); raising concurrency further takes memory from the KV pool one-for-one. +- **Flags.** The cells are the model card's TP=2 recipe without `--mamba-track-interval 64`: the default of 256 tokens satisfies the constraints (a multiple of the 64-token page, at least the 4 draft tokens) and leaves a ~40% larger KV pool (1.48M tokens at 24 concurrent with MTP, 1.07M at 96 without), at the cost of coarser prefix-cache reuse of the recurrent state. `--trust-remote-code` is not needed; the architecture is native to SGLang. +- **Measured.** 100k-token prefill at 2,400–2,840 tok/s; MTP accept length 3.5–3.7 of 4 draft tokens on non-thinking output (lower, ~2.5, on thinking output). +- **Single Spark (file-backed PLE table).** One GB10 holds the checkpoint only if the 47.7 GiB FP8 N-gram table leaves memory entirely: with **PLE Offload = On (NVMe file)** (`--ple-offload-embedding --ple-offload-backend file`, from [#37068](https://github.com/sgl-project/sglang/pull/37068), merged into `qwen4-main-squashed`) SGLang creates a sparse 47.7 GiB file under `$SGLANG_CACHE_DIR/ple/` (relocate with `--ple-offload-dir`; put it on the local NVMe and mount that directory into the container), fills it on boot, and the gather kernel reads rows through the host page tables; the table's resident set stays at 0 while serving, with hot pages in page cache (capped at 8 GiB by `SGLANG_QWEN4_PLE_FILE_RSS_BUDGET_GB`). The 78.3 GiB of experts and dense weights stay resident (the server logs ~80 GB after load, which also counts the CUDA context and allocator overhead) and `--mem-fraction-static 0.85` leaves ~12–18 GB for the pools, so concurrency is pinned low: **8 requests with MTP** (40 fp32 mamba slots of ~113 MB each) or **24 without** (96 slots on `extra_buffer_lazy`). Measured on `4ccff141db`: MTP cell 27.5 tok/s single-stream (TPOT 33.6 ms) and 71.7 tok/s output at 8; no-MTP cell 15.9 tok/s single-stream and 83 tok/s output at 24; host memory never below 10 GiB. **Boot-time caveat:** every boot rewrites the whole table through the mapping; on an already-populated file that is a read-modify-write per 4 KiB page with readahead disabled (`MADV_RANDOM`), ~17 MB/s and ~55 minutes; on a fresh sparse file it fills at GB/s and the boot takes ~10 minutes. Until that is fixed upstream, delete the previous `ple_table_*.bin` before each boot. +- **NVIDIA export (NVFP4 (NVDA)).** `nvidia/Qwen3.8-Flash-Next-NVFP4` is a ModelOpt MIXED_PRECISION checkpoint (NVFP4 experts, FP8 N-gram table, FP8 block-scaled MTP experts) and needs the loader from [#38121](https://github.com/sgl-project/sglang/pull/38121), merged into `qwen4-main-squashed` (the branch the Python install path builds and the `lmsysorg/sglang:dev-qwen38-next-local` image ships); the `qwen38flashnext` image predates it and cannot load this export. Do not pass `--quantization` for it (it resolves to `modelopt_mixed`), and pass `--moe-runner-backend flashinfer_cutlass` explicitly: the mixed-precision auto-default picks `flashinfer_trtllm` on GB10, which the NVFP4 MoE method rejects at autotune. Its MTP experts are FP8 block-scaled with 128-wide blocks and cannot be split across two ranks (640 / 2 = 320), so the two-node low-latency cell reads the MTP draft from the RadixArk export, the same trained head kept in BF16 there. On a single Spark (TP=1, file-backed table) nothing is sharded, so the export's own MTP head loads directly. The single-Spark NVDA cells use the same pins as the RDXA ones (8 requests with MTP, 24 without); the smaller fp8 draft leaves a 174k-token KV pool with MTP, against 93k for the RadixArk cell. Measured on the `qwen4-main-squashed` tip `4ccff141db` (#38121 merged) at TP=2: 47.4 tok/s single-stream with MTP, 253 tok/s output at 96 concurrent without, the same as the RadixArk export. - + - + - + -Pick your hardware + quantization to generate the launch command. +The NVFP4 checkpoint fits a single 96 GB RTX PRO 6000 Blackwell (SM120) only with the 47.7 GiB FP8 N-gram table in **CPU pinned memory** (`--ple-offload-embedding`, the forced setting of the PLE Offload row on this hardware): the other 78 GiB of the checkpoint loads onto the card. After the loader's temporaries are collected, 74.7 GiB stays resident without speculative decoding and 81.8 GiB with it (the draft head is 0.5 GiB; the rest is memory the loader still holds), leaving 19.4 GiB and 12.3 GiB of the 94.2 GiB the process can use. `--mem-fraction-static` keeps (1 − fraction) × 94.2 GiB of that as runtime slack and the pools take the rest: 6.6 GiB of slack and 12.8 GiB of pools at 0.93, 3.8 GiB and 8.3 GiB at 0.96. (The 81 GiB "mem usage" in the load log is the delta before that collection.) The host needs ≥ 64 GB of free RAM for the locked table (plus page cache for the checkpoint) and Docker needs `--ulimit memlock=-1`, or the pinned allocation fails. -import { Deployment } from "/src/snippets/_deployment.jsx"; -import { config } from "/src/snippets/configs/Qwen/qwen3.8-flash-next.jsx"; -import { benchmarks } from "/src/snippets/configs/Qwen/qwen3.8-flash-next-benchmarks.jsx"; +- **Concurrency.** The hybrid model reserves mamba state slots per running request, and the scheduler caps `--max-running-requests` to what the state pool admits: with the default `extra_buffer` strategy (5 fp32 slots per request at 0.109 GiB) that is 3 requests with MTP and 12 without on this card. The cells keep prefix caching on and get to 16 / 64 with three levers: `--mamba-radix-cache-strategy extra_buffer_lazy` (4 slots per request), `SGLANG_OPT_MAMBA_SKIP_DECODE_LOCK=1` (3; a running request's prefix state is no longer pinned in the radix tree during decode, so it can be evicted, which trades cache retention, not numerics), and `--mamba-ssm-dtype bfloat16` (0.055 GiB per slot). `--max-mamba-cache-size` is pinned to requests × 3 (48 / 192). The ceilings from the pool arithmetic are ~20 requests with MTP (each request carries 4 intermediate draft states) and ~70 without; turning prefix caching off (`--disable-radix-cache`, 1 slot per request) reaches 24 / 96 on this card and was verified too, but re-prefills every prompt. +- **Linear-attention kernels.** Left on auto. On SM120 the server resolves decode, prefill and verify to triton — the same GDN kernels the fp32 runs and the DGX Spark cells use; the bf16-state FlashInfer GDN auto-default applies only to SM100. An explicit `--linear-attn-decode-backend flashinfer` also runs on this card and measured the same TPOT (within 0.3 ms) and accuracy, so there is nothing to gain from pinning either. +- **Memory headroom.** With the state pool pinned, the KV pool absorbs the rest of the static budget, so `--mem-fraction-static` is what sets the activation headroom. 4096-token prefill chunks of ShareGPT-length prompts peak 1.5–2.6 GB above the post-graph-capture level, and cells left with 2.4 GB free OOMed in the GDN short-conv during prefill. The cells keep ≥ 4 GB free after graph capture and ≥ 2.3 GB at the measured peak: 0.96 with MTP (78k-token KV pool, ~4.9k per request at 16) and 0.93 without (98k tokens, ~1.5k per request at 64; 0.94 gives 138k tokens with 2.3 GB at peak). `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` is set on both. +- **Accuracy.** Full GSM8K, 1,319 questions, on the `lmsysorg/sglang:dev-qwen38-next-local` image, in two protocols. Chat completions API with thinking off, greedy, an 8,192-token budget, answer parsed from a final "The answer is N" line (the protocol of the DGX Spark rows, and the figure on the benchmark card): 96.9% for the MTP cell and 96.9% for the no-MTP cell (RadixArk export); 97.3% and 97.0% for the NVIDIA export. `python -m sglang.test.run_eval --eval-name gsm8k --num-examples 1319 --max-tokens 16384` (5-shot, greedy, last-number scorer, chat template with thinking on): 97.72% / 97.79% (RadixArk), 97.41% / 97.72% (NVIDIA). Measured on the `4ccff141db` build, which carries the #36811 and #38290 router fixes. +- **Measured (1024-in / 256-out random prompts, `ignore_eos`).** MTP vs no-MTP at 1 request: TTFT 115 vs 116 ms median, TPOT 5.9 vs 11.4 ms, 148 vs 83 tok/s. At 16: TPOT 19.3 vs 25.6 ms, 613 vs 524 tok/s. No-MTP at 64: TPOT 55 ms, 861 tok/s, 3.4 req/s. ShareGPT chat (thinking off, ≤ 512 output tokens): 785 tok/s at 16-way with MTP, 1,258 tok/s at 64-way without. MTP accept length 3.3 of 4 on GSM8K / random prompts, 2.9 on long-form ShareGPT answers. On the `dev-qwen38-next-local` image (`4ccff141db`): 6.0 / 11.44 ms TPOT at 1 request (MTP / no MTP), 685 / 561 tok/s at 16, 909 tok/s at 64 without MTP, accept length 3.16. - +- **NVIDIA export (NVFP4 (NVDA)).** `nvidia/Qwen3.8-Flash-Next-NVFP4` runs on this card with the same shape, pools and flags as the RadixArk cells on the `lmsysorg/sglang:dev-qwen38-next-local` image (it carries the loader from [#38121](https://github.com/sgl-project/sglang/pull/38121); `qwen38flashnext` cannot load this export). Do not pass `--quantization`: the checkpoint resolves to `modelopt_mixed`. Low latency keeps the in-checkpoint MTP head: at TP=1 its fp8 block-scaled experts need no sharding, and #38121 runs them on triton under the `flashinfer_cutlass` pin. The RadixArk BF16 draft measured the same here (accept 3.33 vs 3.31, TPOT 18.5 vs 19.1 ms at 16), so the cell stays single-checkpoint. Verified on that image (`4ccff141db`): full GSM8K in the Accuracy bullet above; 6.02 ms TPOT at 1 request with MTP, 675 output tok/s at 16; 906 output tok/s at 64 without, the same as the RadixArk export. The smaller fp8 draft leaves a ~170k-token KV pool at 16 concurrent. + + ## Playground @@ -122,9 +115,14 @@ Each precision is its own repository: H200, B200, B300, GB300, MI350X, MI355X - NVFP4 + NVFP4 (RDXA) RadixArk/Qwen3.8-Flash-Next-NVFP4 - B200, B300, GB300 (Blackwell only) + B200, B300, GB300, RTX PRO 6000, 1x or 2x DGX Spark (Blackwell only) + + + NVFP4 (NVDA) + nvidia/Qwen3.8-Flash-Next-NVFP4 (ModelOpt MIXED_PRECISION) + RTX PRO 6000, 1x or 2x DGX Spark (needs #38121: the dev-qwen38-next-local image or the Python install path) diff --git a/docs/cookbook/autoregressive/Qwen/Qwen3.mdx b/docs/cookbook/autoregressive/Qwen/Qwen3.mdx index f58397e7d011..4e5d64e5edbc 100644 --- a/docs/cookbook/autoregressive/Qwen/Qwen3.mdx +++ b/docs/cookbook/autoregressive/Qwen/Qwen3.mdx @@ -26,15 +26,13 @@ SGLang offers multiple installation methods. You can choose the most suitable in Please refer to the [official SGLang installation guide](../../../docs/get-started/install) for installation instructions. -For SGLang CPU installation, please refer to the [CPU version installation guide](../../../docs/hardware-platforms/cpu_server#installation). - ## 3. Model Deployment This section provides deployment configurations optimized for different hardware platforms and use cases. ### 3.1 Basic Configuration -The Qwen3 series offers models in various sizes and architectures, optimized for different hardware platforms including NVIDIA GPUs, AMD GPUs, and Intel Xeon CPUs. The recommended launch configurations vary by hardware and model size. +The Qwen3 series offers models in various sizes and architectures, optimized for different hardware platforms including NVIDIA GPUs, AMD GPUs, Intel Arc Pro B-Series GPUs(codename: BMG (Battlemage)), and Intel Xeon CPUs. The recommended launch configurations vary by hardware and model size. **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size, quantization method, and thinking capabilities. diff --git a/docs/cookbook/autoregressive/ThinkingMachines/Inkling-Small.mdx b/docs/cookbook/autoregressive/ThinkingMachines/Inkling-Small.mdx index 5bc7305aa503..ce9315173095 100644 --- a/docs/cookbook/autoregressive/ThinkingMachines/Inkling-Small.mdx +++ b/docs/cookbook/autoregressive/ThinkingMachines/Inkling-Small.mdx @@ -35,7 +35,7 @@ There are two multi-arch (amd64 / arm64) CUDA builds plus a ROCm build; pick the ```bash Command docker pull lmsysorg/sglang:dev-inkling-dspark # CUDA 13 -docker pull lmsysorg/sglang:dev-cu12-inkling-dspark # CUDA 12 +docker pull lmsysorg/sglang:dev-cu12-inkling-dspark # CUDA 12 (final build; lane retired) docker pull lmsysorg/sglang:dev-inkling-small-dgx-spark # DGX Spark (GB10 / SM121) docker pull lmsysorg/sglang-rocm:dev-rocm720-mi35x-inkling-dspark # AMD MI350X / MI355X ``` @@ -309,4 +309,4 @@ To try it, select the **Long Context** strategy in the Deploy panel above for an The **DSpark** deploy strategy is the second speculative-decoding path for Inkling-Small. Unlike **MTP**, which drives Inkling-Small's own multi-layer draft head, DSpark runs a **separate draft checkpoint** — `RadixArk/Inkling-Small-DSpark` — served unquantized alongside the NVFP4 target. -DSpark support ships in the images listed in §1 (`dev-inkling-dspark` for CUDA 13, `dev-cu12-inkling-dspark` for CUDA 12), so no separate build is needed. Verified end-to-end on B200 (TP=8, NVFP4). +DSpark support ships in the images listed in §1 (`dev-inkling-dspark` for CUDA 13, `dev-cu12-inkling-dspark` for CUDA 12 — a final build, since that lane is retired), so no separate build is needed. Verified end-to-end on B200 (TP=8, NVFP4). diff --git a/docs/cookbook/autoregressive/ThinkingMachines/Inkling.mdx b/docs/cookbook/autoregressive/ThinkingMachines/Inkling.mdx index b7a74b9798a1..4ac2d511bade 100644 --- a/docs/cookbook/autoregressive/ThinkingMachines/Inkling.mdx +++ b/docs/cookbook/autoregressive/ThinkingMachines/Inkling.mdx @@ -35,7 +35,7 @@ There are two multi-arch (amd64 / arm64) CUDA builds plus a ROCm build; pick the ```bash Command docker pull lmsysorg/sglang:dev-inkling-dspark # CUDA 13 -docker pull lmsysorg/sglang:dev-cu12-inkling-dspark # CUDA 12 +docker pull lmsysorg/sglang:dev-cu12-inkling-dspark # CUDA 12 (final build; lane retired) docker pull lmsysorg/sglang-rocm:dev-rocm720-mi35x-inkling-dspark # AMD MI350X / MI355X ``` @@ -306,4 +306,4 @@ To try it, select the **Long Context** strategy in the Deploy panel above for an The **DSpark** deploy strategy is the second speculative-decoding path for Inkling. Unlike **MTP**, which drives Inkling's own multi-layer draft head, DSpark runs a **separate draft checkpoint** — `RadixArk/Inkling-DSpark-Preview` — served unquantized alongside the NVFP4 target. -DSpark support ships in the images listed in §1 (`dev-inkling-dspark` for CUDA 13, `dev-cu12-inkling-dspark` for CUDA 12), so no separate build is needed. Verified end-to-end on B200 (TP=8, NVFP4). +DSpark support ships in the images listed in §1 (`dev-inkling-dspark` for CUDA 13, `dev-cu12-inkling-dspark` for CUDA 12 — a final build, since that lane is retired), so no separate build is needed. Verified end-to-end on B200 (TP=8, NVFP4). diff --git a/docs/cookbook/autoregressive/Xiaomi/MiMo-V2.5.mdx b/docs/cookbook/autoregressive/Xiaomi/MiMo-V2.5.mdx index 70e5799fc430..22f34259dd28 100644 --- a/docs/cookbook/autoregressive/Xiaomi/MiMo-V2.5.mdx +++ b/docs/cookbook/autoregressive/Xiaomi/MiMo-V2.5.mdx @@ -2,7 +2,6 @@ title: MiMo-V2.5 metatags: description: "Deploy XiaomiMiMo MiMo-V2.5-Pro (1.02T MoE, text) and MiMo-V2.5 (310B MoE, multimodal) with SGLang — EAGLE speculative decoding, hybrid attention, and 1M-token context." -tag: NEW --- ## 1. Model Introduction diff --git a/docs/cookbook/autoregressive/Xiaomi/MiMo-V2.6.mdx b/docs/cookbook/autoregressive/Xiaomi/MiMo-V2.6.mdx new file mode 100644 index 000000000000..475f269eaf47 --- /dev/null +++ b/docs/cookbook/autoregressive/Xiaomi/MiMo-V2.6.mdx @@ -0,0 +1,124 @@ +--- +title: MiMo-V2.6 +description: "Deploy MiMo-V2.6-Flash and Pro with SGLang on NVIDIA H200 and B300 GPUs, using MXFP4 MoE weights, BF16 routing, DFlash decoding, and a 1M-token context window." +tag: NEW +--- + +## Deployment + +The recipes below target **MiMo-V2.6-Flash on 4× H200 or B300** and **MiMo-V2.6-Pro on 8× H200 or B300**, each on a single node. Both use MXFP4 MoE weights, a BF16 MoE router, and DFlash speculative decoding. + + + + + +BF16 MoE routing and MXFP4 expert loading for MiMo V2.6 landed in [SGLang PR #40448](https://github.com/sgl-project/sglang/pull/40448) and are on `main`, so a recent nightly already carries them. Prepare an NVIDIA CUDA environment with FlashAttention 4, DeepGEMM, and DeepEP available. See the [installation guide](/docs/get-started/install) for platform prerequisites. + + + + +Install from source. The commit below is the revision these recipes were captured at; build from `main` instead if you want later fixes. + +```bash Command +git clone https://github.com/sgl-project/sglang.git sglang-mimo-v2.6 +cd sglang-mimo-v2.6 +git checkout 983e643854f15cf9ef4370a49dfd74b6af54c3e3 +python3 -m pip install -e ./python +``` + +Then run the **Python** command from the panel below. Checkpoint paths refer to directories on this host. + + + + +Pull the official nightly image, which already includes the MiMo-V2.6 support: + +```bash Command +docker pull lmsysorg/sglang:dev +``` + + + + + + +Select **Flash** or **Pro** to generate its command. Each variant has one base recipe; **Balanced** identifies that recipe and does not imply a measured throughput or latency optimum. The `/model/...` checkpoint and DFlash paths in the generated command are editable defaults, not download locations — set them to your own paths under **⚙ Env** before launching. Use the panel's cURL example to send a request to port **30000** after the server is ready. + +import { Deployment } from "/src/snippets/_deployment.jsx"; +import { config } from "/src/snippets/configs/XiaomiMiMo/mimo-v2.6.jsx"; + + + +## Playground + +Experiment with reasoning and tool-call parsers, or disable speculative decoding for comparison. These overrides inherit the selected Flash or Pro recipe; they require separate validation. Parallelism and attention backends remain at the PR settings. + +import { Playground } from "/src/snippets/_playground.jsx"; + + + +## 1. Model introduction + +**MiMo-V2.6-Flash** is an open-source foundation model developed by Xiaomi. Its Mixture-of-Experts architecture has **309B total parameters and 15B activated per token**, with hybrid attention for computational efficiency. It supports a **1M-token context window** and **native multimodal capabilities**, and is designed for coding, visual understanding, general-purpose assistance, and research in agentic workflows, including complex tasks that require many steps. + +**MiMo-V2.6-Pro** is Xiaomi's flagship foundation model, with **1.02T total parameters and 42B activated per token**, designed for demanding workloads. It also supports a **1M-token context window** and **native multimodal capabilities**, with an emphasis on coding, visual understanding, general-purpose assistance, research, and long-horizon agentic tasks. + + + + + + + + + + + + + + + + + + + + + + + + +
VariantTotal parametersContext windowRecipe (H200 / B300)
MiMo-V2.6-Flash309B (15B active)1M tokens4 GPUs, TP=EP=4
MiMo-V2.6-Pro1.02T (42B active)1M tokens8 GPUs, TP=EP=8
+ +**License:** MIT, continuing the MiMo-V2.5 licensing as confirmed by the model team. + +**Architecture references:** The [MiMo-V2.6-Flash-RL](https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL) and [MiMo-V2.6-Pro-RL](https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-RL) model cards document this generation directly — the hybrid sliding-window/global attention backbone, the omnimodal encoders, and the published evaluation scores — and each repository also ships the MiMo-V2.6 technical report. Both checkpoints declare `store_dtype: mxfp4` alongside `moe_router_dtype: bfloat16` in `config.json`, which is the expert-only MXFP4 quantization and BF16 routing these recipes depend on, and both carry the paired DFlash drafter in a `dflash/` subdirectory. + +## 2. Configuration tips + +### Parallelism and precision + +- **Flash:** `--tp 4 --ep 4`; **Pro:** `--tp 8 --ep 8`. Both keep `--dp 1 --pp-size 1`. TP and EP use the same GPU ranks; do not multiply them to determine the GPU count. +- **MoE compute and communication:** Keep `--moe-runner-backend deep_gemm` on B300 and `--moe-runner-backend marlin` on H200, with `--moe-a2a-backend deepep --deepep-mode auto`, together with `--moe-dense-tp-size 1` and `--enable-dp-lm-head`, as in the PR. +- **Precision:** MXFP4 describes the MoE expert weights, not the entire model. BF16 router weights are selected by the checkpoint's `moe_router_dtype`; the support commit computes router output logits in FP32. The PR relies on checkpoint quantization metadata and does not add a `--quantization` override. +- **Attention:** Keep `--attention-backend fa4` and `--mm-attention-backend fa4` on both H200 and B300. The recipe also enables `--mm-enable-dp-encoder` for multimodal encoding. + +### Context length and memory + +Both commands set `--context-length 1048576`. This is the configured per-request context limit; it does not establish that 64 simultaneous 1M-token requests fit in memory. + +Flash uses `--mem-fraction-static 0.6 --swa-full-tokens-ratio 0.03`; Pro uses `--mem-fraction-static 0.7 --swa-full-tokens-ratio 0.08`. The memory fraction budgets weights and KV cache, while the SWA ratio controls sliding-window versus full-attention KV allocation. Preserve these model-specific values for the base recipe. + +The shared prefill settings are `--chunked-prefill-size 49152 --max-prefill-tokens 65536`. Both commands cap running requests and decode CUDA-graph batch size at **64**, disable prefill CUDA graphs, and use page size **1**. `--enable-cache-report` and `--log-level-http warning` retain the PR's reporting settings. + +### DFlash and caching + +The base recipe uses `--speculative-algorithm DFLASH --speculative-num-draft-tokens 8`. Supply the DFlash checkpoint paired with the selected target model; the draft path is independently editable and does not have to be a subdirectory of the target checkpoint. + +At the pinned commit, DFlash on NVIDIA GPUs requires PP=1 and rejects DP-attention. Do not add `--enable-dp-attention` or copy EAGLE-specific settings from the V2.5 cookbook. + +## 3. Reasoning and tool calling + +Both launch commands enable `--reasoning-parser mimo --tool-call-parser mimo`. The MiMo reasoning parser uses the request's `chat_template_kwargs.enable_thinking` setting: set it to `true` for reasoning or `false` for a direct answer. The command panel's cURL example enables reasoning explicitly. + +For OpenAI-compatible clients, read reasoning from `choices[0].message.reasoning_content`, the final answer from `choices[0].message.content`, and structured tool requests from `choices[0].message.tool_calls`. Preserve both reasoning and answer fields when inspecting responses. Use the checkpoint's generation defaults unless your application needs explicit sampling settings. + +Runtime throughput, latency, accuracy, and example responses for V2.6 will be added when measurements from these checkpoints are available. diff --git a/docs/cookbook/autoregressive/intro.mdx b/docs/cookbook/autoregressive/intro.mdx index 69b565570245..17b8eb4a8664 100644 --- a/docs/cookbook/autoregressive/intro.mdx +++ b/docs/cookbook/autoregressive/intro.mdx @@ -7,6 +7,12 @@ metatags: --- + diff --git a/docs/cookbook/diffusion/Ernie-Image/Ernie-Image.mdx b/docs/cookbook/diffusion/Ernie-Image/Ernie-Image.mdx index a4c420a07062..e8c6668099ac 100644 --- a/docs/cookbook/diffusion/Ernie-Image/Ernie-Image.mdx +++ b/docs/cookbook/diffusion/Ernie-Image/Ernie-Image.mdx @@ -81,3 +81,9 @@ with open("ernie_image.png", "wb") as f: - `--performance-mode auto` keeps conservative defaults while preserving explicit user flags. - If the checkpoint includes a PE component, SGLang loads it automatically with the native Ministral3 runtime. Use `--layerwise-offload-components pe` when the local PE decoder needs to trade latency for lower GPU memory usage. - Treat FSDP, SP/Ulysses/Ring, and TP as explicit benchmark knobs. Measure the target resolution, step count, and GPU type before making them production defaults. + +## 6. Run in ComfyUI + +import { ComfyUISupport } from '/src/snippets/diffusion/comfyui-support.jsx'; + + diff --git a/docs/cookbook/diffusion/FLUX/FLUX.mdx b/docs/cookbook/diffusion/FLUX/FLUX.mdx index 05c0c4debe24..4cb84c7d88d2 100644 --- a/docs/cookbook/diffusion/FLUX/FLUX.mdx +++ b/docs/cookbook/diffusion/FLUX/FLUX.mdx @@ -34,13 +34,13 @@ This section provides deployment configurations optimized for different hardware FLUX models are optimized for high-quality image generation. The recommended launch configurations vary by hardware and model version. -**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and model version. SGLang supports serving FLUX on NVIDIA B200, H200, H100, and AMD MI355X, MI325X, MI300X GPUs and Ascend A2, A3 NPUs. +**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and model version. SGLang supports serving FLUX on NVIDIA B200, H200, H100, and AMD MI355X, MI325X, MI300X GPUs, Ascend A2, A3 NPUs Series NPUs and Intel Arc B-series graphics(codename: BMG (Battlemage)). ### 3.2 Configuration Tips -Currently supported optimizations are listed [here](/docs/sglang-diffusion/compatibility_matrix). +See [Performance Optimization](/docs/sglang-diffusion/performance-optimization) for acceleration features and their runtime requirements. - `--vae-path`: Path to a custom VAE model or HuggingFace model ID (e.g., fal/FLUX.2-Tiny-AutoEncoder). If not specified, the VAE will be loaded from the main model path. - `--num-gpus`: Number of GPUs to use @@ -49,6 +49,31 @@ Currently supported optimizations are listed [here](/docs/sglang-diffusion/compa - `--ulysses-degree`: The degree of DeepSpeed-Ulysses-style SP in USP - `--ring-degree`: The degree of ring attention-style SP in USP +### 3.3 Breakable CUDA graph for FLUX.1-dev + +For repeated `quality=lossless` requests, FLUX.1-dev supports breakable CUDA graph (BCG) execution with the native backend. BCG captures DiT segments during startup and replays them for the warmed resolutions. It can reduce recurring host launch overhead without enabling `torch.compile`. + +The following configuration was validated on two NVIDIA H200 GPUs with BF16 weights, PyTorch 2.13, and CUDA 13.0. Set `HF_TOKEN` to a Hugging Face token with access to the checkpoint before downloading it. + +```bash +CUDA_VISIBLE_DEVICES=0,1 sglang generate \ + --model-path black-forest-labs/FLUX.1-dev \ + --backend sglang \ + --num-gpus 2 --tp-size 2 \ + --component-residency dit=resident \ + --enable-torch-compile false \ + --enable-breakable-cuda-graph \ + --warmup-resolutions 1024x1024 \ + --quality lossless \ + --width 1024 --height 1024 \ + --num-inference-steps 50 --guidance-scale 3.5 --seed 42 \ + --prompt "A futuristic cyberpunk city at night, neon lights reflecting on wet streets" +``` + +Confirm `[Diffusion BCG] captured` in the log, then compare warmed request latency with eager execution on the same GPUs. Capture time and graph memory are additional startup costs. Add other served resolutions to `--warmup-resolutions`; a request with an uncaptured signature falls back to eager. + +FLUX.1-dev uses a fixed 512-token T5 conditioning sequence, so changing `--bcg-text-buckets` does not create additional prompt-length graphs. Its request-gated DiT fusions at `quality=high` and `extra-high` cannot be combined with BCG: the runtime rejects those requests because the captured graph uses the lossless branches. FLUX.2 and quantized transformer overrides require separate validation. + ## 4. API Usage For complete API documentation, please refer to the [official API usage guide](/docs/sglang-diffusion/api/openai_api). @@ -193,6 +218,14 @@ sglang serve --model-path black-forest-labs/FLUX.1-dev - `--vae-cpu-offload`: Use CPU offload for VAE. - `--pin-cpu-memory`: Pin memory for CPU offload. Only added as a temp workaround if it throws "CUDA error: invalid argument". +#### 4.2.3 Known LoRA examples + +Use `--lora-path` at startup or the [LoRA management API](/docs/sglang-diffusion/api/openai_api#lora-management) to load an adapter. Known FLUX examples include: + +- [`dvyio/flux-lora-simple-illustration`](https://huggingface.co/dvyio/flux-lora-simple-illustration) +- [`XLabs-AI/flux-furry-lora`](https://huggingface.co/XLabs-AI/flux-furry-lora) +- [`XLabs-AI/flux-RealismLora`](https://huggingface.co/XLabs-AI/flux-RealismLora) + ## 5. Benchmark ### 5.1 Speedup Benchmark @@ -244,11 +277,11 @@ Test Environment: ============================================================ ``` - + **Server Command**: ```shell Command - #One A3 card has 2 npu chips + #One A3 Series card has 2 npu chips sglang serve --tp-size 2 --sp-degree 1 --model-path black-forest-labs/FLUX.1-dev --num-gpus 2 ``` @@ -330,11 +363,11 @@ Test Environment: ============================================================ ``` - + **Server Command** : ```shell Command - #One A3 card has 2 npu chips + #One A3 Series card has 2 npu chips sglang serve --tp-size 2 --sp-degree 1 --model-path black-forest-labs/FLUX.1-dev --num-gpus 2 ``` @@ -374,3 +407,9 @@ Test Environment: ``` + +## 6. Run in ComfyUI + +import { ComfyUISupport } from '/src/snippets/diffusion/comfyui-support.jsx'; + + diff --git a/docs/cookbook/diffusion/Ideogram/Ideogram4.mdx b/docs/cookbook/diffusion/Ideogram/Ideogram4.mdx index c02530e6a8ce..7b46738e25e5 100644 --- a/docs/cookbook/diffusion/Ideogram/Ideogram4.mdx +++ b/docs/cookbook/diffusion/Ideogram/Ideogram4.mdx @@ -187,3 +187,15 @@ response = client.images.generate( ``` Base Ideogram 4 presets are `V4_DEFAULT_20`, `V4_QUALITY_48`, and `V4_TURBO_12`. The fal variants automatically select `V4_FAST_20` and `V4_INSTANT_8`, respectively. A preset controls both `num_inference_steps` and guidance, so do not set those fields directly. + +You can also produce captions with a separate SRT LLM using +[prompt enhancement](/docs/sglang-diffusion/models_with_pe). Configure an +Ideogram-specific JSON caption template and schema; the generic prose template +is not appropriate for these checkpoints. This does not reproduce fal's +checkpoint-specific prompt expander or establish equivalent image quality. + +## 5. Run in ComfyUI + +import { ComfyUISupport } from '/src/snippets/diffusion/comfyui-support.jsx'; + + diff --git a/docs/cookbook/diffusion/JoyEcho/JoyEcho.mdx b/docs/cookbook/diffusion/JoyEcho/JoyEcho.mdx index e9ab40e44337..f41d7b881688 100644 --- a/docs/cookbook/diffusion/JoyEcho/JoyEcho.mdx +++ b/docs/cookbook/diffusion/JoyEcho/JoyEcho.mdx @@ -15,7 +15,7 @@ import { DiffusionModelTags } from '/src/snippets/diffusion/model-tags.jsx'; Choose JoyEcho over a standard LTX pipeline when shots must share audiovisual memory. Its distilled 832×480 path prioritizes long-form continuity and throughput rather than the higher-resolution two-stage quality modes offered by LTX-2.3. -Use `jdopensource/JoyAI-Echo` as `--model-path`; SGLang materializes the monolithic release through the built-in [JoyAI-Echo overlay](https://huggingface.co/Niehen6174/JoyAI-Echo-overlay). +SGLang materializes the Echo 1.0 monolithic release through the built-in [JoyAI-Echo overlay](https://huggingface.co/Niehen6174/JoyAI-Echo-overlay). Prepare the pinned checkpoint below before running the examples. | Aspect | Standard LTX-2.3 | JoyEcho | | --- | --- | --- | @@ -41,11 +41,33 @@ For platform-specific setup, see the [SGLang Diffusion installation guide](/docs ## 3. Model Deployment +### 3.1 Prepare the Echo 1.0 checkpoint + +The native overlay requires `JoyAI-Echo-release.safetensors`. The upstream repository's Echo 1.5 revision does not contain that file. Download the [Echo 1.0 revision](https://huggingface.co/jdopensource/JoyAI-Echo/tree/4187f9a53c6eff3a76c51e79bd27f70d10f7591b) into the Hugging Face cache: + +```bash +JOY_ECHO_MODEL_PATH=$(python - <<'PY' +from huggingface_hub import snapshot_download + +print(snapshot_download( + repo_id="jdopensource/JoyAI-Echo", + revision="4187f9a53c6eff3a76c51e79bd27f70d10f7591b", + allow_patterns=["JoyAI-Echo-release.safetensors", "*.json", "*.md", "LICENSE"], +)) +PY +) +``` + +Use the returned cache path as `--model-path` and keep `--model-id jdopensource/JoyAI-Echo` when using local weights. The model ID lets BCG select JoyEcho's support policy. Initial startup also downloads and materializes the overlay and its text encoder dependencies. + +### 3.2 Serve the model + JoyEcho uses the default `JoyEchoPipeline` registered for `jdopensource/JoyAI-Echo`. A single high-VRAM GPU (for example H100 or H200) is enough for the common 832x480 / 121-frame / 8-step setting. ```bash sglang serve \ - --model-path jdopensource/JoyAI-Echo + --model-path "$JOY_ECHO_MODEL_PATH" \ + --model-id jdopensource/JoyAI-Echo ``` Optional environment variable for long runs: @@ -58,7 +80,8 @@ For multi-GPU serving, tensor parallelism (TP) and **Ulysses sequence parallelis ```bash sglang serve \ - --model-path jdopensource/JoyAI-Echo \ + --model-path "$JOY_ECHO_MODEL_PATH" \ + --model-id jdopensource/JoyAI-Echo \ --num-gpus 2 \ --ulysses-degree 2 ``` @@ -84,7 +107,8 @@ JoyEcho SP currently targets **Ulysses-only** parallelism (`ulysses_degree=2`, ` ```bash sglang generate \ - --model-path jdopensource/JoyAI-Echo \ + --model-path "$JOY_ECHO_MODEL_PATH" \ + --model-id jdopensource/JoyAI-Echo \ --prompt "A curious raccoon walks through a sunlit forest path" \ --height 480 --width 832 --num-frames 121 --fps 25 \ --num-inference-steps 8 --seed 42 \ @@ -96,7 +120,6 @@ Disable the memory bank for standalone clips with a config file: ```bash cat > /tmp/joy_echo_single.json <<'EOF' { - "model_path": "jdopensource/JoyAI-Echo", "prompt": "A curious raccoon walks through a sunlit forest path", "enable_memory_bank": false, "seed": 42, @@ -108,7 +131,9 @@ cat > /tmp/joy_echo_single.json <<'EOF' } EOF -sglang generate --config /tmp/joy_echo_single.json --save-output +sglang generate --config /tmp/joy_echo_single.json \ + --model-path "$JOY_ECHO_MODEL_PATH" --model-id jdopensource/JoyAI-Echo \ + --save-output ``` ### 4.3 Multi-shot generation @@ -127,7 +152,6 @@ Pass multiple prompts as a list in a config file: ```bash cat > /tmp/joy_echo_4shot.json <<'EOF' { - "model_path": "jdopensource/JoyAI-Echo", "prompt": [ "Shot 0: A raccoon wakes up in a cozy attic.", "Shot 1: The raccoon climbs down and opens the back door.", @@ -145,14 +169,17 @@ cat > /tmp/joy_echo_4shot.json <<'EOF' } EOF -sglang generate --config /tmp/joy_echo_4shot.json --save-output +sglang generate --config /tmp/joy_echo_4shot.json \ + --model-path "$JOY_ECHO_MODEL_PATH" --model-id jdopensource/JoyAI-Echo \ + --save-output ``` You can also pass prompts from a text file (one prompt per line) with `--prompt-path`: ```bash sglang generate \ - --model-path jdopensource/JoyAI-Echo \ + --model-path "$JOY_ECHO_MODEL_PATH" \ + --model-id jdopensource/JoyAI-Echo \ --prompt-path /tmp/joy_echo_shots.txt \ --seed 42 \ --height 480 --width 832 --num-frames 121 --fps 25 \ @@ -169,6 +196,40 @@ sglang generate \ Set `enable_memory_bank=false` when you want independent shots without cross-shot continuity. +### 4.5 Measured two-H200 single-shot configuration + +For short independent clips, keep the text and audio/video components on GPU with `--component-residency=all=resident`. Full-stage profiles showed that this removes repeated host-to-device weight copies between component uses. + +The following configuration was measured on two H200s with Ulysses degree 2, TP1, PyTorch 2.11.0+cu130, 640x384, 33 frames, 8 steps and seed 42. It disables compilation and the memory bank: + +```bash +cat > joy_echo_h200.json <<'EOF' +{"enable_memory_bank": false} +EOF + +CUDA_VISIBLE_DEVICES=0,1 sglang generate \ + --model-path "$JOY_ECHO_MODEL_PATH" --model-id jdopensource/JoyAI-Echo \ + --config joy_echo_h200.json --prompt "A curious raccoon" \ + --width 640 --height 384 --num-frames 33 --num-inference-steps 8 --seed 42 \ + --num-gpus 2 --ulysses-degree 2 \ + --performance-mode manual --enable-torch-compile=false --quality lossless \ + --component-residency=all=resident --warmup-mode request \ + --save-output --perf-dump-path joy_echo_h200.json.perf +``` + +For BCG, add `--enable-breakable-cuda-graph --warmup-resolutions 640x384 --warmup-num-frames 33`. Check for successful `[Diffusion BCG] captured` logs and absence of request signature misses. Keep the same model ID, resolution, frame count and quality as warmup. + +Two fresh-process saved requests per configuration, after request warmup: + +| Lossless mode | Auto residency E2E | All resident E2E | Reduction | Peak reserved per rank, auto → resident | +| --- | ---: | ---: | ---: | ---: | +| Eager | 2.563 s | 2.366 s | 7.65% | 46.43 → 69.32 GiB | +| BCG | 1.229 s | 1.058 s | 13.88% | 48.77–48.88 → 69.32 GiB | + +Loading, warmup and profiling are excluded from these timings. The paired eager profiles remove 71 pinned host-to-device copies (13.07 GB, 278.86 ms on the profiled rank), with the same 77,045 kernel launches. All eight lossless outputs have pixel-identical video frames; audio differences are comparable to baseline repeat variability. These results cover this compact single-shot workload, rather than the default 121-frame or multi-shot memory-bank workload. + +Use `quality=lossless` for this recipe. High-mode output did not pass the separate quality comparison, and high + BCG is rejected by the runtime. If a larger request exceeds available memory, return to auto residency or keep only selected components resident. + ## 5. Practical Tips - Use `--num-inference-steps 8` and `--guidance-scale 1.0` to match the official JoyEcho DMD distilled path. @@ -177,3 +238,9 @@ Set `enable_memory_bank=false` when you want independent shots without cross-sho - For **2-GPU latency**, try **Ulysses SP** (`--num-gpus 2 --ulysses-degree 2`) on both single-shot and multi-shot runs. Use **TP** when you need a different sharding strategy or more than two GPUs. - Set `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` for long multi-shot SP sessions. - JoyEcho outputs per-shot mp4 files with synchronized audio. There is no built-in two-stage HQ upscaling path like LTX-2.3 HQ. + +## 6. Run in ComfyUI + +import { ComfyUISupport } from '/src/snippets/diffusion/comfyui-support.jsx'; + + diff --git a/docs/cookbook/diffusion/Krea/Krea-2.mdx b/docs/cookbook/diffusion/Krea/Krea-2.mdx index 725d2dc1f2b8..afc40cfea585 100644 --- a/docs/cookbook/diffusion/Krea/Krea-2.mdx +++ b/docs/cookbook/diffusion/Krea/Krea-2.mdx @@ -44,7 +44,7 @@ The step count and guidance scale are **request-time** settings (see [API Usage] ### 3.2 Configuration Tips -Currently supported optimizations are listed [here](/docs/sglang-diffusion/compatibility_matrix). +See [Performance Optimization](/docs/sglang-diffusion/performance-optimization) for acceleration features and their runtime requirements. - `--num-gpus`: Number of GPUs to use. - Multi-GPU (tensor and/or sequence parallelism): see [Section 3.3](#3-3-multi-gpu-tensor-and-sequence-parallelism). @@ -334,3 +334,9 @@ Peak Memory Mean (MB): 37466.40 Peak Memory Median (MB): 37466.00 ============================================================ ``` + +## 6. Run in ComfyUI + +import { ComfyUISupport } from '/src/snippets/diffusion/comfyui-support.jsx'; + + diff --git a/docs/cookbook/diffusion/LTX/LTX2 & LTX2.3.mdx b/docs/cookbook/diffusion/LTX/LTX2 & LTX2.3.mdx index 56f22bafe251..99f4f11c83cb 100644 --- a/docs/cookbook/diffusion/LTX/LTX2 & LTX2.3.mdx +++ b/docs/cookbook/diffusion/LTX/LTX2 & LTX2.3.mdx @@ -248,3 +248,9 @@ Some community LoRAs only include weights for transformer blocks. In that case, - Use `--ltx2-two-stage-device-mode resident` on high-VRAM GPUs if latency matters more than memory usage. - Use `--ltx2-two-stage-device-mode original` when comparing against official two-stage behavior. - Keep `--width` and `--height` aligned with the target model resolution; for LTX models, these are output video dimensions. + +## 6. Run in ComfyUI + +import { ComfyUISupport } from '/src/snippets/diffusion/comfyui-support.jsx'; + + diff --git a/docs/cookbook/diffusion/LTX/LTX2.5.mdx b/docs/cookbook/diffusion/LTX/LTX2.5.mdx index 94def6262cbb..6a75c25b40ca 100644 --- a/docs/cookbook/diffusion/LTX/LTX2.5.mdx +++ b/docs/cookbook/diffusion/LTX/LTX2.5.mdx @@ -128,6 +128,7 @@ Selecting weights: | 2 GPUs, long sequences | `--num-gpus 2 --ulysses-degree 2` | Sequence parallel; the memory/long-sequence tool. | | 2 GPUs, large DiT | `--num-gpus 2 --tp-size 2` | Tensor parallel across attention heads. | | 2 GPUs, dev weights | `--num-gpus 2 --enable-cfg-parallel` | Splits the guided and unguided branches across GPUs. Measured 1.77x on denoising (15.1s to 8.5s, 960×544 / 57 frames / 30 steps). | +| 2 GPUs, diffusion decoder | `--num-gpus 2 --ulysses-degree 2` | The decoder's tiles are split across the ranks by default. See [section 4.6.1](#4-6-1-memory-and-multi-gpu). | **CFG parallelism does not apply on the default (distilled) path.** That DiT @@ -340,3 +341,63 @@ uv pip install natten==0.21.6+torch2110cu130 -f https://whl.natten.org/ Nothing else changes if you skip it: the decoder still produces the same video, just slower. + +#### 4.6.1 Memory and multi-GPU + +Two flags govern how the decode is executed. Both default to on, so the numbers +below are what you already get — they matter when you want to turn one off. + +`--diffusion-decoder-tiling` runs the decoder's two expensive stages over +overlapping tiles instead of the whole volume. It is a **memory** control, not a +speed one: it costs wall clock and buys headroom. + +`--diffusion-decoder-parallel-tiling` splits those tiles across the +decode-parallel ranks -- the TP, SP, PP and CFG ranks of one replica, since the +decoder is replicated over all of them. Without it every one of those ranks +decodes every tile and keeps its own identical copy. It only applies on the +tiled path, so it does nothing when tiling is off, and nothing at a single +rank. + +Decoding stage on 2xH200 at 960×544, Ulysses degree 2. "Peak" is the whole +process, not the decode alone: + +| Frames | Tiling | Parallel tiling | Decode | Peak | +| --- | --- | --- | --- | --- | +| 121 | off | *(n/a)* | **3.38s** | 103.2 GB | +| 121 | on | off | 5.68s | 78.2 GB | +| 121 | on | on | 4.08s | 79.3 GB | +| 49 | off | *(n/a)* | **1.93s** | 84.0 GB | +| 49 | on | off | 2.79s | 78.2 GB | +| 49 | on | on | 2.12s | 78.2 GB | + +Reading that: + +- **Untiled is the fastest option** whenever it fits. Tiling exists for the + ~25 GB it saves at 121 frames, which is the difference between fitting on an + 80 GB card and not. +- **Parallel tiling recovers most of tiling's cost** — 1.39x at 121 frames, + 1.32x at 49 — but does not beat an untiled decode. It also adds about 1 GB + for the gather buffers. +- The gain grows with tile count, so it is larger at higher resolution: at + 1920×1088 / 49 frames the decode goes from 12.32s to 7.86s, **1.57x**. + +Output is bitwise identical however many ranks the tiles are split over: every +rank draws the whole grid's noise in the same order, and only the decode is +shared out. Turning *tiling* on or off does change the result slightly near +tile borders, so pick one and stay with it if you need reproducible frames: + +```bash +# Fastest, if the untiled decode fits in VRAM +sglang serve \ + --model-path Lightricks/LTX-2.5-Diffusers \ + --pipeline-class-name LTX2Pipeline \ + --load-diffusion-decoder \ + --diffusion-decoder-tiling false + +# Memory-bound: keep tiling, and split the tiles over both GPUs +sglang serve \ + --model-path Lightricks/LTX-2.5-Diffusers \ + --pipeline-class-name LTX2Pipeline \ + --load-diffusion-decoder \ + --num-gpus 2 --ulysses-degree 2 +``` diff --git a/docs/cookbook/diffusion/LingBot-World/LingBot-World-2.0.mdx b/docs/cookbook/diffusion/LingBot-World/LingBot-World-2.0.mdx index 73f4c5071f0a..384919e1abca 100644 --- a/docs/cookbook/diffusion/LingBot-World/LingBot-World-2.0.mdx +++ b/docs/cookbook/diffusion/LingBot-World/LingBot-World-2.0.mdx @@ -84,7 +84,7 @@ Send this MessagePack map immediately after the WebSocket opens. | `num_inference_steps` | integer | No | Denoising steps per chunk. LingBot defaults to `4` when omitted. | | `guidance_scale` | number | No | Classifier-free guidance scale. Realtime LingBot commonly uses `1`. | | `negative_prompt` | string | No | Negative prompt passed to the diffusion pipeline. | -| `quality` | `"lossless"`, `"high"` | No | `lossless` keeps FP32 VAE decode. `high` uses the validated BF16 decode path for lower per-chunk latency. | +| `quality` | `"lossless"`, `"extra-high"`, `"high"` | No | `lossless` keeps FP32 VAE decode. `extra-high` keeps FP32 decode and enables only any eligible request-gated kernel fusions. `high` includes those fusions and uses the validated BF16 decode path for lower per-chunk latency. | | `max_chunks` | integer | No | Stop after this many chunks. Omit for a continuous session. | | `realtime_causal_sink_size` | integer | No | Number of sink frames/tokens retained in the causal attention window. | | `realtime_causal_kv_cache_num_frames` | integer | No | Number of recent frames retained in the causal KV cache window. | @@ -184,3 +184,9 @@ LingBot World 2.0 uses raw-frame websocket GT plus per-chunk latency guards for - Use the realtime endpoint for interactive sessions: `/v1/realtime_video/generate`. - Prefer WebP preview transport for interactive testing; use raw-frame transport for consistency checks. - Long-running sessions should be validated with raw-frame consistency before changing causal cache, condition sampling, or VAE decode behavior. + +## 7. Run in ComfyUI + +import { ComfyUISupport } from '/src/snippets/diffusion/comfyui-support.jsx'; + + diff --git a/docs/cookbook/diffusion/LingBot-World/LingBot-World.mdx b/docs/cookbook/diffusion/LingBot-World/LingBot-World.mdx index 9ffc1c326277..a3dea5325994 100644 --- a/docs/cookbook/diffusion/LingBot-World/LingBot-World.mdx +++ b/docs/cookbook/diffusion/LingBot-World/LingBot-World.mdx @@ -79,7 +79,7 @@ Send this MessagePack map immediately after the WebSocket opens. | `num_inference_steps` | integer | No | Denoising steps per chunk. LingBot defaults to `4` when omitted. | | `guidance_scale` | number | No | Classifier-free guidance scale. Realtime LingBot commonly uses `1`. | | `negative_prompt` | string | No | Negative prompt passed to the diffusion pipeline. | -| `quality` | `"lossless"`, `"high"` | No | `lossless` keeps FP32 VAE decode. `high` uses the validated BF16 decode path for lower per-chunk latency. | +| `quality` | `"lossless"`, `"extra-high"`, `"high"` | No | `lossless` keeps FP32 VAE decode. `extra-high` keeps FP32 decode and enables only any eligible request-gated kernel fusions. `high` includes those fusions and uses the validated BF16 decode path for lower per-chunk latency. | | `max_chunks` | integer | No | Stop after this many chunks. Omit for a continuous session. | | `realtime_causal_sink_size` | integer | No | Number of sink frames/tokens retained in the causal attention window. | | `realtime_causal_kv_cache_num_frames` | integer | No | Number of recent frames retained in the causal KV cache window. | @@ -159,3 +159,9 @@ LingBot World uses raw-frame websocket GT plus per-chunk latency guards for cons - Use the realtime endpoint for interactive sessions: `/v1/realtime_video/generate`. - Prefer WebP preview transport for interactive testing; use raw-frame transport for consistency checks. - Long-running sessions should be validated with raw-frame consistency before changing causal cache, condition sampling, or VAE decode behavior. + +## 7. Run in ComfyUI + +import { ComfyUISupport } from '/src/snippets/diffusion/comfyui-support.jsx'; + + diff --git a/docs/cookbook/diffusion/LongLive/LongLive-2.0.mdx b/docs/cookbook/diffusion/LongLive/LongLive-2.0.mdx index 4da75d45ca14..0d9a8363dff3 100644 --- a/docs/cookbook/diffusion/LongLive/LongLive-2.0.mdx +++ b/docs/cookbook/diffusion/LongLive/LongLive-2.0.mdx @@ -117,3 +117,9 @@ The image is used as the first-frame condition. - SGLang supports T2V sizes 1280x704, 704x1280, 832x480, and 480x832. - I2V request images follow the Wan TI2V preprocessing path in SGLang. This is different from the original LongLive dataset resize path. - For multi-shot runs, set `num_frames` to match `len(shot_prompts) * chunks_per_shot * 8` latent frames, that is `num_frames = (len(shot_prompts) * chunks_per_shot * 8 - 1) * 4 + 1`. + +## 6. Run in ComfyUI + +import { ComfyUISupport } from '/src/snippets/diffusion/comfyui-support.jsx'; + + diff --git a/docs/cookbook/diffusion/MOVA/MOVA.mdx b/docs/cookbook/diffusion/MOVA/MOVA.mdx index fb0053ac97f9..2f9c9aa74397 100644 --- a/docs/cookbook/diffusion/MOVA/MOVA.mdx +++ b/docs/cookbook/diffusion/MOVA/MOVA.mdx @@ -41,7 +41,7 @@ import { MOVADeployment } from '/src/snippets/diffusion/mova-deployment.jsx' ### 3.2 Configuration Tips -Currently supported optimizations are listed [here](/docs/sglang-diffusion/compatibility_matrix). +See [Performance Optimization](/docs/sglang-diffusion/performance-optimization) for acceleration features and their runtime requirements. - `--num-gpus`: Number of GPUs to use - `--tp`: Tensor parallelism size (should not be larger than 1 if text encoder offload is enabled, as layer-wise offload plus prefetch is faster) @@ -266,3 +266,9 @@ python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ --task image-to-video --dataset vbench --num-prompts 20 --max-concurrency 20 \ --port 30002 ``` + +## 6. Run in ComfyUI + +import { ComfyUISupport } from '/src/snippets/diffusion/comfyui-support.jsx'; + + diff --git a/docs/cookbook/diffusion/MiniMax/MiniMax-H3.mdx b/docs/cookbook/diffusion/MiniMax/MiniMax-H3.mdx index 90cb3f0ff911..6f15081ec4d3 100644 --- a/docs/cookbook/diffusion/MiniMax/MiniMax-H3.mdx +++ b/docs/cookbook/diffusion/MiniMax/MiniMax-H3.mdx @@ -28,7 +28,8 @@ reload the model. The Docker form installs the platform-specific diffusion extra from the source bundled in the image. For conditioned requests, set **Host media directory** under **Variables**; the builder mounts it read-only at `/data/minimax-h3`. -AMD currently offers the Python form, while NVIDIA also offers Docker. +AMD and Grace Blackwell currently offer the Python form; the other NVIDIA +recipes also offer Docker. To use ModelScope through the same normal `sglang serve` path, prefix the copied command with `SGLANG_USE_MODELSCOPE=true` and replace the model path with @@ -60,6 +61,8 @@ or `MiniMax/MiniMax-H3` on ModelScope. Select the checkpoint variant with `--model-variant`: `fl2va` serves both `t2va` and `fl2va`, while `ref2va` serves reference-conditioned requests. SGLang owns the checkpoint-directory mapping; do not point `--model-path` at a manually downloaded subdirectory. +The same FL2VA / Ref2VA split applies in +[ComfyUI integrated mode](#comfyui-integrated-mode). Review the license and usage terms in the MiniMax-H3 model card before production or commercial use. SGLang support does not grant additional model usage rights. @@ -72,6 +75,29 @@ The builder accepts legal custom GPU counts and topologies, marking them Static H3 head or partition violations disable Copy before they reach `sglang serve`. +### Grace Blackwell + +Select **GB300** or **GB200** in the builder. The default is one Linux ARM64 +host with four GPUs, Ulysses4, and resident weights. This follows the +[four-GPU compute-tray layout](https://docs.nvidia.com/dgx/dgxgb200-user-guide/hardware.html); +an NVL72 rack is not a single 72-GPU process host. + +For two four-GPU hosts, set **Nodes = 2** and **GPUs / node = 4**. The builder +emits Ulysses4 inside each host, Ring2 across hosts, and replicated encoders. +Set the same reachable **Head address** on both hosts and use node ranks 0 and +1. Run the generated command once per host, with the same SGLang revision and +dependencies. Let `sglang serve` resolve and download the selected weights. + +GB200 is a **derived, unverified recipe**, not a measured GB300-equivalent +performance claim. Keep the native precision and platform-default attention +for the baseline; the B200/B300 quantization measurements and H200 Cache-DiT +quality audit do not establish coverage on Grace Blackwell. + +GB300 end-to-end coverage currently includes single-host FL2VA text-to-video +with audio, native precision, eager execution, and `quality="lossless"`. +Ref2VA and multi-host commands remain **Unverified**. See the +[GB300 measurements](#gb300-single-host) for the tested workload. + ### Checkpoint and adapter formats Start with the command emitted by the builder. Every row below is an overlay on @@ -86,13 +112,16 @@ may be either a normal style adapter or a timestep-distilled Turbo adapter. | DiT | Official Diffusers component layout | `--component-paths.transformer MiniMaxAI/MiniMax-H3/transformer` (`fl2va`) or `.../transformer_ref` (`ref2va`) | Loads the official component through the native SGLang graph; no Diffusers runtime fallback. | | DiT | [AdaLN-pruned Diffusers component](https://huggingface.co/multimodalart/MiniMax-H3-Pruned) | `--component-paths.transformer multimodalart/MiniMax-H3-Pruned/transformer` or `.../transformer_ref` | Approximate curve-AdaLN architecture; its config and basis metadata are loaded natively. | | DiT | [Full or AdaLN-pruned](https://huggingface.co/Comfy-Org/MiniMax-H3/tree/main/diffusion_models), or [LoRA-merged/remixed](https://huggingface.co/FX-FeiHou/MiniMax-H3-Remix) BF16 safetensors | `--component-weights-paths.transformer OWNER/REPO/path/FILE.safetensors` | Weight-only override for a native full/pruned H3 layout. Match the FL2VA/Ref2VA partition; pruned, merged, or dtype-converted exports are approximate, and any author-specific sampler remains a separate requirement. | +| DiT | FL/Ref hybrid fine-tunes, including [Singularity v1.3](https://huggingface.co/WarmBloodAban/Minimax-h3_Singularity) (full or pruned INT8) | Replace `--model-variant` with `hybrid`; add `--component-weights-paths.transformer OWNER/REPO/FILE.safetensors` | Explicit merged-weight deployment serving `t2va`, `fl2va`, and `ref2va` on one native pipeline. Quantization and pruning are detected from the file; INT8 requires `comfy-kitchen`. See [Singularity](#singularity-hybrid-weights). | | DiT | Comfy FP8 or self-describing MXFP8 safetensors | `--component-weights-paths.transformer OWNER/REPO/path/FILE.safetensors` | Per-layer metadata selects static/dynamic FP8 or MXFP8 automatically. | | DiT | ConvRot INT8, W4A8, W4A4, or mixed W4A4+INT8 safetensors ([INT8](https://huggingface.co/Comfy-Org/MiniMax-H3/tree/main/diffusion_models), [W4A8](https://huggingface.co/Winnougan/MiniMax-H3-INT4_Convrot_ComfyUI), [W4A4](https://huggingface.co/Merserk/MiniMax-H3-INT4-ConvRot)) | `--component-weights-paths.transformer OWNER/REPO/path/FILE.safetensors` | Auto-detected; requires `comfy-kitchen`. TP must preserve each file's ConvRot group boundaries. | | DiT | [NVFP4, optionally mixed with INT8 or FP8](https://huggingface.co/Abiray/Minimax-H3-nvfp4-INT4-INT8-Convrot) | `--component-weights-paths.transformer OWNER/REPO/path/FILE.safetensors` | Auto-detected; NVFP4 execution requires NVIDIA compute capability 10.0+. | | DiT | [AutoRound W4A16 component](https://huggingface.co/Ar4ikov/MiniMax-H3-transformer-W4A16-RTN) | `--component-paths.transformer Ar4ikov/MiniMax-H3-transformer-W4A16-RTN` | Self-describing Diffusers component; SGLang reuses the SRT GPTQ/Marlin backend. The linked export is FL2VA. | | DiT | GGUF, full or AdaLN-pruned ([full](https://huggingface.co/leejet/MiniMax-H3-GGUF), [pruned](https://huggingface.co/unsloth/MiniMax-H3-GGUF)) | `--component-weights-paths.transformer OWNER/REPO/FILE.gguf` | CUDA capacity path; aligned TP and layerwise offload are supported, FSDP and LoRA are not. | +| Text encoder | Architecture-compatible Qwen3-VL BF16 finetune ([Heretic example](https://huggingface.co/llmfan46/Qwen3-VL-32B-Instruct-ultra-uncensored-heretic)) | `--component-paths.text_encoder OWNER/REPO` | Reuses the native H3 extractor, including its vision tower and layer-50 hidden-state selection. Finetuning changes conditioning, not the sampling schedule. | | Text encoder | [Serialized FP8 component](https://huggingface.co/Qwen/Qwen3-VL-32B-Instruct-FP8) | `--component-paths.text_encoder Qwen/Qwen3-VL-32B-Instruct-FP8` | Only eligible language-model linears use FP8; embeddings, norms, and the vision tower keep their declared precision. | -| Text encoder | ConvRot INT8, W4A8, or W4A4 safetensors ([INT8](https://huggingface.co/Comfy-Org/MiniMax-H3/tree/main/text_encoders), [W4A8](https://huggingface.co/Winnougan/MiniMax-H3-INT4_Convrot_ComfyUI), [W4A4](https://huggingface.co/Merserk/MiniMax-H3-INT4-ConvRot)) | `--component-weights-paths.text_encoder OWNER/REPO/path/FILE.safetensors` | Auto-detected; requires `comfy-kitchen`. Unmarked vision and embedding tensors keep their declared precision. | +| Text encoder | ConvRot INT8, W4A8, or W4A4 safetensors ([INT8](https://huggingface.co/Comfy-Org/MiniMax-H3/tree/main/text_encoders), [Heretic INT8](https://huggingface.co/ethanfel/Qwen3-VL-32B-Ultra-Heretic-H3-ComfyUI-INT8-ConvRot), [W4A8](https://huggingface.co/Winnougan/MiniMax-H3-INT4_Convrot_ComfyUI), [W4A4](https://huggingface.co/Merserk/MiniMax-H3-INT4-ConvRot)) | `--component-weights-paths.text_encoder OWNER/REPO/path/FILE.safetensors` | Auto-detected; requires `comfy-kitchen`. INT8/W4A8 files may include a scalar-scale INT8 embedding. Unmarked tensors retain their declared precision. | +| Text encoder | Comfy NVFP4 with dynamic activation quantization ([Heretic example](https://huggingface.co/sakamakismile/Qwen3-VL-32B-Heretic-MiniMax-H3-NVFP4)) | `--component-weights-paths.text_encoder OWNER/REPO/path/FILE.safetensors` | Requires `comfy-kitchen` and NVIDIA compute capability 10.0+. Per-layer metadata selects native NVFP4 matmul; scalar/row-wise INT8 embeddings remain packed. | | Text encoder | [NVFP4-AWQ](https://huggingface.co/Comfy-Org/MiniMax-H3/tree/main/text_encoders) or [Quanto qint8](https://huggingface.co/DeepBeepMeep/MiniMax-H3/tree/main/Qwen3-VL-32B-Instruct) safetensors | `--component-weights-paths.text_encoder OWNER/REPO/path/FILE.safetensors` | Memory-oriented formats: compressed storage is restored, then each active matrix uses BF16/FP16 compute. | | Text encoder | [GGUF Qwen3-VL](https://huggingface.co/DeepBeepMeep/MiniMax-H3/tree/main/Qwen3-VL-32B-Instruct) | `--component-weights-paths.text_encoder OWNER/REPO/FILE.gguf` | CUDA capacity path with encoder TP/layerwise support; encoder FSDP is not supported. | | Text encoder | [Compact Qwen3-VL 4B/8B + ClipProj](https://huggingface.co/NicoLab28/ClipProj-MiniMax-H3) | `--component-paths.text_encoder ENCODER_REPO --component-paths.conditioning_projection PROJECTION.safetensors` | Approximate conditioning replacement. A separate weight-only override may quantize the selected small encoder. | @@ -103,6 +132,11 @@ The rows compose rather than enumerate every cross-product. A Turbo-merged INT8 ConvRot checkpoint, for example, must satisfy both the Turbo sampling contract and the ConvRot storage/backend contract. +H3's Qwen3-VL text encoder also handles image understanding; there is no separate +`image_encoder` component. Select the conditioning checkpoint, not an optional +generation-tail file for prompt rewriting. The `uncensored` or `Heretic` label +describes a weight modification, not a separate loader or a guarantee of output quality. + For H3, the registered component names are `transformer`, `text_encoder`, `video_vae`, and `audio_vae`. The shorter `--transformer-weights-path` and `--text-encoder-path` aliases remain supported. `conditioning_projection` is an @@ -251,6 +285,60 @@ recommended lossless preset enables it implicitly. An explicit `--enable-torch-compile true` remains available for controlled experiments, but do not use it to generate consistency ground truth. +### ComfyUI integrated mode + +The [ComfyUI plugin](/docs/sglang-diffusion/comfyui) can run H3 as a per-step +DiT backend. ComfyUI keeps CLIP, VAE, and the sampler loop. SGLang loads only +the transformer under `--comfyui-mode`. + +Copy `python/sglang/multimodal_gen/apps/ComfyUI_SGLDiffusion` into ComfyUI +`custom_nodes/`. Load a shipped graph and set `SGLDOptions.model_type` to +`minimax_h3`: + +| Graph | Task | DiT filename | +| --- | --- | --- | +| `minimax_h3_t2v_sgld.json` | T2V / I2VA / FL2VA | `minimax_h3_fl2va_*.safetensors` | +| `minimax_h3_r2v_sgld.json` | Reference-to-video | `minimax_h3_ref2va_*.safetensors` | +| `minimax_h3_t2v_sgld_upscaler.json` | Low-res then 3D ×2 | `fl2va` DiT | + +Pruned BF16 / FP8 files from [Comfy-Org/MiniMax-H3](https://huggingface.co/Comfy-Org/MiniMax-H3/tree/main/diffusion_models) +are the usual loader inputs. GGUF goes in `SGLDOptions.transformer_weights_path`. +On a 24 GB card put the 32B CLIP on CPU and enable DiT layerwise offload; use +`kitchen_int8` or GGUF rather than the full unpruned BF16 DiT. + +### Singularity hybrid weights + +[Singularity](https://huggingface.co/WarmBloodAban/Minimax-h3_Singularity) is an +FL/Ref fusion fine-tune, not a complete pipeline repository. Keep the official +model ID for the encoders and VAEs, and select one exact transformer file: + +```bash 2×B300 hybrid checkpoint +pip install comfy-kitchen + +WEIGHTS=WarmBloodAban/Minimax-h3_Singularity/Minimax-h3_Singularity_ref2va_Pruned_v1.3_int8.safetensors +sglang serve --model-path MiniMaxAI/MiniMax-H3 \ + --model-variant hybrid \ + --component-weights-paths.transformer "$WEIGHTS" \ + --num-gpus 2 --ulysses-degree 2 --performance-mode speed \ + --port 30000 +``` + +For the full INT8 checkpoint, replace the filename with +`Minimax-h3_Singularity_ref2va_v1.3_int8.safetensors`. SGLang downloads the +selected file automatically. Do not add an online quantization override. +`hybrid` requires an explicit weight override; it does not convert the official +Ref2VA weights into a multimode model. Request `task` and `conditions` retain +their [usual meanings](#4-generate-video-and-audio), including `conditions: []` +for `t2va` and keyframes for `fl2va`. + +Start with `quality: "lossless"` and the standard 50-point schedule. Here +`lossless` disables additional request-time approximations; it does not undo +INT8 quantization, pruning, or fine-tuning. The author's recommended Ref2V +Turbo LoRA is a separate optional adapter, not an automatically inferred +4-step schedule. Reference fidelity can differ from the official model; see +the [author's guidance](https://huggingface.co/WarmBloodAban/Minimax-h3_Singularity/discussions/1) +before applying the [LoRA sampling contract](#5-lora-recipes). + ### Advanced: precomputed AdaLN cache The [model card](https://huggingface.co/MiniMaxAI/MiniMax-H3) notes that about @@ -297,6 +385,34 @@ values. SGLang rejects a request outside that coverage instead of silently changing conditioning. Cache mode supports the matching unquantized checkpoint only. +### Advanced: online AdaLN rebuild with a host cache + +`--minimax-h3-adaln-online true` needs no prebuilt artifact: the server drops +the 24.2 GiB of `adaln_proj` weights from the GPU and computes each request's +AdaLN outputs from the checkpoint on demand, bit-exact with the resident-weight +path. Because a rebuild pass streams the whole 24.2 GiB, the first request of +every new `(task shape, num_inference_steps, flow_shift, audio_flow_shift)` +combination pays several seconds; after that the plans are served from a +64-slot GPU slab (LRU per plan) backed by a pinned host cache +(`--minimax-h3-adaln-host-cache-gb`, LRU per schedule, default 8 GB per +rank), so mixed-schedule serving does not re-read the checkpoint. Plan sets +that exceed the host budget are simply recomputed on their next occurrence. +Expert escape hatches live in environment variables: +`SGLANG_DIFFUSION_MINIMAX_H3_ADALN_GPU_PLANS` resizes the GPU slab (needed +only beyond 65 inference steps) and `SGLANG_DIFFUSION_MINIMAX_H3_ADALN_FP32` +computes the one-time projections in fp32 (experimental; not bit-comparable +to resident weights, validate end-to-end before production). LoRA adapters +that modify `adaln_proj` are rejected in both cache modes rather than +silently ignored. + +Both cache modes hold values derived from `adaln_proj`, so a runtime weight +update is only accepted when the cache can follow it: online mode takes a disk +update whose target directory carries native `adaln_proj` safetensors, and +rejects everything else (tensor updates, and directories without those +tensors) before a single weight is written. A sidecar is built offline and +cannot be regenerated in the server, so weight updates are rejected outright; +rebuild the sidecar against the new weights and restart. + ### Serve MiniMax-H3 on Ascend NPUs For Ascend NPU, follow the @@ -643,7 +759,201 @@ SGLang projects those adapter factors onto the pruned coordinates at load time. A structurally modified checkpoint without that metadata still fails closed, and packed GGUF weights remain incompatible with LoRA. -## 6. Sampling and output controls +### PDD acceleration LoRAs + +[Alibaba PAI's MiniMax-H3-Acc-LoRAs](https://huggingface.co/alibaba-pai/MiniMax-H3-Acc-LoRAs) +use Parallel Decoding Distillation (PDD) for eight denoiser evaluations. These +checkpoints contain backbone LoRA weights and interval-specific output heads. +Merge the backbone weights offline, then fuse the output heads before serving. +The `--lora-path` recipe above does not perform these preparation steps. + +Match the adapter and original MiniMax-H3 transformer weights to the server's +model variant: + +| Model variant | Adapter file | Tasks | +| --- | --- | --- | +| `fl2va` | `MiniMax-H3-FL2VA-Acc-8Step.safetensors` | `t2va`, `fl2va` | +| `ref2va` | `MiniMax-H3-Ref2VA-Acc-8Step.safetensors` | `ref2va` | + +`BASE_TRANSFORMER_DIR` is the directory containing the original model's +BF16/FP32 transformer weight files, not the model root or the LoRA directory. +If you downloaded `MiniMaxAI/MiniMax-H3` to `/path/to/MiniMax-H3`, use +`/path/to/MiniMax-H3/FL2VA/transformer` for T2VA/FL2VA or +`/path/to/MiniMax-H3/Ref2VA/transformer` for Ref2VA. This directory contains +`model-*.safetensors`, `model.safetensors.index.json`, and `config.json`. +The builder reads these files and writes the merged weights to +`$PDD_DIR/transformer`, leaving the original weights unchanged. + +The following example prepares FL2VA +weights and serves them on eight NVIDIA CUDA GPUs with TP2 + Ulysses4: + +```bash Command +BASE_TRANSFORMER_DIR=/path/to/MiniMax-H3/FL2VA/transformer +LORA_DIR=/path/to/h3-acc-loras +PDD_DIR=/path/to/h3-fl2va-pdd + +hf download alibaba-pai/MiniMax-H3-Acc-LoRAs \ + MiniMax-H3-FL2VA-Acc-8Step.safetensors --local-dir "$LORA_DIR" + +python3 -m sglang.multimodal_gen.tools.build_minimax_h3_pdd_weights \ + "$BASE_TRANSFORMER_DIR" \ + "$LORA_DIR/MiniMax-H3-FL2VA-Acc-8Step.safetensors" \ + "$PDD_DIR" + +python3 -m sglang.multimodal_gen.tools.fuse_minimax_h3_pdd_heads \ + "$PDD_DIR" --video-shift 12.0 --audio-shift 3.0 + +SGLANG_DIFFUSION_MINIMAX_H3_PDD_HEADS="$PDD_DIR/pdd_fused_heads.safetensors" \ +sglang serve \ + --model-path MiniMaxAI/MiniMax-H3 \ + --model-variant fl2va \ + --transformer-weights-path "$PDD_DIR/transformer" \ + --num-gpus 8 \ + --tp-size 2 \ + --ulysses-degree 4 \ + --performance-mode speed \ + --port 30010 +``` + +Keep `pdd_config.json` beside `pdd_fused_heads.safetensors`, outside the +`transformer/` directory. For Ref2VA, use its adapter and base transformer, +a separate output directory, and `--model-variant ref2va`. + +Use the requests in [Generate video and audio](#4-generate-video-and-audio) +with `num_inference_steps: 9`, `flow_shift: 12.0`, and `audio_flow_shift: 3.0`. +The nine sigma grid points give eight denoiser evaluations. These settings +must match the fused heads; the server rejects a different step count or +shift. Use `quality: "lossless"` in requests. + +## 6. FastH3: 4-step distilled preview + +[FastVideo/FastVideo-FastH3-4-step-Preview-v1-VSA-DataFree](https://huggingface.co/FastVideo/FastVideo-FastH3-4-step-Preview-v1-VSA-DataFree) +is a 4-step DMD2 distillation of MiniMax-H3, trained data-free with Video +Sparse Attention (VSA) at 0.9 sparsity and 64-token tiles. Only the T2VA +capability was distilled: requests must use `task: "t2va"`, and `fl2va` / +`ref2va` requests are rejected. The checkpoint inherits the MiniMax-H3 +Community License. + +Pass the repository directly to `--model-path`. The flat native-Diffusers +upload is materialized into the base-H3 layout through a registered model +overlay; the only non-symlink step is a one-time re-serialization of the +roughly 10 GB video VAE on first launch. + +```bash 4×B300 VSA-H3 +sglang serve \ + --model-path FastVideo/FastVideo-FastH3-4-step-Preview-v1-VSA-DataFree \ + --num-gpus 4 \ + --attention-backend video_sparse_attn_h3 \ + --attention-backend-config '{"VSA_sparsity": 0.9}' \ + --port 30010 +``` + +Requests use the same asynchronous video endpoint as the base model, with +`task: "t2va"`, `conditions: []`, and a target such as +`{"short_edge": 768, "aspect_ratio": "16:9", "duration_seconds": 5.0}`. The +request default is `num_inference_steps: 5`: five points on the standard +shift-12/shift-3 sigma grid, i.e. the four distilled DiT evaluations. Any +other step count is rejected. + +`video_sparse_attn_h3` (VSA-H3) is the trained sparse policy: an in-tree +Triton block-sparse kernel (SM90 / SM100 / SM103) over segment-pure prefix +tiles and (4, 4, 4) video tiles, driven by the checkpoint's trained +`to_gate_compress` compression branch. Only the DiT runs sparse; the token +refiner, text encoder, and VAEs keep their dense defaults. Ulysses sequence +parallelism is supported. See +[Attention Backends](/docs/sglang-diffusion/attention_backends) for +`VSA_sparsity`, `vsa_mode`, `vsa_dense_first_n_steps`, and +`vsa_dense_layers`. Every dense backend that runs on base H3 (`fa`, +`torch_sdpa`, ...) also runs on the FastH3 weights without VSA flags, and +`sglang generate` takes the same flags as `sglang serve`. + +Measured latencies for the 4× B300 recipe are in +[FastH3 on B300](#fasth3-on-b300). + +FastH3 rejects deployment options that do not apply to the distilled preview +instead of silently ignoring them: `--model-variant`, `quality: "high"`, +`fl2va` / `ref2va` requests, and, with VSA-H3, `--ring-degree` greater than 1, +`torch.compile`, and breakable CUDA graph execution. + + +The sibling `FastVideo/FastVideo-FastH3-4-step-Preview-v1-LoRA` adapters carry +full-rank `.diff` / `.diff_b` deltas and `set_weight` gate tensors beyond the +LoRA contract; `--lora-path` rejects them with an explicit error. Serve the +merged VSA-DataFree checkpoint above instead. + + + +Upstream labels this checkpoint a preview. Quality gaps versus base H3 on hard +motion and fine detail are properties of the released distillation, not of the +SGLang port. Use base MiniMax-H3 when output quality matters more than +latency. + + +## 7. VDN-H3: hybrid attention, 8-step distill + +[Video DeltaNet](https://openvdn.github.io/) (VDN) is a hybrid attention +architecture that approaches the speed of linear attention while maintaining +near-lossless quality. It splits the video-video attention into two +complementary branches: a sliding-window softmax branch for local frame pairs +(five consecutive latent frames form a chunk, and each chunk attends to +itself, the previous chunk, and the following chunk; the first and last +frames, text and audio stay dense) and a bidirectional linear branch, Video +Delta Attention, for long-range context. +[OpenVDN/vdn-minimax-h3](https://huggingface.co/OpenVDN/vdn-minimax-h3) +applies it to MiniMax-H3: the released `stage-dmd-step-250` checkpoint adds a +4.3 GB linear branch and OpenVDN's own 8-step DMD2 LoRA on the unchanged H3 +backbone. It is the FL2VA partition, so it serves `t2va` and `fl2va` (first, +last, or both keyframes); `ref2va` is rejected. The weights inherit the +MiniMax-H3 Community License. + +Pass the repository directly to `--model-path`. On first launch a registered +overlay prefuses both adapters into the transformer weights (a 62 GB write) +and attaches the linear branch; the Qwen3-VL conditioner and the VAEs are +hard-linked from `MiniMaxAI/MiniMax-H3`. Keep `SGLANG_DIFFUSION_CACHE_ROOT` on +the same filesystem as the Hugging Face cache with at least 90 GB free. + +```bash 8×B200 +sglang serve \ + --model-path OpenVDN/vdn-minimax-h3 \ + --num-gpus 8 \ + --attention-backend hybrid_window_attn_h3 \ + --performance-mode speed \ + --warmup-num-frames 345 \ + --warmup-resolutions 1344x768 \ + --port 30010 +``` + +`--num-gpus 4`, `2` or `1` gives the smaller Ulysses recipes in the benchmark +table; `sglang generate` takes the same flags. `--warmup-num-frames` / +`--warmup-resolutions` warm up at the clip length and canvas you will serve; +without them the first long request pays 2 to 3 s of allocator growth and +kernel setup. + +Requests use the same video endpoint as the base model with `task: "t2va"` +and `conditions: []`, or `task: "fl2va"` with the +[keyframe conditions](#4-generate-video-and-audio), and a target such as +`{"short_edge": 768, "aspect_ratio": "16:9", "duration_seconds": 14.375}` (the +345-frame paper workload). The default `num_inference_steps: 9` is the only +accepted value: nine sigma grid points, eight DiT forwards. Each keyframe adds +about 2,000 dense rows, so an `fl2va` forward costs a little more than `t2va`. + +`hybrid_window_attn_h3` is required: a dense backend would silently skip the +linear branch and the gates and produce the wrong model. On Blackwell +(SM100 / SM103 / SM120) the transformer defaults to online MXFP8; +`--quantization bf16` opts out. Before SM100, `--quantization fp8` is base +H3's per-channel fp8 path, and SM80 runs the bf16 DiT (62 GB; budget for +`--layerwise-offload-components`). Ampere and Ada are enabled but not +benchmarked. Ulysses sequence parallelism is supported; `--model-variant`, +`quality: "high"`, `--ring-degree` greater than 1, `torch.compile`, and +breakable CUDA graph execution are rejected. The backend's +`--attention-backend-config` keys are listed in +[Attention Backends](/docs/sglang-diffusion/attention_backends). + +Measured latencies for 1, 2, 4 and 8 B200s are in +[VDN-H3 on B200](#vdn-h3-on-b200); single-card and PCIe multi-card numbers +for the RTX PRO 6000 are in [VDN-H3 on RTX PRO 6000](#vdn-h3-on-rtx-pro-6000). + +## 8. Sampling and output controls MiniMax-H3 supports more than one output per prompt. The video API accepts `num_outputs_per_prompt` (or OpenAI-compatible `n`) from 1 through 10. Offline @@ -673,17 +983,22 @@ done ### Choose the quality level -`quality` is a request-scoped sampling parameter with two validated levels: +`quality` is a cumulative request-scoped optimization parameter with three +levels: - `"lossless"` (default): the exact reference path. Output is bit-exact against the reference implementation and the CI ground truth. +- `"extra-high"`: includes the global fusion-only tier but does not enable + Cache-DiT or another approximate optimization. MiniMax-H3 currently has no + request-gated fusion site, so its denoise path is the same as `lossless`. - `"high"`: the audited accelerated path. Quality is guaranteed (the audited Cache-DiT configuration measures SSIM 0.931 / PSNR 28.16 dB against `lossless`), but output is no longer bit-identical to the reference. -One resident server serves both levels; a `quality: "high"` request mounts -its audited Cache-DiT policy at the batch boundary, and a later -`quality: "lossless"` request removes the hooks before denoising. +One resident server serves all three levels; a `quality: "high"` request +mounts its audited Cache-DiT policy at the batch boundary, and a later +`quality: "lossless"` or `quality: "extra-high"` request removes the hooks +before denoising. Start the validated server once: @@ -720,6 +1035,19 @@ omitting the field is equivalent. + + +The global fusion-only tier. It does not enable MiniMax-H3 Cache-DiT and +currently follows the same H3 denoise path as `lossless`. + +```json Request field +{ + "quality": "extra-high" +} +``` + + + The audited accelerated path. Use it when you can trade bit-exactness for @@ -740,6 +1068,7 @@ The measured trade-off is: | `quality` | Mean
inference
latency | Speedup | SSIM vs
lossless | PSNR vs
lossless | Expected
trade-off | | --- | ---: | ---: | ---: | ---: | --- | | `lossless` | 75.10 s | 1.00× | 1.000 | exact | Native reference path | +| `extra-high` | Not separately measured | — | Same H3 denoise path | Same H3 denoise path | Fusion-only tier; no H3-specific request-gated site yet | | `high` | 53.70 s | 1.40× | 0.931 | 28.16 dB | Smallest same-seed visual change | These numbers use 1344×768, 124-frame, 24 fps T2VA with 50 inference steps, @@ -765,9 +1094,11 @@ name, for example `sglang generate --quality high`. For manually tuned Cache-DiT experiments outside that validated path, omit -the request `quality` field and set the process-wide environment controls -directly. An explicit `quality: "lossless"` request overrides those controls -and restores native denoising: +the request `quality` field and set `--enable-cache-dit` or the +process-wide `SGLANG_CACHE_DIT_*` defaults. Any explicit `quality`, including +`"lossless"` and `"extra-high"`, takes H3 off the generic Cache-DiT path. The +24 GB layerwise recipe above can use the same switch; skipped blocks are +not streamed. ```bash Command SGLANG_CACHE_DIT_ENABLED=true \ @@ -787,15 +1118,17 @@ sglang serve \ ``` -Cache-DiT skips selected block computation and is approximate. It cannot be -combined with FSDP inference or DiT layerwise offload. Breakable CUDA graph -execution takes precedence and leaves Cache-DiT disabled. Tune the cache -thresholds only after comparing both video and audio quality on the target -task profile. A real B200 request has completed, but the `quality: "high"` -path above remains fail-closed to the audited 4×H200 workload. +Cache-DiT skips selected block computation and is approximate. It cannot +be combined with FSDP inference. DiT layerwise offload is compatible: +skipped blocks are not streamed, and the first layer after a skip may +sync-load. Breakable CUDA graph execution takes precedence and leaves +Cache-DiT disabled. Tune the cache thresholds only after comparing both +video and audio quality on the target task profile. A real B200 request +has completed, but the `quality: "high"` path above remains fail-closed +to the audited 4×H200 workload. -## 7. Feature contracts and advanced recipes +## 9. Feature contracts and advanced recipes The generated command already contains the recommended topology and encoder setting. Use the detailed reference below only when applying an optional @@ -816,7 +1149,7 @@ listed hardware and topology; it is not inherited by a similar GPU family. | Tensor parallelism | Verified: B200 TP2 + Ulysses4; H100 TP2 + Ulysses2 and TP4 + Ulysses1 | `--tp-size` may be combined with Ulysses when the TP-local head count remains divisible by the Ulysses degree. On 4×H100, TP2 + Ulysses2 is the measured speed default. | | FSDP inference | Verified: 4× B200 and 4× H100 + Ulysses4 | Preserves H3's mixed BF16/FP32 parameter policy. B200 completed the exact eager comparison; H100 completed consecutive real requests at about 57 GB peak memory per GPU. | | Resident components | Verified: B200, H200, 4×H100 with TP, and 1/2/4/8× MI300X and MI355X | This is the recommended single-request latency path when the complete workload fits. | -| CPU and layerwise offload | Verified: 2× RTX 5090 TP2; 1× RTX 4090 24 GB | The 5090 lossless recipe keeps 20 DiT blocks plus both VAE encoders resident, streams the remaining DiT blocks, text encoder, and video VAE decoder blocks, and leaves the small audio VAE resident. The 4090 recipe streams DiT and the text encoder with zero resident DiT layers and **omits `vae`** from `--layerwise-offload-components`. | +| CPU and layerwise offload | Verified: 2× RTX 5090 TP2; 1× RTX 4090 24 GB | The 5090 lossless recipe keeps 20 DiT blocks plus both VAE encoders resident, streams the remaining DiT blocks, text encoder, and video VAE decoder blocks, and leaves the small audio VAE resident. The 4090 recipe streams DiT and the text encoder with zero resident DiT layers and **omits `vae`** from `--layerwise-offload-components`. Compatible with Cache-DiT; skipped blocks are not streamed. | | Breakable CUDA graph | Verified: B200 Ref2VA, opt-in | Matching eager output was observed for the captured signature, without a measured speedup. Re-capture for other shapes and reference sets. | | `torch.compile` | Measured: H200, opt-in | Steady-state benefit was below measurement noise, while startup increased and numerical output changed. Do not use it for consistency ground truth. | @@ -954,6 +1287,112 @@ fails closed when the selected format, projection, or topology is incompatible. + + +Cube sparse attention applies TopK sparsity only to H3's 3D visual streams. +Text, audio, standalone reference images, and the text-only token refiner stay +dense. It runs on pure PyTorch plus FlexAttention, so it has no third-party +kernel dependency. + +Select it for the H3 transformer with +`--component-attention-backends transformer=cube_sparse_attn` and pass +`--attention-backend-config` with both `local_cube_size` and +`topk_ratio_list`. Scoping the backend leaves the text encoder on its native +backend: + +```bash Command +sglang serve \ + --model-path MiniMaxAI/MiniMax-H3 \ + --model-variant ref2va \ + --num-gpus 8 \ + --ulysses-degree 8 \ + --performance-mode speed \ + --component-attention-backends transformer=cube_sparse_attn \ + --attention-backend-config '{"local_cube_size": [4, 4, 4], "topk_ratio_list": [1.0, 1.0, 0.8, 0.7, 0.6, 0.5, 0.5]}' \ + --port 30010 +``` + +- `local_cube_size` is the `(T, H, W)` cube that groups neighboring latent + tokens into one attention block. It must have exactly three entries. +- `topk_ratio_list` sets the per-step keep ratio and must have exactly one + entry per denoise step, each in `(0, 1]`. A ratio of `1.0` keeps a step + on H3's native dense attention path; smaller values select the FlexAttention + sparse path and drop more blocks. The example above matches a request with + `num_inference_steps: 8`, whose endpoint-inclusive sigma schedule has seven + denoise updates. + +Cube labeling is coordinate-driven. FL2VA keyframes share the target video's +position grid, so a keyframe token and a target token at the same `(T, H, W)` +coordinate receive the same semantic cube label. Duplicate coordinates do not +extend the temporal grid; a semantic cube can therefore span multiple physical +attention blocks. In Ref2VA, standalone reference images remain dense, while +reference videos and the target video contribute to one global TopK candidate +pool rather than receiving separate per-stream quotas. + + +Cube sparse attention is an approximate backend and is not a consistency +ground-truth mode. `topk_ratio_list` length must equal the denoise step count +or the server rejects the request. Cube sparse attention does not support Ring +parallelism; use `--ulysses-degree` without `--ring-degree`. FlexAttention's +routing overhead can outweigh sparse-kernel savings on short sequences, so +benchmark latency as well as visual and audio quality on the target workload. + + + + + + +SubBlock sparse attention applies training-free block sparsity to MiniMax-H3's +long, non-causal DiT self-attention. The BF16 path supports CUDA SM90, SM100, +and SM120. Other compute capabilities, including SM103/B300, are rejected. +Ulysses sequence parallelism is supported; Ring parallelism is not. + +The global backend selects the H3 DiT lazily, so keep the Qwen text encoder on +its compatible dense backend with a component override: + +```bash Command +sglang serve \ + --model-path MiniMaxAI/MiniMax-H3 \ + --model-variant fl2va \ + --num-gpus 8 \ + --ulysses-degree 8 \ + --attention-backend subblock_sparse_attn \ + --component-attention-backends text_encoder=fa \ + --attention-backend-config '{"sparsity": 0.75, "skip_first_steps": 10}' \ + --port 30010 +``` + +Use `text_encoder=torch_sdpa` on SM120. The default configuration drops at +most 75% of key blocks after the first 10 denoise forwards; short sequences, +cross-attention, and unsupported shapes continue on the dense path. Tune +`sparsity` and `skip_first_steps` together with visual and audio quality checks. + +On H100/H200 (SM90), `compute_mode=sage_fp8` switches the sparse kernel to +online INT8 Q/K and FP8 P/V compute. Install the optional kernel and add the +compute mode: + +```bash Command +pip install git+https://github.com/thu-ml/SpargeAttn.git --no-build-isolation + +sglang serve \ + --model-path MiniMaxAI/MiniMax-H3 \ + --model-variant fl2va \ + --num-gpus 8 \ + --ulysses-degree 8 \ + --attention-backend subblock_sparse_attn \ + --component-attention-backends text_encoder=fa \ + --attention-backend-config '{"compute_mode": "sage_fp8", "sparsity": 0.75}' \ + --port 30010 +``` + + +Both SubBlock routing and `sage_fp8` are approximate. Use dense BF16 attention +for consistency ground truth, and validate the selected sparsity on the target +resolution, duration, task, and checkpoint. + + + + The picker explicitly writes `--encoder-parallel auto` in every single-node @@ -983,7 +1422,7 @@ fold decision is not node-boundary aware: -## 8. Configuration notes +## 10. Configuration notes - MiniMax-H3 produces the canonical 24 fps output; request duration is expressed through `target.duration_seconds`. - `target.duration_seconds` must be between 4 and 15 seconds, inclusive. The command picker defaults to the verified 5-second profile. @@ -992,7 +1431,7 @@ fold decision is not node-boundary aware: - V2V uses `task: "ref2va"` with a `video` or `video_audio` reference; it is served by the `Ref2VA` partition and is not a separate public task value. - `conditions[].start_time_seconds` selects a non-negative offset for a video reference. Its visual and audio streams are always sought together. - Ref2VA condition order is semantic and must match the one-based material tags in the prompt. For Ref2VA, `target.aspect_ratio: "auto"` resolves to the model's 16:9 fallback rather than inheriting a reference asset's geometry. -- The distilled pipeline uses a single denoising branch, so CFG parallelism does not apply. Do not enable it: `--enable-cfg-parallel true` or `--cfg-parallel-size` greater than 1 is rejected instead of duplicating the positive branch. Explicitly disabling CFG, or setting its size to 1, remains a valid no-op. +- The distilled pipeline uses a single denoising branch, so CFG parallelism does not apply. Keep it disabled: a CFG degree greater than 1 is accepted for request compatibility, but the extra ranks only recompute the same branch and do not accelerate the request. Explicitly disabling CFG, or setting its size to 1, remains a valid no-op. - The released visual VAE quality recipe uses overlapping tiled decode. SGLang keeps that recipe by default and distributes complete tiles across the decode group; this changes scheduling, not the computation inside each tile. - H3 rejects `--vae-config.parallel-decode-mode spatial` and `spatial_shard`: validation found output mismatches. Use the default released tiled recipe. - Keep the default `--encoder-parallel auto`. With the server’s default `batching_max_size` of 1, single-node H100/H200/B200/B300 recipes with peer-to-peer access fold the Qwen text encoder over otherwise idle Ulysses ranks. This is separate from DiT tensor parallelism. A pure-TP recipe already shards the encoder over its TP group and does not add a world fold. @@ -1002,7 +1441,7 @@ fold decision is not node-boundary aware: - `speed` keeps model components resident, while `auto` applies the model-aware 120 GiB residency threshold. `memory` prioritizes avoiding OOM and includes the executable VAE decoder in its default layerwise set. A measured recipe with sufficient headroom can opt into `--component-residency vae=resident`; the 2×H100 CI recipe does this because the VAE's 4.8 GiB/GPU cost avoids repeated decoder transfers during tiled decode. DiT residency and prefetch knobs remain scoped to the DiT. Use `speed` only after confirming that the complete target workload fits. - Breakable CUDA graph execution is an explicit opt-in, not part of the recommended `speed` preset. It requires `--enable-breakable-cuda-graph`, every served size in `--warmup-resolutions`, and `--bcg-text-buckets` that cover the live H3 condition sequence. The validated 1344×768 Ref2VA recipe uses 5504; other task profiles and reference sets may need a different value. It preserves eager output for matching captured signatures, but graph capture consumes additional GPU memory and may provide little latency benefit when Ulysses attention and collectives dominate, so benchmark it on the target topology before enabling it. -## 9. Benchmarks +## 11. Benchmarks The picker exposes resident and FSDP profiles on NVIDIA datacenter GPUs. GPU counts are properties of the selected recipes, not a claim that every platform @@ -1013,13 +1452,42 @@ the configurations with collected measurements: | --- | --- | --- | | B300 | 8× Ulysses8 resident | 8× FSDP + Ulysses8; the 8-GPU sweep is not a minimum-GPU claim. | | B200 | 8× Ulysses8 resident | 4× FSDP + Ulysses4 | +| GB300 | 4× Ulysses4 resident (FL2VA T2VA) | 2 nodes × 4× Ulysses4×Ring2 is unverified. | +| GB200 | 4× Ulysses4 resident, derived/unverified | No performance measurements. | | H200 | 4× Ulysses4 resident | 4× FSDP + Ulysses4; 4× TP2 + Ulysses2; 2 nodes × 8× Ulysses8×Ring2 cross-node | | H100 | 4× TP2 + Ulysses2 resident | 4× TP4 + Ulysses1; 4× FSDP + Ulysses4 | | Ascend NPU | 8 NPUs, TP2 + SP4, Laser Attention | 4 NPUs, TP2 + SP2, Laser Attention | | MI300X / MI355X | 8× Ulysses8 resident | 1×, 2×, and 4× scaling runs | -| RTX 5090 | 2× TP2 + layerwise offload | — | +| RTX 5090 | 1× layerwise offload, measured on a physical desktop | 2× TP2 + layerwise offload | | RTX 4090 24 GB | 1× layerwise offload + `kitchen_int8` | Approximate attention backends are opt-in | +### GB300 single-host + +Four requests on **1 host × 4 NVIDIA GB300**, using Linux ARM64, SGLang main +[`15d2cbcc90fc`](https://github.com/sgl-project/sglang/commit/15d2cbcc90fc66b4d08745994cc707a45b067ffc), +PyTorch 2.13.0+cu130, and NCCL 2.29.7. The source checkout was installed with +`python3 -m pip install -e "python[diffusion]" --upgrade` inside +`lmsysorg/sglang:latest`; this is a source-validation environment, not a claim +that the image's bundled release contains that revision. + +The command matches the GB300 default in the builder: Ulysses4, resident +weights, automatic encoder folding, native BF16/FP32, platform-default +Dynamic cuDNN/FA attention, and eager execution. The model was downloaded by +`sglang serve` from `MiniMaxAI/MiniMax-H3`. + +The workload uses the builder's cat-band prompt, FL2VA T2VA, a 5-second request +at a 768-pixel short edge, 50 inference steps, `quality="lossless"`, one output, +and seed 1101. Every MP4 contained video and audio and passed a full decode check. + +| First full request after server warmup | Following three requests | Warm median | +| ---: | --- | ---: | +| 35.16 s | 33.11 / 33.10 / 33.10 s | **33.10 s** | + +These are client-observed completion times, including up to one second of +status-polling delay. Download, model loading, and server warmup are excluded. +This is a deployment smoke/latency measurement, not a cross-framework quality +audit, a minimum-GPU claim, or an extrapolation to GB200/NVL72 scaling. + ### Ascend NPU topology comparison Both topologies used Laser Attention and the explicit Cache-DiT configuration @@ -1113,6 +1581,119 @@ python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ | Ref2VA | FP8 | fold | 112.0 s | 34.44 s | **27.12 s** | 52,816 MB | | Ref2VA | FP8 | replicate | 116.0 s | 33.42 s | **27.12 s** | 93,396 MB | +### FastH3 on B300 + +The same 4× B300 host served [FastH3](#6-fasth3-4-step-distilled-preview) +with the VSA-H3 recipe above (1344×768 at 24 fps with audio, `task: "t2va"`, +`num_inference_steps: 5`, seed 1000, eager BF16, Ulysses4, `VSA_sparsity` 0.9). +E2E is the client wall clock of a `/v1/videos` request including decode, +muxing, and file output, median of three requests after one warm request; +the stage columns are the server timings of the same request. H3 aligns the +requested durations to 124, 243, and 362 frames. Client RTF is E2E divided by +the video duration: + +| Requested / aligned | Encoder | Denoise (4 forwards) | Decode | Transport + MP4 | E2E | Client RTF | Peak/GPU | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| 5 s / 124 | 0.07 s | 2.18 s | 0.87 s | 0.9 s | **4.1 s** | 0.79 | 95,744 MB | +| 10 s / 243 | 0.07 s | 4.83 s | 1.71 s | 1.3 s | **8.0 s** | 0.79 | 102,666 MB | +| 15 s / 362 | 0.07 s | 8.80 s | 2.56 s | 1.8 s | **13.3 s** | 0.88 | 110,774 MB | + +All three requests finish faster than playback. Dense FA on the same weights +and topology takes 3.77 / 9.84 / 18.45 s (`sglang generate`, stage sum): it is +competitive at 5 s, and VSA-H3 pulls ahead from 10 s on. At 5 s, +TP2 + Ulysses2 (3.42 s, 62,290 MB), FSDP + Ulysses4 (3.42 s, 50,984 MB), and +online `--quantization fp8` (2.93 s, 64,204 MB) trade a little latency for +peak memory. + +### VDN-H3 on B200 + +An 8× B200 (SM100, 183 GB) host served [VDN-H3](#7-vdn-h3-hybrid-attention-8-step-distill) +on the paper workload: 1344×768 at 24 fps with audio for 14.375 s (345 frames, +102 latent frames, about 104k packed rows), `task: "t2va"`, +`num_inference_steps: 9` (8 DiT forwards), seed 1000, `hybrid_window_attn_h3`, +eager, and the warmup run at the served clip shape +(`--warmup-num-frames 345 --warmup-resolutions 1344x768`), so every forward +of the served request is steady state; without those flags the first forward +pays 2 to 3 s of allocator growth and kernel setup. "Steady +s/NFE" is the mean of forwards 2 to 8. The OpenVDN reference rows ran the released +inference stack (`8nfe_tuned_fp8.yaml`, `infer_ulysses.py`) on the same host +and the same clip length, with its Ulysses layout (`parallel.softmax_ranks`) +swept at every GPU count rather than assumed. Every row below comes from one +campaign, one arm at a time on an otherwise idle node; a repeated arm moves by +about 3% across nodes, so treat smaller differences as noise: + +| Config | Steady s/NFE | Denoise (8 forwards) | Decode | Peak/GPU | +| --- | ---: | ---: | ---: | ---: | +| OpenVDN published, FP8, 8× B200 Ulysses 5+3 ([openvdn.github.io](https://openvdn.github.io/)) | 1.40 | 11.2 s | — | — | +| OpenVDN reference, BF16 (`8nfe_tuned.yaml`), 1× B200 | 7.66 | 61.2 s | — | — | +| OpenVDN reference, FP8, 1× B200 | 6.25 | 50.0 s | — | — | +| OpenVDN reference, FP8, 2× B200 Ulysses (standard / 1+1 branch-parallel) | 4.87 / 5.04 | 39.0 / 40.3 s | — | — | +| OpenVDN reference, FP8, 4× B200 Ulysses (3+1 branch-parallel) | 2.59 | 20.7 s | — | — | +| OpenVDN reference, FP8, 8× B200 Ulysses (5+3 / 4+4 / standard) | 1.40 / 1.47 / 1.52 | 11.2 / 11.7 / 12.2 s | — | — | +| SGLang, `--quantization bf16`, 1× B200 | 7.45 | 58.8 s | 8.8 s | 156,298 MB | +| SGLang, `--quantization fp8` (→ online `mxfp8` on SM100+), 1× B200 | 6.03 | 47.6 s | 8.7 s | 128,418 MB | +| SGLang default (online `mxfp8`), 2× B200 Ulysses2, `--performance-mode speed`, served-shape warmup | **3.30** | **25.9 s** | 4.7 s | 106,738 MB | +| SGLang, `--quantization bf16`, 4× B200 Ulysses4, `--performance-mode speed`, served-shape warmup | 2.06 | 16.2 s | 2.3 s | 108,434 MB | +| SGLang default (online `mxfp8`), 4× B200 Ulysses4, `--performance-mode speed`, served-shape warmup | **1.67** | **13.1 s** | 2.4 s | 85,244 MB | +| SGLang default (online `mxfp8`), 8× B200 Ulysses8, `--performance-mode speed`, served-shape warmup | **0.88** | **6.9 s** | 1.5 s | 79,972 MB | +| SGLang, per-channel fp8 weight scales (the online `fp8` path of other models; on SM100+ `--quantization fp8` resolves to `mxfp8`, so this arm selects the per-channel path explicitly), 8× B200 Ulysses8 | 0.98 | 7.7 s | 1.5 s | 79,504 MB | + +On 8× B200 the SGLang Ulysses8 path runs the paper workload at 0.88 s/NFE +(7.0 GPU-seconds per NFE) against the published 1.40 s/NFE of OpenVDN's 5+3 +branch-parallel layout; the whole request (text encoding, 8 forwards, joint +decode) completes in about 8.7 s after warmup. At this GPU count the step is +launch- and copy-bound in the linear branch rather than FLOP-bound, and the +transformer runs online `mxfp8` by default (e4m3 with one E8M0 scale per 32 +elements into cuBLASLt's block-scaled GEMM, with the activation quant fused +into the adaLN and SwiGLU producers). + +Against the published 8× B200 headline (1.40 s/NFE, 11.2 +GPU-seconds per NFE), the SGLang 4× B200 run spends 6.7 GPU-seconds per +NFE, with half the GPUs and half the all-to-all fan-out. At equal GPU count +SGLang is 1.48× faster at two cards (3.30 vs 4.87 s/NFE), 1.55× at four +(1.67 vs 2.59) and 1.60× at eight (0.88 vs 1.40), while on one card the two +stacks are within 4% of each other (6.03 vs 6.25 s/NFE quantized, 7.45 vs 7.66 +in BF16). The advantage is therefore parallel scaling rather than per-GPU +kernel work: SGLang holds 86-91% parallel efficiency from 2 to 8 cards against +the reference stack's 56-64%, and sweeping the reference stack's own layout +does not close it (its best split is 8% better than standard Ulysses at eight +cards, 1.40 against 1.52 s/NFE). + +On one B200 both single-GPU rows run with the DiT resident under the auto +memory policy (128 GB peak quantized, 156 GB in BF16, of the card's 183 GB), +so no `--layerwise-offload-components` flag is needed at this clip length and +the single-GPU rows are compute-bound. + +### VDN-H3 on RTX PRO 6000 + +The same paper workload (1344×768, 345 frames, 8 DiT forwards, seed 1000, +`hybrid_window_attn_h3`, `--performance-mode speed`, served-shape warmup) on +RTX PRO 6000 Blackwell Server Edition cards (SM120, 96 GB, PCIe, no NVLink). +The window runs on FA4's sm120 kernel and the transformer defaults to online +`mxfp8` (a cutlass sm120 block-scaled GEMM); `--quantization fp8` maps to it. +One card needs `--layerwise-offload-components text_encoder`: the fp8 DiT +(33 GB) and the bf16 text encoder (48 GB) do not both fit, and the whole-module +`--text-encoder-cpu-offload` re-homes the encoder in one piece and runs out of +memory when it is used. Multi-card Ulysses goes over PCIe; the all-to-all is +the bulk of the step there (54% of GPU time at 8 cards), so the per-card +efficiency drops with count. A 2-card run should use a pair on one PCIe +switch (`nvidia-smi topo -m` shows `PIX`): a pair across the CPU root complex +measured 15.7 to 17.0 s/NFE on a shared host. + +| Config | Steady s/NFE | Denoise (8 forwards) | Decode | Peak/GPU | +| --- | ---: | ---: | ---: | ---: | +| 1× RTX PRO 6000, `--layerwise-offload-components text_encoder` (default `mxfp8`) | **20.27** | **160.0 s** | 20.6 s | 95,790 MB | +| 1× RTX PRO 6000, same, per-channel `fp8` path (before this mapping) | 22.85 | 180.3 s | 20.7 s | 95,160 MB | +| 2× RTX PRO 6000 Ulysses2, one PCIe switch (default `mxfp8`) | 13.61 | 107.4 s | 10.6 s | 93,846 MB | +| 2× RTX PRO 6000 Ulysses2, one PCIe switch, per-channel `fp8` | 14.97 | 118.1 s | 10.6 s | 94,700 MB | +| 4× RTX PRO 6000 Ulysses4 (default `mxfp8`) | 7.98 | 63.0 s | 5.6 s | 82,178 MB | +| 4× RTX PRO 6000 Ulysses4, per-channel `fp8` | 8.23 | 65.1 s | 5.6 s | 82,422 MB | +| 8× RTX PRO 6000 Ulysses8 (default `mxfp8`) | 5.43 | 43.1 s | 3.5 s | 77,704 MB | +| 8× RTX PRO 6000 Ulysses8, per-channel `fp8` | 5.47 | 43.4 s | 3.5 s | 77,236 MB | + +For comparison the same code on B200 measures 6.03 / 3.30 / 1.67 / 0.88 s/NFE +at 1 / 2 / 4 / 8 cards (NVLink, 86% parallel efficiency at 8). + ### H200 topology comparison The same four-card H200 host completed both lossless resident placements with @@ -1214,9 +1795,11 @@ Ulysses2 was the fastest; TP4 used the least memory: | FSDP + Ulysses4 | 13.36 s | 57.01 GB | | TP4 + Ulysses1 | 13.86 s | 49.80 GB | -### RTX 5090 capacity run +### RTX 5090 capacity run (two cards) -The verified two-card RTX 5090 host used TP2 with layerwise offload. The full +The single-card recipe is a different one — see *RTX 5090 single card, physical +desktop* below; the twenty resident layers here are half a layer per GPU under +TP2 and do not transfer to one card. The verified two-card RTX 5090 host used TP2 with layerwise offload. The full 50-step, 1344×768, 5-second request completed in 559.67 seconds: 525.05 seconds of denoising and 33.61 seconds of decoding, with a 26.3 GiB sampled peak per GPU. @@ -1249,6 +1832,15 @@ cards and a Host RAM selector: pick your budget and it emits this command with your tier's measured expectations attached as comments. The table below is the same data in one view. +**Unified-memory machines are the exception**: on a DGX Spark (GB10, 128 GB +shared between CPU and GPU) launch with **no flags at all**. The deployment +still exceeds the pool, automatic offload engages on its own, and its placement +beat the explicit recipe above by 2.1× on the denoise (12.1 vs 25.8 s/it +measured on the same box) — the VRAM/host split that justifies every flag in +this section does not exist there. Expect ~12 min of load, a ~5.5 min text +encoding stage per request (steady-state compute on this chip, not a stall), +and ~12 min per warm 480P request. + **Two budgets, and what each one buys** | | 12 GB VRAM + 32 GB host | host free, VRAM 16 GB | @@ -1384,16 +1976,33 @@ Under that cap, Recipe A wins the whole request at every host size: Same GPU, same load window, unpruned bf16 checkpoints, outputs verified. All figures are anchored at 480P — activations grow with the pixel count, so at 768P drop the resident DiT layers to 0 first, then `video_vae` to 24 if the -decode still collides. And the host convention holds the weights in page -cache; a physical 32 GB machine re-reads them from disk each step, so the -page cache cannot hold the per-step weight sweep, so every step re-reads it -from disk and the drive becomes the denoise clock: a real desktop 4090 with a -990 Pro measured 38 s/step, reading 52.9 GB per step (faulted sequentially, so -almost none of it shows in majflt — measure `read_bytes`, not major faults). -Two things cut that read directly: resident DiT layers (~1 GB/step each — on a -physically small host raise them as far as VRAM allows, the opposite of the -capped-host guidance above), and more RAM (64 GB caches the sweep and returns -to the quoted times). The +decode still collides. + +One convention behind that table is the one a home machine does not share: the +host sizes above are `psutil` patches on a 2 TB lab machine, so the kernel kept +the whole checkpoint in page cache and nothing was read from disk. A physical +host of that size re-reads the streamed weights every step, and what that costs +is computable — `bytes per step = (50 - resident DiT layers - layers the host +pin budget covered) x 1.23 GiB`. The courier overlaps that read with compute, so +the drive stops being the denoise clock once its sequential rate exceeds those +bytes divided by the card's step time. A 32 GB host covers roughly 13 layers, +which puts a 24 GB card at ~38 GiB/step and a 12/16 GB card at ~46 GiB/step: a +PCIe 4.0 NVMe (6–7 GiB/s) clears both, a PCIe 3.0 drive (~3.2 GiB/s) becomes the +clock, and a SATA SSD is not usable for this model. Resident DiT layers remove +1.23 GiB/step each, but only while the streamed set is larger than the pin +budget — once the read fits under the step's compute time, more of them buy +nothing and only cost VRAM. + +An earlier revision of this page recorded 38 s/step on a physical 4090 with a +990 Pro, reading 52.9 GB per step — about 1.4 GB/s, and faulted sequentially, so +almost none of it showed in `majflt` (measure `read_bytes`, not major faults). +That rate was the old path, not the drive: the pinned-store pool rounded every +block up to a power of two, so the pin budget over-committed and the machine had +to run with pinning off, and the layers that stayed mapped faulted in 4 KiB at a +time. Exact-size pinning and O_DIRECT reads through the courier removed both, +and the physical 5090 run below shows the drive fully hidden behind compute. A +4090 on a 32 GB host is derived to return to its compute wall (~8.5 s/step) on a +PCIe 4.0 drive; that machine has not been re-measured. The VRAM axis holds too: capped at 16 GiB the same recipe wins ~250 vs 292–301 s, and at 24 GiB (with `--dit-layerwise-resident-layers 6` — measured at a 22 GiB cap so a desktop's own allocations fit; a headless card can raise it @@ -1410,7 +2019,9 @@ Stage by stage under the cap: text encoding is even (both stream the same 48 GB Qwen3VL), the denoise leads at 32–48 GB hosts and sits within run-to-run variance of ComfyUI at 64 GB (162 vs 159 s), and the decode leads everywhere. Two ComfyUI notes that still matter: `--fast-disk` measured no -faster than its default here, and stacking +faster than its default on this lab host — though on a physical 60 GB desktop +its default is OOM-killed while loading (55.6 GB of anonymous memory) and +`--fast-disk` is what makes it run at all — and stacking `--novram --cache-none --disable-pinned-memory` made things strictly worse (69.1 GiB anonymous, 750 s requests) — the adaptive default is the right configuration on a small host. @@ -1421,6 +2032,61 @@ i.e. an int8 DiT and an NVFP4 text encoder, and its pruned bf16 file is 40.2 GB against the unpruned 66.3 GB. Those are different weights, so it is not a like-for-like comparison with the recipes above. +### RTX 5090 single card, physical desktop + +Every consumer figure above this point was measured with at least one axis +simulated: a hard allocator cap standing in for VRAM, a `psutil` patch standing +in for host RAM, and a lab machine whose page cache held the whole checkpoint. +This run had none of that — an RTX 5090 (32 GB) in a Ryzen 9 9950X desktop with +60 GB of RAM, an 8 GB swap file and a Samsung 9100 PRO on PCIe 5.0, on Ubuntu +24.04. The workload matches the comparison above: FL2VA T2VA, 864×480, 124 +frames, 20 steps, cfg 1.0, euler_ancestral, sigma shift 12.0/3.0, same seeds and +keyframes on both engines, unpruned bf16 weights on both sides. + +```bash 32 GB VRAM + 60 GB host, lossless +PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \ +sglang serve --model-path MiniMaxAI/MiniMax-H3 --model-variant fl2va \ + --performance-mode memory \ + --layerwise-offload-components dit,text_encoder,vae \ + --dit-layerwise-resident-layers 14 \ + --layerwise-resident-layers video_vae=36 \ + --warmup-resolutions 864x480 --warmup-num-frames 124 +``` + +| | text encoding | denoise step | decode | request | +| --- | ---: | ---: | ---: | ---: | +| sglang | **4.2 s** | 5.14–5.17 s | **6.4 s** | **112.2 / 112.1 s** | +| ComfyUI `--fast-disk --cache-none` | 11.3 s | 5.17–5.30 s | 7.2 s | 145.9 / 145.6 / 140.9 s | + +Three things a capped run cannot settle: + +**The denoise is at the card's compute wall, not the drive's.** The two engines +are within noise of each other per step, and prefetch depth 2 and cuDNN SDPA +each measured as no change. The 51.3 GiB the DiT reads from the drive per +request is fully overlapped — across a whole request the compute thread waited +0.9 s for the courier. + +**Host RAM decides how much traffic there is to hide.** At 60 GB the pin budget +covers 34 of the 50 DiT layers, so only about two layers per step come off the +disk. The 46 GiB text encoder is never pinned — streamed layers take the budget +first — and is read whole, once per request, at the drive's sequential rate: +that is the 4.2 s stage, against 12.1–12.9 s when the same bytes go through the +page cache. + +**Neither engine's default command survives this machine.** ComfyUI's is +OOM-killed during load (55.6 GB of anonymous memory) and needs `--fast-disk`; +sglang's was too, until pinned stores were allocated at their exact size instead +of the allocator's next power of two. Even now the automatic placement keeps no +resident DiT layers and streams the VAE on every decode, so the flags above are +what reach the numbers in the table. + +Two caveats to carry into your own machine. The run finished with its 8 GB swap +file full, because the pin budget takes 95% of the memory available after +loading and there is no smaller setting yet — on a machine that is also your +desktop, expect that pressure and give the card more resident layers, since a +resident layer is one the host no longer has to pin. And these are 480P figures: +at 768P drop the resident DiT layers first, then `video_vae` to 24. + ### RTX 4090 24 GB single-GPU run One RTX 4090 D 24 GB completed the 1344×768, 107-frame, 20-NFE T2VA @@ -1440,7 +2106,8 @@ timed (seed 42); only the timed pass is reported. GPU peak stayed about `kitchen_int8` + FA changes Linear numerics only. The `sol_attn` / `sage_attn` / hybrid rows also change the attention algorithm, so speed and pixel fidelity rank in opposite orders there. Default remains -`kitchen_int8` + `fa`. +`kitchen_int8` + `fa`. Cache-DiT can share this layerwise recipe; omit +`quality` and see the quality-level section. ### AMD Instinct task and scaling runs @@ -1477,3 +2144,9 @@ For a measured lower-count AMD deployment, set both `--num-gpus` and `--ulysses-degree` to 4, 2, or 1. AITER packed attention matched segment-wise BF16 SDPA at cosine similarity `0.9999991655` on MI355X and `0.9999991059` on MI300X. + +## 10. Run in ComfyUI + +import { ComfyUISupport } from '/src/snippets/diffusion/comfyui-support.jsx'; + + diff --git a/docs/cookbook/diffusion/Qwen-Image/Qwen-Image-2.1.mdx b/docs/cookbook/diffusion/Qwen-Image/Qwen-Image-2.1.mdx new file mode 100644 index 000000000000..46cc9ad90975 --- /dev/null +++ b/docs/cookbook/diffusion/Qwen-Image/Qwen-Image-2.1.mdx @@ -0,0 +1,244 @@ +--- +title: Qwen-Image 2.1 +description: "Run Qwen-Image 2.1 text-to-image and image-conditioned generation with SGLang Diffusion." +tag: NEW +--- + +import { DiffusionModelTags } from '/src/snippets/diffusion/model-tags.jsx'; +import { Deployment } from '/src/snippets/_deployment.jsx'; +import { config } from '/src/snippets/configs/Qwen/qwen-image-2.1.jsx'; + + + +## 1. Quick start + +Install the runtime dependencies with `uv pip install "sglang[diffusion]" --prerelease=allow`, +then install this integration from its source checkout with +`uv pip install -e "python[diffusion]"`. The picker uses `Qwen/Qwen-Image-2.1`; +you can also set a local checkpoint directory under **Variables**. The recipes +target NVIDIA CUDA on Linux; the picker marks which single-GPU workloads have +been verified with the full checkpoint. + + + +Use **Setup** to select text-to-image, single-image editing, or multi-image +editing. **Server** controls placement, attention, encoder scheduling, VAE +tiling, graph execution, and request batching. **Request** controls the background, resolution, +steps, and output count. Set reference PNG paths under **Variables**; edits +upload files from the machine running cURL, so they need not exist on the server. + +Hardware selection applies the recommended placement for that GPU. H200, +B200, RTX PRO 6000 96GB, and DGX Spark keep weights resident; RTX 5090 and RTX 4090 +offload selected components to fit the full pipeline. +Untested topologies and feature combinations remain selectable and are labeled +**Unverified**. Invalid topology combinations disable Copy. This integration +currently uses the Python/source command; no published Docker image is verified. + +Both request modes return base64 PNGs. To save all returned images, append +`> response.json` to the request command, then run: + +```bash Command +python - <<'PY' +import base64 +import json +from pathlib import Path + +for i, item in enumerate(json.loads(Path("response.json").read_text())["data"]): + Path(f"output-{i}.png").write_bytes(base64.b64decode(item["b64_json"])) +PY +``` + +### Recommended hardware settings + +The picker defaults to native BF16/FP32 precision, exact attention, eager +execution, and full-image VAE decoding. +Commands omit default values, including one GPU, encoder auto scheduling, and +batch size one. Explicit placement and attention overrides preserve each recipe. + +| GPU | Placement / attention | Generation | Edit | Peak VRAM | +| --- | --- | --- | --- | --- | +| H200 141GB | Resident / FlashAttention | 4.48 s | 5.29 s | 38.4 GiB | +| B200 192GB | Resident / FlashAttention | 2.46 s | 3.02 s | 38.5 GiB | +| RTX PRO 6000 96GB | Resident / Torch SDPA | 8.03 s | 9.63 s | 38.4 GiB | +| RTX 4090 24GB | DiT and VAE resident, encoder layerwise offload / FlashAttention | 18.68 s | 21.68 s | 22.7 GiB | +| DGX Spark 128GB unified | Resident / Torch SDPA | 35.36 s | 42.23 s | — (unified) | + +Measured on 2026-09-20 at 1024×1024, 40 steps, CFG 1, and one RGBA PNG per +request. Times are median HTTP latency after warmup, including PNG serialization +and excluding startup; VRAM is the sampled request-phase peak. Prompts and +software versions affect both latency and memory use. + +RTX 5090 uses DiT layerwise offload and Torch SDPA; its recipe has not been +retested with the updated checkpoint. Both RTX 5090 and RTX PRO 6000 use SDPA +when FlashAttention is selected in this runtime. CPU offload requires host RAM. + +### DGX Spark + +Select **DGX Spark** for one GB10 GPU on Linux ARM64 with CUDA 13. Use the +source installation above. The recommended configuration keeps all components +resident, uses native BF16/FP32 precision, and lets the runtime select Torch SDPA: + +```bash Command +sglang serve \ + --model-path Qwen/Qwen-Image-2.1 \ + --performance-mode speed +``` + +The [128 GB unified memory](https://docs.nvidia.com/dgx/dgx-spark/hardware.html) +is shared by the CPU and GPU. CPU offload is unnecessary for the verified +single-image 1024×1024 workload. Keep full-image VAE decoding and eager execution. +Generation, editing, transparent generation, and transparent editing were +verified with PyTorch 2.13.0+cu130. Spark reports no separate VRAM usage in `nvidia-smi`. +This recipe covers one Spark; multi-node deployment and batching remain unverified. + +### Batching + +Keep **Request batching → Off** and **Outputs → 1** for interactive use. +Batching increases individual request latency and does not guarantee higher +throughput. Measure your workload before enabling it. + +Cross-request batching merges compatible text-to-image requests. Image edits +run separately; **Outputs** controls multiple images within one request. +On RTX 4090, selecting multiple outputs or request batching switches to DiT +layerwise offload for memory headroom. Restart with the updated **Server** command. + +Batching preserves native precision but can change floating-point rounding and +output pixels, even with the same seed. See +[Inference batching](/docs/sglang-diffusion/dynamic_batching) for admission rules +and metrics. + +## 2. Model capabilities + +Qwen-Image 2.1 supports text-to-image generation, single- and multi-image editing, +and RGBA output. Use one checkpoint for all modes. + +For multi-round editing, send the previous output as the next reference image. +The server does not retain conversation state. Condition-prefix KV caches are +reused within one request; cross-request and dialogue-history caching are not +implemented. + +## 3. Checkpoint layout + +The checkpoint directory must contain `model_index.json` and the `processor`, +`text_encoder`, `transformer`, `vae`, and `scheduler` subdirectories. The processor +includes the Qwen3-VL tokenizer assets; no separate tokenizer directory is needed. +Use `--model-id Qwen-Image-2.1` when your local checkpoint directory has another +name. Older Qwen-Image and Qwen-Image-Edit transformer/VAE weights are incompatible. + +Keep SGLang's installed dependencies. Its native encoder preserves the +reference's Transformers 4.57.3 conditioning semantics without requiring a +runtime-wide downgrade. + +### Transparent PNG output + +Choose **Transparent / alpha** under Request and describe an isolated subject +on a transparent background in the prompt. The picker adds this instruction +and selects PNG. `background: "transparent"` alone does not change conditioning +or remove the background; JPEG cannot retain alpha. + +PNG references retain their alpha channel during editing; RGB references +receive an opaque alpha channel. The model predicts continuous alpha values, +including partly transparent edges, without thresholding or background removal. + +## 4. Offline requests + +Defaults are 1024×1024, 40 steps, CFG 1, and seed 42; output saving is enabled. +For GPUs that need offload, also pass the placement flags from the picker. + +### Text-to-image + +```bash Command +sglang generate \ + --model-path Qwen/Qwen-Image-2.1 \ + --prompt "A capybara reading a book by candlelight" +``` + +### Image-conditioned editing + +```bash Command +sglang generate \ + --model-path Qwen/Qwen-Image-2.1 \ + --image-path /path/to/input.png \ + --prompt "Move the scene to a snowy mountain at sunrise" +``` + +Height and width must be positive multiples of 32. Reference images preserve +their aspect ratio and are resized to approximately the requested output area; +the same resized image feeds the VLM and VAE. Image labels are deterministic +(`Picture 1`, `Picture 2`, and so on). Multiple outputs receive independent +noise seeds and independent prefix caches. + +## 5. Runtime features + +The default is 40 Euler flow-matching steps with CFG disabled. For CFG, provide +`--negative-prompt` and `--guidance-scale` greater than one. The API requires a +text prompt; precomputed embeddings alone are insufficient. + +- **Parallelism:** TP, Ulysses, Ring, CFG parallelism, and encoder folding are + available in the picker. The target token count, `(height / 16) × (width / 16)`, + must be divisible by the SP degree. Ring requires FlashAttention or SageAttention. +- **Memory:** use the hardware's recommended placement. **All components + layerwise** also streams encoder and VAE blocks, trading transfers for lower + device memory. +- **VAE:** full-image decoding is the default. Tiling can change pixels near + boundaries. With two or more GPUs, **Spatial shard** distributes full-image + decoding without enabling tiling; floating-point rounding can still differ. + +See the [compatibility inventory](/docs/sglang-diffusion/compatibility_matrix) +for configuration support and the +[performance guide](/docs/sglang-diffusion/performance-optimization) for shared +runtime options. + +### Quantization + +Native precision is the default. Quantization changes image and alpha values; +check quality on your own prompts and reference images. Set compatible component +paths under **Variables** when choosing an exported format. Adding quantization +metadata to native weights does not convert them. + +For online FP8, use `--component-quantizations.transformer fp8`, +`--component-quantizations.text_encoder fp8`, or both. + +### Serialized FP8 components + +Select a **Serialized FP8** option and set the exported component directories. +Each directory needs its architecture `config.json`, weights, and quantization +metadata. Use `--component-paths.transformer` and/or +`--component-paths.text_encoder`; omit online quantization flags. +See the [quantization guide](/docs/sglang-diffusion/quantization) for formats. + +### GGUF components + +Select a **GGUF** option and set the `.gguf` files. The picker uses +`--component-weights-paths.transformer` and/or +`--component-weights-paths.text_encoder`, retaining architecture configs from the +base checkpoint. Each file must contain the entire component with native tensor +names. GGUF reduces weight storage but does not guarantee lower latency. +See the [GGUF guide](/docs/sglang-diffusion/quantization#gguf). + +### NVFP4 components + +NVFP4 requires Blackwell and compatible ModelOpt exports. Select the component +directories using `--component-paths.transformer` and/or +`--component-paths.text_encoder`. Keep the FlashInfer backend at `auto` on +RTX 5090, RTX PRO 6000, and DGX Spark: TensorRT-LLM FP4 GEMM does not support SM12.x. +These GPUs remain unverified for this model's NVFP4 exports. See the +[NVFP4 guide](/docs/sglang-diffusion/quantization#modelopt-nvfp4). + +### LoRA and execution options + +Use `--lora-path` and `--lora-merge-mode dynamic|merge` or the runtime adapter APIs. +Diffusers adapter keys prefixed with `transformer.` map to the native DiT. + +Keep eager execution as the default. Breakable CUDA Graph replay requires +matching resolution and condition-prefix length; unseen shapes run eagerly. +Text buckets alone do not guarantee replay. SageAttention and Cache-DiT can +change numerical results and require quality checks for your workload. + +### Cache-DiT + +Enable `--enable-cache-dit true` or `SGLANG_CACHE_DIT_ENABLED=true`. 2.1 prefix +KV is per layer: each block slices caches by `_layer_id`. Cache-DiT wraps +`transformer_blocks` and forwards the same extras to every layer; without that +slice, later layers reuse layer 0 and the image collapses to color noise. +See the [Cache-DiT guide](/docs/sglang-diffusion/cache_dit). diff --git a/docs/cookbook/diffusion/Qwen-Image/Qwen-Image-Edit.mdx b/docs/cookbook/diffusion/Qwen-Image/Qwen-Image-Edit.mdx index 50d0d82b5487..5b7c92e1e394 100644 --- a/docs/cookbook/diffusion/Qwen-Image/Qwen-Image-Edit.mdx +++ b/docs/cookbook/diffusion/Qwen-Image/Qwen-Image-Edit.mdx @@ -35,7 +35,7 @@ Qwen-Image-Edit-2511 is a 20B parameter model optimized for image editing tasks. ### 3.2 Configuration Tips -Currently supported optimizations are listed [here](/docs/sglang-diffusion/compatibility_matrix). +See [Performance Optimization](/docs/sglang-diffusion/performance-optimization) for acceleration features and their runtime requirements. - `--vae-path`: Path to a custom VAE model or HuggingFace model ID (e.g., fal/FLUX.2-Tiny-AutoEncoder). If not specified, the VAE will be loaded from the main model path. - `--num-gpus`: Number of GPUs to use @@ -44,6 +44,44 @@ Currently supported optimizations are listed [here](/docs/sglang-diffusion/compa - `--ulysses-degree`: The degree of DeepSpeed-Ulysses-style SP in USP - `--ring-degree`: The degree of ring attention-style SP in USP +### 3.3 Decompose an image into layers on H200 + +`Qwen/Qwen-Image-Layered` returns separate RGBA images. For this model, +`--num-frames 4` requests four output layers. The CLI saves all four PNGs, +and `DiffGenerator.generate()` returns one result per layer. + +On Linux with NVIDIA CUDA and two H200 GPUs, you can run the conditional and +unconditional branches on separate GPUs: + +```bash Command +CUDA_VISIBLE_DEVICES=0,1 sglang generate \ + --model-path Qwen/Qwen-Image-Layered \ + --num-gpus 2 \ + --cfg-parallel-size 2 \ + --tp-size 1 \ + --ulysses-degree 1 \ + --quality lossless \ + --enable-torch-compile false \ + --warmup-mode request \ + --width 640 --height 640 --num-frames 4 \ + --num-inference-steps 50 --guidance-scale 4.0 --seed 42 \ + --image-path https://raw.githubusercontent.com/QwenLM/Qwen-Image-Layered/main/assets/test_images/4.png \ + --prompt "a high quality, cute halloween themed illustration, consistent style and lighting" \ + --output-path outputs/qwen-layered \ + --save-output +``` + +For a single H200, set `CUDA_VISIBLE_DEVICES=0`, `--num-gpus 1`, and +`--cfg-parallel-size 1`. Both configurations use eager execution. Layered +does not currently support breakable CUDA graph; enabling BCG falls back to +eager execution. + +The Layered CFG policy gathers the branch predictions before applying the +single-GPU arithmetic order, preserving BF16 rounding and alpha values in the +validated fixed-seed example. Each GPU still holds a full DiT replica, so CFG +parallelism reduces request latency without reducing the model memory needed +on each GPU. + ## 4. API Usage For complete API documentation, please refer to the [official API usage guide](/docs/sglang-diffusion/api/openai_api). @@ -189,6 +227,13 @@ sglang serve --model-path Qwen/Qwen-Image-Edit-2511 - `--vae-cpu-offload`: Use CPU offload for VAE. - `--pin-cpu-memory`: Pin memory for CPU offload. Only added as a temp workaround if it throws "CUDA error: invalid argument". +#### 4.2.3 Known LoRA examples + +Use `--lora-path` at startup or the [LoRA management API](/docs/sglang-diffusion/api/openai_api#lora-management) to load an adapter. Known Qwen-Image-Edit examples include: + +- [`ostris/qwen_image_edit_inpainting`](https://huggingface.co/ostris/qwen_image_edit_inpainting) +- [`lightx2v/Qwen-Image-Edit-2511-Lightning`](https://huggingface.co/lightx2v/Qwen-Image-Edit-2511-Lightning) + ## 5. Benchmark Test Environment: @@ -270,3 +315,9 @@ Peak Memory Mean (MB): 47971.49 Peak Memory Median (MB): 47971.29 ============================================================ ``` + +## 6. Run in ComfyUI + +import { ComfyUISupport } from '/src/snippets/diffusion/comfyui-support.jsx'; + + diff --git a/docs/cookbook/diffusion/Qwen-Image/Qwen-Image.mdx b/docs/cookbook/diffusion/Qwen-Image/Qwen-Image.mdx index 4703ca5e1358..881dfa9c5e4a 100644 --- a/docs/cookbook/diffusion/Qwen-Image/Qwen-Image.mdx +++ b/docs/cookbook/diffusion/Qwen-Image/Qwen-Image.mdx @@ -27,7 +27,7 @@ This section provides deployment configurations optimized for different hardware ### 3.1 Basic Configuration -Qwen-Image is a text-to-image model. The recommended launch configurations vary by hardware. SGLang supports serving Qwen-Image on NVIDIA B200, B300, H200, H100, AMD MI300X, MI325X, MI355X GPUs and Ascend A2, A3 NPUs. +Qwen-Image is a text-to-image model. The recommended launch configurations vary by hardware. SGLang supports serving Qwen-Image on NVIDIA B200, B300, H200, H100, AMD MI300X, MI325X, MI355X GPUs and Ascend A2/A3 Series NPUs. **Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform. @@ -55,9 +55,47 @@ sglang generate \ --save-output ``` -### 3.2 Configuration Tips +### 3.2 Fixed-resolution latency on two H200 GPUs -Currently supported optimizations are listed [here](/docs/sglang-diffusion/compatibility_matrix). +For `Qwen/Qwen-Image-2512` at 1024x1024, use breakable CUDA graph (BCG) to +reduce launch overhead across graph-safe DiT segments while retaining explicit +breakpoints around unsupported operations. This recipe was validated on two +NVIDIA H200 GPUs with 50 denoising steps and no classifier-free guidance: + +```bash Command +sglang serve \ + --model-path Qwen/Qwen-Image-2512 \ + --model-type diffusion \ + --num-gpus 2 \ + --tp-size 2 \ + --performance-mode speed \ + --dit-layerwise-offload false \ + --enable-torch-compile false \ + --enable-breakable-cuda-graph \ + --warmup-mode server \ + --warmup-resolutions 1024x1024 +``` + +Declare every production resolution in `--warmup-resolutions`. A request at an +uncaptured resolution runs eagerly, so omitting `1024x1024` removes the gain +from this recipe. Graph capture used about 5 GB more peak memory per GPU in the +validation run. + +On CUDA, the TP path dispatches supported collectives through SRT +CustomAllReduceV2. At 1024x1024, Qwen-Image reduces 24 MiB row-parallel +outputs; the diffusion runtime reserves a 32 MiB V2 workspace so these +collectives do not fall back to NCCL. If profiling shows large NCCL all-reduce +kernels again, first confirm that V2 is enabled and the requested shape fits +the workspace. + +BCG changed floating-point execution order but not the sampling algorithm. The +fixed-seed output measured 0.984 SSIM and 39.7 dB PSNR against eager output; use +eager execution when you require bit-exact output. Regional `torch.compile` was +also tested on this profile and did not improve steady-state latency. + +### 3.3 Configuration Tips + +See [Performance Optimization](/docs/sglang-diffusion/performance-optimization) for acceleration features and their runtime requirements. - `--vae-path`: Path to a custom VAE model or HuggingFace model ID (e.g., fal/FLUX.2-Tiny-AutoEncoder). If not specified, the VAE will be loaded from the main model path. - `--num-gpus`: Number of GPUs to use @@ -211,6 +249,15 @@ sglang serve --model-path Qwen/Qwen-Image - `--vae-cpu-offload`: Use CPU offload for VAE. - `--pin-cpu-memory`: Pin memory for CPU offload. Only added as a temp workaround if it throws "CUDA error: invalid argument". +#### 4.2.3 Known LoRA examples + +Use `--lora-path` at startup or the [LoRA management API](/docs/sglang-diffusion/api/openai_api#lora-management) to load an adapter. Known Qwen-Image examples include: + +- [`lightx2v/Qwen-Image-Lightning`](https://huggingface.co/lightx2v/Qwen-Image-Lightning) +- [`flymy-ai/qwen-image-realism-lora`](https://huggingface.co/flymy-ai/qwen-image-realism-lora) +- [`prithivMLmods/Qwen-Image-HeadshotX`](https://huggingface.co/prithivMLmods/Qwen-Image-HeadshotX) +- [`starsfriday/Qwen-Image-EVA-LoRA`](https://huggingface.co/starsfriday/Qwen-Image-EVA-LoRA) + ## 5. Benchmark Test Environment: @@ -264,11 +311,11 @@ Test Environment: ============================================================ ``` - + **Server Command**: ```shell Command - #One A3 card has 2 npu chips + #One A3 Series card has 2 npu chips sglang serve --tp-size 2 --sp-degree 1 --model-path Qwen/Qwen-Image --num-gpus 2 ``` @@ -344,7 +391,7 @@ Test Environment: ============================================================ ``` - + **Benchmark Command**: ```shell Command @@ -381,3 +428,9 @@ Test Environment: ``` + +## 6. Run in ComfyUI + +import { ComfyUISupport } from '/src/snippets/diffusion/comfyui-support.jsx'; + + diff --git a/docs/cookbook/diffusion/README.mdx b/docs/cookbook/diffusion/README.mdx index c26aaa21c75a..df988182c836 100644 --- a/docs/cookbook/diffusion/README.mdx +++ b/docs/cookbook/diffusion/README.mdx @@ -69,4 +69,4 @@ validation then catches MDX, navigation, and internal-link problems. For runtime concepts and platform support, see the [SGLang Diffusion documentation](/docs/sglang-diffusion/index) and -[compatibility matrix](/docs/sglang-diffusion/compatibility_matrix). +[supported model catalog](/docs/sglang-diffusion/compatibility_matrix). diff --git a/docs/cookbook/diffusion/SANA-WM/SANA-WM.mdx b/docs/cookbook/diffusion/SANA-WM/SANA-WM.mdx index 970894f4fe36..4983ea663c26 100644 --- a/docs/cookbook/diffusion/SANA-WM/SANA-WM.mdx +++ b/docs/cookbook/diffusion/SANA-WM/SANA-WM.mdx @@ -192,7 +192,7 @@ Notes on launch behavior: - **Default endpoint** is `127.0.0.1:30000` (`--host` / `--port` override). - **CPU offload flags are optional.** `--dit-cpu-offload`, `--text-encoder-cpu-offload`, and `--image-encoder-cpu-offload` are available; defaults are auto-adjusted from GPU memory (GPUs under 30 GB get more aggressive offloading). -- **Multi-GPU realtime.** Prefer explicit sequence parallelism (`--sp-degree` equal to the number of GPUs for a single session). Do not enable CFG parallel for the realtime profile: the default realtime request uses `guidance_scale=1.0`, while CFG parallel requires active cond/uncond branches. +- **Multi-GPU realtime.** Prefer explicit sequence parallelism (`--sp-degree` equal to the number of GPUs for a single session). Do not enable CFG parallel for the realtime profile: the default request uses `guidance_scale=1.0`, so the extra CFG ranks only recompute the same branch. - **FSDP.** Use `--use-fsdp-inference` only when you specifically need weight sharding for memory. For the low-latency realtime profile, prefer keeping components resident and using SP first. - **Warmup.** Server warmup is **automatically skipped** for the realtime pipeline — a synthetic warmup request has no WebSocket session, so the server detects the registered realtime adapter and skips it. No explicit `--warmup-mode` setting is needed. @@ -351,7 +351,7 @@ Header fields: **Encodings.** `application/x-raw-rgb` is uncompressed RGB24 (3 × uint8, `bytes_per_frame = width*height*3`). `application/x-raw-rgb-delta-gzip` is the zlib-compressed **per-frame XOR delta** against the preceding frame (each frame in the batch is XOR'd against the previous one; sent by default). `realtime_output_format: "raw"` forces uncompressed RGB; `"webp"` / `"jpeg"` send preview-encoded frames. -delta-gzip must be restored **frame-by-frame**: decompress the payload, then for each frame XOR it against the already-restored previous frame (the first frame of a batch references the last frame of the previous batch). See `restore_delta_gzip_raw_rgb_payload` in `runtime/utils/realtime_video.py`. The `"raw"` format below avoids this. +delta-gzip must be restored **frame-by-frame**: decompress the payload, then for each frame XOR it against the already-restored previous frame (the first frame of a batch references the last frame of the previous batch). See `restore_delta_gzip_raw_rgb_payload` in `runtime/realtime/video.py`. The `"raw"` format below avoids this. ### Minimal client example @@ -487,3 +487,9 @@ At WebSocket `init` the realtime adapter fills SANA-WM defaults that differ from `guidance_scale` applies to the dense path (§4) only; the distilled streaming path uses `streaming_cfg_scale` (default `1.0`, i.e. no CFG) so a `guidance_scale` override never accidentally enables CFG on the streaming stage. `denoising_step_list = (1000, 960, 889, 727, 0)` is the official 4-step streaming schedule (it must end in 0). + +## 10. Run in ComfyUI + +import { ComfyUISupport } from '/src/snippets/diffusion/comfyui-support.jsx'; + + diff --git a/docs/cookbook/diffusion/SenseNova/SenseNova-U1.5-8B-MoT.mdx b/docs/cookbook/diffusion/SenseNova/SenseNova-U1.5-8B-MoT.mdx new file mode 100644 index 000000000000..f5006040c167 --- /dev/null +++ b/docs/cookbook/diffusion/SenseNova/SenseNova-U1.5-8B-MoT.mdx @@ -0,0 +1,184 @@ +--- +title: SenseNova-U1.5-8B-MoT +metatags: + description: "Run SenseNova-U1.5-8B-MoT text-to-image generation with SGLang's native multimodal generation runtime." +--- + +import { DiffusionModelTags } from '/src/snippets/diffusion/model-tags.jsx'; + + + +## 1. Model Introduction + +[SenseNova-U1.5-8B-MoT](https://huggingface.co/sensenova/SenseNova-U1.5-8B-MoT) is a text-to-image model in the SenseNova-U1 family. It is useful for high-resolution poster, infographic, and enterprise visual-generation workloads where prompt following, layout structure, and Chinese text rendering are important. + +Built on [NEO-Unify](https://huggingface.co/blog/sensenova/neo-unify), this release improves composition, color harmony, material rendering, and local visual detail for native high-resolution generation. It also focuses on clearer Chinese and English text rendering, stronger infographic layout structure, and more reliable execution of complex prompts with object counts, spatial relationships, styles, and multiple constraints. + +## 2. SGLang-diffusion Installation + +SGLang-diffusion offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. + +Please refer to the [official SGLang-diffusion installation guide](../../../docs/sglang-diffusion/installation) for installation instructions. + +## 3. Model Deployment + +### 3.1 Offline Image Generation + +SenseNova-U1.5-8B-MoT is a text-to-image model. You can run a single offline generation job with `sglang generate`: + +```bash Command +sglang generate \ + --model-path sensenova/SenseNova-U1.5-8B-MoT \ + --prompt "A cinematic mountain lake at sunrise, realistic photography." \ + --width 2048 \ + --height 2048 \ + --guidance-scale 4.0 \ + --num-inference-steps 50 \ + --seed 42 \ + --output-file-path output_sensenova_u15.png +``` + +### 3.2 Recommended Resolutions + +SenseNova-U1 was trained around the 2K-pixel resolution tier. The SGLang pipeline only supports positive `--width` and `--height` values that are divisible by 32, and generation quality may degrade for untrained size combinations. + +Recommended resolution tiers: + +| Aspect ratio | Resolution | +| --- | --- | +| 1:1 | `2048 x 2048` | +| 16:9 / 9:16 | `2720 x 1536` / `1536 x 2720` | +| 3:2 / 2:3 | `2496 x 1664` / `1664 x 2496` | +| 4:3 / 3:4 | `2368 x 1760` / `1760 x 2368` | +| 2:1 / 1:2 | `2880 x 1440` / `1440 x 2880` | +| 3:1 / 1:3 | `3456 x 1152` / `1152 x 3456` | +| 9:16 4K | `2304 x 4096` | + +## 4. API Usage + +For complete API documentation, please refer to the [official API usage guide](../../../docs/sglang-diffusion/api/openai_api). + +### 4.1 Generate an Image + +```python Example +import base64 +from openai import OpenAI + +client = OpenAI(api_key="EMPTY", base_url="http://localhost:30000/v1") + +response = client.images.generate( + model="sensenova/SenseNova-U1.5-8B-MoT", + prompt="A clean technology poster with structured layout and clear typography.", + n=1, + response_format="b64_json", +) + +image_bytes = base64.b64decode(response.data[0].b64_json) +with open("output_sensenova_u15.png", "wb") as f: + f.write(image_bytes) +``` + +## 5. Advanced usage + +### 5.1 Dynamic request batching + +Start the server with dynamic batching enabled: + +```bash Command +sglang serve \ + --model-path sensenova/SenseNova-U1.5-8B-MoT \ + --port 30000 \ + --batching-max-size 2 \ + --batching-delay-ms 100 +``` + +Submit requests concurrently so they arrive within the batching delay. Requests +in one batch must use the same resolution, inference steps, guidance settings, +and output options. Requests with multiple outputs or `think_mode=true` are +executed sequentially. + +### 5.2 Ascend NPU optimizations + +On Ascend NPU, SenseNova-U1 automatically uses fused inference attention, +RMSNorm, and SwiGLU MLP operators for supported inputs. FIA and SwiGLU fall back +when their requirements are not met. Run one complete warmup request before +benchmarking because the fused MLP packs its gate and up projection weights on +first use. + +## 6. Benchmark + +### 6.1 Speedup benchmark + +#### 6.1.1 Single-run profile + + + + + This is a single-run profile for smoke validation rather than a formal benchmark. Environment: NVIDIA A800; workload: vbench text-to-image, 1 prompt, concurrency 1, 1 output per prompt; warmup: none; repetitions: 1; dispersion: not reported; baseline: none; revisions: PR checkout and resolved model snapshot at run time; output validation: request completed and returned one image. + + **Server Command**: + + ```shell Command + sglang serve \ + --model-path sensenova/SenseNova-U1.5-8B-MoT \ + --port 30000 + ``` + + **Benchmark Command**: + + ```shell Command + python3 -m sglang.multimodal_gen.benchmarks.bench_serving \ + --dataset vbench \ + --task text-to-image \ + --num-prompts 1 \ + --max-concurrency 1 \ + --port 30000 + ``` + + **Result**: + + ```text Output + ================= Serving Benchmark Result ================= + Task: text-to-image + Model: /workspace/models/ModelScope/SenseNova/SenseNova-U1.5-8B-MoT + Dataset: vbench + -------------------------------------------------- + Benchmark duration (s): 13.10 + Request rate: inf + Max request concurrency: 1 + Successful requests: 1/1 + Completed outputs: 1 + Outputs per prompt: 1 + -------------------------------------------------- + Request throughput (req/s): 0.08 + Output throughput (outputs/s): 0.08 + Latency Mean (s): 13.10 + Latency Median (s): 13.10 + Latency P90 (s): 13.10 + Latency P95 (s): 13.10 + Latency P99 (s): 13.10 + -------------------------------------------------- + Peak Memory Max (MB): 34326.00 + Peak Memory Mean (MB): 34326.00 + Peak Memory Median (MB): 34326.00 + ------------------------------------------------------------ + ``` + + + + Environment: one Ascend 910C; workload: four concurrent 2048 x 2048 + text-to-image requests, 50 denoising steps, CFG 4, BF16. + + | Batch size | NPU operators | Duration (s) | Throughput (images/s) | Mean latency (s) | Peak reserved memory (MB) | + |---:|---|---:|---:|---:|---:| + | 1 | Disabled | 234.28 | 0.01707 | 146.47 | 35698 | + | 2 | Disabled | 234.83 | 0.01703 | 176.06 | 37870 | + | 1 | FIA + RMSNorm + SwiGLU | 205.94 | 0.01942 | 128.80 | 35718 | + | 2 | FIA + RMSNorm + SwiGLU | 195.02 | 0.02051 | 146.33 | 37850 | + + Optimized B1 improves throughput by 13.76% over unoptimized B1. Optimized + B2 improves throughput by 20.42% over unoptimized B2 and by 5.60% over + optimized B1. + + + diff --git a/docs/cookbook/diffusion/Wan/Wan2.1.mdx b/docs/cookbook/diffusion/Wan/Wan2.1.mdx index bd1e85726bea..2c5640526340 100644 --- a/docs/cookbook/diffusion/Wan/Wan2.1.mdx +++ b/docs/cookbook/diffusion/Wan/Wan2.1.mdx @@ -27,7 +27,7 @@ This section provides deployment configurations optimized for different hardware ### 3.1 Basic Configuration -The Wan2.1 series offers models in multiple sizes and resolutions. SGLang supports Wan2.1 deployment on NVIDIA B200, B300, H200, H100, and AMD MI300X, MI325X, MI355X GPUs and Ascend A2, A3 NPUs. The recommended launch configurations vary by hardware, model size, and memory headroom. +The Wan2.1 series offers models in multiple sizes and resolutions. SGLang supports Wan2.1 deployment on NVIDIA B200, B300, H200, H100, and AMD MI300X, MI325X, MI355X GPUs and Ascend A2/A3 Series NPUs. The recommended launch configurations vary by hardware, model size, and memory headroom. **Interactive Command Generator**: Use the configuration selector below to automatically generate an appropriate deployment command for your model variant and options. @@ -217,11 +217,11 @@ You can use the built-in SGLang diffusion benchmark script to evaluate Wan2.1 pe ``` - + **Server Command**: ```bash Command - #One A3 card has 2 npu chips. Benchmark was did with two A3 cards + #One A3 Series card has 2 npu chips. Benchmark was done with two A3 Series cards sglang serve \ --model-path /models/Wan-AI/Wan2.1-T2V-14B-Diffusers/ \ --tp-size 2 \ @@ -321,11 +321,11 @@ You can use the built-in SGLang diffusion benchmark script to evaluate Wan2.1 pe ``` - + **Server Command**: ```bash Command - #One A3 card has 2 npu chips. Benchmark was did with two Atlas 3 cards + #One A3 Series card has 2 npu chips. Benchmark was done with two A3 Series cards SGLANG_CACHE_DIT_FN=2 \ SGLANG_CACHE_DIT_BN=1 \ SGLANG_CACHE_DIT_WARMUP=4 \ @@ -381,3 +381,9 @@ You can use the built-in SGLang diffusion benchmark script to evaluate Wan2.1 pe ``` + +## 6. Run in ComfyUI + +import { ComfyUISupport } from '/src/snippets/diffusion/comfyui-support.jsx'; + + diff --git a/docs/cookbook/diffusion/Wan/Wan2.2.mdx b/docs/cookbook/diffusion/Wan/Wan2.2.mdx index b3dddb0d1134..a24355ed30d2 100644 --- a/docs/cookbook/diffusion/Wan/Wan2.2.mdx +++ b/docs/cookbook/diffusion/Wan/Wan2.2.mdx @@ -15,6 +15,12 @@ import { Wan22Deployment } from '/src/snippets/diffusion/wan22-deployment.jsx'; Choose the A14B MoE checkpoints for maximum T2V or I2V capacity and the 5B TI2V model for a smaller unified 720p-at-24-fps path. MoE reduces active compute relative to total capacity but does not remove the memory cost of loading expert weights, so hardware selection still matters. + +The Wan2.2 TI2V 5B checkpoint currently has known quality issues when it is used +for image-to-video generation. Use `Wan-AI/Wan2.2-I2V-A14B-Diffusers` when I2V +quality is the priority. + + ## 2. SGLang-diffusion Installation SGLang-diffusion offers multiple installation methods. You can choose the most suitable installation method based on your hardware platform and requirements. @@ -29,13 +35,13 @@ This section provides deployment configurations optimized for different hardware The Wan2.2 series offers models in various sizes, architectures and input types, optimized for different hardware platforms. The recommended launch configurations vary by hardware and model size. -**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size. SGLang supports serving Wan2.2 on NVIDIA B200, H200, AMD MI300X, MI325X, MI355X GPUs and Ascend A2, A3 NPUs. +**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size. SGLang supports serving Wan2.2 on NVIDIA B200, H200, AMD MI300X, MI325X, MI355X GPUs and Ascend A2/A3 Series NPUs. ### 3.2 Configuration Tips -Currently supported optimizations are listed [here](/docs/sglang-diffusion/compatibility_matrix). +See [Performance Optimization](/docs/sglang-diffusion/performance-optimization) for acceleration features and their runtime requirements. - `--vae-path`: Path to a custom VAE model or HuggingFace model ID (e.g., fal/FLUX.2-Tiny-AutoEncoder). If not specified, the VAE will be loaded from the main model path. - `--num-gpus {NUM_GPUS}`: Number of GPUs to use @@ -291,10 +297,10 @@ Test Environment: ``` - + **Server Command**: ```shell Command - #One A3 card has 2 npu chips. Using four A3 cards in benchmarking + #One A3 Series card has 2 npu chips. Using four A3 Series cards in benchmarking sglang serve \ --model-path /models/Wan-AI/Wan2.2-T2V-A14B-Diffusers/ \ --tp-size 2 \ @@ -393,11 +399,11 @@ Test Environment: ``` - + **Server Command**: ```shell Command - #One A3 card has 2 npu chips. Using four A3 cards in benchmarking + #One A3 Series card has 2 npu chips. Using four A3 Series cards in benchmarking SGLANG_CACHE_DIT_FN=2 \ SGLANG_CACHE_DIT_BN=1 \ SGLANG_CACHE_DIT_WARMUP=4 \ @@ -453,3 +459,9 @@ Test Environment: ``` + +## 6. Run in ComfyUI + +import { ComfyUISupport } from '/src/snippets/diffusion/comfyui-support.jsx'; + + diff --git a/docs/cookbook/diffusion/Z-Image/Z-Image-Turbo.mdx b/docs/cookbook/diffusion/Z-Image/Z-Image-Turbo.mdx index 7196cda2df03..b5dc01e4ac1e 100644 --- a/docs/cookbook/diffusion/Z-Image/Z-Image-Turbo.mdx +++ b/docs/cookbook/diffusion/Z-Image/Z-Image-Turbo.mdx @@ -29,13 +29,13 @@ This section provides deployment configurations optimized for different hardware Z-Image-Turbo is optimized for high-quality image generation with only 8 inference steps. The recommended launch configurations vary by hardware. -**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform. +**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and model version. SGLang supports serving Z-Image-Turbo on NVIDIA B200, H200, H100, and AMD MI355X, MI325X, MI300X GPUs, Ascend A2, A3 NPUs and Intel Arc Pro B-Series GPUs(codename: BMG (Battlemage)). ### 3.2 Configuration Tips -Currently supported optimizations are listed [here](/docs/sglang-diffusion/compatibility_matrix). +See [Performance Optimization](/docs/sglang-diffusion/performance-optimization) for acceleration features and their runtime requirements. - `--vae-path`: Path to a custom VAE model or HuggingFace model ID (e.g., fal/FLUX.2-Tiny-AutoEncoder). If not specified, the VAE will be loaded from the main model path. - `--num-gpus`: Number of GPUs to use @@ -189,6 +189,13 @@ sglang serve --model-path Tongyi-MAI/Z-Image-Turbo - `--vae-cpu-offload`: Use CPU offload for VAE. - `--pin-cpu-memory`: Pin memory for CPU offload. Only added as a temp workaround if it throws "CUDA error: invalid argument". +#### 4.2.3 Known LoRA examples + +Use `--lora-path` at startup or the [LoRA management API](/docs/sglang-diffusion/api/openai_api#lora-management) to load an adapter. Known Z-Image-Turbo examples include: + +- [`tarn59/pixel_art_style_lora_z_image_turbo`](https://huggingface.co/tarn59/pixel_art_style_lora_z_image_turbo) +- [`wcde/Z-Image-Turbo-DeJPEG-Lora`](https://huggingface.co/wcde/Z-Image-Turbo-DeJPEG-Lora) + ## 5. Benchmark Test Environment: @@ -242,11 +249,11 @@ Test Environment: ============================================================ ``` - + **Server Command**: ```shell Command - #One A3 card has 2 npu chips + #One A3 Series card has 2 npu chips sglang serve --model-path Tongyi-MAI/Z-Image-Turbo --tp-size 2 --sp-degree 1 --num-gpus 2 ``` @@ -322,7 +329,7 @@ Test Environment: ============================================================ ``` - + **Benchmark Command**: ```shell Command @@ -359,3 +366,9 @@ Test Environment: ``` + +## 6. Run in ComfyUI + +import { ComfyUISupport } from '/src/snippets/diffusion/comfyui-support.jsx'; + + diff --git a/docs/cookbook/diffusion/inclusionAI/Ming-Image.mdx b/docs/cookbook/diffusion/inclusionAI/Ming-Image.mdx new file mode 100644 index 000000000000..216d96af418e --- /dev/null +++ b/docs/cookbook/diffusion/inclusionAI/Ming-Image.mdx @@ -0,0 +1,115 @@ +--- +title: Ming-Image +tag: NEW +description: "Deploy Ming-Image Design and Design-Layer with SGLang Diffusion for text-to-image, RGBA image editing, and ordered transparent layer decomposition." +--- + +import { DiffusionModelTags } from '/src/snippets/diffusion/model-tags.jsx'; +import { Deployment } from '/src/snippets/_deployment.jsx'; +import { config } from '/src/snippets/configs/inclusionAI/ming-image.jsx'; + + + +## 1. Quick start + +Install the dependencies, then this integration from its source checkout: + +```bash Installation +uv pip install "sglang[diffusion]" --prerelease=allow +uv pip install -e "python[diffusion]" --no-deps +``` + +Choose **Design** for generation or editing and **Design-Layer** for decomposition. +The two checkpoints require separate servers. Start with one H200 on Linux/CUDA; +the picker marks unverified hardware and feature combinations explicitly. + + + +Requests return base64 PNGs, preserving all four RGBA channels. For decomposition, +each returned image is one ordered layer, not an independent generation. The +internal composite frame is omitted from the response. Save all entries in +`data`, rather than only the first image. +`n` selects independent generations (or sets of layers), processed sequentially. + +## 2. Model capabilities + +[Ming-Image Design](https://huggingface.co/inclusionAI/Ming-Image-0.1-Design) +generates images from text and edits a single reference image. It targets design +assets, layouts, and transparent RGBA output. Use the prompt prefix +`RGBA, 4-channel, transparent background` to request transparency; RGBA output +alone does not guarantee a transparent background. + +[Design-Layer](https://huggingface.co/inclusionAI/Ming-Image-0.1-Design-Layer) +decomposes one reference into ordered transparent layers. Supply `num_layers`, +or specify a count with `into N layers` or `Number of layers: N` in the prompt; +the prompt count takes precedence. Defaults are 12 steps with guidance 1 for +Design and guidance 2 for Design-Layer. The model uses zero unconditional +embeddings and does not support custom negative prompts. + +Both checkpoints use the native Bailing multimodal MoE encoder, query connector, +joint DiT, and four-channel Qwen-style VAE. No remote modeling code is executed. +The original checkpoint directory layout is accepted without conversion. + +## 3. Offline generation + +```bash Design +sglang generate --model-path inclusionAI/Ming-Image-0.1-Design \ + --prompt "A minimalist exhibition poster with a red geometric sculpture" \ + --height 1024 --width 1024 --save-output +``` + +```bash Layer decomposition +sglang generate --model-path inclusionAI/Ming-Image-0.1-Design-Layer \ + --image-path /path/to/input.png --num-layers 4 --save-output +``` + +Design defaults to 2048 by 2048; 1024 by 1024 reduces latency and activation +memory. Decomposition defaults to the official 1024-resolution aspect buckets; +use `--height 512 --width 512` for the 512 buckets. Reference-image requests +select the closest official bucket and restore the input aspect ratio in the +returned PNGs. Multiple reference images are not supported. + +For a renamed local checkpoint directory, also provide +`--model-id inclusionAI/Ming-Image-0.1-Design` or the corresponding Layer ID. + +## 4. Runtime features + +DiT parallelism reuses the native TP and sequence-parallel attention layers. +The DiT has 30 attention heads: TP multiplied by Ulysses must divide 30. +Encoder folding has its own head constraints; two-way folding is compatible +with the 16-head language tower, 16-head vision tower, and 12-head connector. +Do not infer encoder compatibility from the DiT's head count. + +Use the standard component residency controls for `transformer`, `text_encoder`, +and `vae`. Layerwise offload trades host memory and transfer time for lower VRAM. +The default eager path does not use approximate caching or quantized attention. +Cache-DiT and SageAttention require workload-specific quality validation. +VAE tiling is disabled by default to match the official inference path. Enable +`--vae-tiling true` when decode memory is constrained; tiling can change pixels. + +Breakable CUDA graphs preserve Ming's checkpoint-specific padding and RoPE +offsets. They capture exact prompt shapes; an uncaptured shape runs eagerly. +Requests with different prompts are not merged into a dynamic batch. A fixed +seed does not imply bit-identical output across attention backends, GPU types, +or parallel topologies. + +Full-checkpoint validation covers generation, editing, and layer decomposition +on H200, including repeated HTTP requests before and after warmup. Tested +resolutions include 2048-square generation and four-layer 1024-square +decomposition. See the [compatibility notes](/docs/sglang-diffusion/compatibility_matrix#ming-image-integration-status) +for the tested feature scope. Other GPU families and quantized checkpoints +have not been validated for this model. + +See [performance optimization](/docs/sglang-diffusion/performance-optimization) +for shared runtime controls and the +[official repository](https://github.com/inclusionAI/Ming-Image) for model examples. + +## 5. Run in ComfyUI + +import { ComfyUISupport } from '/src/snippets/diffusion/comfyui-support.jsx'; + + + +Use the generic image node for Design generation. The ComfyUI integration has +not been validated for ordered multi-layer output; use the HTTP API above for +Design-Layer decomposition. diff --git a/docs/cookbook/diffusion/intro.mdx b/docs/cookbook/diffusion/intro.mdx index 37772941be68..48067a89fdba 100644 --- a/docs/cookbook/diffusion/intro.mdx +++ b/docs/cookbook/diffusion/intro.mdx @@ -28,9 +28,15 @@ Image models generate one image request as a bounded denoising job, usually with + + +#### Gemma 4 on `aiter` (ROCm) + +Gemma 4's hybrid attention mixes sliding-window layers (`head_dim` 256) with +full-attention layers (`head_dim` 512) and shares KV across layers. The `aiter` +backend supports this and is allow-listed for Gemma 4, so you can serve it with +`--attention-backend aiter` on MI300X/MI350X. Under the hood `aiter` routes the +`head_dim`-512 prefill through the Triton `unified_attention` kernel (the CK +batch-prefill path caps `head_dim` at 256), honors each layer's softmax scale, +and resolves the sliding-window KV pool for speculative-decode draft workers. +Non–sliding-window models (`head_dim` ≤ 256) are unaffected and keep using the +CK path. + + +For Gemma 4 on `aiter`, use bf16 KV cache (`--kv-cache-dtype auto`). Do **not** +use `--kv-cache-dtype fp8_e4m3`: batched decode currently returns incorrect +output under fp8 KV on this path (a pre-existing aiter fp8-decode issue, tracked +separately). bf16 KV is correct at every batch size. + + For deployment examples, see the [DeepSeek V3.2 deployment guide](/cookbook/autoregressive/DeepSeek/DeepSeek-V3_2). ### Hybrid attention (different backends for prefill vs decode) (Experimental) @@ -664,15 +683,6 @@ python3 -m sglang.launch_server \ --trust-remote-code ``` -- Cutlass MLA -```bash Command -python3 -m sglang.launch_server \ - --tp 8 \ - --model deepseek-ai/DeepSeek-R1 \ - --attention-backend cutlass_mla \ - --trust-remote-code -``` - - Ascend ```bash Command python3 -m sglang.launch_server \ @@ -701,13 +711,6 @@ python3 -m sglang.launch_server \ --attention-backend flex_attention ``` -- Dual Chunk FlashAttention -```bash Command -python3 -m sglang.launch_server \ - --model Qwen/Qwen2.5-14B-Instruct-1M \ - --attention-backend dual_chunk_flash_attn -``` - - Torch Native ```bash Command python3 -m sglang.launch_server \ diff --git a/docs/docs/advanced_features/checkpoint_engine.mdx b/docs/docs/advanced_features/checkpoint_engine.mdx index 59472e6094cc..6c9b1ba8e4b5 100644 --- a/docs/docs/advanced_features/checkpoint_engine.mdx +++ b/docs/docs/advanced_features/checkpoint_engine.mdx @@ -25,7 +25,7 @@ pip install 'checkpoint-engine[p2p]' The system consists of two main components: -1. **SGLang Server**: Runs with `--wait-for-initial-weights` flag to wait for weights before becoming ready +1. **SGLang Server**: Runs with `--checkpoint-engine-wait-weights-before-ready` flag to wait for weights before becoming ready 2. **Checkpoint Engine Workers**: Separate processes (managed by torchrun) that load and distribute model weights The checkpoint engine uses a parameter server architecture with support for: @@ -43,7 +43,7 @@ python -m sglang.launch_server \ --model-path Qwen/Qwen3-8B \ --tp 8 \ --load-format dummy \ - --wait-for-initial-weights + --checkpoint-engine-wait-weights-before-ready ``` **Terminal 2 - Run Checkpoint Engine:** @@ -75,7 +75,7 @@ python -m sglang.launch_server \ --model-path Qwen/Qwen3-8B \ --tp 8 \ --load-format dummy \ - --wait-for-initial-weights \ + --checkpoint-engine-wait-weights-before-ready \ --host [IP] ``` @@ -110,7 +110,7 @@ python -m sglang.launch_server \ --model-path Qwen/Qwen3-8B \ --tp 8 \ --load-format dummy \ - --wait-for-initial-weights \ + --checkpoint-engine-wait-weights-before-ready \ --host [IP] ``` @@ -147,7 +147,7 @@ python -m sglang.launch_server \ --model-path Qwen/Qwen3-8B \ --tp 8 \ --load-format dummy \ - --wait-for-initial-weights \ + --checkpoint-engine-wait-weights-before-ready \ --host [IP] \ --dist-init-addr [IP]:9120 \ --nnodes 2 \ @@ -185,7 +185,7 @@ python -m sglang.launch_server \ --model-path Qwen/Qwen3-8B \ --tp 8 \ --load-format dummy \ - --wait-for-initial-weights \ + --checkpoint-engine-wait-weights-before-ready \ --host [IP] \ --dist-init-addr [IP]:9120 \ --nnodes 2 \ @@ -220,7 +220,7 @@ torchrun --nproc-per-node 8 \ ### SGLang Server Options - `--load-format dummy`: Use dummy format for initial loading (allows overlapping with other tasks) -- `--wait-for-initial-weights`: Wait for checkpoint engine to provide weights before becoming ready +- `--checkpoint-engine-wait-weights-before-ready`: Wait for checkpoint engine to provide weights before becoming ready - `--host`: Host address for multi-node setups - `--dist-init-addr`: Distributed initialization address for tensor parallelism diff --git a/docs/docs/advanced_features/cuda_graph_for_multi_modal_encoder.mdx b/docs/docs/advanced_features/cuda_graph_for_multi_modal_encoder.mdx index 1ee4639792e1..b1c1aed28998 100644 --- a/docs/docs/advanced_features/cuda_graph_for_multi_modal_encoder.mdx +++ b/docs/docs/advanced_features/cuda_graph_for_multi_modal_encoder.mdx @@ -61,14 +61,17 @@ SGLANG_VIT_ENABLE_CUDA_GRAPH=1 \ python3 -m sglang.launch_server \ --model Qwen/Qwen3-VL-8B-Instruct ``` -Or you can run CUDA Graph for ViT together with Piecewise CUDA Graph feature by both setting env variable `SGLANG_VIT_ENABLE_CUDA_GRAPH=1` and setting `--enable-piecewise-cuda-graph`, for example: +To run the ViT graph together with Piecewise CUDA Graph over the decoder's +prefill, ask for the prefill backend explicitly. Qwen3-VL is on neither +multimodal allowlist, so the default backend resolves prefill to `disabled` for +it; naming a backend locks the choice and skips that rule: ```shell Command SGLANG_VIT_ENABLE_CUDA_GRAPH=1 \ python3 -m sglang.launch_server \ --model Qwen/Qwen3-VL-8B-Instruct \ - --piecewise-cuda-graph-max-tokens 4096 \ - --enable-piecewise-cuda-graph \ - --piecewise-cuda-graph-compiler eager + --cuda-graph-backend-prefill tc_piecewise \ + --cuda-graph-max-bs-prefill 4096 \ + --cuda-graph-tc-compiler eager ``` ## Known supported models diff --git a/docs/docs/advanced_features/dcp.mdx b/docs/docs/advanced_features/dcp.mdx index e7bd9ed62a0c..262b69c4c295 100644 --- a/docs/docs/advanced_features/dcp.mdx +++ b/docs/docs/advanced_features/dcp.mdx @@ -15,7 +15,7 @@ On Kimi K3, DCP applies only to MLA layers. Request-indexed KDA state is unchang | Setting | Behavior | | --- | --- | | `--dcp-size N` | Enables DCP and widens virtual capacity and page size by `N`. Alias: `--decode-context-parallel-size`. | -| `--dcp-comm-backend` | Selects the MLA partial-output merge: `ag_rs`, `a2a`, or `fi_a2a`. | +| `--dcp-comm-backend` | Selects the MLA partial-output merge: `ag_rs`, `a2a`, or `fi_a2a`. Unset resolves to `fi_a2a` where the DCP group shares an MNNVL domain on Blackwell, otherwise `a2a` on CUDA/ROCm. | | `--dcp-replicate-q-proj` | Removes the query all-gather for supported MLA weights. Use `--no-dcp-replicate-q-proj` to disable a model-specific default. | | `--enable-dp-attention` | Composes only when DCP groups nest inside attention TP. | @@ -44,7 +44,7 @@ sglang serve \ --port 30000 ``` -Kimi K3 on 32 GPUs with DPA and DCP 8. The model override enables replicated Q by default and picks `fi_a2a` on MNNVL systems, otherwise `a2a`: +Kimi K3 on 32 GPUs with DPA and DCP 8. The model override enables replicated Q by default; the communication backend resolves generically to `fi_a2a` on MNNVL systems, otherwise `a2a`: ```bash Command sglang serve \ @@ -130,12 +130,12 @@ Normally each layer all-gathers its absorbed and rotary query components. `--dcp | Backend | DCP collectives per MLA layer | Notes | | --- | --- | --- | -| `ag_rs` | Query AG + LSE AG + FP32 output RS | Generic fallback | +| `ag_rs` | Query AG + LSE AG + FP32 output RS | Fallback for platforms without the a2a path; opt in with `--dcp-comm-backend ag_rs` | | `a2a`, gathered Q | Query AG + packed NCCL A2A | Two collectives | | `a2a`, replicated Q | One packed NCCL A2A | Canonical low-latency path | -| `fi_a2a`, replicated Q | One FlashInfer MNNVL A2A | Requires CUDA, SM90+, and MNNVL fabric | +| `fi_a2a`, replicated Q | One FlashInfer MNNVL A2A | Requires Blackwell and a DCP group inside one MNNVL domain (GB200/GB300 fabric, or a single node) | -Kimi K3 enables replicated Q by default and picks `fi_a2a` on MNNVL systems, otherwise `a2a`. Its decode backend is `cutedsl_mla`. The generic defaults remain `ag_rs` and model-resolved query replication. +When `--dcp-comm-backend` is unset, every DCP launch resolves it the same way: `fi_a2a` where the DCP group shares one MNNVL domain on Blackwell, otherwise `a2a` on CUDA/ROCm, and `ag_rs` only on platforms without the a2a path. Query replication stays model-resolved: Kimi K3 enables it by default on the a2a family, and its decode backend is `cutedsl_mla`. DCP adds a context-independent decode collective while cutting context-dependent KV storage and reads by about `c`. Extend is outside that decode cost model: it may gather cached prefix shards, restore token order, and append new tokens, and that work grows with prefix length. diff --git a/docs/docs/advanced_features/hicache_best_practices.mdx b/docs/docs/advanced_features/hicache_best_practices.mdx index df91d0aacd5b..91980e827d40 100644 --- a/docs/docs/advanced_features/hicache_best_practices.mdx +++ b/docs/docs/advanced_features/hicache_best_practices.mdx @@ -7,6 +7,10 @@ metatags: SGLang HiCache extends the traditional RadixAttention with a three-tier hierarchical KV caching system that dramatically improves performance for long-context and multi-turn conversation scenarios. By intelligently managing KV caches across GPU memory, host memory, and external storage backends, HiCache addresses the fundamental capacity bottleneck that limits cache hit rates in conventional systems. + +L1 and L2 are private to a single inference instance; only L3 can be shared. Host memory cannot be pooled across instances or across hosts, not even for two instances on the same node. Raising `--hicache-ratio` or `--hicache-size` only enlarges the instance's own private L2. Cross-instance reuse is the job of L3, which needs `--hicache-storage-backend` and a backend configured for the scope you want: the `file` backend defaults to the node-local `/tmp/hicache`, whereas `mooncake`, `hf3fs`, `nixl` and `aibrix` reach cluster scope when every instance shares the same namespace. See [Tier Sharing Scope](/docs/advanced_features/hicache_design#tier-sharing-scope). + + ## Configuration Guidelines ## Core HiCache Parameters diff --git a/docs/docs/advanced_features/hicache_design.mdx b/docs/docs/advanced_features/hicache_design.mdx index 0265ea02906e..9654c33b81b3 100644 --- a/docs/docs/advanced_features/hicache_design.mdx +++ b/docs/docs/advanced_features/hicache_design.mdx @@ -18,6 +18,52 @@ Inspired by the classic three-level cache design of modern CPUs, HiCache organiz In many modern CPU architectures, the small but fast L1 and L2 caches are private to each core, enabling rapid access to the hottest data, while the larger L3 cache is shared across all cores to significantly reduce redundancy within the cache. Similarly, in HiCache, the L1 and L2 KV caches are private to each inference instance, whereas the L3 KV cache is shared among all inference instances within the cluster. +### Tier Sharing Scope + +Which tier is private and which is shared decides where a cache hit can come from, so it is worth stating explicitly: + +| Tier | Medium | Scope | Shared across instances? | +|---|---|---|---| +| **L1** | GPU memory (HBM) | One inference instance | No | +| **L2** | Host memory (CPU DRAM) | One inference instance, on its own node | **No** | +| **L3** | Storage backend (`file`, `mooncake`, `hf3fs`, `nixl`, `aibrix`, or a custom one) | Whatever the backend is configured to span | **Only if the backend is set up for it** | + +```mermaid +flowchart TB + subgraph N0["Node A"] + subgraph I0["Instance 0"] + L1A["L1 - GPU memory
private"] + L2A["L2 - host memory
private to this instance"] + L1A --- L2A + end + subgraph I1["Instance 1"] + L1B["L1 - GPU memory
private"] + L2B["L2 - host memory
private to this instance"] + L1B --- L2B + end + end + + subgraph N1["Node B"] + subgraph I2["Instance 2"] + L1C["L1 - GPU memory
private"] + L2C["L2 - host memory
private to this instance"] + L1C --- L2C + end + end + + L3["L3 - storage backend
shared by the instances the backend is configured to span"] + + L2A --> L3 + L2B --> L3 + L2C --> L3 +``` + +**L2 is node-local and instance-private.** It is host memory owned by one inference instance's process, so two instances never read each other's L2, not even two instances on the same node. A KV cache produced by instance 0 becomes visible to instance 1 only after it reaches L3. + +This is the answer to a common question: HiCache cannot pool the host memory of several machines into one larger L2. Growing `--hicache-ratio` or `--hicache-size` only makes each instance's own private L2 larger. Cross-instance reuse is the job of L3, so it needs `--hicache-storage-backend`. + +L3 is the tier that *can* be shared, but whether it actually is depends on the backend and how it is configured. The `file` backend writes to `/tmp/hicache` by default (`SGLANG_HICACHE_FILE_BACKEND_STORAGE_DIR` overrides it), which is node-local unless that path is a shared mount. Distributed backends such as `mooncake`, `hf3fs`, `nixl` and `aibrix` give cluster-wide scope, but only when every instance is pointed at the same namespace and configuration. + ### HiRadixTree: Metadata Organization in HiCache For KV cache data organization, HiCache builds upon the RadixTree structure introduced in RadixAttention and proposes HiRadixTree. In RadixAttention, each node of the RadixTree corresponds to the KV cache of a consecutive span of tokens in GPU memory. A path from the root to a leaf node represents the prefix of a request, and shared prefixes across multiple requests can reuse the same nodes, thereby avoiding redundant storage. diff --git a/docs/docs/advanced_features/hicache_storage_runtime_attach_detach.mdx b/docs/docs/advanced_features/hicache_storage_runtime_attach_detach.mdx index 1f36e067764d..54da9d41813c 100644 --- a/docs/docs/advanced_features/hicache_storage_runtime_attach_detach.mdx +++ b/docs/docs/advanced_features/hicache_storage_runtime_attach_detach.mdx @@ -26,7 +26,7 @@ The control path is: 3. **Scheduler** (`python/sglang/srt/managers/scheduler.py`) - Performs a **strict idle check** - Calls `tree_cache.attach_storage_backend(...)` / `detach_storage_backend(...)` -4. **HiRadixCache** (`python/sglang/srt/mem_cache/hiradix_cache.py`) +4. **UnifiedRadixCache** (`python/sglang/srt/mem_cache/unified_radix_cache.py`) - Parses `hicache_storage_backend_extra_config_json` (supports both backend config and prefetch knobs) - Calls `cache_controller.attach_storage_backend(...)` / `detach_storage_backend(...)` 5. **HiCacheController** (`python/sglang/srt/managers/cache_controller.py`) @@ -130,4 +130,4 @@ Notes: - **Host KV layout constraints still apply**: for example, Mooncake still requires layouts like `page_first/page_first_direct/page_head`; if the server's HiCache host-memory layout does not satisfy the backend requirements, attach will fail with an error - **Observability**: - After attach, `server_args.hicache_storage_backend*` is updated on both the tokenizer and scheduler sides - - If metrics are enabled, attach will create a storage metrics collector in `HiRadixCache` on demand + - If metrics are enabled, attach will create a storage metrics collector in `UnifiedRadixCache` on demand diff --git a/docs/docs/advanced_features/hisparse_guide.mdx b/docs/docs/advanced_features/hisparse_guide.mdx index 294ecba6dd5b..54c412bafd23 100644 --- a/docs/docs/advanced_features/hisparse_guide.mdx +++ b/docs/docs/advanced_features/hisparse_guide.mdx @@ -6,7 +6,7 @@ metatags: HiSparse reduces per-request GPU memory consumption during the decode phase by maintaining only a small "hot" KV buffer on GPU while keeping complete KV data in CPU pinned memory. Combined with PD disaggregation, it enables significantly higher decode concurrency. -> **Prerequisites**: HiSparse works with models that use **DeepSeek Sparse Attention (DSA)** architectures (e.g., DeepSeek-V3.2, GLM-5.1) and **DeepSeek V4**. These models natively select a subset of tokens for attention, making it possible to keep only the top-k KV on GPU while storing the full KV in host memory — without accuracy loss. Additionally, HiSparse currently requires **PD disaggregation mode** and is enabled on the **decode instance** only. +> **Prerequisites**: HiSparse works with models that use **DeepSeek Sparse Attention (DSA)** architectures (e.g., DeepSeek-V3.2, GLM-5.1), **DeepSeek V4**, and **MiniMax M3**. These models natively select a subset of tokens for attention, making it possible to keep only the top-k KV on GPU while storing the full KV in host memory — without accuracy loss. Additionally, HiSparse currently requires **PD disaggregation mode** and is enabled on the **decode instance** only. ## Why HiSparse? @@ -165,6 +165,15 @@ python3 -m sglang.launch_server \ > **Note**: For DSA models, `--kv-cache-dtype` defaults to `auto`, which resolves to `fp8_e4m3` on SM100+ (Blackwell) and `bfloat16` on older architectures. The DSA decode backend is automatically selected based on KV dtype (`bfloat16` → `flashmla_sparse`, `fp8_e4m3` → `flashmla_kv`), except for GLM DSA models on SM120/SM121 with `fp8_e4m3`, which use `flashinfer_sparse_mla`. DSA backend flags apply only to DSA models; DeepSeek V4 uses its own `dsv4` attention backend. +### MiniMax M3 + +Dense-layer K/V and index K stay on GPU; sparse-layer K/V use host memory plus a GPU working set. + +- Use TP 4 or greater, with the same TP size and PP 1 on both PD instances. +- Use `--attention-backend triton`, `--mm-attention-backend triton_attn`, `--disable-prefill-cuda-graph`, and `--disable-radix-cache`. +- Set `device_buffer_size` to at least 2048 in `--hisparse-config`; `top_k` does not override the model's selection width. +- PD retraction backup is not supported. Use `--num-reserved-decode-tokens` to reserve capacity for the expected output length. + ### Benchmark ```bash Command diff --git a/docs/docs/advanced_features/overview.mdx b/docs/docs/advanced_features/overview.mdx index 2a2dbe5f8337..a7773d97c763 100644 --- a/docs/docs/advanced_features/overview.mdx +++ b/docs/docs/advanced_features/overview.mdx @@ -16,5 +16,6 @@ description: Advanced configuration, optimization, and deployment features for S - [PD Disaggregation](./pd_disaggregation) - [Pipeline Parallelism](./pipeline_parallelism) - [HiCache](./hicache_best_practices) +- [SGLang Simulator](./sglang_simulator) - [Observability](./observability) - [And more…](./server_arguments) diff --git a/docs/docs/advanced_features/pd_disaggregation.mdx b/docs/docs/advanced_features/pd_disaggregation.mdx index d2a781042f91..7fd878bc94c6 100644 --- a/docs/docs/advanced_features/pd_disaggregation.mdx +++ b/docs/docs/advanced_features/pd_disaggregation.mdx @@ -28,6 +28,25 @@ When you need to profile prefill or decode workers in PD disaggregation mode, pl For deploying PD disaggregation at scale with load balancing and fault tolerance, SGLang provides a router. The router can distribute requests between prefill and decode instances using various routing policies. For detailed information on setting up routing with PD disaggregation, including configuration options and deployment patterns, see the [SGLang Model Gateway (former Router)](./sgl_model_gateway#prefill-decode-disaggregation). +### Responses API under HTTP PD + +HTTP PD supports foreground generation through `POST /v1/responses`, including streaming generation with `stream=true` through the HTTP PD router. This requires both the router's Responses dispatch support and serving workers that accept Responses PD routing metadata. + +Store-backed and stateful Responses features are **not supported under PD**: + +- Response persistence (`store`) and response history. +- Conversation chaining with `previous_response_id`. +- Background Responses (`background=true`), including background streams. +- Retrieval by response ID (`GET /v1/responses/{id}`). +- Cancel-by-ID (`POST /v1/responses/{id}/cancel`). +- Other workflows requiring shared process-local Responses history. + +Prefill and decode run in separate serving processes. Their process-local Responses stores cannot provide coherent shared state: both legs need the same conversation history to reconstruct the prompt, and prefill's stored response is not available to decode. Send explicit conversation history in foreground requests instead of relying on stored response IDs. + +Built-in tools (`web_search`, `code_interpreter`) are also unsupported under PD: each tool call requires another generation, but the router dispatches exactly one prefill/decode pair per HTTP request. + +Serving-side capability enforcement is tracked in [#39122](https://github.com/sgl-project/sglang/pull/39122). That change introduces opt-in standalone storage with `--enable-response-store`, disallows it under PD, and rejects stateful requests with HTTP 400 before generation. These checks require a serving version containing that change; older versions must not be assumed to reject every unsupported workflow at admission. The HTTP PD router also rejects detached background requests because it does not implement their retrieval/cancel lifecycle. + ## Mooncake ### Requirements @@ -36,6 +55,39 @@ For deploying PD disaggregation at scale with load balancing and fault tolerance uv pip install mooncake-transfer-engine ``` +### AWS EFA + +On an x86_64 AWS EFA instance, replace the standard CUDA 13 Mooncake package with the EFA build. The two packages install the same `mooncake` Python module, so do not install them side by side. + +The following commands replace the package in an SGLang CUDA 13 environment while preserving its existing Python dependencies: + +```bash +python3 -m pip uninstall -y \ + mooncake-transfer-engine-cuda13 \ + mooncake-transfer-engine-efa-cuda13 +python3 -m pip install --no-deps \ + mooncake-transfer-engine-efa-cuda13==0.3.13.post1 + +export LD_LIBRARY_PATH="/opt/amazon/efa/lib:/opt/amazon/efa/lib64:${LD_LIBRARY_PATH:-}" +export MOONCAKE_PROTOCOL=efa + +python3 -c 'from mooncake import engine; assert engine.SUPPORT_EFA' +fi_info -p efa -t FI_EP_RDM +``` + +The EFA wheel expects a compatible AWS EFA userspace installation on the host and does not bundle `libfabric` or `libefa`. The current wheel is available only for x86_64. It is built without Mooncake EP/PG and the NVLink/MNNVL components, so use the standard Mooncake CUDA package for those features. + +The current libfabric EFA provider cannot transfer CUDA VMM-backed buffers. Until [libfabric issue #12853](https://github.com/ofiwg/libfabric/issues/12853) is resolved, apply these settings to both prefill and decode workers: + +- Do not set `SGLANG_MOONCAKE_CUSTOM_MEM_POOL` to `NVLINK` or `BAREX`. + `INTRA_NODE_NVLINK` is supported because it uses the default CUDA allocator. +- Do not enable `expandable_segments:True` in `PYTORCH_CUDA_ALLOC_CONF` or `PYTORCH_ALLOC_CONF`. +- Post-capture KV sizing is automatically disabled for the EFA protocol because + its KV cache uses CUDA VMM allocations. +- Do not set `MC_FORCE_MNNVL` or `NCCL_MNNVL_ENABLE`; EFA is not an MNNVL transport. + +SGLang validates the first two constraints at startup to avoid a later `fi_read` or `fi_write` failure from libfabric. + ### IB Device Configuration `--disaggregation-ib-device` supports the following formats when using the Mooncake backend: @@ -180,6 +232,11 @@ The `SGLANG_MOONCAKE_CUSTOM_MEM_POOL` environment variable enables the custom me Sets the number of parallel transfer queues. KVCache transfer requests from multiple decode instances will be sharded into these queues so that they can share the threads and the transfer bandwidth at the same time. If it is set to 1, then we transfer requests one by one according to fcfs strategy `4` + + **`SGLANG_MOONCAKE_MAX_TRANSFER_BATCH_INDICES`** + Opt-in limit for the number of KV cache indices represented by one synchronous all-layer Mooncake transfer batch. Set it to a positive value to slice larger index arrays into ordered sub-batches before contiguous address ranges are formed. The custom-memory-pool layerwise path is unchanged. + 0 (disabled) + **`SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT`** Timeout (seconds) for receiving destination KV indices during request initialization @@ -563,7 +620,7 @@ python3 -m sglang.launch_server \ --disaggregation-bootstrap-port 8996 \ --base-gpu-id 8 \ --disable-radix-cache \ - --cuda-graph-bs 1 2 4 8 10 12 14 16 \ + --cuda-graph-bs-decode 1 2 4 8 10 12 14 16 \ --speculative-draft-model-quantization unquant \ --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \ --enable-multi-layer-eagle \ @@ -676,9 +733,8 @@ do --port 8001 --trust-remote-code --nnodes 1 --node-rank 0 --tp-size 16 --dp-size 16 \ --mem-fraction-static 0.8 --max-running-requests 448 --attention-backend ascend --device npu --quantization modelslim \ --moe-a2a-backend deepep --enable-dp-attention --deepep-mode low_latency --enable-dp-lm-head \ - --cuda-graph-bs 2 4 6 8 10 12 14 16 18 20 22 24 26 28 --disaggregation-transfer-backend ascend --watchdog-timeout 9000 --context-length 8192 \ - --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \ - --prefill-round-robin-balance --disable-shared-experts-fusion --dtype bfloat16 --tokenizer-worker-num 4 \ + --cuda-graph-bs-decode 2 4 6 8 10 12 14 16 18 20 22 24 26 28 --disaggregation-transfer-backend ascend --watchdog-timeout 9000 --context-length 8192 \ + --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --disable-shared-experts-fusion --dtype bfloat16 --tokenizer-worker-num 4 \ --load-balance-method round_robin NODE_RANK=$i break diff --git a/docs/docs/advanced_features/piecewise_cuda_graph.mdx b/docs/docs/advanced_features/piecewise_cuda_graph.mdx index d1c3e7dd8761..4f03665e824c 100644 --- a/docs/docs/advanced_features/piecewise_cuda_graph.mdx +++ b/docs/docs/advanced_features/piecewise_cuda_graph.mdx @@ -10,7 +10,7 @@ Standard CUDA graphs capture the entire model forward pass as a single graph. Th Piecewise CUDA Graph (PCG) solves this by splitting the model's computation graph into pieces (roughly one per layer) at "split points" (e.g., MoE dispatch ops). Each piece is captured as a separate CUDA graph for a set of pre-defined token lengths. At runtime, the input is padded to the nearest captured size, and each piece is replayed. This eliminates kernel launch overhead for prefill/extend while still supporting dynamic shapes. -Recently we **enabled PCG by default**, which means that the old `--enable-piecewise-cuda-graph` flag is deprecated. Use `--disable-piecewise-cuda-graph` to turn it off. +PCG is **enabled by default**. Pass `--cuda-graph-backend-prefill=disabled` to turn it off. ## Usage @@ -26,7 +26,7 @@ python3 -m sglang.launch_server \ ```bash python3 -m sglang.launch_server \ --model-path meta-llama/Llama-3.1-8B-Instruct \ - --disable-piecewise-cuda-graph + --cuda-graph-backend-prefill=disabled ``` ### Custom capture sizes @@ -34,7 +34,7 @@ python3 -m sglang.launch_server \ ```bash python3 -m sglang.launch_server \ --model-path meta-llama/Llama-3.1-8B-Instruct \ - --piecewise-cuda-graph-max-tokens 2048 + --cuda-graph-max-bs-prefill 2048 ``` ### Server Args @@ -54,41 +54,31 @@ python3 -m sglang.launch_server \ - --disable-piecewise-cuda-graph - False - Disable PCG for extend/prefill. - - - --enforce-piecewise-cuda-graph - False - Force-enable PCG, skipping all auto-disable conditions. For testing only. + --cuda-graph-backend-prefill + None (auto) + Backend for the prefill phase. Choices: full, breakable, tc_piecewise, disabled. Pass disabled to turn PCG off for extend/prefill, or tc_piecewise to force it on, skipping all auto-disable conditions (testing only). - --piecewise-cuda-graph-max-tokens + --cuda-graph-max-bs-prefill None (auto) Maximum token count to capture. Defaults to chunked_prefill_size (non-MLA) or 2048 (MLA). - --piecewise-cuda-graph-tokens + --cuda-graph-bs-prefill None (auto) Explicit list of token lengths to capture. Auto-generated if not set. - --piecewise-cuda-graph-compiler + --cuda-graph-tc-compiler "eager" Compiler backend for the captured subgraphs. Choices: eager, inductor. - - --enable-piecewise-cuda-graph - — - Deprecated. PCG is now enabled by default. Use --enforce-piecewise-cuda-graph to skip auto-disable conditions. - ## Bug Report -PCG is enabled by default but is still in an experimental stage. Since PCG relies on `torch.compile` to trace the model's forward pass, most bugs are introduced by torch compile tracing failures (e.g., untraceable ops, dynamic control flow, or graph breaks). If you encounter any issues related to PCG, please disable it by adding `--disable-piecewise-cuda-graph` to your launch command and report the bug at [GitHub Issues](https://github.com/sgl-project/sglang/issues/new/choose). We greatly appreciate your help in improving this feature. +PCG is enabled by default but is still in an experimental stage. Since PCG relies on `torch.compile` to trace the model's forward pass, most bugs are introduced by torch compile tracing failures (e.g., untraceable ops, dynamic control flow, or graph breaks). If you encounter any issues related to PCG, please disable it by adding `--cuda-graph-backend-prefill=disabled` to your launch command and report the bug at [GitHub Issues](https://github.com/sgl-project/sglang/issues/new/choose). We greatly appreciate your help in improving this feature. ### For Users @@ -96,11 +86,11 @@ If you see an error message like the following during server startup, it is a PC ``` Piecewise CUDA Graph is enabled by default as an experimental feature. -To work around this error, add --disable-piecewise-cuda-graph to your launch command. +To work around this error, add --cuda-graph-backend-prefill=disabled to your launch command. Please report this issue at https://github.com/sgl-project/sglang/issues/new/choose ``` -To work around it, add `--disable-piecewise-cuda-graph` to your launch command. When filing a bug report, please include: +To work around it, add `--cuda-graph-backend-prefill=disabled` to your launch command. When filing a bug report, please include: 1. The full error traceback 2. Model name and quantization method 3. Launch command with all arguments @@ -228,7 +218,7 @@ The default capture schedule is auto-generated with increasing granularity: -For the auto-generated schedule, sizes are capped at `--piecewise-cuda-graph-max-tokens`. The default cap is `chunked_prefill_size` for non-MLA models and `2048` for MLA backend models. If `--max-total-tokens` is set, the cap is further limited to not exceed it. Additionally, Llama-2 models are auto-capped at 4096 tokens as a temporary workaround. +For the auto-generated schedule, sizes are capped at `--cuda-graph-max-bs-prefill`. The default cap is `chunked_prefill_size` for non-MLA models and `2048` for MLA backend models. If `--max-total-tokens` is set, the cap is further limited to not exceed it. Additionally, Llama-2 models are auto-capped at 4096 tokens as a temporary workaround. ## Compatibility @@ -247,7 +237,7 @@ PCG is auto-disabled in the following scenarios. We are actively working on expa - PD disaggregation - Expert distribution recorder / EPLB -Use `--enforce-piecewise-cuda-graph` to skip all auto-disable checks (for testing/debugging only). +Use `--cuda-graph-backend-prefill=tc_piecewise` to skip all auto-disable checks (for testing/debugging only). ## Code Reference diff --git a/docs/docs/advanced_features/quantization.mdx b/docs/docs/advanced_features/quantization.mdx index cd32e5055175..b56d155077c5 100644 --- a/docs/docs/advanced_features/quantization.mdx +++ b/docs/docs/advanced_features/quantization.mdx @@ -30,7 +30,7 @@ The following table summarizes quantization method support across NVIDIA and AMD Method NVIDIA GPUs AMD GPUs (MI300X/MI325X/MI350X) - Ascend NPUs (A2/A3/A5) + Ascend NPUs (A2/A3 Series, 950PR/DT Series) Notes @@ -46,22 +46,22 @@ The following table summarizes quantization method support across NVIDIA and AMD mxfp4 Yes Yes - Yes (A5) - On GPU: requires CDNA3/CDNA4 with MXFP support (uses Aiter). On Ascend NPU (A5): W4A4 MXFP4 for Qwen3 dense and MoE LLMs (MXFP4 weights + activations) — dense models support online dual-level MXFP4; offline W4A4_MXFP4 dense and MoE checkpoints (single-level) are auto-detected via modelslim + Yes (950PR/DT Series) + On GPU: requires CDNA3/CDNA4 with MXFP support (uses Aiter). On Ascend NPU (950PR/DT Series): W4A4 MXFP4 for Qwen3 dense and MoE LLMs (MXFP4 weights + activations) — dense models support online dual-level MXFP4; offline W4A4_MXFP4 dense and MoE checkpoints (single-level) are auto-detected via modelslim. On Intel GPUs (XPU): W4A16 MoE experts on Xe2/BMG via sgl-kernel-xpu, enabled automatically with --device xpu (see Intel GPUs (XPU)) mxfp8 No No - Yes (A5 for Diffusion, LLM Dense Linear and LLM MoE) - Ascend NPU only; online + offline MXFP8 for Diffusion models (e.g., Wan2.2), LLM Dense Linear, and LLM MoE (FusedMoE, e.g. Qwen3-30B-A3B) on A5 series; uses CANN npu_dynamic_mx_quant / npu_quant_matmul (dense) and npu_grouped_matmul_swiglu_quant_v2 / npu_grouped_matmul (MoE) kernels + Yes (950PR/DT Series for Diffusion, LLM Dense Linear and LLM MoE) + Ascend NPU only; online + offline MXFP8 for Diffusion models (e.g., Wan2.2), LLM Dense Linear, and LLM MoE (FusedMoE, e.g. Qwen3-30B-A3B) on 950PR/DT Series; uses CANN npu_dynamic_mx_quant / npu_quant_matmul (dense) and npu_grouped_matmul_swiglu_quant_v2 / npu_grouped_matmul (MoE) kernels mxfp_w4a8 No No - Yes (A5) - Ascend NPU only; online W4A8 for Qwen3 dense LLM (MXFP4 weights + MXFP8 activations) on A5 series; offline W4A8_MXFP dense and MoE checkpoints are auto-detected via modelslim + Yes (950PR/DT Series) + Ascend NPU only; online W4A8 for Qwen3 dense LLM (MXFP4 weights + MXFP8 activations) on 950PR/DT Series; offline W4A8_MXFP dense and MoE checkpoints are auto-detected via modelslim blockwise_int8 @@ -89,14 +89,14 @@ The following table summarizes quantization method support across NVIDIA and AMD Yes Yes Yes - Uses Triton dequantize on AMD (vs. optimized CUDA kernels on NVIDIA). Uses CANN kernels on Ascend + Uses a JIT-compiled CUDA kernel on NVIDIA, Triton dequantize on AMD. Uses CANN kernels on Ascend gptq + No + No Yes - Yes - Yes - Uses Triton or vLLM kernels on AMD. Uses CANN kernels on Ascend + Removed on NVIDIA and AMD GPUs — use gptq_marlin instead. Uses CANN kernels on Ascend. Still supported on Intel CPUs with AMX compressed-tensors @@ -923,9 +923,9 @@ SGLang supports quantization methods based on the advanced algorithm [auto-round ## Diffusion Model Quantization on Ascend NPU -SGLang-Diffusion supports MXFP8 quantization for diffusion models (such as Wan2.2) on Ascend A5 NPUs, in both online and offline (ModelSlim) modes. This is separate from the LLM serving path and uses the `sglang serve` / `sglang generate` CLI. +SGLang-Diffusion supports MXFP8 quantization for diffusion models (such as Wan2.2) on Ascend 950PR/DT Series NPUs, in both online and offline (ModelSlim) modes. This is separate from the LLM serving path and uses the `sglang serve` / `sglang generate` CLI. -**Requirements:** Ascend A5, CANN ≥ 8.0.RC3 +**Requirements:** Ascend 950PR/DT Series, CANN ≥ 8.0.RC3 ### Online MXFP8 diff --git a/docs/docs/advanced_features/quantized_kv_cache.mdx b/docs/docs/advanced_features/quantized_kv_cache.mdx index a8b5d82f676a..b4fc132b2c8c 100644 --- a/docs/docs/advanced_features/quantized_kv_cache.mdx +++ b/docs/docs/advanced_features/quantized_kv_cache.mdx @@ -30,9 +30,10 @@ SGLang supports the following quantized KV cache formats: FP4 quantization is currently experimental. -[OCP (Open Compute Project)](https://www.opencompute.org) specifies MXFP4 (Microscaling FP4), a 4-bit floating-point format: +[OCP (Open Compute Project)](https://www.opencompute.org) specifies MXFP4 (Microscaling FP4), a 4-bit floating-point format. SGLang exposes two experimental E2M1 KV-cache recipes: -- **E2M1** (1 sign bit, 2 exponent bits, 1 mantissa bit): Uses block-based microscaling where tensors are divided into blocks of consecutive elements, with each block sharing a single 8-bit exponential scaling factor. While OCP specifies blocks of 32 elements, SGLang's current implementation uses blocks of 16 elements for KV cache quantization. +- **`nvfp4`**: NVIDIA FP4 with 16-value blocks, E4M3 block scales, and a per-tensor global scale. +- **`fp4_mx_block16`**: An E2M1 block-size-16 compatibility recipe. It is distinct from the standard OCP MXFP4 block-size-32 format. ## Usage @@ -62,6 +63,38 @@ python3 -m sglang.launch_server \ --kv-cache-dtype fp4_mx_block16 \ ``` +### SM100 native NVFP4 recipes + +On SM100, prefill can either consume packed NVFP4 directly or dequantize it into an FP8 E4M3 workspace. Select the online dequantization dtype with `--prefill-kv-cache-dequant-dtype`; SGLang chooses the corresponding attention implementation. + +For native NVFP4 prefill and decode: + +```bash Command +python3 -m sglang.launch_server \ + --model-path Qwen/Qwen3.5-35B-A3B-FP8 \ + --tp-size 1 \ + --kv-cache-dtype nvfp4 \ + --prefill-kv-cache-dequant-dtype nvfp4 \ + --page-size 16 +``` + +For FP8 prefill backed by a temporary dequantization workspace, with native NVFP4 decode: + +```bash Command +python3 -m sglang.launch_server \ + --model-path Qwen/Qwen3.5-35B-A3B-FP8 \ + --tp-size 1 \ + --kv-cache-dtype nvfp4 \ + --prefill-kv-cache-dequant-dtype fp8_e4m3 \ + --page-size 16 +``` + +`nvfp4` means that prefill consumes the packed native NVFP4 cache directly without any additional dequantization. The default value, `auto`, selects this native NVFP4 mode on SM100 and FP8 E4M3 dequantization on SM120. Decode consumes native NVFP4 in both recipes. The native recipe avoids the FP8 workspace and its token-linear scale copy; the FP8 recipe retains both the linear scales used during prefill and the physical scale layout used during decode. KV data remains stored as packed FP4 either way. + + +Native NVFP4 prefill requires SM100, a page size divisible by 4, and an attention head dimension divisible by 64. TRT-LLM GenMHA uses FP8 query and output buffers internally; SGLang converts the result back to the model activation dtype. Top-k-1 EAGLE/EAGLE3/NEXTN and breadth-1 NGRAM speculative decoding are supported: an EAGLE-family draft worker uses `trtllm_mha`, and target verification consumes the physical NVFP4 cache directly in eager execution and CUDA Graphs. Set `--speculative-ngram-max-bfs-breadth=1` for NGRAM. With the mixed FlashInfer-prefill/TRT-LLM-decode recipe, SGLang resolves `--speculative-attention-mode` to `decode` so verification does not depend on FlashInfer's transient dequantization workspace. Other speculative algorithms, PD disaggregation, hierarchical KV cache, and LMCache are not currently supported by the SM100 native NVFP4 path. SM120 XQA continues to use its architecture-specific linear scale layout and BF16 query/output path. + + ### Scaling Factors FP8 quantization requires scaling factors to properly quantize and dequantize the KV cache. @@ -114,6 +147,8 @@ FP4 and FP8 quantization require additional memory for block-based scaling facto This enables longer context lengths or more concurrent requests within the same memory budget. +For native `nvfp4`, each logical scalar costs 0.5 bytes of packed FP4 data plus 1/16 byte of block-scale storage, compared with 2 bytes for BF16. The resulting theoretical KV-token capacity multiplier is `2 / 0.5625 = 3.56×`. The mixed SM100 recipe owns a second scale layout and a shared one-layer FP8 prefill workspace, so its exact capacity depends on the number of full-attention layers. SGLang includes those auxiliary buffers in its KV-pool sizing calculation. + ### Accuracy Impact #### FP8 Accuracy diff --git a/docs/docs/advanced_features/radix_eviction_policy.mdx b/docs/docs/advanced_features/radix_eviction_policy.mdx new file mode 100644 index 000000000000..787430be3faf --- /dev/null +++ b/docs/docs/advanced_features/radix_eviction_policy.mdx @@ -0,0 +1,91 @@ +--- +title: "Radix Cache Eviction Policies" +metatags: + description: "Choose how the radix cache picks which prefix KV to drop under memory pressure, and tune a policy with --radix-eviction-policy-config." +--- + +When the KV cache pool is full, the radix cache reclaims space by evicting cached prefixes. The eviction policy decides which prefix goes first. The default, `lru`, is the right choice for most workloads; the others trade recency for hit frequency, reuse history, or request priority. + +## How eviction picks a victim + +Eviction only ever considers **evictable leaves**: nodes whose KV is present, unlocked (no in-flight request holds them), and not shadowed by a child that still holds KV. The root is never evictable. + +The policy scores each candidate and the **lowest score is evicted first**. Once a leaf is evicted, its parent may become a leaf and re-enter the candidate set, so eviction walks a branch from its tip toward the root. + +A policy only scores. It does not decide how much to free, and it cannot pin KV in memory — a node protected by the policy is still evicted if reclaiming everything else is not enough. + +## Available policies + +Select one with `--radix-eviction-policy`. All of them fall back to least-recently-used order within a tie. + +| Policy | Evicts first | Use when | +|---|---|---| +| `lru` (default) | The prefix unused for longest. | General serving. Matches how prefix reuse decays with time. | +| `lfu` | The prefix with the fewest cache hits, then the least recent. | A small set of prompts is reused far more than the rest, and you want those to survive bursts of one-off traffic. | +| `slru` | Prefixes still in the probationary segment, then the least recent within a segment. | Like `lfu`, but you want a hard floor on how much a one-off prefix can displace a proven one. See [`slru` parameters](#slru-parameters). | +| `priority` | The prefix belonging to the lowest-priority request, then the least recent. | You run [priority scheduling](/docs/advanced_features/server_arguments) and want cache retention to follow the same ranking as admission. | +| `tlru` | The tail of a conversation beyond what its next prefill needs to meet the TTFT budget ("TEL-safe" tokens), then the least recent. | Agentic / multi-turn workloads where tail TTFT matters more than raw hit rate. See [`tlru` parameters](#tlru-parameters). Requires the unified radix cache (the default tree). | + +Notes on the scoring inputs: + +- **Hit count** (`lfu`, `slru`) counts how many times a node was matched by a later request. It is not incremented for chunked-prefill steps, for evicted nodes, or under write-back HiCache. +- **Request priority** (`priority`) is the `priority` field of the request that inserted the prefix; a node reached by several requests keeps the highest priority among them. Without priority scheduling every node is priority `0` and this policy is equivalent to `lru`. + +`fifo`, `mru`, and `filo` also exist in the policy registry but are not offered on the command line. They are reachable only by out-of-tree code that extends the choice list, and exist for experiments rather than serving. + +## Tuning a policy + +Some policies take parameters. Pass them as a json object to `--radix-eviction-policy-config`; the keys are the policy's own, so they are only valid for the policy you selected. + +```bash Command +python3 -m sglang.launch_server \ + --model-path MODEL_PATH \ + --radix-eviction-policy slru \ + --radix-eviction-policy-config '{"protected_threshold": 4}' +``` + +Omit the flag to accept every default. An unrecognized key fails at startup rather than being ignored: + +``` +TypeError: SLRUStrategy.__init__() got an unexpected keyword argument 'protected_treshold' +``` + + +`--radix-eviction-policy-config` is not supported by the experimental Rust tree core (`SGLANG_UNIFIED_RADIX_TREE_CORE_BACKEND=rust`), which builds its strategy from the policy name alone. Passing both fails at startup. + + +### Policy parameters + +`slru` and `tlru` take parameters. `lru`, `lfu`, and `priority` take none, so `--radix-eviction-policy-config` has no effect with them and any key is an error. + +#### `slru` parameters + +`slru` splits the cache into a **probationary** segment and a **protected** segment. A prefix enters probationary, and is promoted to protected once it has been hit enough times. Everything probationary is evicted before anything protected. + +| Key | Type | Default | Meaning | +|---|---|---|---| +| `protected_threshold` | int | `2` | Hit count at which a prefix is promoted to the protected segment. | + +Raising it makes promotion harder, so the protected set stays small and closer to your genuinely hot prefixes; a prefix hit three times stays probationary at `4` but is protected at the default `2`. Lowering it to `1` promotes any prefix that is reused even once, which approaches `lru` with a one-hit grace period. + +#### `tlru` parameters + +`tlru` (Tail-Optimized LRU, [arXiv:2510.15152](https://arxiv.org/abs/2510.15152)) protects only the cached history a conversation's next prefill needs to stay under a TTFT budget; the rest of its tail is evicted first, and eviction continues in plain recency order once the TEL-safe tokens run out. + +| Key | Type | Meaning | +|---|---|---| +| `threshold` | int | Tail-latency threshold ξ, in tokens: a conversation only keeps enough cache to hold its next prefill under ξ uncached tokens. Convert from a TTFT target by dividing it by the measured ms per uncached token. The paper states ξ in blocks, so multiply its values by `--page-size`. | +| `next_prompt_estimate` | int | Estimated tokens the next turn of a conversation will add; use the trace's empirical mean. | + +Both keys are required in practice: `threshold` must be greater than `next_prompt_estimate` (only the difference affects behaviour), and values at or above it reduce T-LRU to plain LRU, which startup rejects. + +```bash Command +python3 -m sglang.launch_server \ + --model-path MODEL_PATH \ + --radix-eviction-policy tlru \ + --radix-eviction-policy-config '{"threshold": 4096, "next_prompt_estimate": 512}' +``` + +## Which policy to pick + +Start with `lru` and change it only against a measured cache hit rate — the counters are exposed under `--enable-metrics`. `lfu` and `slru` help when your traffic has a stable hot set that a recency-only policy keeps flushing; they hurt when prefix popularity shifts over time, because a prefix that earned a high hit count keeps its advantage after it stops being useful. diff --git a/docs/docs/advanced_features/server_arguments.mdx b/docs/docs/advanced_features/server_arguments.mdx index 496000c896c2..2a09a0c8c0ce 100644 --- a/docs/docs/advanced_features/server_arguments.mdx +++ b/docs/docs/advanced_features/server_arguments.mdx @@ -519,6 +519,12 @@ Please consult the documentation below and [server_args.py](https://github.com/s How conservative the schedule policy is. A larger value means more conservative scheduling. Use a larger value if you see requests being retracted frequently. `1.0` Type: float + + + `--retraction-policy` + The decode retraction policy to use when the KV cache is full. 'length' preserves the existing behavior and retracts short-output, long-input requests first. 'priority' retracts lower-priority requests first, using the same priority direction as priority scheduling. + `length` + length, priority `--page-size` @@ -662,8 +668,8 @@ Please consult the documentation below and [server_args.py](https://github.com/s `--dcp-comm-backend` - Communication backend for the DCP attention reduction: AllGather + ReduceScatter, fused NCCL All-to-All, or FlashInfer MNNVL All-to-All. - `ag_rs` + Communication backend for the DCP attention reduction: AllGather + ReduceScatter, fused NCCL All-to-All, or FlashInfer MNNVL All-to-All. Unset resolves to fi_a2a where the DCP group shares an MNNVL domain on Blackwell, else a2a on CUDA/ROCm, else ag_rs. + None ag_rs, a2a, fi_a2a @@ -792,12 +798,6 @@ Please consult the documentation below and [server_args.py](https://github.com/s `False` bool flag (set to enable) - - `--stream-output` - [Deprecated] Use --incremental-streaming-output instead. - — - Type: str - `--enable-streaming-session` Enable streaming session mode and StreamingSession wrapper. @@ -937,12 +937,6 @@ Please consult the documentation below and [server_args.py](https://github.com/s The buckets of end-to-end request latency, specified as a list of floats. None List[float] - - - `--collect-tokens-histogram` - Collect prompt/generation tokens histogram. - False - bool flag (set to enable) `--prompt-tokens-buckets` @@ -1069,6 +1063,16 @@ Please consult the documentation below and [server_args.py](https://github.com/s +## Responses storage + +Responses storage is now disabled by default, including on standalone servers. Add `--enable-response-store` to retain responses for retrieval, `previous_response_id` chaining, and background requests. Storage is process-local and in-memory, has no TTL or size limit, and is lost on restart. + +Foreground generation and streaming still accept the API's default `store: true` without retaining responses or message history when server storage is disabled. The response's `store` field echoes the client request; the server flag controls persistence. With storage enabled, `store: false` opts out for the current request while allowing it to read a stored predecessor. + +Background requests, including streaming requests, require both `--enable-response-store` and `store: true`. Without the flag, chaining, background requests, retrieval, and cancellation return HTTP 400. Cancellation applies to detached background requests; active streams have no stored response until completion. + +Combining `--enable-response-store` with `--disaggregation-mode=prefill` or `decode` fails at startup. PD clients must send explicit conversation history in foreground requests. Built-in web search and code interpreter calls are also unsupported under PD. + ## API related @@ -1121,6 +1125,12 @@ Please consult the documentation below and [server_args.py](https://github.com/s + + + + + + @@ -1142,15 +1152,15 @@ Please consult the documentation below and [server_args.py](https://github.com/s - + - + - + - + @@ -1418,19 +1428,19 @@ Please consult the documentation below and [server_args.py](https://github.com/s - + - + - + @@ -1510,18 +1520,6 @@ Please consult the documentation below and [server_args.py](https://github.com/s - - - - - - - - - - - -
When the HuggingFace tokenizer has multiple chat templates (e.g., 'default', 'tool_use', 'rag'), specify which named template to use. If not set, the first available template is used. `None` Type: str
`--default-chat-template-kwargs`A JSON object used as defaults for chat_template_kwargs by Chat Completions, Responses, Anthropic-compatible requests, and chat tokenization. Direct Ollama and embedding paths do not consume these defaults. For ordinary template keys, explicit per-request values take precedence. Keys are specific to the selected model's chat template; reasoning_effort is normalized separately.`None`Type: JSON / Dict
`--completion-template`
`--reasoning-parser`Specify the parser for reasoning models. Supported parsers: [deepseek-r1, deepseek-v3, glm45, gpt-oss, kimi, qwen3, qwen3-thinking, step3].Specify the parser for reasoning models. Use `auto` to detect the parser from the model's chat template. `None`deepseek-r1, deepseek-v3, glm45, gpt-oss, kimi, qwen3, qwen3-thinking, step3auto, apertus2509, deepseek-r1, deepseek-v3, deepseek-v4, dots, glm45, ling3, hunyuan, gpt-oss, k2_horizon, kimi, kimi_k2, kimi_k3, mimo, muse, poolside_v1, qwen3, qwen3-thinking, minimax, minimax-append-think, minimax-m3, step3, step3p5, mistral, nemotron_3, interns1, gemma4, gigachat35, inkling, cohere_command4
`--tool-call-parser`Specify the parser for handling tool-call interactions. Supported parsers: [deepseekv3, deepseekv31, glm, glm45, glm47, gpt-oss, kimi_k2, llama3, mistral, pythonic, qwen, qwen25, qwen3_coder, step3].Specify the parser for handling tool-call interactions. Use `auto` to detect the parser from the model's chat template. `None`deepseekv3, deepseekv31, glm, glm45, glm47, gpt-oss, kimi_k2, llama3, mistral, pythonic, qwen, qwen25, qwen3_coder, step3, gigachat3auto, apertus2509, cohere_command4, deepseekv3, deepseekv31, deepseekv32, deepseekv4, dots, glm, glm45, glm47, gpt-oss, k2_horizon, kimi_k2, kimi_k3, lfm2, ling3, llama3, mimo, minicpm5, mistral, muse, poolside_v1, pythonic, qwen, qwen25, qwen3_coder, spark25, step3, step3p5, minimax-m2, minimax-m3, trinity, interns1, hermes, hunyuan, gigachat3, gigachat35, gemma4, inkling
`--tool-server``--attention-backend` Choose the kernels for attention layers. `None`triton, torch_native, flex_attention, dsa, nsa, dsv4, compressed, cutlass_mla, fa3, fa4, flashinfer, flashmla, trtllm_mla, cutedsl_mla, tokenspeed_mla, trtllm_mha, dual_chunk_flash_attn, aiter, wave, intel_amx, ascend, intel_xputriton, torch_native, flex_attention, dsa, nsa, dsv4, compressed, fa3, fa4, flashinfer, flashmla, trtllm_mla, cutedsl_mla, tokenspeed_mla, trtllm_mha, aiter, wave, intel_amx, ascend, intel_xpu
`--prefill-attention-backend` Choose the kernels for prefill attention layers (have priority over --attention-backend). `None`triton, torch_native, flex_attention, dsa, nsa, dsv4, compressed, cutlass_mla, fa3, fa4, flashinfer, flashmla, trtllm_mla, cutedsl_mla, tokenspeed_mla, trtllm_mha, dual_chunk_flash_attn, aiter, wave, intel_amx, ascend, intel_xputriton, torch_native, flex_attention, dsa, nsa, dsv4, compressed, fa3, fa4, flashinfer, flashmla, trtllm_mla, cutedsl_mla, tokenspeed_mla, trtllm_mha, aiter, wave, intel_amx, ascend, intel_xpu
`--decode-attention-backend` Choose the kernels for decode attention layers (have priority over --attention-backend). `None`triton, torch_native, flex_attention, dsa, nsa, dsv4, compressed, cutlass_mla, fa3, fa4, flashinfer, flashmla, trtllm_mla, cutedsl_mla, tokenspeed_mla, trtllm_mha, dual_chunk_flash_attn, aiter, wave, intel_amx, ascend, intel_xputriton, torch_native, flex_attention, dsa, nsa, dsv4, compressed, fa3, fa4, flashinfer, flashmla, trtllm_mla, cutedsl_mla, tokenspeed_mla, trtllm_mha, aiter, wave, intel_amx, ascend, intel_xpu
`--sampling-backend``None` Type: str
`--nsa-prefill-backend`[Deprecated] Use --dsa-prefill-backend instead.Autoflashmla_sparse, flashmla_kv, flashmla_auto, fa3, tilelang, aiter, trtllm
`--nsa-decode-backend`[Deprecated] Use --dsa-decode-backend instead.Autoflashmla_sparse, flashmla_kv, flashmla_auto, fa3, tilelang, aiter, trtllm
@@ -1650,12 +1648,6 @@ Please consult the documentation below and [server_args.py](https://github.com/s `None` Type: int - - `--speculative-dflash-draft-window-size` - — - — - Type: int - @@ -1808,9 +1800,9 @@ Please consult the documentation below and [server_args.py](https://github.com/s `--flashinfer-mxfp4-moe-precision` - Choose the computation precision of flashinfer mxfp4 moe + Choose the computation precision of flashinfer mxfp4 moe. On SM90, `fp8` selects the Humming-style MXFP4-weight x FP8-activation path introduced by FlashInfer #3738 and requires FlashInfer >= 0.6.18. `default` - default, bf16 + default, bf16, fp8 `--enable-flashinfer-allreduce-fusion` @@ -1832,11 +1824,17 @@ Please consult the documentation below and [server_args.py](https://github.com/s `--deepep-dispatcher-output-dtype` - Select DeepEP dispather output dtype, could be bf16, fp8, int8 (only Ascend A2/A3 NPU), nvfp4 or auto. Default is auto, which follows a priority order (server argument → deprecated env var → input_global_scale check → dispatcher_output_dtype from quant_config → flashinfer/cutlass backend → NPU BF16 default → GPU FP8 default). + Select DeepEP dispatcher output dtype, could be bf16, fp8, int8 (only Ascend A2/A3 Series NPUs), nvfp4 or auto. Default is auto, which follows a priority order (server argument → deprecated env var → input_global_scale check → dispatcher_output_dtype from quant_config → flashinfer/cutlass backend → NPU BF16 default → GPU FP8 default). `auto` auto, bf16, fp8, int8, nvfp4 - + + --flashinfer-a2a-dispatch-type + Select FlashInfer A2A dispatcher activation dtype. When omitted, it falls back to the SGLANG_MOE_NVFP4_DISPATCH environment variable. Explicit auto selects mxfp8 for --quantization mxfp8, nvfp4 for modelopt FP4 or hybrid NVFP4 MoE checkpoints, and bf16 otherwise. The SGLANG_MOE_NVFP4_DISPATCH environment variable cannot be set with this argument. + None + auto, bf16, nvfp4, mxfp8 + + `--ep-num-redundant-experts` Allocate this number of redundant experts in expert parallel. `0` @@ -1939,12 +1937,6 @@ Please consult the documentation below and [server_args.py](https://github.com/s bool flag (set to enable) - `--elastic-ep-rejoin` - Indicates that this process is a relaunched elastic EP rank that should rejoin an existing process group during rank recovery. - `False` - bool flag (set to enable) - - `--flashinfer-allreduce-fusion-backend` Enable FlashInfer allreduce fusion and choose backend. Defaults to auto. 'auto': choose mnnvl on SM90 single-node systems and SM100/SM103 single-node or multi-node systems; choose trtllm otherwise. 'trtllm': available on single-node systems only. 'mnnvl': available on SM90 single-node systems and SM100/SM103 single-node or multi-node systems via MNNVL fabric. Fuses allreduce with Residual + RMSNorm for supported MoE models. `None` @@ -2318,6 +2310,24 @@ Please consult the documentation below and [server_args.py](https://github.com/s Number of layers per group in offloading. `-1` Type: int + + + `--ple-offload-embedding` + Keep the Qwen4-Exp PLE n-gram embedding table in host memory instead of device memory; the gather kernel reads it from a host pointer. Enabled by default for BF16 Qwen4-Exp on CUDA; `--no-ple-offload-embedding` disables it. + `None` + Type: bool + + + `--ple-offload-backend` + Host storage for the offloaded PLE table: `pinned` (CPU pinned memory) or `file` (a sparse file-backed mmap under `--ple-offload-dir`, read directly by the gather kernel). Use `file` on unified-memory devices such as GB10 / DGX Spark, where pinned host memory comes out of the same pool as the weights; it requires a device that reports `cudaDevAttrPageableMemoryAccessUsesHostPageTables`. + `pinned` + Type: str + + + `--ple-offload-dir` + Directory for the file-backed PLE table when `--ple-offload-backend` is `file`. The file is sparse and rewritten on every startup; keep it on fast local storage (NVMe). Rewriting an existing table can be much slower than filling a fresh sparse file. Before restarting, stop all servers using that directory and remove its `ple_table_*.bin` files to avoid this startup cost. + `$SGLANG_CACHE_DIR/ple/` + Type: str `--offload-num-in-group` @@ -2420,6 +2430,12 @@ Please consult the documentation below and [server_args.py](https://github.com/s `None` Type: int + + `--cuda-graph-max-seq-len-prefill` + Longest sequence a prefill CUDA graph replay admits; longer batches run eager prefill. Folds into `cuda_graph_config[prefill].max_seq_len`. + `None` + Type: int + `--cuda-graph-bs-decode` Explicit list of batch sizes to capture for the decode CUDA graph. @@ -2582,30 +2598,12 @@ Please consult the documentation below and [server_args.py](https://github.com/s `32` Type: int - - `--cuda-graph-max-bs-decode` - Deprecated alias for --cuda-graph-max-bs-decode. - `None` - Type: int - - - `--cuda-graph-bs` - Deprecated alias for --cuda-graph-bs-decode. - `None` - List[int] - `--disable-cuda-graph` Deprecated. Use --cuda-graph-backend-decode=disabled and/or --cuda-graph-backend-prefill=disabled. False bool flag (set to enable) - - `--enable-breakable-cuda-graph` - Deprecated alias for --cuda-graph-backend-prefill=breakable. - False - bool flag (set to enable) - `--disable-prefill-cuda-graph` Disable the prefill-phase CUDA graph. Convenience for --cuda-graph-backend-prefill=disabled. @@ -2618,36 +2616,6 @@ Please consult the documentation below and [server_args.py](https://github.com/s `False` bool flag (set to enable) - - `--disable-piecewise-cuda-graph` - Deprecated alias for --cuda-graph-backend-prefill=disabled. - False - bool flag (set to enable) - - - `--enforce-piecewise-cuda-graph` - Deprecated alias for --cuda-graph-backend-prefill=tc_piecewise. Explicitly setting the prefill backend now skips the auto-disable cascade automatically. - False - bool flag (set to enable) - - - `--piecewise-cuda-graph-tokens` - Deprecated alias for --cuda-graph-bs-prefill. - `None` - List[int] - - - `--piecewise-cuda-graph-compiler` - Deprecated alias for --cuda-graph-tc-compiler. - eager - eager, inductor - - - `--piecewise-cuda-graph-max-tokens` - Deprecated alias for --cuda-graph-max-bs-prefill. - 4096 - Type: int - `--enable-p2p-check` @@ -2745,18 +2713,18 @@ Please consult the documentation below and [server_args.py](https://github.com/s `False` bool flag (set to enable) - - `--keep-mm-feature-on-device` - Keep multimodal feature tensors on device after processing to save D2H copy. - `False` - bool flag (set to enable) - `--enable-return-hidden-states` Enable returning hidden states with responses. `False` bool flag (set to enable) + + `--return-hidden-states-mode` + Set the maximum hidden-state return mode supported by the server. `last` allows requests with return_hidden_states=False or `last`; `full` also allows return_hidden_states=True. + `None` + last, full + `--enable-return-routed-experts` Enable returning routed experts of each layer with responses. @@ -2801,7 +2769,7 @@ Please consult the documentation below and [server_args.py](https://github.com/s `--cp-strategy` - Sharding strategy for prefill CP. zigzag is the former in-seq-split mode; interleave is the former round-robin-split mode. + Sharding strategy for prefill CP. zigzag assigns each rank one early and one late sequence block; interleave assigns token indices modulo the CP size. None zigzag, interleave @@ -2865,42 +2833,6 @@ Please consult the documentation below and [server_args.py](https://github.com/s `False` bool flag (set to enable) - - `--enable-dsa-prefill-context-parallel` - [Deprecated] Use --enable-prefill-cp instead. - — - Type: str - - - `--enable-nsa-prefill-context-parallel` - [Deprecated] Use --enable-prefill-cp instead. - — - Type: str - - - `--enable-prefill-context-parallel` - [Deprecated] Use --enable-prefill-cp instead. - — - Type: str - - - `--dsa-prefill-cp-mode` - [Deprecated] Use --cp-strategy {zigzag,interleave} instead. 'in-seq-split' maps to 'zigzag'; 'round-robin-split' maps to 'interleave'. - `round-robin-split` - in-seq-split, round-robin-split - - - `--nsa-prefill-cp-mode` - [Deprecated] Use --cp-strategy instead. - Auto - in-seq-split, round-robin-split - - - `--prefill-cp-mode` - [Deprecated] Use --cp-strategy {zigzag,interleave} instead. 'in-seq-split' maps to 'zigzag'. - `in-seq-split` - in-seq-split - `--enable-fused-moe-sum-all-reduce` Enable fused moe triton and sum all reduce. @@ -3100,6 +3032,12 @@ Please consult the documentation below and [server_args.py](https://github.com/s For VLM, load weights for the language model only. `False` bool flag (set to enable) + + + `--language-model-only` + Skip the multimodal encoder entirely: its weights are never loaded and the tower is never built, freeing that GPU memory for KV cache. Multimodal requests are rejected. Unlike --language-only this is a standalone mode, not part of encoder/decoder disaggregation. + `False` + bool flag (set to enable) `--encoder-transfer-backend` @@ -3461,12 +3399,6 @@ Please consult the documentation below and [server_args.py](https://github.com/s - - `--prefill-round-robin-balance` - Note: Note: --prefill-round-robin-balance is deprecated now. - `None` - N/A - diff --git a/docs/docs/advanced_features/session_radix_cache.mdx b/docs/docs/advanced_features/session_radix_cache.mdx index 4db1ed4107ba..e1f2d0e70e50 100644 --- a/docs/docs/advanced_features/session_radix_cache.mdx +++ b/docs/docs/advanced_features/session_radix_cache.mdx @@ -13,7 +13,7 @@ Session references are soft protection, not memory pins. Referenced KV can still This feature is implemented only by `UnifiedRadixCache`. ```bash Command -SGLANG_ENABLE_UNIFIED_RADIX_TREE=1 python3 -m sglang.launch_server \ +python3 -m sglang.launch_server \ --model-path MODEL_PATH \ --enable-session-radix-cache ``` diff --git a/docs/docs/advanced_features/sglang_for_rl.mdx b/docs/docs/advanced_features/sglang_for_rl.mdx index 37f242a8e4a0..1c2c663dd6a3 100644 --- a/docs/docs/advanced_features/sglang_for_rl.mdx +++ b/docs/docs/advanced_features/sglang_for_rl.mdx @@ -328,6 +328,21 @@ This path trades some I/O overhead for simplicity and flexibility. It integrates > **Note:** The diffusion engine (SGLang-Diffusion) does not currently support hot refit (updating weights while inference is in progress). The diffusion scheduler processes one request at a time and completes the entire inference before handling the next request, so weight updates and inference never run concurrently. +### Weight Update Session + +`update_weights_from_tensor` and `update_weights_from_distributed` must run inside a weight-update session. The update calls only write checkpoint-format values; `end_weight_update` runs every quantized layer's `process_weights_after_loading` once, turning them into kernel layout (Marlin repack, MXFP8 shuffles, ...), and runs `post_load_weights` if no weights were loaded through the engine during the session (P2P/RDMA writes). `begin_weight_update` first restores the few schemes whose kernel layout changes parameter shapes (today compressed-tensors W4A16 MoE on Marlin) so the checkpoint-format values fit. One session can span many update calls, so the finalize cost is paid once per refit rather than once per bucket. A call outside a session, a second `begin_weight_update`, or an `end_weight_update` without one is rejected with `success: false` (HTTP 400) and leaves the weights untouched. + +**Endpoints:** `POST /begin_weight_update`, `POST /end_weight_update` + +`begin_weight_update` accepts `selector`: `"target"`, `"draft"`, or `"all"` (default). Under speculative decoding the same selector on the update endpoints picks which model runners receive the weights; `end_weight_update` finalizes the runners the session opened. + +```python +engine.begin_weight_update() +for bucket in buckets: + engine.update_weights_from_tensor(bucket) +engine.end_weight_update() +``` + ### Update Weights from Tensor **When to use:** @@ -394,7 +409,7 @@ This strategy requires the training process and rollout engine to share access t **Note:** The serialized tensor payloads must be created with `MultiprocessingSerializer.serialize(...)` and should be base64-safe strings. -**Python Engine API:** `engine.update_weights_from_tensor(named_tensors, load_format=None, flush_cache=True)` +**Python Engine API:** `engine.update_weights_from_tensor(named_tensors, load_format=None, flush_cache=True)`, inside `begin_weight_update()` / `end_weight_update()`. ### Update Weights from Distributed Group @@ -575,7 +590,9 @@ Training workers gather weights (typically on TP rank 0), broadcast them to the **Python Engine APIs:** - `engine.init_weights_update_group(...)` +- `engine.begin_weight_update(selector="all")` - `engine.update_weights_from_distributed(names, dtypes, shapes, ...)` +- `engine.end_weight_update()` - `engine.destroy_weights_update_group(group_name)` ### Per-Token Weight Version Attribution diff --git a/docs/docs/advanced_features/sglang_simulator.mdx b/docs/docs/advanced_features/sglang_simulator.mdx new file mode 100644 index 000000000000..44567b92feac --- /dev/null +++ b/docs/docs/advanced_features/sglang_simulator.mdx @@ -0,0 +1,88 @@ +--- +title: "SGLang Simulator" +metatags: + description: "Run SGLang scheduling and KV-cache simulations without loading model weights or executing GPU kernels." +--- + +SGLang Simulator reuses SGLang's scheduler, request lifecycle, and KV-cache implementation while replacing model forward execution with a latency predictor. Use it to compare scheduling and cache configurations on timestamped or synthetic workloads without loading model weights. + +## Supported scope + +SGLang Simulator tracks the current `main` branch and recent SGLang releases. The current integration is validated with `v0.5.16`, `v0.5.17`, `v0.5.18`, and `main`. + +The initial upstream scope uses one simulated worker with `tp_size=1`, `ep_size=1`, `dp_size=1`, and `pp_size=1`. A simulator configuration can describe a larger target system for latency prediction, but the SGLang runtime process topology remains single-worker. + +The simulator supports: + +- synthetic request rates, ShareGPT workloads, and timestamped Autobench traces; +- OFFLINE logical-time simulation and BLOCKING wall-clock replay; +- AIConfigurator, ML, and replay latency predictors; +- SGLang prefix caching and [HiCache](/docs/advanced_features/hicache); and +- serving-compatible TTFT, TPOT, ITL, throughput, and cache-hit metrics. + +## Install from the SGLang repository + +Use the simulator and SGLang source from the same monorepo checkout: + +```bash +python3 -m pip install -e tools/sglang-simulator +export PYTHONPATH="$PWD/tools/sglang-simulator/src:$PWD/python" +``` + +AIConfigurator is optional. Install the validated extra only when you use an AIConfigurator predictor: + +```bash +python3 -m pip install -e "tools/sglang-simulator[aic]" +``` + +## Start a simulator server + +Choose a fresh output directory for every run. The server owns the simulation mode and writes metrics to this directory. + +```bash +export SGLANG_USE_CPU_ENGINE=1 +export CUDA_VISIBLE_DEVICES="" +export SGLANG_SIMULATOR_OUTPUT_MODE=OFFLINE +export SGLANG_SIMULATOR_OUTPUT_DIR=/tmp/sglang-simulator-quickstart + +python3 -m sglang_simulator.simulation.sglang.launch_server \ + --model-path tools/sglang-simulator/test/assets/qwen3-8b \ + --tokenizer-path tools/sglang-simulator/examples/assets/tokenizer \ + --sim-config-path tools/sglang-simulator/examples/sim_configs/replay.json \ + --port 30000 +``` + +`OFFLINE` advances the simulator's logical clock without sleeping. `BLOCKING` also sleeps for predicted forward and cache-load latency, which is useful when a client must observe simulated wall-clock pacing. + +## Send a workload + +In another terminal, export the same output directory and run the simulator-aware serving benchmark from the repository root: + +```bash +export PYTHONPATH="$PWD/tools/sglang-simulator/src:$PWD/python" +export SGLANG_SIMULATOR_OUTPUT_DIR=/tmp/sglang-simulator-quickstart + +python3 benchmark/simulator/bench_serving.py \ + --simulator-mode offline \ + --backend sglang \ + --base-url http://127.0.0.1:30000 \ + --model tools/sglang-simulator/test/assets/qwen3-8b \ + --tokenizer tools/sglang-simulator/examples/assets/tokenizer \ + --dataset-name sharegpt \ + --dataset-path tools/sglang-simulator/examples/workloads/sharegpt-example.json \ + --sharegpt-output-len 4 \ + --num-prompts 3 \ + --output-file /tmp/sglang-simulator-quickstart/benchmark.json +``` + +The benchmark injects logical arrival metadata into each request and displays the server-side simulator metrics. For timestamped traffic, use the simulator-owned Autobench JSONL format and add `--use-trace-timestamps`. + +## Read the results + +The output directory contains: + +- `metrics.json`: aggregate latency, throughput, and cache metrics; +- `request.jsonl`: per-request timing and cache information; and +- `iteration.jsonl`: scheduler batch composition and predicted iteration latency. + +Use a unique output directory for each run so metrics from separate experiments are not mixed. See the [SGLang Simulator source README](https://github.com/sgl-project/sglang/tree/main/tools/sglang-simulator) for simulator configuration fields, predictor examples, and maintained tests. diff --git a/docs/docs/advanced_features/speculative_decoding.mdx b/docs/docs/advanced_features/speculative_decoding.mdx index 084a2260e577..004e9ef75b81 100644 --- a/docs/docs/advanced_features/speculative_decoding.mdx +++ b/docs/docs/advanced_features/speculative_decoding.mdx @@ -1,9 +1,9 @@ --- title: "Speculative Decoding" metatags: - description: "SGLang speculative decoding: EAGLE-2/EAGLE-3, MTP, DFLASH, draft model configuration, and overlap-scheduler guidance." + description: "SGLang speculative decoding: EAGLE-2/EAGLE-3, MTP, UNO, DFLASH, draft model configuration, and overlap-scheduler guidance." --- -SGLang provides several speculative decoding options, including EAGLE-2/EAGLE-3, MTP, DFLASH, classic draft-model decoding, and an NGRAM-based variant. Our implementation aims to maximize speed and efficiency and is considered to be among the fastest in open-source LLM engines. +SGLang provides several speculative decoding options, including EAGLE-2/EAGLE-3, MTP, UNO, DFLASH, classic draft-model decoding, and an NGRAM-based variant. Our implementation aims to maximize speed and efficiency and is considered to be among the fastest in open-source LLM engines. ## Summary @@ -15,6 +15,7 @@ SGLang provides several speculative decoding options, including EAGLE-2/EAGLE-3, - [EAGLE-2 Decoding via Frequency-Ranked Speculative Sampling](#eagle-2-decoding-via-frequency-ranked-speculative-sampling) - [EAGLE-3 Decoding](#eagle-3-decoding) - [Multi Token Prediction](#multi-token-prediction) +- [UNO decoding](#uno-decoding) - [DFlash Decoding](#dflash-decoding) - [Standalone Speculative Decoding (Small Draft Model)](#standalone-speculative-decoding-small-draft-model) - [Speculative Decoding V2 (Overlap Scheduler)](#speculative-decoding-v2-overlap-scheduler) @@ -30,6 +31,7 @@ SGLang provides several speculative decoding options, including EAGLE-2/EAGLE-3, - **Workload acceptance changes over time**: Use [**Adaptive speculative decoding**](./adaptive_speculative_decoding) on top of **EAGLE** with `--speculative-eagle-topk 1`. - **Lower `lm_head` overhead for EAGLE-2**: Enable **FR-Spec** with `--speculative-token-map`. - **Model is MTP-enabled**: Use **MTP via speculative decoding** (often with small `speculative_num_steps/topk/num_draft_tokens`, see the example section). +- **You have a UNO adapter for the target model**: Use **UNO** with `--speculative-algorithm UNO` and `--uno-lora-path ...`. - **You have a DFlash draft checkpoint**: Use **DFLASH** with `--speculative-algorithm DFLASH` and `--speculative-draft-model-path ...`. - **You have a smaller draft LLM**: Use **STANDALONE** (`--speculative-algorithm STANDALONE`). - **No extra model available**: Use **NGRAM** (`--speculative-algorithm NGRAM`, CUDA-only). @@ -86,6 +88,13 @@ SGLang provides several speculative decoding options, including EAGLE-2/EAGLE-3, See Multi Token Prediction section Uses speculative workflow; draft path may be auto-handled for some models + + UNO + Target model with a trained UNO LoRA (linear chain or tree) + No + --speculative-algorithm UNO + --uno-lora-path ... + CUDA and FA3; currently requires TP=PP=1 + DFLASH DFlash draft model (linear block verification) @@ -434,6 +443,88 @@ print(response.json()) --- +## UNO decoding + +UNO reuses the target transformer for both passes of each speculative decode cycle instead of loading a separate draft model. During the draft forward, the first row of each request uses the base weights and the remaining `B - 1` rows use a trained UNO LoRA. Target verification is entirely base-only. + +UNO provides two sampling modes: + +- **Linear** constructs and verifies one chain of draft tokens, similar to DFlash's proposal layout. +- **Tree** expands multiple candidates at each draft depth and uses SGLang's EAGLE tree-verification path. + +For UNO modes, the three numbers are `B/K/V`: draft-forward width, candidates kept per expansion, and verification width. Thus, `Linear 8/1/8` uses an 8-token draft width, one candidate per expansion, and an 8-token verification width. UNO's `K` controls proposal-tree breadth and is unrelated to the request sampling parameter `top_k`. The command-line mapping depends on the mode: + +- In linear mode, `B` and `V` are both `--speculative-num-draft-tokens`; set `--speculative-num-steps 1` and `--speculative-eagle-topk 1`. +- In tree mode, `B` is `--speculative-num-steps + 1`, `K` is `--speculative-eagle-topk`, and `V` is `--speculative-num-draft-tokens`. + +Use an adapter trained for the exact target-model checkpoint. The current integration has been validated with `Qwen/Qwen3-8B`; unsupported LoRA target layers are rejected at startup. Set the adapter's local path or Hugging Face repository before starting the server: + +### Linear 8/1/8 + +```bash Command +export UNO_LORA_PATH="s-sahoo/uno-qwen3-8B" + +sglang serve \ + --model-path Qwen/Qwen3-8B \ + --speculative-algorithm UNO \ + --uno-lora-path "$UNO_LORA_PATH" \ + --speculative-num-steps 1 \ + --speculative-eagle-topk 1 \ + --speculative-num-draft-tokens 8 \ + --attention-backend fa3 \ + --tp 1 \ + --host 0.0.0.0 \ + --port 30000 +``` + +### Tree 8/32/8 + +The following command selects tree mode because `K` is greater than one: + +```bash Command +export UNO_LORA_PATH="s-sahoo/uno-qwen3-8B" + +sglang serve \ + --model-path Qwen/Qwen3-8B \ + --speculative-algorithm UNO \ + --uno-lora-path "$UNO_LORA_PATH" \ + --speculative-num-steps 7 \ + --speculative-eagle-topk 32 \ + --speculative-num-draft-tokens 8 \ + --attention-backend fa3 \ + --tp 1 \ + --host 0.0.0.0 \ + --port 30000 +``` + +Send requests through the standard OpenAI-compatible API: + +```python Example +import openai + +client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None") + +response = client.chat.completions.create( + model="Qwen/Qwen3-8B", + messages=[{"role": "user", "content": "Explain speculative decoding."}], + max_tokens=128, +) + +print(response.choices[0].message.content) +``` + +### Key requirements and limitations + +- UNO requires CUDA with FA3 for prefill and decode, tensor and pipeline parallel sizes of 1, and no DP attention or context parallelism. +- `--uno-lora-path` loads UNO's fixed internal adapter and cannot be combined with request-selectable Multi-LoRA serving. +- UNO manages its own stochastic verification; do not set `--speculative-use-rejection-sampling`. +- Grammar decoding, returned logprobs or hidden states, sampling penalties, `min_p`, logit bias, custom logit processors, strict thinking, and deterministic inference are not yet supported. +- In tree mode, both speculative acceptance thresholds must remain at 1.0, `V` must be at least `B` and at most 128, and `V * K` must not exceed 2048. SGLang validates the remaining tree-capacity and EAGLE parent-representation constraints at startup. +- Mixed chunked prefill is disabled for UNO. +- Ordinary overlap scheduling is supported. Tree mode does not yet support PDMux or the separate `--enable-two-batch-overlap` feature. + +--- + ## DFlash Decoding SGLang also supports **DFLASH** speculative decoding using a dedicated draft model checkpoint. Compared with EAGLE-style tree verification, DFLASH verifies a linear draft block and is configured around a block size / draft window. This path is useful when the target model has a matching DFlash draft checkpoint, such as `meta-llama/Llama-3.1-8B-Instruct` with `z-lab/LLaMA3.1-8B-Instruct-DFlash-UltraChat`. @@ -470,7 +561,7 @@ Relevant parameters: None - --speculative-dflash-draft-window-size + --speculative-draft-window-size Draft KV sliding-window size. Must be >= speculative-num-draft-tokens when set. None @@ -753,7 +844,7 @@ Below is a comprehensive list of all speculative decoding parameters available i --speculative-algorithm str None - Algorithm to use: DFLASH, EAGLE, EAGLE3, STANDALONE, NGRAM, NEXTN (alias of EAGLE) + Algorithm to use: UNO, DFLASH, EAGLE, EAGLE3, STANDALONE, NGRAM, NEXTN (alias of EAGLE) --speculative-draft-model-path @@ -761,6 +852,12 @@ Below is a comprehensive list of all speculative decoding parameters available i None Path to the draft model weights + + --uno-lora-path + str + None + UNO-only path or Hugging Face repository for the draft LoRA checkpoint + --speculative-draft-model-revision str @@ -777,19 +874,19 @@ Below is a comprehensive list of all speculative decoding parameters available i --speculative-num-steps int None (auto-chosen when omitted) - Autoregressive drafting depth + Autoregressive drafting depth. Some algorithms auto-tune it; tree UNO requires it and uses B = speculative_num_steps + 1. --speculative-eagle-topk int None (auto-chosen when omitted) - Branching factor per drafting step + Branching factor per drafting step. Some algorithms auto-tune it; UNO resolves an omitted value to 1, and values greater than one select UNO tree mode. --speculative-num-draft-tokens int None (auto-chosen when omitted) - Maximum number of draft tokens for verification + Maximum number of draft tokens for verification. Some algorithms auto-tune it; UNO requires it and uses it as linear width or tree verification width V. --speculative-dflash-block-size @@ -798,7 +895,7 @@ Below is a comprehensive list of all speculative decoding parameters available i DFlash-only alias of --speculative-num-draft-tokens - --speculative-dflash-draft-window-size + --speculative-draft-window-size int None DFlash-only draft KV sliding-window size diff --git a/docs/docs/basic_usage/native_api.mdx b/docs/docs/basic_usage/native_api.mdx index eff2293166d9..1673efc5b5d3 100644 --- a/docs/docs/basic_usage/native_api.mdx +++ b/docs/docs/basic_usage/native_api.mdx @@ -15,6 +15,7 @@ Apart from the OpenAI compatible APIs, the SGLang Runtime also provides its nati - `/encode`(embedding model) - `/v1/rerank`(cross encoder rerank model) - `/v1/score`(decoder-only scoring) +- `/v1/decisions`(typed decisions on a chat model) - `/classify`(reward model) - `/start_expert_distribution_record` - `/stop_expert_distribution_record` @@ -256,6 +257,8 @@ Parameters: - `item_first`: Whether items come first in concatenation order (default: False) - `model`: Model name +`query` and `items` also accept token ids, with `query` set to `[]` when each item is a complete prompt. `label_token_ids` can be one list per item. `temperature` divides the label logits before the softmax and needs `apply_softmax`, and `return_token_logprobs` adds `token_logprobs`, the full-vocabulary log-probabilities of the labels. + The response contains `scores` - a list of probability lists, one per item, each in the order of `label_token_ids`. ```python Example @@ -293,6 +296,20 @@ for item, scores in zip(items, response_json["scores"]): terminate_process(score_process) ``` +## v1/decisions (typed decisions on a chat model) + +Answer typed questions about an input with per-option probabilities, without generating. The server renders each question with the model's chat template and thinking turned off, assigns one-token answer labels, and scores them through the same path as `v1/score`. + +Parameters: +- `input`: Text, object, or array the questions are about +- `questions`: List of questions, each with a unique `id` and a `type` of `choice` (with `options`), `score` (with `levels`), or `yes_no` +- `temperature`: Divides the label logits before the softmax (default: 1) +- `chat_template_kwargs`: Extra chat template arguments +- `prompt_format_version`: Optional pin of the server-owned prompt wording +- `return_prompt_token_ids`: Return the scored prompt and label token ids for replay through `v1/score` (default: False) + +The response contains `answers` keyed by question id, each with `type`, `probabilities`, and `label_mass`, plus `choice` for a choice question or `score` for a score question. A yes or no answer is read from `probabilities["yes"]`. See [Decision models](/docs/supported-models/decision_models) for examples, the full reference, and error cases. + ## Classify (reward model) SGLang Runtime also supports reward models. Here we use a reward model to classify the quality of pairwise generations. diff --git a/docs/docs/basic_usage/openai_api_completions.mdx b/docs/docs/basic_usage/openai_api_completions.mdx index 449b07700c58..be4e68c3d1a1 100644 --- a/docs/docs/basic_usage/openai_api_completions.mdx +++ b/docs/docs/basic_usage/openai_api_completions.mdx @@ -117,10 +117,10 @@ Some models support internal reasoning or thinking processes that can be exposed #### Basic Usage -To enable reasoning output, you need to: -1. Launch the server with the appropriate reasoning parser -2. Set the model-specific parameter in `chat_template_kwargs` -3. Optionally use `separate_reasoning: False` to not get reasoning content separately (default to `True`) +To use reasoning output: +1. Launch the server with the appropriate reasoning parser. +2. Use the model-specific parameter in `chat_template_kwargs` when you need to override the model's chat-template default. +3. Optionally use `separate_reasoning: False` to keep reasoning in the response content instead of returning it separately (defaults to `True`). **Note for Qwen3-Thinking models:** These models always generate thinking content and do not support the `enable_thinking` parameter. Use `--reasoning-parser qwen3-thinking` or `--reasoning-parser qwen3` to parse the thinking content. @@ -176,9 +176,30 @@ Thus, the total count is **3**. **Answer:** 3. ``` -Setting `"enable_thinking": False` (or omitting it) will result in `reasoning_content` being `None`. Qwen3-Thinking models always generate reasoning content and don't support the `enable_thinking` parameter. +Setting `"enable_thinking": False` disables reasoning for Qwen3. Omitting the parameter follows the model's chat-template default; Qwen3 enables thinking by default. Qwen3-Thinking models always generate reasoning content and do not support the `enable_thinking` parameter. +#### Set a Default for OpenAI Chat Completions + +If most OpenAI chat-completion requests should use the same chat-template mode, set a default when launching the server: + +```bash Command +python -m sglang.launch_server \ + --model-path Qwen/Qwen3-4B \ + --reasoning-parser qwen3 \ + --default-chat-template-kwargs '{"enable_thinking": false}' +``` + +`--default-chat-template-kwargs` accepts a JSON object and merges it into each OpenAI chat-completion request's `chat_template_kwargs`. For ordinary template keys such as `enable_thinking`, the precedence is: + +1. An explicit value in the request's `chat_template_kwargs` +2. The value in `--default-chat-template-kwargs` +3. The model's chat-template default + +`reasoning_effort` follows a separate normalization path, so the precedence list above does not apply to it. Configure `reasoning_effort` in either the server defaults or individual requests, not both. + +The accepted keys are model-specific. For example, Qwen3 uses `enable_thinking`, while other model families may use `thinking` or may not expose a reasoning toggle. The reasoning parser controls how reasoning tokens are separated from the response; it does not by itself enable reasoning in the chat template. + #### Logit Bias Support SGLang supports the `logit_bias` parameter for both chat completions and completions APIs. This parameter allows you to modify the likelihood of specific tokens being generated by adding bias values to their logits. The bias values can range from -100 to 100, where: diff --git a/docs/docs/developer_guide/benchmark_and_profiling.mdx b/docs/docs/developer_guide/benchmark_and_profiling.mdx index fa9963d22167..ac4890a4e47f 100644 --- a/docs/docs/developer_guide/benchmark_and_profiling.mdx +++ b/docs/docs/developer_guide/benchmark_and_profiling.mdx @@ -59,18 +59,19 @@ Use `bench_serving` by default unless there are specific needs. ``` - Pass `--enable-multi-batch` and set `--batch-size` to a multiple of the server's `--max-running-requests` to stabilize throughput measurements. Surplus requests are queued by the scheduler and promoted batch-by-batch, amortizing per-request prefill and first-step transients into steady-state decode. Under this flag, only `overall_throughput` is authoritative; `input_throughput`, `output_throughput`, `last_ttft`, and ITL include cross-batch queueing in their denominators and should be treated as informational. + - Pass `--disable-ignore-eos` when benchmarking with real prompts, where forcing decode past EOS would shift the output distribution toward gibberish. Requests then early-stop, so the whole-run `output_throughput`/ITL include the decaying-batch tail; the report gains steady-state columns measured only over the window where every request is still decoding (from the last request's first token to the first request's finish). Replay recorded traffic with `--dataset-name sharegpt` or `--dataset-name custom --dataset-path ` (recorded completion lens are ignored; `--output-len` is the shared `max_new_tokens` cap). For OpenAI-format traces with per-request parameters, use `bench_serving`. - Pass `--lora-name ` to route every prompt through a pre-loaded LoRA adapter. Requires the server to be launched with `--enable-lora --lora-paths =`. **`bench_offline_throughput`** directly instantiates the `Engine` object in-process (no HTTP server) and submits all requests at once via `engine.generate()`. The engine's scheduler handles batching and execution. This measures maximum achievable throughput without any network overhead. ```bash Command - python3 -m sglang.bench_offline_throughput --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --num-prompts 10 + python3 -m sglang.benchmark.offline_throughput --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --num-prompts 10 ``` **`bench_one_batch`** is the lowest-level tool. It directly instantiates a `ModelRunner` and calls `extend()` / `decode()` on a fixed static batch, bypassing the scheduler entirely. The prefill and decode phases are run separately, making profiling easier but rendering the metrics unrealistic. Because there is no dynamic batching, it may run out of memory for batch sizes that a real server can handle (a real server chunks prefill into smaller batches). This is best suited for profiling individual kernel performance. ```bash Command - python3 -m sglang.bench_one_batch --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --batch-size 32 --input-len 256 --output-len 32 + python3 -m sglang.benchmark.one_batch --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --batch-size 32 --input-len 256 --output-len 32 ``` ## Profile with PyTorch Profiler @@ -139,16 +140,16 @@ python -m sglang.bench_serving --backend sglang --model meta-llama/Llama-3.1-8B- - Make sure `SGLANG_TORCH_PROFILER_DIR` is set on all worker nodes before starting the servers - For more details on setting up PD disaggregation, see [PD Disaggregation Guide](../advanced_features/pd_disaggregation) -### Profile a server with `sglang.bench_offline_throughput` +### Profile a server with `sglang.benchmark.offline_throughput` ```bash Command export SGLANG_TORCH_PROFILER_DIR=/root/sglang/profile_log # profile one batch with bench_one_batch.py # batch size can be controlled with --batch argument -python3 -m sglang.bench_one_batch --model-path meta-llama/Llama-3.1-8B-Instruct --batch 32 --input-len 1024 --output-len 10 --profile +python3 -m sglang.benchmark.one_batch --model-path meta-llama/Llama-3.1-8B-Instruct --batch 32 --input-len 1024 --output-len 10 --profile # profile multiple batches with bench_offline_throughput.py -python -m sglang.bench_offline_throughput --model-path meta-llama/Llama-3.1-8B-Instruct --dataset-name random --num-prompts 10 --profile --mem-frac=0.8 +python -m sglang.benchmark.offline_throughput --model-path meta-llama/Llama-3.1-8B-Instruct --dataset-name random --num-prompts 10 --profile --mem-frac=0.8 ``` ### Profile a server with `sglang.profiler` @@ -375,7 +376,7 @@ RuntimeError: !stack.empty() INTERNAL ASSERT FAILED at "/pytorch/torch/csrc/auto This is likely a PyTorch Bug reported in [Bug: vLLM Profiler](https://github.com/vllm-project/vllm/issues/18240) and [Bug: torch.profiler.profile](https://github.com/pytorch/pytorch/issues/101632). As a workaround, you may disable `with_stack` with an environment variable such as follows: ```bash Command export SGLANG_PROFILE_WITH_STACK=False -python -m sglang.bench_offline_throughput --model-path meta-llama/Llama-3.1-8B-Instruct --dataset-name random --num-prompts 10 --profile --mem-frac=0.8 +python -m sglang.benchmark.offline_throughput --model-path meta-llama/Llama-3.1-8B-Instruct --dataset-name random --num-prompts 10 --profile --mem-frac=0.8 ``` ### View traces @@ -419,7 +420,7 @@ Additionally, if you want to locate the SGLang Python source code through the cu 2. To profile a single batch, use ```bash Command - nsys profile --trace-fork-before-exec=true --cuda-graph-trace=node python3 -m sglang.bench_one_batch --model meta-llama/Meta-Llama-3-8B --batch-size 64 --input-len 512 + nsys profile --trace-fork-before-exec=true --cuda-graph-trace=node python3 -m sglang.benchmark.one_batch --model meta-llama/Meta-Llama-3-8B --batch-size 64 --input-len 512 ``` 3. To profile a server, e.g. @@ -570,7 +571,7 @@ In the Nsight Systems GUI, you'll see: 2. You can benchmark a model with modified configs (e.g., less layers) by using `--json-model-override-args`. For example, you can benchmark a model with only 2 layers and 2 kv heads using: ```bash Command - python -m sglang.bench_one_batch --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --batch 32 --input-len 256 --output-len 32 --load-format dummy --json-model-override-args '{"num_hidden_layers": 1, "num_key_value_heads": 1}' + python -m sglang.benchmark.one_batch --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --batch 32 --input-len 256 --output-len 32 --load-format dummy --json-model-override-args '{"num_hidden_layers": 1, "num_key_value_heads": 1}' ``` 3. You can use `--python-backtrace=cuda` to see python call stack for all CUDA kernels, as in PyTorch Profiler. (Caveat: this can cause inaccurately long kernel runtimes for CUDA event based timing) diff --git a/docs/docs/developer_guide/contribution_guide.mdx b/docs/docs/developer_guide/contribution_guide.mdx index f320cbf682dc..08b212d31591 100644 --- a/docs/docs/developer_guide/contribution_guide.mdx +++ b/docs/docs/developer_guide/contribution_guide.mdx @@ -122,14 +122,17 @@ For CI to run on a pull request, it must have the "run-ci" label. Authorized use - `/tag-run-ci-label`: Adds the "run-ci" label. Only **future** commits trigger CI; the current commit is unaffected. Add the `extra` argument (`/tag-run-ci-label extra`) to additionally apply the "run-ci-extra" label, opting the PR into the extra test workflow (`pr-test-extra.yml`). - `/rerun-failed-ci`: Reruns workflows from the latest commit with conclusion **failed, skipped, cancelled, or timed out**. - `/tag-and-rerun-ci`: Runs both. Use this on a fresh PR to kick off CI on the current commit — `/tag-run-ci-label` alone won't. Accepts the same `extra` argument (`/tag-and-rerun-ci extra`). -- `/rerun-test [ ...]`: Reruns one or more specific tests directly. A spec may select a file, class, or method using `::[.]`. Multiple specs and file globs are supported. Examples: `/rerun-test test_srt_endpoint.py`, `/rerun-test registered/core/test_srt_endpoint.py::TestSRTEndpoint.test_simple_decode`, `/rerun-test test_a.py test_b.py`, or `/rerun-test test_*backend*.py`. -- `/rerun-group [ ...]`: Expands one or more registered test groups (for example, `/rerun-group hicache`) and dispatches their tests through the same selective-rerun workflow. +- `/run-full-ci`: Baseline **and** extra CI. Short form of `/tag-and-rerun-ci extra` — applies both labels, then reruns everything at the current commit that didn't succeed. +- `/run-extra-ci`: Extra CI **only**. Applies both labels, then reruns just `PR Test Extra`, leaving baseline CI runs alone. Use it when baseline is already green and you only need the extra tier. +- `/rerun-test [ ...]`: Reruns one or more specific tests directly. A spec may select a file, class, or method using `::[.]`. Multiple specs and file globs are supported. `--changed` (short form `-c`) adds every test file the PR itself adds or modifies (under `test/registered/` or the multimodal test directory), so a PR that touches several tests can rerun them all without listing them. Examples: `/rerun-test test_srt_endpoint.py`, `/rerun-test registered/core/test_srt_endpoint.py::TestSRTEndpoint.test_simple_decode`, `/rerun-test test_a.py test_b.py`, `/rerun-test test_*backend*.py`, `/rerun-test --changed`, or `/rerun-test -c`. +- `/rerun-group [ ...]`: Expands one or more registered test groups (for example, `/rerun-group rust-server`) and dispatches their tests through the same selective-rerun workflow. Groups are defined in [`scripts/ci/rerun_test_groups.json`](https://github.com/sgl-project/sglang/blob/main/scripts/ci/rerun_test_groups.json). The rerun commands have the following permission rules: | Command | PR author | Other users | Additional rule for fork PRs | | --- | --- | --- | --- | | `/rerun-failed-ci` | Always allowed on their own PR | Must have `can_rerun_failed_ci` in `CI_PERMISSIONS.json` | None | +| `/run-full-ci`, `/run-extra-ci` | Not auto-allowed -- these apply labels, which the PR-author allowance does not cover | Must have both `can_tag_run_ci_label` and `can_rerun_failed_ci` in `CI_PERMISSIONS.json` (every listed user has both) | None | | `/rerun-test`, `/rerun-group` | No special allowance; judged as any other commenter | Must have `cooldown_interval_minutes: 0` in `CI_PERMISSIONS.json`, or `write`/`admin` repository permission | None -- the same rule applies wherever the PR comes from | @@ -138,10 +141,27 @@ Selective reruns do not build or install a PR-local `sglang-kernel` wheel. If a If you have permission, the [Slash Command Handler](https://github.com/sgl-project/sglang/actions/workflows/slash-command-handler.yml) will run your command and react with a 👍 to your comment. It may take up to a few minutes for the reaction to appear. Here’s a usage [example](https://github.com/sgl-project/sglang/pull/14253#issuecomment-3599509302). +If a command isn't recognized, the handler reacts 😕 instead, and replies with the closest matching command when there is one. A comment with no reaction at all means the handler never saw it — check the command is spelled as listed above. + To avoid spamming a PR with too many `/rerun-failed-ci` comments, you can also trigger the command by editing an existing comment and adding any suffix (e.g., `/rerun-failed-ci try again`). If you don’t have permission and you’re not the PR author, please ask maintainers to trigger CI for you. +### CI control labels + +Four labels change how a PR's CI runs. Label before you trigger: +`max-concurrency` fixes the shard fan-out when the run is created. + +| Label | Effect | +| --- | --- | +| `bypass-fail-fast` | A job failing no longer aborts its siblings; the run continues instead of stopping at the first root-cause failure. A failing `lint` still stops everything. | +| `parallel-stages` | Stages stop waiting on each other. `base-a`, `base-b` and `base-c` dispatch together, the way a scheduled run does. | +| `max-concurrency` | A suite fans out to all of its shards at once rather than a third of them. | +| `highest-priority` | All three of the above, and the PR is also skipped by the batch-cancel workflow, sorted first in the active-runs report, and never closed as stale. | + +Each spends extra runner capacity, so they unblock a specific PR rather than +serve as a default. + ### CI rate limits Due to CI scheduling and limited resources, higher-priority PRs may preempt running jobs. In such cases, you may need to rerun the tests. @@ -201,9 +221,9 @@ Develop SGLang's customized DeepGEMM package on the [`dev` branch of `sgl-projec ### Update sgl-deep-ep -Develop `sgl-deep-ep` in [`sgl-project/DeepEP`](https://github.com/sgl-project/DeepEP). Use the implementation branch for the target platform: `sgl-deepep` for CUDA 13 on x86_64 or aarch64, `sgl-deepep-cu12-x86` for CUDA 12.9 on x86_64, or `sgl-deepep-cu12-arm` for CUDA 12.9 on aarch64. Merge packaging changes into `sgl-deepep-packaging`. The [`sgl-deep-ep` README](https://github.com/sgl-project/DeepEP/blob/sgl-deepep-packaging/sgl_deep_ep/README.md) describes the platform prerequisites and release matrix. +Develop `sgl-deep-ep` in [`sgl-project/DeepEP`](https://github.com/sgl-project/DeepEP). Use `sgl-deepep`, the implementation branch for CUDA 13 on x86_64 or aarch64. Merge packaging changes into `sgl-deepep-packaging`. The [`sgl-deep-ep` README](https://github.com/sgl-project/DeepEP/blob/sgl-deepep-packaging/sgl_deep_ep/README.md) describes the platform prerequisites and release matrix. -To validate locally, check out the selected implementation branch as `DeepEP-source` and the packaging branch as `DeepEP-packaging`. Install the required build dependencies first; CUDA 12.9 builds also require GDRCopy. The following CUDA 13 example builds a wheel for the host architecture, installs that exact wheel, and verifies that its guarded package import succeeds: +To validate locally, check out the selected implementation branch as `DeepEP-source` and the packaging branch as `DeepEP-packaging`. Install the required build dependencies first. The following example builds a wheel for the host architecture, installs that exact wheel, and verifies that its guarded package import succeeds: ```bash DEEPEP_OUTPUT_DIR="$(mktemp -d)" @@ -218,17 +238,13 @@ python3 -m pip install --force-reinstall --no-deps \ python3 -c "import deep_ep; print(deep_ep.__file__)" ``` -Use `12.9` instead of `13.0` for a CUDA 12.9 build. The import check validates packaging and binary loading, but not communication correctness. On a configured multi-GPU host, also run the test appropriate for the implementation branch: +The import check validates packaging and binary loading, but not communication correctness. On a configured multi-GPU host, also run: ```bash -# CUDA 13 implementation branch python3 DeepEP-source/tests/elastic/test_ep.py --num-processes 8 - -# CUDA 12.9 implementation branches -python3 DeepEP-source/tests/test_intranode.py --num-processes 8 ``` -Adjust `--num-processes` to the available GPUs and run the internode or low-latency tests when those transports changed. After local validation, ask the SGLang team to run the [sgl-deep-ep release workflow](https://github.com/sgl-project/sglang/actions/workflows/release-whl-deepep.yml) with the new version, CUDA target, and packaging ref. After verifying the published wheels for the supported Python versions and architectures, update the `sgl-deep-ep` pin in `python/pyproject.toml` before landing dependent SGLang changes. +Adjust `--num-processes` to the available GPUs and run the internode or low-latency tests when those transports changed. After local validation, ask the SGLang team to run the [sgl-deep-ep release workflow](https://github.com/sgl-project/sglang/actions/workflows/release-whl-deepep.yml) with the new version and packaging ref. After verifying the published wheels for the supported Python versions and architectures, update the `sgl-deep-ep` pin in `python/pyproject.toml` before landing dependent SGLang changes. ## Tips for newcomers diff --git a/docs/docs/developer_guide/development_guide_using_docker.mdx b/docs/docs/developer_guide/development_guide_using_docker.mdx index 70e69aa28b72..0e94bccd0860 100644 --- a/docs/docs/developer_guide/development_guide_using_docker.mdx +++ b/docs/docs/developer_guide/development_guide_using_docker.mdx @@ -108,7 +108,7 @@ docker exec -it sglang_zhyncs /bin/zsh ```bash Command # Change batch size, input, output and add `disable-cuda-graph` (for easier analysis) # e.g. DeepSeek V3 -nsys profile -o deepseek_v3 python3 -m sglang.bench_one_batch --batch-size 1 --input 128 --output 256 --model deepseek-ai/DeepSeek-V3 --trust-remote-code --tp 8 --disable-cuda-graph +nsys profile -o deepseek_v3 python3 -m sglang.benchmark.one_batch --batch-size 1 --input 128 --output 256 --model deepseek-ai/DeepSeek-V3 --trust-remote-code --tp 8 --disable-cuda-graph ``` ## Evaluation diff --git a/docs/docs/developer_guide/evaluating_new_models.mdx b/docs/docs/developer_guide/evaluating_new_models.mdx index 34ad0a869884..d06ba18e54b0 100644 --- a/docs/docs/developer_guide/evaluating_new_models.mdx +++ b/docs/docs/developer_guide/evaluating_new_models.mdx @@ -40,7 +40,7 @@ python -m sglang.test.run_eval \ ```bash Command python benchmark/hellaswag/bench_sglang.py \ - --host http://127.0.0.1 \ + --host 127.0.0.1 \ --port 30000 \ --num-questions 200 \ --num-shots 20 diff --git a/docs/docs/get-started/install.mdx b/docs/docs/get-started/install.mdx index 669bcf482263..b7d715566977 100644 --- a/docs/docs/get-started/install.mdx +++ b/docs/docs/get-started/install.mdx @@ -29,15 +29,11 @@ uv pip install --prerelease=allow sglang Some of SGLang's dependencies only publish pre-releases on PyPI, so without `--prerelease=allow` uv older than 0.12.0 silently installs SGLang 0.5.9. On [uv 0.12.0](https://github.com/astral-sh/uv/releases/tag/0.12.0) and newer the flag is a harmless no-op. -The major version of Cuda is 13 by default. To install sglang under Cuda 12 with pip or uv, please try the following commands: -```bash Command -pip install --upgrade pip -pip install uv -uv pip install --prerelease=allow sglang -uv pip install --force-reinstall torch==2.13.0 torchaudio==2.11.0 torchvision --index-url https://download.pytorch.org/whl/cu129 -uv pip install --force-reinstall sglang-kernel --index-url https://docs.sglang.ai/whl/cu129/ -uv pip install --force-reinstall sgl-deep-gemm --index-url https://docs.sglang.ai/whl/cu129/ --no-deps -``` + +SGLang requires CUDA 13. The CUDA 12 (`cu129`) wheels and images are retired, +because PyTorch 2.14 publishes no CUDA 12.9 builds. SGLang 0.5.19 is the last +release with a CUDA 12 lane. + ### Nightly builds @@ -49,13 +45,6 @@ pip install uv uv pip install --prerelease=allow --index-strategy unsafe-best-match --extra-index-url https://docs.sglang.ai/whl/cu130/ sglang ``` -To install a nightly build under Cuda 12, swap the index to `cu129`: -```bash Command -pip install --upgrade pip -pip install uv -uv pip install --prerelease=allow --index-strategy unsafe-best-match --extra-index-url https://docs.sglang.ai/whl/cu129/ sglang -``` - ### Quick fixes to common problems - If you encounter `OSError: CUDA_HOME environment variable is not set`. Please set it to your CUDA install root with either of the following solutions: 1. Use `export CUDA_HOME=/usr/local/cuda-` to set the `CUDA_HOME` environment variable. @@ -113,7 +102,7 @@ docker run --gpus all \ You can also find the nightly docker images [here](https://hub.docker.com/r/lmsysorg/sglang/tags?name=nightly). Notes: -- SGLang is shipped with CUDA 13 environment by default. To run SGLang on CUDA 12 environment, please use images with `-cu12` or `-cu129` suffix, such as `lmsysorg/sglang:latest-cu129` or `lmsysorg/sglang:dev-cu12`. +- SGLang images ship a CUDA 13 environment. The `-cu12` / `-cu129` images are no longer published; `lmsysorg/sglang:v0.5.19-cu129` is the last CUDA 12 tag. ## Method 4: Using Kubernetes diff --git a/docs/docs/get-started/quickstart.mdx b/docs/docs/get-started/quickstart.mdx index 4cf766a0337e..57773eaed4f3 100644 --- a/docs/docs/get-started/quickstart.mdx +++ b/docs/docs/get-started/quickstart.mdx @@ -61,7 +61,7 @@ pip install -e "python" --env "HF_TOKEN=" \ --ipc=host \ lmsysorg/sglang:latest \ - python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --host 0.0.0.0 --port 30000 + sglang serve --model-path meta-llama/Llama-3.1-8B-Instruct --host 0.0.0.0 --port 30000 ``` For production deployments, use the smaller **runtime** variant (~40% size reduction): @@ -74,7 +74,7 @@ pip install -e "python" --env "HF_TOKEN=" \ --ipc=host \ lmsysorg/sglang:latest-runtime \ - python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --host 0.0.0.0 --port 30000 + sglang serve --model-path meta-llama/Llama-3.1-8B-Instruct --host 0.0.0.0 --port 30000 ``` @@ -93,7 +93,7 @@ export CUDA_HOME=/usr/local/cuda- Start the SGLang server with a model. Here we use `qwen/qwen2.5-0.5b-instruct` as a lightweight example: ```bash -python3 -m sglang.launch_server --model-path qwen/qwen2.5-0.5b-instruct --host 0.0.0.0 --port 30000 +sglang serve --model-path qwen/qwen2.5-0.5b-instruct --host 0.0.0.0 --port 30000 ``` Wait until you see `The server is fired up and ready to roll!` in the terminal output. diff --git a/docs/docs/hardware-platforms/ascend-npus/development/contribution_guide.mdx b/docs/docs/hardware-platforms/ascend-npus/development/contribution_guide.mdx index 92d7d539712d..acaf5e683cf7 100644 --- a/docs/docs/hardware-platforms/ascend-npus/development/contribution_guide.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/development/contribution_guide.mdx @@ -122,7 +122,6 @@ Also, do not rely on the "Latency/Output throughput" from this script, as it is GSM8K is too easy for state-of-the-art models nowadays. Please try your own more challenging accuracy tests. You can find additional accuracy eval examples in: - [test_eval_accuracy_large.py](https://github.com/sgl-project/sglang/blob/main/test/manual/eval/test_eval_accuracy_large.py) -- [test_gpt_oss_1gpu.py](https://github.com/sgl-project/sglang/blob/main/test/manual/core/test_gpt_oss_1gpu.py) ## Benchmark the speed diff --git a/docs/docs/hardware-platforms/ascend-npus/development/operator_development.mdx b/docs/docs/hardware-platforms/ascend-npus/development/operator_development.mdx index 979aa65eea7b..9dee53fbcb86 100644 --- a/docs/docs/hardware-platforms/ascend-npus/development/operator_development.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/development/operator_development.mdx @@ -37,7 +37,7 @@ sgl-kernel-npu/ │ ├── op_host/ # Host-side code (validation, tiling, launch) │ │ ├── .cpp │ │ └── tiling/ # Optional: tiling data -│ └── op_kernel/ # Device-side code (Ascend C kernel on AICore) +│ └── op_kernel/ # Device-side code (Ascend C kernel on AI Core) │ └── _kernel.cpp ├── include/ │ └── sgl_kenel_npu_ops.h # C++ interface declarations @@ -62,7 +62,7 @@ sgl-kernel-npu/ A complete Ascend C operator consists of two parts: -- **Device part**: Kernel code running on the NPU AICore, responsible for actual +- **Device part**: Kernel code running on the NPU AI Core, responsible for actual computation. Written using the Ascend C API. - **Host part**: Code running on the CPU, responsible for parameter validation, data pre-processing, tiling, and kernel launch. @@ -86,7 +86,7 @@ csrc// ### Step 2: Write the Device-side Kernel (op_kernel) -Device-side code runs on AICore and follows the Ascend C programming model. The +Device-side code runs on AI Core and follows the Ascend C programming model. The core structure is a class with `Init()` and `Process()` methods, plus an `extern "C"` entry function. @@ -155,7 +155,7 @@ extern "C" __global__ __aicore__ void helloworld( **Key points:** -- Class methods must be marked with `__aicore__`, indicating they run on AICore. +- Class methods must be marked with `__aicore__`, indicating they run on AI Core. - Use `AscendC::TPipe` + `AscendC::TQue` to build a pipeline that overlaps data movement and computation. - The entry function must be declared `extern "C" __global__ __aicore__`. The diff --git a/docs/docs/hardware-platforms/ascend-npus/development/operator_performance_optimizing.mdx b/docs/docs/hardware-platforms/ascend-npus/development/operator_performance_optimizing.mdx index cb2291fb5299..7ab0d2def447 100644 --- a/docs/docs/hardware-platforms/ascend-npus/development/operator_performance_optimizing.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/development/operator_performance_optimizing.mdx @@ -96,7 +96,7 @@ Below is the field-by-field breakdown of the operator performance record, aligne | Field | Value | Definition (per official docs) | |-------|-------|--------------------------------| -| Input Shapes | 163840,1024;128,1024;16384;;;;128 | **Input Shapes**: Dimensions of each input tensor, separated by semicolons; empty values represent scalar inputs. Breakdown: 7 inputs with shapes `[163840,1024]`, `[128,1024]`, `[163840]`, 3 scalars, and `[128]`. | +| Input Shapes | 163840,1024;128,1024;163840;;;;128 | **Input Shapes**: Dimensions of each input tensor, separated by semicolons; empty values represent scalar inputs. Breakdown: 7 inputs with shapes `[163840,1024]`, `[128,1024]`, `[163840]`, 3 scalars, and `[128]`. | | Input Data Types | INT32;FLOAT;FLOAT;DT_UNDEFINED;DT_UNDEFINED;DT_UNDEFINED;INT64 | **Input Data Types**: Data types of inputs, in the same order as input shapes. | | Input Formats | ND;ND;ND;NULL;NULL;NULL;ND | **Input Formats**: Memory layout of inputs; `ND` stands for N-dimensional tensor format, `NULL` corresponds to scalar/undefined inputs. | | Output Shapes | 163840,512;163840 | **Output Shapes**: Dimensions of two output tensors, separated by semicolons: `[163840,512]` and `[163840]`. | @@ -165,7 +165,7 @@ Must be vectorized. ### Tips -1. Ascend 910 series usually has only 40 or 48 vector cores. If the number of grids exceeds 40 or 48 vector cores, the grids will be delivered in a queue, resulting in a long waiting time. Therefore, the number of cores for high-performance implementation does not exceed the number of vector cores. +1. A2/A3 Series products usually have only 40 or 48 vector cores. If the number of grids exceeds 40 or 48 vector cores, the grids will be delivered in a queue, resulting in a long waiting time. Therefore, the number of cores for high-performance implementation does not exceed the number of vector cores. 2. Try to use up all the UB as much as possible. Move a large block size at a time to ensure that the bound is in the MTE. No Redundant Copy. 3. If the offset is a negative number, the current triton-ascend considers it as a discrete memory access scenario. As a result, the performance severely deteriorates, and the data is read from the entire DMA block instead of being read in scalar mode. 4. The UB of the Ascend hardware requires that the size of the tail axis of the tensor can be exactly divided by 32bytes. If the length of the tail axis is insufficient, the length of the tail axis is automatically supplemented. For example, the performance deteriorates exponentially due to automatic supplementation for the Tensor whose shape is (2048, 3). In this situation, you can perform the transposition operation to change the alignment axis to a lower dimension. In addition, the transposition operation is affected by the automatic supplement rule. Therefore, special skills are also required to avoid supplementation. diff --git a/docs/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation.mdx b/docs/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation.mdx index 265fb3315a85..5b441474ef7c 100644 --- a/docs/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/evaluation/accuracy_evaluation.mdx @@ -23,7 +23,7 @@ Ensure sufficient disk space before proceeding. The Docker image requires at lea First, launch the SGLang environment using the provided container image: - + ```shell Command export IMAGE=quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 @@ -46,7 +46,7 @@ docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \ ``` - + ```shell Command export IMAGE=quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 @@ -214,8 +214,8 @@ For more details, refer to the [EvalScope documentation](https://evalscope.readt ### SGLang Server Startup Failure -1. Verify device mapping: A2 uses `davinci[0-7]`, A3 uses `davinci[0-15]` -2. Confirm image tag matches device type: A2 uses `...-910b`, A3 uses `...-a3` +1. Verify device mapping: A2 Series uses `davinci[0-7]`, A3 Series uses `davinci[0-15]` +2. Confirm image tag matches device type: A2 Series uses `...-910b`, A3 Series uses `...-a3` 3. Check NPU status with `npu-smi info` 4. First run requires model download; set `HF_ENDPOINT=https://hf-mirror.com` if network access is restricted @@ -267,6 +267,7 @@ This is a common workaround for corporate proxy environments. If it does not res If you cannot obtain the CA certificate, download datasets manually as shown in [Download Dataset Error](#download-dataset-error) below. ### EvalScope Request Retry Timeout + If EvalScope keeps retrying requests with errors like: ``` 2026-06-22 03:09:03 - evalscope - WARNING: Attempt 4 / 5 failed: ....... Retrying... diff --git a/docs/docs/hardware-platforms/ascend-npus/evaluation/performance_testing.mdx b/docs/docs/hardware-platforms/ascend-npus/evaluation/performance_testing.mdx index 251fb5fdb598..a5e3e108e754 100644 --- a/docs/docs/hardware-platforms/ascend-npus/evaluation/performance_testing.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/evaluation/performance_testing.mdx @@ -6,7 +6,7 @@ metatags: This page walks through performance testing your SGLang deployment on Ascend NPUs. We cover three model types — text generation (`Qwen/Qwen2.5-7B-Instruct`), multimodal vision (`Qwen/Qwen2.5-VL-7B-Instruct`), and embedding (`Qwen/Qwen3-Embedding-8B`) — in both online and offline serving modes. You can use [Evalscope](https://evalscope.readthedocs.io/en/latest/), [AISBench](https://ais-bench-benchmark.readthedocs.io/en/latest/), or SGLang's built-in benchmarking tools. -The benchmark output examples in this guide are for illustration only. Actual performance depends on your hardware (e.g., Atlas 800I A2 vs A3), model version, SGLang version, and deployment configuration. Always run benchmarks on your own hardware to obtain accurate performance data. +The benchmark output examples in this guide are for illustration only. Actual performance depends on your hardware (e.g., Ascend A2 Series vs Ascend A3 Series), model version, SGLang version, and deployment configuration. Always run benchmarks on your own hardware to obtain accurate performance data. ## 1. Prepare @@ -750,7 +750,7 @@ SGLang's `Engine` API runs inference in-process, without an HTTP server, letting `bench_offline_throughput` uses the `Engine` API internally and measures pure inference throughput without HTTP overhead: ```shell Command -python -m sglang.bench_offline_throughput \ +python -m sglang.benchmark.offline_throughput \ --model-path Qwen/Qwen2.5-7B-Instruct \ --dataset-name random \ --random-input-len 1024 \ diff --git a/docs/docs/hardware-platforms/ascend-npus/faq.mdx b/docs/docs/hardware-platforms/ascend-npus/faq.mdx index 2e803d201738..8d602e828b05 100644 --- a/docs/docs/hardware-platforms/ascend-npus/faq.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/faq.mdx @@ -42,13 +42,13 @@ Thread 0x0000fff873f6f120 (most recent call first): (SGLangEngine pid=3872176) [ERROR] 2026-04-14-12:14:41 (PID:3874122, Device:0, RankID:-1) ERR00100 PTA call acl api failed (SGLangEngine pid=3872176) [Error]: An internal error occurs in the runtime module on the host. (SGLangEngine pid=3872176) Rectify the fault based on the error information in the ascend log. -(SGLangEngine pid=3872176) [PID: 3874122] 2026-04-14-12:14:41.897.548 AclNN_Runtime_Error(EZ9903): aclrtLaunchKerneWithHostArgs failed, return: 507000 +(SGLangEngine pid=3872176) [PID: 3874122] 2026-04-14-12:14:41.897.548 AclNN_Runtime_Error(EZ9903): aclrtLaunchKernelWithHostArgs failed, return: 507000 (SGLangEngine pid=3872176) Solution: In this scenario, collect the plog when the fault occurs and locate the fault based on the plog. (SGLangEngine pid=3872176) TraceBack (most recent call last): (SGLangEngine pid=3872176) Check kernel task failed, stream_id=2028, task_id=48, retCode=0x7080005.[FUNC:LaunchKernel][FILE:context.cc][LINE:1585] (SGLangEngine pid=3872176) rtsLaunchKernelWithHostArgs execution failed, reason=kernel type error[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:61] (SGLangEngine pid=3872176) rtsLaunchKernelWithHostArgs failed, runtime result = 507000.[FUNC:ReportCallError][FILE:Log_inner.cpp][LINE:148] -(SGLangEngine pid=3872176) aclrtLaunchKerneWWithHostArgs failed, return: 507000 +(SGLangEngine pid=3872176) aclrtLaunchKernelWithHostArgs failed, return: 507000 (SGLangEngine pid=3872176) Launch kernel failed. (SGLangEngine pid=3872176) #### KernelLaunch failed: /home/850b160/cann-8.5.8/opp/built-in/op_impl/ai_core/tbe//kernel/ascend910_93/ops_legacy/fill/Fill_41dadce325bOf810d03359af2a38990b_high_performance.o (SGLangEngine pid=3872176) Kernel Run failed. opType: 18, Fill @@ -109,7 +109,7 @@ prefill 7.log:3094:RuntimeError: ACL stream synchronize failed, error code:50703 [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.381.705 davinci kernel task.cc:14241170327 GetArasInfo:[AIC INFO] aras(0 to 9) after execute:0x3fffffb9000, 0, 0, 0x12c9323ff600, 0x12c93231ee00, 0x12c93f1d8800,0x12c93f3ff600,0x12c958200000,0x100000003, 0xaaaa00000001. [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.381.710 [davinci kernel task.cc:1427]170327 GetArgsInfo:tilingKey = 0, print 1 Times totalLen=(10*8), argsSize=80, blockDim=3 [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.381.717 [davinci kernel task.cc:1468]170327 PrintErrorInfoForDavinciTask:[AIC INFO] after execute:arqs print end -[ERROR] RUNTIME(164629,):2026-04-14-05:34:12.381.751 davinci kernel task.cc:14981170327 PrintErrorInfoForDavinciTask:[DFX INFO]Aicore kernel execute failed, device id=13, stream id=43, report stream id=43, task id=13145, flip num=56, fault kernel_name=alloc_extend _kernel_18, fault kernel info ext=alloc_extend_kernel, program id=141, hash=14069671779787989248. +[ERROR] RUNTIME(164629,):2026-04-14-05:34:12.381.751 davinci kernel task.cc:14981170327 PrintErrorInfoForDavinciTask:[DFX INFO]Aicore kernel execute failed, device id=13, stream id=43, report stream id=43, task id=13145, flip num=56, fault kernel_name=alloc_extend_kernel_18, fault kernel info ext=alloc_extend_kernel, program id=141, hash=14069671779787989248. [ERROR] IDEDD(164629,):2026-04-14-05:34:12.381.823 [dump manager.cpp:41][tid:170327] An exception callback message is received. [ERROR] IDEDD(164629,):2026-04-14-05:34:12.381.971 [kernel info collector.cpp:384][tid:170327] Get error register information. coreNum=0 [ERROR] IDEDD(164629,):2026-04-14-05:34:12.381.981 kernel info collector.cpp:4771tid:1703271 It is Non-SuperKernel. functionCount=1, qlobalCount=1 @@ -119,7 +119,7 @@ prefill 7.log:3094:RuntimeError: ACL stream synchronize failed, error code:50703 [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.383.911 [stream.cc:1335]170327 GetError:AIV Kernel happen error, retCode=0x31. [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.383.929 [stream.cc:1335]170327 GetError:[AIC_INFO] after execute:args print end [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.383.936 stream.cc:13351170327 GetError: DFX INFO1Aicore kernel execute failed, device id=13, stream id=43, report stream id=43, task id=13145, tlip num=56, - fault kernel_name=alloc_extend_kernel_18, ault kernel info ext=alloc_extend_kernel, program id=141, hash=14069671779787989248. + fault kernel_name=alloc_extend_kernel_18, fault kernel info ext=alloc_extend_kernel, program id=141, hash=14069671779787989248. [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.383.943 [stream.cc:3549]170327 EnterFailureAbort:stream id=43 enter failure abort. [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.383.973 [stars_engine.cc:1427]170327 StarsResumeRtsa:stop scheduling in abort failure mode: stream id=43, sq id=6,sq head=801, task id=13145, taskType=66. [ERROR] RUNTIME(164629,):2026-04-14-05:34:12.383.972 [stream.cc:1463]164629 SynchronizeExecutedTask:context is abort, status=0x715005e. diff --git a/docs/docs/hardware-platforms/ascend-npus/getting-started/installation.mdx b/docs/docs/hardware-platforms/ascend-npus/getting-started/installation.mdx index 399ec006b36f..2de76d2973aa 100644 --- a/docs/docs/hardware-platforms/ascend-npus/getting-started/installation.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/getting-started/installation.mdx @@ -42,8 +42,8 @@ Each CANN version requires its own matched set of component versions — do not MemFabric - 1.0.8 - `pip install memfabric-hybrid==1.0.8` + 1.1.4 + `pip install memfabric-hybrid==1.1.4` Triton @@ -52,13 +52,13 @@ Each CANN version requires its own matched set of component versions — do not SGLang NPU Kernel - 2026.05.01.post3 - link + 2026.9.0 + link MemFabric-zbal - 1.1.1 - `pip install memfabric-zbal==1.1.1` + 1.1.2 + `pip install memfabric-zbal==1.1.2` @@ -137,7 +137,7 @@ Ensure sufficient disk space before pulling images. Each Docker image requires a You can obtain the dependency of a specified version of CANN through an image. - + ```bash Command # CANN 9.0.0 @@ -148,7 +148,7 @@ docker pull quay.io/ascend/cann:9.1.0-a3-ubuntu22.04-py3.12 ``` - + ```bash Command # CANN 9.0.0 @@ -298,7 +298,7 @@ SGLang images for the CANN 9.1.0 set have not been published yet. For now, pleas - + ```bash Command # Stable release @@ -309,7 +309,7 @@ docker pull quay.io/ascend/sglang:main-cann9.0.0-a3 ``` - + ```bash Command # Stable release @@ -332,7 +332,7 @@ cd sglang/docker # Build the docker image # Replace with the target architecture, e.g., amd64, arm64. # Optional build arguments: -# --build-arg DEVICE_TYPE=910b # Required for Atlas 800I A2 +# --build-arg DEVICE_TYPE=910b # Required for Ascend A2 Series # --build-arg APTMIRROR= # Use a custom APT mirror to improve download speed # If there are network errors, please modify the Dockerfile to add ARG HTTP_PROXY/HTTPS_PROXY and set them as ENV. docker build --build-arg TARGETARCH= -t -f npu.Dockerfile . @@ -343,7 +343,7 @@ docker build --build-arg TARGETARCH= -t -f npu.Dockerfile __Notice:__ `--privileged` and `--network=host` are required by RDMA, which is typically needed by Ascend NPU clusters. - + ```bash Command # Create a shortcut 'drun' to launch a privileged Docker container @@ -366,7 +366,7 @@ drun --env "HF_TOKEN=" \ ``` - + ```bash Command # Create a shortcut 'drun' to launch a privileged Docker container @@ -444,7 +444,7 @@ python3 -m sglang.launch_server \ 1. Launch Prefill Server - + ```bash Command # Enabling CPU Affinity @@ -468,7 +468,7 @@ python3 -m sglang.launch_server \ ``` - + ```bash Command # Enabling CPU Affinity @@ -498,7 +498,7 @@ python3 -m sglang.launch_server \ 2. Launch Decode Server - + ```bash Command # PREFILL_IP: IP address of the first Prefill Server @@ -518,7 +518,7 @@ python3 -m sglang.launch_server \ ``` - + ```bash Command # PREFILL_IP: IP address of the first Prefill Server diff --git a/docs/docs/hardware-platforms/ascend-npus/getting-started/quick_start.mdx b/docs/docs/hardware-platforms/ascend-npus/getting-started/quick_start.mdx index 6f82fcf38739..65a696cd6dad 100644 --- a/docs/docs/hardware-platforms/ascend-npus/getting-started/quick_start.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/getting-started/quick_start.mdx @@ -14,10 +14,10 @@ see [SGLang installation with NPUs support](/docs/hardware-platforms/ascend-npus ### Supported Devices -- Atlas 800I A2 inference series (Atlas 800I A2) -- Atlas 800I A3 inference series (Atlas 800I A3) +- Ascend A2 Series Products +- Ascend A3 Series Products -To identify your device, run `npu-smi info -l`: A3 reports `Chip Count: 2` per NPU, while A2 reports `Chip Count: 1` per NPU. +To identify your device, run `npu-smi info -l`: A3 Series reports `Chip Count: 2` per NPU, while A2 Series reports `Chip Count: 1` per NPU. For hardware details, see the [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ### Docker @@ -48,7 +48,7 @@ If you have already downloaded model weights to a local path (e.g., `/path/to/mo adding `--volume /path/to/model:/path/to/model` to the `docker run` command below. - + ```shell Command # Choose one (uncomment the line you want): @@ -73,7 +73,7 @@ docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \ ``` - + ```shell Command # Choose one (uncomment the line you want): diff --git a/docs/docs/hardware-platforms/ascend-npus/mindspore_backend.mdx b/docs/docs/hardware-platforms/ascend-npus/mindspore_backend.mdx index b242eefa845c..2988c0e9febb 100644 --- a/docs/docs/hardware-platforms/ascend-npus/mindspore_backend.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/mindspore_backend.mdx @@ -33,9 +33,9 @@ pip install -e . Current SGLang-MindSpore supports Qwen3 and DeepSeek V3/R1 models. This doc uses Qwen3-8B as an example. -### Offline infer +### Offline Inference -Use the following script for offline infer: +Use the following script for offline inference: ```python Offline Inference @@ -107,7 +107,7 @@ python3 -m sglang.launch_server \ ## Troubleshooting -#### Debug Mode +### Debug Mode Enable sglang debug logging by log-level argument. @@ -132,7 +132,7 @@ export GLOG_v=0 # DEBUG ``` -#### Explicitly select devices +### Explicitly select devices Use the following environment variable to explicitly select the devices to use. @@ -142,7 +142,7 @@ export ASCEND_RT_VISIBLE_DEVICES=4,5,6,7 # to set device ``` -#### Some communication environment issues +### Some communication environment issues In case of some environment with special communication environment, users need set some environment variables. @@ -152,7 +152,7 @@ export MS_ENABLE_LCCL=off # current not support LCCL communication mode in SGLan ``` -#### Some dependencies of protobuf +### Some dependencies of protobuf In case of some environment with special protobuf version, users need set some environment variables to avoid binary version mismatch. diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_r1.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_r1.mdx index 46898890a6e9..6c2a8bb081d3 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_r1.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_r1.mdx @@ -7,25 +7,25 @@ metatags: This page focuses on optimal configuration and benchmark results for DeepSeek-R1 on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [DeepSeek-R1 Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_r1). -On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. +On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. ### Low Latency | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | -| DeepSeek-R1 | Atlas 800I A3 | 32 | PD Disaggregation | 3.5k+1.5k | 20ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-w8a8-2p1d-32p-in3k5-out1k5-20ms) | -| DeepSeek-R1 | Atlas 800I A3 | 32 | PD Disaggregation | 3.5k+1k | 19.0ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-w8a8-2p1d-32p-in3k5-out1k-19-0ms) | -| DeepSeek-R1 | Atlas 800I A3 | 32 | PD Disaggregation | 3.9k+1k | 19.0ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-w8a8-2p1d-32p-in3k9-out1k-19-0ms) | -| DeepSeek-R1 | Atlas 800I A3 | 32 | PD Disaggregation | 6k+1.6k | 20.5ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-w8a8-2p1d-32p-in6k-out1k6-20-5ms) | +| DeepSeek-R1 | Ascend A3 Series Products | 32 | PD Disaggregation | 3.5k+1.5k | 20ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-w8a8-2p1d-32p-in3k5-out1k5-20ms) | +| DeepSeek-R1 | Ascend A3 Series Products | 32 | PD Disaggregation | 3.5k+1k | 19.0ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-w8a8-2p1d-32p-in3k5-out1k-19-0ms) | +| DeepSeek-R1 | Ascend A3 Series Products | 32 | PD Disaggregation | 3.9k+1k | 19.0ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-w8a8-2p1d-32p-in3k9-out1k-19-0ms) | +| DeepSeek-R1 | Ascend A3 Series Products | 32 | PD Disaggregation | 6k+1.6k | 20.5ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-w8a8-2p1d-32p-in6k-out1k6-20-5ms) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | -| DeepSeek-R1 | Atlas 800I A3 | 16 | PD Disaggregation | 3.5k+1.5k | 50ms | W4A8 INT8 | [Optimal Configuration](#deepseek-r1-w4a8-1p1d-16p-in3k5-out1k5-50ms) | -| DeepSeek-R1 | Atlas 800I A3 | 8 | PD Mixed | 3.5k+1.5k | 50ms | W4A8 INT8 | [Optimal Configuration](#deepseek-r1-w4a8-8p-in3k5-out1k5-50ms) | -| DeepSeek-R1 | Atlas 800I A3 | 32 | PD Disaggregation | 3.5k+1.5k | 50ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-w8a8-2p1d-32p-in3k5-out1k5-50ms) | +| DeepSeek-R1 | Ascend A3 Series Products | 16 | PD Disaggregation | 3.5k+1.5k | 50ms | W4A8 INT8 | [Optimal Configuration](#deepseek-r1-w4a8-1p1d-16p-in3k5-out1k5-50ms) | +| DeepSeek-R1 | Ascend A3 Series Products | 8 | PD Mixed | 3.5k+1.5k | 50ms | W4A8 INT8 | [Optimal Configuration](#deepseek-r1-w4a8-8p-in3k5-out1k5-50ms) | +| DeepSeek-R1 | Ascend A3 Series Products | 32 | PD Disaggregation | 3.5k+1.5k | 50ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-w8a8-2p1d-32p-in3k5-out1k5-50ms) | ## Optimal Configuration @@ -33,7 +33,7 @@ On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend **Model**: DeepSeek-R1 -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 16 @@ -151,7 +151,6 @@ do export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=78 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 - export SGLANG_ENABLE_SPEC_V2=1 export TASK_QUEUE_ENABLE=1 python3 -m sglang.launch_server \ @@ -169,14 +168,13 @@ do --enable-dp-attention \ --deepep-mode low_latency \ --enable-dp-lm-head \ - --cuda-graph-bs 2 4 6 8 10 12 14 16 18 20 22 24 26 \ + --cuda-graph-bs-decode 2 4 6 8 10 12 14 16 18 20 22 24 26 \ --watchdog-timeout 9000 \ --context-length 8192 \ --speculative-algorithm NEXTN \ --speculative-num-steps 2 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 3 \ - --prefill-round-robin-balance \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --tokenizer-worker-num 4 \ @@ -234,7 +232,7 @@ python -m sglang.bench_serving \ **Model**: DeepSeek-R1 -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 8 @@ -280,7 +278,6 @@ export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=72 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 -export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_NPU_USE_MLAPO=1 export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=200 export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1 @@ -296,7 +293,7 @@ python3 -m sglang.launch_server \ --device npu \ --quantization modelslim \ --watchdog-timeout 9000 \ - --cuda-graph-bs 4 8 12 14 \ + --cuda-graph-bs-decode 4 8 12 14 \ --mem-fraction-static 0.9 \ --max-running-requests 224 \ --context-length 8188 \ @@ -342,7 +339,7 @@ python -m sglang.bench_serving \ **Model**: DeepSeek-R1 -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 32 @@ -459,7 +456,6 @@ do export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=12 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 - export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_SCHEDULER_SKIP_ALL_GATHER=1 export TASK_QUEUE_ENABLE=1 @@ -481,7 +477,7 @@ do --deepep-mode low_latency \ --enable-dp-lm-head \ --moe-dense-tp 1 \ - --cuda-graph-bs 2 4 6 \ + --cuda-graph-bs-decode 2 4 6 \ --watchdog-timeout 9000 \ --context-length 8192 \ --speculative-algorithm NEXTN \ @@ -489,7 +485,6 @@ do --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --tokenizer-worker-num 4 \ - --prefill-round-robin-balance \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --load-balance-method round_robin \ @@ -546,7 +541,7 @@ python -m sglang.bench_serving \ **Model**: DeepSeek-R1 -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 32 @@ -667,7 +662,6 @@ do export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=64 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 - export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_LM_HEAD_TP=8 export SGLANG_NPU_FUSED_MOE_MODE=1 export TASK_QUEUE_ENABLE=1 @@ -689,7 +683,7 @@ do --enable-dp-attention \ --deepep-mode low_latency \ --moe-dense-tp 1 \ - --cuda-graph-bs 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 \ + --cuda-graph-bs-decode 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 \ --watchdog-timeout 9000 \ --context-length 8192 \ --speculative-algorithm NEXTN \ @@ -697,7 +691,6 @@ do --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 2 \ --tokenizer-worker-num 4 \ - --prefill-round-robin-balance \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --load-balance-method round_robin \ @@ -754,7 +747,7 @@ python -m sglang.bench_serving \ **Model**: DeepSeek-R1 -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 32 @@ -871,7 +864,6 @@ do export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=12 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 - export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_SCHEDULER_SKIP_ALL_GATHER=1 export TASK_QUEUE_ENABLE=1 @@ -893,7 +885,7 @@ do --deepep-mode low_latency \ --enable-dp-lm-head \ --moe-dense-tp 1 \ - --cuda-graph-bs 2 4 6 \ + --cuda-graph-bs-decode 2 4 6 \ --watchdog-timeout 9000 \ --context-length 8192 \ --speculative-algorithm NEXTN \ @@ -901,7 +893,6 @@ do --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --tokenizer-worker-num 4 \ - --prefill-round-robin-balance \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --load-balance-method round_robin \ @@ -958,7 +949,7 @@ python -m sglang.bench_serving \ **Model**: DeepSeek-R1 -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 32 @@ -1075,7 +1066,6 @@ do export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=12 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 - export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_SCHEDULER_SKIP_ALL_GATHER=1 export TASK_QUEUE_ENABLE=1 @@ -1097,7 +1087,7 @@ do --deepep-mode low_latency \ --enable-dp-lm-head \ --moe-dense-tp 1 \ - --cuda-graph-bs 2 4 6 \ + --cuda-graph-bs-decode 2 4 6 \ --watchdog-timeout 9000 \ --context-length 8192 \ --speculative-algorithm NEXTN \ @@ -1105,7 +1095,6 @@ do --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --tokenizer-worker-num 4 \ - --prefill-round-robin-balance \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --load-balance-method round_robin \ @@ -1162,7 +1151,7 @@ python -m sglang.bench_serving \ **Model**: DeepSeek-R1 -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 32 @@ -1278,7 +1267,6 @@ do export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=16 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 - export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_SCHEDULER_SKIP_ALL_GATHER=1 export TASK_QUEUE_ENABLE=1 @@ -1300,14 +1288,13 @@ do --deepep-mode low_latency \ --enable-dp-lm-head \ --moe-dense-tp 1 \ - --cuda-graph-bs 2 4 6 \ + --cuda-graph-bs-decode 2 4 6 \ --watchdog-timeout 9000 \ --speculative-algorithm NEXTN \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --tokenizer-worker-num 4 \ - --prefill-round-robin-balance \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --load-balance-method round_robin \ diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_v3_2.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_v3_2.mdx index bf08bdec5cea..eedc6777bd8a 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_v3_2.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_v3_2.mdx @@ -7,21 +7,21 @@ metatags: This page focuses on optimal configuration and benchmark results for DeepSeek-V3.2 on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [DeepSeek-V3.2 Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_v3_2). -On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. +On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. ### Low Latency | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | -| DeepSeek-V3.2 | Atlas 800I A3 | 32 | PD Disaggregation | 128k+1k | 26ms | W8A8 INT8 | [Optimal Configuration](#deepseek-v3-2-w8a8-1p1d-32p-in128k-out1k-26ms) | -| DeepSeek-V3.2 | Atlas 800I A3 | 32 | PD Disaggregation | 128k+1k | 26ms | W8A8 INT8 | [Optimal Configuration](#deepseek-v3-2-w8a8-1p1d-32p-in128k-out1k-bs8) | +| DeepSeek-V3.2 | Ascend A3 Series Products | 32 | PD Disaggregation | 128k+1k | 26ms | W8A8 INT8 | [Optimal Configuration](#deepseek-v3-2-w8a8-1p1d-32p-in128k-out1k-26ms) | +| DeepSeek-V3.2 | Ascend A3 Series Products | 32 | PD Disaggregation | 128k+1k | 26ms | W8A8 INT8 | [Optimal Configuration](#deepseek-v3-2-w8a8-1p1d-32p-in128k-out1k-bs8) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | -| DeepSeek-V3.2 | Atlas 800I A3 | 32 | PD Disaggregation | 128k+1k | 107ms | W8A8 INT8 | [Optimal Configuration](#deepseek-v3-2-w8a8-1p1d-32p-in128k-out1k-bs16) | +| DeepSeek-V3.2 | Ascend A3 Series Products | 32 | PD Disaggregation | 128k+1k | 107ms | W8A8 INT8 | [Optimal Configuration](#deepseek-v3-2-w8a8-1p1d-32p-in128k-out1k-bs16) | ## Optimal Configuration @@ -31,7 +31,7 @@ On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend **Model**: DeepSeek-V3.2 -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 32 @@ -147,7 +147,6 @@ do export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=8 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 - export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_SCHEDULER_SKIP_ALL_GATHER=1 export TASK_QUEUE_ENABLE=0 @@ -229,7 +228,7 @@ python -m sglang.bench_serving \ **Model**: DeepSeek-V3.2 -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 32 @@ -345,7 +344,6 @@ do export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=8 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 - export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_SCHEDULER_SKIP_ALL_GATHER=1 export TASK_QUEUE_ENABLE=0 @@ -426,7 +424,7 @@ python -m sglang.bench_serving \ **Model**: DeepSeek-V3.2 -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 32 @@ -542,7 +540,6 @@ do export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=8 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 - export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_SCHEDULER_SKIP_ALL_GATHER=1 export TASK_QUEUE_ENABLE=0 diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_v4_flash.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_v4_flash.mdx index 63a3b6628b66..6e8ff2e502cf 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_v4_flash.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_v4_flash.mdx @@ -7,16 +7,16 @@ metatags: This page focuses on optimal configuration and benchmark results for DeepSeek-V4-Flash on the Ascend NPU. -On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. +On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | -| DeepSeek-V4-Flash | Atlas 800I A3 | 16 | PD Disaggregation | 8k+1k | 50ms | W8A8 INT8 | [Optimal Configuration](#deepseek-v4-flash-w8a8-1p1d-16p-in8k-out1k-50ms) | -| DeepSeek-V4-Flash | Atlas 800I A3 | 8 | PD Mixed | 32k+1k | 50ms | W8A8 INT8 | [Optimal Configuration](#deepseek-v4-flash-w8a8-8p-in32k-out1k-50ms) | -| DeepSeek-V4-Flash | Atlas 800I A3 | 8 | PD Mixed | 8k+1k | 50ms | W8A8 INT8 | [Optimal Configuration](#deepseek-v4-flash-w8a8-8p-in8k-out1k-50ms) | +| DeepSeek-V4-Flash | Ascend A3 Series Products | 16 | PD Disaggregation | 8k+1k | 50ms | W8A8 INT8 | [Optimal Configuration](#deepseek-v4-flash-w8a8-1p1d-16p-in8k-out1k-50ms) | +| DeepSeek-V4-Flash | Ascend A3 Series Products | 8 | PD Mixed | 32k+1k | 50ms | W8A8 INT8 | [Optimal Configuration](#deepseek-v4-flash-w8a8-8p-in32k-out1k-50ms) | +| DeepSeek-V4-Flash | Ascend A3 Series Products | 8 | PD Mixed | 8k+1k | 50ms | W8A8 INT8 | [Optimal Configuration](#deepseek-v4-flash-w8a8-8p-in8k-out1k-50ms) | ## Optimal Configuration @@ -26,7 +26,7 @@ On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend **Model**: DeepSeek-V4-Flash -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 16 @@ -75,7 +75,6 @@ export INF_NAN_MODE_FORCE_DISABLE=1 export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DSV4_FP4_EXPERTS=False export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 -export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_OPT_BF16_FP32_GEMM_ALGO=torch export SGLANG_OPT_DEEPGEMM_HC_PRENORM=False export SGLANG_OPT_FP8_WO_A_GEMM=0 @@ -185,7 +184,7 @@ do --quantization modelslim \ --enable-dp-lm-head \ --kv-cache-dtype bfloat16 \ - --cuda-graph-bs 1 2 4 8 16 24 36 40 48 56 \ + --cuda-graph-bs-decode 1 2 4 8 16 24 36 40 48 56 \ --speculative-algorithm EAGLE \ --speculative-num-steps 2 \ --speculative-eagle-topk 1 \ @@ -237,7 +236,7 @@ python -m sglang.bench_serving \ **Model**: DeepSeek-V4-Flash -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 8 @@ -289,7 +288,6 @@ export SGLANG_DSV4_FP4_EXPERTS=False export SGLANG_DSV4_NPU_FUSED_COMPRESSOR=1 export SGLANG_DSV4_NPU_FUSED_COMPRESSOR_PREFILL=0 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 -export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 export SGLANG_NPU_USE_MULTI_STREAM=1 export SGLANG_OPT_BF16_FP32_GEMM_ALGO=torch @@ -329,7 +327,7 @@ python3 -m sglang.launch_server \ --enable-dp-lm-head \ --kv-cache-dtype auto \ --skip-server-warmup \ - --cuda-graph-bs 1 2 4 8 \ + --cuda-graph-bs-decode 1 2 4 8 \ --speculative-algorithm EAGLE \ --speculative-num-steps 2 \ --speculative-eagle-topk 1 \ @@ -363,7 +361,7 @@ python -m sglang.bench_serving \ **Model**: DeepSeek-V4-Flash -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 8 @@ -415,7 +413,6 @@ export SGLANG_DSV4_FP4_EXPERTS=False export SGLANG_DSV4_NPU_FUSED_COMPRESSOR=1 export SGLANG_DSV4_NPU_FUSED_COMPRESSOR_PREFILL=1 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 -export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 export SGLANG_NPU_USE_MULTI_STREAM=1 export SGLANG_OPT_BF16_FP32_GEMM_ALGO=torch @@ -455,7 +452,7 @@ python3 -m sglang.launch_server \ --enable-dp-lm-head \ --kv-cache-dtype auto \ --skip-server-warmup \ - --cuda-graph-bs 1 2 4 8 10 \ + --cuda-graph-bs-decode 1 2 4 8 10 \ --speculative-algorithm EAGLE \ --speculative-num-steps 2 \ --speculative-eagle-topk 1 \ diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/glm_5_1.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/glm_5_1.mdx index 04472d42f680..276fa9080a15 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/glm_5_1.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/glm_5_1.mdx @@ -7,26 +7,26 @@ metatags: This page focuses on optimal configuration and benchmark results for GLM-5.1 on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [GLM-5.1 Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_1). -On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. +On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. ### Low Latency | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | TTFT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | --- | -| GLM-5.1 | Atlas 800I A3 | 32 | PD Disaggregation | 65k+1.5k (90% prefix cache hit rate) | 25ms | - | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-1p1d-32p-in65k-out1k5-prefix90-25ms) | +| GLM-5.1 | Ascend A3 Series Products | 32 | PD Disaggregation | 65k+1.5k (90% prefix cache hit rate) | 25ms | - | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-1p1d-32p-in65k-out1k5-prefix90-25ms) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | TTFT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | --- | -| GLM-5.1 | Atlas 800I A3 | 16 | PD Mixed | 3.5k+1.5k | 50ms | - | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-16p-in3k5-out1k5-50ms) | -| GLM-5.1 | Atlas 800I A3 | 32 | PD Disaggregation | 128k+1k | 56.4ms | 13.1s | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-1p1d-32p-in128k-out1k-56-4ms) | -| GLM-5.1 | Atlas 800I A3 | 32 | PD Disaggregation | 16k+1k | 50ms | - | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-1p1d-32p-in16k-out1k-50ms) | -| GLM-5.1 | Atlas 800I A3 | 32 | PD Disaggregation | 64k+1k | 55.2ms | 7.58s | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-1p1d-32p-in64k-out1k-55-2ms) | -| GLM-5.1 | Atlas 800I A3 | 32 | PD Disaggregation | 64k+1k | 50ms | - | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-1p1d-32p-in64k-out1k-50ms) | -| GLM-5.1 | Atlas 800I A3 | 48 | PD Disaggregation | 128k+1k (90% prefix cache hit rate) | 50ms | - | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-2p1d-48p-in128k-out1k-prefix90-50ms) | -| GLM-5.1 | Atlas 800I A3 | 48 | PD Disaggregation | 64k+1k (90% prefix cache hit rate) | 50ms | - | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-4p1d-48p-in64k-out1k-prefix90-50ms) | +| GLM-5.1 | Ascend A3 Series Products | 16 | PD Mixed | 3.5k+1.5k | 50ms | - | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-16p-in3k5-out1k5-50ms) | +| GLM-5.1 | Ascend A3 Series Products | 32 | PD Disaggregation | 128k+1k | 56.4ms | 13.1s | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-1p1d-32p-in128k-out1k-56-4ms) | +| GLM-5.1 | Ascend A3 Series Products | 32 | PD Disaggregation | 16k+1k | 50ms | - | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-1p1d-32p-in16k-out1k-50ms) | +| GLM-5.1 | Ascend A3 Series Products | 32 | PD Disaggregation | 64k+1k | 55.2ms | 7.58s | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-1p1d-32p-in64k-out1k-55-2ms) | +| GLM-5.1 | Ascend A3 Series Products | 32 | PD Disaggregation | 64k+1k | 50ms | - | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-1p1d-32p-in64k-out1k-50ms) | +| GLM-5.1 | Ascend A3 Series Products | 48 | PD Disaggregation | 128k+1k (90% prefix cache hit rate) | 50ms | - | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-2p1d-48p-in128k-out1k-prefix90-50ms) | +| GLM-5.1 | Ascend A3 Series Products | 48 | PD Disaggregation | 64k+1k (90% prefix cache hit rate) | 50ms | - | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-4p1d-48p-in64k-out1k-prefix90-50ms) | ## Optimal Configuration @@ -36,7 +36,7 @@ On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend **Model**: GLM-5.1 -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 16 @@ -83,7 +83,6 @@ export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32 export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 -export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 @@ -155,7 +154,7 @@ python -m sglang.bench_serving \ **Model**: GLM-5.1 -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 32 @@ -281,7 +280,6 @@ do export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=16 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 - export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_NPU_USE_MULTI_STREAM=1 export SGLANG_SPEC_ENABLE_OVERLAP_REFLOW=1 export TASK_QUEUE_ENABLE=0 @@ -307,12 +305,11 @@ do --served-model-name glm-5 \ --moe-a2a-backend deepep \ --deepep-mode low_latency \ - --cuda-graph-bs 1 2 3 \ + --cuda-graph-bs-decode 1 2 3 \ --disaggregation-transfer-backend ascend \ --watchdog-timeout 9000 \ --context-length 180000 \ --tokenizer-worker-num 16 \ - --prefill-round-robin-balance \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --load-balance-method round_robin \ @@ -364,7 +361,7 @@ python -m sglang.bench_serving \ **Model**: GLM-5.1 -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 32 @@ -483,7 +480,6 @@ do export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=64 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 - export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_SPEC_ENABLE_OVERLAP_REFLOW=1 export TASK_QUEUE_ENABLE=0 @@ -507,12 +503,11 @@ do --served-model-name glm-5 \ --moe-a2a-backend deepep \ --deepep-mode low_latency \ - --cuda-graph-bs 1 2 3 \ + --cuda-graph-bs-decode 1 2 3 \ --disaggregation-transfer-backend ascend \ --watchdog-timeout 9000 \ --context-length 180000 \ --tokenizer-worker-num 4 \ - --prefill-round-robin-balance \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --load-balance-method round_robin \ @@ -567,7 +562,7 @@ python -m sglang.bench_serving \ **Model**: GLM-5.1 -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 32 @@ -694,7 +689,6 @@ do export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=16 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 - export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_NPU_USE_MULTI_STREAM=1 export SGLANG_SPEC_ENABLE_OVERLAP_REFLOW=1 export TASK_QUEUE_ENABLE=0 @@ -719,12 +713,11 @@ do --served-model-name glm-5 \ --moe-a2a-backend deepep \ --deepep-mode low_latency \ - --cuda-graph-bs 1 2 3 \ + --cuda-graph-bs-decode 1 2 3 \ --disaggregation-transfer-backend ascend \ --watchdog-timeout 9000 \ --context-length 180000 \ --tokenizer-worker-num 16 \ - --prefill-round-robin-balance \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --load-balance-method round_robin \ @@ -777,7 +770,7 @@ python -m sglang.bench_serving \ **Model**: GLM-5.1 -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 32 @@ -921,12 +914,11 @@ do --served-model-name glm-5 \ --moe-a2a-backend deepep \ --deepep-mode low_latency \ - --cuda-graph-bs 1 2 3 \ + --cuda-graph-bs-decode 1 2 3 \ --disaggregation-transfer-backend ascend \ --watchdog-timeout 9000 \ --context-length 180000 \ --tokenizer-worker-num 16 \ - --prefill-round-robin-balance \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --load-balance-method round_robin \ @@ -977,7 +969,7 @@ python -m sglang.bench_serving \ **Model**: GLM-5.1 -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 32 @@ -1096,7 +1088,6 @@ do export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=64 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 - export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_SPEC_ENABLE_OVERLAP_REFLOW=1 export TASK_QUEUE_ENABLE=0 @@ -1120,12 +1111,11 @@ do --served-model-name glm-5 \ --moe-a2a-backend deepep \ --deepep-mode low_latency \ - --cuda-graph-bs 1 2 3 \ + --cuda-graph-bs-decode 1 2 3 \ --disaggregation-transfer-backend ascend \ --watchdog-timeout 9000 \ --context-length 180000 \ --tokenizer-worker-num 4 \ - --prefill-round-robin-balance \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --load-balance-method round_robin \ @@ -1185,7 +1175,7 @@ python -m sglang.bench_serving \ **Model**: GLM-5.1 -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 48 @@ -1306,7 +1296,6 @@ do export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=24 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 - export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_SPEC_ENABLE_OVERLAP_REFLOW=1 export TASK_QUEUE_ENABLE=0 @@ -1330,12 +1319,11 @@ do --served-model-name glm-5 \ --moe-a2a-backend deepep \ --deepep-mode low_latency \ - --cuda-graph-bs 1 2 3 4 5 6 \ + --cuda-graph-bs-decode 1 2 3 4 5 6 \ --disaggregation-transfer-backend ascend \ --watchdog-timeout 9000 \ --context-length 180000 \ --tokenizer-worker-num 32 \ - --prefill-round-robin-balance \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --load-balance-method round_robin \ @@ -1396,7 +1384,7 @@ python -m sglang.bench_serving \ **Model**: GLM-5.1 -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 48 @@ -1516,7 +1504,6 @@ do export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=40 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 - export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_SPEC_ENABLE_OVERLAP_REFLOW=1 export TASK_QUEUE_ENABLE=0 @@ -1540,12 +1527,11 @@ do --served-model-name glm-5 \ --moe-a2a-backend deepep \ --deepep-mode low_latency \ - --cuda-graph-bs 1 2 3 4 5 6 7 8 9 10 \ + --cuda-graph-bs-decode 1 2 3 4 5 6 7 8 9 10 \ --disaggregation-transfer-backend ascend \ --watchdog-timeout 9000 \ --context-length 180000 \ --tokenizer-worker-num 4 \ - --prefill-round-robin-balance \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --load-balance-method round_robin \ diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/glm_5_2.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/glm_5_2.mdx index 50334f60dd34..f74bfc74549d 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/glm_5_2.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/glm_5_2.mdx @@ -7,14 +7,14 @@ metatags: This page focuses on optimal configuration and benchmark results for GLM-5.2 on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [GLM-5.2 Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_2). -On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. +On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | -| GLM-5.2 | Atlas 800I A3 | 32 | PD Disaggregation | 16k+1k | 50ms | W4A8 INT8 | [Optimal Configuration](#glm-5-2-w4a8-3p1d-32p-in16k-out1k-50ms) | +| GLM-5.2 | Ascend A3 Series Products | 32 | PD Disaggregation | 16k+1k | 50ms | W4A8 INT8 | [Optimal Configuration](#glm-5-2-w4a8-3p1d-32p-in16k-out1k-50ms) | ## Optimal Configuration @@ -24,7 +24,7 @@ On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend **Model**: GLM-5.2 -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 32 @@ -148,7 +148,6 @@ do export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=40 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 - export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_NPU_USE_MULTI_STREAM=1 export SGLANG_SPEC_ENABLE_OVERLAP_REFLOW=1 export TASK_QUEUE_ENABLE=0 @@ -171,12 +170,11 @@ do --served-model-name glm-5 \ --moe-a2a-backend deepep \ --deepep-mode low_latency \ - --cuda-graph-max-bs 8 \ + --cuda-graph-max-bs-decode 8 \ --disaggregation-transfer-backend ascend \ --watchdog-timeout 9000 \ --context-length 180000 \ --tokenizer-worker-num 8 \ - --prefill-round-robin-balance \ --disable-shared-experts-fusion \ --dtype bfloat16 \ --load-balance-method round_robin \ diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/kimi_k2_6.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/kimi_k2_6.mdx index 56bee38e73cf..5419b1792a21 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/kimi_k2_6.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/kimi_k2_6.mdx @@ -7,27 +7,27 @@ metatags: This page focuses on optimal configuration and benchmark results for Kimi-K2.6 on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [Kimi-K2.6 Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/kimi_k2_6). -On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. +On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. ### Low Latency | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | TTFT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | --- | -| Kimi-K2.6 | Atlas 800I A3 | 8 | PD Mixed | 3.5k+1.5k | 20ms | - | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-8p-in3k5-out1k5-20ms) | +| Kimi-K2.6 | Ascend A3 Series Products | 8 | PD Mixed | 3.5k+1.5k | 20ms | - | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-8p-in3k5-out1k5-20ms) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | TTFT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | --- | -| Kimi-K2.6 | Atlas 800I A3 | 16 | PD Mixed | 64k+1k | 100ms | - | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-16p-in64k-out1k-100ms) | -| Kimi-K2.6 | Atlas 800I A3 | 16 | PD Disaggregation | 128k+1k | 100ms | - | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-1p1d-16p-in128k-out1k-100ms) | -| Kimi-K2.6 | Atlas 800I A3 | 16 | PD Disaggregation | 128k+1k (90% prefix cache hit rate) | 100ms | 5s | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-1p1d-16p-in128k-out1k-prefix90-100ms) | -| Kimi-K2.6 | Atlas 800I A3 | 16 | PD Disaggregation | 64k+1.5k | 100ms | - | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-1p1d-16p-in64k-out1k5-100ms) | -| Kimi-K2.6 | Atlas 800I A3 | 16 | PD Disaggregation | 64k+1.5k (90% prefix cache hit rate) | 100ms | 3s | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-1p1d-16p-in64k-out1k5-prefix90-100ms) | -| Kimi-K2.6 | Atlas 800I A3 | 8 | PD Mixed | 1024x1024 (30)+1024 | 50ms | - | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-8p-in1024x1024-30-out1024-50ms) | -| Kimi-K2.6 | Atlas 800I A3 | 8 | PD Mixed | 1080p_30+256 | 50ms | - | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-8p-in1080p-30-out256-50ms) | -| Kimi-K2.6 | Atlas 800I A3 | 8 | PD Mixed | 3.5k+1.5k | 50ms | - | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-8p-in3k5-out1k5-50ms) | +| Kimi-K2.6 | Ascend A3 Series Products | 16 | PD Mixed | 64k+1k | 100ms | - | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-16p-in64k-out1k-100ms) | +| Kimi-K2.6 | Ascend A3 Series Products | 16 | PD Disaggregation | 128k+1k | 100ms | - | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-1p1d-16p-in128k-out1k-100ms) | +| Kimi-K2.6 | Ascend A3 Series Products | 16 | PD Disaggregation | 128k+1k (90% prefix cache hit rate) | 100ms | 5s | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-1p1d-16p-in128k-out1k-prefix90-100ms) | +| Kimi-K2.6 | Ascend A3 Series Products | 16 | PD Disaggregation | 64k+1.5k | 100ms | - | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-1p1d-16p-in64k-out1k5-100ms) | +| Kimi-K2.6 | Ascend A3 Series Products | 16 | PD Disaggregation | 64k+1.5k (90% prefix cache hit rate) | 100ms | 3s | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-1p1d-16p-in64k-out1k5-prefix90-100ms) | +| Kimi-K2.6 | Ascend A3 Series Products | 8 | PD Mixed | 1024x1024 (30)+1024 | 50ms | - | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-8p-in1024x1024-30-out1024-50ms) | +| Kimi-K2.6 | Ascend A3 Series Products | 8 | PD Mixed | 1080p_30+256 | 50ms | - | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-8p-in1080p-30-out256-50ms) | +| Kimi-K2.6 | Ascend A3 Series Products | 8 | PD Mixed | 3.5k+1.5k | 50ms | - | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-8p-in3k5-out1k5-50ms) | ## Optimal Configuration @@ -37,7 +37,7 @@ On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend **Model**: Kimi-K2.6 -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 16 @@ -123,7 +123,7 @@ do --dp-size 32 \ --moe-a2a-backend deepep \ --deepep-mode auto \ - --cuda-graph-bs 1 \ + --cuda-graph-bs-decode 1 \ --disable-radix-cache \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ @@ -162,7 +162,7 @@ python -m sglang.bench_serving \ **Model**: Kimi-K2.6 -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 16 @@ -277,7 +277,6 @@ do export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=64 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 - export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_NPU_USE_MLAPO=1 export SGLANG_NPU_USE_MULTI_STREAM=1 @@ -305,7 +304,7 @@ do --sampling-backend ascend \ --moe-a2a-backend deepep \ --deepep-mode auto \ - --cuda-graph-bs 1 2 4 6 8 16 \ + --cuda-graph-bs-decode 1 2 4 6 8 16 \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 3 \ @@ -358,7 +357,7 @@ python -m sglang.bench_serving \ **Model**: Kimi-K2.6 -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 16 @@ -476,7 +475,6 @@ do export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=64 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 - export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_NPU_USE_MLAPO=1 export SGLANG_NPU_USE_MULTI_STREAM=1 @@ -504,7 +502,7 @@ do --sampling-backend ascend \ --moe-a2a-backend deepep \ --deepep-mode auto \ - --cuda-graph-bs 1 2 4 6 8 16 \ + --cuda-graph-bs-decode 1 2 4 6 8 16 \ --speculative-algorithm EAGLE3 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --speculative-num-steps 3 \ @@ -561,7 +559,7 @@ python -m sglang.bench_serving \ **Model**: Kimi-K2.6 -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 16 @@ -674,7 +672,6 @@ do export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=64 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 - export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_NPU_USE_MLAPO=1 export SGLANG_NPU_USE_MULTI_STREAM=1 @@ -702,7 +699,7 @@ do --sampling-backend ascend \ --moe-a2a-backend deepep \ --deepep-mode auto \ - --cuda-graph-bs 16 \ + --cuda-graph-bs-decode 16 \ --reasoning-parser kimi_k2 \ --tool-call-parser kimi_k2 \ --speculative-algorithm EAGLE3 \ @@ -755,7 +752,7 @@ python -m sglang.bench_serving \ **Model**: Kimi-K2.6 -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 16 @@ -868,7 +865,6 @@ do export HCCL_SOCKET_IFNAME= export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=64 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 - export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_NPU_USE_MLAPO=1 export SGLANG_NPU_USE_MULTI_STREAM=1 @@ -896,7 +892,7 @@ do --sampling-backend ascend \ --moe-a2a-backend deepep \ --deepep-mode auto \ - --cuda-graph-bs 1 2 4 6 8 \ + --cuda-graph-bs-decode 1 2 4 6 8 \ --reasoning-parser kimi_k2 \ --tool-call-parser kimi_k2 \ --speculative-algorithm EAGLE3 \ @@ -955,7 +951,7 @@ python -m sglang.bench_serving \ **Model**: Kimi-K2.6 -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 8 @@ -1001,7 +997,6 @@ export HCCL_OP_EXPANSION_MODE=AIV export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=112 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 -export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_NPU_USE_MULTI_STREAM=1 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 @@ -1068,7 +1063,7 @@ python -m sglang.bench_serving \ **Model**: Kimi-K2.6 -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 8 @@ -1112,7 +1107,6 @@ export HCCL_OP_EXPANSION_MODE=AIV export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 -export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_NPU_USE_MULTI_STREAM=1 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 @@ -1179,7 +1173,7 @@ python -m sglang.bench_serving \ **Model**: Kimi-K2.6 -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 8 @@ -1228,7 +1222,6 @@ export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=96 export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 -export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_NPU_USE_MLAPO=1 export SGLANG_NPU_USE_MULTI_STREAM=1 export SGLANG_SET_CPU_AFFINITY=1 @@ -1294,7 +1287,7 @@ python -m sglang.bench_serving \ **Model**: Kimi-K2.6 -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 8 @@ -1344,7 +1337,6 @@ export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=96 export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 -export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/mimo_v2_5_pro.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/mimo_v2_5_pro.mdx new file mode 100644 index 000000000000..c18f549457b3 --- /dev/null +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/mimo_v2_5_pro.mdx @@ -0,0 +1,169 @@ +--- +title: "MiMo-V2.5-Pro" +metatags: + description: "Best Practice for MiMo-V2.5-Pro on Ascend NPU" +--- + + +This page focuses on the deployment of MiMo-V2.5-Pro (FP4) with DFlash speculative decoding in PD disaggregation mode on the Ascend NPU. + +On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. + + +### Model Deployment + +MiMo-V2.5-Pro-FP4-DFlash is an MXFP4-quantized checkpoint with a built-in DFlash draft model (located in the `dflash/` subdirectory of the weights). The following example deploys it in 1P1D mode (1 prefill node + 1 decode node, TP8 + DP2 per node). + +#### Common environment setup (both nodes) + +```bash Command +# ============================================================ +# Before running, update the following variables: +# ASCEND_MF_STORE_URL: prefill node IP with port +# HCCL_SOCKET_IFNAME / GLOO_SOCKET_IFNAME: network interface name +# ============================================================ + +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 + +export SGLANG_SET_CPU_AFFINITY=1 +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING + +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh + +export HCCL_BUFFSIZE=300 +export HCCL_OP_EXPANSION_MODE=AIV +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export STREAMS_PER_DEVICE=32 +export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 + +# Use the AscendC flash attention +export ASCEND_USE_FIA=1 + +# PD disaggregation transfer config +export ASCEND_MF_STORE_URL="tcp://:24669" +export ASCEND_MF_TRANSFER_PROTOCOL="device_urma" + +# DeepEP +export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32 +export HCCL_SOCKET_IFNAME= +export GLOO_SOCKET_IFNAME= +export HCCL_HOST_SOCKET_PORT_RANGE=auto + +MODEL_PATH=/path/to/MiMo-V2.5-Pro-FP4-DFlash +``` + +#### Prefill node + +```bash Command +export DEEPEP_HCCL_BUFFSIZE=2500 +# Enable chunked dispatch for long sequences +export DEEPEP_NORMAL_LONG_SEQ_ROUND=10 +export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=4096 +export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=0 + +python3 -m sglang.launch_server \ + --model-path $MODEL_PATH \ + --attention-backend ascend \ + --device npu \ + --tp-size 8 --nnodes 1 --node-rank 0 \ + --chunked-prefill-size 8192 \ + --trust-remote-code --port 10001 \ + --host --max-running-requests 32 \ + --mem-fraction-static 0.90 \ + --swa-full-tokens-ratio 0.3 \ + --disaggregation-mode prefill --disaggregation-transfer-backend ascend \ + --disaggregation-bootstrap-port 8996 \ + --disable-piecewise-cuda-graph \ + --dp-size 2 --enable-dp-attention --enable-dp-lm-head \ + --moe-a2a-backend deepep --deepep-mode normal +``` + +#### Decode node + +```bash Command +# Use eagle_worker_v2 and overlap plan stream to hide the draft/target preparation +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 +# DFlash draft model has a longer context length than the derived value +export SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 + +export DEEPEP_HCCL_BUFFSIZE=1200 + +python3 -m sglang.launch_server \ + --model-path $MODEL_PATH \ + --speculative-draft-model-path $MODEL_PATH/dflash \ + --attention-backend ascend \ + --device npu \ + --tp-size 8 --nnodes 1 --node-rank 0 \ + --trust-remote-code --port 20001 \ + --host --max-running-requests 32 \ + --mem-fraction-static 0.88 \ + --swa-full-tokens-ratio 0.3 \ + --cuda-graph-bs 1 2 4 8 12 16 \ + --disaggregation-mode decode --disaggregation-transfer-backend ascend \ + --disaggregation-bootstrap-port 8996 \ + --moe-a2a-backend deepep --deepep-mode low_latency \ + --dp-size 2 --enable-dp-attention --enable-dp-lm-head \ + --speculative-algorithm DFLASH \ + --speculative-num-draft-tokens 8 +``` + +#### Router + +```bash Command +python -m sglang_router.launch_router \ + --pd-disaggregation \ + --policy cache_aware \ + --prefill http://:10001 \ + --decode http://:20001 \ + --host 127.0.0.1 \ + --port 6688 \ + --health-check-interval-secs 3600 --mini-lb +``` + +### Benchmark + +We tested it based on the `RANDOM` dataset. + +#### Benchmark Prefill Node (TTFT) + +```bash Command +python3 -m sglang.bench_serving \ + --backend sglang \ + --host 127.0.0.1 \ + --port 6688 \ + --model /path/to/MiMo-V2.5-Pro-FP4-DFlash \ + --dataset-name random \ + --tokenize-prompt \ + --random-input-len 16000 \ + --random-output-len 1 \ + --request-rate 0.4 \ + --random-range-ratio 1 \ + --num-prompts 128 \ + --max-concurrency 32 +``` + +#### Benchmark Decode Node (TPOT) + +```bash Command +python3 -m sglang.bench_serving \ + --backend sglang \ + --host 127.0.0.1 \ + --port 6688 \ + --model /path/to/MiMo-V2.5-Pro-FP4-DFlash \ + --dataset-name random \ + --tokenize-prompt \ + --random-input-len 16000 \ + --random-output-len 1000 \ + --request-rate inf \ + --random-range-ratio 1 \ + --num-prompts 128 \ + --max-concurrency 32 +``` diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/mimo_v2_flash.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/mimo_v2_flash.mdx index 1a5d4f6cb9bf..8599e2c4f605 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/mimo_v2_flash.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/mimo_v2_flash.mdx @@ -7,22 +7,22 @@ metatags: This page focuses on optimal configuration and benchmark results for MiMo-V2-Flash on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [MiMo-V2-Flash Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/mimo_v2_flash). -On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. +On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. ### Low Latency | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | TTFT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | --- | -| MiMo-V2-Flash | Atlas 800I A3 | 12 | PD Disaggregation | 16k+1k | 20ms | - | W8A8 INT8 | [Optimal Configuration](#mimo-v2-flash-1p1d-12p-in16k-out1k-tpot-20ms) | -| MiMo-V2-Flash | Atlas 800I A3 | 12 | PD Disaggregation | 32k+1k | 20ms | - | W8A8 INT8 | [Optimal Configuration](#mimo-v2-flash-1p1d-12p-in32k-out1k-tpot-20ms) | +| MiMo-V2-Flash | Ascend A3 Series Products | 12 | PD Disaggregation | 16k+1k | 20ms | - | W8A8 INT8 | [Optimal Configuration](#mimo-v2-flash-1p1d-12p-in16k-out1k-tpot-20ms) | +| MiMo-V2-Flash | Ascend A3 Series Products | 12 | PD Disaggregation | 32k+1k | 20ms | - | W8A8 INT8 | [Optimal Configuration](#mimo-v2-flash-1p1d-12p-in32k-out1k-tpot-20ms) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | TTFT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | --- | -| MiMo-V2-Flash | Atlas 800I A3 | 12 | PD Disaggregation | 16k+1 | - | 5s | W8A8 INT8 | [Optimal Configuration](#mimo-v2-flash-1p1d-12p-in16k-out1-ttft-5s) | -| MiMo-V2-Flash | Atlas 800I A3 | 12 | PD Disaggregation | 32k+1 | - | 5s | W8A8 INT8 | [Optimal Configuration](#mimo-v2-flash-1p1d-12p-in32k-out1-ttft-5s) | +| MiMo-V2-Flash | Ascend A3 Series Products | 12 | PD Disaggregation | 16k+1 | - | 5s | W8A8 INT8 | [Optimal Configuration](#mimo-v2-flash-1p1d-12p-in16k-out1-ttft-5s) | +| MiMo-V2-Flash | Ascend A3 Series Products | 12 | PD Disaggregation | 32k+1 | - | 5s | W8A8 INT8 | [Optimal Configuration](#mimo-v2-flash-1p1d-12p-in32k-out1-ttft-5s) | ## Optimal Configuration @@ -32,7 +32,7 @@ On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend **Model**: MiMo-V2-Flash -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 12 @@ -83,7 +83,6 @@ export SGLANG_DEEPEP_BF16_DISPATCH=0 export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=3600 export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=3600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0 -export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 @@ -129,7 +128,7 @@ do --disaggregation-transfer-backend ascend \ --disable-radix-cache \ --disable-cuda-graph \ - --disable-piecewise-cuda-graph \ + --cuda-graph-backend-prefill=disabled \ --dp-size 2 break fi @@ -159,7 +158,7 @@ do --max-running-requests 64 \ --mem-fraction-static 0.8 \ --swa-full-tokens-ratio 0.3 \ - --cuda-graph-bs 1 2 4 8 12 16 20 24 28 32 \ + --cuda-graph-bs-decode 1 2 4 8 12 16 20 24 28 32 \ --disaggregation-transfer-backend ascend \ --speculative-algorithm EAGLE \ --speculative-num-steps 3 \ @@ -217,7 +216,7 @@ python -m sglang.bench_serving \ **Model**: MiMo-V2-Flash -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 12 @@ -268,7 +267,6 @@ export SGLANG_DEEPEP_BF16_DISPATCH=0 export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=3600 export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=3600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0 -export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 @@ -314,7 +312,7 @@ do --disaggregation-transfer-backend ascend \ --disable-radix-cache \ --disable-cuda-graph \ - --disable-piecewise-cuda-graph \ + --cuda-graph-backend-prefill=disabled \ --dp-size 2 break fi @@ -344,7 +342,7 @@ do --max-running-requests 32 \ --mem-fraction-static 0.8 \ --swa-full-tokens-ratio 0.3 \ - --cuda-graph-bs 1 2 4 8 12 16 \ + --cuda-graph-bs-decode 1 2 4 8 12 16 \ --disaggregation-transfer-backend ascend \ --speculative-algorithm EAGLE \ --speculative-num-steps 3 \ @@ -402,7 +400,7 @@ python -m sglang.bench_serving \ **Model**: MiMo-V2-Flash -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 12 @@ -453,7 +451,6 @@ export SGLANG_DEEPEP_BF16_DISPATCH=0 export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=3600 export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=3600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0 -export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 @@ -499,7 +496,7 @@ do --disaggregation-transfer-backend ascend \ --disable-radix-cache \ --disable-cuda-graph \ - --disable-piecewise-cuda-graph \ + --cuda-graph-backend-prefill=disabled \ --dp-size 2 break fi @@ -529,7 +526,7 @@ do --max-running-requests 64 \ --mem-fraction-static 0.8 \ --swa-full-tokens-ratio 0.3 \ - --cuda-graph-bs 1 2 4 8 12 16 20 24 28 32 \ + --cuda-graph-bs-decode 1 2 4 8 12 16 20 24 28 32 \ --disaggregation-transfer-backend ascend \ --speculative-algorithm EAGLE \ --speculative-num-steps 3 \ @@ -587,7 +584,7 @@ python -m sglang.bench_serving \ **Model**: MiMo-V2-Flash -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 12 @@ -638,7 +635,6 @@ export SGLANG_DEEPEP_BF16_DISPATCH=0 export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=3600 export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=3600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0 -export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 @@ -684,7 +680,7 @@ do --disaggregation-transfer-backend ascend \ --disable-radix-cache \ --disable-cuda-graph \ - --disable-piecewise-cuda-graph \ + --cuda-graph-backend-prefill=disabled \ --dp-size 2 break fi @@ -714,7 +710,7 @@ do --max-running-requests 64 \ --mem-fraction-static 0.8 \ --swa-full-tokens-ratio 0.3 \ - --cuda-graph-bs 1 2 4 8 12 16 20 24 28 32 \ + --cuda-graph-bs-decode 1 2 4 8 12 16 20 24 28 32 \ --disaggregation-transfer-backend ascend \ --speculative-algorithm EAGLE \ --speculative-num-steps 3 \ diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/minimax_m2_5.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/minimax_m2_5.mdx index e842e3cd3059..4fdf7a210e79 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/minimax_m2_5.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/minimax_m2_5.mdx @@ -7,23 +7,23 @@ metatags: This page focuses on optimal configuration and benchmark results for MiniMax-M2.5 on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [MiniMax-M2.5 Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/minimax_m2_5). -On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. +On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. ### Low Latency | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | -| MiniMax-M2.5 | Atlas 800I A3 | 8 | PD Mixed | 128k+1k (90% prefix cache hit rate) | 24.44ms | W8A8 INT8 | [Optimal Configuration](#minimax-m2-5-w8a8-8p-in128k-out1k-prefix90-24-44ms) | -| MiniMax-M2.5 | Atlas 800I A3 | 8 | PD Mixed | 3.5k+1.5k | 20ms | W8A8 INT8 | [Optimal Configuration](#minimax-m2-5-w8a8-8p-in3k5-out1k5-20ms) | +| MiniMax-M2.5 | Ascend A3 Series Products | 8 | PD Mixed | 128k+1k (90% prefix cache hit rate) | 24.44ms | W8A8 INT8 | [Optimal Configuration](#minimax-m2-5-w8a8-8p-in128k-out1k-prefix90-24-44ms) | +| MiniMax-M2.5 | Ascend A3 Series Products | 8 | PD Mixed | 3.5k+1.5k | 20ms | W8A8 INT8 | [Optimal Configuration](#minimax-m2-5-w8a8-8p-in3k5-out1k5-20ms) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | -| MiniMax-M2.5 | Atlas 800I A3 | 4 | PD Mixed | 32k+1k | 50ms | W8A8 INT8 | [Optimal Configuration](#minimax-m2-5-w8a8-4p-in32k-out1k-50ms) | -| MiniMax-M2.5 | Atlas 800I A3 | 4 | PD Mixed | 64k+1k (90% prefix cache hit rate) | 50ms | W8A8 INT8 | [Optimal Configuration](#minimax-m2-5-w8a8-4p-in64k-out1k-prefix90-50ms) | -| MiniMax-M2.5 | Atlas 800I A3 | 8 | PD Mixed | 3.5k+1.5k | 50ms | W8A8 INT8 | [Optimal Configuration](#minimax-m2-5-w8a8-8p-in3k5-out1k5-50ms) | +| MiniMax-M2.5 | Ascend A3 Series Products | 4 | PD Mixed | 32k+1k | 50ms | W8A8 INT8 | [Optimal Configuration](#minimax-m2-5-w8a8-4p-in32k-out1k-50ms) | +| MiniMax-M2.5 | Ascend A3 Series Products | 4 | PD Mixed | 64k+1k (90% prefix cache hit rate) | 50ms | W8A8 INT8 | [Optimal Configuration](#minimax-m2-5-w8a8-4p-in64k-out1k-prefix90-50ms) | +| MiniMax-M2.5 | Ascend A3 Series Products | 8 | PD Mixed | 3.5k+1.5k | 50ms | W8A8 INT8 | [Optimal Configuration](#minimax-m2-5-w8a8-8p-in3k5-out1k5-50ms) | ## Optimal Configuration @@ -33,7 +33,7 @@ On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend **Model**: MiniMax-M2.5 -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 4 @@ -85,7 +85,6 @@ export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=640 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 -export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 export SGLANG_EXTERNAL_MODEL_PACKAGE=custom_eagle3 export SGLANG_SET_CPU_AFFINITY=1 @@ -105,7 +104,7 @@ python3 -m sglang.launch_server \ --max-running-requests 18 \ --chunked-prefill-size -1 \ --max-prefill-tokens 32768 \ - --cuda-graph-bs 2 4 6 8 10 12 14 16 18 24 \ + --cuda-graph-bs-decode 2 4 6 8 10 12 14 16 18 24 \ --moe-a2a-backend deepep \ --deepep-mode auto \ --quantization modelslim \ @@ -145,7 +144,7 @@ python -m sglang.bench_serving \ **Model**: MiniMax-M2.5 -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 4 @@ -212,7 +211,7 @@ python3 -m sglang.launch_server \ --prefill-max-requests 10 \ --chunked-prefill-size 67072 \ --max-prefill-tokens 67000 \ - --cuda-graph-bs 2 4 8 12 16 18 20 22 24 26 \ + --cuda-graph-bs-decode 2 4 8 12 16 18 20 22 24 26 \ --moe-a2a-backend ascend_fuseep \ --deepep-mode auto \ --quantization modelslim \ @@ -254,7 +253,7 @@ python -m sglang.bench_serving \ **Model**: MiniMax-M2.5 -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 8 @@ -303,7 +302,6 @@ export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=160000 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 -export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_EXTERNAL_MODEL_PACKAGE=custom_eagle3 export SGLANG_NPU_FUSED_MOE_MODE=2 export SGLANG_SET_CPU_AFFINITY=1 @@ -324,7 +322,7 @@ python3 -m sglang.launch_server \ --prefill-max-requests 4 \ --chunked-prefill-size 160000 \ --max-prefill-tokens 80000 \ - --cuda-graph-bs 2 4 6 8 \ + --cuda-graph-bs-decode 2 4 6 8 \ --moe-a2a-backend ascend_fuseep \ --deepep-mode auto \ --quantization modelslim \ @@ -366,7 +364,7 @@ python -m sglang.bench_serving \ **Model**: MiniMax-M2.5 -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 8 @@ -414,7 +412,6 @@ export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=204800 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 -export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_EXTERNAL_MODEL_PACKAGE=custom_eagle3 export SGLANG_NPU_FUSED_MOE_MODE=2 export SGLANG_SET_CPU_AFFINITY=1 @@ -437,7 +434,7 @@ python3 -m sglang.launch_server \ --prefill-max-requests 3 \ --chunked-prefill-size -1 \ --max-prefill-tokens 8192 \ - --cuda-graph-bs 1 2 3 4 5 6 \ + --cuda-graph-bs-decode 1 2 3 4 5 6 \ --moe-a2a-backend ascend_fuseep \ --deepep-mode auto \ --quantization modelslim \ @@ -473,7 +470,7 @@ python -m sglang.bench_serving \ **Model**: MiniMax-M2.5 -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 8 @@ -541,7 +538,7 @@ python3 -m sglang.launch_server \ --enable-prefill-delayer \ --chunked-prefill-size 196608 \ --max-prefill-tokens 8192 \ - --cuda-graph-bs 1 2 4 8 12 16 20 \ + --cuda-graph-bs-decode 1 2 4 8 12 16 20 \ --moe-a2a-backend ascend_fuseep \ --fuseep-mode 2 \ --quantization modelslim \ diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_235b_a22b.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_235b_a22b.mdx index b73f67f46263..6c3a3f49a18b 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_235b_a22b.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_235b_a22b.mdx @@ -7,20 +7,20 @@ metatags: This page focuses on optimal configuration and benchmark results for Qwen3-235B-A22B on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [Qwen3-235B-A22B Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_235b_a22b). -On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. +On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. ### Low Latency | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | -| Qwen3-235B-A22B | Atlas 800I A3 | 8 | PD Mixed | 11k+1.5k | 8ms | BF16 | [Optimal Configuration](#qwen3-235b-a22b-bf16-8p-in11k-out1k5-8ms) | +| Qwen3-235B-A22B | Ascend A3 Series Products | 8 | PD Mixed | 11k+1.5k | 8ms | BF16 | [Optimal Configuration](#qwen3-235b-a22b-bf16-8p-in11k-out1k5-8ms) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | -| Qwen3-235B-A22B | Atlas 800I A3 | 8 | PD Mixed | 3.5k+1.5k | 50.1ms | W8A8 INT8 | [Optimal Configuration](#qwen3-235b-a22b-w8a8-8p-in3k5-out1k5-50-1ms) | +| Qwen3-235B-A22B | Ascend A3 Series Products | 8 | PD Mixed | 3.5k+1.5k | 50.1ms | W8A8 INT8 | [Optimal Configuration](#qwen3-235b-a22b-w8a8-8p-in3k5-out1k5-50-1ms) | ## Optimal Configuration @@ -30,7 +30,7 @@ On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend **Model**: Qwen3-235B-A22B -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 8 @@ -77,7 +77,6 @@ export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 -export SGLANG_ENABLE_SPEC_V2=1 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ @@ -101,7 +100,7 @@ python3 -m sglang.launch_server \ --enable-dp-lm-head \ --tp 16 \ --mem-fraction-static 0.78 \ - --cuda-graph-bs 1 \ + --cuda-graph-bs-decode 1 \ --reasoning-parser qwen3 \ --tool-call-parser qwen25 ``` @@ -128,7 +127,7 @@ python -m sglang.bench_serving \ **Model**: Qwen3-235B-A22B -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 8 @@ -176,7 +175,6 @@ export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=188416 export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 -export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_NPU_FUSED_MOE_MODE=2 export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=100 export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1 @@ -210,7 +208,7 @@ python3 -m sglang.launch_server \ --enable-dp-attention \ --enable-dp-lm-head \ --mem-fraction-static 0.8 \ - --cuda-graph-bs 1 2 4 8 16 20 24 26 27 \ + --cuda-graph-bs-decode 1 2 4 8 16 20 24 26 27 \ --reasoning-parser qwen3 \ --tool-call-parser qwen25 ``` diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_30b_a3b.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_30b_a3b.mdx index 9324eaa3c08c..d18fb5389325 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_30b_a3b.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_30b_a3b.mdx @@ -7,22 +7,22 @@ metatags: This page focuses on optimal configuration and benchmark results for Qwen3-30B-A3B on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [Qwen3-30B-A3B Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_30b_a3b). -On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. +On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. ### Low Latency | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | -| Qwen3-30B-A3B | Atlas 800I A3 | 1 | PD Mixed | 3.5k+1.5k | 10ms | W8A8 INT8 | [Optimal Configuration](#qwen3-30b-a3b-w8a8-1p-in3k5-out1k5-10ms) | -| Qwen3-30B-A3B | Atlas 800I A3 | 1 | PD Mixed | 6k+1.5k | 10.25ms | W8A8 INT8 | [Optimal Configuration](#qwen3-30b-a3b-w8a8-1p-in6k-out1k5-bs16) | +| Qwen3-30B-A3B | Ascend A3 Series Products | 1 | PD Mixed | 3.5k+1.5k | 10ms | W8A8 INT8 | [Optimal Configuration](#qwen3-30b-a3b-w8a8-1p-in3k5-out1k5-10ms) | +| Qwen3-30B-A3B | Ascend A3 Series Products | 1 | PD Mixed | 6k+1.5k | 10.25ms | W8A8 INT8 | [Optimal Configuration](#qwen3-30b-a3b-w8a8-1p-in6k-out1k5-bs16) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | -| Qwen3-30B-A3B | Atlas 800I A3 | 1 | PD Mixed | 1k+100 | 10000ms | BF16 | [Optimal Configuration](#qwen3-30b-a3b-bf16-1p-in1k-out100) | -| Qwen3-30B-A3B | Atlas 800I A3 | 1 | PD Mixed | 3.5k+1.5k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-30b-a3b-w8a8-1p-in3k5-out1k5-50ms) | +| Qwen3-30B-A3B | Ascend A3 Series Products | 1 | PD Mixed | 1k+100 | 10000ms | BF16 | [Optimal Configuration](#qwen3-30b-a3b-bf16-1p-in1k-out100) | +| Qwen3-30B-A3B | Ascend A3 Series Products | 1 | PD Mixed | 3.5k+1.5k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-30b-a3b-w8a8-1p-in3k5-out1k5-50ms) | ## Optimal Configuration @@ -32,7 +32,7 @@ On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend **Model**: Qwen3-30B-A3B -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 1 @@ -81,7 +81,6 @@ export INF_NAN_MODE_FORCE_DISABLE=1 export PYTORCH_NPU_ALLOC_CONF=expandable_segments:False export SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 -export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=200 export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1 export SGLANG_USE_MAX_DP_ATT=1 @@ -109,7 +108,7 @@ python3 -m sglang.launch_server \ --enable-dp-attention \ --dp-size 2 \ --mem-fraction-static 0.85 \ - --cuda-graph-bs 1 2 4 8 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76 80 84 \ + --cuda-graph-bs-decode 1 2 4 8 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76 80 84 \ --dtype bfloat16 \ --reasoning-parser qwen3 \ --tool-call-parser qwen @@ -138,7 +137,7 @@ python -m sglang.bench_serving \ **Model**: Qwen3-30B-A3B -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 1 @@ -186,7 +185,6 @@ export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 -export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=200 export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1 @@ -211,7 +209,7 @@ python3 -m sglang.launch_server \ --speculative-num-draft-tokens 4 \ --tp-size 2 \ --mem-fraction-static 0.87 \ - --cuda-graph-bs 1 5 15 40 70 100 120 130 140 146 150 154 156 158 160 162 \ + --cuda-graph-bs-decode 1 5 15 40 70 100 120 130 140 146 150 154 156 158 160 162 \ --dtype bfloat16 \ --reasoning-parser qwen3 \ --tool-call-parser qwen @@ -239,7 +237,7 @@ python -m sglang.bench_serving \ **Model**: Qwen3-30B-A3B -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 1 @@ -311,7 +309,7 @@ python3 -m sglang.launch_server \ --speculative-num-draft-tokens 4 \ --tp-size 2 \ --mem-fraction-static 0.87 \ - --cuda-graph-bs 1 5 15 40 70 100 120 130 140 146 150 154 156 158 160 162 \ + --cuda-graph-bs-decode 1 5 15 40 70 100 120 130 140 146 150 154 156 158 160 162 \ --dtype bfloat16 \ --reasoning-parser qwen3 \ --tool-call-parser qwen @@ -339,7 +337,7 @@ python -m sglang.bench_serving \ **Model**: Qwen3-30B-A3B -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 1 @@ -386,7 +384,6 @@ export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 -export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_SET_CPU_AFFINITY=1 export TRANSFORMERS_VERBOSITY=error @@ -411,7 +408,7 @@ python3 -m sglang.launch_server \ --max-prefill-tokens 35000 \ --tp-size 2 \ --mem-fraction-static 0.6 \ - --cuda-graph-bs 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 \ + --cuda-graph-bs-decode 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 \ --dtype bfloat16 \ --reasoning-parser qwen3 \ --tool-call-parser qwen diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_32b.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_32b.mdx index 62d332d0983f..2a42f7d8eea8 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_32b.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_32b.mdx @@ -7,21 +7,21 @@ metatags: This page focuses on optimal configuration and benchmark results for Qwen3-32B on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [Qwen3-32B Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_32b). -On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. +On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. ### Low Latency | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | -| Qwen3-32B | Atlas 800I A3 | 8 | PD Mixed | 18k+4k | 6ms | BF16 | [Optimal Configuration](#qwen3-32b-bf16-8p-in18k-out4k-6ms) | +| Qwen3-32B | Ascend A3 Series Products | 8 | PD Mixed | 18k+4k | 6ms | BF16 | [Optimal Configuration](#qwen3-32b-bf16-8p-in18k-out4k-6ms) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | -| Qwen3-32B | Atlas 800I A3 | 2 | PD Mixed | 3.5k+1.5k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-32b-w8a8-2p-in3k5-out1k5-50ms) | -| Qwen3-32B | Atlas 800I A2 | 2 | PD Mixed | 3.5k+1.5k | 55ms | W8A8 INT8 | [Optimal Configuration](#qwen3-32b-w8a8-2p-in3k5-out1k5-55ms-a2) | +| Qwen3-32B | Ascend A3 Series Products | 2 | PD Mixed | 3.5k+1.5k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-32b-w8a8-2p-in3k5-out1k5-50ms) | +| Qwen3-32B | Ascend A2 Series Products | 2 | PD Mixed | 3.5k+1.5k | 55ms | W8A8 INT8 | [Optimal Configuration](#qwen3-32b-w8a8-2p-in3k5-out1k5-55ms-a2-series) | ## Optimal Configuration @@ -31,7 +31,7 @@ On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend **Model**: Qwen3-32B -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 8 @@ -100,7 +100,7 @@ python3 -m sglang.launch_server \ --speculative-num-draft-tokens 5 \ --tp-size 16 \ --mem-fraction-static 0.72 \ - --cuda-graph-bs 1 \ + --cuda-graph-bs-decode 1 \ --dtype bfloat16 \ --reasoning-parser qwen3 \ --tool-call-parser qwen @@ -128,7 +128,7 @@ python -m sglang.bench_serving \ **Model**: Qwen3-32B -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 2 @@ -199,7 +199,7 @@ python3 -m sglang.launch_server \ --speculative-num-draft-tokens 4 \ --tp-size 4 \ --mem-fraction-static 0.845 \ - --cuda-graph-bs 16 32 64 72 88 90 92 94 96 97 98 99 100 101 \ + --cuda-graph-bs-decode 16 32 64 72 88 90 92 94 96 97 98 99 100 101 \ --dtype bfloat16 \ --reasoning-parser qwen3 \ --tool-call-parser qwen @@ -223,11 +223,11 @@ python -m sglang.bench_serving \ --seed 1 ``` -### Qwen3-32B W8A8 2P IN3K5 OUT1K5 55ms A2 +### Qwen3-32B W8A8 2P IN3K5 OUT1K5 55ms A2 Series **Model**: Qwen3-32B -**Hardware**: Atlas 800I A2 +**Hardware**: Ascend A2 Series Products **Cards**: 2 @@ -298,7 +298,7 @@ python3 -m sglang.launch_server \ --speculative-num-draft-tokens 4 \ --tp-size 4 \ --mem-fraction-static 0.845 \ - --cuda-graph-bs 16 32 64 72 88 90 92 94 96 97 98 99 100 101 \ + --cuda-graph-bs-decode 16 32 64 72 88 90 92 94 96 97 98 99 100 101 \ --dtype bfloat16 \ --reasoning-parser qwen3 \ --tool-call-parser qwen diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_5_397b.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_5_397b.mdx index 75724ee1ad1b..767fd2f5a910 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_5_397b.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_5_397b.mdx @@ -9,28 +9,28 @@ This page focuses on optimal configuration and benchmark results for Qwen3.5-397 setup, model weight download, feature configuration, and deployment instructions, etc., see the [Qwen3.5-397B-A17B Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_5_397b). -On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. +On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. ### Low Latency | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | -| Qwen3.5-397B-A17B | Atlas 800I A3 | 8 | PD Mixed | 128k+1k | 20ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-a17b-w4a8-8p-in128k-out1k-20ms) | -| Qwen3.5-397B-A17B | Atlas 800I A3 | 8 | PD Mixed | 16k+1k | 20ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-a17b-w4a8-8p-in16k-out1k-20ms) | -| Qwen3.5-397B-A17B | Atlas 800I A3 | 8 | PD Mixed | 3.5k+1.5k | 22.2ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-a17b-w4a8-8p-in3k5-out1k5-22-2ms) | -| Qwen3.5-397B-A17B | Atlas 800I A3 | 8 | PD Mixed | 64k+1k | 20ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-a17b-w4a8-8p-in64k-out1k-20ms) | +| Qwen3.5-397B-A17B | Ascend A3 Series Products | 8 | PD Mixed | 128k+1k | 20ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-a17b-w4a8-8p-in128k-out1k-20ms) | +| Qwen3.5-397B-A17B | Ascend A3 Series Products | 8 | PD Mixed | 16k+1k | 20ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-a17b-w4a8-8p-in16k-out1k-20ms) | +| Qwen3.5-397B-A17B | Ascend A3 Series Products | 8 | PD Mixed | 3.5k+1.5k | 22.2ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-a17b-w4a8-8p-in3k5-out1k5-22-2ms) | +| Qwen3.5-397B-A17B | Ascend A3 Series Products | 8 | PD Mixed | 64k+1k | 20ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-a17b-w4a8-8p-in64k-out1k-20ms) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | -| Qwen3.5-397B-A17B | Atlas 800I A3 | 8 | PD Mixed | 128k+1k | 50ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-a17b-w4a8-8p-in128k-out1k-50ms) | -| Qwen3.5-397B-A17B | Atlas 800I A3 | 8 | PD Mixed | 128k+1k (90% prefix cache hit rate) | 50ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-a17b-w4a8-8p-in128k-out1k-prefix90-50ms) | -| Qwen3.5-397B-A17B | Atlas 800I A3 | 8 | PD Mixed | 16k+1k | 50ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-a17b-w4a8-8p-in16k-out1k-50ms) | -| Qwen3.5-397B-A17B | Atlas 800I A3 | 8 | PD Mixed | 3.5k+1.5k | 50ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-a17b-w4a8-8p-in3k5-out1k5-50ms) | -| Qwen3.5-397B-A17B | Atlas 800I A3 | 8 | PD Mixed | 64k+1k | 50ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-a17b-w4a8-8p-in64k-out1k-50ms) | -| Qwen3.5-397B-A17B | Atlas 800I A3 | 8 | PD Mixed | 64k+1k (90% prefix cache hit rate) | 50ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-a17b-w4a8-8p-in64k-out1k-prefix90-50ms) | +| Qwen3.5-397B-A17B | Ascend A3 Series Products | 8 | PD Mixed | 128k+1k | 50ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-a17b-w4a8-8p-in128k-out1k-50ms) | +| Qwen3.5-397B-A17B | Ascend A3 Series Products | 8 | PD Mixed | 128k+1k (90% prefix cache hit rate) | 50ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-a17b-w4a8-8p-in128k-out1k-prefix90-50ms) | +| Qwen3.5-397B-A17B | Ascend A3 Series Products | 8 | PD Mixed | 16k+1k | 50ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-a17b-w4a8-8p-in16k-out1k-50ms) | +| Qwen3.5-397B-A17B | Ascend A3 Series Products | 8 | PD Mixed | 3.5k+1.5k | 50ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-a17b-w4a8-8p-in3k5-out1k5-50ms) | +| Qwen3.5-397B-A17B | Ascend A3 Series Products | 8 | PD Mixed | 64k+1k | 50ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-a17b-w4a8-8p-in64k-out1k-50ms) | +| Qwen3.5-397B-A17B | Ascend A3 Series Products | 8 | PD Mixed | 64k+1k (90% prefix cache hit rate) | 50ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-a17b-w4a8-8p-in64k-out1k-prefix90-50ms) | ## Optimal Configuration @@ -40,7 +40,7 @@ On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend **Model**: Qwen3.5-397B-A17B -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 8 @@ -89,7 +89,6 @@ export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 -export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24669 @@ -111,7 +110,7 @@ python3 -m sglang.launch_server \ --trust-remote-code \ --max-running-requests 16 \ --mem-fraction-static 0.6 \ - --cuda-graph-bs 2 3 4 5 6 8 10 12 14 16 \ + --cuda-graph-bs-decode 2 3 4 5 6 8 10 12 14 16 \ --quantization modelslim \ --enable-multimodal \ --moe-a2a-backend deepep \ @@ -154,7 +153,7 @@ python -m sglang.bench_serving \ **Model**: Qwen3.5-397B-A17B -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 8 @@ -203,7 +202,6 @@ export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 -export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24669 @@ -267,7 +265,7 @@ python -m sglang.bench_serving \ **Model**: Qwen3.5-397B-A17B -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 8 @@ -316,7 +314,6 @@ export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 -export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 @@ -330,7 +327,7 @@ python3 -m sglang.launch_server \ --max-prefill-tokens 131072 \ --max-mamba-cache-size 320 \ --prefill-max-requests 10 \ - --mamba-scheduler-strategy extra_buffer \ + --mamba-radix-cache-strategy extra_buffer \ --trust-remote-code \ --max-running-requests 64 \ --mem-fraction-static 0.6 \ @@ -379,7 +376,7 @@ python -m sglang.bench_serving \ **Model**: Qwen3.5-397B-A17B -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 8 @@ -428,7 +425,6 @@ export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 -export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24669 @@ -451,7 +447,7 @@ python3 -m sglang.launch_server \ --max-running-requests 48 \ --mem-fraction-static 0.8 \ --max-total-tokens 210000 \ - --cuda-graph-bs 2 4 6 8 10 12 \ + --cuda-graph-bs-decode 2 4 6 8 10 12 \ --quantization modelslim \ --enable-multimodal \ --moe-a2a-backend deepep \ @@ -497,7 +493,7 @@ python -m sglang.bench_serving \ **Model**: Qwen3.5-397B-A17B -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 8 @@ -546,7 +542,6 @@ export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 -export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24669 @@ -569,7 +564,7 @@ python3 -m sglang.launch_server \ --max-running-requests 144 \ --mem-fraction-static 0.8 \ --max-total-tokens 635000 \ - --cuda-graph-bs 2 4 6 8 12 14 16 18 20 24 26 28 30 32 34 36 \ + --cuda-graph-bs-decode 2 4 6 8 12 14 16 18 20 24 26 28 30 32 34 36 \ --quantization modelslim \ --enable-multimodal \ --moe-a2a-backend deepep \ @@ -615,7 +610,7 @@ python -m sglang.bench_serving \ **Model**: Qwen3.5-397B-A17B -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 8 @@ -664,7 +659,6 @@ export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 -export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24669 @@ -686,7 +680,7 @@ python3 -m sglang.launch_server \ --trust-remote-code \ --max-running-requests 160 \ --mem-fraction-static 0.8 \ - --cuda-graph-bs 2 4 6 8 10 12 14 16 18 20 \ + --cuda-graph-bs-decode 2 4 6 8 10 12 14 16 18 20 \ --quantization modelslim \ --enable-multimodal \ --moe-a2a-backend deepep \ @@ -732,7 +726,7 @@ python -m sglang.bench_serving \ **Model**: Qwen3.5-397B-A17B -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 8 @@ -781,7 +775,6 @@ export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 -export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24669 @@ -803,7 +796,7 @@ python3 -m sglang.launch_server \ --trust-remote-code \ --max-running-requests 432 \ --mem-fraction-static 0.8 \ - --cuda-graph-bs 2 4 6 8 12 16 20 24 28 32 36 40 44 48 50 52 54 \ + --cuda-graph-bs-decode 2 4 6 8 12 16 20 24 28 32 36 40 44 48 50 52 54 \ --quantization modelslim \ --enable-multimodal \ --moe-a2a-backend deepep \ @@ -849,7 +842,7 @@ python -m sglang.bench_serving \ **Model**: Qwen3.5-397B-A17B -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 8 @@ -898,7 +891,6 @@ export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 -export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24669 @@ -921,7 +913,7 @@ python3 -m sglang.launch_server \ --max-running-requests 16 \ --mem-fraction-static 0.6 \ --max-total-tokens 1065000 \ - --cuda-graph-bs 2 4 6 8 10 12 14 16 \ + --cuda-graph-bs-decode 2 4 6 8 10 12 14 16 \ --quantization modelslim \ --enable-multimodal \ --moe-a2a-backend deepep \ @@ -967,7 +959,7 @@ python -m sglang.bench_serving \ **Model**: Qwen3.5-397B-A17B -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 8 @@ -1016,7 +1008,6 @@ export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 -export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_ZBAL_BOOTSTRAP_URL=tcp://127.0.0.1:24669 @@ -1039,7 +1030,7 @@ python3 -m sglang.launch_server \ --max-running-requests 32 \ --mem-fraction-static 0.6 \ --max-total-tokens 1065000 \ - --cuda-graph-bs 2 4 6 8 12 14 16 \ + --cuda-graph-bs-decode 2 4 6 8 12 14 16 \ --quantization modelslim \ --enable-multimodal \ --moe-a2a-backend deepep \ @@ -1085,7 +1076,7 @@ python -m sglang.bench_serving \ **Model**: Qwen3.5-397B-A17B -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 8 @@ -1134,7 +1125,6 @@ export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 -export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 @@ -1147,7 +1137,7 @@ python3 -m sglang.launch_server \ --chunked-prefill-size -1 \ --max-prefill-tokens 65536 \ --max-mamba-cache-size 640 \ - --mamba-scheduler-strategy extra_buffer \ + --mamba-radix-cache-strategy extra_buffer \ --trust-remote-code \ --max-running-requests 128 \ --mem-fraction-static 0.6 \ diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_6_27b.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_6_27b.mdx index 5c52399c757d..1505612f7e1f 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_6_27b.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_6_27b.mdx @@ -7,21 +7,21 @@ metatags: This page focuses on optimal configuration and benchmark results for Qwen3.6-27B on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [Qwen3.6-27B Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_6_27b). -On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. +On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | -| Qwen3.6-27B | Atlas 800I A3 | 1 | PD Mixed | 1024x1024 (30)+1024 | 50ms | BF16 | [Optimal Configuration](#qwen3-6-27b-1p-in1024x1024-30-out1024-50ms) | -| Qwen3.6-27B | Atlas 800I A3 | 1 | PD Mixed | 1080p_30+256 | 50ms | BF16 | [Optimal Configuration](#qwen3-6-27b-1p-in1080p-30-out256-50ms) | -| Qwen3.6-27B | Atlas 800I A3 | 1 | PD Mixed | 64k+1k (90% prefix cache hit rate) | 50ms | BF16 | [Optimal Configuration](#qwen3-6-27b-1p-in64k-out1k-prefix90-50ms) | -| Qwen3.6-27B | Atlas 800I A3 | 1 | PD Mixed | 3.5k+1.5k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-6-27b-w8a8-1p-in3k5-out1k5-50ms) | -| Qwen3.6-27B | Atlas 800I A3 | 1 | PD Mixed | 64k+1k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-6-27b-w8a8-1p-in64k-out1k-50ms) | -| Qwen3.6-27B | Atlas 800I A3 | 2 | PD Mixed | 128k+1k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-6-27b-w8a8-2p-in128k-out1k-50ms) | -| Qwen3.6-27B | Atlas 800I A3 | 2 | PD Mixed | 16k+1k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-6-27b-w8a8-2p-in16k-out1k-50ms) | -| Qwen3.6-27B | Atlas 800I A3 | 2 | PD Mixed | 64k+1k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-6-27b-w8a8-2p-in64k-out1k-50ms) | +| Qwen3.6-27B | Ascend A3 Series Products | 1 | PD Mixed | 1024x1024 (30)+1024 | 50ms | BF16 | [Optimal Configuration](#qwen3-6-27b-1p-in1024x1024-30-out1024-50ms) | +| Qwen3.6-27B | Ascend A3 Series Products | 1 | PD Mixed | 1080p_30+256 | 50ms | BF16 | [Optimal Configuration](#qwen3-6-27b-1p-in1080p-30-out256-50ms) | +| Qwen3.6-27B | Ascend A3 Series Products | 1 | PD Mixed | 64k+1k (90% prefix cache hit rate) | 50ms | BF16 | [Optimal Configuration](#qwen3-6-27b-1p-in64k-out1k-prefix90-50ms) | +| Qwen3.6-27B | Ascend A3 Series Products | 1 | PD Mixed | 3.5k+1.5k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-6-27b-w8a8-1p-in3k5-out1k5-50ms) | +| Qwen3.6-27B | Ascend A3 Series Products | 1 | PD Mixed | 64k+1k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-6-27b-w8a8-1p-in64k-out1k-50ms) | +| Qwen3.6-27B | Ascend A3 Series Products | 2 | PD Mixed | 128k+1k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-6-27b-w8a8-2p-in128k-out1k-50ms) | +| Qwen3.6-27B | Ascend A3 Series Products | 2 | PD Mixed | 16k+1k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-6-27b-w8a8-2p-in16k-out1k-50ms) | +| Qwen3.6-27B | Ascend A3 Series Products | 2 | PD Mixed | 64k+1k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-6-27b-w8a8-2p-in64k-out1k-50ms) | ## Optimal Configuration @@ -31,7 +31,7 @@ On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend **Model**: Qwen3.6-27B -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 1 @@ -96,7 +96,7 @@ python3 -m sglang.launch_server \ --max-running-requests 60 \ --max-mamba-cache-size 60 \ --mem-fraction-static 0.74 \ - --cuda-graph-bs 2 4 8 14 16 24 26 32 36 37 40 42 44 45 46 50 52 60 \ + --cuda-graph-bs-decode 2 4 8 14 16 24 26 32 36 37 40 42 44 45 46 50 52 60 \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ @@ -134,7 +134,7 @@ python -m sglang.bench_serving \ **Model**: Qwen3.6-27B -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 1 @@ -194,7 +194,7 @@ python3 -m sglang.launch_server \ --max-running-requests 38 \ --max-mamba-cache-size 38 \ --mem-fraction-static 0.7 \ - --cuda-graph-bs 1 2 4 8 10 12 16 20 24 28 30 32 35 38 \ + --cuda-graph-bs-decode 1 2 4 8 10 12 16 20 24 28 30 32 35 38 \ --enable-prefill-delayer \ --prefill-delayer-queue-min-ratio 0.45 \ --prefill-delayer-max-delay-ms 5500 \ @@ -235,7 +235,7 @@ python -m sglang.bench_serving \ **Model**: Qwen3.6-27B -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 1 @@ -279,7 +279,6 @@ export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 -export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 @@ -292,12 +291,12 @@ python3 -m sglang.launch_server \ --device npu \ --chunked-prefill-size 32768 \ --max-prefill-tokens 32768 \ - --mamba-scheduler-strategy extra_buffer \ + --mamba-radix-cache-strategy extra_buffer \ --trust-remote-code \ --max-running-requests 20 \ --max-mamba-cache-size 160 \ --mem-fraction-static 0.82 \ - --cuda-graph-bs 1 2 5 10 15 17 19 20 \ + --cuda-graph-bs-decode 1 2 5 10 15 17 19 20 \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ --speculative-algorithm NEXTN \ @@ -335,7 +334,7 @@ python -m sglang.bench_serving \ **Model**: Qwen3.6-27B -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 1 @@ -397,7 +396,7 @@ python3 -m sglang.launch_server \ --max-running-requests 64 \ --max-mamba-cache-size 74 \ --mem-fraction-static 0.7 \ - --cuda-graph-bs 2 8 16 32 40 45 50 54 \ + --cuda-graph-bs-decode 2 8 16 32 40 45 50 54 \ --enable-multimodal \ --quantization modelslim \ --mm-attention-backend ascend_attn \ @@ -433,7 +432,7 @@ python -m sglang.bench_serving \ **Model**: Qwen3.6-27B -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 1 @@ -491,7 +490,7 @@ python3 -m sglang.launch_server \ --max-running-requests 6 \ --max-mamba-cache-size 16 \ --mem-fraction-static 0.6 \ - --cuda-graph-bs 1 2 4 5 6 \ + --cuda-graph-bs-decode 1 2 4 5 6 \ --quantization modelslim \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ @@ -525,7 +524,7 @@ python -m sglang.bench_serving \ **Model**: Qwen3.6-27B -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 2 @@ -587,7 +586,7 @@ python3 -m sglang.launch_server \ --max-running-requests 6 \ --max-mamba-cache-size 7 \ --mem-fraction-static 0.63 \ - --cuda-graph-bs 1 2 4 5 6 \ + --cuda-graph-bs-decode 1 2 4 5 6 \ --enable-multimodal \ --quantization modelslim \ --mm-attention-backend ascend_attn \ @@ -619,7 +618,7 @@ python -m sglang.bench_serving \ **Model**: Qwen3.6-27B -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 2 @@ -662,7 +661,6 @@ export GLOO_SOCKET_IFNAME= export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 -export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=50 export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1 export SGLANG_SET_CPU_AFFINITY=1 @@ -682,7 +680,7 @@ python3 -m sglang.launch_server \ --max-running-requests 37 \ --max-mamba-cache-size 74 \ --mem-fraction-static 0.7 \ - --cuda-graph-bs 1 2 3 4 6 8 10 12 14 16 18 20 21 23 24 25 26 27 28 29 30 31 33 35 37 \ + --cuda-graph-bs-decode 1 2 3 4 6 8 10 12 14 16 18 20 21 23 24 25 26 27 28 29 30 31 33 35 37 \ --quantization modelslim \ --dtype bfloat16 \ --mamba-ssm-dtype bfloat16 \ @@ -717,7 +715,7 @@ python -m sglang.bench_serving \ **Model**: Qwen3.6-27B -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 2 @@ -779,7 +777,7 @@ python3 -m sglang.launch_server \ --max-running-requests 28 \ --max-mamba-cache-size 50 \ --mem-fraction-static 0.7 \ - --cuda-graph-bs 2 4 6 \ + --cuda-graph-bs-decode 2 4 6 \ --enable-multimodal \ --quantization modelslim \ --mm-attention-backend ascend_attn \ diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_6_35b_a3b.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_6_35b_a3b.mdx index 6bf66c156dc0..91ab777df8ee 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_6_35b_a3b.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_6_35b_a3b.mdx @@ -7,27 +7,27 @@ metatags: This page focuses on optimal configuration and benchmark results for Qwen3.6-35B-A3B on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [Qwen3.6-35B-A3B Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_6_35b_a3b). -On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. +On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. ### Low Latency | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | -| Qwen3.6-35B-A3B | Atlas 800I A3 | 1 | PD Mixed | 254k+1k | 16.1ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in254k-out1k) | +| Qwen3.6-35B-A3B | Ascend A3 Series Products | 1 | PD Mixed | 254k+1k | 16.1ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in254k-out1k) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | -| Qwen3.6-35B-A3B | Atlas 800I A3 | 1 | PD Mixed | 1024x1024 (30)+1024 | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in1024x1024-30-out1024-50ms) | -| Qwen3.6-35B-A3B | Atlas 800I A3 | 1 | PD Mixed | 1080p_30+256 | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in1080p-30-out256-50ms) | -| Qwen3.6-35B-A3B | Atlas 800I A3 | 1 | PD Mixed | 128k+1k | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in128k-out1k-50ms) | -| Qwen3.6-35B-A3B | Atlas 800I A3 | 1 | PD Mixed | 128k+1k (90% prefix cache hit rate) | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in128k-out1k-prefix90-50ms) | -| Qwen3.6-35B-A3B | Atlas 800I A3 | 1 | PD Mixed | 3.5k+1.5k | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in3k5-out1k5-50ms) | -| Qwen3.6-35B-A3B | Atlas 800I A3 | 1 | PD Mixed | 64k+1k | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in64k-out1k-50ms) | -| Qwen3.6-35B-A3B | Atlas 800I A3 | 1 | PD Mixed | 64k+1k (90% prefix cache hit rate) | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in64k-out1k-prefix90-50ms) | -| Qwen3.6-35B-A3B | Atlas 800I A3 | 2 | PD Mixed | 984k+1k | 40.91ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-2p-in984k-out1k) | +| Qwen3.6-35B-A3B | Ascend A3 Series Products | 1 | PD Mixed | 1024x1024 (30)+1024 | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in1024x1024-30-out1024-50ms) | +| Qwen3.6-35B-A3B | Ascend A3 Series Products | 1 | PD Mixed | 1080p_30+256 | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in1080p-30-out256-50ms) | +| Qwen3.6-35B-A3B | Ascend A3 Series Products | 1 | PD Mixed | 128k+1k | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in128k-out1k-50ms) | +| Qwen3.6-35B-A3B | Ascend A3 Series Products | 1 | PD Mixed | 128k+1k (90% prefix cache hit rate) | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in128k-out1k-prefix90-50ms) | +| Qwen3.6-35B-A3B | Ascend A3 Series Products | 1 | PD Mixed | 3.5k+1.5k | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in3k5-out1k5-50ms) | +| Qwen3.6-35B-A3B | Ascend A3 Series Products | 1 | PD Mixed | 64k+1k | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in64k-out1k-50ms) | +| Qwen3.6-35B-A3B | Ascend A3 Series Products | 1 | PD Mixed | 64k+1k (90% prefix cache hit rate) | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in64k-out1k-prefix90-50ms) | +| Qwen3.6-35B-A3B | Ascend A3 Series Products | 2 | PD Mixed | 984k+1k | 40.91ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-2p-in984k-out1k) | ## Optimal Configuration @@ -37,7 +37,7 @@ On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend **Model**: Qwen3.6-35B-A3B -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 1 @@ -82,7 +82,6 @@ export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 -export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_VIT_ENABLE_CUDA_GRAPH=1 export STREAMS_PER_DEVICE=32 @@ -103,7 +102,7 @@ python3 -m sglang.launch_server \ --max-running-requests 120 \ --max-mamba-cache-size 120 \ --mem-fraction-static 0.85 \ - --cuda-graph-bs 4 16 32 48 64 110 165 \ + --cuda-graph-bs-decode 4 16 32 48 64 110 165 \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ @@ -142,7 +141,7 @@ python -m sglang.bench_serving \ **Model**: Qwen3.6-35B-A3B -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 1 @@ -185,7 +184,6 @@ export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 -export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_SET_CPU_AFFINITY=1 export SGLANG_VIT_ENABLE_CUDA_GRAPH=1 export STREAMS_PER_DEVICE=32 @@ -205,7 +203,7 @@ python3 -m sglang.launch_server \ --max-running-requests 42 \ --max-mamba-cache-size 42 \ --mem-fraction-static 0.75 \ - --cuda-graph-bs 4 8 16 24 48 64 80 \ + --cuda-graph-bs-decode 4 8 16 24 48 64 80 \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ @@ -244,7 +242,7 @@ python -m sglang.bench_serving \ **Model**: Qwen3.6-35B-A3B -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 1 @@ -308,7 +306,7 @@ python3 -m sglang.launch_server \ --max-running-requests 3 \ --max-mamba-cache-size 3 \ --mem-fraction-static 0.9 \ - --cuda-graph-bs 1 2 3 \ + --cuda-graph-bs-decode 1 2 3 \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ @@ -344,7 +342,7 @@ python -m sglang.bench_serving \ **Model**: Qwen3.6-35B-A3B -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 1 @@ -402,11 +400,11 @@ python3 -m sglang.launch_server \ --max-prefill-tokens 65536 \ --trust-remote-code \ --enable-prefill-delayer \ - --mamba-scheduler-strategy extra_buffer \ + --mamba-radix-cache-strategy extra_buffer \ --max-running-requests 103 \ --max-mamba-cache-size 85 \ --mem-fraction-static 0.85 \ - --cuda-graph-bs 2 4 8 16 32 48 64 80 96 103 \ + --cuda-graph-bs-decode 2 4 8 16 32 48 64 80 96 103 \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ @@ -446,7 +444,7 @@ python -m sglang.bench_serving \ **Model**: Qwen3.6-35B-A3B -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 1 @@ -489,7 +487,6 @@ export HCCL_OP_EXPANSION_MODE=AIV export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 -export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 @@ -507,7 +504,7 @@ python3 -m sglang.launch_server \ --max-running-requests 1 \ --max-mamba-cache-size 6 \ --mem-fraction-static 0.65 \ - --cuda-graph-bs 1 \ + --cuda-graph-bs-decode 1 \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ @@ -543,7 +540,7 @@ python -m sglang.bench_serving \ **Model**: Qwen3.6-35B-A3B -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 1 @@ -606,7 +603,7 @@ python3 -m sglang.launch_server \ --max-running-requests 122 \ --max-mamba-cache-size 122 \ --mem-fraction-static 0.9 \ - --cuda-graph-bs 4 16 32 64 96 116 120 122 \ + --cuda-graph-bs-decode 4 16 32 64 96 116 120 122 \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ @@ -642,7 +639,7 @@ python -m sglang.bench_serving \ **Model**: Qwen3.6-35B-A3B -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 1 @@ -705,7 +702,7 @@ python3 -m sglang.launch_server \ --max-running-requests 10 \ --max-mamba-cache-size 20 \ --mem-fraction-static 0.65 \ - --cuda-graph-bs 2 4 8 12 14 16 \ + --cuda-graph-bs-decode 2 4 8 12 14 16 \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ @@ -741,7 +738,7 @@ python -m sglang.bench_serving \ **Model**: Qwen3.6-35B-A3B -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 1 @@ -800,11 +797,11 @@ python3 -m sglang.launch_server \ --max-total-tokens 470784 \ --max-prefill-tokens 65536 \ --trust-remote-code \ - --mamba-scheduler-strategy extra_buffer \ + --mamba-radix-cache-strategy extra_buffer \ --max-running-requests 40 \ --max-mamba-cache-size 200 \ --mem-fraction-static 0.9 \ - --cuda-graph-bs 2 8 16 24 32 36 40 \ + --cuda-graph-bs-decode 2 8 16 24 32 36 40 \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ @@ -844,7 +841,7 @@ python -m sglang.bench_serving \ **Model**: Qwen3.6-35B-A3B -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 2 @@ -888,7 +885,6 @@ export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 -export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_SET_CPU_AFFINITY=1 export STREAMS_PER_DEVICE=32 @@ -906,7 +902,7 @@ python3 -m sglang.launch_server \ --max-running-requests 1 \ --max-mamba-cache-size 6 \ --mem-fraction-static 0.68 \ - --cuda-graph-bs 1 \ + --cuda-graph-bs-decode 1 \ --enable-multimodal \ --mm-attention-backend ascend_attn \ --dtype bfloat16 \ diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_8b.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_8b.mdx index 82696fb69218..bdc42844350f 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_8b.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_8b.mdx @@ -7,21 +7,21 @@ metatags: This page focuses on optimal configuration and benchmark results for Qwen3-8B on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [Qwen3-8B Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_8b). -On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. +On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. ### Low Latency | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | -| Qwen3-8B | Atlas 800I A3 | 1 | PD Mixed | 3.5k+1.5k | 5ms | W8A8 INT8 | [Optimal Configuration](#qwen3-8b-w8a8-1p-in3k5-out1k5-5ms) | -| Qwen3-8B | Atlas 800I A3 | 1 | PD Mixed | 6k+1.5k | 11.79ms | W8A8 INT8 | [Optimal Configuration](#qwen3-8b-w8a8-1p-in6k-out1k5-bs16) | +| Qwen3-8B | Ascend A3 Series Products | 1 | PD Mixed | 3.5k+1.5k | 5ms | W8A8 INT8 | [Optimal Configuration](#qwen3-8b-w8a8-1p-in3k5-out1k5-5ms) | +| Qwen3-8B | Ascend A3 Series Products | 1 | PD Mixed | 6k+1.5k | 11.79ms | W8A8 INT8 | [Optimal Configuration](#qwen3-8b-w8a8-1p-in6k-out1k5-bs16) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | -| Qwen3-8B | Atlas 800I A3 | 1 | PD Mixed | 3.5k+1.5k | 37ms | W8A8 INT8 | [Optimal Configuration](#qwen3-8b-w8a8-1p-in3k5-out1k5-37ms) | +| Qwen3-8B | Ascend A3 Series Products | 1 | PD Mixed | 3.5k+1.5k | 37ms | W8A8 INT8 | [Optimal Configuration](#qwen3-8b-w8a8-1p-in3k5-out1k5-37ms) | ## Optimal Configuration @@ -31,7 +31,7 @@ On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend **Model**: Qwen3-8B -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 1 @@ -95,7 +95,7 @@ python3 -m sglang.launch_server \ --chunked-prefill-size 16384 \ --tp-size 1 \ --mem-fraction-static 0.85 \ - --cuda-graph-bs 8 12 24 36 48 51 55 60 63 64 66 68 70 \ + --cuda-graph-bs-decode 8 12 24 36 48 51 55 60 63 64 66 68 70 \ --dtype bfloat16 \ --speculative-draft-model-quantization unquant \ --speculative-algorithm EAGLE3 \ @@ -129,7 +129,7 @@ python -m sglang.bench_serving \ **Model**: Qwen3-8B -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 1 @@ -175,7 +175,6 @@ export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 -export SGLANG_ENABLE_SPEC_V2=1 python3 -m sglang.launch_server \ --model-path $MODEL_PATH \ @@ -192,7 +191,7 @@ python3 -m sglang.launch_server \ --chunked-prefill-size -1 \ --tp-size 2 \ --mem-fraction-static 0.894 \ - --cuda-graph-bs 1 \ + --cuda-graph-bs-decode 1 \ --dtype bfloat16 \ --speculative-draft-model-quantization unquant \ --speculative-algorithm EAGLE3 \ @@ -227,7 +226,7 @@ python -m sglang.bench_serving \ **Model**: Qwen3-8B -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 1 @@ -289,7 +288,7 @@ python3 -m sglang.launch_server \ --chunked-prefill-size -1 \ --tp-size 2 \ --mem-fraction-static 0.894 \ - --cuda-graph-bs 1 5 15 16 \ + --cuda-graph-bs-decode 1 5 15 16 \ --dtype bfloat16 \ --speculative-draft-model-quantization unquant \ --speculative-algorithm EAGLE3 \ diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_next_80b_a3b_instruct.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_next_80b_a3b_instruct.mdx index 15bd5d67eacf..f0f6e8e68aae 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_next_80b_a3b_instruct.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_next_80b_a3b_instruct.mdx @@ -7,21 +7,21 @@ metatags: This page focuses on optimal configuration and benchmark results for Qwen3-Next-80B-A3B-Instruct on the Ascend NPU. For environment setup, model weight download, feature configuration, and deployment instructions, etc., see the [Qwen3-Next-80B-A3B-Instruct Model Tutorial](/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_next_80b_a3b_instruct). -On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. +On the A3 Series, each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend NPU Reference](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware) for details. ### Low Latency | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | -| Qwen3-Next-80B-A3B-Instruct | Atlas 800I A3 | 1 | PD Mixed | 3.5k+1.5k | 20ms | W8A8 INT8 | [Optimal Configuration](#qwen3-next-80b-a3b-instruct-w8a8-1p-in3k5-out1k5-20ms) | -| Qwen3-Next-80B-A3B-Instruct | Atlas 800I A3 | 2 | PD Mixed | 6k+1.5k | 15.62ms | W8A8 INT8 | [Optimal Configuration](#qwen3-next-80b-a3b-instruct-w8a8-2p-in6k-out1k5-bs16) | +| Qwen3-Next-80B-A3B-Instruct | Ascend A3 Series Products | 1 | PD Mixed | 3.5k+1.5k | 20ms | W8A8 INT8 | [Optimal Configuration](#qwen3-next-80b-a3b-instruct-w8a8-1p-in3k5-out1k5-20ms) | +| Qwen3-Next-80B-A3B-Instruct | Ascend A3 Series Products | 2 | PD Mixed | 6k+1.5k | 15.62ms | W8A8 INT8 | [Optimal Configuration](#qwen3-next-80b-a3b-instruct-w8a8-2p-in6k-out1k5-bs16) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | -| Qwen3-Next-80B-A3B-Instruct | Atlas 800I A3 | 2 | PD Mixed | 3.5k+1.5k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-next-80b-a3b-instruct-w8a8-2p-in3k5-out1k5-50ms) | +| Qwen3-Next-80B-A3B-Instruct | Ascend A3 Series Products | 2 | PD Mixed | 3.5k+1.5k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-next-80b-a3b-instruct-w8a8-2p-in3k5-out1k5-50ms) | ## Optimal Configuration @@ -31,7 +31,7 @@ On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend **Model**: Qwen3-Next-80B-A3B-Instruct -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 1 @@ -82,7 +82,6 @@ export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=400 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 -export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 export SGLANG_NPU_USE_MULTI_STREAM=0 export SGLANG_SET_CPU_AFFINITY=1 @@ -116,7 +115,7 @@ python3 -m sglang.launch_server \ --speculative-draft-model-quantization unquant \ --chunked-prefill-size -1 \ --max-running-requests 2 \ - --cuda-graph-bs 2 \ + --cuda-graph-bs-decode 2 \ --mamba-ssm-dtype bfloat16 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --reasoning-parser qwen3 \ @@ -145,7 +144,7 @@ python -m sglang.bench_serving \ **Model**: Qwen3-Next-80B-A3B-Instruct -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 2 @@ -195,7 +194,6 @@ export HCCL_SOCKET_IFNAME= export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=330 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 -export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 export SGLANG_NPU_USE_MULTI_STREAM=0 export SGLANG_SET_CPU_AFFINITY=1 @@ -235,7 +233,7 @@ python3 -m sglang.launch_server \ --enable-dp-lm-head \ --moe-a2a-backend deepep \ --deepep-mode auto \ - --cuda-graph-bs 1 2 3 4 5 6 7 8 10 12 14 16 18 20 22 24 26 28 30 32 40 44 48 52 56 60 64 72 80 88 96 104 112 120 128 136 144 150 \ + --cuda-graph-bs-decode 1 2 3 4 5 6 7 8 10 12 14 16 18 20 22 24 26 28 30 32 40 44 48 52 56 60 64 72 80 88 96 104 112 120 128 136 144 150 \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder ``` @@ -262,7 +260,7 @@ python -m sglang.bench_serving \ **Model**: Qwen3-Next-80B-A3B-Instruct -**Hardware**: Atlas 800I A3 +**Hardware**: Ascend A3 Series Products **Cards**: 2 @@ -349,7 +347,7 @@ python3 -m sglang.launch_server \ --speculative-draft-model-quantization unquant \ --chunked-prefill-size -1 \ --max-running-requests 16 \ - --cuda-graph-bs 2 4 8 \ + --cuda-graph-bs-decode 2 4 8 \ --mamba-ssm-dtype bfloat16 \ --speculative-draft-model-path $DRAFT_MODEL_PATH \ --reasoning-parser qwen3 \ diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_r1.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_r1.mdx index e5833c23e635..15a893cdca61 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_r1.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_r1.mdx @@ -27,7 +27,7 @@ v0.5.16 or a later version. | Expert Parallelism | `--ep-size 16 \`
`--moe-a2a-backend deepep \`
`--deepep-mode auto` | | PD Disaggregation | `--disaggregation-mode prefill \`
`--disaggregation-transfer-backend ascend` | | Quantization | `--quantization modelslim` | -| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 4 8 20 21 22` | +| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 4 8 20 21 22` | | Speculative Decoding | `--speculative-algorithm NEXTN \`
`--speculative-num-steps 2 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 3` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | | DP LM Head | `--enable-dp-lm-head` | @@ -68,17 +68,17 @@ For multi-node deployment, download the weights to a shared directory accessible - [DeepSeek-R1-0528-W8A8](https://modelers.cn/models/Modelers_Park/DeepSeek-R1-0528-w8a8) (Quantized version, 647GB) -The W4A8 variant (376GB) can be deployed on 8 × 64GB of device memory (`--tp-size 8`), which corresponds to one full A2 -node or 8 dies on A3 (4 cards). +The W4A8 variant (376GB) can be deployed on 8 × 64GB of device memory (`--tp-size 8`), which corresponds to one full A2 Series +node or 8 dies on A3 Series (4 cards). The W8A8 variant (647GB) can be deployed on 16 × 64GB of device memory (`--tp-size 16`), which corresponds to one full -A3 node (8 cards, 16 dies) or two A2 nodes. +A3 Series node (8 cards, 16 dies) or two A2 Series nodes. This is the minimum recommended configuration. For optimized configurations, see [Best practices](#best-practices), which may require additional cards or nodes. -For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3), +For the hardware specifications (memory per die, dies per card, and the difference between A2 Series and A3 Series), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation @@ -94,7 +94,7 @@ Both **stable releases** and **daily builds** are available. The following comma For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). - + ```bash Command docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 @@ -129,7 +129,7 @@ quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 ``` - + ```bash Command docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 @@ -171,14 +171,14 @@ quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 Single-node deployment completes both prefill and decode within the same node (PD mixed mode), suitable for scenarios with limited hardware resources. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see -[DeepSeek-R1 Best Practice — W4A8 8P PD Mixed On A3](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_r1#single-node-pd-mixed). +[DeepSeek-R1 Best Practice — W4A8 8P PD Mixed On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_r1#single-node-pd-mixed). ### Multi-node PD disaggregation deployment PD disaggregation splits the prefill and decode stages onto separate nodes, reducing interference and improving throughput for high-concurrency scenarios. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see -[DeepSeek-R1 Best Practice — W8A8 32P PD Disaggregation On A3](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_r1#pd-disaggregation). +[DeepSeek-R1 Best Practice — W8A8 32P PD Disaggregation On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_r1#pd-disaggregation). ## Functional verification diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_v3_2.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_v3_2.mdx index 0a1363732e13..c3a0592fb0c8 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_v3_2.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_v3_2.mdx @@ -30,7 +30,7 @@ recommended to use v0.5.16 or a later version. | PD Disaggregation | `--disaggregation-mode prefill \`
`--disaggregation-transfer-backend ascend` | | Quantization | `--quantization modelslim` | | Speculative Decoding | `--speculative-algorithm NEXTN \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4` | -| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 1 2 3 4 5 6 7 8` | +| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 1 2 3 4 5 6 7 8` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | | DP LM Head | `--enable-dp-lm-head` | @@ -67,13 +67,13 @@ For multi-node deployment, download the weights to a shared directory accessible The W8A8 variant (694.47GB) can be deployed on 16 × 64GB of device memory (`--tp-size 16`), which corresponds to one full -A3 node (8 cards, 16 dies) or two A2 nodes. +A3 Series node (8 cards, 16 dies) or two A2 Series nodes. This is the minimum recommended configuration. For optimized configurations, see [Best practices](#best-practices), which may require additional cards or nodes. -For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3), +For the hardware specifications (memory per die, dies per card, and the difference between A2 Series and A3 Series), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation @@ -89,7 +89,7 @@ Both **stable releases** and **daily builds** are available. The following comma For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). - + ```bash Command docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 @@ -124,7 +124,7 @@ quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 ``` - + ```bash Command docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 @@ -166,7 +166,7 @@ quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 PD disaggregation splits the prefill and decode stages onto separate nodes, reducing interference and improving throughput for high-concurrency scenarios. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see -[DeepSeek-V3.2 Best Practice — PD Disaggregation On A3](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_v3_2#pd-disaggregation). +[DeepSeek-V3.2 Best Practice — PD Disaggregation On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_v3_2#pd-disaggregation). ## Functional verification diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_v4_flash.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_v4_flash.mdx index 545bfbd18fa8..bffa9c5330e1 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_v4_flash.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_v4_flash.mdx @@ -27,7 +27,7 @@ recommended to use v0.5.16 or a later version. | Expert Parallelism | `--ep-size 16 \`
`--moe-a2a-backend deepep \`
`--deepep-mode auto` | | PD Disaggregation | `--disaggregation-mode prefill \`
`--disaggregation-transfer-backend ascend` | | Quantization | `--quantization modelslim` | -| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 1 2 4 8 16 24 36 40 48 56` | +| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 1 2 4 8 16 24 36 40 48 56` | | Speculative Decoding | `--speculative-algorithm EAGLE \`
`--speculative-num-steps 2 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 3` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | | DP LM Head | `--enable-dp-lm-head` | @@ -65,14 +65,14 @@ For multi-node deployment, download the weights to a shared directory accessible - [DeepSeek-V4-Flash-w8a8-mtp](https://www.modelscope.cn/models/Eco-Tech/DeepSeek-V4-Flash-w8a8-mtp) (W8A8 quantized version, 300GB) -The W8A8 variant (300GB) can be deployed on 8 × 64GB of device memory (`--tp-size 8`), which corresponds to one full A2 -node or 8 dies on A3 (4 cards). +The W8A8 variant (300GB) can be deployed on 8 × 64GB of device memory (`--tp-size 8`), which corresponds to one full A2 Series +node or 8 dies on A3 Series (4 cards). This is the minimum recommended configuration. For optimized configurations, see [Best practices](#best-practices), which may require additional cards or nodes. -For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3), +For the hardware specifications (memory per die, dies per card, and the difference between A2 Series and A3 Series), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation @@ -88,7 +88,7 @@ Both **stable releases** and **daily builds** are available. The following comma For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). - + ```bash Command docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 @@ -123,7 +123,7 @@ quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 ``` - + ```bash Command docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 @@ -163,8 +163,8 @@ quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 ### Single-node PD mixed deployment Single-node deployment completes both prefill and decode within the same node (PD mixed mode), suitable for scenarios -with limited hardware resources. In such scenarios, DeepSeek-V4-Flash can be launched with half of an Atlas 800I A3 -node (4 cards, 8 dies) or a full Atlas 800I A2 node (8 cards). The configuration below was validated on Atlas 800I A3 +with limited hardware resources. In such scenarios, DeepSeek-V4-Flash can be launched with half of an Ascend A3 Series +node (4 cards, 8 dies) or a full Ascend A2 Series node (8 cards). The configuration below was validated on an Ascend A3 Series node (4 cards, 8 dies) with `--tp-size 8`. ```bash Command @@ -188,7 +188,6 @@ export INF_NAN_MODE_FORCE_DISABLE=1 export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True export SGLANG_DSV4_FP4_EXPERTS=False export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 -export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 export SGLANG_OPT_BF16_FP32_GEMM_ALGO=torch export SGLANG_OPT_DEEPGEMM_HC_PRENORM=False @@ -226,14 +225,14 @@ python3 -m sglang.launch_server \ ``` For the complete benchmark data and additional single-node configurations, see -[DeepSeek-V4-Flash Best Practice — PD Mixed On A3](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_v4_flash#single-node-pd-mixed). +[DeepSeek-V4-Flash Best Practice — PD Mixed On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_v4_flash#single-node-pd-mixed). ### Multi-node PD disaggregation deployment PD disaggregation splits the prefill and decode stages onto separate nodes, reducing interference and improving throughput for high-concurrency scenarios. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see -[DeepSeek-V4-Flash Best Practice — PD Disaggregation On A3](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_v4_flash#pd-disaggregation). +[DeepSeek-V4-Flash Best Practice — PD Disaggregation On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/deepseek_v4_flash#pd-disaggregation). ## Functional verification diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_1.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_1.mdx index 5d8ec0fa9d4f..84a3121f74c4 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_1.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_1.mdx @@ -31,7 +31,7 @@ v0.5.16 or a later version. | PD Disaggregation | `--disaggregation-mode prefill \`
`--disaggregation-transfer-backend ascend` | | Quantization | `--quantization modelslim` | | Chunked Prefill | auto based on device memory, or set explicit value;
disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 16384` | -| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 1 2 3 4 5 6` | +| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 1 2 3 4 5 6` | | Speculative Decoding | `--speculative-algorithm NEXTN \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4 \`
`--speculative-draft-model-quantization unquant` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | | DP LM Head | `--enable-dp-lm-head` | @@ -71,13 +71,13 @@ For multi-node deployment, download the weights to a shared directory accessible We recommend deploying the W4A8 variant for reduced resource usage and higher throughput. -It (420.17GB) can be deployed on 8 × 64GB of device memory (`--tp-size 8`), which corresponds to one full A2 node or 8 dies on A3 (4 cards). +It (420.17GB) can be deployed on 8 × 64GB of device memory (`--tp-size 8`), which corresponds to one full A2 Series node or 8 dies on A3 Series (4 cards). This is the minimum recommended configuration. For optimized configurations, see [Best practices](#best-practices), which may require additional cards or nodes. -For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3), +For the hardware specifications (memory per die, dies per card, and the difference between A2 Series and A3 Series), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation @@ -93,7 +93,7 @@ Both **stable releases** and **daily builds** are available. The following comma For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). - + ```bash Command docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 @@ -128,7 +128,7 @@ quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 ``` - + ```bash Command docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 @@ -167,18 +167,18 @@ quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 ### Multi-node PD mixed deployment -Multi-node deployment distributes the model across multiple Atlas 800I A3 nodes using tensor parallelism while keeping +Multi-node deployment distributes the model across multiple Ascend A3 Series nodes using tensor parallelism while keeping prefill and decode on the same nodes (PD mixed mode), suitable for scenarios that need more device memory than a single node can provide. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see -[GLM-5.1 Best Practice — Multi-node PD Mixed On A3](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/glm_5_1#multi-node-pd-mixed). +[GLM-5.1 Best Practice — Multi-node PD Mixed On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/glm_5_1#multi-node-pd-mixed). ### Multi-node PD disaggregation deployment PD disaggregation splits the prefill and decode stages onto separate nodes, reducing interference and improving throughput for high-concurrency scenarios. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see -[GLM-5.1 Best Practice — PD Disaggregation On A3](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/glm_5_1#pd-disaggregation). +[GLM-5.1 Best Practice — PD Disaggregation On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/glm_5_1#pd-disaggregation). ## Functional verification diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_2.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_2.mdx index 8b82e41fadb7..4a18b869c20f 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_2.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/glm_5_2.mdx @@ -23,7 +23,7 @@ multi-node deployment, prefill-decode disaggregation, feature configuration, and | PD Disaggregation | `--disaggregation-mode prefill \`
`--disaggregation-transfer-backend ascend` | | Quantization | `--quantization modelslim` | | Chunked Prefill | auto based on device memory, or set explicit value;
disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 16384` | -| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs`; e.g., `--cuda-graph-bs 16` | +| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 16` | | Speculative Decoding | `--speculative-algorithm NEXTN \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4 \`
`--speculative-draft-model-quantization unquant` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | | DP LM Head | `--enable-dp-lm-head` | @@ -63,13 +63,13 @@ For multi-node deployment, download the weights to a shared directory accessible We recommend deploying the W8A8 variant for reduced resource usage and higher throughput. -It (774.08GB) can be deployed on 16 × 64GB of device memory (`--tp-size 16`), which corresponds to one full A3 node (8 cards, 16 dies) or two A2 nodes. +It (774.08GB) can be deployed on 16 × 64GB of device memory (`--tp-size 16`), which corresponds to one full A3 Series node (8 cards, 16 dies) or two A2 Series nodes. This is the minimum recommended configuration. For optimized configurations, see [Best practices](#best-practices), which may require additional cards or nodes. -For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3), +For the hardware specifications (memory per die, dies per card, and the difference between A2 Series and A3 Series), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation @@ -82,7 +82,7 @@ The dependencies required for the NPU runtime environment have been integrated i online platform. You can directly pull it. - + ```bash Command docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 @@ -117,7 +117,7 @@ quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 ``` - + ```bash Command docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 @@ -164,7 +164,7 @@ quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 ### Single-node deployment -Quantized model `GLM-5.2-w8a8` can be deployed on one Atlas 800I A3 node. +Quantized model `GLM-5.2-w8a8` can be deployed on one Ascend A3 Series node. Run the following script to execute online inference. @@ -220,7 +220,7 @@ python3 -m sglang.launch_server \ --mem-fraction-static 0.7 \ --port 8000 \ --served-model-name glm-5 \ - --cuda-graph-bs 16 \ + --cuda-graph-bs-decode 16 \ --quantization modelslim \ --speculative-draft-model-quantization unquant \ --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \ @@ -229,7 +229,7 @@ python3 -m sglang.launch_server \ ### Multi-node deployment -Quantized model `GLM-5.2-w8a8` can be deployed on two Atlas 800I A3 nodes. +Quantized model `GLM-5.2-w8a8` can be deployed on two Ascend A3 Series nodes. Modify the IP addresses of the two nodes, then run the same script on both nodes. @@ -302,7 +302,7 @@ do --mem-fraction-static 0.8 \ --port 8000 \ --served-model-name glm-5 \ - --cuda-graph-max-bs 32 \ + --cuda-graph-max-bs-decode 32 \ --moe-a2a-backend deepep \ --deepep-mode auto \ --speculative-draft-model-quantization unquant \ @@ -421,8 +421,8 @@ do --port 8003 --trust-remote-code --nnodes 1 --node-rank $i --tp-size 16 --dp-size 16 --ep-size 16 \ --mem-fraction-static 0.8 --max-running-requests 128 --attention-backend ascend --device npu --quantization modelslim \ --served-model-name glm-5 --moe-a2a-backend deepep --enable-dp-attention --deepep-mode low_latency \ - --cuda-graph-max-bs 4 --disaggregation-transfer-backend ascend --watchdog-timeout 9000 --context-length 180000 \ - --tokenizer-worker-num 4 --prefill-round-robin-balance --disable-shared-experts-fusion --dtype bfloat16 --load-balance-method round_robin \ + --cuda-graph-max-bs-decode 4 --disaggregation-transfer-backend ascend --watchdog-timeout 9000 --context-length 180000 \ + --tokenizer-worker-num 4 --disable-shared-experts-fusion --dtype bfloat16 --load-balance-method round_robin \ --speculative-draft-model-quantization unquant \ --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 NODE_RANK=$i diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/hy3.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/hy3.mdx index 0ad0b420a48c..a695bb7237f7 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/hy3.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/hy3.mdx @@ -13,15 +13,15 @@ parameters, developed by the Tencent Hy Team. It features 192 experts with top-8 outperforms similar-size models and rivals flagship open-source models with 2–5× the parameters, with notable improvements in agentic workflows, long-context tasks, and production reliability. -This document demonstrates the deployment of Hy3 on Ascend NPUs using SGLang, including single-node (Atlas 800I A3) and -multi-node (Atlas 800I A2) PD mixed mode and speculative decoding. +This document demonstrates the deployment of Hy3 on Ascend NPUs using SGLang, including single-node (Ascend A3 Series) and +multi-node (Ascend A2 Series) PD mixed mode and speculative decoding. ## Supported features | Feature | Example usage | |--------------------------|----------------------------------------------------------------------------------------------------------------------------------------| | Tensor Parallelism | `--tp-size 16` | -| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs`; e.g. `--cuda-graph-bs 4 8 16 20 24 28 32 36 40` | +| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs-decode`; e.g. `--cuda-graph-bs-decode 4 8 16 20 24 28 32 36 40` | | Speculative Decoding | `--speculative-algorithm EAGLE \`
`--speculative-num-steps 2 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 3` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | | Reasoning Mode | `extra_body={"chat_template_kwargs": {"reasoning_effort": "high"}}` — deep chain-of-thought;
`"reasoning_effort": "low"` — short thinking chain;
`"reasoning_effort": "no_think"` — direct response, no thinking | @@ -60,12 +60,12 @@ Hy3 has 295B total parameters. If you need to download model weights, check the The BF16 variant (597.60GB) can be deployed on 16 × 64GB of device memory (`--tp-size 16`), which corresponds to one full -Atlas 800I A3 node (8 cards × 2 dies) or two Atlas 800I A2 nodes (8 cards × 1 die each). +Ascend A3 Series node (8 cards × 2 dies) or two Ascend A2 Series nodes (8 cards × 1 die each). It is recommended to download the model weights to a shared directory accessible from within the container. -For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3), +For the hardware specifications (memory per die, dies per card, and the difference between A2 Series and A3 Series), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation @@ -84,7 +84,7 @@ For details, see - + ```bash Command docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 @@ -119,7 +119,7 @@ quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 ``` - + ```bash Command docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 @@ -160,7 +160,7 @@ quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 ### Single-node online deployment Single-node deployment completes both prefill and decode within the same node (PD mixed mode). Hy3 with 295B -parameters requires all 16 logical NPUs on an Atlas 800I A3 server (8 physical NPUs × 2 logical cores each, 64 GB HBM +parameters requires all 16 logical NPUs on an Ascend A3 Series server (8 physical NPUs × 2 logical cores each, 64 GB HBM per logical NPU). Set the following environment variables before launching the server: @@ -185,7 +185,6 @@ export HCCL_SOCKET_IFNAME=lo export GLOO_SOCKET_IFNAME=lo # SGLang settings -export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 ``` @@ -209,17 +208,17 @@ python3 -m sglang.launch_server \ --base-gpu-id 0 \ --prefill-max-requests 40 \ --max-running-requests 40 \ - --cuda-graph-bs 4 8 16 20 24 28 32 36 40 \ + --cuda-graph-bs-decode 4 8 16 20 24 28 32 36 40 \ --speculative-algorithm EAGLE \ --speculative-num-steps 2 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 3 ``` -### Multi-node PD mixed deployment on Atlas 800I A2 +### Multi-node PD mixed deployment on Ascend A2 Series -Each Atlas 800I A2 node provides 8 NPUs (Ascend 910B, 64 GB HBM per NPU). Hy3 requires all 16 logical NPUs across -**2 × Atlas 800I A2** nodes in PD mixed mode (prefill and decode are colocated on the same 2-node cluster). +Each Ascend A2 Series node provides 8 NPUs (64 GB HBM per NPU). Hy3 requires all 16 logical NPUs across +**2 × Ascend A2 Series** nodes in PD mixed mode (prefill and decode are colocated on the same 2-node cluster). On each node, set the following environment variables before launching the server: @@ -249,7 +248,6 @@ export HCCL_SOCKET_IFNAME= export GLOO_SOCKET_IFNAME= # SGLang settings -export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 ``` @@ -286,7 +284,7 @@ for i in "${!NODE_IPS[@]}"; do --base-gpu-id 0 \ --prefill-max-requests 40 \ --max-running-requests 40 \ - --cuda-graph-bs 4 8 16 20 24 28 32 36 40 \ + --cuda-graph-bs-decode 4 8 16 20 24 28 32 36 40 \ --speculative-algorithm EAGLE \ --speculative-num-steps 2 \ --speculative-eagle-topk 1 \ @@ -359,7 +357,7 @@ For performance data and benchmark commands, see [Performance Testing on Ascend ### Best practice configuration reference Hy3 has no standalone best practice page yet. The [Single-node online deployment](#single-node-online-deployment) and -[Multi-node PD mixed deployment on Atlas 800I A2](#multi-node-pd-mixed-deployment-on-atlas-800i-a2) sections above provide +[Multi-node PD mixed deployment on Ascend A2 Series](#multi-node-pd-mixed-deployment-on-ascend-a2-series) sections above provide the ready-to-use scripts for the supported PD mixed cases. For the full catalog of optimization features and their parameter and compatibility details, diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/kimi_k2_6.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/kimi_k2_6.mdx index 35d50fb65d1e..7d8bec441194 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/kimi_k2_6.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/kimi_k2_6.mdx @@ -30,7 +30,7 @@ v0.5.16 or a later version. | PD Disaggregation | `--disaggregation-mode prefill \`
`--disaggregation-transfer-backend ascend` | | Quantization | `--quantization modelslim` | | Chunked Prefill | auto based on device memory, or set explicit value;
disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 32768` | -| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 1 2 4 8 12 16 24 32 48 64 96 120` | +| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 1 2 4 8 12 16 24 32 48 64 96 120` | | Speculative Decoding | `--speculative-algorithm EAGLE3 \`
`--speculative-draft-model-path /path/to/draft-model-weights \`
`--speculative-num-steps 4 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 5 \`
`--speculative-draft-model-quantization unquant` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | | DP LM Head | `--enable-dp-lm-head` | @@ -73,13 +73,13 @@ For multi-node deployment, download the weights to a shared directory accessible We recommend deploying the W4A8 variant for reduced resource usage and higher throughput. -It (535.91GB) can be deployed on 16 × 64GB of device memory (`--tp-size 16`), which corresponds to one full A3 node (8 cards, 16 dies) or two A2 nodes. +It (535.91GB) can be deployed on 16 × 64GB of device memory (`--tp-size 16`), which corresponds to one full A3 Series node (8 cards, 16 dies) or two A2 Series nodes. This is the minimum recommended configuration. For optimized configurations, see [Best practices](#best-practices), which may require additional cards or nodes. -For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3), +For the hardware specifications (memory per die, dies per card, and the difference between A2 Series and A3 Series), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation @@ -95,7 +95,7 @@ Both **stable releases** and **daily builds** are available. The following comma For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). - + ```bash Command docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 @@ -130,7 +130,7 @@ quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 ``` - + ```bash Command docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 @@ -172,22 +172,22 @@ quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 Single-node deployment completes both prefill and decode within the same node (PD mixed mode), suitable for scenarios with limited hardware resources. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see -[Kimi K2.6 Best Practice — PD Mixed On A3](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/kimi_k2_6#single-node-pd-mixed). +[Kimi K2.6 Best Practice — PD Mixed On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/kimi_k2_6#single-node-pd-mixed). ### Multi-node online deployment -Multi-node deployment distributes the model across multiple Atlas 800I A3 nodes using tensor parallelism while keeping +Multi-node deployment distributes the model across multiple Ascend A3 Series nodes using tensor parallelism while keeping prefill and decode on the same nodes (PD mixed mode), suitable for scenarios that need more device memory than a single node can provide. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see -[Kimi-K2.6 Best Practice — Multi-node On A3](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/kimi_k2_6#multi-node-pd-mixed). +[Kimi-K2.6 Best Practice — Multi-node On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/kimi_k2_6#multi-node-pd-mixed). ### Multi-node PD disaggregation deployment PD disaggregation splits the prefill and decode stages onto separate nodes, reducing interference and improving throughput for high-concurrency scenarios. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see -[Kimi-K2.6 Best Practice — PD Disaggregation On A3](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/kimi_k2_6#pd-disaggregation). +[Kimi-K2.6 Best Practice — PD Disaggregation On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/kimi_k2_6#pd-disaggregation). ## Functional verification diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/mimo_v2_flash.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/mimo_v2_flash.mdx index e42eeeec9e19..a497850d5514 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/mimo_v2_flash.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/mimo_v2_flash.mdx @@ -26,7 +26,7 @@ v0.5.16 or a later version. | Expert Parallelism | `--moe-a2a-backend deepep \`
`--deepep-mode low_latency` | | PD Disaggregation | `--disaggregation-mode prefill \`
`--disaggregation-transfer-backend ascend` | | Quantization | `--quantization modelslim` | -| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs`; e.g., `--cuda-graph-bs 1 2 4 8 12 16 20 24 28 32` | +| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 1 2 4 8 12 16 20 24 28 32` | | Speculative Decoding | `--speculative-algorithm EAGLE \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4 \`
`--enable-multi-layer-eagle` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0` | | DP LM Head | `--enable-dp-lm-head` | @@ -64,14 +64,14 @@ For multi-node deployment, download the weights to a shared directory accessible - [MiMo-V2-Flash-W8A8](https://www.modelscope.cn/models/iridiumine/MiMo-V2-Flash-W8A8) (Quantized version, 311.5GB) -The W8A8 variant (311.5GB) can be deployed on 8 × 64GB of device memory (`--tp-size 8`), which corresponds to one full A2 -node or 8 dies on A3 (4 cards). +The W8A8 variant (311.5GB) can be deployed on 8 × 64GB of device memory (`--tp-size 8`), which corresponds to one full A2 Series +node or 8 dies on A3 Series (4 cards). This is the minimum recommended configuration. For optimized configurations, see [Best practices](#best-practices), which may require additional cards or nodes. -For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3), +For the hardware specifications (memory per die, dies per card, and the difference between A2 Series and A3 Series), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation @@ -87,7 +87,7 @@ Both **stable releases** and **daily builds** are available. The following comma For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). - + ```bash Command docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 @@ -122,7 +122,7 @@ quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 ``` - + ```bash Command docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 @@ -164,7 +164,7 @@ quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 PD disaggregation splits the prefill and decode stages onto separate nodes, reducing interference and improving throughput for high-concurrency scenarios. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see -[MiMo-V2-Flash Best Practice — W8A8 24P PD Disaggregation On A3](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/mimo_v2_flash#pd-disaggregation). +[MiMo-V2-Flash Best Practice — W8A8 24P PD Disaggregation On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/mimo_v2_flash#pd-disaggregation). ## Functional verification diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/minimax_m2_5.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/minimax_m2_5.mdx index 29640527639a..01d1f16051cd 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/minimax_m2_5.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/minimax_m2_5.mdx @@ -28,7 +28,7 @@ v0.5.16 or a later version. | Expert Parallelism | `--ep-size 16 \`
`--moe-a2a-backend ascend_fuseep \`
`--deepep-mode auto` | | Quantization | `--quantization modelslim` | | Chunked Prefill | auto based on device memory, or set explicit value;
disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 8192` | -| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 8 16 24 32 48 64 80` | +| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 8 16 24 32 48 64 80` | | Speculative Decoding | `--speculative-algorithm EAGLE3 \`
`--speculative-draft-model-path /path/to/draft-model-weights \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4 \`
`--speculative-draft-model-quantization unquant` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | @@ -64,14 +64,14 @@ Before downloading model weights, check the model size to reserve enough disk sp - [MiniMax-M2.5-eagle-model](https://www.modelscope.cn/models/sgl-npu/MiniMax-M2.5-eagel-model-0318) (EAGLE3 draft model for speculative decoding) -The W8A8 variant (230.82GB) can be deployed on 8 × 64GB of device memory (`--tp-size 8`), which corresponds to one full A2 -node or 8 dies on A3 (4 cards). +The W8A8 variant (230.82GB) can be deployed on 8 × 64GB of device memory (`--tp-size 8`), which corresponds to one full A2 Series +node or 8 dies on A3 Series (4 cards). This is the minimum recommended configuration. For optimized configurations, see [Best practices](#best-practices), which may require additional cards or nodes. -For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3), +For the hardware specifications (memory per die, dies per card, and the difference between A2 Series and A3 Series), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation @@ -87,7 +87,7 @@ Both **stable releases** and **daily builds** are available. The following comma For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). - + ```bash Command docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 @@ -122,7 +122,7 @@ quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 ``` - + ```bash Command docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 @@ -164,7 +164,7 @@ quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 Single-node deployment completes both prefill and decode within the same node (PD mixed mode), suitable for scenarios with limited hardware resources. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see -[MiniMax-M2.5 Best Practice — PD Mixed On A3](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/minimax_m2_5#single-node-pd-mixed). +[MiniMax-M2.5 Best Practice — PD Mixed On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/minimax_m2_5#single-node-pd-mixed). ## Functional verification diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_235b_a22b.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_235b_a22b.mdx index f75e7ed6bfe8..a9af4e443282 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_235b_a22b.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_235b_a22b.mdx @@ -30,7 +30,7 @@ v0.5.16 or a later version. | PD Disaggregation | `--disaggregation-mode prefill \`
`--disaggregation-transfer-backend ascend` | | Quantization | `--quantization modelslim` | | Chunked Prefill | auto based on device memory, or set explicit value;
disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 94208` | -| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 1 2 4 8 16 20 24 26 27` | +| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 1 2 4 8 16 20 24 26 27` | | Speculative Decoding | `--speculative-algorithm EAGLE3 \`
`--speculative-draft-model-path /path/to/draft-model-weights \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4 \`
`--speculative-draft-model-quantization unquant` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | | DP LM Head | `--enable-dp-lm-head` | @@ -71,15 +71,15 @@ For multi-node deployment, download the weights to a shared directory accessible The BF16 variant (470.21GB) can be deployed on 16 × 64GB of device memory (`--tp-size 16`), which corresponds to one full -A3 node (8 cards, 16 dies) or two A2 nodes. +A3 Series node (8 cards, 16 dies) or two A2 Series nodes. -The W8A8 variant (236.80GB) can be deployed on 8 × 64GB (`--tp-size 8`), which corresponds to one full A2 node or 8 dies on A3 (4 cards). +The W8A8 variant (236.80GB) can be deployed on 8 × 64GB (`--tp-size 8`), which corresponds to one full A2 Series node or 8 dies on A3 Series (4 cards). This is the minimum recommended configuration. For optimized configurations, see [Best practices](#best-practices), which may require additional cards or nodes. -For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3), +For the hardware specifications (memory per die, dies per card, and the difference between A2 Series and A3 Series), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation @@ -95,7 +95,7 @@ Both **stable releases** and **daily builds** are available. The following comma For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). - + ```bash Command docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 @@ -130,7 +130,7 @@ quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 ``` - + ```bash Command docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 @@ -172,13 +172,13 @@ quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 Single-node deployment completes both prefill and decode within the same node (PD mixed mode), suitable for scenarios with limited hardware resources. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see -[Qwen3-235B-A22B Best Practice — PD Mixed On A3](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_235b_a22b#single-node-pd-mixed). +[Qwen3-235B-A22B Best Practice — PD Mixed On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_235b_a22b#single-node-pd-mixed). ### Multi-node PD disaggregation deployment -#### 256K long-sequence PD disaggregation on 2 x Atlas 800I A3 (without CP) +#### 256K long-sequence PD disaggregation on 2 x Ascend A3 Series nodes (without CP) -This configuration uses PD disaggregation for **256K long-sequence inference** on 2 x Atlas 800I A3 with context parallel disabled. +This configuration uses PD disaggregation for **256K long-sequence inference** on 2 x Ascend A3 Series nodes with context parallel disabled. The following command is based on the **W8A8** quantized model. 1. Set the shared environment variables on both prefill and decode nodes: @@ -291,7 +291,7 @@ python3 -m sglang_router.launch_router \ --prometheus-port 29010 ``` -#### Prefill Context Parallel (PCP) on 2 x Atlas 800I A3 +#### Prefill Context Parallel (PCP) on 2 x Ascend A3 Series nodes This configuration enables **Prefill Context Parallel** (`--enable-prefill-context-parallel`) to split the context across CP ranks during prefill, reducing per-device memory pressure and improving TTFT for long sequences. diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_30b_a3b.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_30b_a3b.mdx index f13f56614e6e..fc96c6a67832 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_30b_a3b.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_30b_a3b.mdx @@ -26,7 +26,7 @@ v0.5.16 or a later version. | Tensor Parallelism | `--tp-size 2` | | Data Parallelism | `--dp-size 2` | | Quantization | `--quantization modelslim` | -| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 1 5 15 40 70 100 120 130 140 146 150 154 156 158 160 162` | +| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 1 5 15 40 70 100 120 130 140 146 150 154 156 158 160 162` | | Speculative Decoding | `--speculative-algorithm EAGLE3 \`
`--speculative-draft-model-path /path/to/draft-model-weights \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4 \`
`--speculative-draft-model-quantization unquant` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | @@ -64,13 +64,13 @@ For multi-node deployment, download the weights to a shared directory accessible - [Qwen3-a3B_eagle3](https://www.modelscope.cn/models/vllm-ascend/Qwen3-a3B_eagle3) (EAGLE3 draft model for speculative decoding) -Both variants fit within a single 64GB die. For single-node deployment, `--tp-size 1` is sufficient on either A2 or A3. +Both variants fit within a single 64GB die. For single-node deployment, `--tp-size 1` is sufficient on either A2 Series or A3 Series. This is the minimum recommended configuration. For optimized configurations, see [Best practices](#best-practices), which may require additional cards or nodes. -For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3), +For the hardware specifications (memory per die, dies per card, and the difference between A2 Series and A3 Series), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation @@ -86,7 +86,7 @@ Both **stable releases** and **daily builds** are available. The following comma For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). - + ```bash Command docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 @@ -121,7 +121,7 @@ quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 ``` - + ```bash Command docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 @@ -163,7 +163,7 @@ quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 Single-node deployment completes both prefill and decode within the same node (PD mixed mode), suitable for scenarios with limited hardware resources. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see -[Qwen3-30B-A3B Best Practice — PD Mixed On A3](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_30b_a3b#single-node-pd-mixed). +[Qwen3-30B-A3B Best Practice — PD Mixed On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_30b_a3b#single-node-pd-mixed). ## Functional verification diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_32b.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_32b.mdx index 835fdd765d31..77b6a95fec6e 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_32b.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_32b.mdx @@ -24,7 +24,7 @@ version. |-------------------------------|-----------------------------------------------------------------------------------------------| | Tensor Parallelism | `--tp-size 4` | | Quantization | `--quantization modelslim` | -| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 1 4 8 16` | +| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 1 4 8 16` | | Speculative Decoding | `--speculative-algorithm EAGLE3 \`
`--speculative-draft-model-path /path/to/draft-model-weights \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4 \`
`--speculative-draft-model-quantization unquant` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | @@ -62,7 +62,7 @@ For multi-node deployment, download the weights to a shared directory accessible - [Eagle3-Qwen3-32B-zh](https://www.modelscope.cn/models/Zjcxy-SmartAI/Eagle3-Qwen3-32B-zh) (EAGLE3 draft model for speculative decoding) -The BF16 variant (65.54GB) can be deployed on 2 × 64GB of device memory (`--tp-size 2`), which corresponds to 2 cards on A2 or 1 card (2 dies) on A3. +The BF16 variant (65.54GB) can be deployed on 2 × 64GB of device memory (`--tp-size 2`), which corresponds to 2 cards on A2 Series or 1 card (2 dies) on A3 Series. The W8A8 variant (42.77GB) fits within a single die. @@ -70,7 +70,7 @@ The W8A8 variant (42.77GB) fits within a single die. This is the minimum recommended configuration. For optimized configurations, see [Best practices](#best-practices), which may require additional cards or nodes. -For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3), +For the hardware specifications (memory per die, dies per card, and the difference between A2 Series and A3 Series), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation @@ -86,7 +86,7 @@ Both **stable releases** and **daily builds** are available. The following comma For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). - + ```bash Command docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 @@ -121,7 +121,7 @@ quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 ``` - + ```bash Command docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 @@ -163,7 +163,7 @@ quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 Single-node deployment completes both prefill and decode within the same node (PD mixed mode), suitable for scenarios with limited hardware resources. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see -[Qwen3-32B Best Practice — PD Mixed On A3](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_32b#single-node-pd-mixed). +[Qwen3-32B Best Practice — PD Mixed On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_32b#single-node-pd-mixed). ## Functional verification diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_5_397b.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_5_397b.mdx index 6a7181bfa6cf..21fd78ddd1ce 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_5_397b.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_5_397b.mdx @@ -28,7 +28,7 @@ use v0.5.16 or a later version. | Data Parallelism | `--dp-size 8` | | Expert Parallelism | `--ep-size 16 \`
`--moe-a2a-backend deepep \`
`--deepep-mode auto` | | Quantization | `--quantization modelslim` | -| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 2 4 6 8 10 12 14 16 18 20` | +| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 2 4 6 8 10 12 14 16 18 20` | | Speculative Decoding | `--speculative-algorithm NEXTN \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4 \`
`--speculative-draft-model-quantization unquant` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | | DP LM Head | `--enable-dp-lm-head` | @@ -65,14 +65,14 @@ For multi-node deployment, download the weights to a shared directory accessible - [Eco-Tech/Qwen3.5-397B-A17B-w4a8-mtp](https://www.modelscope.cn/models/Eco-Tech/Qwen3.5-397B-A17B-w4a8-mtp) (W4A8 quantized version with MTP, 235.88GB) -The W4A8 variant (235.88GB) can be deployed on 8 × 64GB of device memory (`--tp-size 8`), which corresponds to one full A2 -node or 8 dies on A3 (4 cards). +The W4A8 variant (235.88GB) can be deployed on 8 × 64GB of device memory (`--tp-size 8`), which corresponds to one full A2 Series +node or 8 dies on A3 Series (4 cards). This is the minimum recommended configuration. For optimized configurations, see [Best practices](#best-practices), which may require additional cards or nodes. -For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3), +For the hardware specifications (memory per die, dies per card, and the difference between A2 Series and A3 Series), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation @@ -88,7 +88,7 @@ Both **stable releases** and **daily builds** are available. The following comma For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). - + ```bash Command docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 @@ -123,7 +123,7 @@ quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 ``` - + ```bash Command docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 @@ -165,7 +165,7 @@ quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 Single-node deployment completes both prefill and decode within the same node (PD mixed mode), suitable for scenarios with limited hardware resources. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see -[Qwen3.5-397B Best Practice — PD Mixed On A3](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_5_397b#single-node-pd-mixed). +[Qwen3.5-397B Best Practice — PD Mixed On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_5_397b#single-node-pd-mixed). ## Functional verification diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_6_27b.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_6_27b.mdx index b4366bce5b48..96d914aa9865 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_6_27b.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_6_27b.mdx @@ -26,7 +26,7 @@ v0.5.16 or a later version. | Tensor Parallelism | `--tp-size 2` | | Quantization | `--quantization modelslim` | | Chunked Prefill | auto based on device memory, or set explicit value;
disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 32768` | -| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 2 8 16 32 48` | +| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 2 8 16 32 48` | | Speculative Decoding | `--speculative-algorithm NEXTN \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | @@ -62,13 +62,13 @@ For multi-node deployment, download the weights to a shared directory accessible - [Eco-Tech/Qwen3.6-27B-w8a8](https://www.modelscope.cn/models/Eco-Tech/Qwen3.6-27B-w8a8) (W8A8 quantized version, 36.45GB) -The W8A8 variant (36.45GB) fits within a single 64GB die. For single-node deployment, `--tp-size 1` is sufficient on either A2 or A3. +The W8A8 variant (36.45GB) fits within a single 64GB die. For single-node deployment, `--tp-size 1` is sufficient on either A2 Series or A3 Series. This is the minimum recommended configuration. For optimized configurations, see [Best practices](#best-practices), which may require additional cards or nodes. -For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3), +For the hardware specifications (memory per die, dies per card, and the difference between A2 Series and A3 Series), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation @@ -84,7 +84,7 @@ Both **stable releases** and **daily builds** are available. The following comma For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). - + ```bash Command docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 @@ -119,7 +119,7 @@ quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 ``` - + ```bash Command docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 @@ -161,7 +161,7 @@ quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 Single-node deployment completes both prefill and decode within the same node (PD mixed mode), suitable for scenarios with limited hardware resources. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see -[Qwen3.6-27B Best Practice — PD Mixed On A3](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_6_27b#single-node-pd-mixed). +[Qwen3.6-27B Best Practice — PD Mixed On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_6_27b#single-node-pd-mixed). ## Functional verification diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_6_35b_a3b.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_6_35b_a3b.mdx index 95d268049354..70370e652918 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_6_35b_a3b.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_6_35b_a3b.mdx @@ -26,7 +26,7 @@ version**. |-------------------------------|-----------------------------------------------------------------------------------------------| | Tensor Parallelism | `--tp-size 2` | | Chunked Prefill | auto based on device memory, or set explicit value;
disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 16384` | -| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 4 8 16 24 32 48 64 80 96 112 120` | +| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 4 8 16 24 32 48 64 80 96 112 120` | | Speculative Decoding | `--speculative-algorithm NEXTN \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | @@ -62,13 +62,13 @@ For multi-node deployment, download the weights to a shared directory accessible - [Eco-Tech/Qwen3.6-35B-A3B-w8a8](https://www.modelscope.cn/models/Eco-Tech/Qwen3.6-35B-A3B-w8a8) (W8A8 quantized version, 39.81GB) -The W8A8 variant (39.81GB) fits within a single 64GB die. For single-node deployment, `--tp-size 1` is sufficient on either A2 or A3. +The W8A8 variant (39.81GB) fits within a single 64GB die. For single-node deployment, `--tp-size 1` is sufficient on either A2 Series or A3 Series. This is the minimum recommended configuration. For optimized configurations, see [Best practices](#best-practices), which may require additional cards or nodes. -For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3), +For the hardware specifications (memory per die, dies per card, and the difference between A2 Series and A3 Series), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation @@ -84,7 +84,7 @@ Both **stable releases** and **daily builds** are available. The following comma For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). - + ```bash Command docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 @@ -119,7 +119,7 @@ quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 ``` - + ```bash Command docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 @@ -161,7 +161,7 @@ quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 Single-node deployment completes both prefill and decode within the same node (PD mixed mode), suitable for scenarios with limited hardware resources. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see -[Qwen3.6-35B-A3B Best Practice — PD Mixed On A3](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_6_35b_a3b#single-node-pd-mixed). +[Qwen3.6-35B-A3B Best Practice — PD Mixed On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_6_35b_a3b#single-node-pd-mixed). ## Functional verification diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_8_max.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_8_max.mdx index b90657af5ccd..4c398e08c539 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_8_max.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_8_max.mdx @@ -28,7 +28,7 @@ source. | Expert Parallelism | `--moe-a2a-backend deepep \`
`--deepep-mode auto` | | Quantization | `--quantization modelslim` | | Chunked Prefill | auto based on device memory, or set explicit value;
disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 8192` | -| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 16` | +| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 16` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | | DP LM Head | `--enable-dp-lm-head` | @@ -66,7 +66,7 @@ For multi-node deployment, download the weights to a shared directory accessible We recommend deploying the W4A8 variant for reduced resource usage and higher throughput. -The validated configuration in this tutorial deploys the W4A8 variant on 4 Atlas 800I A3 nodes (`--tp-size 64`, 64 dies +The validated configuration in this tutorial deploys the W4A8 variant on 4 Ascend A3 Series nodes (`--tp-size 64`, 64 dies in total). The BF16 checkpoint alone weighs approximately 4.8TB and requires additional nodes. @@ -86,7 +86,7 @@ The following command is based on the daily build tag, which contains the latest For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). - + ```bash Command docker pull quay.io/ascend/sglang:main-cann9.0.0-a3 @@ -133,7 +133,7 @@ quay.io/ascend/sglang:main-cann9.0.0-a3 ### Multi-node online deployment -Multi-node deployment distributes the model across multiple Atlas 800I A3 nodes using tensor parallelism while keeping +Multi-node deployment distributes the model across multiple Ascend A3 Series nodes using tensor parallelism while keeping prefill and decode on the same nodes (PD mixed mode), suitable for scenarios that need more device memory than a single node can provide. The validated configuration deploys the W4A8 quantized checkpoint on 4 nodes with `--tp-size 64` (16 dies per node), DP attention (`--dp-size 4`), and DeepEP in `auto` mode. @@ -198,7 +198,6 @@ do # overlap schedule export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 - export SGLANG_ENABLE_SPEC_V2=1 export SGLANG_RAGGED_VERIFY_MODE=static sglang serve \ @@ -215,7 +214,7 @@ do --enable-dp-attention --dp-size 4 --enable-dp-lm-head \ --mem-fraction-static 0.8 \ --chunked-prefill-size 8192 \ - --cuda-graph-bs 16 \ + --cuda-graph-bs-decode 16 \ --disable-radix-cache \ --max-running-requests 64 \ --host 0.0.0.0 \ diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_8b.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_8b.mdx index 3b0826358394..375ae88cf70f 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_8b.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_8b.mdx @@ -26,7 +26,7 @@ version. | Tensor Parallelism | `--tp-size 2` | | Quantization | `--quantization modelslim` | | Chunked Prefill | auto based on device memory, or set explicit value;
disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 8192` | -| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 1 2 4 6 9 10 15 16` | +| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 1 2 4 6 9 10 15 16` | | Speculative Decoding | `--speculative-algorithm EAGLE3 \`
`--speculative-draft-model-path /path/to/draft-model-weights \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4 \`
`--speculative-draft-model-quantization unquant` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | @@ -65,13 +65,13 @@ For multi-node deployment, download the weights to a shared directory accessible We recommend deploying the W8A8 variant for reduced resource usage and higher throughput. -It (11.27GB) fits within a single 64GB die, so `--tp-size 1` is sufficient on either A2 or A3. +It (11.27GB) fits within a single 64GB die, so `--tp-size 1` is sufficient on either A2 Series or A3 Series. This is the minimum recommended configuration. For optimized configurations, see [Best practices](#best-practices), which may require additional cards or nodes. -For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3), +For the hardware specifications (memory per die, dies per card, and the difference between A2 Series and A3 Series), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation @@ -87,7 +87,7 @@ Both **stable releases** and **daily builds** are available. The following comma For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). - + ```bash Command docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 @@ -122,7 +122,7 @@ quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 ``` - + ```bash Command docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 @@ -164,7 +164,7 @@ quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 Single-node deployment completes both prefill and decode within the same node (PD mixed mode), suitable for scenarios with limited hardware resources. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see -[Qwen3-8B Best Practice — PD Mixed On A3](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_8b#single-node-pd-mixed). +[Qwen3-8B Best Practice — PD Mixed On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_8b#single-node-pd-mixed). ## Functional verification diff --git a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_next_80b_a3b_instruct.mdx b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_next_80b_a3b_instruct.mdx index b1740ecaca09..be6c47b99fb6 100644 --- a/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_next_80b_a3b_instruct.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/model-deployment/tutorials/qwen3_next_80b_a3b_instruct.mdx @@ -27,7 +27,7 @@ v0.5.16 or a later version. | Data Parallelism | `--dp-size 2` | | Expert Parallelism | `--ep-size 4 \`
`--moe-a2a-backend deepep \`
`--deepep-mode auto` | | Quantization | `--quantization modelslim` | -| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs 2 4 8` | +| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs-decode` or `--cuda-graph-max-bs-decode`; e.g., `--cuda-graph-bs-decode 2 4 8` | | Speculative Decoding | `--speculative-algorithm NEXTN \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4 \`
`--speculative-draft-model-quantization unquant \`
`--speculative-draft-model-path /path/to/draft-model-weights` | | Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` | | DP LM Head | `--enable-dp-lm-head` | @@ -66,14 +66,14 @@ For multi-node deployment, download the weights to a shared directory accessible We recommend deploying the W8A8 variant as the main model for reduced resource usage and higher throughput. -It (84.90GB) can be deployed on 2 × 64GB (`--tp-size 2`), which corresponds to 2 cards on A2 or 1 card (2 dies) on A3. +It (84.90GB) can be deployed on 2 × 64GB (`--tp-size 2`), which corresponds to 2 cards on A2 Series or 1 card (2 dies) on A3 Series. The BF16 weights serve as the EAGLE3 draft model (set `--speculative-draft-model-path` to the BF16 weight path). This is the minimum recommended configuration. For optimized configurations, see [Best practices](#best-practices), which may require additional cards or nodes. -For the hardware specifications (memory per die, dies per card, and the difference between A2 and A3), +For the hardware specifications (memory per die, dies per card, and the difference between A2 Series and A3 Series), see [Ascend NPU Reference — Hardware](/docs/hardware-platforms/ascend-npus/reference/glossary#hardware). ## Installation @@ -89,7 +89,7 @@ Both **stable releases** and **daily builds** are available. The following comma For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/faq#8-docker-image-versions-stable-release-vs-daily-build). - + ```bash Command docker pull quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 @@ -124,7 +124,7 @@ quay.io/ascend/sglang:cann9.0.0-a3-v0.5.16 ``` - + ```bash Command docker pull quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 @@ -166,7 +166,7 @@ quay.io/ascend/sglang:cann9.0.0-910b-v0.5.16 Single-node deployment completes both prefill and decode within the same node (PD mixed mode), suitable for scenarios with limited hardware resources. This scenario is already covered in the best practice. For the complete, optimized deployment commands and benchmark data, see -[Qwen3-Next-80B-A3B-Instruct Best Practice — PD Mixed On A3](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_next_80b_a3b_instruct#single-node-pd-mixed). +[Qwen3-Next-80B-A3B-Instruct Best Practice — PD Mixed On A3 Series](/docs/hardware-platforms/ascend-npus/model-deployment/best-practices/qwen3_next_80b_a3b_instruct#single-node-pd-mixed). ## Functional verification diff --git a/docs/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning.mdx b/docs/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning.mdx index 2115219156a5..2b5c45fa05fa 100644 --- a/docs/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/optimization/parameter_tuning.mdx @@ -755,9 +755,9 @@ see [Quantization on Ascend](/docs/hardware-platforms/ascend-npus/optimization/q Breaks large prefill computations into smaller, fixed-size chunks for better scheduling interleaving with decode batches. This prevents long prefill requests from blocking decode latency. Use `-1` to disable chunked prefill on dedicated prefill nodes. Has partial compatibility with most features because chunk boundaries introduce scheduling complexity. -### NPU Graph (`--cuda-graph-bs`) +### NPU Graph (`--cuda-graph-bs-decode`) -Captures the compute graph on NPU and replays it to eliminate kernel launch overhead, analogous to CUDA Graph on NVIDIA GPUs. Internally uses `torch.npu.NPUGraph`. Most effective for decode with stable, predictable batch sizes (via `--cuda-graph-bs`). Has partial compatibility with most features because graph capture requires fixed control flow and tensor shapes. +Captures the compute graph on NPU and replays it to eliminate kernel launch overhead, analogous to CUDA Graph on NVIDIA GPUs. Internally uses `torch.npu.NPUGraph`. Most effective for decode with stable, predictable batch sizes (via `--cuda-graph-bs-decode`). Has partial compatibility with most features because graph capture requires fixed control flow and tensor shapes. `--enable-torch-compile` is incompatible with NPU Graph. When torch.compile is enabled, NPU Graph must be disabled via `--disable-cuda-graph`. @@ -767,6 +767,8 @@ Captures the compute graph on NPU and replays it to eliminate kernel launch over Reduces per-token latency by predicting multiple future tokens in a single forward pass, then verifying them against the model. Ascend supports `NEXTN` (for DeepSeek models, using the model's own hidden states) and `EAGLE3` (for Qwen MoE models, using a separate draft model). Controlled by `--speculative-num-steps`, `--speculative-eagle-topk`, and `--speculative-num-draft-tokens`. On Ascend, `SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` enables the optimized overlap scheduler for speculative decoding. +For non-greedy requests, Ascend target verification applies request-level `temperature`, `top_k`, and `top_p`. The default target-only mode supports linear and tree-shaped candidates. Classic rejection sampling is available with `--speculative-use-rejection-sampling` for linear chains configured with `--speculative-eagle-topk 1`. + ### PrefixCache (`--disable-radix-cache`) Reuses KV cache across requests that share common prompt prefixes (Radix Cache), reducing repeated prefill computation and lowering time-to-first-token. Enabled by default; disable with `--disable-radix-cache` when prefix reuse is not expected (e.g., dedicated prefill nodes in PD disaggregation, or random-input benchmarks). diff --git a/docs/docs/hardware-platforms/ascend-npus/optimization/profiling.mdx b/docs/docs/hardware-platforms/ascend-npus/optimization/profiling.mdx index 2373fbb12b9d..828d47b3c76f 100644 --- a/docs/docs/hardware-platforms/ascend-npus/optimization/profiling.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/optimization/profiling.mdx @@ -422,7 +422,7 @@ them via: ```text [2026-05-19 13:23:17] Stop profiling... [2026-05-19 13:23:17] [WARNING] [350443] profiler.py: Incorrect schedule: Stop profiler while current state is RECORD which may result in incomplete parsed data. -[rank0]:[W519 13:23:17.084812760 compiler_depend.ts:3136] Warning: The indexFromRank 0is not equal indexFromCurDevice 4 , which might be normal if the number of devices on your collective communication server is inconsistent.Otherwise, you need to check if the current device is correct when calling the interface.If it's incorrect, it might have introduced an error. (function operator()) +[rank0]:[W519 13:23:17.084812760 compiler_depend.ts:3136] Warning: The indexFromRank 0 is not equal indexFromCurDevice 4 , which might be normal if the number of devices on your collective communication server is inconsistent.Otherwise, you need to check if the current device is correct when calling the interface.If it's incorrect, it might have introduced an error. (function operator()) [2026-05-19 13:23:17] [INFO] [352725] profiler.py: Start parsing profiling data: /tmp/1779196995.6948605/localhost.localdomain_350443_20260519132315700_ascend_pt [2026-05-19 13:23:22] [INFO] [352734] profiler.py: CANN profiling data parsed in a total time of 0:00:04.022310 [2026-05-19 13:23:32] [INFO] [352725] profiler.py: All profiling data parsed in a total time of 0:00:14.305669 diff --git a/docs/docs/hardware-platforms/ascend-npus/optimization/quantization.mdx b/docs/docs/hardware-platforms/ascend-npus/optimization/quantization.mdx index 17048ebf9f0f..1ac1b7cd91ef 100644 --- a/docs/docs/hardware-platforms/ascend-npus/optimization/quantization.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/optimization/quantization.mdx @@ -13,9 +13,9 @@ SGLang supports **mix-bits** quantization (independently defines and loads each Quantization scheme Layer type - A2 Supported - A3 Supported - Ascend 950 Products Supported + A2 Series Supported + A3 Series Supported + 950PR/DT Series Supported Diffusion models @@ -133,9 +133,9 @@ SGLang supports **mix-bits** quantization (independently defines and loads each Quantization scheme Layer type - A2 Supported - A3 Supported - Ascend 950 Products Supported + A2 Series Supported + A3 Series Supported + 950PR/DT Series Supported @@ -169,9 +169,9 @@ GPTQ on Ascend support Quantization scheme Layer type - A2 Supported - A3 Supported - Ascend 950 Products Supported + A2 Series Supported + A3 Series Supported + 950PR/DT Series Supported @@ -212,9 +212,9 @@ GPTQ on Ascend support Quantization scheme Layer type - A2 Supported - A3 Supported - Ascend 950 Products Supported + A2 Series Supported + A3 Series Supported + 950PR/DT Series Supported @@ -255,9 +255,9 @@ Compressed-tensors (LLM Compressor) on Ascend support: Quantization scheme Layer type - A2 Supported - A3 Supported - Ascend 950 Products Supported + A2 Series Supported + A3 Series Supported + 950PR/DT Series Supported @@ -298,9 +298,9 @@ Compressed-tensors (LLM Compressor) on Ascend support: Quantization type Layer type - A2 Supported - A3 Supported - Ascend 950 Products Supported + A2 Series Supported + A3 Series Supported + 950PR/DT Series Supported @@ -351,7 +351,7 @@ python3 -m sglang.launch_server \ **MXFP8 for LLM dense models (e.g., Qwen3 / Qwen3.5):** -LLM dense W8A8 MXFP8 Linear support on Ascend was added in [PR #22352](https://github.com/sgl-project/sglang/pull/22352). Requires Ascend 950 Products or newer (`npu_dynamic_mx_quant` is not available on A2 / A3). +LLM dense W8A8 MXFP8 Linear support on Ascend was added in [PR #22352](https://github.com/sgl-project/sglang/pull/22352). Requires Ascend 950PR/DT Series or newer (`npu_dynamic_mx_quant` is not available on A2/A3 Series). - Online MXFP8 quantization (BF16/FP16 weights → MXFP8 at load time): @@ -381,7 +381,7 @@ python3 -m sglang.launch_server \ **MXFP8 for LLM MoE models (e.g. Qwen3-30B-A3B / Qwen3.5 MoE):** -LLM MoE W8A8 MXFP8 (FusedMoE) support builds on the dense MXFP8 path. Requires Ascend A5 series or newer — the fused MoE MX kernels (`npu_grouped_matmul_swiglu_quant_v2`, `npu_dynamic_mx_quant`) are A5-only. +LLM MoE W8A8 MXFP8 (FusedMoE) support builds on the dense MXFP8 path. Requires Ascend 950PR/DT Series or newer — the fused MoE MX kernels (`npu_grouped_matmul_swiglu_quant_v2`, `npu_dynamic_mx_quant`) are only available on the 950PR/DT Series. - Online MXFP8 quantization (BF16/FP16 expert weights → MXFP8 at load time): @@ -410,11 +410,11 @@ python3 -m sglang.launch_server \ > - Offline path: `ModelSlimMXFP8MoEScheme` (one instance per weight group) loads `float8_e4m3fn` expert weights + uint8 (e8m0, exponent + 127) block scales. The scale is reshaped `[E, N, K/32] → [E, N, K/64, 2]` (contiguous pairing, matching `npu_dynamic_mx_quant`) then transposed. > - Forward: `AscendTPDispatcher` runs `npu_moe_init_routing_v2(quant_mode=3)`, which fuses the per-token MX activation quant into routing (e4m3 payload + e8m0 block scale, reshaped to the pair-split layout). `AscendRunnerCore` then runs gmm1 `npu_grouped_matmul_swiglu_quant_v2` (cumulative `group_list`; fuses gate/up + swiglu + requant, so no separate activation step) → gmm2 `npu_grouped_matmul` (count `group_list`). The UE8M0 (`float8_e8m0fnu`) scale dtypes are passed explicitly; the e4m3 `x`/`weight` dtypes are left implicit. > - **Router gate**: msmodelslim may also quantize `mlp.gate` (`W8A8_MXFP8`). The gate is a `ReplicatedLinear`, so its quantization must be **description-driven**: for the offline `modelslim` path the gate is passed the quant config and dequantized correctly; the online path keeps it in BF16. Loading a quantized gate as BF16 without its block scale scrambles routing and produces garbage output. -> - Where the activation quant happens depends on the dispatcher. On `ascend_tp` it is fused into routing as described above. DeepEP has no MXFP8 dispatch dtype, so it keeps dispatching BF16 and gmm1 quantizes the hidden states itself via `npu_dynamic_mx_quant` before the fused kernel — the two paths reach the same gmm1 input. Only the `ascend_tp` path has been validated end-to-end on Ascend A5. +> - Where the activation quant happens depends on the dispatcher. On `ascend_tp` it is fused into routing as described above. DeepEP has no MXFP8 dispatch dtype, so it keeps dispatching BF16 and gmm1 quantizes the hidden states itself via `npu_dynamic_mx_quant` before the fused kernel — the two paths reach the same gmm1 input. Only the `ascend_tp` path has been validated end-to-end on the Ascend 950PR/DT Series. **MXFP4 W4A8 for LLM dense models (e.g., Qwen3 / Qwen3.5):** -LLM dense W4A8 (MXFP4 4-bit weights + MXFP8 8-bit activations) Linear support was added in [PR #23650](https://github.com/sgl-project/sglang/pull/23650). Requires Ascend 950 Products or newer. +LLM dense W4A8 (MXFP4 4-bit weights + MXFP8 8-bit activations) Linear support was added in [PR #23650](https://github.com/sgl-project/sglang/pull/23650). Requires Ascend 950PR/DT Series or newer. - Online W4A8 quantization (BF16/FP16 weights → MXFP4 at load time): @@ -436,7 +436,7 @@ python3 -m sglang.launch_server \ **ModelSlim W4A8 MXFP4 for LLM MoE models:** -SGLang auto-detects offline ModelSlim `W4A8_MXFP` MoE checkpoints from `quant_model_description.json`; do not pass `--quantization`. This path requires Ascend A5 or newer. +SGLang auto-detects offline ModelSlim `W4A8_MXFP` MoE checkpoints from `quant_model_description.json`; do not pass `--quantization`. This path requires Ascend 950PR/DT Series or newer. ```bash Command MODEL_PATH=/path/to/w4a8-mxfp4-moe-model @@ -453,7 +453,7 @@ python3 -m sglang.launch_server \ **MXFP4 W4A4 for LLM dense models (e.g. Qwen3 / Qwen3.5):** -LLM dense W4A4 (MXFP4 4-bit weights + 4-bit activations) Linear support was added in [PR #23795](https://github.com/sgl-project/sglang/pull/23795). Requires Ascend A5 series (Ascend 950) or newer — the dual-level online path uses the `DualLevelQuantBatchMatmul` op, which A2/A3 lack. On the Ascend NPU backend `--quantization mxfp4` selects this W4A4 path (on GPU the same flag selects the upstream OCP MXFP4 MoE config instead). +LLM dense W4A4 (MXFP4 4-bit weights + 4-bit activations) Linear support was added in [PR #23795](https://github.com/sgl-project/sglang/pull/23795). Requires Ascend 950PR/DT Series or newer — the dual-level online path uses the `DualLevelQuantBatchMatmul` op, which A2/A3 Series lack. On the Ascend NPU backend `--quantization mxfp4` selects this W4A4 path (on GPU the same flag selects the upstream OCP MXFP4 MoE config instead). - Online W4A4 quantization (BF16/FP16 weights → dual-level MXFP4 at load time): @@ -472,11 +472,11 @@ python3 -m sglang.launch_server \ > - **Online** (`NPUDualLevelMXFP4LinearMethod`) uses **dual-level** MXFP4: both weights and activations are quantized with a fine FP8 (E4M3) L0 block scale plus a coarser L1 scale via `npu_dynamic_dual_level_mx_quant`, and the matmul runs via `npu_dual_level_quant_matmul` (weight in FRACTAL_NZ). Dual-level captures per-block dynamic range far better than a single UE8M0 (power-of-2) scale, which is what made an earlier single-level RTN online path degenerate (greedy decoding could loop without emitting EOS). > - **Offline** (`ModelSlimMXFP4Scheme` → `NPUSingleLevelMXFP4OfflineLinearMethod`) is **single-level**: msmodelslim's `W4A4_MXFP4` checkpoint ships single-level UE8M0 block scales (block_size = 32), so the matmul runs via `npu_quant_matmul(..., x1_dtype=x2_dtype=torch_npu.float4_e2m1fn_x2, group_sizes=[1, 1, 32])`. The online and offline paths therefore use different matmul kernels — they no longer share the matmul path. > - As with W4A8, the packed-FP4 dtype passed to the NPU ops (`dst_type` / `x2_dtype`) must be resolved from `torch_npu.float4_e2m1fn_x2` (an int enum), not the `torch.float4_e2m1fn_x2` dtype object, which recent op-plugin builds reject. -> - Validated end-to-end on Ascend A5 hardware. +> - Validated end-to-end on Ascend 950PR/DT Series hardware. **ModelSlim W4A4 MXFP4 for LLM MoE models:** -SGLang auto-detects offline ModelSlim `W4A4_MXFP4` MoE checkpoints from `quant_model_description.json`; do not pass `--quantization`. This path requires Ascend 950 products or newer. +SGLang auto-detects offline ModelSlim `W4A4_MXFP4` MoE checkpoints from `quant_model_description.json`; do not pass `--quantization`. This path requires Ascend 950PR/DT Series or newer. ```bash Command MODEL_PATH=/path/to/w4a4-mxfp4-moe-model @@ -493,9 +493,9 @@ python3 -m sglang.launch_server \ ## Diffusion Model Quantization on Ascend NPU -SGLang-Diffusion supports MXFP8 online and offline quantization for diffusion models (such as Wan2.2) on Ascend NPUs. MXFP8 requires Ascend 950 Products; the ModelSlim W8A8/W4A4 schemes work on A2/A3. +SGLang-Diffusion supports MXFP8 online and offline quantization for diffusion models (such as Wan2.2) on Ascend NPUs. MXFP8 requires Ascend 950PR/DT Series; the ModelSlim W8A8/W4A4 schemes work on A2/A3 Series. -**Requirements for MXFP8:** CANN ≥ 8.0.RC3, Ascend 950 Products +**Requirements for MXFP8:** CANN ≥ 8.0.RC3, Ascend 950PR/DT Series @@ -504,8 +504,8 @@ SGLang-Diffusion supports MXFP8 online and offline quantization for diffusion mo - - + + diff --git a/docs/docs/hardware-platforms/ascend-npus/optimization/ring_sp_performance.mdx b/docs/docs/hardware-platforms/ascend-npus/optimization/ring_sp_performance.mdx index 4e1337bad3f3..3a693c644b86 100644 --- a/docs/docs/hardware-platforms/ascend-npus/optimization/ring_sp_performance.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/optimization/ring_sp_performance.mdx @@ -1,7 +1,7 @@ --- title: "Ring-SP Performance (Wan2.1-T2V-1.3B)" metatags: - description: "This page reports Ring-SP performance on Ascend NPU with torchnpu==2.10.0." + description: "This page reports Ring-SP performance on Ascend NPU with torch_npu==2.10.0." --- This page reports Ring-SP performance on Ascend NPU with `torch_npu==2.10.0`. diff --git a/docs/docs/hardware-platforms/ascend-npus/reference/glossary.mdx b/docs/docs/hardware-platforms/ascend-npus/reference/glossary.mdx index fc72acc5869c..c2e9cd746f8e 100644 --- a/docs/docs/hardware-platforms/ascend-npus/reference/glossary.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/reference/glossary.mdx @@ -13,23 +13,23 @@ you will encounter throughout the Ascend NPU documentation. Refer back here when SGLang supports the following Ascend inference hardware: -| Hardware | Chip | Devices | Dies per card | Memory configuration | -| ------------------ | ----------- | ------- | ------------- | ------------------------------------ | -| Atlas 800I A2 (A2) | Ascend 910B | 8 | 1 | 8(cards) × 1(die/card) × 64(GB/die) | -| Atlas 800I A3 (A3) | Ascend 910C | 16 | 2 | 8(cards) × 2(dies/card) × 64(GB/die) | +| Hardware | Devices | Dies per card | Memory configuration | +| ------------------------- | ------- | ------------- | ------------------------------------ | +| Ascend A2 Series Products | 8 | 1 | 8(cards) × 1(die/card) × 64(GB/die) | +| Ascend A3 Series Products | 16 | 2 | 8(cards) × 2(dies/card) × 64(GB/die) | -Throughout these docs, **A2** and **A3** are used as shorthand for the hardware above. -Docker image tags use `910b` for A2 and `a3` for A3. +Throughout these docs, **A2 Series** and **A3 Series** are used as shorthand for the hardware above. +Docker image tags use `910b` for the A2 Series and `a3` for the A3 Series. For example, `cann9.0.0-910b-v0.5.16` and `cann9.0.0-a3-v0.5.16`. -On A3, each card has 2 dies, giving 16 devices vs 8 on A2. -Benchmark pages use "Cards" to refer to physical cards, so `Cards: 4` on A3 means `--tp-size 8`. +On the A3 Series, each card has 2 dies, giving 16 devices vs 8 on the A2 Series. +Benchmark pages use "Cards" to refer to physical cards, so `Cards: 4` on the A3 Series means `--tp-size 8`. -From a deployment perspective, the two key differences between A2 and A3 are: +From a deployment perspective, the two key differences between the A2 Series and the A3 Series are: 1. dies per card — which drives both total memory and `--tp-size` configuration -2. PD disaggregation — A2 requires setting `export ASCEND_MF_TRANSFER_PROTOCOL="device_rdma"`, while A3 uses the default protocol. +2. PD disaggregation — the A2 Series requires setting `export ASCEND_MF_TRANSFER_PROTOCOL="device_rdma"`, while the A3 Series uses the default protocol. ### NPU @@ -37,10 +37,10 @@ From a deployment perspective, the two key differences between A2 and A3 are: **NPU** stands for Neural Processing Unit. Each NPU device is a single `davinci` core. The terms "NPU" and "davinci" are used interchangeably in commands and error logs. -On A2, devices are numbered `/dev/davinci0` through `/dev/davinci7` (8 devices). -On A3, devices are numbered `/dev/davinci0` through `/dev/davinci15` (16 devices). +On the A2 Series, devices are numbered `/dev/davinci0` through `/dev/davinci7` (8 devices). +On the A3 Series, devices are numbered `/dev/davinci0` through `/dev/davinci15` (16 devices). -On either an A2 or A3 server, run `npu-smi info` to view NPU information such as device +On either an A2 Series or an A3 Series server, run `npu-smi info` to view NPU information such as device health, memory usage, and chip status. If the command is not found or reports no devices, the driver is likely not installed. Follow the [Ascend driver installation guide](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/900/softwareinst/instg/instg_0005.html?OS=openEuler&InstallType=local) @@ -63,7 +63,7 @@ to install it. | **W4A8** | 4-bit weights, 8-bit activations | | **W4A16** | 4-bit weights, 16-bit activations | | **BF16** | Brain Floating Point 16 — 16-bit format optimized for ML workloads | -| **FP8** | 8-bit Floating Point — not supported on A2/A3 | +| **FP8** | 8-bit Floating Point — not supported on the A2/A3 Series | | **INT8** | 8-bit Integer quantization | To apply quantization, use `--quantization modelslim` for W8A8 INT8, or load a pre-quantized diff --git a/docs/docs/hardware-platforms/ascend-npus/reference/support_features.mdx b/docs/docs/hardware-platforms/ascend-npus/reference/support_features.mdx index c200b1fd2144..d66366ba923c 100644 --- a/docs/docs/hardware-platforms/ascend-npus/reference/support_features.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/reference/support_features.mdx @@ -32,97 +32,97 @@ click [Server Arguments](../../../advanced_features/server_arguments). - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - +
quant_type in JSON Scheme class ModeA2/A3 SupportedAscend 950 Products SupportedA2/A3 Series Supported950PR/DT Series Supported Trigger
`--model-path`
`--model`		 `None` Type: strA2, A3A2/A3 Series
`--tokenizer-path` `None` Type: strA2, A3A2/A3 Series
`--tokenizer-mode` `auto` `auto`, `slow`A2, A3A2/A3 Series
`--tokenizer-backend` `huggingface` `huggingface`, `fastokens`A2, A3A2/A3 Series
`--tokenizer-worker-num` `1` Type: intA2, A3A2/A3 Series
`--detokenizer-worker-num` `1` Type: intA2, A3A2/A3 Series
`--skip-tokenizer-init` `False` bool flag (set to enable)A2, A3A2/A3 Series
`--load-format` `auto` `auto`, `safetensors`, `gguf`A2, A3A2/A3 Series
`--model-loader-extra-config` `{}` Type: strA2, A3A2/A3 Series
`--trust-remote-code` `False` bool flag (set to enable)A2, A3A2/A3 Series
`--context-length` `None` Type: intA2, A3A2/A3 Series
`--is-embedding` `False` bool flag (set to enable)A2, A3A2/A3 Series
`--enable-multimodal` `None` bool flag (set to enable)A2, A3A2/A3 Series
`--revision` `None` Type: strA2, A3A2/A3 Series
`--model-impl` `auto` `auto`, `sglang`,
`transformers`		A2, A3A2/A3 Series
`--model-config-parser` `auto` Type: strA2, A3A2/A3 Series
@@ -150,37 +150,37 @@ click [Server Arguments](../../../advanced_features/server_arguments). `--host` `127.0.0.1` Type: str - A2, A3 + A2/A3 Series `--port` `30000` Type: int - A2, A3 + A2/A3 Series `--skip-server-warmup` `False` bool flag (set to enable) - A2, A3 + A2/A3 Series `--warmups` `None` Type: str - A2, A3 + A2/A3 Series `--nccl-port` `None` Type: int - A2, A3 + A2/A3 Series `--fastapi-root-path` `None` Type: str - A2, A3 + A2/A3 Series `--grpc-mode` @@ -214,31 +214,31 @@ click [Server Arguments](../../../advanced_features/server_arguments). `--ssl-keyfile` `None` Type: str - A2, A3 + A2/A3 Series `--ssl-certfile` `None` Type: str - A2, A3 + A2/A3 Series `--ssl-keyfile-password` `None` Type: str - A2, A3 + A2/A3 Series `--enable-ssl-refresh` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series `--enable-http2` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series @@ -267,13 +267,13 @@ click [Server Arguments](../../../advanced_features/server_arguments). `--dtype` `auto` `auto`,
`float16`,
`bfloat16` - A2, A3 + A2/A3 Series `--quantization` `None` `modelslim` - A2, A3 + A2/A3 Series `--quantization-param-path` @@ -285,13 +285,13 @@ click [Server Arguments](../../../advanced_features/server_arguments). `--kv-cache-dtype` `auto` `auto` - A2, A3 + A2/A3 Series `--enable-fp32-lm-head` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series `--modelopt-quant` @@ -355,169 +355,163 @@ click [Server Arguments](../../../advanced_features/server_arguments). `--mem-fraction-static` `None` Type: float - A2, A3 + A2/A3 Series `--max-running-requests` `None` Type: int - A2, A3 + A2/A3 Series `--prefill-max-requests` `None` Type: int - A2, A3 + A2/A3 Series `--max-queued-requests` `None` Type: int - A2, A3 + A2/A3 Series `--max-total-tokens` `None` Type: int - A2, A3 + A2/A3 Series `--chunked-prefill-size` `None` Type: int - A2, A3 + A2/A3 Series `--max-prefill-tokens` `16384` Type: int - A2, A3 + A2/A3 Series `--schedule-policy` `fcfs` `lpm`,
`fcfs`,
`random` - A2, A3 + A2/A3 Series `--enable-priority-scheduling` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series `--disable-priority-preemption` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series `--default-priority-value` `None` Type: int - A2, A3 + A2/A3 Series `--schedule-low-priority-values-first` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series `--priority-scheduling-preemption-threshold` `10` Type: int - A2, A3 + A2/A3 Series `--retraction-policy` `length` `length`,
`priority` - A2, A3 + A2/A3 Series `--schedule-conservativeness` `1.0` Type: float - A2, A3 + A2/A3 Series `--page-size` `128` Type: int - A2, A3 + A2/A3 Series `--swa-full-tokens-ratio` `0.8` Type: float - A2, A3 - - - `--disable-hybrid-swa-memory` - `False` - bool flag
(set to enable) - A2, A3 + A2/A3 Series `--radix-eviction-policy` `lru` `lru`,
`lfu` - A2, A3 + A2/A3 Series `--enable-prefill-delayer` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series `--prefill-delayer-max-delay-passes` `30` Type: int - A2, A3 + A2/A3 Series `--prefill-delayer-token-usage-low-watermark` `None` Type: float - A2, A3 + A2/A3 Series `--prefill-delayer-forward-passes-buckets` `None` List[float] - A2, A3 + A2/A3 Series `--prefill-delayer-wait-seconds-buckets` `None` List[float] - A2, A3 + A2/A3 Series `--prefill-delayer-queue-min-ratio` `None` Type: float - A2, A3 + A2/A3 Series `--prefill-delayer-max-delay-ms` `None` Type: float - A2, A3 + A2/A3 Series `--abort-on-priority-when-disabled` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series `--enable-dynamic-chunking` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series @@ -545,151 +539,151 @@ click [Server Arguments](../../../advanced_features/server_arguments). `--device` `None` Type: str - A2, A3 + A2/A3 Series `--tensor-parallel-size`
`--tp-size` `1` Type: int - A2, A3 + A2/A3 Series `--pipeline-parallel-size`
`--pp-size` `1` Type: int; Currently `2` not supported; Cannot be used together with TP - A2, A3 + A2/A3 Series `--attention-context-parallel-size`
`--attn-cp-size` `1` Type: int; must be equal to --tp-size - A2, A3 + A2/A3 Series `--moe-data-parallel-size`
`--moe-dp-size` `1` Type: int - A2, A3 + A2/A3 Series `--pp-max-micro-batch-size` `None` Type: int - A2, A3 + A2/A3 Series `--pp-async-batch-depth` `0` Type: int - A2, A3 + A2/A3 Series `--stream-interval` `1` Type: int - A2, A3 + A2/A3 Series `--batch-notify-size` `16` Type: int - A2, A3 + A2/A3 Series `--incremental-streaming-output` `False` bool flag (set to enable) - A2, A3 + A2/A3 Series `--stream-response-default-include-usage` `False` bool flag (set to enable) - A2, A3 + A2/A3 Series `--enable-streaming-session` `False` bool flag (set to enable) - A2, A3 + A2/A3 Series `--enable-session-radix-cache` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series `--random-seed` `None` Type: int - A2, A3 + A2/A3 Series `--constrained-json-whitespace-pattern` `None` Type: str - A2, A3 + A2/A3 Series `--constrained-json-disable-any-whitespace` `False` bool flag (set to enable) - A2, A3 + A2/A3 Series `--watchdog-timeout` `300` Type: float - A2, A3 + A2/A3 Series `--soft-watchdog-timeout` `300` Type: float - A2, A3 + A2/A3 Series `--dist-timeout` `None` Type: int - A2, A3 + A2/A3 Series `--download-dir` `None` Type: str - A2, A3 + A2/A3 Series `--model-checksum` `None` Type: str - A2, A3 + A2/A3 Series `--base-gpu-id` `0` Type: int - A2, A3 + A2/A3 Series `--gpu-id-step` `1` Type: int - A2, A3 + A2/A3 Series `--sleep-on-idle` `False` bool flag (set to enable) - A2, A3 + A2/A3 Series `--load-snapshot-publish-interval` `15` Type: int - A2, A3 + A2/A3 Series `--use-ray` @@ -701,7 +695,7 @@ click [Server Arguments](../../../advanced_features/server_arguments). `--custom-sigquit-handler` `None` Only for engine - A2, A3 + A2/A3 Series @@ -730,127 +724,121 @@ click [Server Arguments](../../../advanced_features/server_arguments). `--log-level` `info` Type: str - A2, A3 + A2/A3 Series `--log-level-http` `None` Type: str - A2, A3 + A2/A3 Series `--log-requests` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series `--log-requests-level` `2` `0`, `1`, `2`, `3` - A2, A3 + A2/A3 Series `--log-requests-format` text `text`, `json` - A2, A3 + A2/A3 Series `--crash-dump-folder` `None` Type: str - A2, A3 + A2/A3 Series `--enable-metrics` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series `--enable-mfu-metrics` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series `--enable-metrics-for-all-schedulers` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series `--tokenizer-metrics-custom-labels-header` `x-custom-labels` Type: str - A2, A3 + A2/A3 Series `--tokenizer-metrics-allowed-custom-labels` `None` List[str] - A2, A3 + A2/A3 Series `--extra-metric-labels` `None` Type: JSON/Dict - A2, A3 + A2/A3 Series `--bucket-time-to-first-token` `None` List[float] - A2, A3 + A2/A3 Series `--bucket-inter-token-latency` `None` List[float] - A2, A3 + A2/A3 Series `--bucket-e2e-request-latency` `None` List[float] - A2, A3 - - - `--collect-tokens-histogram` - `False` - bool flag
(set to enable) - A2, A3 + A2/A3 Series `--prompt-tokens-buckets` `None` List[str] - A2, A3 + A2/A3 Series `--generation-tokens-buckets` `None` List[str] - A2, A3 + A2/A3 Series `--gc-warning-threshold-secs` `0.0` Type: float - A2, A3 + A2/A3 Series `--decode-log-interval` `40` Type: int - A2, A3 + A2/A3 Series `--enable-request-time-stats-logging` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series `--kv-events-config` @@ -862,49 +850,49 @@ click [Server Arguments](../../../advanced_features/server_arguments). `--enable-forward-pass-metrics` `False` bool flag (set to enable) - A2, A3 + A2/A3 Series `--forward-pass-metrics-worker-id` `` Type: str - A2, A3 + A2/A3 Series `--forward-pass-metrics-ipc-name` `None` Type: str - A2, A3 + A2/A3 Series `--enable-trace` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series `--trace-modules` `request` Type: str - A2, A3 + A2/A3 Series `--otlp-traces-endpoint` `localhost:4317` Type: str - A2, A3 + A2/A3 Series `--log-requests-target` `None` Type: str - A2, A3 + A2/A3 Series `--uvicorn-access-log-exclude-prefixes` `[]` List[str] - A2, A3 + A2/A3 Series @@ -932,13 +920,13 @@ click [Server Arguments](../../../advanced_features/server_arguments). `--export-metrics-to-file` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series `--export-metrics-to-file-dir` `None` Type: str - A2, A3 + A2/A3 Series @@ -966,43 +954,43 @@ click [Server Arguments](../../../advanced_features/server_arguments). `--api-key` `None` Type: str - A2, A3 + A2/A3 Series `--admin-api-key` `None` Type: str - A2, A3 + A2/A3 Series `--served-model-name` `None` Type: str - A2, A3 + A2/A3 Series `--weight-version` `default` Type: str - A2, A3 + A2/A3 Series `--chat-template` `None` Type: str - A2, A3 + A2/A3 Series `--hf-chat-template-name` `None` Type: str - A2, A3 + A2/A3 Series `--completion-template` `None` Type: str - A2, A3 + A2/A3 Series `--file-storage-path` @@ -1014,55 +1002,55 @@ click [Server Arguments](../../../advanced_features/server_arguments). `--enable-cache-report` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series `--reasoning-parser` `None` `deepseek-r1`
`deepseek-v3`
`glm45`
`gpt-oss`
`kimi`
`qwen3`
`qwen3-thinking`
`step3` - A2, A3 + A2/A3 Series `--default-chat-template-kwargs` `None` Type: JSON / Dict - A2, A3 + A2/A3 Series `--strip-thinking-cache` `False` bool flag (set to enable) - A2, A3 + A2/A3 Series `--enable-strict-thinking` `False` bool flag (set to enable) - A2, A3 + A2/A3 Series `--tool-call-parser` `None` `llama3`
`pythonic`
`qwen`
`qwen3_coder` - A2, A3 + A2/A3 Series `--sampling-defaults` `model` `openai`, `model` - A2, A3 + A2/A3 Series `--asr-max-buffer-seconds` `60` Type: int - A2, A3 + A2/A3 Series `--asr-max-concurrent-sessions` `32` Type: int - A2, A3 + A2/A3 Series @@ -1090,13 +1078,13 @@ click [Server Arguments](../../../advanced_features/server_arguments). `--data-parallel-size`
`--dp-size` `1` Type: int - A2, A3 + A2/A3 Series `--load-balance-method` `auto` `auto`,
`round_robin`,
`follow_bootstrap_room`,
`total_requests`,
`total_tokens` - A2, A3 + A2/A3 Series @@ -1124,19 +1112,19 @@ click [Server Arguments](../../../advanced_features/server_arguments). `--dist-init-addr`
`--nccl-init-addr` `None` Type: str - A2, A3 + A2/A3 Series `--nnodes` `1` Type: int - A2, A3 + A2/A3 Series `--node-rank` `0` Type: int - A2, A3 + A2/A3 Series @@ -1164,13 +1152,13 @@ click [Server Arguments](../../../advanced_features/server_arguments). `--json-model-override-args` `{}` Type: str - A2, A3 + A2/A3 Series `--preferred-sampling-params` `None` Type: str - A2, A3 + A2/A3 Series @@ -1199,61 +1187,61 @@ click [Server Arguments](../../../advanced_features/server_arguments). `--enable-lora` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series `--enable-lora-overlap-loading` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series `--max-lora-rank` `None` Type: int - A2, A3 + A2/A3 Series `--lora-target-modules` `None` `all` - A2, A3 + A2/A3 Series `--lora-paths` `None` Type: List[str] /
JSON objects - A2, A3 + A2/A3 Series `--max-loras-per-batch` `8` Type: int - A2, A3 + A2/A3 Series `--max-loaded-loras` `None` Type: int - A2, A3 + A2/A3 Series `--lora-eviction-policy` `lru` `lru`,
`fifo` - A2, A3 + A2/A3 Series `--lora-backend` `csgmv` `triton`,
`csgmv`,
`ascend`,
`torch_native` - A2, A3 + A2/A3 Series `--experts-shared-outer-loras` `None` Type: bool - A2, A3 + A2/A3 Series `--lora-use-virtual-experts` @@ -1265,13 +1253,13 @@ click [Server Arguments](../../../advanced_features/server_arguments). `--lora-strict-loading` `False` Type: bool - A2, A3 + Special for GPU `--lora-drain-wait-threshold` `0.0` Type: float - A2, A3 + A2/A3 Series `--max-lora-chunk-size` @@ -1306,43 +1294,43 @@ click [Server Arguments](../../../advanced_features/server_arguments). `--attention-backend` `None` `ascend` - A2, A3 + A2/A3 Series `--prefill-attention-backend` `None` `ascend` - A2, A3 + A2/A3 Series `--decode-attention-backend` `None` `ascend` - A2, A3 + A2/A3 Series `--sampling-backend` `None` `pytorch`,
`ascend` - A2, A3 + A2/A3 Series `--grammar-backend` `None` `xgrammar`,
`outlines`,
`llguidance` - A2, A3 + A2/A3 Series `--radix-cache-backend` `None` Type: str - A2, A3 + A2/A3 Series `--mm-attention-backend` `None` `ascend_attn` - A2, A3 + A2/A3 Series `--dsa-prefill-backend` @@ -1373,6 +1361,16 @@ click [Server Arguments](../../../advanced_features/server_arguments). ## Speculative decoding +Ascend NPU supports non-greedy target verification for the supported EAGLE-based speculative algorithms. Request-level +`temperature`, `top_k`, and `top_p` are applied when constructing the target probability distribution. Request-level +`top_k` controls token sampling and is separate from `--speculative-eagle-topk`, which controls the draft tree branching +factor. + +The default target-only mode supports both linear and tree-shaped draft candidates. Classic rejection sampling is +available for EAGLE3 and NEXTN with `--speculative-use-rejection-sampling` and requires +`--speculative-eagle-topk 1`. Rejection sampling uses the draft proposal distribution and cannot be combined with +non-default speculative acceptance thresholds or deterministic inference. + @@ -1395,103 +1393,109 @@ click [Server Arguments](../../../advanced_features/server_arguments). - + - + - + - + - + - + - + - + - + - + + + + + + + - + - + - + - + - + - + - + - +
`--speculative-algorithm` `None` `EAGLE`,
`EAGLE3`,
`NEXTN`		A2, A3A2/A3 Series
`--speculative-draft-model-path`
`--speculative-draft-model`		 `None` Type: strA2, A3A2/A3 Series
`--speculative-draft-model-revision` `None` Type: str,
`branch name`,
`tag name`,
`commit id`		A2, A3A2/A3 Series
`--speculative-draft-load-format` `auto` `auto`,
`dummy`		A2, A3A2/A3 Series
`--speculative-num-steps` `None` Type: intA2, A3A2/A3 Series
`--speculative-eagle-topk` `None` `1` (the only supported value on Ascend NPU)A2, A3A2/A3 Series
`--speculative-num-draft-tokens` `None` Type: intA2, A3A2/A3 Series
`--speculative-dflash-block-size` `None` Type: intA2, A3A2/A3 Series
`--speculative-accept-threshold-single` `1.0` Type: floatSpecial for GPUA2/A3 Series
`--speculative-accept-threshold-acc` `1.0` Type: floatSpecial for GPUA2/A3 Series
`--speculative-use-rejection-sampling``False`bool flag
(requires `--speculative-eagle-topk 1`)		A2/A3 Series
`--speculative-token-map` `None` Type: str; requires `--tp-size=1`A2, A3A2/A3 Series
`--speculative-attention-mode` `prefill` `prefill`,
`decode`		A2, A3A2/A3 Series
`--speculative-moe-runner-backend` `None` `auto`A2, A3A2/A3 Series
`--speculative-moe-a2a-backend` `None` `ascend_fuseep` (the only supported value on Ascend NPU)A2, A3A2/A3 Series
`--speculative-draft-attention-backend` `None` `ascend`A2, A3A2/A3 Series
`--speculative-dflash-draft-window-size`
`--speculative-draft-window-size`		`--speculative-draft-window-size` `None` Type: intA2, A3A2/A3 Series
`--speculative-draft-model-quantization` `None` `unquant` (the only supported value for speculative decoding on Ascend NPU)A2, A3A2/A3 Series
@@ -1532,67 +1536,67 @@ click [Server Arguments](../../../advanced_features/server_arguments). `--speculative-ngram-min-bfs-breadth` `1` Type: int - A2, A3 + Experimental `--speculative-ngram-max-bfs-breadth` `10` Type: int - A2, A3 + Experimental `--speculative-ngram-match-type` `BFS` `BFS`,
`PROB` - A2, A3. `BFS` uses recency-based expansion; `PROB` uses frequency-based expansion. + Experimental. `BFS` uses recency-based expansion; `PROB` uses frequency-based expansion. `--speculative-ngram-max-trie-depth` `18` Type: int - A2, A3 + Experimental `--speculative-ngram-capacity` `10000000` Type: int - A2, A3 + Experimental `--speculative-ngram-external-corpus-path` `None` Type: str - A2, A3 + Experimental `--speculative-ngram-external-sam-budget` `0` Type: int - A2, A3 + Experimental `--speculative-ngram-external-corpus-max-tokens` `10000000` Type: int - A2, A3 + Experimental `--speculative-adaptive` `False` bool flag (set to enable) - A2, A3 + A2/A3 Series `--speculative-adaptive-config` `None` Type: str - A2, A3 + A2/A3 Series `--speculative-skip-dp-mlp-sync` `False` bool flag (set to enable) - A2, A3 + A2/A3 Series @@ -1621,24 +1625,24 @@ click [Server Arguments](../../../advanced_features/server_arguments). `--expert-parallel-size`
`--ep-size`
`--ep` `1` Type: int - A2, A3 + A2/A3 Series `--moe-a2a-backend` `none` `none`,
`deepep`,
`ascend_fuseep`(It is incompatible with eplb) - A2, A3 + A2/A3 Series `--moe-runner-backend` `auto` `auto`, `triton` - A2, A3 + A2/A3 Series `--flashinfer-mxfp4-moe-precision` `default` - `default`,
`bf16` + `default`,
`bf16`,
`fp8` Special for GPU @@ -1651,20 +1655,20 @@ click [Server Arguments](../../../advanced_features/server_arguments). `--deepep-mode` `auto` `normal`,
`low_latency`,
`auto` - A2, A3 + A2/A3 Series `--fuseep-mode` `2` `1`,
`2` - A2, A3 + A2/A3 Series `--deepep-dispatcher-output-dtype` `auto` `auto`,
`bf16`,
`int8`
(When enabling DeepEP for a quantized model, set the dispatcher output dtype according to your model’s quantization. If the value is int8, you must also set the environment variable:DEEP_NORMAL_MODE_USE_INT8_QUANT=1) - A2, A3 + A2/A3 Series `--deepep-config` @@ -1676,73 +1680,73 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD `--ep-num-redundant-experts` `0` Type: int - A2, A3 + A2/A3 Series `--ep-dispatch-algorithm` `None` `static`,
`dynamic`,
`fake` - A2, A3 + A2/A3 Series `--init-expert-location` `trivial` `trivial`,
``,
``,
`` - A2, A3 + A2/A3 Series `--enable-eplb` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series `--eplb-algorithm` `deepseek` `auto`,
`deepseek` - A2, A3 + A2/A3 Series `--eplb-rebalance-num-iterations` `1000` Type: int - A2, A3 + A2/A3 Series `--eplb-rebalance-layers-per-chunk` `None` Type: int - A2, A3 + A2/A3 Series `--eplb-min-rebalancing-utilization-threshold` `1.0` Type: float - A2, A3 + A2/A3 Series `--expert-distribution-recorder-mode` `None` `stat`,
`stat_approx`,
`per_pass`,
`per_token` - A2, A3 + A2/A3 Series `--expert-distribution-recorder-buffer-size` `None` Type: int - A2, A3 + A2/A3 Series `--expert-balancedness-report-mode` off off, server_log, prometheus, both - A2, A3 + A2/A3 Series `--moe-dense-tp-size` `None` `1` - A2, A3 + A2/A3 Series `--elastic-ep-backend` @@ -1760,7 +1764,7 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD `--enable-waterfill` `False` bool flag (set to enable) - A2, A3 + A2/A3 Series @@ -1788,31 +1792,31 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD `--max-mamba-cache-size` `None` Type: int - A2, A3 + A2/A3 Series `--mamba-ssm-dtype` `float32` `float32`,
`bfloat16`,
`float16` - A2, A3 + A2/A3 Series `--mamba-full-memory-ratio` `0.9` Type: float - A2, A3 + A2/A3 Series `--mamba-radix-cache-strategy` `auto` `auto`,
`no_buffer`,
`extra_buffer` - A2, A3 + A2/A3 Series `--mamba-track-interval` `256` Type: int - A2, A3 + A2/A3 Series @@ -1841,43 +1845,43 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD `--enable-hierarchical-cache` `False` bool flag
(set to enable).
Currently, mamba cache is not supported. - A2, A3 + A2/A3 Series `--hicache-ratio` `2.0` Type: float - A2, A3 + A2/A3 Series `--hicache-size` `0` Type: int - A2, A3 + A2/A3 Series `--hicache-write-policy` `write_through` `write_back`,
`write_through`,
`write_through_selective` - A2, A3 + A2/A3 Series `--hicache-io-backend` `kernel` `kernel_ascend`,
`direct` - A2, A3 + A2/A3 Series `--hicache-mem-layout` `layer_first` `page_first_direct`,
`page_first_kv_split` - A2, A3 + A2/A3 Series `--hicache-storage-backend` `None` `file` - A2, A3 + A2/A3 Series `--hicache-storage-prefetch-policy` @@ -1952,13 +1956,13 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD `--dllm-algorithm` `None` Type: str - A2, A3 + A2/A3 Series `--dllm-algorithm-config` `None` Type: str - A2, A3 + A2/A3 Series @@ -1986,31 +1990,31 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD `--cpu-offload-gb` `0` Type: int - A2, A3 + A2/A3 Series `--offload-group-size` `-1` Type: int (DeepSeek only) - A2, A3 + A2/A3 Series `--offload-num-in-group` `1` Type: int (DeepSeek only) - A2, A3 + A2/A3 Series `--offload-prefetch-step` `1` Type: int (DeepSeek only) - A2, A3 + A2/A3 Series `--offload-mode` `cpu` `cpu` (DeepSeek only)
`meta` (DeepSeek only)
`sharded_gpu` (DeepSeek only, only support tp=1 dp>1) - A2, A3 + A2/A3 Series @@ -2040,91 +2044,85 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD `--disable-radix-cache` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series `--cuda-graph-config` `None` Type: JSON - A2, A3 + A2/A3 Series `--cuda-graph-backend-decode` `None` `full`,
`disabled` - A2, A3 + A2/A3 Series `--cuda-graph-backend-prefill` `None` - `disabled` - A2, A3 + `disabled`, `tc_piecewise`
(`tc_piecewise` currently supports Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct) + A2/A3 Series `--cuda-graph-max-bs-decode` `None` Type: int - A2, A3 + A2/A3 Series `--cuda-graph-max-bs-prefill` `None` Type: int - A2, A3 + A2/A3 Series `--cuda-graph-bs-decode` `None` Type: List[int] - A2, A3 + A2/A3 Series `--cuda-graph-bs-prefill` `None` Type: List[int] - A2, A3 + A2/A3 Series `--disable-prefill-cuda-graph` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series `--disable-decode-cuda-graph` `False` bool flag
(set to enable) - A2, A3 - - - `--cuda-graph-bs` - `None` - List[int] - A2, A3 + A2/A3 Series `--disable-cuda-graph` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series `--disable-cuda-graph-padding` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series `--enable-profile-cuda-graph` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series `--enable-cudagraph-gc` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series `--enable-nccl-nvls` @@ -2148,19 +2146,19 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD `--enable-tokenizer-batch-encode` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series `--disable-tokenizer-batch-decode` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series `--disable-custom-all-reduce` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series `--enable-mscclpp` @@ -2172,7 +2170,7 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD `--pre-warm-nccl` `False` bool flag (set to enable) - A2, A3 + A2/A3 Series `--enable-torch-symm-mem` @@ -2184,31 +2182,31 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD `--disable-overlap-schedule` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series `--enable-mixed-chunk` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series `--enable-dp-attention` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series `--enable-dp-attention-local-control-broadcast` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series `--enable-dp-lm-head` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series `--enable-two-batch-overlap` @@ -2232,43 +2230,19 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD `--enable-torch-compile` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series `--enable-torch-compile-debug-mode` `False` bool flag
(set to enable) - A2, A3 - - - `--enforce-piecewise-cuda-graph` - `False` - bool flag
(set to enable);
Currently, Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct models are supported. - A2, A3 - - - `--piecewise-cuda-graph-tokens` - `None` - Type: JSON
list - A2, A3 - - - `--piecewise-cuda-graph-compiler` - `eager` - `eager` - A2, A3 + A2/A3 Series `--torch-compile-max-bs` `32` Type: int - A2, A3 - - - `--piecewise-cuda-graph-max-tokens` - `None` - Type: int - A2, A3 + A2/A3 Series `--enable-p2p-check` @@ -2298,37 +2272,37 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD `--delete-ckpt-after-loading` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series `--enable-memory-saver` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series `--enable-weights-cpu-backup` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series `--enable-draft-weights-cpu-backup` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series `--allow-auto-truncate` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series `--enable-custom-logit-processor` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series `--flashinfer-mla-disable-ragged` @@ -2340,79 +2314,67 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD `--disable-shared-experts-fusion` `True` bool flag
(set to enable) - A2, A3 - - - `--enforce-shared-experts-fusion` - `False` - bool flag
(set to enable) - A2, A3 + A2/A3 Series `--disable-chunked-prefix-cache` `True` bool flag
(set to enable) - A2, A3 + A2/A3 Series `--image-processor-backend` `auto` `auto`, `torchvision`, `pil` - A2, A3 + A2/A3 Series `--disable-fast-image-processor` `False` bool flag
(set to enable) - A2, A3 - - - `--keep-mm-feature-on-device` - `False` - bool flag
(set to enable) - A2, A3 + A2/A3 Series `--enable-return-hidden-states` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series `--enable-return-routed-experts` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series `--scheduler-recv-interval` `1` Type: int - A2, A3 + A2/A3 Series `--numa-node` `None` List[int] - A2, A3 + A2/A3 Series `--enable-deterministic-inference` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series `--rl-on-policy-target` `None` `fsdp` - A2, A3 + A2/A3 Series `--disable-attn-tp-gather` `False` bool flag (set to enable) - A2, A3 + A2/A3 Series `--enable-layerwise-nvtx-marker` @@ -2424,19 +2386,19 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD `--enable-attn-tp-input-scattered` `False` bool flag
(set to enable) - A2, A3 + Experimental `--enable-prefill-cp` `False` bool flag (set to enable) - A2, A3 + A2/A3 Series `--cp-strategy` `None` `zigzag` - A2, A3 + A2/A3 Series `--enable-fused-qk-norm-rope` @@ -2448,13 +2410,13 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD `--enable-precise-embedding-interpolation` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series `--gc-threshold` `None` List[int] - A2, A3 + A2/A3 Series @@ -2482,19 +2444,19 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD `--enable-dynamic-batch-tokenizer` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series `--dynamic-batch-tokenizer-batch-size` `32` Type: int - A2, A3 + A2/A3 Series `--dynamic-batch-tokenizer-batch-timeout` `0.002` Type: float - A2, A3 + A2/A3 Series @@ -2522,19 +2484,19 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD `--debug-tensor-dump-output-folder` `None` Type: str - A2, A3 + A2/A3 Series `--debug-tensor-dump-layers` `None` List[int] - A2, A3 + A2/A3 Series `--debug-tensor-dump-input-file` `None` Type: str - A2, A3 + A2/A3 Series @@ -2563,19 +2525,19 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD `--disaggregation-mode` `null` `null`,
`prefill`,
`decode` - A2, A3 + A2/A3 Series `--disaggregation-transfer-backend` `mooncake` `ascend` (default `mooncake` not supported on NPU, must be specified manually) - A2, A3 + A2/A3 Series `--disaggregation-bootstrap-port` `8998` Type: int - A2, A3 + A2/A3 Series `--disaggregation-ib-device` @@ -2587,25 +2549,25 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD `--disaggregation-decode-enable-offload-kvcache` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series `--num-reserved-decode-tokens` `512` Type: int - A2, A3 + A2/A3 Series `--disaggregation-decode-polling-interval` `1` Type: int - A2, A3 + A2/A3 Series `--optimistic-prefill-attempts` `0` Type: int - A2, A3 + A2/A3 Series @@ -2633,43 +2595,43 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD `--enable-adaptive-dispatch-to-encoder` `False` bool flag
(set to enable adaptive dispatch) - A2, A3 + A2/A3 Series `--encoder-only` `False` bool flag
(set to launch an encoder-only server) - A2, A3 + A2/A3 Series `--language-only` `False` bool flag
(set to load weights for the language model only) - A2, A3 + A2/A3 Series `--encoder-transfer-backend` `zmq_to_scheduler` `zmq_to_scheduler`,
`zmq_to_tokenizer`,
`mooncake` - A2, A3 + A2/A3 Series `--encoder-urls` `[]` List[str]
(List of encoder server urls) - A2, A3 + A2/A3 Series `--encoder-bootstrap-port` `8997` Type: int - A2, A3 + A2/A3 Series `--encoder-register-urls` `[]` List[str] - A2, A3 + A2/A3 Series @@ -2698,55 +2660,55 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD `--custom-weight-loader` `None` List[str] - A2, A3 + A2/A3 Series `--weight-loader-disable-mmap` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series `--weight-loader-prefetch-checkpoints` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series `--weight-loader-prefetch-num-threads` `4` Type: int - A2, A3 + A2/A3 Series `--remote-instance-weight-loader-seed-instance-ip` `None` Type: str - A2, A3 + A2/A3 Series `--remote-instance-weight-loader-seed-instance-service-port` `None` Type: int - A2, A3 + A2/A3 Series `--remote-instance-weight-loader-send-weights-group-ports` `None` Type: JSON
list - A2, A3 + A2/A3 Series `--remote-instance-weight-loader-backend` `nccl` `transfer_engine`,
`nccl`,
`modelexpress` - A2, A3 + A2/A3 Series `--weight-loader-drop-cache-after-load` `False` bool flag (set to enable) - A2, A3 + A2/A3 Series `--remote-instance-weight-loader-start-seed-via-transfer-engine` @@ -2821,37 +2783,37 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD `--enable-broadcast-mm-inputs-process` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series `--mm-process-config` `None` Type: JSON / Dict - A2, A3 + A2/A3 Series `--mm-enable-dp-encoder` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series `--mm-processor-worker-num` `0` Type: int - A2, A3 + A2/A3 Series `--mm-io-worker-num` `0` Type: int - A2, A3 + A2/A3 Series `--limit-mm-data-per-request` `None` Type: JSON / Dict - A2, A3 + A2/A3 Series @@ -2879,19 +2841,19 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD `--decrypted-config-file` `None` Type: str - A2, A3 + A2/A3 Series `--decrypted-draft-config-file` `None` Type: str - A2, A3 + A2/A3 Series `--enable-prefix-mm-cache` `False` bool flag
(set to enable) - A2, A3 + A2/A3 Series @@ -2920,13 +2882,13 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD `--forward-hooks` `None` Type: JSON list - A2, A3 + A2/A3 Series `--enable-quant-communications` `False` bool flag (set to enable)
(When using DeepEP with a quantized model, you must set --deepep-dispatcher-output-dtype) - A2, A3 + A2/A3 Series @@ -2954,7 +2916,7 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD `--msprobe-dump-config` `None` Type: str - A2, A3 + A2/A3 Series @@ -2982,7 +2944,7 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD `--config` `None` Type: str - A2, A3 + A2/A3 Series diff --git a/docs/docs/hardware-platforms/ascend-npus/reference/support_models.mdx b/docs/docs/hardware-platforms/ascend-npus/reference/support_models.mdx index 32743c162152..1840161bb1a6 100644 --- a/docs/docs/hardware-platforms/ascend-npus/reference/support_models.mdx +++ b/docs/docs/hardware-platforms/ascend-npus/reference/support_models.mdx @@ -20,8 +20,8 @@ You are welcome to enable various models based on your business requirements. Models Model Family - A2 Supported - A3 Supported + Ascend A2 Series Products Supported + Ascend A3 Series Products Supported @@ -391,12 +391,6 @@ You are welcome to enable various models based on your business requirements. ✅ ✅ - - FLM/Tele-FLM - Tele FLM (52B-1T) - ✅ - ✅ - bigcode/starcoder2-7b StarCoder2 @@ -421,30 +415,6 @@ You are welcome to enable various models based on your business requirements. ✅ ✅ - - LiquidAI/LFM2.5-1.2B-Instruct - LFM2 (230M, 350M, 1.2B) - ✅ - ✅ - - - LiquidAI/LFM2-8B-A1B - LFM2-MoE (8B-A1B, 24B-A2B) - ✅ - ✅ - - - tiiuae/Falcon-H1-34B-Instruct - Falcon-H1 (0.5B-34B) - ✅ - ✅ - - - ibm-granite/granite-4.0-h-micro - IBM Granite 4.0 (Hybrid, Dense) - ✅ - ✅ - ibm-granite/granite-4.0-micro IBM Granite 4.0 (Hybrid, Dense) @@ -479,8 +449,8 @@ You are welcome to enable various models based on your business requirements. Models Model Family (Variants) - A2 Supported - A3 Supported + Ascend A2 Series Products Supported + Ascend A3 Series Products Supported @@ -628,36 +598,12 @@ You are welcome to enable various models based on your business requirements. ✅ ✅ - - lmms-lab/LLaVA-NeXT-Video-7B - LLaVA video variants (LLaVA-NeXT-Video, LLaVA-OneVision) - ✅ - ✅ - - - ZhipuAI/GLM-OCR - GLM-OCR - ✅ - ✅ - - - AI-ModelScope/whisper-large-v3 - Whisper - ✅ - ✅ - Qwen/Qwen3-ASR-1.7B Qwen3-ASR (0.6B, 1.7B) ✅ ✅ - - LiquidAI/LFM2.5-VL-1.6B - LFM2-VL - ✅ - ✅ - @@ -674,8 +620,8 @@ You are welcome to enable various models based on your business requirements. Models Model Family - A2 Supported - A3 Supported + Ascend A2 Series Products Supported + Ascend A3 Series Products Supported @@ -703,12 +649,6 @@ You are welcome to enable various models based on your business requirements. ✅ ✅ - - AI-ModelScope/clip-vit-large-patch14-336 - CLIP - ✅ - ✅ - BAAI/bge-large-en-v1.5 BGE @@ -731,8 +671,8 @@ You are welcome to enable various models based on your business requirements. Models Model Family - A2 Supported - A3 Supported + Ascend A2 Series Products Supported + Ascend A3 Series Products Supported @@ -776,8 +716,8 @@ You are welcome to enable various models based on your business requirements. Models Model Family - A2 Supported - A3 Supported + Ascend A2 Series Products Supported + Ascend A3 Series Products Supported diff --git a/docs/docs/hardware-platforms/mthreads_gpu.mdx b/docs/docs/hardware-platforms/mthreads_gpu.mdx index ce31173c23b6..f16dc17ef97e 100644 --- a/docs/docs/hardware-platforms/mthreads_gpu.mdx +++ b/docs/docs/hardware-platforms/mthreads_gpu.mdx @@ -4,7 +4,20 @@ metatags: description: "Run SGLang on Moore Threads GPUs." --- -This document describes how run SGLang on Moore Threads GPUs. If you encounter issues or have questions, please [open an issue](https://github.com/sgl-project/sglang/issues). +This document describes how to run SGLang on Moore Threads GPUs. If you +encounter issues or have questions, please [open an issue](https://github.com/sgl-project/sglang/issues). + +SGLang uses the MUSA runtime through +[`torch_musa`](https://github.com/MooreThreads/torch_musa) and +[`torchada`](https://github.com/MooreThreads/torchada). The +`python[all_musa]` extra installs the MUSA torch, Triton, TileLang, MATE, and +runtime Python stack from the Moore Threads package index, with public PyPI as +the fallback for common Python dependencies. + +## Prerequisites + +Install the Moore Threads driver and MUSA toolkit before installing SGLang. For +MTT S5000, follow the [official driver installation guide](https://docs.mthreads.com/driver-linux-server/driver-linux-server-doc-online/MTT_S5000/install_guide). ## Install SGLang @@ -12,18 +25,92 @@ You can install SGLang using one of the methods below. ### Install from Source -```bash -# Use the default branch +Start from an environment with the prerequisites above installed. + +```bash Command git clone https://github.com/sgl-project/sglang.git cd sglang -# Compile sgl-kernel -pip install --upgrade pip +python -m pip install --upgrade pip "setuptools<82" wheel +cp python/pyproject_other.toml python/pyproject.toml +python -m pip install -e "python[all_musa]" \ + --index-url https://dl.mthreads.com/repo/api/pypi/pypi/simple \ + --extra-index-url https://pypi.org/simple \ + --trusted-host dl.mthreads.com \ + --no-build-isolation + +# Ensure the MUSA Triton wheel (with triton.backends.mtgpu) is installed. +python -m pip install --no-cache-dir --force-reinstall --no-deps \ + --index-url https://dl.mthreads.com/repo/api/pypi/pypi/simple \ + --trusted-host dl.mthreads.com \ + triton==3.2.0 +python -c 'import triton.backends.mtgpu' + cd python/sglang/kernels/aot -python setup_musa.py install +cp pyproject_musa.toml pyproject.toml +MTGPU_TARGET=mp_31 python setup_musa.py install +``` + +### Install Using Docker + +The Dockerfile installs the MUSA runtime and Python stack declared by +`python/pyproject_other.toml`. + +```bash Command +git clone https://github.com/sgl-project/sglang.git +cd sglang + +docker build -f docker/musa.Dockerfile -t sglang:main-musa . +``` + +Run the image with MUSA devices exposed by the host container toolkit. + +```bash Command +alias drun='docker run -it --rm --network=host \ + --env MTHREADS_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \ + --env MTHREADS_DRIVER_CAPABILITIES=all \ + --shm-size=32g \ + -v ~/.cache/huggingface:/root/.cache/huggingface' + +drun sglang:main-musa \ + python3 -c 'import triton.backends.mtgpu; print("MUSA Triton backend OK")' +``` + +Launch the server: + +```bash Command +drun sglang:main-musa \ + python3 -m sglang.launch_server \ + --model-path Qwen/Qwen3-8B \ + --host 0.0.0.0 \ + --port 30000 +``` + +When the server displays `The server is fired up and ready to roll!`, startup is +successful. + +## Verify + +You can send an OpenAI-compatible request to the engine: + +```bash Command +curl http://127.0.0.1:30000/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "Qwen/Qwen3-8B", + "messages": [{"role": "user", "content": "Hello!"}], + "max_tokens": 32 + }' +``` + +Or run a benchmark: -# Install sglang python package along with diffusion support -cd ../../../.. -rm -f python/pyproject.toml && mv python/pyproject_other.toml python/pyproject.toml -pip install -e "python[all_musa]" +```bash Command +drun sglang:main-musa \ + python3 -m sglang.bench_serving \ + --backend sglang \ + --dataset-name random \ + --num-prompts 4000 \ + --random-input 128 \ + --random-output 128 ``` diff --git a/docs/docs/hardware-platforms/overview.mdx b/docs/docs/hardware-platforms/overview.mdx index 3129f50843ce..157af87b0831 100644 --- a/docs/docs/hardware-platforms/overview.mdx +++ b/docs/docs/hardware-platforms/overview.mdx @@ -10,3 +10,4 @@ description: Platform-specific guides for running SGLang on GPUs, TPUs, NPUs, CP - [NVIDIA Jetson Orin](./nvidia_jetson) - [TPU](./tpu) - [XPU](./xpu) +- [Moore Threads GPUs](./mthreads_gpu) diff --git a/docs/docs/hardware-platforms/plugin.mdx b/docs/docs/hardware-platforms/plugin.mdx index dda26d0ef7c1..2e4f78c411a3 100644 --- a/docs/docs/hardware-platforms/plugin.mdx +++ b/docs/docs/hardware-platforms/plugin.mdx @@ -19,19 +19,19 @@ The framework provides two plugin types, both discovered via Python's standard ` Plugin Type - Entry Point Group + Entry Point Groups Purpose Hardware Platform Plugin - sglang.srt.platforms + sglang.srt.platforms
sglang.multimodal_gen.platforms Register a custom hardware platform (device operations, KV cache pools, attention backends, graph capture, compilation backends, etc.) General Plugin - sglang.srt.plugins + sglang.srt.plugins
sglang.multimodal_gen.plugins Inject hooks (before/after/around/replace) into any function/method, or replace entire classes @@ -39,44 +39,38 @@ The framework provides two plugin types, both discovered via Python's standard ` ### Principles -- **Non-intrusive**: Existing CUDA/ROCm/NPU/XPU code remains unchanged. OOT code paths are added alongside existing hardware-specific logic. -- **Zero configuration**: Plugins are automatically discovered after `pip install`, no sglang code changes required. -- **Environment variable control**: `SGLANG_PLATFORM` selects or validates the active platform plugin; `SGLANG_PLUGINS` (comma-separated) controls which general plugins to load. +- **Non-intrusive**: Built-in platforms remain the fallback when no OOT platform activates. +- **Install-time discovery**: Plugins are discovered from Python entry points after installation. +- **Environment variable control**: `SGLANG_PLATFORM` selects an SRT platform, `SGLANG_DIFFUSION_PLATFORM_OVERRIDE` selects a diffusion platform, and `SGLANG_PLUGINS` filters general plugins in either hook group. -### Current Scope & Future Direction +### Current scope -The plugin system currently targets **out-of-tree (OOT) hardware platforms** — enabling new devices to integrate with SGLang without any changes to the main repository. The main-repo hardware paths (CUDA, ROCm, NPU, XPU, etc.) continue to use the existing `is_cuda()`/`is_npu()`/… utility functions. - -As the plugin interfaces mature and stabilize, in-tree hardware backends can be gradually migrated to the same plugin architecture. This would replace the scattered `if device == "cuda" … elif device == "npu" …` branches throughout the codebase with a single polymorphic dispatch through the platform interface, making each hardware backend self-contained and the core engine hardware-agnostic. +The platform plugin system targets **out-of-tree (OOT) hardware platforms**. Diffusion support is experimental and covers the seams documented below, not every device-specific branch. ## Architecture -### Platform Hierarchy +### Runtime-specific platform interfaces -The platform hierarchy uses a DeviceMixin pattern to share device operations between SRT (LLM inference) and Multimodal subsystems: +SRT and diffusion have separate platform base classes and platform identity types in `sglang.srt.platforms` and `sglang.multimodal_gen.runtime.platforms`. A package supporting both runtimes should define separate platform classes and use each runtime's own `PlatformEnum.OOT` value. +Each platform entry point resolves to a zero-argument activation callback: + +```python +def activate() -> str | None: + """Return the platform class qualname when this hardware is available.""" ``` -DeviceMixin (shared device identity + operations) -├── SRTPlatform(DeviceMixin) # + graph runner, KV pool, … -│ └── MySRTPlatform(SRTPlatform, MyDeviceMixin) # OOT plugin -└── MMPlatform(DeviceMixin) # + attention backend, VAE, … (future) - └── MyMMPlatform(MMPlatform, MyDeviceMixin) # OOT plugin -``` -Key design points: -- **DeviceMixin** provides platform identity queries (`is_cuda()`, `is_npu()`, etc.) and device operations (`set_device()`, `get_device_name()`, etc.) -- **SRTPlatform** adds SRT-specific factory methods, capability flags, and lifecycle hooks -- OOT plugins implement a **device mixin** (vendor-specific operations) and compose it with **SRTPlatform** via multiple inheritance -- All methods are **instance methods** (not classmethods), called through the `current_platform` singleton -- Device operations and factory methods raise `NotImplementedError` by default (fail-fast) -- Capability flags use safe conservative defaults (`False`/`pass`) -- Methods are annotated `[Active]` (called by SGLang core) or `[Planned]` (reserved for future migration) +Return a fully qualified class name when the provider can run, or `None` otherwise. Keep activation import-safe: do not access `current_platform` or initialize runtime or device state. Put required backend setup in the platform's `init_backend()` method and reserve general plugins for hooks that the platform interface cannot express. + +Explicit selection enumerates entry-point metadata and imports only the selected callback. Automatic selection invokes installed platform callbacks to determine which provider is active. + +Diffusion reserves `cpu`, `cuda`, `rocm`, `xpu`, `mps`, `npu`, and `musa`. Automatic discovery also rejects duplicate entry-point names; explicit selection validates only the selected name and does not import unrelated providers. -### Platform Discovery (`current_platform`) +#### SRT selection `current_platform` is a **lazy singleton** in `sglang.srt.platforms`. On first access it resolves the active platform through the following priority chain: -``` +```text entry_points("sglang.srt.platforms") → Enumerate ALL plugins by name (metadata only) │ ├─ SGLANG_PLATFORM set (front-loading filter): @@ -94,9 +88,43 @@ entry_points("sglang.srt.platforms") → Enumerate ALL plugins by name (metadat └─ N activated → RuntimeError (must set SGLANG_PLATFORM) ``` +#### Diffusion selection + +SGLang Diffusion resolves its platform in this order: + +1. When `SGLANG_DIFFUSION_PLATFORM_OVERRIDE` names `cpu`, `cuda`, `rocm`, `mps`, `npu`, or `musa`, select that built-in platform without hardware probing. XPU remains automatic-only, preserving the existing selector behavior. +2. When it contains another name, load only the matching `sglang.multimodal_gen.platforms` entry point. An unknown name or a callback that returns `None` is an error. +3. When it is unset, activate installed diffusion platform plugins. No active plugin continues to built-in detection, one selects that plugin, and multiple active plugins raise an error that asks you to set the selector. An activation callback that raises aborts startup rather than falling back, so a broken vendor runtime cannot silently run the job on a built-in platform. +4. Try built-in platforms in order: MPS, XPU, ROCm, CUDA, NPU, MUSA, then CPU. + +The existing override variable is therefore the single explicit selector for supported built-in aliases and OOT entry-point names. + +Selection resolves lazily, the first time anything in a process touches `current_platform`, so it needs no call site and happens in every process automatically. SGLang Diffusion also records which distribution supplied the selected platform, and skips the hooks of every other installed platform package. + +#### Required platform initialization vs. optional hooks + +The two mechanisms have different failure semantics, and it matters which one you use: + +| | Platform contract | General plugin | +| --- | --- | --- | +| Entry-point group | `sglang.multimodal_gen.platforms` | `sglang.multimodal_gen.plugins` | +| Delivery | methods on your `Platform` subclass | hooks that monkey-patch a target | +| Activation | lazy `current_platform`, plus guarded `init_backend()` in each worker | registration with `load_plugins()`, then explicit `apply_plugin_hooks()` per process | +| A failure | aborts startup | aborts startup if the plugin ships in the selected platform's distribution; otherwise logged | + +Anything your hardware needs in order to be correct belongs on the `Platform` subclass, so a broken platform cannot silently serve. Reach for a general plugin only when the `Platform` interface has no seam for what you need — and please report that gap. + +A plugin shipped in the selected platform's own distribution is treated as part of that platform's contract: a failure to load it, to run its callback, or to apply any hook it registered aborts startup rather than leaving the platform half-initialized. Plugins from any other installed package stay best-effort, so a broken third party cannot take the server down. An explicit `SGLANG_PLUGINS` allowlist can disable any general plugin, including one from the selected platform package; required hardware setup therefore belongs in `init_backend()`. + ### Plugin Loading Flow -`load_plugins()` discovers and executes general plugins, then applies all registered hooks. It is called at four points: +Each runtime has a process-local hook registry. SRT retains its single +`load_plugins()` activation step. Diffusion separates registration +(`load_plugins()`) from target resolution (`apply_plugin_hooks()`), because +resolving a dotted hook target can import that target's entire module graph. +Both runtimes honor `SGLANG_PLUGINS`. + +The loader is called at these SRT entry points: @@ -113,7 +141,7 @@ entry_points("sglang.srt.platforms") → Enumerate ALL plugins by name (metadat - + @@ -135,9 +163,9 @@ entry_points("sglang.srt.platforms") → Enumerate ALL plugins by name (metadat
cli/serve.py serve()cli/serve.py serve() Main Before prepare_server_args()
-> **Note**: `load_plugins()` is idempotent (guarded by `_plugins_loaded` flag). In spawn'd subprocesses the flag resets, so plugins are correctly re-loaded. +> **Note**: Diffusion plugin registration and hook application each run once per process. Spawned subprocesses start from a blank interpreter and establish their own hooks — nothing the parent patched survives the spawn boundary. SRT's `load_plugins()` performs both phases in one call. -``` +```text load_plugins() ├── _get_excluded_dists() → compute dists to skip (via SGLANG_PLATFORM) ├── load_plugins_by_group("sglang.srt.plugins", → discover entry_points, filter by SGLANG_PLUGINS @@ -147,15 +175,76 @@ load_plugins() └── HookRegistry.apply_hooks() → monkey-patch targets ``` +Diffusion launchers call `apply_plugin_hooks()`, which first performs registration +if necessary and then resolves and patches targets. Scheduler children use a +stricter lifecycle in `runtime/managers/worker_bootstrap.py`: + +```text +spawn unpickles SchedulerProcessSpec (stdlib types + opaque ServerArgs bytes) + -> initialize_current_platform() + -> load_plugins() # discover callbacks and register hooks only + -> apply_plugin_hooks() # target imports are allowed from here onward + -> materialize ServerArgs + -> resolve and invoke the patched run_scheduler_process +``` + +HTTP children likewise finish plugin registration and hook application before +importing `runtime.launch_server` or materializing `ServerArgs`. This matters for +class replacements and other hooks whose effects cannot be retroactively applied +to classes or registrations created while importing the server module graph. + +The bootstrap module is the `mp.Process` target and imports no diffusion runtime +module at module scope. `ServerArgs` is serialized inside `ServerArgsPayload`, so +the multiprocessing unpickler cannot import pipeline configuration modules +before the target starts. Workers always use a local `spawn` context, independent +of an embedding application's global multiprocessing setting. + +This ordering makes backend initialization precede plugin callback imports, +hook-target resolution, `ServerArgs` materialization, and worker imports. A +platform activation module necessarily loads before its own `init_backend()`; +activation modules must therefore remain import-safe. Prefer platform methods +and registries for required behavior. + +Each activation phase runs once per process behind a lock: a re-entrant call +from a plugin callback returns and lets the outer call finish, another thread +waits for it, and a failure is terminal. A callback must not hand activation to +a second thread and join it — that deadlocks. + +#### Offline scripts and the spawn boundary + +`spawn` re-executes the launching script's module scope in every child *before* +it unpickles the target's arguments, so an offline script's own imports run +ahead of that child's platform initialization and `ServerArgsPayload` cannot +help. The supported script layout is: + +```python +from sglang.multimodal_gen import DiffGenerator # a proxy: imports nothing yet + +if __name__ == "__main__": + generator = DiffGenerator.from_pretrained(model_path="...") +``` + +`DiffGenerator` on the `sglang.multimodal_gen` facade is a lazy proxy, so +binding it at module scope costs no diffusion import and the child still reaches +`initialize_current_platform()` with a clean module table. Module scope also +stays open to `envs` and `runtime.platforms`, including `runtime.platforms.plugins`. +These modules remain import-safe so a plugin can subclass `Platform` and register +hooks before backend initialization. Everything else, including `SamplingParams` and `PipelineConfig` +from the same facade, belongs inside the `if __name__ == "__main__":` guard or +inside the function that uses it. A child +that finds runtime modules already imported names them in a warning: hook +application can still patch those modules, but classes and registrations +created while importing them are already past reach. + --- ## Plugin Type 1: Hardware Platform Plugin ### Description -A hardware platform plugin registers an `SRTPlatform` subclass that tells SGLang how to interact with a specific hardware backend. +A hardware platform plugin registers an SRT `SRTPlatform` subclass, a diffusion `Platform` subclass, or both. The selected class tells that runtime how to interact with a specific hardware backend. -### Quick Start +### SRT quick start **1. Create a minimal package:** @@ -230,7 +319,7 @@ pip install -e my_platform_plugin/ python -c "from sglang.srt.platforms import current_platform; print(current_platform)" ``` -### Platform Interface Reference +### SRT platform interface reference #### Identity Queries (from DeviceMixin) @@ -578,7 +667,7 @@ python -c "from sglang.srt.platforms import current_platform; print(current_plat -### Environment Variables +### Platform and plugin environment variables @@ -598,18 +687,138 @@ python -c "from sglang.srt.platforms import current_platform; print(current_plat - +
SGLANG_PLUGINSComma-separated whitelist of general plugin names to load (group: sglang.srt.plugins). If unset, all discovered general plugins are loaded.Comma-separated whitelist of general plugin names to load from either hook group. It also filters automatic SRT platform discovery when SGLANG_PLATFORM is unset; explicit platform selection ignores it.
+### Add diffusion support + +A package may support SRT, SGLang Diffusion, or both. Diffusion uses a zero-argument `Platform` subclass and separate platform and hook entry points: + +```toml +[project.entry-points."sglang.multimodal_gen.platforms"] +my_device = "my_platform_plugin:activate_diffusion" + +[project.entry-points."sglang.multimodal_gen.plugins"] +my_device = "my_platform_plugin:register_hooks" +``` + +Keep activation import-safe and return the fully qualified class name only when the backend is available: + +```python +def activate_diffusion() -> str | None: + try: + import my_device_runtime + except ImportError: + return None + if not my_device_runtime.is_available(): + return None + return "my_platform_plugin.diffusion_platform.MyDiffusionPlatform" +``` + +The referenced class must be zero-argument constructible: + +```python +from sglang.multimodal_gen.runtime.platforms import Platform, PlatformEnum + +class MyDiffusionPlatform(Platform): + _enum = PlatformEnum.OOT + device_name = "my_device" + device_type = "my_device" + dispatch_key = "PrivateUse1" + + def get_dispatch_key_name(self) -> str: + return "my_device" +``` + +Set `dispatch_key` to the PyTorch dispatcher key used by direct `torch.library` registrations. Keep it separate from `get_dispatch_key_name()`, which selects `CustomOp` implementations. Bring up the remaining contract in dependency order: + +1. Implement `get_device_name()`, `get_device_total_memory()`, and `get_available_gpu_memory()` before constructing `ServerArgs`. +2. Implement `get_device()` and `get_local_torch_device()` before worker binding. +3. Configure distributed initialization, attention selection, and custom-op implementations for supported workloads. +4. Run an end-to-end workload and audit model-specific kernels, compilation, and remaining device-family branches. + +Register custom-op implementations from `init_backend()`, which runs once in each worker before any pipeline module is constructed: + +```python +class MyDiffusionPlatform(Platform): + ... + + def init_backend(self) -> None: + from my_platform_plugin.ops import rms_norm_forward + from sglang.multimodal_gen.runtime.layers.custom_op import CustomOp + from sglang.multimodal_gen.runtime.layers.layernorm import RMSNorm + + CustomOp.register_oot_forward( + RMSNorm, fn=rms_norm_forward, platform_key="my_device" + ) +``` + +`init_backend()` runs at most once per worker process, before the worker implementation is imported. Raising aborts startup and the failed initialization is not retried in that process, because registrations and other backend side effects may be only partially reversible. Custom-op dispatch is resolved when each operation is constructed, after backend initialization has completed. The selected callable therefore remains stable when the operation is compiled instead of changing on the first compiled call. + +The function receives the operation instance before its normal arguments. Registration matches the exact operation class and the value returned by `get_dispatch_key_name()`. Without a registration, SGLang looks for a matching `forward_()` implementation on the operation, then falls back to `forward_oot()`, whose base implementation calls `forward_native()`. Returning `"cuda"`, for example, lets a CUDA-compatible OOT platform reuse operation-specific `forward_cuda()` implementations without reporting CUDA platform identity. + +The `sglang.multimodal_gen.plugins` entry point remains for seams the `Platform` interface does not cover. Its callbacks run in launchers and workers before scheduler or model construction, so use a `BEFORE` or `AROUND` hook on `GPUWorker.init_device_and_model()` for rank-aware initialization. Hooks from platform distributions other than the selected one are skipped. Because these ship in your platform's distribution, a failure at any stage — load, callback, or hook application — aborts startup, the same as `init_backend()`. + +#### Diffusion platform contract + +**Active** interfaces are called through `current_platform`. **Compatibility** interfaces are retained adapters; unlisted `Platform` methods are not a stable OOT contract. + +| Status | Area | Interface | +| --- | --- | --- | +| Active | Initialization | `init_backend()` | +| Active | Configuration | `apply_server_args_defaults()` | +| Active | Device and memory | `get_device()`, `set_device()`, `get_local_torch_device()`, `get_device_name()`, `get_device_uuid()`, `get_device_total_memory()`, `get_available_gpu_memory()`, `get_device_capability()` | +| Active | Distributed | `get_torch_distributed_backend_str()`, `supports_distributed_device_id()`, `get_all_to_all_communicator_cls()`, `get_cpu_architecture()` | +| Active | Dispatch and model | `dispatch_key` (read by `get_torch_library_dispatch_key()`), `get_dispatch_key_name()`, `get_attn_backend_cls_str()`, `verify_model_arch()`, `optimize_vae()` | +| Active | Execution | `get_compile_backend()`, `get_compile_options()`, `inference_mode()`, `seed_everything()`, `enable_dit_layerwise_offload_by_default()` | +| Compatibility | Communicator | `get_device_communicator_cls()` remains the fallback used by `get_all_to_all_communicator_cls()` for existing platform subclasses. New OOT platforms should override the latter. | + +Install the package, restart Python to refresh entry-point metadata, and verify explicit selection: + +```bash +pip install -e my_platform_plugin/ +SGLANG_DIFFUSION_PLATFORM_OVERRIDE=my_device python -c \ + "from sglang.multimodal_gen.runtime.platforms import current_platform; print(current_platform)" +``` + +### Diffusion platform limitations + +- SRT and diffusion require separate platform classes. +- Only one external platform can be active per process; set the runtime selector if multiple callbacks activate. +- `get_all_to_all_communicator_cls()` controls only `all_to_all_4D()`, not every collective, graph-capture, or synchronization path; `get_device_communicator_cls()` remains for compatibility. +- Compile settings affect only `build_torch_compile_kwargs()` callers; audit static `@torch.compile` decorators and other direct compile paths. +- Explicit attention selectors accept only `AttentionBackendEnum` names. Returning a custom backend works when none is selected; selecting it by name requires a hook or downstream patch. +- Device-family branches remain outside the interface. Each supported workload needs a native fallback or an actionable unsupported-feature error. +- The diffusion `PlatformEnum.OOT` identifies an external provider. Override built-in identity predicates only after auditing every enabled branch. + --- ## Plugin Type 2: General Plugin ### Description -General function plugins inject behavior into sglang **without requiring a custom platform**. Use cases include: +General function plugins inject behavior into SRT or diffusion **without requiring a custom platform**. The two runtimes have separate entry-point groups and hook registries: + +```python +# SRT plugin +from sglang.srt.plugins.hook_registry import HookType, plugin_hook + +# Diffusion plugin +from sglang.multimodal_gen.runtime.platforms.plugins import HookType, plugin_hook +``` + +Register an SRT plugin under `sglang.srt.plugins` and a diffusion plugin under +`sglang.multimodal_gen.plugins`, and import the hook API from the matching +module above. Each runtime applies only its own registry, so a hook registered +through the other runtime's API never runs. The examples below use SRT. + +The diffusion hook API lives alongside platform discovery in +`sglang.multimodal_gen.runtime.platforms.plugins`; +`sglang.multimodal_gen.plugins` is its entry-point group, not a Python module path. + +Use cases include: - **Observability**: Add logging, metrics, and tracing to any function - **Behavior modification**: Modify function arguments or return values @@ -766,7 +975,7 @@ Target paths use fully-qualified dotted notation. Both formats are supported: - **Dotted**: `sglang.srt.managers.scheduler.Scheduler.__init__` - **Entry-points style**: `sglang.srt.managers.scheduler:Scheduler.__init__` (colon treated as dot) -### Common Hook Targets +### Common SRT Hook Targets @@ -837,7 +1046,7 @@ Target paths use fully-qualified dotted notation. Both formats are supported: - + @@ -847,6 +1056,22 @@ Target paths use fully-qualified dotted notation. Both formats are supported: + + + + + + + + + + + + + + + + diff --git a/docs/docs/hardware-platforms/tpu.mdx b/docs/docs/hardware-platforms/tpu.mdx index b3d7f2516823..f386ebf9c3b6 100644 --- a/docs/docs/hardware-platforms/tpu.mdx +++ b/docs/docs/hardware-platforms/tpu.mdx @@ -165,7 +165,7 @@ SGLang-JAX provides comprehensive TPU-optimized features for production LLM serv ## Optimized Model List -The following models have been tested and optimized for TPU deployment: +SGLang-JAX supports the following model families. Use the linked TPU cookbooks for deployment instructions.
sglang/srt/platforms/device_mixin.pyPlatformEnum + DeviceMixin base classDeviceMixin base class and SRT platform identity types
sglang/srt/platforms/interface.pysglang/srt/platforms/__init__.py current_platform lazy singleton + discovery logic
sglang/multimodal_gen/runtime/platforms/interface.pyDiffusion Platform base class
sglang/multimodal_gen/runtime/platforms/__init__.pyDiffusion current_platform lazy singleton and built-in fallback order
sglang/multimodal_gen/runtime/platforms/plugins.pyDiffusion plugin registration, explicit hook-application phase, and hook registry
sglang/multimodal_gen/runtime/managers/worker_bootstrap.pyImport-neutral process specifications and spawn targets that initialize the backend before resolving runtime hooks
sglang/srt/plugins/__init__.py load_plugins() + load_plugins_by_group()
@@ -175,45 +175,53 @@ The following models have been tested and optimized for TPU deployment: - + - + - + - - + + - - + + - - + + - - + + - + - + + + + + + + + + - - + +
Model FamilyPerformance StatusCookbook
Qwen 3⭐ Recommended for productionQwen3
Qwen 3 MoE⭐ Best performanceQwen3 MoE
Qwen 2Needs improvementQwen2.5-VLQwen2.5-VL
Qwen 2 MoENeeds improvementLlama/LLaMALlama 3.1 / Llama 3.3 70B
Qwen 1.5Needs improvementDeepSeek V3 / R1DeepSeek V3 / DeepSeek R1
Llama/LLaMANeeds improvementKimi-LinearKimi-Linear
Grok-2Needs improvementGrok-2
Gemma 2Verified on TPUGemma 2
MiMo-V2-FlashMiMo-V2-Flash
MiMo-V2.5-ProMiMo-V2.5-Pro
Bailing MoENeeds improvementLing-2.6Ling-2.6
diff --git a/docs/docs/hardware-platforms/xpu.mdx b/docs/docs/hardware-platforms/xpu.mdx index a017751502cd..336e22a5ace8 100644 --- a/docs/docs/hardware-platforms/xpu.mdx +++ b/docs/docs/hardware-platforms/xpu.mdx @@ -7,39 +7,11 @@ The document addresses how to set up the [SGLang](https://github.com/sgl-project Specifically, SGLang is optimized for [Intel® Arc™ Pro B-Series Graphics](https://www.intel.com/content/www/us/en/ark/products/series/242616/intel-arc-pro-b-series-graphics.html) and [ Intel® Arc™ B-Series Graphics](https://www.intel.com/content/www/us/en/ark/products/series/240391/intel-arc-b-series-graphics.html). -## Optimized Model List - -A list of LLMs have been optimized on Intel GPU, and more are on the way: - - - - - - - - - - - - - - - - - - - - - - - - - - -
Model NameBF16
Llama-3.2-3B[meta-llama/Llama-3.2-3B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct)
Llama-3.1-8B[meta-llama/Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct)
Qwen2.5-1.5B[Qwen/Qwen2.5-1.5B](https://huggingface.co/Qwen/Qwen2.5-1.5B)
- -**Note:** The model identifiers listed in the table above -have been verified on [Intel® Arc™ B580 Graphics](https://www.intel.com/content/www/us/en/products/sku/241598/intel-arc-b580-graphics/specifications.html). +A number of popular LLMs are optimized and run efficiently on XPU, including the most notable open-source models like Llama series, Qwen series, and Diffusion model series like FLUX and Z-Image. +Please check the [SGLang Cookbook pages](https://docs.sglang.io/cookbook/intro) in which the support status and example commands can be found. + +Quantized MoE models are covered separately in +[MXFP4 MoE Quantization](#mxfp4-moe-quantization) below. ## Installation @@ -48,14 +20,24 @@ have been verified on [Intel® Arc™ B580 Graphics](https://www.intel.com/conte Currently SGLang XPU only supports installation from source. Please refer to ["Getting Started on Intel GPU"](https://docs.pytorch.org/docs/stable/notes/get_start_xpu.html) to install XPU dependency. ```bash Command +# Below commands shows Ubuntu/Debian OS as example, other OS requires similar command change +# add sudo if required to install below packages +# Install curl and protobuf compiler + +apt-get update && apt-get install -y curl protobuf-compiler + +# Install Rust (non-interactive) +curl --proto '=https' --tlsv1.2 --retry 3 --retry-delay 2 -sSf https://sh.rustup.rs | sh -s -- -y +# Make cargo available in the current shell +source "$HOME/.cargo/env" + # Create and activate a conda environment conda create -n sgl-xpu python=3.12 -y conda activate sgl-xpu # Set PyTorch XPU as primary pip install channel to avoid installing the larger CUDA-enabled version and prevent potential runtime issues. pip3 install torch==2.13.0+xpu torchvision==0.28.0+xpu torchaudio==2.11.0+xpu --index-url https://download.pytorch.org/whl/xpu -pip3 install xgrammar --no-deps # xgrammar will introduce CUDA-enabled triton which might conflict with XPU -pip3 install apache-tvm-ffi # xgrammar requires apache-tvm-ffi +pip3 install xgrammar==0.1.33 --no-deps # xgrammar will introduce CUDA-enabled triton which might conflict with XPU # Clone the SGLang code git clone https://github.com/sgl-project/sglang.git @@ -72,6 +54,20 @@ pip install -v . --extra-index-url https://download.pytorch.org/whl/xpu ### Install Using Docker +It is recommended to use Docker for setting up the SGLang environment. + +#### Pull from Docker Hub + +Pull the prebuilt docker image of SGLang package releases from `lmsysorg/sglang` repository. +The [XPU image tags](https://hub.docker.com/r/lmsysorg/sglang/tags?name=xpu) end with `xpu` suffix. +The image pulling command is like: + +```bash Command +docker pull lmsysorg/sglang:v0.5.20-xpu +``` + +#### Build from Dockerfile + [The SGLang XPU Dockerfile](https://github.com/sgl-project/sglang/blob/main/docker/xpu.Dockerfile) is provided to facilitate the installation. Replace `` below with your [HuggingFace access token](https://huggingface.co/docs/hub/en/security-tokens). @@ -108,7 +104,6 @@ Example command to launch SGLang serving: sglang serve \ --model-path \ --trust-remote-code \ - --disable-overlap-schedule \ --device xpu \ --host 0.0.0.0 \ --tp 2 \ # using multi GPUs @@ -116,6 +111,40 @@ sglang serve \ --page-size \ # intel_xpu attention backend supports [32, 64, 128] ``` +## MXFP4 MoE Quantization + +Native MXFP4 MoE checkpoints (OCP microscaling FP4: packed `e2m1` weights plus +per-32-element `ue8m0` block scales) run on Intel GPUs through the +`sgl-kernel-xpu` W4A16 grouped GEMM. The expert weights stay in the checkpoint's +packed layout end to end — there is no dequantization to BF16 — so GPT-OSS-20B +loads in roughly 13 GB rather than the ~42 GB a BF16 upcast would need, which is +what lets it fit on a single 24 GB card. + +The `mxfp4` method is registered automatically on `--device xpu`, which already +requires `sgl-kernel-xpu` — no extra flag is needed: + +```bash +sglang serve \ + --model-path openai/gpt-oss-20b \ + --device xpu \ + --attention-backend intel_xpu +``` + +**Tested models:** + +| Model | Notes | +|:---|:---| +| [openai/gpt-oss-20b](https://huggingface.co/openai/gpt-oss-20b) | 32 experts, top-k 4, hidden 2880, intermediate 2880; clamped-swiglu activation (`gemm1_alpha` 1.702, limit 7.0) with expert biases | + +**Requirements and limitations:** + +| Item | Status | +|---|---| +| GPU architecture | Xe2 / BMG only (Intel® Arc™ B-Series and Arc™ Pro B-Series) | +| Quantized layers | Fused MoE experts only; attention, router, embeddings and LM head stay BF16 (per the checkpoint's `modules_to_not_convert`) | +| Activations | BF16 (W4A16). Clamped swiglu (GPT-OSS) and plain SiLU are supported | +| Expert parallelism | Not yet validated on this path | + ## Benchmarking with Requests You can benchmark the performance via the `bench_serving` script. @@ -279,10 +308,63 @@ via `--cuda-graph-backend-prefill` or `--cuda-graph-config`. | Feature | Status | |---|---| -| Memory saver (`--enable-memory-saver`) | Not yet supported | | Two-batch overlap (`--enable-two-batch-overlap`) | Not yet supported | | Speculative decoding | Not yet implemented | +## Memory Saver (release/resume memory occupation) on Intel XPU [Experimental] + +SGLang can temporarily release most of the GPU memory it holds — model weights +and/or KV cache — and reclaim it later without restarting the process. This is +the same `release_memory_occupation` / `resume_memory_occupation` feature +available on CUDA, used for RL rollout/training hand-off and for freeing the +device between inference bursts. + +This is backed by the [`torch_memory_saver`](https://github.com/fzyzcjy/torch_memory_saver) +package — the same package used on CUDA — which gained an Intel XPU backend +built natively on Level Zero (keeping virtual addresses fixed while +releasing/re-committing physical pages via `zeVirtualMemUnmap` / +`zeVirtualMemMap`). + +**Install `torch_memory_saver`.** Unlike CUDA (prebuilt wheel), the XPU backend +is built from source against your local oneAPI + `torch+xpu` runtime (the `.so` +links `libsycl.so.`, which must match the installed `intel-sycl-rt`). +`TMS_PLATFORM=xpu` forces the XPU backend, and `--no-build-isolation` lets the +build import your installed `torch` so it can match the `libsycl` major to it: + +The published wheels are CUDA-only, so install from git and let it build. The ref +below is the `v0.0.10b2` release, pinned so the build is reproducible. + +```bash +source /opt/intel/oneapi/setvars.sh +TMS_PLATFORM=xpu pip install --no-build-isolation \ + git+https://github.com/fzyzcjy/torch_memory_saver.git@a5c99f11b18ebb8e9fda71a68812e476ae49e417 +``` + +**Use it** by launching with `--enable-memory-saver` (the XPU backend is +selected automatically); optionally add `--enable-weights-cpu-backup` to keep +weights in host RAM across a release: + +```bash +python -m sglang.launch_server --model-path Qwen/Qwen3-0.6B \ + --trust-remote-code --device xpu --enable-memory-saver +``` + +```bash +# Release GPU memory while idle, then reclaim it (server must be idle). +curl -X POST http://127.0.0.1:30000/release_memory_occupation +curl -X POST http://127.0.0.1:30000/resume_memory_occupation +``` + +The Python engine API (`engine.release_memory_occupation(tags=...)` / +`engine.resume_memory_occupation(tags=...)`) and the `weights` / `kv_cache` tags +behave the same as on CUDA. Pauseable CUDA-graph capture is not used on XPU, so +the `cuda_graph` tag is a no-op there. + +> **Verifying memory was freed:** neither `torch.xpu.memory_allocated()` nor +> `torch.xpu.mem_get_info()` drops when physical pages are released — the first is +> allocator accounting, and the second stays flat because torch keeps the freed block +> cached. Query actual device memory via sysman (`ZES_ENABLE_SYSMAN=1`) instead. + ## Prefill-Decode (P/D) Disaggregation on Intel XPU [Experimental] SGLang supports prefill-decode disaggregation on Intel XPU using the [NIXL](https://github.com/ai-dynamo/nixl) KV-transfer backend. diff --git a/docs/docs/references/environment_variables.mdx b/docs/docs/references/environment_variables.mdx index 7f69d703c1e7..f013ac4ce4f9 100644 --- a/docs/docs/references/environment_variables.mdx +++ b/docs/docs/references/environment_variables.mdx @@ -83,6 +83,36 @@ SGLang supports various environment variables that can be used to configure its Cache directory for model weights and other data. Also the default root for compiled-kernel caches: Triton, Inductor, FlashInfer, the CUDA driver and DeepGEMM are pointed under it unless their own env vars (`TRITON_CACHE_DIR`, `TORCHINDUCTOR_CACHE_DIR`, `FLASHINFER_WORKSPACE_BASE`, `CUDA_CACHE_PATH`, `SGLANG_DG_CACHE_DIR`) are set explicitly ~/.cache/sglang + + SGLANG_QWEN4_PLE_FILE_DIR + Default directory for the file-backed Qwen4-Exp PLE table (`--ple-offload-backend file`); one subdirectory per model path is created under it + {SGLANG_CACHE_DIR}/ple + + + SGLANG_QWEN4_PLE_FILE_PREFETCH + Hint the page cache (`posix_fadvise(WILLNEED)`) with the rows a prefill-sized PLE gather is about to read from the file-backed table. Set to `0` to disable + 1 + + + SGLANG_QWEN4_PLE_FILE_SKIP_DEVICE_CHECK + Skip the load-time check that the device reads pageable host memory through the host page tables (`cudaDevAttrPageableMemoryAccessUsesHostPageTables`) before using the file-backed PLE table; only for devices that cannot be queried + 0 + + + SGLANG_QWEN4_PLE_FILE_RSS_BUDGET_GB + Cap (GiB) on the resident set of the file-backed PLE mapping. A row fault maps a whole page-cache folio, so the mapping otherwise creeps towards the full table and consumes the free memory that sizes the KV pool; over the cap the pages are dropped with `MADV_DONTNEED` and stay in the page cache. Set to `0` to disable + 8.0 + + + SGLANG_QWEN4_PLE_FILE_RSS_INTERVAL_S + How often the file-backed PLE mapping's resident set is compared against `SGLANG_QWEN4_PLE_FILE_RSS_BUDGET_GB` + 30.0 + + + SGLANG_CUTE_AOT_CACHE_DIR + Trusted directory for persistent CuTe DSL AOT objects shared across process restarts. Artifacts are namespaced by source, runtime ABI, host platform, and target GPU architecture. Set to an empty string to keep compilation process-local. Only use a directory writable by trusted users because SGLang loads cached object files into the process. + {`{SGLANG_CACHE_DIR}`}/cute_aot + SGLANG_PREFETCH_BLOCK_SIZE_MB Block size (in MB) for sequential checkpoint prefetch reads that warm the OS page cache before workers load weights via mmap @@ -321,6 +351,21 @@ SGLang supports various environment variables that can be used to configure its The maximum number of dispatched tokens on each GPU for --moe-a2a-backend=flashinfer `"1024"` + + SGLANG_FLASHINFER_MEGAMOE_COMBINE_DTYPE + FlashInfer NVFP4 MegaMOE cross-rank combine wire format. Supported values are bf16, mxfp8, and nvfp4. Quantized formats are incompatible with SGLANG_FLASHINFER_MEGAMOE_IN_KERNEL_FC2_REDUCE=1. + "bf16" + + + SGLANG_FLASHINFER_MEGAMOE_MAX_TOKENS_PER_RANK + Per-rank FlashInfer MegaMOE symmetric-workspace token capacity. A value of 0 derives the capacity from the runtime token limits. + 0 + + + SGLANG_FLASHINFER_MEGAMOE_IN_KERNEL_FC2_REDUCE + Use the CuTe DSL MegaMOE in-kernel FC2 reduction. This can reduce workspace size and improve large-batch performance, but BF16 atomic accumulation is nondeterministic. It is incompatible with quantized combine dtypes. + "false" + `SGLANG_DEEPEP_LL_COMBINE_SEND_NUM_SMS` Number of SMs used for DeepEP combine when single batch overlap is enabled @@ -367,12 +412,12 @@ SGLang supports various environment variables that can be used to configure its true - SGLANG_ENABLE_LOGITS_PROCESSER_CHUNK + SGLANG_ENABLE_LOGPROB_CHUNK Process logits in chunks to reduce peak memory. - false + true - SGLANG_LOGITS_PROCESSER_CHUNK_SIZE + SGLANG_LOGPROB_CHUNK_SIZE Chunk size (in tokens) used when logits-processor chunking is enabled. 2048 @@ -446,9 +491,9 @@ SGLang supports various environment variables that can be used to configure its "auto" - SGLANG_MORI_FP8_COMB - Use FP8 for combine - "false" + SGLANG_MORI_COMBINE_DTYPE + Combine dtype: auto, bf16, fp8, fp4, or fp8_direct_cast + "auto" MORI_DISABLE_AUTO_XGMI @@ -518,7 +563,7 @@ SGLang supports various environment variables that can be used to configure its SGLANG_DSA_FUSE_TOPK - Fuse the operation of picking topk logits and picking topk indices from page table. SGLANG_NSA_FUSE_TOPK is a deprecated alias. + Fuse the operation of picking topk logits and picking topk indices from page table. true @@ -533,7 +578,7 @@ SGLang supports various environment variables that can be used to configure its SGLANG_DSA_PREFILL_DENSE_ATTN_KV_LEN_THRESHOLD - When the maximum kv len in current prefill batch exceeds this value, the sparse mla kernel will be applied, else it falls back to dense MHA implementation. Default to the index topk of model (2048 for DeepSeek V3.2). SGLANG_NSA_PREFILL_DENSE_ATTN_KV_LEN_THRESHOLD is a deprecated alias. + When the maximum kv len in current prefill batch exceeds this value, the sparse mla kernel will be applied, else it falls back to dense MHA implementation. Default to the index topk of model (2048 for DeepSeek V3.2). 2048 @@ -563,7 +608,7 @@ SGLang supports various environment variables that can be used to configure its SGLANG_DSA_HIP_DISABLE_PRESHUFFLE - Disable weight pre-shuffle on the HIP DSA path. SGLANG_NSA_HIP_DISABLE_PRESHUFFLE is a deprecated alias. + Disable weight pre-shuffle on the HIP DSA path. false @@ -710,6 +755,11 @@ SGLang supports various environment variables that can be used to configure its Default reasoning_effort for the DeepSeek V4 chat encoder when a request does not set it. The preview profile accepts high and max; the official profile accepts low, high, and max. The profile is detected from the bundled encoder. Override it with --json-model-override-args '{"dsv4_reasoning_effort_profile":"official"}'. "" + + SGLANG_DSV41_REASONING_EFFORT + Default reasoning_effort for the DeepSeek-V4.1 chat encoder when a request does not set it: one of low, high, xhigh, max, or an integer budget in [1, 100]. Unset means the encoder default high. + "" + SGLANG_DSV4_USE_BF16_KV_QUANT_SOURCE For DeepSeek V4, quantize the SWA FP8 KV cache from BF16-rounded values instead of FP32 registers. This matches trainer-side QAT and the DSA prefill-CP path, at the cost of an extra BF16 KV materialization and separate cache-store kernels. @@ -759,10 +809,15 @@ SGLang supports various environment variables that can be used to configure its Enable FlashInfer TRT-LLM or CuTe DSL v2 (no A2A or FlashInfer A2A) per-token FP32 activation scaling for serialized modelopt_fp4 checkpoints; checkpoint activation scales are treated as 1 false + + SGLANG_FLASHINFER_CUTEDSL_NVFP4_W4A16 + Use BF16 activations and outputs with FlashInfer CuTe DSL NVFP4 weights on NVIDIA SM100-family GPUs (SM100/SM103). Serialized dense linear layers use this mode with --fp4-gemm-backend flashinfer_cutedsl. MoE layers use it with --moe-runner-backend flashinfer_cutedsl; the MoE path supports online weight quantization and serialized ModelOpt NVFP4 weights with either no A2A or FlashInfer A2A, and honors SGLANG_FLASHINFER_MOE_FUSED_FINALIZE. + false + SGLANG_FLASHINFER_MOE_FUSED_FINALIZE - Use FlashInfer's fused atomic CUTLASS and CuTe DSL MoE finalize for best performance. Deterministic inference overrides this to false. - true + Use FlashInfer's fused atomic CUTLASS and CuTe DSL MoE finalize. Disabled by default for better numerical accuracy. Set to 1 to opt in for more aggressive performance. Deterministic inference overrides this to false. + false FLASHINFER_NVFP4_4OVER6 @@ -1359,16 +1414,6 @@ SGLang supports various environment variables that can be used to configure its Multiplier applied to the sliding-window-attention eviction interval. 1.0 - - SGLANG_ENABLE_UNIFIED_RADIX_TREE - Use the unified radix-tree cache implementation. - false - - - SGLANG_EXPERIMENTAL_CPP_RADIX_TREE - Use the experimental C++ radix-tree implementation. - false - SGLANG_RADIX_FORCE_MISS Force radix-cache misses (debugging/benchmarking). @@ -1743,11 +1788,6 @@ SGLang supports various environment variables that can be used to configure its Output directory for the expert-distribution recorder. /tmp - - SGLANG_ENABLE_EPLB_BALANCEDNESS_METRIC - Removed. Use --expert-balancedness-report-mode=prometheus to emit the EPLB balancedness metric. - false - SGLANG_LOG_EXPERT_LOCATION_METADATA Log expert-location metadata. @@ -1797,6 +1837,11 @@ SGLang supports various environment variables that can be used to configure its Use AITER FP8 per-token quantization. false + + SGLANG_AITER_HONOR_EXPLICIT_MEM_FRACTION + Use an explicitly passed --mem-fraction-static as-is on AITER with context length above 8192, instead of scaling it by 0.85 to reserve non-static attention workspace. Skipping the reserve can OOM long-context serving; set it only when the scaled fraction is too small to hold the model weights. + false + SGLANG_USE_AITER_MOE_GU_ITLV Select the AITER MoE gate/up tile layout: true interleaves, false uses the separated layout. @@ -1822,6 +1867,16 @@ SGLang supports various environment variables that can be used to configure its Disable linear-layer quantization on ROCm. false + + SGLANG_ROCM_K3_FUSE_KDA_INPROJ + Kimi-K3 on ROCm: fold the KDA [f_a|b] tail into the wide [q,k,v,g] projection so the whole input projection is one GEMM. Applies to unquantized weights only; falls back to the split projection otherwise. + true + + + SGLANG_ROCM_K3_FUSE_KDA_INPROJ_MAX_TOKENS + Token count above which SGLANG_ROCM_K3_FUSE_KDA_INPROJ stops applying and the split projection runs instead. The merged shape is only faster while the projection is bandwidth bound. + 256 + diff --git a/docs/docs/references/production_metrics.mdx b/docs/docs/references/production_metrics.mdx index f15f51c9f67c..ed5f7748034b 100644 --- a/docs/docs/references/production_metrics.mdx +++ b/docs/docs/references/production_metrics.mdx @@ -7,6 +7,8 @@ SGLang exposes the following metrics via Prometheus. You can enable it by adding An example of the monitoring dashboard is available in [examples/monitoring/grafana.json](https://github.com/sgl-project/sglang/blob/main/examples/monitoring/grafana/dashboards/json/sglang-dashboard.json). +## Language model metrics + Here is an example of the metrics: ```text Output @@ -134,6 +136,12 @@ sglang:spec_num_steps{model_name="meta-llama/Llama-3.1-8B-Instruct"} 3.0 sglang:spec_num_draft_tokens{model_name="meta-llama/Llama-3.1-8B-Instruct"} 4.0 ``` +## Diffusion metrics + +SGLang Diffusion exposes request, queue, stage and LoRA metrics with +`--enable-metrics`. See [Diffusion production metrics](/docs/sglang-diffusion/production_metrics) +for the metric reference, counting semantics and disaggregated scraping setup. + ## Setup Guide This section describes how to set up the monitoring stack (Prometheus + Grafana) provided in the `examples/monitoring` directory. diff --git a/docs/docs/sglang-diffusion/api/cli.mdx b/docs/docs/sglang-diffusion/api/cli.mdx index 95b4287470f8..faf961c866aa 100644 --- a/docs/docs/sglang-diffusion/api/cli.mdx +++ b/docs/docs/sglang-diffusion/api/cli.mdx @@ -64,7 +64,7 @@ sglang serve \ --port 30010 ``` -For request and response examples, see [OpenAI-Compatible API](./openai_api). +For request and response examples, see [OpenAI-Compatible API](/docs/sglang-diffusion/api/openai_api). Use `sglang generate --help` and `sglang serve --help` for the full argument list. The CLI help output is the source of truth for exhaustive flags. @@ -76,8 +76,11 @@ Use `sglang generate --help` and `sglang serve --help` for the full argument lis - `--model-path {MODEL}`: model path or Hugging Face model ID - `--served-model-name {NAME}`: stable model name exposed by serving APIs. Defaults to `--model-id` when set, otherwise `--model-path`. -- `--model-variant {NAME}`: semantic checkpoint variant to load when one model repository contains multiple weight partitions. The pipeline maps this stable name to the repository layout before loading; for example, MiniMax-H3 accepts `fl2va` and `ref2va`. This is a server/load-time choice, unlike a request's `task`. +- `--model-variant {NAME}`: semantic checkpoint variant to load when one model repository contains multiple weight partitions. The pipeline maps this stable name to the repository layout before loading; for example, MiniMax-H3 accepts `fl2va` and `ref2va`, or `hybrid` with explicit merged transformer weights to serve all three tasks (see the [H3 cookbook](/cookbook/diffusion/MiniMax/MiniMax-H3#singularity-hybrid-weights)). This is a server/load-time choice, unlike a request's `task`. - `--minimax-h3-adaln-cache-path {FILE}`: advanced MiniMax-H3-only inference cache. It replaces the checkpoint's AdaLN projection weights with precomputed outputs and only accepts requests whose exact FP32 timestep plan is included in the cache. It requires unquantized weights and the matching model variant. +- `--minimax-h3-adaln-online {true,false}`: rebuild MiniMax-H3 AdaLN outputs from the checkpoint on demand instead of keeping the 24.2 GiB of `adaln_proj` weights resident. Works with any step count or schedule; requires the unquantized native-layout checkpoint. Built plans live in a GPU slab with per-plan LRU eviction and, by default, a pinned-host cache so previously seen schedules swap back in over PCIe instead of re-reading the checkpoint. +- `--minimax-h3-adaln-plan-width {N}`: widest timestep plan the online slab is sized for (default 4 covers every task; t2va needs 2, fl2va 3). +- `--minimax-h3-adaln-host-cache-gb {GB}`: pinned host memory per rank caching built AdaLN plans (default 8; 0 disables). A 50-step schedule needs about 0.9 (t2va) / 1.33 (fl2va) / 1.77 (ref2va) GB; over-capacity plan sets simply recompute. Expert escape hatches (GPU slot count, experimental fp32 rebuild) are the `SGLANG_DIFFUSION_MINIMAX_H3_ADALN_*` environment variables. - `--model-subfolder {PATH}`: advanced direct override for a component subfolder inside the model repository. Prefer `--model-variant` when the pipeline exposes semantic routing. If both are supplied, they must resolve to the same weight partition. - `--lora-path {PATH}` and `--lora-nickname {NAME}`: load a LoRA adapter from a local path, Hugging Face repo/subfolder, or exact Hub file URL - `--lora-weight-name {FILE}`: select one adapter file from a repository that contains multiple LoRA revisions. The Hub download is filtered to that file plus JSON metadata, so unused weights are not downloaded. @@ -86,27 +89,33 @@ Use `sglang generate --help` and `sglang serve --help` for the full argument lis - `--lora-merge-mode {auto|merge|dynamic}`: choose how LoRA is applied. `auto` statically merges regular weights and uses dynamic LoRA for FSDP-sharded weights to avoid full-gather peaks. - `--num-gpus {N}`: number of GPUs to use - `--performance-mode {manual|auto|speed|memory}` / `--mode`: preset for latency/throughput and memory defaults. `auto` is the default and dispatches residency from selected-GPU headroom and workload type: image DiTs stay resident above the 45 GiB threshold, while video DiT placement remains model-specific. It uses FSDP only for validated DiT-offload replacement paths. `speed` keeps `torch.compile` disabled unless a model-specific deployment config opts in after validation; pass `--enable-torch-compile true` to enable it explicitly. Use `manual` to keep performance-related server args under explicit user control. Explicit offload, FSDP, and parallelism flags take precedence in all modes. -- `--direct-gpu-weight-loading {true|false}`: opt into direct GPU loading for an unquantized, GPU-resident, TP=1 DiT by materializing its complete checkpoint state dict on GPU. Startup impact is model-dependent, so benchmark the target model before deployment. Disabled by default because checkpoint and model weights coexist temporarily, substantially increasing peak GPU memory. It is incompatible with DiT CPU/layerwise offload and FSDP. +- `--direct-gpu-weight-loading {true|false}`: opt into direct GPU loading for an unquantized, GPU-resident, TP=1 DiT by materializing its complete checkpoint state dict on GPU. Compatible tensors become parameter storage without another GPU copy, while tensors requiring transforms can still need temporary allocations. It is disabled by default; startup time and peak GPU memory remain model-dependent, so benchmark the target model before deployment. It is incompatible with DiT CPU/layerwise offload and FSDP. - `--tp-size {N}`: tensor parallelism size. Depending on the pipeline, it can shard the DiT, one or more encoders, or both. - `--sp-degree {N}`: sequence parallelism size - `--dp-size {N}` (alias `--data-parallel-size`): number of data-parallel replicas. Each replica is a full copy of the engine on `num_gpus / N` GPUs with its own ingress; generation requests round-robin across replicas, realtime sessions stick to the replica holding their state, and control operations (weights, LoRA, memory occupation, shutdown) apply to every replica. Combines with the other parallelism axes (`num_gpus = dp × cfg × tp × sp`); monolithic serving only. - `--ulysses-degree {N}` and `--ring-degree {N}`: USP parallelism controls - `--kv-gather-degree {N}`: sequence-parallel degree that splits rows inside attention and exchanges with one K/V all-gather (queries stay local) instead of Ulysses all-to-all. Non-causal attention only; does not compose with `--ulysses-degree`/`--ring-degree` yet. When no SP degree is set explicitly, `sp_degree=2` defaults to `kv_gather_degree=2` (its measured-win zone) and higher degrees default to Ulysses; under that auto assignment, attention calls the gather path cannot take fall back to the Ulysses exchange, while an explicit degree fails instead of degrading. -- `--enable-cfg-parallel {true|false}`: enable or explicitly disable CFG parallelism -- `--encoder-parallel {auto|fold|dp|replicate}`: how text/image encoders use the GPUs in each DiT replica. `auto` TP-folds an encoder wide enough to benefit, selects batch DP when it can engage, and otherwise keeps the existing encoder TP layout; `fold` shards across the full replica whenever dimensions allow; `dp` splits a batched encode across encoder copies and composes with encoder TP; `replicate` disables folding and batch DP. Encoder collectives never cross `--dp-size` replicas. See [Encoder Parallelism](/docs/sglang-diffusion/encoder_parallel). +- `--enable-cfg-parallel {true|false}`: enable or explicitly disable CFG parallelism. Requests that disable CFG remain valid on a CFG-parallel server, but extra CFG ranks redundantly recompute the single active branch. +- `--encoder-parallel {auto|fold|dp|replicate}`: how native encoders use the GPUs in each DiT replica. `auto` TP-folds a native text/image encoder wide enough to benefit, selects batch DP for an explicitly supported native text encoder when it can engage, and otherwise keeps the existing encoder TP layout; `fold` shards native text/image encoders across the full replica whenever dimensions allow; `dp` splits a batched encode across supported native text encoder copies and composes with encoder TP; `replicate` disables folding and batch DP. Encoder collectives never cross `--dp-size` replicas. See [Encoder Parallelism](/docs/sglang-diffusion/encoder_parallel). - `--warmup-mode {off|request|server}`: control startup warmup for `sglang serve`; `off` skips warmup, `request` primes the request path, and `server` runs a full synthetic server warmup before serving traffic - `--enable-torch-compile {true|false}`: compile native diffusion hot paths. When no warmup mode is configured, this also enables server warmup so first real requests do not pay compile latency. - `--offload-during-compile {true|false}`: when compile warmup is active, temporarily layerwise-offload DiT weights and move resident non-DiT components off-device so `max-autotune` fits on tighter-memory GPUs; the configured serving residency is restored before real traffic. Skipped under existing layerwise offload, Cache-DiT, or FSDP. -- `--enable-breakable-cuda-graph {true|false}`: capture supported DiT forwards as breakable CUDA graph segments to reduce launch overhead. Requires `--warmup-resolutions` for every served resolution because each resolution is captured separately. A `quality=high` request is rejected when it would mount request-scoped DiT fusions that were not present during lossless graph capture; VAE-only high-quality paths remain compatible. +- `--enable-breakable-cuda-graph {true|false}`: capture supported DiT forwards as breakable CUDA graph segments to reduce launch overhead. Requires `--warmup-resolutions` for every served resolution because each resolution is captured separately. An `extra-high` or `high` request is rejected when it would mount request-scoped DiT fusions that were not present during lossless graph capture; VAE-only request-gated paths remain compatible. - `--bcg-text-buckets {N...}`: prompt-length padding buckets for breakable CUDA graph capture/replay reuse. - `--attention-backend {BACKEND}`: attention backend for native SGLang and diffusers pipelines +- `--enable-attention-backend-autotune {true|false}`: for SGLang-native pipelines, benchmark compatible attention backends on each layer's first sufficiently large input and keep a backend only when it is both numerically compatible and measurably faster. Disabled by default and currently validated on SM90 and SM12x. Explicit `--attention-backend`, component overrides, and model-required backends are never replaced. - `--component-attention-backends {MAP}`: per-component attention backend overrides, for example `text_encoder=torch_sdpa,transformer=fa` - `--attention-backend-config {CONFIG}`: attention backend configuration -- `--srt-encoder-url {HTTPADDRESS}`: address of SGLang srt server with AR model for GLM-Image like models. See [Models with AR Stage](../models_with_ar). +- `--srt-encoder-url {HTTPADDRESS}`: address of SGLang srt server with AR model for GLM-Image like models. See [Models with AR Stage](/docs/sglang-diffusion/models_with_ar). - `--srt-encoder-timeout {SECONDS}`: Timeout in seconds for HTTP requests to the SGLang encoder server - `--srt-encoder-connection-timeout {SECONDS}`: TCP connection timeout in seconds for SGLang encoder server - `--scheduler-rpc-timeout {SECONDS}`: optional end-to-end deadline for an internal scheduler RPC, including scheduler queue time. It is unset by default so valid long-running and queued video jobs are not failed by the transport layer. Set it only when the deployment requires a bounded request deadline; caller cancellation and server shutdown remain effective without it. -- `--pe-server-url {HTTPADDRESS}`: url of SGLang server hosting the PE model (e.g., for ERNIE-Image). See [Models with Prompt Enhancement](../models_with_pe). +- `--enable-trace`: export OpenTelemetry traces for the diffusion scheduler and worker stages. +- `--otlp-traces-endpoint {HOST:PORT}`: OTLP collector endpoint used with `--enable-trace` (default: `localhost:4317`). +- `--otlp-service-name {NAME}`: `service.name` attached to exported traces. Resolution order is this flag, `OTEL_SERVICE_NAME`, then `sglang-diffusion`. +- `--enable-metrics`: expose Prometheus metrics at `/metrics` (default: disabled). Includes request counts, queue time, host-side stage timing and LoRA state, separated by role and DP replica. See [Production metrics](/docs/sglang-diffusion/production_metrics) for metric semantics and disaggregated scraping. +- `--pe-server-url {HTTPADDRESS}`: url of SGLang server hosting the PE model (e.g., for ERNIE-Image). See [Models with Prompt Enhancement](/docs/sglang-diffusion/models_with_pe). +- `--prompt-enhancer-config {JSON_FILE}`: configure an independent SRT LLM/VLM for opt-in HTTP prompt rewriting. Requests use `enhance_prompt=true`; no enhancer runs by default. See [Prompt enhancement](/docs/sglang-diffusion/models_with_pe). ### Sampling and output @@ -114,11 +123,12 @@ Use `sglang generate --help` and `sglang serve --help` for the full argument lis - `--image-path {PATH} [{PATH} ...]`: input image(s) for image-to-video or image-to-image generation - `--num-inference-steps {STEPS}` and `--seed {SEED}` - `--num-outputs-per-prompt {N}` / `--num-outputs {N}`: generate multiple outputs for each prompt. A scalar seed expands as `seed + output_index`. -- `--quality {lossless,high}`: request-level quality. `lossless` (default) keeps the exact reference path, bit-exact against the reference implementation; `high` opts into the model-owned validated accelerated path, whose quality stays guaranteed but is not bit-exact. Support and validated deployment constraints are model-specific. +- `--quality {lossless,extra-high,high}`: cumulative request-level optimization tier. `lossless` (default) keeps the selected deployment's reference path and all unconditional bit-exact replacements. `extra-high` adds only request-gated DiT/VAE kernel fusions; the tier does not itself enable sparse, caching, or other approximate paths. `high` includes the complete `extra-high` set and may also enable model-owned approximate optimizations. Separately configured quantization, attention, or caching options still apply. Support and validation constraints are model-specific. +- `skip_softmax_params` (online request only): explicitly enables lossy BLASST/Skip-Softmax attention for compatible FA self-attention layers. See [Attention Backends](../attention_backends#request-scoped-skip-softmax). - `--height {HEIGHT}`, `--width {WIDTH}`, `--num-frames {N}`, `--fps {FPS}` - `--output-path {PATH}`, `--output-file-name {NAME}`, `--save-output`, `--return-frames` -For frame interpolation and upscaling, see [Post-Processing](./post_processing). +For frame interpolation and upscaling, see [Post-Processing](/docs/sglang-diffusion/api/post_processing). ### Quantization @@ -131,6 +141,8 @@ pipeline's registered module name: | --- | --- | --- | --- | | Replace a component | `--component-paths. {MODEL}` | `---path {MODEL}` | Load the replacement component's configuration and weights | | Replace only its weights | `--component-weights-paths. {WEIGHTS}` | `---weights-path {WEIGHTS}` | Retain the base component configuration and replace its weights | +| Select component precision | `--component-precisions. {DTYPE}` | — | Use a loader-supported exact parameter and execution dtype | +| Direct-load an eligible component | `--component-direct-gpu-weight-loading.` | None | Use that component's audited direct-GPU loader; it must stay resident | | Quantize an unquantized component online | `--component-quantizations. {METHOD}` | `---quantization {METHOD}` | Apply a method supported by that component's native loader | | Keep selected component layers unquantized | `--component-quantization-ignored-layers. {PATTERN...}` | None | Pass component-local ignored-layer patterns to its online quantizer | @@ -160,12 +172,25 @@ The same contract applies to every weighted component: path routing is generic, while quantized materialization is capability-based. Native auxiliary loaders whose current materializer expects plain state dicts reject unsupported quantization metadata before model construction. See -[Quantized Component Repositories](../quantization#quantized-component-repositories) +[Quantized Component Repositories](/docs/sglang-diffusion/quantization#quantized-component-repositories) for the current component matrix. A model cookbook is the source of truth for published, model-specific checkpoint examples; for example, all H3 sources and their exact overlays are kept in one [MiniMax-H3 compatibility table](/cookbook/diffusion/MiniMax/MiniMax-H3#checkpoint-and-adapter-formats). +Exact precision overrides are capability-based. Native text and image encoders, +standard VAE components, and native plain-state components support them; +other component loaders reject the option instead of accepting a dtype that +their execution stage would not honor. + +Direct-GPU loading is also capability-based. The existing +`--direct-gpu-weight-loading` remains the primary DiT path. The component form +currently supports standard native `vae` and `video_vae` state dicts on CUDA: +it streams each safetensors tensor directly to the resident +module, rather than materializing a complete CPU state dict. It rejects custom +Diffusers `auto_map` classes, quantized checkpoints, tied state entries, and +any component-offload or layerwise-offload placement. + Component overrides accept a local component directory, a standalone Hub repository, or a Hub component subfolder written as `owner/repo/subfolder`. For transformer and native encoder loaders, an explicit weight filename keeps @@ -176,8 +201,8 @@ For supported realtime causal video models, `--kv-cache-quant {off|int4|int2}` compresses completed KV-cache chunks independently of transformer weight quantization. It is lossy and disabled by default. -See [Realtime and Causal Video Models](../realtime_models) for the runtime and -model scope, and [Quantization](../quantization) for supported quantization +See [Realtime and Causal Video Models](/docs/sglang-diffusion/realtime_models) for the runtime and +model scope, and [Quantization](/docs/sglang-diffusion/quantization) for supported quantization families and examples. ### Request logging @@ -248,7 +273,7 @@ sglang generate \ HTTP server-only arguments are ignored by `sglang generate`. -For supported native pipelines, set `SGLANG_CACHE_DIT_ENABLED=true` to enable Cache-DiT. For the diffusers backend, use `--backend diffusers --cache-dit-config ...`. See [Cache-DiT](../cache_dit). +For supported native pipelines, set `SGLANG_CACHE_DIT_ENABLED=true` to enable Cache-DiT. It can run with DiT layerwise offload; it cannot run with FSDP. For the diffusers backend, use `--backend diffusers --cache-dit-config ...`. See [Cache-DiT](/docs/sglang-diffusion/cache_dit). For supported image pipelines, breakable CUDA graph can be enabled with `--enable-breakable-cuda-graph`, but you must declare every served resolution in `--warmup-resolutions` so warmup captures matching graph signatures. @@ -294,13 +319,42 @@ The available modes are: - `resident`: keep the complete component on the accelerator. - `component-offload`: keep the complete component on CPU between uses, moving it to the accelerator before each declared use and back to CPU afterward. +- `snapshot-offload`: retain CPU parameter storage while a complete component runs on GPU. At the end of a use, restore the CPU parameters instead of copying GPU weights back. Mutable buffers still move back to CPU. - `layerwise-offload`: keep component weights on CPU and stream its declared layers during execution. Selectors match exact loaded component keys from `model_index.json`, including names such as `transformer_2`, `audio_vae`, and `connectors`. The group selectors `dit`, `text_encoder`, `image_encoder`, and `vae` are also available, together with `all`. An exact key overrides a matching group, and a group overrides `all`. Components without a matching canonical selector retain their explicit legacy setting or automatic/model default. The existing `--dit-cpu-offload`, `--text-encoder-cpu-offload`, `--image-encoder-cpu-offload`, `--vae-cpu-offload`, and `--cpu-offload-components` options remain supported. New and legacy options may be mixed: `--component-residency` wins only for components it matches, while unmatched legacy settings remain effective. Legacy layerwise selectors take precedence over legacy component-offload selectors for the same component. Explicit `--dit-layerwise-offload false` makes the DiT resident unless another explicit DiT selector, such as `--dit-cpu-offload true` or `--component-residency dit=component-offload`, selects a different mode. -Layerwise selection is strict. A native weighted component selected for `layerwise-offload` must declare its layer structure; otherwise startup fails with the unsupported component name instead of silently changing modes. FSDP applies only to resident components. The Diffusers backend supports only pipeline-wide `all=resident` and `all=component-offload`. +Layerwise selection is strict. A native weighted component selected for `layerwise-offload` must declare its layer structure; otherwise startup fails with the unsupported component name instead of silently changing modes. Explicit non-resident placement also requires a request-time component-use declaration, so it cannot silently select a module that the pipeline does not manage. FSDP applies only to resident components. The Diffusers backend supports only pipeline-wide `all=resident` and `all=component-offload`. + +#### Snapshot offload + +On NVIDIA CUDA GPUs with separate host and device memory, explicitly select +`--component-residency vae=snapshot-offload` to avoid repeated weight D2H transfers. +Existing defaults and `component-offload` behavior are unchanged. + +Snapshot offload retains a full host copy during GPU execution. With +`--pin-cpu-memory` enabled (the default), it shares a per-worker host pin budget +with layerwise offload. Layerwise initialization claims its allowance first; +snapshot weights use the remainder. Shared parameter storage is pinned once and +reused across requests; its allowance is returned when the storage is released. +Pinning also checks current host headroom, including the cgroup limit. + +Weights that do not fit the budget, or have pinning disabled, retain their +existing CPU storage. Checkpoint mmap stays file-backed unless pinning, dtype +conversion, or weight mutation materializes a copy. Pageable H2D can be slower, +and pinning temporarily needs space for both the original and pinned storage. +This budget limits new pinned weight allocations, not total process RAM or the +CUDA host allocator's cache. Benchmark repeated requests with the actual host +storage and workload; avoiding D2H does not guarantee an end-to-end speedup. + +The whole component must still fit on GPU. Unlike layerwise offload with all layers +retained across denoising steps, snapshot offload does not require a layer-structure +declaration and also covers parameters outside transformer blocks. Weight updates, +LoRA merge/unmerge, and sleep restore the CPU weights before mutation or release. +FSDP-managed components, shared-memory GPUs, and snapshot-offloaded DiTs with +breakable CUDA graphs are not supported. ### Layerwise Offload Tuning @@ -316,23 +370,27 @@ sglang generate \ Values passed to the compatibility option `--layerwise-offload-components` must match loaded component keys, such as `transformer`, `text_encoder`, `image_encoder`, `vae`, `condition_image_encoder`, `spatial_upsampler`, or `vocoder`. Its `default` group selects text encoders, image encoders, and VAEs. Use `all` to select every layerwise-offloadable component. -Layerwise tuning options such as `--dit-offload-prefetch-size`, `--dit-layerwise-resident-layers`, and `--dit-layerwise-residency-policy` continue to control the streamed layer working set. Prefer the smallest component set that solves the memory issue because layerwise offload can increase latency. +Layerwise tuning options such as `--dit-offload-prefetch-size`, `--dit-layerwise-resident-layers`, `--dit-layerwise-residency-policy`, and `--dit-layerwise-residency-lifetime` continue to control the streamed layer working set. Prefer the smallest component set that solves the memory issue because layerwise offload can increase latency. DiT layerwise offload can run with Cache-DiT: skipped blocks are not streamed, and the first layer after a skip may sync-load. Cache-DiT remains incompatible with FSDP. -Those three set the default for every streamed component. To give one component its own values, use the `component=value` forms, which also accept JSON: +Those four set the default for every streamed component. To give one component its own values, use the `component=value` forms, which also accept JSON: ```bash sglang serve --model-path \ --layerwise-offload-components dit,text_encoder \ --layerwise-prefetch-size text_encoder=2 \ - --layerwise-resident-layers text_encoder=4 \ - --layerwise-residency-policy text_encoder=strided + --layerwise-resident-layers transformer=4 \ + --layerwise-residency-policy transformer=strided \ + --layerwise-residency-lifetime transformer=permanent ``` +The `component=value` maps take the loaded component's name — `transformer`, `text_encoder`, `video_vae` — not a group alias: `--layerwise-offload-components` understands `dit`, but `--layerwise-resident-layers dit=4` matches no component and is silently ignored. + - `--layerwise-prefetch-size`: how many layers to fetch ahead. Fractional values are a share of the stack, `>= 1` an absolute count. Deeper prefetch overlaps more of the transfer with compute, at the cost of staging buffers. -- `--layerwise-resident-layers`: how many layers stay on the GPU instead of being streamed. Resident layers are transferred once at startup, so they are removed from every pass. Fractional values are a share of the stack. +- `--layerwise-resident-layers`: how many layers stay on the GPU instead of being streamed. Fractional values are a share of the stack. Resident layers pay for a component that runs its layers many times per request — a DiT across the denoise steps, a video VAE across the latent chunks — and do nothing for one that runs them once per request, such as a text encoder. - `--layerwise-residency-policy`: `leading` keeps the first layers, `strided` spreads them across the stack so the transfers do not arrive as one burst. +- `--layerwise-residency-lifetime`: how long the resident layers stay. `forward` (default) places them when the component starts running its layers for a request and releases them when it finishes, so they are transferred once per request and keep a copy in host memory. `permanent` places them at load time, keeps no host copy, and never releases them: one transfer for the life of the server. -A component without an entry keeps the group default, so adding these options changes nothing until one is set. Both knobs trade VRAM for transfer, and the return differs by component: a DiT earns it back once per denoising step, a text encoder or VAE once per request. Measure before raising either. +A component without an entry keeps the group default, so adding these options changes nothing until one is set. All of these trade VRAM for transfer. The lifetime decides *whose* VRAM: with `forward`, only the component that is running holds its resident layers, so the peak is the largest single component's set; with `permanent`, every component's resident layers are on the GPU at once, so the peak is their sum. Pick `permanent` when that sum fits alongside the activations — it also frees the pinned host memory the copies would have held — and `forward` when it does not. Measure before raising any of them. ## Serve @@ -379,7 +437,7 @@ export SGLANG_S3_SECRET_ACCESS_KEY=your-secret-key export SGLANG_S3_ENDPOINT_URL=https://minio.example.com ``` -See [Environment Variables](../environment_variables) for the full set of storage options. +See [Environment Variables](/docs/sglang-diffusion/environment_variables) for the full set of storage options. ## Component Path Overrides @@ -401,7 +459,7 @@ selected loader must support that serialized format. Native plain-state loaders fail closed; library-managed components inherit the corresponding Transformers or Diffusers support. The transformer-specific `--quantization` flag does not select the format of component checkpoints; their own metadata does. See -[Quantized Component Repositories](../quantization#quantized-component-repositories). +[Quantized Component Repositories](/docs/sglang-diffusion/quantization#quantized-component-repositories). ## Component Attention Backend Overrides @@ -414,7 +472,7 @@ sglang generate \ --component-attention-backends text_encoder=torch_sdpa ``` -The component key must match a pipeline module key such as `text_encoder`, `text_encoder_2`, `transformer`, `transformer_2`, or `connectors`. Component overrides take precedence over the global `--attention-backend` only while that component is being constructed and otherwise fail if the component cannot satisfy them. Sparse self-attention backends use a compatible dense backend for cross-attention layers. The global backend remains strict for DiT components, while auxiliary components may fall back to a compatible backend. +The component key must match a pipeline module key such as `text_encoder`, `text_encoder_2`, `transformer`, `transformer_2`, or `connectors`. Component overrides take precedence over the global `--attention-backend` while that component is being constructed and fail if the component cannot satisfy them. A native component may explicitly defer backend selection until first use; components with fixed attention reject the override. Sparse self-attention backends use a compatible dense backend for cross-attention layers. The global backend remains strict for DiT components, while auxiliary components may fall back to a compatible backend. The Diffusers backend supports only the global backend passthrough. You can also pass dotted CLI entries: diff --git a/docs/docs/sglang-diffusion/api/openai_api.mdx b/docs/docs/sglang-diffusion/api/openai_api.mdx index fb33c21b2684..54aa18a9ea3d 100644 --- a/docs/docs/sglang-diffusion/api/openai_api.mdx +++ b/docs/docs/sglang-diffusion/api/openai_api.mdx @@ -96,6 +96,14 @@ curl -sS "http://localhost:30010/v1/models/wan-t2v" ## Endpoints +### Prompt enhancement + +Image generation, image edits and video creation accept `enhance_prompt=true` +when the server is configured with `--prompt-enhancer-config`. An independent +SRT service rewrites the prompt before diffusion runs. The default is `false`; +image responses return `data[].revised_prompt`, and video responses return +`revised_prompt`. See [deployment and model compatibility](/docs/sglang-diffusion/models_with_pe). + ### Model-specific request fields The image and video request schemas contain only OpenAI fields and stable @@ -129,7 +137,7 @@ The server implements an OpenAI-compatible Images API under the `/v1/images` nam #### Request quality -`quality` selects a model-owned sampling level when that model advertises one: use `lossless` for the reference path or `high` for a validated accelerated path. Omit it (or send OpenAI's default `auto`) to keep the runtime default. It is distinct from `output_quality`, which controls only output-file compression. The same extension is accepted by image edits and video requests. +`quality` selects a cumulative request-level optimization tier: `lossless` keeps the selected deployment's reference path and all unconditional bit-exact replacements; `extra-high` additionally enables only request-gated DiT/VAE kernel fusions; `high` includes the full `extra-high` set and may also enable model-owned sparse, caching, lower-precision, or other approximate paths. The tier does not override separately configured quantization, attention, or caching options. Omit it (or send OpenAI's default `auto`) to keep the `lossless` runtime default. It is distinct from `output_quality`, which controls only output-file compression. The same extension is accepted by image edits and video requests. **Python Example (b64_json response):** diff --git a/docs/docs/sglang-diffusion/attention_backends.mdx b/docs/docs/sglang-diffusion/attention_backends.mdx index 26e2c123a345..5ba2e5cb1f7d 100644 --- a/docs/docs/sglang-diffusion/attention_backends.mdx +++ b/docs/docs/sglang-diffusion/attention_backends.mdx @@ -9,7 +9,7 @@ This document describes the attention backends available in sglang diffusion (`s Attention backends are defined by `AttentionBackendEnum` (`sglang.multimodal_gen.runtime.platforms.interface.AttentionBackendEnum`) and selected via the CLI flag `--attention-backend`. -Backend selection is performed by the shared attention layers (e.g. `LocalAttention` / `USPAttention` / `UlyssesAttention` in `sglang.multimodal_gen.runtime.layers.attention.layer`). `--attention-backend` is strict for the diffusion transformer / DiT. Auxiliary components such as encoders and VAEs use it when compatible, then fall back to a component default or a platform-compatible backend. Use `--component-attention-backends` when an auxiliary component must use a specific backend; incompatible component overrides fail unless a sparse backend is being replaced for cross-attention. +Backend selection is performed by the shared attention layers (e.g. `LocalAttention` / `USPAttention` / `UlyssesAttention` in `sglang.multimodal_gen.runtime.layers.attention.layer`). `--attention-backend` is strict for the diffusion transformer / DiT. Auxiliary components such as encoders and VAEs use it when compatible, then fall back to a component default or a platform-compatible backend. Use `--component-attention-backends` when an auxiliary component must use a specific backend; incompatible overrides, including an override for a component that constructs no SGLang attention layer, fail unless a sparse backend is being replaced for cross-attention. When using the diffusers backend, `--attention-backend` is passed through to diffusers' `set_attention_backend` (e.g., `flash`, `_flash_3_hub`, `sage`, `xformers`, `native`). @@ -64,6 +64,11 @@ For SGLang-native pipelines, the CLI accepts the lowercase names of `AttentionBa `SAGE_ATTN_3` Requires SageAttention3 installed per upstream instructions. + + `sparge_attn` + `SPARGE_ATTN` + Training-free sparse SageAttention2. CUDA SM80/86/87/89/90, FP16/BF16, head dim 64/128, and square self-attention with sequence length at least 128. Other attention shapes use dense SDPA. Install pip install git+https://github.com/thu-ml/SpargeAttn.git --no-build-isolation; configure retained block ratio with topk. + `sol_attn` `SOL_ATTN` @@ -74,6 +79,21 @@ For SGLang-native pipelines, the CLI accepts the lowercase names of `AttentionBa `VIDEO_SPARSE_ATTN` Requires vsa. Configure sparsity via --attention-backend-config. + + `subblock_sparse_attn` + `SUBBLOCK_SPARSE_ATTN` + Training-free SubBlock sparsity for MiniMax-H3 on CUDA SM90, SM100, and SM120. The default compute mode is BF16; compute_mode=sage_fp8 is an approximate SM90-only path that requires SpargeAttn. See the MiniMax-H3 recipe. + + + `video_sparse_attn_h3` + `VIDEO_SPARSE_ATTN_H3` + Video Sparse Attention for MiniMax-H3 / FastH3 (VSA-H3). In-tree Triton block-sparse kernel (SM90 / SM100 / SM103); no external package. Configure via --attention-backend-config. + + + `hybrid_window_attn_h3` + `HYBRID_WINDOW_ATTN_H3` + VDN-H3 hybrid attention: chunk-aligned window softmax (exact, gated) plus the Video Delta linear branch over the window complement. The window runs as a union of FlashAttention varlen calls (FA4 on SM100 / SM103 / SM120, FA3 on SM90; on SM80 / SM86 / SM89 the same FA3 build runs its Sm80 mainloop at FA2-class speed). Configure via --attention-backend-config. + `vmoba_attn` `VMOBA_ATTN` @@ -119,6 +139,11 @@ For SGLang-native pipelines, the CLI accepts the lowercase names of `AttentionBa RAIN_FUSION_ATTN Requires attentions which can be installed with sgl_kernel_npu; available only for NPU. + + fp8_fa_sm120 + FP8_FA_SM120 + FP8 (E4M3) dense attention for SM120 GPUs (GeForce RTX 50, RTX PRO Blackwell), written in CuTe-DSL. Q/K/V are quantized per head on the fly; the output stays BF16. Non-causal, batch 1, head dim 128, BF16 inputs; other calls use torch_cudnn_sdpa. Opt-in only: outputs differ from BF16 attention by about 5% relative RMS per call. Each new sequence length compiles once (about 10 s). + @@ -138,6 +163,25 @@ Sparse backends selected for self-attention similarly fall back to a compatible dense backend for cross-attention. Explicit component overrides are otherwise strict. +Model and layer backend sets guide automatic selection; they are not positive +allowlists for explicit requests. A backend selected with +`--attention-backend` or `--component-attention-backends` can therefore be used +without first adding it to every model's set. It still fails closed when the +platform cannot resolve it or when it lacks a required operation such as dense +cross-attention, packed variable-length attention, or ring-attention LSE. +Model paths that require one backend for correctness declare it as required; +those layers keep that backend even when the surrounding component is +overridden. + +For an automatic, measurement-based choice, pass +`--enable-attention-backend-autotune true`. On each layer's first sufficiently +large input, the native runtime times compatible candidates and switches only +when the output remains within its numerical guard and the measured gain exceeds +the noise margin. The tuner is disabled by default and currently validated on +SM90 and SM12x. It does not replace a backend selected explicitly through +`--attention-backend` or `--component-attention-backends`, or a backend required +by the model for correctness. + ## Configuration Some backends require additional configuration. You can pass these parameters via `--attention-backend-config`. This argument accepts: @@ -213,6 +257,133 @@ Some backends require additional configuration. You can pass these parameters vi +**SpargeAttention (`sparge_attn`)** + +| Parameter | Type | Description | Default | +|---|---|---|---| +| `topk` | `float` | Fraction of predicted attention blocks retained per head. Higher values preserve more attention work and generally improve quality. Must be in `(0, 1]`. | `0.5` | + +SpargeAttention is approximate even when `topk=1`: the recommended upstream +kernel quantizes attention through SageAttention2. Validate output quality and +end-to-end latency on the target model and resolution before deployment. + +**Video Sparse Attention for H3 (`video_sparse_attn_h3`)** + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ParameterTypeDescriptionDefault
`VSA_sparsity``float`Fraction of video tiles excluded from the top-k selection (0.0 - 1.0). `0.9` is the FastH3 trained policy.`0.9`
`vsa_mode``str``exempt`: non-video keys (text/audio prefix tiles) are always selected. `compete`: they compete with video tiles in the top-k.`exempt`
`vsa_dense_first_n_steps``int`Use dense attention for the first N denoising steps.`0`
`vsa_dense_layers``list[int]`Layer indices kept dense, e.g. `[0, 1]`.`[]`
`vsa_tile_size``int`Kernel tile size. Only `64` (the trained (4, 4, 4) geometry) is accepted.`64`
+ +VSA-H3 constraints: + +- Only the DiT runs sparse; the token refiner, text encoder, and VAEs keep + their dense defaults. An explicit `--component-attention-backends + text_encoder=fa` is rejected because the H3 text encoder has SDPA-only layers. +- Uses the checkpoint's trained `to_gate_compress` compression branch. Base + MiniMax-H3 weights load zero gates and run pure sparse. +- Ulysses sequence parallelism is supported; `--ring-degree` greater than 1, + `torch.compile`, and breakable CUDA graph execution are rejected. + +**Hybrid window attention for VDN-H3 (`hybrid_window_attn_h3`)** + +[VDN-H3](/cookbook/diffusion/MiniMax/MiniMax-H3#7-vdn-h3-hybrid-attention-8-step-distill) +replaces every DiT block's dense self-attention with two branches. The +softmax branch is an exact softmax over a chunk-aligned frame window (frame +`t` belongs to chunk `t // 5` and attends to chunks `c - 1 .. c + 1`; frames +0 and F-1 are dense as rows and columns; every text / audio pair stays dense) +scaled by a per-head sigmoid gate. The linear branch (a frame-wise Video Delta +rule recurrence, forward and reverse over frames) covers exactly the window's +complement and is driven by the attention module. The mask is request-static, +so the metadata is built once per request. The checkpoint's +`transformer/config.json` carries the window geometry; the backend reads it. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ParameterTypeDescriptionDefault
`vdn_h3_dense_smoke``bool`Allows a dense transformer backend (`fa`) on the VDN-H3 weights for the base-H3 + LoRA equivalence smoke. The gates and the linear branch are skipped: a study, not a sample.`false`
`vdn_max_gather_rows``int`Upper bound on the gathered K/V rows per window FlashAttention call; consecutive chunk groups fill one call up to it. Splitting changes no query's kept key set, only the pass count and the gather's peak memory.`200000`
+ +Hybrid window attention constraints: + +- VDN-H3 requires this backend for the transformer; a dense backend on these + weights would silently skip the linear branch and the gates, so it is + rejected unless `vdn_h3_dense_smoke` is set. The token refiner, text + encoder, and VAEs keep dense attention. +- Base MiniMax-H3 and FastH3 checkpoints have no linear branch and are + rejected by this backend. +- Ulysses sequence parallelism is supported (QK-norm + RoPE run after the + all-to-all on the head shard); `--ring-degree` greater than 1, + `torch.compile`, and breakable CUDA graph execution are rejected. + **V-MoBA (`vmoba_attn`)** @@ -494,6 +665,16 @@ Some backends require additional configuration. You can pass these parameters vi + + + + + + + + + + @@ -514,6 +695,16 @@ Some backends require additional configuration. You can pass these parameters vi + + + + + + + + + + @@ -604,6 +795,16 @@ Some backends require additional configuration. You can pass these parameters vi + + + + + + + + + +
❌ CUDA-only (optional dependency).
`sparge_attn`YesNoNoNo❌❌CUDA SM80/86/87/89/90 only. Requires SpargeAttn; head dim 64/128 and square self-attention with sequence length at least 128.
`sol_attn` Yes❌ CUDA-only. Requires vsa. Configure sparsity via --attention-backend-config.
`video_sparse_attn_h3`YesNoNoNo❌❌CUDA-only (SM90 / SM100 / SM103). In-tree Triton kernel, no external dependency. Configure via --attention-backend-config.
sla_attn Yes✅ NPU-only. Requires attentions from sgl_kernel_npu Configuration via --attention-backend-config.
fp8_fa_sm120Yes❌❌❌❌❌CUDA SM120 only; falls back to torch_cudnn_sdpa on other devices and for unsupported calls.
@@ -642,6 +843,10 @@ Use this override when the fallback must be pinned: unlike the global backend, an incompatible component override raises an error instead of selecting another backend. The one role-based exception is a sparse self-attention backend, which uses a compatible dense backend for cross-attention layers in the same component. +The component must construct SGLang-selectable attention or explicitly defer +selection until first use; components with fixed attention reject the override. +Per-component overrides apply only to native pipelines. The Diffusers backend +accepts the global `--attention-backend` passthrough instead. ### Per-request override (denoise loop) @@ -674,6 +879,77 @@ dense switching; under ring parallelism the target must be ring-capable. Note `sage_attn` / `sage_attn_3` are lossy (quantized attention) — validate quality on your workload. +### Request-scoped Skip Softmax + +Skip Softmax (BLASST) keeps the QK matmul, but skips the exponential, +softmax-state update, V load, and PV matmul for attention tiles whose estimated +softmax mass is below a threshold. It is an explicit lossy optimization backed +by the FlashInfer kernels shipped with SGLang's pinned dependencies. See the +[BLASST paper](https://arxiv.org/abs/2512.12087) and NVIDIA's +[video-generation study](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/blogs/tech_blog/blog28_Accelerating_Video_Generation_with_GEMM_Quantization_Attention_Quantization_and_Skip_Softmax_Attention_in_TensorRT-LLM.md) +for the algorithm and measured quality/performance trade-offs. + +Pass `skip_softmax_params` in one image or video request. The runtime routes +compatible self-attention layers through the FA/FlashInfer path for that batch; +cross-attention keeps its normal backend, and the next request restores the +server default. + +```python +client.videos.create( + model="", + prompt="...", + extra_body={ + "skip_softmax_params": { + "threshold_scale_factor": 500.0, + "start_step": 14, + } + }, +) +``` + +- `threshold_scale_factor` is required and must be positive. The kernel uses + `threshold_scale_factor / context_length`; larger values skip more work and + usually increase quality loss. +- `start_step` is the zero-based denoising step at which sparse execution + starts. It defaults to `0`; keeping early high-noise steps dense is generally + safer. + +There is intentionally no default threshold. Calibrate the threshold and start +step against output-quality metrics for each model, resolution, step count, and +deployment backend; values tuned for one workload are not portable. + +Current support is FP16/BF16, head dimension 128 or 256, and unmasked +self-attention on Hopper SM90 and Blackwell SM100/SM103/SM107. Ulysses sequence +parallelism is supported because the kernel runs after its all-to-all. Ring +Attention, attention masks, `torch.compile`, and breakable CUDA graphs reject +the request instead of silently running dense attention. Models that merely +offer an FA backend do not automatically qualify: the runtime still checks the +GPU, dtype, head dimension, attention role, and execution mode. + +### Using SpargeAttention + +Install the optional CUDA extension, then select the backend explicitly: + +```bash +pip install ninja +pip install git+https://github.com/thu-ml/SpargeAttn.git --no-build-isolation + +sglang generate \ + --model-path \ + --prompt "..." \ + --attention-backend sparge_attn \ + --attention-backend-config topk=0.5 +``` + +`sparge_attn` applies to compatible self-attention layers. Cross-attention uses +the existing dense fallback; short self-attention and asymmetric Q/KV also use +dense SDPA. LTX-2, LTX-2.3, and LTX-2.5 have compatible video (128) and audio +(64) head dimensions, so their sufficiently long, unmasked self-attention uses +SpargeAttention while prompt and audio-video cross-attention remain dense. +Ulysses sequence parallelism is supported, but ring attention is not because the +upstream kernel does not expose the softmax LSE needed for ring merging. K/V-gather +SP therefore follows the normal sparse-backend rule and uses Ulysses instead. + ### Sage then Sol hybrid `sol_attn` keeps the first `dense_steps` steps dense. Set diff --git a/docs/docs/sglang-diffusion/cache_dit.mdx b/docs/docs/sglang-diffusion/cache_dit.mdx index 14d62e0ca271..9ca452450db1 100644 --- a/docs/docs/sglang-diffusion/cache_dit.mdx +++ b/docs/docs/sglang-diffusion/cache_dit.mdx @@ -11,8 +11,12 @@ SGLang integrates [Cache-DiT](https://github.com/vipshop/cache-dit), a caching a - **DBCache (Dual Block Cache)**: Dynamically decides when to cache transformer blocks based on residual differences - **TaylorSeer**: Uses Taylor expansion for calibration to optimize caching decisions +- **DMD Calibrator**: An **exponential-basis** forecasting calibrator (Dynamic Mode Decomposition, not Distribution Matching Distillation) that serves as a drop-in alternative to TaylorSeer's polynomial basis; strongest on flow-matching models - **SCM (Step Computation Masking)**: Step-level caching control for additional speedup +Cache-DiT also ships **SVDQuant** W4A4 (int4 / NVFP4) dynamic quantization, which can be combined +with DBCache caching (see [Quantization](#quantization)). + ## Basic Usage Cache-DiT is a **per-request** switch: each request decides whether to run @@ -47,8 +51,11 @@ client.images.generate( `SGLANG_CACHE_DIT_ENABLED` server default). `cache_dit_params` accepts the DBCache knobs (`Fn_compute_blocks`, `Bn_compute_blocks`, `max_warmup_steps`, `residual_diff_threshold`, `max_continuous_cached_steps`, `enable_taylorseer`, -`taylorseer_order`), the SCM knobs (`scm_preset`, `scm_compute_bins`, -`scm_cache_bins`, `scm_policy`), and a nested `secondary` dict with the DBCache +`taylorseer_order`), the DMD knobs (`enable_dmd`, `dmd_history`, `dmd_rank`, +`dmd_ridge`, `dmd_svd_precision`; DMD and TaylorSeer are mutually exclusive +calibrators and cannot be enabled together), the SCM knobs (`scm_preset`, +`scm_compute_bins`, `scm_cache_bins`, `scm_policy`), and a nested `secondary` +dict with the DBCache knobs for the second transformer of dual-DiT models (unset secondary keys inherit the request's primary values, then the `SGLANG_CACHE_DIT_SECONDARY_*` defaults). @@ -134,6 +141,49 @@ cache_config: enable_sperate_cfg: true # e.g, Qwen-Image, Wan, Chroma, Ovis-Image, etc. ``` +- DBCache + DMD Calibrator + +Instead of TaylorSeer, you can use the DMD calibrator: an **exponential-basis** forecasting +calibrator that serves as a drop-in alternative to TaylorSeer's polynomial basis. DMD models +the cached feature stream as a linear dynamical system (`Y_{t+1} ~= A @ Y_t`), forecasts +cached features from the fitted eigen-modes, and stays accurate over longer cache skips where +polynomial extrapolation diverges. DMD here refers to Dynamic Mode Decomposition (Schmid +2010), **not** Distribution Matching Distillation. DMD works best on flow-matching models +(e.g., FLUX), while TaylorSeer is often better on DDPM-style models — try both. DMD and +TaylorSeer are mutually exclusive — enable only one of `enable_dmd` / `enable_taylorseer`: + +```yaml Config +cache_config: + max_warmup_steps: 8 + warmup_interval: 2 + max_cached_steps: -1 + max_continuous_cached_steps: 2 + Fn_compute_blocks: 1 + Bn_compute_blocks: 0 # Bn=0 since the DMD calibrator replaces the Bn calibrator + residual_diff_threshold: 0.12 + enable_dmd: true + dmd_history: 6 # snapshot window length, 5-6 typical + dmd_svd_precision: "medium" # "low", "medium" or "high" +``` + +A `dmd_history` window of 5–6 snapshots is typically the sweet spot — longer histories do not +always help, because the feature dynamics drift across timesteps. With fewer than 4 uniformly +spaced snapshots available, DMD transparently falls back to the Taylor expansion it maintains +internally. See the +[Cache-DiT DMD documentation](https://cache-dit.readthedocs.io/en/latest/user_guide/CACHE_API/#dmd-calibrator-dynamic-mode-decomposition) +for the mathematical principle and quantitative comparisons. A ready-made config is available +at +[examples/configs/cache_dmd.yaml](https://github.com/vipshop/cache-dit/blob/main/examples/configs/cache_dmd.yaml) +in the Cache-DiT repository. Apply it with the same `--cache-dit-config` flag: + +```bash +sglang generate \ + --backend diffusers \ + --model-path Qwen/Qwen-Image \ + --cache-dit-config cache_dmd.yaml \ + --prompt "A beautiful sunset over the mountains" +``` + ### Distributed inference - 1D Parallelism @@ -300,6 +350,81 @@ sglang generate \ --prompt "A beautiful sunset over the mountains" ``` +#### SVDQuant (W4A4 int4 / NVFP4) + +SVDQuant is Cache-DiT's built-in W4A4 PTQ quantization (weights and activations in int4 or +NVFP4, with smoothed low-rank branches). It can be freely combined with DBCache caching and +the DMD calibrator for the largest speedups. + +::::note +SVDQuant requires a cache-dit build **with CUDA extension support** — a plain +`pip install cache-dit` does NOT include it. Install one of: + +```bash Command +# Option 1: prebuilt CUDA 13 wheel +pip install cache-dit-cu13== --no-deps + +# Option 2: build from source with SVDQuant enabled +git clone https://github.com/vipshop/cache-dit +cd cache-dit +export CUDA_HOME=/usr/local/cuda +CACHE_DIT_BUILD_SVDQUANT=1 pip install ".[quantization]" --no-build-isolation +``` +:::: + +Valid `quant_type` values are `svdq_int4_r{32,64,128,256}_dq` (int4 W4A4) and +`svdq_nvfp4_r{32,64,128,256}_dq` (NVFP4 W4A4; requires a Blackwell GPU). Example config +combining SVDQuant NVFP4 with DBCache + DMD (see +[examples/configs/blackwell/cache_dmd_svdq.yaml](https://github.com/vipshop/cache-dit/blob/main/examples/configs/blackwell/cache_dmd_svdq.yaml)): + +```yaml Config +cache_config: + max_warmup_steps: 8 + warmup_interval: 2 + max_cached_steps: -1 + max_continuous_cached_steps: 2 + Fn_compute_blocks: 1 + Bn_compute_blocks: 0 + residual_diff_threshold: 0.12 + enable_dmd: true + dmd_history: 6 + dmd_svd_precision: "medium" +quantize_config: + quant_type: "svdq_nvfp4_r128_dq" # nvfp4 for Blackwell; use svdq_int4_r128_dq for int4 + svdq_kwargs: + quantize_device: "cuda" + fused_mlp: true + exclude_layers: + - "embedder" + - "embed" + verbose: false +``` + +For int4 W4A4 (pre-Blackwell GPUs), the same config with +`quant_type: "svdq_int4_r128_dq"` is available at +[examples/configs/cache_dmd_svdq.yaml](https://github.com/vipshop/cache-dit/blob/main/examples/configs/cache_dmd_svdq.yaml) +(add `runtime_kernel: "v2"` to `svdq_kwargs`). + +Enable `torch.compile` for the best SVDQuant performance, and make sure `--warmup-steps` +covers the compile warmup (use the same value as `--num-inference-steps`): + +```bash Command +sglang generate \ + --backend diffusers \ + --model-path black-forest-labs/FLUX.1-dev \ + --num-inference-steps=28 \ + --warmup-mode request \ + --warmup-steps 28 \ + --cache-dit-config cache_dmd_svdq.yaml \ + --enable-torch-compile \ + --dit-cpu-offload false \ + --text-encoder-cpu-offload false \ + --prompt "A beautiful sunset over the mountains" +``` + +You can verify from the log that the quantization is active: +`[Cache-DiT] SVDQuant Type: svdq_nvfp4_r128_dq, Rank: 128`. + ### Combined Configs: Cache + Parallelism + Quantization You can also combine all the above configs together in a single yaml file `combined.yaml` that contains: @@ -418,12 +543,89 @@ TaylorSeer improves caching accuracy using Taylor expansion: +### DMD Calibrator Configuration + +DMD (Dynamic Mode Decomposition, Schmid 2010 — **not** Distribution Matching Distillation) is +an **exponential-basis** forecasting calibrator and a drop-in alternative to TaylorSeer's +polynomial basis. At each full-compute step it records a snapshot of the computed features; at +a cached step it identifies a linear propagator from the recent snapshot window (one economy +SVD with rank truncation, then eigendecomposition) and forecasts the current features via +eigenvalue powers — cheap to advance, and stable over longer cache skips where polynomial +extrapolation diverges. It typically improves both speed and quality over pure DBCache. +**DMD and TaylorSeer are mutually exclusive** (enabling both raises a `ValueError`); DMD is +best for flow-matching models, TaylorSeer for DDPM-style ones. See the +[Cache-DiT DMD documentation](https://cache-dit.readthedocs.io/en/latest/user_guide/CACHE_API/#dmd-calibrator-dynamic-mode-decomposition) +for details: + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ParameterEnv VariableDefaultDescription
Enable`SGLANG_CACHE_DIT_DMD`falseEnable the DMD calibrator
History`SGLANG_CACHE_DIT_DMD_HISTORY`6Snapshot window length; 5-6 typical. Needs >= 4 uniformly spaced snapshots, otherwise DMD falls back to TaylorSeer
Rank`SGLANG_CACHE_DIT_DMD_RANK`0SVD truncation rank; 0 = automatic (drop modes below 1e-4 of the leading singular value)
Ridge`SGLANG_CACHE_DIT_DMD_RIDGE`1e-8Tikhonov regularization added to the inverted singular values
SVD Precision`SGLANG_CACHE_DIT_DMD_SVD_PRECISION`mediumSVD precision: "low", "medium" or "high"
+ +Usage (SGLD backend, env-driven): + +```bash Command +SGLANG_CACHE_DIT_ENABLED=true \ +SGLANG_CACHE_DIT_DMD=true \ +sglang generate --model-path black-forest-labs/FLUX.1-dev \ + --prompt "A curious raccoon in a forest" +``` + +On the diffusers backend, enable DMD from the yaml config instead +(`enable_dmd: true` in `cache_config`, see +[Diffusers Backend](#diffusers-backend)); DMD can also be set per request via +`cache_dit_params: {"enable_dmd": true}`. + ### Combined Configuration Example DBCache and TaylorSeer are complementary strategies that work together, you can configure both sets of parameters simultaneously: -```bash +```bash Command SGLANG_CACHE_DIT_ENABLED=true \ SGLANG_CACHE_DIT_FN=2 \ SGLANG_CACHE_DIT_BN=1 \ @@ -496,7 +698,7 @@ SCM is configured with presets: **Usage** -```bash +```bash Command SGLANG_CACHE_DIT_ENABLED=true \ SGLANG_CACHE_DIT_SCM_PRESET=medium \ sglang generate --model-path Qwen/Qwen-Image \ @@ -507,7 +709,7 @@ sglang generate --model-path Qwen/Qwen-Image \ For fine-grained control over which steps to compute vs cache: -```bash +```bash Command SGLANG_CACHE_DIT_ENABLED=true \ SGLANG_CACHE_DIT_SCM_COMPUTE_BINS="8,3,3,2,2" \ SGLANG_CACHE_DIT_SCM_CACHE_BINS="1,2,2,2,3" \ @@ -581,7 +783,7 @@ SGLang Diffusion x Cache-DiT supports almost all models originally supported in Qwen - Qwen-Image, Qwen-Image-Edit + Qwen-Image, Qwen-Image-Edit, Qwen-Image 2.1 Hunyuan @@ -605,7 +807,8 @@ SGLang Diffusion x Cache-DiT supports almost all models originally supported in ## Limitations - **SGLang-native pipelines**: Distributed Cache-DiT paths exist for supported pipelines. Hybrid SP+TP configurations add communication and cache coordination overhead, so validate them on the target model and hardware before using them as production defaults. -- **SCM minimum steps**: SCM requires >= 8 inference steps to be effective +- **DiT layerwise offload**: Compatible. Skipped blocks are not streamed, and the first layer after a skip may sync-load. Still incompatible with `--use-fsdp-inference`. +- **SCM minimum steps**: SCM requires >= 8 inference steps to be effective. Some pipelines report `steps - 1` NFEs (for example MiniMax-H3 `num_inference_steps=8` is 7 NFEs), which trips the upstream `steps_mask` assertion; use at least 9 requested steps or custom bins. - **Model support**: The model must be registered in Cache-DiT's `BlockAdapterRegister` or have an SGLang custom block adapter. @@ -616,6 +819,18 @@ SGLang Diffusion x Cache-DiT supports almost all models originally supported in For models with < 8 inference steps (e.g., DMD distilled models), SCM will be automatically disabled. DBCache acceleration still works. +### SVDQuant unavailable or load failure + +SVDQuant cases raise `svdq_is_available() = False` or +`undefined symbol: ... materialize_cow_storage ...` when the installed cache-dit has no CUDA +extension, or the prebuilt wheel was compiled against an incompatible torch. Fix: reinstall +from the `cache-dit-cu13` wheel matching your torch version, or build cache-dit from source +with `CACHE_DIT_BUILD_SVDQUANT=1` (see [Quantization](#quantization)). Quick self-check: + +```bash Command +python -c "from cache_dit.quantization.svdquant import svdq_is_available, svdq_get_load_error as e; print(svdq_is_available(), e())" +``` + ## References - [Cache-DiT](https://github.com/vipshop/cache-dit) diff --git a/docs/docs/sglang-diffusion/caching-acceleration.mdx b/docs/docs/sglang-diffusion/caching-acceleration.mdx index bdcdd041fa37..041898339f68 100644 --- a/docs/docs/sglang-diffusion/caching-acceleration.mdx +++ b/docs/docs/sglang-diffusion/caching-acceleration.mdx @@ -54,7 +54,7 @@ advanced strategies like DBCache and TaylorSeer. It can achieve up to **1.69x sp See [Cache-DiT](./cache_dit) for detailed configuration. -Cache-DiT currently cannot be combined with `--use-fsdp-inference`. Keep FSDP disabled when enabling Cache-DiT, or use other residency/offload controls instead. +Cache-DiT currently cannot be combined with `--use-fsdp-inference`. Keep FSDP disabled when enabling Cache-DiT. DiT layerwise offload is compatible: skipped blocks are not streamed, and the first layer after a skip may sync-load. ### Quick Start diff --git a/docs/docs/sglang-diffusion/ci_perf.mdx b/docs/docs/sglang-diffusion/ci_perf.mdx index 2f87e63586bc..80cdc2e76fcb 100644 --- a/docs/docs/sglang-diffusion/ci_perf.mdx +++ b/docs/docs/sglang-diffusion/ci_perf.mdx @@ -2,6 +2,37 @@ title: "CI Performance Baselines" description: "Generate and update diffusion performance baselines used in CI." --- +## Sequential Request Checks + +Server cases can exercise request-to-request state without restarting the server: + +```python +DiffusionTestCase( + id="image_t2i", + server_args=server_args, + sampling_params=sampling_params, + perf_repeat_requests=2, +) +``` + +Every request is checked, including the first one. Performance checks include +load/runtime peak memory; consistency checks include audio when +`expect_audio_output=True`. A failure in either request fails the case, even if +the other succeeds. Reports retain the request index instead of overwriting the +first result with the second. + +Keep workloads small, but regenerate baselines and GT whenever sampling +parameters change. MiniMax-H3 T2VA and ref2va use different checkpoint +partitions, so they remain separate cases, each issuing two requests to its +own server. +The T2VA case includes VAE layerwise offload to exercise weight restoration +between requests; ref2va retains the resident VAE path. + +Use `SGLANG_GEN_BASELINE=1 pytest -s ... -k ` to collect baselines in +the same sequential order, retaining the worst observed value of each metric. +`SGLANG_GEN_GT=1` with `SGLANG_GT_OUTPUT_DIR` saves +GT; repeated requests share one GT. + ## Perf Baseline Generation Script `python/sglang/multimodal_gen/test/scripts/gen_perf_baselines.py` starts a local diffusion server, issues requests for selected test cases, aggregates stage/denoise-step/E2E timings from the perf log, and writes the results back to the `scenarios` section of `perf_baselines.json`. @@ -31,3 +62,12 @@ Specify input/output paths and timeout: ```bash python python/sglang/multimodal_gen/test/scripts/gen_perf_baselines.py --baseline python/sglang/multimodal_gen/test/server/perf_baselines.json --out /tmp/perf_baselines.json --timeout 600 ``` + +## Nightly Comparison Telemetry + +The nightly comparison reports the median of three client-side measurements. +SGLang requests also write server perf dumps for stage-level timing. An +unreadable dump does not invalidate a completed client measurement, but the +dashboard reports server sample coverage (for example, `2/3`) and computes the +server total and stage medians only from readable dumps. Treat incomplete +coverage as a telemetry warning rather than a performance regression by itself. diff --git a/docs/docs/sglang-diffusion/comfyui.mdx b/docs/docs/sglang-diffusion/comfyui.mdx index 5dbc525d1b87..f9a13e70051b 100644 --- a/docs/docs/sglang-diffusion/comfyui.mdx +++ b/docs/docs/sglang-diffusion/comfyui.mdx @@ -7,7 +7,7 @@ The [ComfyUI SGLDiffusion plugin](https://github.com/sgl-project/sglang/tree/mai **Server mode** talks to a standalone `sglang serve` process over HTTP. ComfyUI sends prompts and receives images or video. This is the same path as the [OpenAI-compatible API](/docs/sglang-diffusion/api/openai_api). -**Integrated mode** keeps ComfyUI's CLIP, VAE, and sampler loop. SGLang loads only the DiT and runs one forward per sampler step. The worker starts the native Flux / Qwen-Image / Z-Image pipeline under `--comfyui-mode`. A single-file ComfyUI `.safetensors` is loaded through a checkpoint spec. Each `apply_model` call is translated by a per-model adapter. +**Integrated mode** keeps ComfyUI's CLIP, VAE, and sampler loop. SGLang loads only the DiT and runs one forward per sampler step. The worker starts the native Flux / Qwen-Image / Z-Image / MiniMax-H3 pipeline under `--comfyui-mode`. A single-file ComfyUI `.safetensors` (or a GGUF overlay) is loaded through a checkpoint spec. Each `apply_model` call is translated by a per-model adapter. Integrated mode does not ship a separate `comfyui_*` pipeline class per model. @@ -25,7 +25,7 @@ Example workflows live next to the plugin under `workflows/`. 2. Set `num_gpus`, `tp_size`, `model_type`, or compile flags with `SGLDiffusion Options`. 3. Connect the loaded model to a standard ComfyUI sampler. -Supported integrated-mode families: Flux, Qwen-Image, Z-Image. Qwen-Image edit is experimental. +Supported integrated-mode families: Flux, Qwen-Image, Z-Image, MiniMax-H3. Qwen-Image edit is experimental. ## How a sampler step reaches the worker @@ -44,5 +44,8 @@ Integrated mode currently supports: | Z-Image | `lumina2` | `ZImagePipeline` | `Z-Image-Turbo` | | Qwen-Image | `qwen_image` | `QwenImagePipeline` | `Qwen-Image`, `Qwen-Image-2512` | | Qwen-Image edit | `qwen_image_edit` | `QwenImageEditPlusPipeline` | `Qwen-Image-Edit-2511` (experimental) | +| MiniMax-H3 | `minimax_h3` | `MiniMaxH3Pipeline` | `minimax_h3_fl2va_*.safetensors`, `minimax_h3_ref2va_*.safetensors` | + +MiniMax-H3 example graphs: `minimax_h3_t2v_sgld.json` (T2V / I2VA / FL2VA on the FL2VA DiT), `minimax_h3_r2v_sgld.json` (reference-to-video; use a `ref2va` file), `minimax_h3_t2v_sgld_upscaler.json` (two-pass 3D ×2). Set `SGLDOptions.model_type` to `minimax_h3`. On 24 GB, keep the 32B CLIP on CPU and prefer pruned BF16, GGUF, or `kitchen_int8` with DiT layerwise offload. See the [MiniMax-H3 cookbook](/cookbook/diffusion/MiniMax/MiniMax-H3#comfyui-integrated-mode). To add a model, register a checkpoint spec under `runtime/loader/comfyui_checkpoints/` and a `ComfyUIModelAdapter`. Do not add another ComfyUI pipeline class. diff --git a/docs/docs/sglang-diffusion/compatibility_matrix.mdx b/docs/docs/sglang-diffusion/compatibility_matrix.mdx index c3148725719f..28413ea67dfc 100644 --- a/docs/docs/sglang-diffusion/compatibility_matrix.mdx +++ b/docs/docs/sglang-diffusion/compatibility_matrix.mdx @@ -1,912 +1,174 @@ --- -title: "Supported Models and Optimization Compatibility" -description: "Check supported SGLang Diffusion models and their optimization compatibility." +title: "Supported Models" +description: "Browse model families and public checkpoints supported by SGLang Diffusion." --- -This page tracks supported SGLang Diffusion model families and their optimization compatibility. It also covers long-tail models that do not yet have dedicated cookbook recipes. -For model-specific usage recipes, start from the [Diffusion Cookbook](/cookbook/diffusion/intro). Cookbook pages cover the primary models with examples; this page keeps the compact support and compatibility inventory. +import { DiffusionModelCatalog } from '/src/snippets/diffusion/model-catalog.jsx'; -## Supported model inventory - -Pass the `Hugging Face Model ID` to `--model-path` for `sglang generate` or `sglang serve`. Python API users can pass the same ID to SGLang Diffusion model-loading helpers. - -Missing checkpoint aliases do not imply that a model family is unsupported. The runtime registry may also accept detector-based aliases or local model directories that match the same family. - -Rows are grouped when a family shares the same runtime path or optimization support. Use the detailed matrix below when you need per-optimization compatibility. - - - -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Model familyModel IDs
FLUX
black-forest-labs/FLUX.1-devblack-forest-labs/FLUX.2-devblack-forest-labs/FLUX.2-dev-NVFP4black-forest-labs/FLUX.2-klein-4Bblack-forest-labs/FLUX.2-klein-9Bblack-forest-labs/FLUX.2-klein-base-4Bblack-forest-labs/FLUX.2-klein-base-9B
Z-Image
Tongyi-MAI/Z-ImageTongyi-MAI/Z-Image-Turbo
Qwen-Image
Qwen/Qwen-ImageQwen/Qwen-Image-2512Qwen/Qwen-Image-EditQwen/Qwen-Image-Edit-2509Qwen/Qwen-Image-Edit-2511Qwen/Qwen-Image-Layered
LongCat-Image
meituan-longcat/LongCat-Imagemeituan-longcat/LongCat-Image-Editmeituan-longcat/LongCat-Image-Edit-Turbo
SD3 / SD3.5
stabilityai/stable-diffusion-3-mediumstabilityai/stable-diffusion-3-medium-diffusersstabilityai/stable-diffusion-3.5-mediumstabilityai/stable-diffusion-3.5-medium-diffusersstabilityai/stable-diffusion-3.5-largestabilityai/stable-diffusion-3.5-large-diffusers
SANA
Efficient-Large-Model/SANA1.5_1.6B_1024px_diffusersEfficient-Large-Model/SANA1.5_4.8B_1024px_diffusersEfficient-Large-Model/Sana_1600M_1024px_diffusersEfficient-Large-Model/Sana_600M_1024px_diffusersEfficient-Large-Model/Sana_1600M_512px_diffusersEfficient-Large-Model/Sana_600M_512px_diffusers
FireRed-Image
FireRedTeam/FireRed-Image-Edit-1.0FireRedTeam/FireRed-Image-Edit-1.1
JoyAI-Image
jdopensource/JoyAI-Image-Edit-Diffusers
Other image pipelines
zai-org/GLM-Imagetencent/Hunyuan3D-2baidu/ERNIE-Imagebaidu/ERNIE-Image-Turboideogram-ai/ideogram-4-fp8ideogram-ai/ideogram-4-nf4Comfy-Org/Ideogram-4fal/ideogram-v4-fastfal/ideogram-v4-instant
-
- - -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Model familyModel IDsResolution / modeOptimization support
FastWan
FastVideo/FastWan2.1-T2V-1.3B-DiffusersFastVideo/FastWan2.2-TI2V-5B-FullAttn-DiffusersFastVideo/FastWan2.2-TI2V-5B-Diffusers
480p / 720pVSA
SANA-Video
Efficient-Large-Model/SANA-Video_2B_480p_diffusers
T2V, 480pNo dedicated optimization listed
LingBot Video MoE
robbyant/lingbot-video-moe-30b-a3b
T2V, 480pNo dedicated optimization listed
Wan2.2
Wan-AI/Wan2.2-TI2V-5B-DiffusersWan-AI/Wan2.2-T2V-A14B-Diffusersnvidia/Wan2.2-T2V-A14B-Diffusers-NVFP4Wan-AI/Wan2.2-I2V-A14B-Diffusers
TI2V / T2V / I2V, 480p / 720pSageLaserBSARain Fusion
LongLive 2.0
Rabinovich/LongLive-2.0-5B-Diffusers
T2V / I2V, 480p / 720pNo dedicated optimization listed
HunyuanVideo
hunyuanvideo-community/HunyuanVideoFastVideo/FastHunyuan-diffusers
720×1280 / 544×960TileSageSVG2
Wan2.1
Wan-AI/Wan2.1-T2V-1.3B-DiffusersWan-AI/Wan2.1-T2V-14B-DiffusersWan-AI/Wan2.1-I2V-14B-480P-DiffusersWan-AI/Wan2.1-I2V-14B-720P-Diffusers
T2V / I2V, 480p / 720pTeaCacheTileSageSVG2LaserBSARain Fusion
TurboWan
IPostYellow/TurboWan2.1-T2V-1.3B-DiffusersIPostYellow/TurboWan2.1-T2V-14B-DiffusersIPostYellow/TurboWan2.1-T2V-14B-720P-DiffusersIPostYellow/TurboWan2.2-I2V-A14B-Diffusers
480p / 720pTeaCacheSLASageSLA
MOVA
OpenMOSS-Team/MOVA-360pOpenMOSS-Team/MOVA-720p
Video-audio, 360p / 720p; local MOVA detector aliases are also supported.No dedicated optimization listed
MiniMax-H3
MiniMaxAI/MiniMax-H3Checkpoint formats
T2VA / FL2VA / Ref2VA, 768p at 24 fps with synchronized audioCache-DiTSageOnline FP8GGUF
Wan2.1 Fun
weizhou03/Wan2.1-Fun-1.3B-InP-Diffusers
480p inpaintingTeaCacheTileSageSVG2
Helios
BestWishYsh/Helios-BaseBestWishYsh/Helios-MidBestWishYsh/Helios-Distilled
720pNo dedicated optimization listed
LTX-2
Lightricks/LTX-2Lightricks/LTX-2.3
One-stage, two-stage, TI2V, HQNo dedicated optimization listed
LTX-2.5
Lightricks/LTX-2.5-Diffusers
One-stage, two-stage, TI2V, auto-duration, diffusion decodeNo dedicated optimization listed
Cosmos3
nvidia/Cosmos3-Nanonvidia/Cosmos3-Supernvidia/Cosmos3-Super-Text2Imagenvidia/Cosmos3-Super-Image2Video
T2V / I2V / T2INo dedicated optimization listed
-
- - -
- - - - - - - - - - - - - - - - - - - - -
Model familyModel IDs / detectorNotes
LingBotWorld
robbyant/lingbot-world-fast-diffusers
Realtime world model with causal state and control tokens.
SANA-WM
Efficient-Large-Model/SANA-WM_bidirectionalEfficient-Large-Model/SANA-WM_streaming
World-model pipeline with bidirectional and streaming checkpoints.
-
- - - - -Wan2.2 TI2V 5B currently has known quality issues when used for I2V generation. - - -## Optimization compatibility - -The detailed video matrix uses these symbols: +## Ming-Image integration status -- ✅ = Full compatibility -- ❌ = No compatibility -- ⭕ = Does not apply to this model +Native support covers `inclusionAI/Ming-Image-0.1-Design` (generation and +single-image editing) and `inclusionAI/Ming-Image-0.1-Design-Layer` (ordered RGBA +layers). The checkpoints are not interchangeable. No remote modeling code is +used; checkpoint directories do not need conversion to a Diffusers layout. - +H200 full-checkpoint checks include 2048-square generation, 1024-square +four-layer decomposition, and repeated 512-square HTTP generation and editing. +The HTTP checks cover cold starts, request warmup, layerwise offload, opt-in VAE +tiling, and one or two sequential generations per request, +requiring identical pixels across repeated requests with fixed inputs. -### Video Generation Models +Two-H200 functional checks cover DiT TP2, Ulysses2, Ring2 with FlashAttention, +CFG parallelism for Design-Layer, encoder folding, and spatial VAE decoding. +Single-GPU checks cover DiT and encoder layerwise offload, Cache-DiT, +SageAttention, actual breakable CUDA graph replay, and dynamic LoRA load/remove +with a synthetic adapter. These checks do not establish quality equivalence for lossy +optimizations or bit-identical output across parallel topologies. Quantized +checkpoints and other GPU families remain unverified. -Optimization columns are abbreviated to keep the matrix readable: +Run the opt-in HTTP regression tests on a CUDA host with sufficient memory: -- `Tea` = TeaCache -- `Tile` = Sliding Tile Attention -- `Sage` = Sage Attention -- `VSA` = Video Sparse Attention -- `SLA` = Sparse Linear Attention -- `SageSLA` = Sage Sparse Linear Attention -- `SVG2` = Sparse Video Gen 2 -- `LA` = Laser Attention -- `BSA` = Block Sparse Attention -- `RF` = Rain Fusion Attention - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Model NameHugging Face Model IDResolutionTeaTileSageVSASLASageSLASVG2LABSARF
FastWan2.1 T2V 1.3B`FastVideo/FastWan2.1-T2V-1.3B-Diffusers`480p⭕⭕⭕✅❌❌❌❌❌❌
SANA-Video 2BEfficient-Large-Model/SANA-Video_2B_480p_diffusers480p❌❌❌❌❌❌❌❌❌❌
LingBot Video MoE 30B-A3Brobbyant/lingbot-video-moe-30b-a3b480p❌❌❌❌❌❌❌❌❌❌
FastWan2.2 TI2V 5BFastVideo/FastWan2.2-TI2V-5B-FullAttn-Diffusers
FastVideo/FastWan2.2-TI2V-5B-Diffusers		720p⭕⭕⭕✅❌❌❌❌❌❌
Wan2.2 TI2V 5B`Wan-AI/Wan2.2-TI2V-5B-Diffusers`720p⭕⭕✅⭕❌❌❌✅✅✅
LongLive 2.0 5BRabinovich/LongLive-2.0-5B-Diffusers480p
720p		❌❌❌❌❌❌❌❌❌❌
Wan2.2 T2V A14BWan-AI/Wan2.2-T2V-A14B-Diffusers
nvidia/Wan2.2-T2V-A14B-Diffusers-NVFP4		480p
720p		❌❌✅⭕❌❌❌✅✅✅
Wan2.2 I2V A14B`Wan-AI/Wan2.2-I2V-A14B-Diffusers`480p
720p		❌❌✅⭕❌❌❌✅✅✅
HunyuanVideo`hunyuanvideo-community/HunyuanVideo`720×1280
544×960		❌✅✅⭕❌❌✅❌❌❌
FastHunyuan`FastVideo/FastHunyuan-diffusers`720×1280
544×960		❌✅✅⭕❌❌✅❌❌❌
Wan2.1 T2V 1.3B`Wan-AI/Wan2.1-T2V-1.3B-Diffusers`480p✅✅✅⭕❌❌✅✅✅✅
Wan2.1 T2V 14B`Wan-AI/Wan2.1-T2V-14B-Diffusers`480p, 720p✅✅✅⭕❌❌✅✅✅✅
Wan2.1 I2V 480P`Wan-AI/Wan2.1-I2V-14B-480P-Diffusers`480p✅✅✅⭕❌❌✅✅✅✅
Wan2.1 I2V 720P`Wan-AI/Wan2.1-I2V-14B-720P-Diffusers`720p✅✅✅⭕❌❌✅✅✅✅
TurboWan2.1 T2V 1.3B`IPostYellow/TurboWan2.1-T2V-1.3B-Diffusers`480p✅❌❌❌✅✅⭕❌❌❌
TurboWan2.1 T2V 14B`IPostYellow/TurboWan2.1-T2V-14B-Diffusers`480p✅❌❌❌✅✅⭕❌❌❌
TurboWan2.1 T2V 14B 720P`IPostYellow/TurboWan2.1-T2V-14B-720P-Diffusers`720p✅❌❌❌✅✅⭕❌❌❌
TurboWan2.2 I2V A14B`IPostYellow/TurboWan2.2-I2V-A14B-Diffusers`720p✅❌❌❌✅✅⭕❌❌❌
Wan2.1 Fun 1.3B InPweizhou03/Wan2.1-Fun-1.3B-InP-Diffusers480p✅✅✅⭕❌❌✅❌❌❌
Helios BaseBestWishYsh/Helios-Base720p❌❌❌❌❌❌❌❌❌❌
Helios MidBestWishYsh/Helios-Mid720p❌❌❌❌❌❌❌❌❌❌
Helios DistilledBestWishYsh/Helios-Distilled720p❌❌❌❌❌❌❌❌❌❌
LTX-2 (one/two-stage/TI2V)Lightricks/LTX-2768×512
1536×1024		❌❌❌❌❌❌❌❌❌❌
MiniMax-H3 (T2VA / FL2VA / Ref2VA image, audio, video/V2V)MiniMaxAI/MiniMax-H3768p · 24 fps❌❌✅❌❌❌❌❌❌❌
LTX-2.3 (one/two-stage/TI2V/HQ)Lightricks/LTX-2.3768×512
1536×1024
1920×1088 (HQ default)		❌❌❌❌❌❌❌❌❌❌
LTX-2.5 (one/two-stage/TI2V)Lightricks/LTX-2.5-Diffusers960×544 (default)
1920×1088 (two-stage)		❌❌❌❌❌❌❌❌❌❌
Cosmos3-Nano (T2V / I2V / T2I)nvidia/Cosmos3-Nano720p · 480p
1024×1024 (T2I)		❌❌❌❌❌❌❌❌❌❌
Cosmos3-Super (T2V / I2V / T2I)nvidia/Cosmos3-Super
nvidia/Cosmos3-Super-Text2Image
nvidia/Cosmos3-Super-Image2Video		720p · 480p
1024×1024 (T2I)		❌❌❌❌❌❌❌❌❌❌
- -**Note**: - -1. Wan2.2 TI2V 5B has some quality issues when performing I2V generation. We are working on fixing this issue. -2. SageSLA is based on SpargeAttn. Install it first with `pip install git+https://github.com/thu-ml/SpargeAttn.git --no-build-isolation` -3. LTX pipeline selection: - - One-stage: `--pipeline-class-name LTX2Pipeline` - - Two-stage: `--pipeline-class-name LTX2TwoStagePipeline` - - Two-stage HQ: `--pipeline-class-name LTX2TwoStageHQPipeline` (HQ defaults to 1920×1088; you can still override `--width/--height`) - - LTX-2 and LTX-2.3 support both T2V and TI2V (`--image-path`) on one-stage and two-stage pipelines (including HQ). - - The spatial upsampler and distilled LoRA are auto-resolved from the model snapshot by default, and can still be overridden with `--spatial-upsampler-path` and `--distilled-lora-path`. - - For LTX models, the `Resolutions` column uses output video `width×height` semantics, matching `sglang generate --width ... --height ...`. -4. LTX-2 / LTX-2.3 two-stage also supports `--ltx2-two-stage-device-mode {original,resident}`: - - `original` keeps official two-stage semantics without the premerged stage-2 transformer path. - - `resident` usually provides the best latency/throughput but uses much more VRAM. - - Default is auto: `resident` on H200/high-memory CUDA GPUs, otherwise `original`. -5. Cosmos3 ships in two sizes — `nvidia/Cosmos3-Nano` (16B) and - `nvidia/Cosmos3-Super` (64B). Both share the same pipeline; the only - difference is transformer depth and width, picked up from - `transformer/config.json` at load time. A single checkpoint serves T2V, - I2V (`--image-path`), and T2I (`--num-frames 1`). - - - -## Supported Components - -SGLang Diffusion supports overriding individual pipeline components with -`---path`. The value can be either a Hugging Face repo ID or a local -component directory. - -The same overrides can also be provided in config files through -`component_paths.`. - -Component path overrides are generic, but quantized checkpoint support is -loader-specific. Native SGLang loaders that only materialize plain state dicts -reject detected quantization metadata before weight loading instead of silently -casting the checkpoint into an unquantized module. Library-managed components -inherit the corresponding Transformers or Diffusers support. See -[Quantized Component Repositories](./quantization#quantized-component-repositories). - -### Common Syntax - -CLI: - -```bash Command -sglang generate \ - --model-path black-forest-labs/FLUX.2-dev \ - --vae-path black-forest-labs/FLUX.2-small-decoder \ - --transformer-path /models/flux2/transformer +```bash +SGLANG_TEST_MING_IMAGE=1 python -m pytest -q \ + python/sglang/multimodal_gen/test/server/test_server_ming_image.py ``` -Config file: - -```yaml Config -model_path: black-forest-labs/FLUX.2-dev -component_paths: - vae: black-forest-labs/FLUX.2-small-decoder - transformer: /models/flux2/transformer +See the [Ming-Image cookbook](/cookbook/diffusion/inclusionAI/Ming-Image) for +launch commands, request parameters, and model-specific limits. + +## Qwen-Image 2.1 integration status + +Qwen-Image 2.1 has a separate native pipeline for text-to-image and optional +reference-image conditioning. Full-checkpoint validation on H200 includes +1024×1024, 40-step generation and editing. Reference comparisons use +Transformers 4.57.3, BF16 and matching Torch SDPA settings with cuDNN SDPA +disabled. The native encoder explicitly preserves that version's pre-final-norm +hidden state and vision position interpolation semantics. + +Full-checkpoint functional checks on up to two H200 GPUs cover DiT TP, Ulysses, +Ring + FA, CFG parallel, DiT layerwise offload, FA/Sage, parallel +tiled VAE decode, Cache-DiT, breakable CUDA graphs, and multiple reference +images and outputs. One H200 with resident weights also passed HTTP generation +and editing, including multiple outputs and reference images. These are +functional checks, not a broad quality evaluation of lossy optimizations. + +Additional full-checkpoint checks on B200 cover native encoder TP, combined +TP2 × Ulysses2 and TP2 × Ring2 + FA on four GPUs, and +`--layerwise-offload-components all`. Two-GPU VAE `spatial_shard` decode passed +with TP2, CFG parallelism, and all-component offload. These use 512px/4-step +editing, plus a 1024px/4-step comparison of spatial and full VAE decode. +Single-GPU 1024px/40-step generation, editing, and transparent PNG output also +passed at source revision `d629d234f1`. + +Two-B200 component regressions assert actual encoder weight sharding and compare +text and image conditioning with a single-rank reference. VAE regressions cover +both decoder structures and even/odd latent heights in FP32 and FP64. FP64 +spatial decode matches the full decoder; BF16 full-checkpoint output is not +bit-exact. In the tested image, spatial versus full decode gave RGBA PSNR +60.69 dB; this is one sample, not a general quality guarantee. + +Request-local prefix KV reuse and new-request graph replay have CUDA regression +tests, including batched targets and independent variable-length prefixes. +Compatible text-to-image requests support opt-in dynamic batching; image-edit +requests remain separate, while multiple outputs within one request are supported. +See the cookbook's [batching guidance](/cookbook/diffusion/Qwen-Image/Qwen-Image-2.1#batching) +for deployment settings and floating-point reproducibility limits. +The deployment picker marks only its exact tested HTTP combinations as +verified, including H200, B200, RTX PRO 6000 96GB, RTX 5090, and RTX 4090. +CLI-only combinations remain Unverified in the picker. + +On `1eab5de5990`, one RTX PRO 6000 Blackwell Server Edition passed repeated +1024px/40-step HTTP generation, editing, transparent generation, and transparent +input editing with resident weights or DiT layerwise offload and Torch SDPA. +Repeated requests and corresponding outputs across placements matched RGBA +pixels; transparent outputs retained alpha from 0 to 255. Its SM120 +RoPE, normalization, fusion-gate, and model regressions also passed. This check +does not establish multi-GPU or quantized-checkpoint coverage on that platform. + +Disaggregated encoder, denoiser, and decoder roles passed full-checkpoint HTTP +generation, variable-length prompts, image editing, and CFG on three B200 GPUs, +using Mooncake's same-host TCP transport. At 512px/4 steps, their RGBA pixels +matched the monolithic server exactly. Multi-host RDMA and multi-rank roles +are not covered by that check. + +Online FP8 for the DiT, encoder, and both together passed 1024px/40-step HTTP +generation and editing on one B200; both together also passed transparent PNG +generation. Serialized E4M3FN FP8 component folders with scalar weight scales +and dynamic activations passed generation, editing, and transparent output for +each component and both together. The combined export also passed separate +checks with TP2 plus encoder folding and with single-GPU all-component +layerwise offload. The 476 loaded quantized matrices and +scales matched their checkpoint values. Quantization and TP change numerical +results. Revision `f1f3366c7c` fixes vision rotary initialization rounding during +CPU/GPU transfer. In 1024px/40-step B200 checks, all-component layerwise offload +matched resident generation, editing, and transparent RGBA pixels exactly +for both native precision and the combined serialized export. Revision +`81c8c550fa` additionally preserves FP8 weights and FP32 buffers during +whole-encoder CPU offload; separate checks of that mode also matched resident +pixels. Resident output was unchanged by these fixes. See the cookbook for +sample quality metrics. + +Native-name Q4_0 GGUF files passed 1024px/40-step B200 HTTP generation, +editing, and transparent output for DiT-only, encoder-only, and both together. +The 476 packed matrices matched their exported bytes. The combined export also +passed TP2 with encoder folding; separate all-component layerwise and +whole-encoder CPU offload checks matched resident RGBA pixels exactly. +Quantization and TP change output pixels, including alpha. These checks cover +the documented private exports, not arbitrary community checkpoints. Other +GGUF types and hardware remain unverified for this model. + +Calibrated ModelOpt-format NVFP4 component directories also passed the same +six-configuration B200 HTTP matrix: each component, both together, two offload +modes, and TP2 with encoder folding. All 476 packed matrices and their scales +matched the exports after runtime layout transforms. Both offload modes +matched resident RGBA pixels exactly. NVFP4 and TP2 change image and alpha +values; this private export uses max calibration on only six requests, without +SVDQuant or AWQ. See the cookbook for measured errors. Native FP4 requires +Blackwell, and this model's NVFP4 exports remain unverified on RTX PRO 6000 and +RTX 5090. + +Synthetic Diffusers-format LoRA passed dynamic loading, merging, +and removal on one B200 and TP2 with encoder folding; both removal paths +restored the base image exactly. Trained-adapter quality remains unverified. +Platform latency measurements +and their exact workloads are in the cookbook. Breakable graphs +replay only matching warmup signatures, including condition-prefix length; +other shapes run eagerly. VAE tiling is opt-in and can change numerical results. +Do not infer compatibility from the older Qwen-Image row. + +Use the public `Qwen/Qwen-Image-2.1` checkpoint; no Hugging Face token is required. +See the [Qwen-Image 2.1 cookbook](/cookbook/diffusion/Qwen-Image/Qwen-Image-2.1) +for checkpoint layout and usage. The standard two-GPU E2E suite includes `qwen_image21_t2i_tp2` +with TP2, 1024px/40-step generation, two requests, and image consistency checks. + +The additional opt-in HTTP cases check repeated generation, editing, and real +RGBA alpha output from a local checkpoint: + +```bash +export SGLANG_QWEN_IMAGE21_TEST_MODEL=/models/Qwen-Image-2.1 +export SGLANG_QWEN_IMAGE21_TEST_IMAGE=/path/to/reference.png +python -m pytest -q python/sglang/multimodal_gen/test/server/test_server_qwen_image21.py ``` -Use the component name from the pipeline's `model_index.json` or the native pipeline's registered module name: +The module skips without the model variable; only editing skips when the image +variable is absent. Credentials stay in the runner environment. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Component TypeSupported KeysNotes
VAEvae, video_vae, audio_vaevae is the common image-generation override
Transformer / DiTtransformer, video_dit, audio_dittransformer is the standard override for the main denoiser
Text / Preprocesstext_encoder, text_encoder_2, tokenizer, processor, image_processorReplacement encoders often need matching preprocessing assets
Auxiliaryscheduler, spatial_upsampler, vocoder, connectors, dual_tower_bridge, image_encoder, vision_language_encoderOnly valid for pipelines that expose these components
- -### Known Component Repos - -The table below lists concrete Hugging Face component repos that are already used in SGLang Diffusion docs or tests. It is not an exhaustive catalog of all compatible component repos. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Base ModelOverride KeyExample RepoNotes
black-forest-labs/FLUX.2-devvaeblack-forest-labs/FLUX.2-small-decoderDecoder-only FLUX.2 VAE override
black-forest-labs/FLUX.2-devvaefal/FLUX.2-Tiny-AutoEncoderExisting tested custom VAE path
- -### VAE - -- `--vae-path` is the common image-generation override. -- `--video-vae-path` and `--audio-vae-path` are only relevant for pipelines with separate video or audio VAEs. - -### Transformer / DiT - -- `--transformer-path` is the standard override for the main denoising transformer. -- For a pre-quantized main transformer, prefer `--transformer-path` or `--transformer-weights-path`; see [Quantization](./quantization). -- `--video-dit-path` and `--audio-dit-path` are only for pipelines that split denoisers by modality. - -### Text Encoders and Preprocessors - -- `--text-encoder-path` and `--text-encoder-2-path` override primary and secondary text encoders. -- `--tokenizer-path`, `--processor-path`, and `--image-processor-path` are useful when the replacement encoder requires matching preprocessing assets. - -### Auxiliary Components - -- `--scheduler-path` is only relevant when the pipeline exposes a scheduler component. -- `--spatial-upsampler-path` is mainly for two-stage pipelines such as `LTX2TwoStagePipeline`. -- `--vocoder-path`, `--connectors-path`, `--dual-tower-bridge-path`, `--image-encoder-path`, and `--vision-language-encoder-path` are only valid for pipelines that expose those components. - -### Notes - -1. Component overrides are only valid when the target pipeline actually uses - that component. -2. The override key should match the component name in the pipeline's - `model_index.json` or the native pipeline's registered module name. -3. An override path does not enable quantization by itself. A pre-quantized - component must include metadata for a format supported by its selected - loader. `--quantization` overrides the transformer loader; other component - checkpoints select their serialized format through their own metadata. - -## Verified LoRA Examples - -This section lists example LoRAs that have been explicitly tested and verified with each base model in the **SGLang Diffusion** pipeline. - - -LoRAs that are not listed here are not necessarily incompatible. -In practice, most standard LoRAs are expected to work, especially those following common Diffusers or SD-style conventions. -The entries below simply reflect configurations that have been manually validated by the SGLang team. - - -### Verified LoRAs by Base Model - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Base ModelSupported LoRAs
MiniMax-H3`larryvrh/MiniMax-H3-Turbo-Lora`
`lightx2v/Minimax-h3-Turbo`
`fal/MiniMax-H3-Realism-People-LoRA`
Wan2.2`lightx2v/Wan2.2-Distill-Loras`
`Cseti/wan2.2-14B-Arcane_Jinx-lora-v1`
Wan2.1`lightx2v/Wan2.1-Distill-Loras`
Z-Image-Turbo`tarn59/pixel_art_style_lora_z_image_turbo`
`wcde/Z-Image-Turbo-DeJPEG-Lora`
Qwen-Image`lightx2v/Qwen-Image-Lightning`
`flymy-ai/qwen-image-realism-lora`
`prithivMLmods/Qwen-Image-HeadshotX`
`starsfriday/Qwen-Image-EVA-LoRA`
Qwen-Image-Edit`ostris/qwen_image_edit_inpainting`
`lightx2v/Qwen-Image-Edit-2511-Lightning`
Flux`dvyio/flux-lora-simple-illustration`
`XLabs-AI/flux-furry-lora`
`XLabs-AI/flux-RealismLora`
- -## Special requirements +## Supported model inventory -### Sliding Tile Attention +Use a listed checkpoint as `--model-path` with `sglang generate` or +`sglang serve`. This registry-backed list contains known public entry points; +family detection may also support compatible local directories. Open the linked +Cookbook recipe for launch commands, optimizations, adapters, and model-specific +notes. -- Currently, only Hopper GPUs (H100s) are supported. + + + + + + + + + + + diff --git a/docs/docs/sglang-diffusion/contributing.mdx b/docs/docs/sglang-diffusion/contributing.mdx index f447518f63e7..f219b6889546 100644 --- a/docs/docs/sglang-diffusion/contributing.mdx +++ b/docs/docs/sglang-diffusion/contributing.mdx @@ -8,9 +8,10 @@ This guide outlines the requirements for contributing to the SGLang Diffusion mo ## Contributor Guides -- [Support New Models](./support_new_models): implementation guide for adding new diffusion pipelines +- [Support New Models](/docs/sglang-diffusion/support_new_models): implementation guide for adding new diffusion pipelines +- [Helper ownership](/docs/sglang-diffusion/support_new_models#place-helpers-with-their-owners): where to put shared and model-specific utilities - [CI Performance](./ci_perf): update and regenerate perf baselines - +- [SGLang Plugin System](/docs/hardware-platforms/plugin): package a hardware platform outside the SGLang repository ## On AI-Assisted ("Vibe Coding") PRs @@ -75,3 +76,13 @@ Consider adding tests to the `pr-test` or `nightly-test` suites to safeguard you Please run the according testcase, then update/add the baseline to `perf_baselines.json` by following the instruction in console if applicable. See [test](https://github.com/sgl-project/sglang/tree/main/python/sglang/multimodal_gen/test) for examples + +## Out-of-tree platform changes + +Use the `sglang.multimodal_gen.platforms` entry-point group when a hardware integration can live in a separately installed package. The [plugin guide](/docs/hardware-platforms/plugin#add-diffusion-support) describes the activation callback, diffusion platform contract, and selection behavior. + +When you change the platform interface or discovery implementation in SGLang: + +- Add focused CPU-only tests with a fake OOT platform. +- Run the relevant SRT plugin tests when changing shared hook-registry behavior. +- Document workload-specific kernels or optimized paths that still require vendor integration. diff --git a/docs/docs/sglang-diffusion/deployment_cookbook.mdx b/docs/docs/sglang-diffusion/deployment_cookbook.mdx index c917bfb5c8e7..d7f911c29b07 100644 --- a/docs/docs/sglang-diffusion/deployment_cookbook.mdx +++ b/docs/docs/sglang-diffusion/deployment_cookbook.mdx @@ -132,7 +132,7 @@ See [OpenAI API: Served model name](/docs/sglang-diffusion/api/openai_api#served The modes tune native pipeline components declared to the component residency manager. DiTs, text/image encoders, VAEs, vocoders, adapters, and upsamplers can use layerwise offload when their native module declares its executable layer structure. Explicitly selecting an unsupported component fails at startup instead of falling back to another residency mode. -For direct control, assign one of `resident`, `component-offload`, or `layerwise-offload` with `--component-residency COMPONENT=MODE`: +For direct control, assign one of `resident`, `component-offload`, `snapshot-offload`, or `layerwise-offload` with `--component-residency COMPONENT=MODE`: ```bash sglang generate \ @@ -185,6 +185,43 @@ In this example, `auto` will not re-enable FSDP. The same applies to parallelism **Layerwise offload** streams the declared layers of any supported native weighted component. It lowers peak accelerator memory further, but may increase latency and lower throughput. +### Which components to stream + +`--layerwise-offload-components` takes a list, and more is not better. Streaming pays one host-to-device transfer of a component's weights **every time that component runs**, overlapped with that run's compute. What matters is how often it runs per request, not how large it is. + +- The **DiT** runs once per denoising step. Its transfer amortizes over every step and hides behind attention, so it is the component layerwise offload is for. +- A **video VAE** runs once per temporal chunk, not once: H3's decoder re-runs whole for each chunk, so streaming its blocks pays their transfer per chunk. +- A **text encoder** runs once. Keep it resident if it fits; otherwise stream it with its blocks resident. + +There are three placements, not two. A component can be in the list and still hold its blocks — `--layerwise-offload-components dit,text_encoder,vae` together with `--layerwise-resident-layers video_vae=36` transfers them once instead of per pass, and gives the VRAM back when the component finishes rather than holding it through denoise. On MiniMax-H3 at 864x480 / 124 frames that is 13 s of decode against 150 s streamed. + +That hand-back is the default lifetime, `forward`, and it is what lets the DiT's resident layers and the VAE's share one budget: each set is on the GPU only while its component runs. When both sets fit at once, `--layerwise-residency-lifetime video_vae=permanent` (or `--dit-layerwise-residency-lifetime permanent` for the DiT) places them at load instead — no transfer per request, and no host copy pinned for a reload that never comes. The trade is peak memory: `permanent` adds every component's resident set together, `forward` only the largest. On a card where the sum does not fit, `permanent` fails at startup rather than slowly; keep `forward` there. + +On a consumer card the trade is usually bad, so measure before reaching for it. Qwen-Image-2.1 on one RTX 5090 (1024px, 40 steps, three interleaved rounds, five requests each), making the text encoder's resident set permanent: + +| `text_encoder` resident | p50 | vs `forward` | peak at load | of a 31.4 GiB card | +|---|---|---|---|---| +| none (default) | 13.53 s | — | 25168 MiB | 78% | +| 0.3 permanent | 13.45 s | −0.6% | 29570 MiB | 92% | +| 0.5 permanent | 13.41 s | −0.8% | 31480 MiB | 98% | +| 0.8 permanent | 13.37 s | −1.2% | 32092 MiB | 100% | + +Every round reproduces to 0.1% and the ordering is consistent, so the win is real — and it is 0.6% to 1.2% in exchange for nearly all of the card's remaining headroom, on a machine that also has a desktop to run. The faster the card, the smaller the share the saved transfer represents: the same flag measured −2.8% on a slower box whose baseline was 14.14 s. Output stays within the usual floating-point reordering (0.5 is bit-identical to the default; 0.8 deviates by at most 1/255 with no pixel beyond one LSB). + +`permanent` earns its memory where one use makes many passes and the sum still fits — a datacenter card, or a component whose set is small next to the budget. + +So do not read "drop it from the list" as the fix for a one-shot component: that keeps it resident for the whole process. With no resident blocks on either side, moving H3's 10.4 GB video VAE out of the list took decode from 39.9 s to 5.3 s on one RTX 4090 — but it also added 5.8 GB of peak, which is the budget a 12 GB card does not have. + +### How deep to prefetch + +`--dit-offload-prefetch-size` is not monotonic. Deeper prefetch hides more of the copy, but its staging buffers crowd out activations, so latency turns back up and peak memory keeps climbing. On the H3 configuration above, 1/2/3/4 layers measured 17.3 / 15.3 / 15.8 / 16.7 s of denoise, with the deepest setting reaching 96% of a 24 GB card. Sweep two or three values rather than trusting the default. + +### When not to bother + +Before tuning residency or prefetch, measure whether the transfer is exposed — and measure it rather than inferring it from the checkpoint size. Bytes over bandwidth bounds what *could* be exposed, not what is, because prefetch exists to hide exactly that. + +A 1.3B video DiT moves about 2.6 GB a step, on the order of a tenth of a one-second step if none of it overlapped, and its residency settings measure flat: it overlaps, so there is nothing to recover. A 50-layer model at 1.4 GB a layer moves 66 GB a step, and there the same flags decide whether the model runs at all. Sweep two or three values on the target configuration and keep the measured winner. + **FSDP** shards DiT weights across multiple GPUs and all-gathers weights during forward. It can reduce DiT CPU offload cost on multi-GPU deployments, especially for validated Wan I2V workloads. FSDP sharding granularity matters. SGLang Diffusion prefers sharding direct repeated transformer block entries such as `transformer_blocks.0` or `blocks.0`. Coarser sharding lowers wrapper count but can increase all-gather peak memory; finer sharding can reduce transient memory but adds communication and scheduling overhead. If a model does not define an explicit sharding rule, the loader falls back to repeated block class names and common direct numbered block paths. diff --git a/docs/docs/sglang-diffusion/disaggregation.mdx b/docs/docs/sglang-diffusion/disaggregation.mdx index b8d70d5a6d0f..e9b721ffb621 100644 --- a/docs/docs/sglang-diffusion/disaggregation.mdx +++ b/docs/docs/sglang-diffusion/disaggregation.mdx @@ -223,7 +223,7 @@ sglang serve \ --model-path ./zai-org/GLM-Image/vision_language_encoder/ \ --tokenizer-path ./zai-org/GLM-Image/processor/ \ --enable-multimodal \ - --cuda-graph-max-bs 28 \ + --cuda-graph-max-bs-decode 28 \ --device npu \ --attention-backend ascend \ --disable-fast-image-processor \ @@ -424,24 +424,25 @@ If not specified, parallelism is auto-derived from `--num-gpus`. ## Python API -For programmatic single-machine deployment, `launch_pool_disagg_server()` is available: +For programmatic single-machine deployment, `launch_pool_disagg_server()` is available. It spawns its workers, and each child re-executes this script's module scope before initializing its platform, so keep the diffusion imports inside the guard: ```python -from sglang.multimodal_gen.runtime.server_args import ServerArgs -from sglang.multimodal_gen.runtime.launch_server import launch_pool_disagg_server - -server_args = ServerArgs.from_kwargs( - model_path="Wan-AI/Wan2.1-T2V-14B-Diffusers", - denoiser_sp=4, denoiser_ulysses=2, denoiser_ring=2, - disagg_ib_device="mlx5_0", -) - -launch_pool_disagg_server( - server_args, - encoder_gpus=[[0]], - denoiser_gpus=[[1, 2, 3, 4], [5, 6, 7, 8]], - decoder_gpus=[[0]], -) +if __name__ == "__main__": + from sglang.multimodal_gen.runtime.launch_server import launch_pool_disagg_server + from sglang.multimodal_gen.runtime.server_args import ServerArgs + + server_args = ServerArgs.from_kwargs( + model_path="Wan-AI/Wan2.1-T2V-14B-Diffusers", + denoiser_sp=4, denoiser_ulysses=2, denoiser_ring=2, + disagg_ib_device="mlx5_0", + ) + + launch_pool_disagg_server( + server_args, + encoder_gpus=[[0]], + denoiser_gpus=[[1, 2, 3, 4], [5, 6, 7, 8]], + decoder_gpus=[[0]], + ) ``` ## Architecture diff --git a/docs/docs/sglang-diffusion/dynamic_batching.mdx b/docs/docs/sglang-diffusion/dynamic_batching.mdx index 3fbd54089320..14c845e51e22 100644 --- a/docs/docs/sglang-diffusion/dynamic_batching.mdx +++ b/docs/docs/sglang-diffusion/dynamic_batching.mdx @@ -45,7 +45,7 @@ Use `--batching-config /path/to/batching_config.json` to load JSON rules when a ## Compatibility -An initial implementation of dynamic batching for T2I and T2V models can be found in [#18764](https://github.com/sgl-project/sglang/pull/18764). The current compatibility grid is below and will be updated as more coverage is added. See [Supported Models and Optimization Compatibility](./compatibility_matrix) for common model IDs and optimization support. +An initial implementation of dynamic batching for T2I and T2V models can be found in [#18764](https://github.com/sgl-project/sglang/pull/18764). The current compatibility grid is below and will be updated as more coverage is added. See [Supported Models](./compatibility_matrix) for common model IDs. `✅` means supported, `❌` means not currently supported, `?` means untested, and `-` means not applicable. @@ -77,6 +77,7 @@ An initial implementation of dynamic batching for T2I and T2V models can be foun GLM-Image (external AR)✅- Qwen Image✅- Qwen Image 2512✅- + Qwen Image 2.1✅❌ Qwen Image Edit-❌ Qwen Image Edit 2509-? Qwen Image Edit 2511-? @@ -98,6 +99,11 @@ An initial implementation of dynamic batching for T2I and T2V models can be foun +Qwen Image 2.1 supports merging compatible text-to-image requests. Image edits +are not merged across requests; `n > 1` still produces multiple outputs within +one edit request. See its [cookbook](/cookbook/diffusion/Qwen-Image/Qwen-Image-2.1#batching) +for deployment settings and batching tradeoffs. + ### Video diff --git a/docs/docs/sglang-diffusion/encoder_parallel.mdx b/docs/docs/sglang-diffusion/encoder_parallel.mdx index 610db498d55f..42aab26878a5 100644 --- a/docs/docs/sglang-diffusion/encoder_parallel.mdx +++ b/docs/docs/sglang-diffusion/encoder_parallel.mdx @@ -6,8 +6,10 @@ metatags: --- While the DiT denoises, the text and image encoders are idle — and while they -encode, the whole DiT replica is idle. `--encoder-parallel` decides how to use -those otherwise-unused GPUs for the encoding stage. +encode, the whole DiT replica is idle. `--encoder-parallel` decides how native +encoders use those otherwise-unused GPUs for the encoding stage. Folding applies +to native text and image encoders; within-replica batch DP currently requires an +explicitly supported native text encoder. ```bash --encoder-parallel {auto,fold,dp,replicate} @@ -15,9 +17,9 @@ those otherwise-unused GPUs for the encoding stage. | Mode | What it does | Use when | | --- | --- | --- | -| `auto` | Picks `fold`, `dp`, or the existing encoder layout from its width and the request's batch width | Default; you want the decision made per encoder | +| `auto` | Picks folding or the existing layout for native text/image encoders, and batch DP for supported native text encoders | Default; you want the decision made per encoder | | `fold` | TP-shards the encoder weights across the idle DiT replica | One wide encoder dominates a single-request encode | -| `dp` | Encoder copies split the prompt batch, then all-gather their outputs inside the replica | Throughput serving with `--batching-max-size > 1` | +| `dp` | Supported native text encoder copies split the prompt batch, then all-gather their outputs inside the replica | Throughput serving with `--batching-max-size > 1` | | `replicate` | Keeps the encoder on its DiT TP group and encodes redundantly across the other replica ranks | You want to disable folding and batch DP | The two accelerated modes are mutually exclusive per encoder: folding shards the diff --git a/docs/docs/sglang-diffusion/environment_variables.mdx b/docs/docs/sglang-diffusion/environment_variables.mdx index 89ece541946d..163dede82041 100644 --- a/docs/docs/sglang-diffusion/environment_variables.mdx +++ b/docs/docs/sglang-diffusion/environment_variables.mdx @@ -23,6 +23,16 @@ description: "Configure SGLang diffusion behavior with environment variables." + + + + + + + + + + @@ -68,11 +78,6 @@ description: "Configure SGLang diffusion behavior with environment variables." - - - - - @@ -187,6 +192,16 @@ description: "Configure SGLang diffusion behavior with environment variables." + + + + + + + + + +
not set Installed package that registers out-of-tree diffusion pipelines and component models. The package is imported once in every process.
SGLANG_DIFFUSION_PLATFORM_OVERRIDEnot setSelect cpu, cuda, rocm, mps, npu, musa, or an installed sglang.multimodal_gen.platforms entry-point name. XPU remains automatic-only. When unset, SGLang Diffusion detects installed platform plugins before trying built-in platforms.
SGLANG_PLUGINSnot setComma-separated allowlist for installed sglang.multimodal_gen.plugins hooks. When unset, all discovered diffusion hooks load.
SGLANG_DIFFUSION_TARGET_DEVICE cudaINFO Default logging level
SGLANG_DIFFUSION_WORKER_MULTIPROC_METHODforkMultiprocess context for workers (fork or spawn)
SGLANG_DIFFUSION_IPC_A2A truefalse Experimental opt-in for fused W8A8 FP8 GEMM in diffusion weight-only FP8 linears. When disabled, FP8 weights are dequantized to the compute dtype before matmul. Enabling this dynamically quantizes activations to FP8 and may change output quality.
SGLANG_DIFFUSION_ENABLE_MXFP8_ATTENTIONfalseEnable Ascend MXFP8 FA for supported non-causal self-attention layers. Applies to online MXFP8Config and offline ModelSlim W8A8_MXFP8 checkpoints. Unsupported calls continue to use the regular attention path.
SGLANG_DIFFUSION_MXFP8_FA_HEAD_CHUNK_SIZE4Maximum number of attention heads processed by each Ascend MXFP8 FA call. Smaller chunks may improve performance for large workloads but add kernel launches; the optimal value depends on the model and input shape. Set to 0 to disable head splitting.
@@ -253,6 +268,31 @@ See [cache-dit documentation](./cache_dit) for detailed configuration. 1 TaylorSeer order (1 or 2) + + `SGLANG_CACHE_DIT_DMD` + false + Enable the DMD (Dynamic Mode Decomposition) calibrator (mutually exclusive with TaylorSeer) + + + `SGLANG_CACHE_DIT_DMD_HISTORY` + 6 + DMD snapshot window length (5-6 typical; needs >= 4 uniformly spaced snapshots, otherwise DMD falls back to TaylorSeer) + + + `SGLANG_CACHE_DIT_DMD_RANK` + 0 + DMD SVD truncation rank (0 = automatic) + + + `SGLANG_CACHE_DIT_DMD_RIDGE` + 1e-8 + DMD Tikhonov regularization added to the inverted singular values + + + `SGLANG_CACHE_DIT_DMD_SVD_PRECISION` + medium + DMD SVD precision (low/medium/high) + `SGLANG_CACHE_DIT_SCM_PRESET` none @@ -329,6 +369,31 @@ For dual-transformer models (e.g., Wan2.2 with high/low-noise experts), these va (from primary) TaylorSeer order (1 or 2) + + SGLANG_CACHE_DIT_SECONDARY_DMD + (from primary) + Enable the DMD calibrator (mutually exclusive with TaylorSeer) + + + SGLANG_CACHE_DIT_SECONDARY_DMD_HISTORY + (from primary) + DMD snapshot window length + + + SGLANG_CACHE_DIT_SECONDARY_DMD_RANK + (from primary) + DMD SVD truncation rank (0 = automatic) + + + SGLANG_CACHE_DIT_SECONDARY_DMD_RIDGE + (from primary) + DMD Tikhonov regularization term + + + SGLANG_CACHE_DIT_SECONDARY_DMD_SVD_PRECISION + (from primary) + DMD SVD precision (low/medium/high) + diff --git a/docs/docs/sglang-diffusion/fused_kernels.mdx b/docs/docs/sglang-diffusion/fused_kernels.mdx index 25ac51cbdf09..ce0052aa92fe 100644 --- a/docs/docs/sglang-diffusion/fused_kernels.mdx +++ b/docs/docs/sglang-diffusion/fused_kernels.mdx @@ -4,26 +4,44 @@ description: "The fused CUDA/Triton kernels SGLang Diffusion ships, what each on tag: "preserve" --- -Diffusion transformers and VAEs spend a large share of their non-GEMM time on short elementwise chains — adaLN modulate, residual gating, QK-norm, RoPE, norm epilogues — each of which is a separate kernel launch and a separate HBM round trip in eager PyTorch. SGLang Diffusion replaces these chains with fused kernels under [`sglang/kernels/ops/diffusion`](https://github.com/sgl-project/sglang/tree/main/python/sglang/kernels/ops/diffusion). +Diffusion transformers and VAEs spend a large share of their non-GEMM time on short elementwise and selection chains — adaLN modulate, residual gating, QK-norm, RoPE, norm epilogues, and MoE routing — each of which can require several kernel launches and HBM round trips in eager PyTorch. SGLang Diffusion replaces these chains with fused kernels under [`sglang/kernels/ops/diffusion`](https://github.com/sgl-project/sglang/tree/main/python/sglang/kernels/ops/diffusion). This page is an inventory: what each kernel fuses, what its numerical contract is, and which models use it. It is not a lever you tune — most of these kernels are on by default and require no flag. The one switch is `--quality`, described below. -## Two numerical contracts +## Numerical contracts and quality tiers -Multi-step denoising amplifies a per-step rounding difference into visible quality loss, so "close enough" and "bit-exact" are different products here. Every kernel in the package falls into one of two classes. +Multi-step denoising amplifies a per-step rounding difference into visible quality loss, so "close enough" and "bit-exact" are different products here. The `quality` switch distinguishes unconditional bit-exact replacements from non-bit-exact eager-chain fusions: **Bit-exact — mounted unconditionally.** The kernel reproduces every rounding boundary of the eager chain, so `torch.equal` holds against the reference. Some go quite far to get there: the fused LayerNorm+modulate kernel replicates PyTorch's `vectorized_layer_norm_kernel` down to its Welford update order, guarded reciprocal, and warp-fold tree; the fused RMSNorm+scale/shift kernel replicates FlashInfer's CuTe-DSL `RMSNormKernel` fragment order and `shfl.bfly` fold. Because the dispatch they replicate can change underneath them, each one still verifies itself against the live eager chain on first sight and falls back permanently on any mismatch. -**Not bit-exact — request-gated.** These differ from eager only at half-precision rounding-order level, but that is enough to matter, so they are mounted only for `quality="high"` requests, at batch boundaries, all-or-nothing per transformer. The default `quality="lossless"` runs the unmodified reference chain. +**Not bit-exact — request-gated.** These differ from eager only at half-precision rounding-order level, but that is enough to matter, so they are mounted only for `quality="extra-high"` and `quality="high"` requests, at batch boundaries, all-or-nothing per transformer. The default `quality="lossless"` runs the unmodified reference chain. + +**Model/checkpoint-native.** Generic close-contract kernels, sparse operators, and FP8/NVFP4 producers can be part of a model implementation or a separately selected deployment path. They are documented in the inventory, but `quality` does not select or undo those choices. + +**Selection-equivalent routing — enabled unconditionally.** LingBot Video's fused group-limited top-k returns the same selected expert-id set as its guarded CUDA `torch.topk(..., sorted=False)` reference chain. The order of those ids is not part of either path's contract. Because the selected experts are unchanged, this path does not depend on the request `quality` tier. -A plain fp32 single-pass norm fusion looks harmless and is not. On ERNIE-Image it moved the 50-step trajectory to 18.83 dB PSNR at `quality=high`, which is what motivated the bit-exact rewrite of that path. +A plain fp32 single-pass norm fusion looks harmless and is not. On ERNIE-Image it moved the 50-step trajectory to 18.83 dB PSNR, which is what motivated the bit-exact rewrite of that path. + + +The quality levels are cumulative: + +| `quality` | Included optimization set | +| --- | --- | +| `lossless` | The selected deployment's reference path plus every unconditional bit-exact replacement | +| `extra-high` | Everything in `lossless`, plus request-gated DiT and VAE kernel fusions; this level does not itself enable sparse, caching, or another approximate path | +| `high` | Everything in `extra-high`, plus any model-owned high-only optimization, such as an audited Cache-DiT policy or lower-precision VAE decode | + +If a model has no eligible request-gated fusion, `extra-high` can execute the same path as `lossless`. Likewise, `high` adds only the model-specific high-only paths that the active pipeline implements. + + +`quality` is not a master precision switch. A quantized checkpoint, an explicitly selected approximate attention backend, or an independently enabled cache remains active at every quality tier. ## Enabling the request-gated set ```bash -sglang generate --model-path MODEL_PATH --prompt "..." --quality high +sglang generate --model-path MODEL_PATH --prompt "..." --quality extra-high ``` The server default stays `lossless`; the OpenAI-compatible endpoints carry it per request. Images: @@ -31,7 +49,7 @@ The server default stays `lossless`; the OpenAI-compatible endpoints carry it pe ```bash curl -X POST http://${HOST}:${PORT}/v1/images/generations \ -H 'Content-Type: application/json' \ - -d '{"model": "MODEL_PATH", "prompt": "...", "quality": "high"}' + -d '{"model": "MODEL_PATH", "prompt": "...", "quality": "extra-high"}' ``` Video, same field: @@ -39,7 +57,7 @@ Video, same field: ```bash curl -X POST http://${HOST}:${PORT}/v1/videos \ -H 'Content-Type: application/json' \ - -d '{"model": "MODEL_PATH", "prompt": "...", "quality": "high"}' + -d '{"model": "MODEL_PATH", "prompt": "...", "quality": "extra-high"}' ``` @@ -50,54 +68,70 @@ The `quality` field in a **video response** body is unrelated. It is Sora-compat Do not combine request-gated DiT fusions with `--enable-breakable-cuda-graph`. -BCG warmup captures the lossless module branches before a high-quality request -mounts its DiT fusions, so replay would bypass the requested kernels. SGLang -rejects this combination for models with eligible DiT quality sites. Models -whose high-quality path changes only VAE decode remain allowed because BCG -captures the DiT only. +BCG warmup captures the lossless module branches before an `extra-high` or +`high` request mounts its DiT fusions, so replay would bypass the requested +kernels. SGLang rejects this combination for models with eligible DiT quality +sites. Models whose request-gated path changes only VAE decode remain allowed +because BCG captures the DiT only. -These fusion families mount under `quality="high"`: +These fusion families mount under both `quality="extra-high"` and +`quality="high"`: | Fusion | What it folds | | --- | --- | | Linear + tanh-GELU | Bias-add and GELU into the GEMM epilogue (cublasLt), removing the `[tokens, 4*dim]` intermediate round trip | +| Wan NVFP4 linear + GELU | Bias-add and GELU fused into Wan's NVFP4 FFN projection output | +| Qwen-Image added-QKV | Added Q/K/V projections fused into joint-buffer production | | LayerNorm + modulate | `layer_norm(x, weight=(1 + scale), bias=shift)` in place of affine-free LN plus a separate modulate | | LTX-2 RMSNorm + modulate | `rms_norm(x) * (1 + scale) + shift` in one launch | | Gate RMSNorm (BF16-native) | `RMSNorm + tanh + mul + add` in one pass | | HunyuanVideo strided QK RMSNorm | Per-head QK RMSNorm over the packed QKV layout | | LingBot Video fused RMSNorm | Replaces the handwritten cast, square, mean, rsqrt, and multiply chain with existing Triton RMSNorm kernels | +| LingBot Video per-token gated residual + RMSNorm modulate | Folds `residual + gate * update` (per-token `[B, S, 1]` gate) and the `rmsnorm(x) * (1 + scale) + shift` adaLN chain (strided `[B, S, 6D]` chunk views) into single kernels | | SANA-Video BF16-input linear attention | Keeps the first linear-attention GEMM's inputs in BF16 with FP32 accumulation/output; the second GEMM remains FP32 | +| FLUX-family VAE fast paths | Channels-last decode, GroupNorm(+SiLU), upsample, and attention replacements for FLUX.2 and AutoencoderKL-based FLUX.1, Z-Image, and SD3 pipelines | +| Wan VAE RMSNorm + SiLU | Replaces the channel-first RMSNorm/SiLU chain while keeping the decode in `channels_last_3d` | ## Kernel inventory -34 operators are registered in the kernel registry across 38 implementations (some operators carry several backends). Backends are named by provenance, not device: `JIT` compiles under nvcc *and* hipcc, `TRITON` runs on CUDA and ROCm, `CUTE_DSL` needs CUTLASS, `FLYDSL` is ROCm gfx950 only, `AOT` comes from the `sgl_kernel` wheel. +45 operators are registered in the kernel registry across 51 implementations (some operators carry several backends). Backends are named by provenance, not device: `KDA` identifies Kernel Design Agents implementations, `JIT` compiles under nvcc *and* hipcc, `TRITON` identifies Triton sources, `CUTE_DSL` needs CUTLASS, `FLYDSL` is ROCm gfx950 only, and `AOT` comes from the `sgl_kernel` wheel. Per-operator capability metadata determines which devices can load each implementation. ### Normalization | Operator | Backend | Contract | Replaces | | --- | --- | --- | --- | | `rmsnorm_scale_shift` | Triton | bit-exact | RMSNorm + `* (1 + scale) + shift` (4 kernels) | -| `scale_residual_norm_scale_shift` | Triton / CuTe-DSL / FlyDSL | bit-exact (Triton) | the above plus the preceding `residual + gate * update` | +| `scale_residual_norm_scale_shift` | KDA / Triton / CuTe-DSL / FlyDSL | bit-exact (Triton) or backend-specific close contract | the above plus the preceding `residual + gate * update` | +| `scale_residual_norm_scale_shift_nvfp4` | JIT CUDA | matches the selected NVFP4 producer contract | Qwen residual LayerNorm/modulation + FC1 NVFP4 quantization | | `layernorm_modulate` | Triton | bit-exact | affine-free LayerNorm + adaLN modulate | +| `try_fused_fp32_layernorm_bf16` | KDA / Triton | bit-exact against the verified native dispatch | LingBot World causal FP32 LayerNorm on Hopper with FP32 affine or adaLN post-ops, including BF16 input/output conversions; other devices retain native LayerNorm | | `qk_head_layernorm` | Triton | bit-exact | per-head LayerNorm on q/k | | `qk_rmsnorm_native` | Triton | bit-exact | Z-Image per-head QK RMSNorm | -| `norm_scale_shift` | CuTe-DSL / FlyDSL | fp32 statistics | LN-or-RMS + scale/shift, many broadcast modes | +| `norm_scale_shift` | KDA / CuTe-DSL / FlyDSL | backend-specific close contract | LN-or-RMS + scale/shift, many broadcast modes | | `rmsnorm_scale`, `rmsnorm_tanh_residual` | Triton | bf16-native statistics | `RMSNorm(x) * scale`, `x + tanh(gate) * RMSNorm(y)` | | `apply_group_norm_silu` | Triton | close | `GroupNorm + SiLU`, NCHW-contiguous | | `group_norm_silu_4d`, `group_norm_silu_rows` | Triton | close | channels-last GroupNorm(+SiLU); what lets a VAE decoder run channels_last end to end with no `nchwToNhwc` transposes | | `wan_rmsnorm_silu` | Triton | close | Wan VAE `channels_last_3d` RMSNorm + SiLU | +| `wan_norm_silu_post` | JIT CUDA | bit-exact, first-sight verified | Wan VAE FP32 divide, scale, affine, and SiLU after the unchanged native L2 reduction; dense NCDHW and `channels_last_3d` | ### adaLN modulation and gating | Operator | Backend | Contract | Replaces | | --- | --- | --- | --- | | `modulate_scale_shift` | JIT CUDA | bit-exact | `x * (1 + scale) + shift` | -| `residual_gate_add` | JIT CUDA | bit-exact | `residual + gate * update` | +| `residual_gate_add` | KDA (JIT CUDA) | bit-exact | `residual + gate * update` | +| `try_fused_scaled_residual_bf16` | Triton | bit-exact against the verified native dispatch | LingBot World causal BF16 residual with an FP32 per-frame gate; skips the unused normalized output before camera conditioning | | `timestep_embedding` | JIT CUDA | close | sinusoidal timestep embedding | | `temb_table_slices` | Triton | bit-exact | see note below | | `ltx2_ada_values` | Triton | bit-exact | LTX-2 nine-way adaLN value split, slices come out contiguous | +LingBot World camera conditioning also uses `modulate_scale_shift` with one +affine row per token. The causal normalization and residual paths verify each +new signature outside CUDA graph capture and fall back to their native chains +on mismatch or unsupported layouts. These paths preserve `lossless` arithmetic; +kernel replay support does not enable BCG for a model configuration that disables it. + `temb_table_slices` is worth knowing about. The eager `(scale_shift_table + temb.float()).chunk(6, dim=2)` materializes roughly 8 GB of fp32 at 704p/121f *and* hands six **strided** slices downstream, whose `.contiguous()` calls then copy each one again. The fused kernel produces the six slices in one pass, each naturally contiguous, so the downstream copies become no-ops. @@ -107,9 +141,12 @@ These fusion families mount under `quality="high"`: | Operator | Backend | Contract | Replaces | | --- | --- | --- | --- | | `fused_inplace_qknorm_rope` | JIT CUDA | one bf16 rounding step vs the split baseline; exact with `round_norm_before_rope=True` | separate QK-norm kernel + RoPE | +| `flux2_qkv_epilogue` | KDA (JIT CUDA) | bit-exact against its selected BF16 reference chain | FLUX.2 QK RMSNorm + RoPE + joint text/image QKV packing | +| `qwen_qkv_epilogue` | JIT CUDA | bit-exact against its selected BF16 reference chain | Qwen-Image QK RMSNorm + RoPE + joint QKV writes on SM100+ | | `rope_rotate_half` | Triton | bit-exact | `chunk` → `cat(-x2, x1)` → two muls + add → `cat(tail)`, about 7 kernels per projection | | `interleaved_rope_fp64` | JIT CUDA | bit-exact | paired SANA-Video Q/K RoPE with fp64 tables, about 14 eager kernels | -| `ltx2_qknorm_split_rope` | JIT CUDA | close (validated on B200) | LTX-2 QK-norm + split RoPE | +| `helios_qk_rope` | JIT CUDA | bit-exact | paired in-place Helios Q/K RoPE with transposed frequency layout | +| `ltx2_qknorm_split_rope` | KDA (JIT CUDA) | close (validated on B200) | LTX-2 QK-norm + split RoPE | | `ltx25_decoder_rope` | JIT CUDA | bit-exact | paired LTX-2.5 decoder 3D RoPE from cached compact axis tables | | `hunyuan_qkv_rope_pack` | Triton | bit-exact | QKV pack and RoPE in one pass | @@ -128,6 +165,12 @@ These fusion families mount under `quality="high"`: | `sparse_linear_attn_fwd` | Triton | block-map, compression, and forward for sparse linear attention | | `bigdn` | Triton | Sana-WM bidirectional gated delta-net; the chunkwise form splits phase A along the KV and Z streams so two blocks stay resident per SM, and stores `(I - P)` so phase B's MMA folds the identity-add in | +### MoE routing + +| Operator | Backend | Contract | Replaces | +| --- | --- | --- | --- | +| `group_limited_topk` | Triton | selected expert-id set matches the guarded CUDA reference; output order is unspecified | LingBot Video's per-group top-2 reduction, group top-k, mask construction, masked expert scores, and final expert top-k | + ### Data movement Every kernel here only moves values (plus zero fill, plus at most one same-order add), so each is bitwise identical to the aten chain it replaces. @@ -137,10 +180,38 @@ Every kernel here only moves values (plus zero fill, plus at most one same-order | `usp_merge_heads` | JIT CUDA | USP all-to-all output head merge (`permute` + `contiguous`) | | `pack_qkv_destination_major` | Triton | Ulysses destination-major QKV pack | | `varlen_pack_qkv`, `varlen_scatter_to_padded` | Triton | varlen gather/scatter around the masked attention path | -| `causal_conv3d_cat_pad` | JIT CUDA / Triton | causal Conv3d `cat` + `pad` | +| `varlen_pack_segmented_qkv` | Triton | varlen gather from a virtual prefix/main Q/K/V sequence | +| `joint_qkv_cat` | Triton | Joy Image Edit's three image/text Q/K/V concatenations, including strided packed-V inputs | +| `causal_conv3d_cat_pad` | KDA (JIT CUDA) / Triton | causal Conv3d `cat` + `pad` | | `cat_pad_channels_last_3d` | Triton | Wan causal VAE `cat + F.pad + contiguous` (three passes plus cache bookkeeping) in one pass | | `dup_up3d_add` | Triton | `repeat_interleave + permute().contiguous() + add` | +Joy Image Edit uses the joint-copy path for eligible CUDA FP16/BF16 image +tensors of at least 32 MiB. It preserves image-first token order, copies +values without arithmetic, and verifies each new shape/stride signature +against the native concatenations before enabling it. Small inputs, +unsupported layouts, gradient-bearing inputs, and unverified signatures +during graph capture use the native path. This does not enable model-level +BCG support. + +For large Hopper BF16 image Q/K with 32 heads of width 128, Joy also uses +the existing out-of-place QK-Norm + RoPE kernel to read the packed projection +directly. This removes two input copies while retaining the original CUDA +arithmetic and contiguous outputs. Each new signature is checked bitwise; +inputs remain intact if the operation fails. Other shapes and platforms, +compilation, unverified capture, and `SGLANG_ENABLE_FUSED_QKNORM_ROPE=0` +retain the original helper. + +### Quantized layout producers + +These kernels preserve the quantized checkpoint path's selected reference operation. They are not a claim that FP8 or NVFP4 is equivalent to an unquantized BF16 checkpoint. + +| Operator | Backend | Replaces | +| --- | --- | --- | +| `flux2_layernorm_modulate_fp8_quant` | KDA (Triton) | FLUX.2 LayerNorm plus adaLN modulation directly into static FP8 output | +| `flux2_token_cat_fp8` | KDA (Triton) | FLUX.2 single-block attention/MLP concatenation plus static FP8 quantization | +| `flux2_token_cat_nvfp4` | JIT CUDA | FLUX.2 single-block attention/MLP concatenation plus NVFP4 quantization | + ## Coverage by model Kernels are written against a specific eager chain in a specific model, so coverage is per-model rather than universal. @@ -148,20 +219,21 @@ Kernels are written against a specific eager chain in a specific model, so cover | Model | Fused paths | | --- | --- | | FLUX.1 | LN+modulate, modulate, residual-gate add, linear+GELU | -| FLUX.2 | LN+modulate, packed SwiGLU, residual-gate add | -| Qwen-Image | linear+GELU, select-0/1 LN modulation | +| FLUX.2 | LN+modulate, fused LN+modulate-to-FP8, packed SwiGLU, gated residual/norm, residual-gate add, QK RMSNorm+RoPE+joint QKV packing, FP8/NVFP4 token-cat producers | +| Qwen-Image | linear+GELU, select-0/1 LN modulation, added-QKV fusion, QK RMSNorm+RoPE+joint QKV writes, residual norm/modulate+NVFP4 producer | | GLM-Image | LN+modulate, per-head qk LN, residual-gate add, linear+GELU | | ERNIE-Image | RMSNorm+scale/shift, residual-gated variant, rotate-half RoPE, residual-gate add | +| Joy Image Edit | Bitwise image/text QKV concatenation; strided image QK-Norm + RoPE on Hopper | | Z-Image | BF16-native RMSNorm scale / tanh-residual, per-head QK RMSNorm | | Ideogram 4 | gate RMSNorm, SwiGLU, rotate-half RoPE, modulate, residual-gate add | | LTX-2 | QK-norm + split RoPE, ada-values split, RMSNorm+modulate, modulate, residual-gate add, linear+GELU | | LTX-2.5 decoder | paired 3D RoPE with shared axis-table cache | -| HunyuanVideo | QKV+RoPE pack, strided QK RMSNorm, linear+GELU | -| LingBot Video MoE | Fused RMSNorm at `quality=high` | +| HunyuanVideo / Helios | QKV+RoPE pack, strided QK RMSNorm, linear+GELU; Helios also has paired in-place Q/K RoPE | +| LingBot Video MoE | Default-on group-limited top-k expert selection; fused RMSNorm, per-token gated residual, and fused RMSNorm+modulate at `quality=extra-high` or `quality=high` | | Sana | LN+modulate, GLUMB bias+SiLU / bias+GLU, residual-gate add | -| SANA-Video | Packed QKV/KV; paired fp64 interleaved RoPE; LN+modulate, GLUMB bias+SiLU / bias+GLU, and residual-gate add during BCG; BF16-input linear attention at `quality=high` | +| SANA-Video | Packed QKV/KV; paired fp64 interleaved RoPE; GLUMB bias+SiLU / bias+GLU in eager and BCG; LN+modulate and residual-gate add during BCG; BF16-input linear attention at `quality=extra-high` or `quality=high` | | Sana-WM | bidirectional gated delta-net, fused QK inverse-RMS | -| Wan | temb table slices; VAE cat+pad and DupUp3D add, `channels_last_3d` RMSNorm+SiLU | +| Wan / LongLive 2 VAE | temb table slices (Wan); VAE cat+pad and DupUp3D add; lossless FP32 normalization post-ops; full `channels_last_3d` RMSNorm+SiLU at `quality=extra-high` or `quality=high` | | Cosmos3 / Krea2 / MiniMax-H3 | QK-norm + RoPE (Krea2 also CuTe-DSL norm+scale/shift; MiniMax-H3 also indexed modulation) | | FLUX.2 VAE / HunyuanVAE / latent upsampler | GroupNorm + SiLU (channels-last two-pass for FLUX.2) | @@ -201,9 +273,9 @@ The package `README.md` carries a selection matrix for the cases where several k ## References -- [Performance Optimization](./performance-optimization) -- [Attention Backends](./attention_backends) -- [Quantization](./quantization) -- [Profiling](./profiling) +- [Performance Optimization](/docs/sglang-diffusion/performance-optimization) +- [Attention Backends](/docs/sglang-diffusion/attention_backends) +- [Quantization](/docs/sglang-diffusion/quantization) +- [Profiling](/docs/sglang-diffusion/profiling) - [`sglang/kernels/ops/diffusion`](https://github.com/sgl-project/sglang/tree/main/python/sglang/kernels/ops/diffusion) — source and selection matrix - [RFC #29630](https://github.com/sgl-project/sglang/issues/29630) — the unified `sglang.kernels` namespace diff --git a/docs/docs/sglang-diffusion/index.mdx b/docs/docs/sglang-diffusion/index.mdx index 1b06e4a22bcb..b6879791dea3 100644 --- a/docs/docs/sglang-diffusion/index.mdx +++ b/docs/docs/sglang-diffusion/index.mdx @@ -30,7 +30,7 @@ sglang serve --model-path Qwen/Qwen-Image --port 30010 ## Start Here - [Installation](/docs/sglang-diffusion/installation): install SGLang Diffusion and platform dependencies -- [Supported Models and Optimization Compatibility](/docs/sglang-diffusion/compatibility_matrix): check supported model families, long-tail coverage, and optimization support +- [Supported Models](/docs/sglang-diffusion/compatibility_matrix): browse supported model families, tasks, and public checkpoints - [CLI](/docs/sglang-diffusion/api/cli): run one-off generation jobs or launch a persistent server - [OpenAI-Compatible API](/docs/sglang-diffusion/api/openai_api): send image and video requests to the HTTP server - [ComfyUI plugin](/docs/sglang-diffusion/comfyui): use SGLang from ComfyUI in server mode or as a per-step DiT backend diff --git a/docs/docs/sglang-diffusion/models_with_ar.mdx b/docs/docs/sglang-diffusion/models_with_ar.mdx index 1d5c38167342..010883045fa0 100644 --- a/docs/docs/sglang-diffusion/models_with_ar.mdx +++ b/docs/docs/sglang-diffusion/models_with_ar.mdx @@ -116,7 +116,7 @@ export HCCL_NPU_SOCKET_PORT_RANGE="24200-24399" ## Best practices -GLM-Image example for Ascend A3 2 cards (4 devices) +GLM-Image example for Ascend A3 Series, 2 cards (4 devices) ```bash # Terminal 1 : server with AR model export HCCL_IF_BASE_PORT=23000 @@ -124,7 +124,7 @@ export HCCL_HOST_SOCKET_PORT_RANGE="23000-23199" export HCCL_NPU_SOCKET_PORT_RANGE="23200-23399" sglang serve --model-path /path/to/zai-org/GLM-Image/vision_language_encoder/ \ --tokenizer-path /path/to/zai-org/GLM-Image/processor/ --enable-multimodal \ ---cuda-graph-bs 1 --device npu --attention-backend ascend --image-processor-backend pil \ +--cuda-graph-bs-decode 1 --device npu --attention-backend ascend --image-processor-backend pil \ --tp-size 4 --port ${PORT} --mem-fraction-static 0.4 ``` Second terminal with diffusion server: diff --git a/docs/docs/sglang-diffusion/models_with_pe.mdx b/docs/docs/sglang-diffusion/models_with_pe.mdx index 777228f83361..ba27a3618fe9 100644 --- a/docs/docs/sglang-diffusion/models_with_pe.mdx +++ b/docs/docs/sglang-diffusion/models_with_pe.mdx @@ -1,76 +1,186 @@ --- -title: "Diffusion Models with Prompt Enhancement (PE)" -description: "Run ERNIE-Image with built-in prompt enhancement or a separate SGLang-served PE model." +title: "Prompt enhancement" +description: "Connect an independently deployed SGLang LLM or VLM to diffusion image and video APIs." --- - -## Quick Start - -By default, the diffusion server loads the PE model in-process with SGLang's native Ministral3 implementation. Deploy the PE model as a separate SGLang server when it needs independent resources or scaling. This document uses `baidu/ERNIE-Image` as an example. - -Run the model with the built-in native PE implementation (default): - -```bash -# Terminal 1: launch server -sglang serve --model-path baidu/ERNIE-Image --port ${PORT} -``` -```bash -# Terminal 2: launch client -curl -X POST http://${HOST}:${PORT}/v1/images/generations \ - -H "Content-Type: application/json" \ - -d '{ - "prompt": "This is a photograph depicting an urban street scene. Shot at eye level, it shows a covered pedestrian or commercial street. Slightly below the center of the frame, a cyclist rides away from the camera toward the background, appearing as a dark silhouette against backlighting with indistinct details. The ground is paved with regular square tiles, bisected by a prominent tactile paving strip running through the scene, whose raised textures are clearly visible under the light. Light streams in diagonally from the right side of the frame, creating a strong backlight effect with a distinct Tyndall effect—visible light beams illuminating dust or vapor in the air and casting long shadows across the street. Several pedestrians appear on the left side and in the distance, some with their backs to the camera and others walking sideways, all rendered as silhouettes or semi-silhouettes. The overall color palette is warm, dominated by golden yellows and dark browns, evoking the atmosphere of dusk or early morning.", - "height": 1264, - "width": 848, - "num_inference_steps": 50, - "guidance_scale": 4.0, - "use_pe": true - }' -``` - -Run the model with an SGLang-served PE model (high performance): - -```bash -# Terminal 1: launch SGLang PE model server -sglang serve --model-path /path/to/baidu/ERNIE-Image/pe/ --port ${PE_PORT} -``` + +Run an SRT server for prompt expansion and a diffusion server for generation. +The diffusion HTTP frontend calls SRT's `/v1/chat/completions` once before +constructing the generation request. The enhancer does not replace the +diffusion model's text encoder or run inside its GPU workers. + +```text +Application -> diffusion HTTP server -> SRT enhancer server + <- rewritten prompt + <- image/video <- diffusion pipeline +``` + +Prompt enhancement is opt-in per request. It changes conditioning and can +change intent or quality; it is not a lossless acceleration feature. Evaluate +the chosen enhancer and template on your workload before enabling it. + +## Deploy two services + +This example uses Linux with NVIDIA CUDA and two GPUs, one for each server. +Install [SGLang](/docs/get-started/install) for SRT and +[SGLang Diffusion](/docs/sglang-diffusion/installation) for diffusion. Separate +environments and hosts are supported; the diffusion host needs HTTP access to SRT. + +Start a text-only enhancer: + +```bash +CUDA_VISIBLE_DEVICES=0 sglang serve \ + --model-path Qwen/Qwen3-4B-Instruct-2507 \ + --port 31000 +``` + +Create `enhancer.json` on the diffusion host: + +```json +{ + "base_url": "http://127.0.0.1:31000/v1", + "model": "Qwen/Qwen3-4B-Instruct-2507" +} +``` + +Start diffusion in another terminal: + ```bash -# Terminal 2: launch diffusion model server with PE server -sglang serve --model-path /path/to/baidu/ERNIE-Image/ \ - --pe-server-url "http://${HOST}:${PE_PORT}" \ - --port ${PORT} +CUDA_VISIBLE_DEVICES=1 sglang serve \ + --model-path Tongyi-MAI/Z-Image-Turbo \ + --prompt-enhancer-config enhancer.json \ + --port 30000 ``` -```bash -# Terminal 3: launch client -curl -X POST http://${HOST}:${PORT}/v1/images/generations \ - -H "Content-Type: application/json" \ - -d '{ - "prompt": "This is a photograph depicting an urban street scene. Shot at eye level, it shows a covered pedestrian or commercial street. Slightly below the center of the frame, a cyclist rides away from the camera toward the background, appearing as a dark silhouette against backlighting with indistinct details. The ground is paved with regular square tiles, bisected by a prominent tactile paving strip running through the scene, whose raised textures are clearly visible under the light. Light streams in diagonally from the right side of the frame, creating a strong backlight effect with a distinct Tyndall effect—visible light beams illuminating dust or vapor in the air and casting long shadows across the street. Several pedestrians appear on the left side and in the distance, some with their backs to the camera and others walking sideways, all rendered as silhouettes or semi-silhouettes. The overall color palette is warm, dominated by golden yellows and dark browns, evoking the atmosphere of dusk or early morning.", - "height": 1264, - "width": 848, - "num_inference_steps": 50, - "guidance_scale": 4.0, - "use_pe": true - }' + +Keep the SRT port separate from diffusion's HTTP and internal ports; diffusion +reserves `--port + 1` for its broker. For separate hosts, bind SRT to its service +interface with `--host` and update +`base_url`. Restrict network access and configure SRT authentication. Do not +expose an unauthenticated enhancer publicly. A shared enhancer can serve +multiple diffusion servers; size its concurrency and memory independently. + +Send a generation request: + +```bash +curl --fail-with-body http://127.0.0.1:30000/v1/images/generations \ + -H 'Content-Type: application/json' \ + -d '{ + "prompt": "A red teapot on a windowsill, morning light", + "enhance_prompt": true, + "size": "512x512", + "seed": 42, + "response_format": "b64_json" + }' ``` -For a memory-constrained in-process deployment, the native PE decoder supports -layerwise offload: +Omit `enhance_prompt` or set it to `false` to use the original prompt without +calling SRT. With the OpenAI Python SDK, pass `extra_body={"enhance_prompt": True}` +to `images.generate()`. + +The same switch works with `/v1/images/edits` and `/v1/videos`, including +multipart requests (`-F 'enhance_prompt=true'`). Image responses expose the +rewritten text as `data[].revised_prompt`; video responses use `revised_prompt`. +Video creation waits for enhancement before returning the queued job. All `n` +outputs share one rewritten prompt. + +Include enhancement in client-observed end-to-end latency. The response's +`inference_time_s` measures diffusion execution, not the preceding SRT call. + +## Configure the enhancer + +The configuration file is read at server startup. Its supported fields are: + +| Field | Meaning | +| --- | --- | +| `base_url` | SRT OpenAI API base URL, including `/v1` | +| `model` | SRT model ID or its configured served model name | +| `system_prompt` | Optional replacement for the built-in, task-aware rewriting instruction | +| `include_images` | Forward reference images to a VLM; defaults to `false` | +| `timeout` | HTTP operation timeout in seconds; defaults to `60` | +| `api_key_env` | Name of an environment variable on the diffusion server containing the SRT API key | +| `generation_kwargs` | SRT chat-completion options, such as `max_tokens`, `temperature`, `chat_template_kwargs`, or `response_format` | + +Default generation options are `max_tokens=512` and `temperature=0`. +`generation_kwargs` cannot replace the model, messages, single-output or +non-streaming settings, or enable tools. Choose a non-thinking model or configure +its supported chat template to suppress reasoning; only message `content` is +used as the diffusion prompt. + +The enhancer receives a JSON text message containing `task` (`image`, +`image_edit`, or `video`) and `prompt`. It returns the replacement prompt as +text. Seed, negative prompt, references and diffusion sampling options are not +rewritten. The diffusion frontend reuses HTTP connections and makes no enhancer +calls during synthetic warmup. + +Timeouts return HTTP 504. Upstream failures, empty responses and truncated +completions return HTTP 502 before diffusion is queued. There is no automatic +retry or fallback to the original prompt. An unconfigured server rejects +`enhance_prompt=true` with HTTP 400. + +## Choose a model combination + +There is no enhancer or diffusion model allowlist in this integration. Protocol +compatibility does not establish quality: use a template appropriate for the +target checkpoint and verify prompt adherence. + +| Diffusion input | Enhancer requirements | +| --- | --- | +| Natural-language image/video prompts, such as Z-Image, FLUX, Qwen-Image, Wan, LTX or MiniMax-H3 | An SRT-served instruction model with chat completions; the default template is a starting point | +| Image editing or image-to-video | Text-only instruction rewriting, or an SRT VLM with `include_images=true` for image-aware rewriting | +| Structured captions, such as Ideogram 4 or LingBot-Video-MoE | A model-specific `system_prompt` and preferably schema-constrained `response_format`; the default prose template is unsuitable | +| ERNIE-Image or LongCat with built-in rewriting | Disable the built-in rewriter for the request when using this generic enhancer, to avoid rewriting twice | + +For a VLM, deploy an image-capable SRT model and set `include_images=true` in +its configuration. Uploaded images are sent as data URLs, so the SRT server +does not need access to the diffusion host's filesystem. HTTP image URLs must +be reachable from SRT. Text-only models should leave this option disabled. +Audio and video references are not forwarded to the enhancer; it cannot inspect +their content. Image edits forward the supplied images; video creation forwards +the primary image reference, not model-specific reference bundles. + +For [Ideogram 4](/cookbook/diffusion/Ideogram/Ideogram4), instruct the enhancer +to produce the checkpoint's JSON caption schema, including +`high_level_description` and `compositional_deconstruction`. Set the matching +JSON schema through `generation_kwargs.response_format`. JSON text is passed +through unchanged, not converted to prose. Valid JSON alone does not guarantee +the correct caption schema or equivalent output to a hosted service's expander. + +This integration applies to the three HTTP image/video endpoints above, +including their normal parallelism, caching and disaggregated diffusion paths. +It does not add enhancement to offline `sglang generate`, realtime streaming, +mesh or action endpoints. For those workflows, call SRT in your application +first and pass its returned text as the diffusion prompt. For reproducibility, +store the rewritten prompt and replay it with `enhance_prompt=false`; a fixed +diffusion seed does not fix the enhancer's output. + +## ERNIE-Image's checkpoint-provided enhancer + +ERNIE-Image also has a separate, existing PE integration. By default, diffusion +loads its native Ministral3 enhancer in-process: ```bash -sglang serve --model-path baidu/ERNIE-Image \ - --layerwise-offload-components pe \ - --port ${PORT} +sglang serve --model-path baidu/ERNIE-Image --port 30000 ``` -This option streams PE decoder layers from CPU and can increase prompt-enhancement -latency. It does not apply when `--pe-server-url` selects an external PE server. +Use `use_pe=false` to skip it for a request. For a memory-constrained deployment, +`--layerwise-offload-components pe` streams the native PE decoder from CPU. + +To serve the checkpoint-provided PE through SRT, use a local checkpoint's PE +directory with its tokenizer and retain the ERNIE-specific protocol: + +```bash +CUDA_VISIBLE_DEVICES=0 sglang serve \ + --model-path /models/ERNIE-Image/pe --port 31000 + +CUDA_VISIBLE_DEVICES=1 sglang serve \ + --model-path /models/ERNIE-Image \ + --pe-server-url http://127.0.0.1:31000 --port 30000 +``` -## Support matrix - -| Model | Built-in PE | SGLang PE Server | -|-------|-------------|------------------| -| ERNIE-Image | ✅ | ✅ | - -## Ascend NPU Environment +`--pe-server-url` uses the checkpoint's PE tokenizer/template and SRT's +`/generate`; it is not the generic chat-completion integration. Keep using it +when you need ERNIE's trained PE behavior. If you instead use +`--prompt-enhancer-config`, send both `enhance_prompt=true` and `use_pe=false`. +For LongCat, use `enable_prompt_rewrite=false` with the generic enhancer. +These request flags skip native rewriting, not loading its model components. -See [Diffusion models with AR stage like GLM-Image](/docs/sglang-diffusion/models_with_ar#ascend-npu-env). +For Ascend setup, see [models with AR stages](/docs/sglang-diffusion/models_with_ar#ascend-npu-env). diff --git a/docs/docs/sglang-diffusion/performance-optimization.mdx b/docs/docs/sglang-diffusion/performance-optimization.mdx index e0206083d937..abc900c94ad1 100644 --- a/docs/docs/sglang-diffusion/performance-optimization.mdx +++ b/docs/docs/sglang-diffusion/performance-optimization.mdx @@ -12,11 +12,28 @@ The docs use "output-preserving" instead of promising bit-exact "lossless" becau ## Start Here -1. Pick a serving or generation mode from [Deployment and Performance Modes](./deployment_cookbook). `--performance-mode auto` is the default; use `speed` when the model fits in GPU memory and latency matters most, `memory` when GPU memory is the bottleneck, and `manual` when every performance flag should be explicit. -2. Choose the right attention backend from [Attention Backends](./attention_backends). -3. Use [Sequence Parallelism](./ring_sp_performance) only when the model and video shape benefit from sequence splitting. -4. Use [Inference Batching](./dynamic_batching) for concurrent compatible requests during serving. -5. Use [Profiling](./profiling) before changing several levers at once. +1. Pick a serving or generation mode from [Deployment and Performance Modes](/docs/sglang-diffusion/deployment_cookbook). `--performance-mode auto` is the default; use `speed` when the model fits in GPU memory and latency matters most, `memory` when GPU memory is the bottleneck, and `manual` when every performance flag should be explicit. +2. Choose the right attention backend from [Attention Backends](/docs/sglang-diffusion/attention_backends). +3. Use [Sequence Parallelism](/docs/sglang-diffusion/ring_sp_performance) only when the model and video shape benefit from sequence splitting. +4. Use [Inference Batching](/docs/sglang-diffusion/dynamic_batching) for concurrent compatible requests during serving. +5. Use [Profiling](/docs/sglang-diffusion/profiling) before changing several levers at once. + +## Choose a request quality tier + +`--quality` is cumulative: a broader tier never drops an optimization from a +stricter tier. + +| Tier | Optimization boundary | +| --- | --- | +| `lossless` (default) | The selected deployment's reference execution plus all unconditional bit-exact replacements | +| `extra-high` | Everything in `lossless`, plus only request-gated DiT/VAE kernel fusions; the tier does not itself enable sparse, caching, or other approximate paths | +| `high` | Everything in `extra-high`, plus model-owned high-only paths such as an audited Cache-DiT policy or lower-precision VAE decode | + +Use `extra-high` when you want to isolate fusion wins from approximate +acceleration. A tier may be a no-op when the active model has no eligible path. +Separately configured quantization, attention, or caching options still apply. +See [Fused Kernels](/docs/sglang-diffusion/fused_kernels) for the current +request-gated families and their numerical contracts. ## Output-Preserving / Lossless-Style Levers @@ -39,37 +56,42 @@ These settings should preserve model behavior while changing residency, parallel --performance-mode You want a safe preset for speed or memory without overriding explicit flags. - Deployment and Performance Modes + Deployment and Performance Modes + + + Breakable CUDA graph + A supported pipeline serves a fixed set of shapes and eager execution is launch-bound. + CLI reference Offload, FSDP, CFG parallelism GPU memory, multi-GPU residency, or CFG branch splitting is the main bottleneck. - Deployment and Performance Modes + Deployment and Performance Modes Sequence parallelism Long image/video sequences need sequence-level parallelism. - Sequence Parallelism + Sequence Parallelism --encoder-parallel Text/image encoding is a visible share of the request and the DiT replica sits idle during it. - Encoder Parallelism + Encoder Parallelism Attention backend Kernel choice dominates DiT latency or memory. - Attention Backends + Attention Backends Fused kernels You want to know which elementwise chains are already fused, or to opt into the request-gated set. - Fused Kernels + Fused Kernels Dynamic batching Serving many compatible requests concurrently. - Inference Batching + Inference Batching @@ -95,22 +117,22 @@ These techniques can change the denoising path, numerical representation, or gen Cache-DiT Skips selected DiT block or step computation based on cache decisions. - Cache-DiT + Cache-DiT TeaCache Reuses residuals when consecutive denoising steps are similar enough. - TeaCache + TeaCache Progressive resolution Runs early denoising at lower latent resolution for supported pipelines. - Progressive Resolution Generation + Progressive Resolution Generation Quantization Uses lower-precision transformer weights or activations. - Quantization + Quantization @@ -119,19 +141,42 @@ These techniques can change the denoising path, numerical representation, or gen 1. Establish a baseline with the target model, resolution, frame count, step count, and GPU type. 2. Select `--performance-mode` and explicit residency or parallelism flags. -3. Tune attention backend and batching for the deployment pattern. -4. Profile if the bottleneck is unclear. -5. Add caching, progressive resolution, or quantization only after comparing output quality against your acceptance target. +3. Compare `quality=lossless` with `quality=extra-high` to isolate the request-gated fusion set. +4. Compare breakable CUDA graph against eager execution for supported fixed-shape pipelines. Pass every served resolution to `--warmup-resolutions` and confirm capture in the server log. Models with request-gated DiT fusions cannot combine those fusions with a graph captured from the lossless branches. +5. Tune attention backend and batching for the deployment pattern. +6. Profile if the bottleneck is unclear. +7. Add `quality=high`, caching, progressive resolution, or quantization only after comparing output quality against your acceptance target. + +## Per-model tuning starting points + +Warmup and breakable CUDA graph (BCG) solve different problems. `--warmup-mode request` runs a warmup copy derived from the first request to prime one-time compilation and caches; it does not remove recurring Python launches from each denoising step. BCG captures supported DiT segments and can reduce that recurring launch overhead for captured shapes. Use the table below as a first experiment, then keep a lever only when profiling confirms that it addresses the active bottleneck. + +| Observed bottleneck or constraint | Example models | First experiment | What to verify | +| --- | --- | --- | --- | +| The first execution of a shape pays substantial compilation or cache setup | ERNIE-Image-Turbo, GLM-Image, FastWan / TurboWan family, FLUX.2-klein-4B, LingBot-World, LongLive, Cosmos3, SANA, LongCat-Image-Edit-Turbo | Compare `--warmup-mode request` with `off` | Separate warmup time, first-real-request latency, and warmed steady-state latency. Request-based warmup is primarily a benchmark aid and still consumes time before the first real request completes. | +| A supported fixed-shape pipeline shows recurring host-launch gaps between GPU kernels | GLM-Image, Z-Image-Turbo, Qwen-Image-2512, Ideogram-4, LongCat-Image, LTX-2 / LTX-2.3, SANA-1.5 1.6B, JoyEcho | Compare eager execution with `--enable-breakable-cuda-graph` | Confirm graph capture and replay in the server log, then compare steady-state latency and GPU memory. Pass additional production shapes through `--warmup-resolutions`; without it, BCG captures only the model's default warmup resolution. | +| The model does not fit while both DiT stages are resident | Wan2.2-T2V-A14B, Wan2.2-I2V-A14B, LingBot-Video-MoE | Start with `--dit-layerwise-offload` | Verify peak GPU memory first, then measure the transfer overhead. Add warmup separately if first-run compilation is also material. | +| Dynamic masks, metadata, or image conditions make graph replay ineffective | Qwen-Image-Edit family, FireRed-Image-Edit | Keep eager execution as the baseline | Profile before attempting BCG. Dynamic per-step host work can outweigh graph replay savings, and unsupported pipelines fall back to eager execution. | +| Kernels or collectives already dominate a many-step workload | Qwen-Image, FLUX.1-dev, FLUX.2-dev, LTX-2, Wan2.1 14B, HunyuanVideo, MOVA | Warm the baseline, then profile before enabling another lever | A small host gap limits the benefit available from BCG. Prioritize kernels, attention, parallelism, or residency according to the trace. | +| A one-shot or very few-step workload cannot amortize warmup | FastVideo-FastH3 (4-step) | Start with `--warmup-mode off` | Compare end-to-end latency including warmup. Use request warmup only when intentionally separating cold-start setup from the measured request. | + +The example assignments come from single-GPU NVIDIA B300/GB300 profiles and are directional rather than an exhaustive compatibility list. A model can match more than one row as resolution, frame count, step count, parallelism, or GPU type changes; the measured bottleneck takes precedence over the model name. + +BCG is enabled only for model and pipeline configurations accepted by the runtime support check. It captures the default warmup shape automatically. Use `--warmup-resolutions` for additional served resolutions; for video and variable prompt lengths, set `--warmup-num-frames` and `--bcg-text-buckets` to cover the intended workload. Requests that do not match a captured signature fall back to eager execution. + +Warmup and BCG do not intentionally trade output quality for speed, but that is not a guarantee of byte-identical output. Different execution paths can introduce numerical differences, and some pipelines customize the scheduler or schedule used by a synthetic warmup request. Before production rollout, compare output hashes when bitwise stability is required, otherwise run the project's quality acceptance check. + +Performance results are configuration-dependent. Record the exact checkpoint revision, GPU, precision, resolution, frame count, step count, parallelism, command line, and whether the measurement includes warmup. Re-profile with [Profiling](/docs/sglang-diffusion/profiling) on the target workload rather than transferring a percentage from another model or shape. ## Diagnostics -[Profiling](./profiling) is not an optimization technique by itself. It belongs in the performance workflow because it tells you which stage, kernel, or denoising step is worth optimizing before you change multiple levers. +[Profiling](/docs/sglang-diffusion/profiling) is not an optimization technique by itself. It belongs in the performance workflow because it tells you which stage, kernel, or denoising step is worth optimizing before you change multiple levers. ## References -- [Deployment and Performance Modes](./deployment_cookbook) -- [Attention Backends](./attention_backends) -- [Fused Kernels](./fused_kernels) -- [Sequence Parallelism](./ring_sp_performance) -- [Caching Strategies](./caching-acceleration) -- [Profiling](./profiling) +- [Deployment and Performance Modes](/docs/sglang-diffusion/deployment_cookbook) +- [Attention Backends](/docs/sglang-diffusion/attention_backends) +- [Fused Kernels](/docs/sglang-diffusion/fused_kernels) +- [Sequence Parallelism](/docs/sglang-diffusion/ring_sp_performance) +- [Caching Strategies](/docs/sglang-diffusion/caching-acceleration) +- [Profiling](/docs/sglang-diffusion/profiling) diff --git a/docs/docs/sglang-diffusion/production_metrics.mdx b/docs/docs/sglang-diffusion/production_metrics.mdx new file mode 100644 index 000000000000..3e9c95020d29 --- /dev/null +++ b/docs/docs/sglang-diffusion/production_metrics.mdx @@ -0,0 +1,69 @@ +--- +title: "Production Metrics" +description: "Monitor SGLang Diffusion requests, queues, stages and LoRA state with Prometheus." +--- + +## Enable metrics + +Enable metrics on an NVIDIA CUDA deployment, for example: + +```bash +sglang serve --model-path black-forest-labs/FLUX.2-klein-4B \ + --num-gpus 1 --enable-metrics --port 30000 +curl http://localhost:30000/metrics +``` + +Metrics are opt-in. Disabled metrics do not scan queues or collect LoRA status. +Enabled metrics add host-side bookkeeping, not GPU synchronization or collectives. + +## Metric reference + +All diffusion metrics carry `role` and `replica` labels. `replica` is the +scheduler endpoint; only each DP replica's leader publishes, so TP/SP ranks do +not multiply request counts. The table lists additional labels. + +| Metric | Type | Labels | Description | +| --- | --- | --- | --- | +| `sglang:diffusion_num_queue_reqs` | Gauge | none | Original generation requests waiting for their first dispatch. | +| `sglang:diffusion_num_running_reqs` | Gauge | none | Number of diffusion generation requests dispatched by the scheduler and not yet finished. | +| `sglang:diffusion_requests_total` | Counter | `status`, `is_warmup` | Completed diffusion generation requests. Dynamic batches are counted per original scheduler request after the merged output is split. | +| `sglang:diffusion_request_latency_seconds` | Histogram | `status`, `is_warmup` | Scheduler acceptance to completion, excluding HTTP preprocessing, media encoding and response delivery. | +| `sglang:diffusion_queue_time_seconds` | Histogram | `is_warmup` | Time spent waiting in the diffusion scheduler queue. | +| `sglang:diffusion_generation_batch_size` | Histogram | `stop_reason` | Generation batch size selected by the diffusion scheduler at dispatch time. | +| `sglang:diffusion_stage_host_latency_seconds` | Histogram | `stage` | Host wall time around a stage, not GPU kernel execution time. Step labels are normalized to `DenoisingStep`. | +| `sglang:diffusion_lora_loaded_adapters` | Gauge | none | Number of loaded diffusion LoRA adapters. | +| `sglang:diffusion_lora_active_modules` | Gauge | none | Number of diffusion modules with active LoRA adapters. | +| `sglang:diffusion_lora_active_adapters` | Gauge | none | Number of unique active diffusion LoRA adapters. | +| `sglang:diffusion_lora_module_active` | Gauge | `module` | Whether a diffusion module currently has an active LoRA adapter. | + +Request counts refer to original scheduler requests, not generated images, +denoising steps or distributed shards. `status` is `success` or `error`; +`is_warmup` is `true` or `false`. Queue and running gauges include warmup. +Stage observations include warmup and count stage invocations, not requests; +asynchronous GPU work can complete in a later stage. For synchronized diagnostic +timings, use `SGLANG_DIFFUSION_SYNC_STAGE_PROFILING=1` separately, accepting its +synchronization overhead. LoRA gauges update at startup and after LoRA control operations. + +## Disaggregated serving + +In disaggregated serving, the head (`role="server"`) records the original request +lifecycle, including role handoffs and errors. Queue time ends at the first +encoder dispatch; intermediate waits remain part of request latency. Workers +report their own stage and LoRA metrics, not duplicate completed requests. +The generation-batch histogram currently describes monolithic scheduling only. + +Single-host pool mode exposes all child metrics through the head's `/metrics`. +A scrape aggregates only processes sharing that host's metrics directory. +For standalone remote roles, pass `--enable-metrics` to each process and scrape +its `--host`/`--port` as well as the head. Each role serves a metrics-only HTTP +endpoint. Use a separate, empty `PROMETHEUS_MULTIPROC_DIR` per server launch if +you set it yourself; otherwise SGLang creates and owns a temporary directory. +Do not share this directory between independent servers or reuse stale files. + +## Query throughput + +Successful, non-warmup request throughput: + +```promql +sum(rate(sglang:diffusion_requests_total{status="success",is_warmup="false"}[5m])) +``` diff --git a/docs/docs/sglang-diffusion/quantization.mdx b/docs/docs/sglang-diffusion/quantization.mdx index b526a0059dce..90873d9c8a93 100644 --- a/docs/docs/sglang-diffusion/quantization.mdx +++ b/docs/docs/sglang-diffusion/quantization.mdx @@ -72,7 +72,7 @@ paths: | Component path | Quantized checkpoint behavior | | --- | --- | | `transformer`, `transformer_2`, `unconditional_transformer`, `audio_dit`, `video_dit` | Uses the SGLang transformer quantization adapters documented below. | -| `text_encoder*`, `image_encoder*` | Passes detected metadata to the registered native encoder. Loading continues only when that implementation constructs compatible quantized layers; unknown or incompatible combinations fail closed. | +| `text_encoder*`, `image_encoder*` | Prefers a compatible registered native encoder. Otherwise, a standard top-level `quantization_config` recognized by the installed Transformers version is delegated to its `from_pretrained` path; unsupported formats and metadata locations fail closed. | | `vae`, `video_vae`, `audio_vae` | A standard top-level Diffusers `quantization_config` is delegated to `AutoModel.from_pretrained`. Native-only VAEs and nested/compression metadata fail closed. | | Library-managed Transformers or Diffusers components | Delegates to the upstream `from_pretrained` path and inherits its format support and validation behavior. The local PE model uses this path; compatible formats remain model-specific. | | Native auxiliary components that load raw state dicts | Quantized checkpoints are rejected before model construction until that component has a quantized materialization implementation. This includes connectors, duration heads, bridges, diffusion decoders, sound tokenizers, spatial upsamplers, and vocoders. | @@ -210,12 +210,12 @@ backend. Auto-detected; omit --quantization. Each layer dispatches to its serialized W4A4 or INT8 ConvRot kernel. CUDA requires SM75+; TP must preserve each format's quantization and ConvRot group boundaries. Offload is supported and FSDP is not. - comfy-nvfp4-full-precision - Safetensors with serialized nvfp4 and optional row-wise int8_tensorwise layer metadata + comfy-nvfp4 + Safetensors with serialized nvfp4 and optional scalar/row-wise int8_tensorwise layer metadata --component-weights-paths.text_encoder MiniMax-H3 native Qwen3-VL encoder - None - Auto-detected; omit explicit quantization. Preserves packed storage, high-nibble-first weights, swizzled block scales, and AWQ input pre-scales. Each active NVFP4 matrix is dequantized for BF16/FP16 compute, so this is a memory path rather than a native FP4 speed path. + comfy-kitchen for NVFP4 matmul + Auto-detected; omit explicit quantization. Preserves high-nibble-first weights, swizzled block scales, and optional AWQ input pre-scales. Layers declaring full_precision_matrix_mult use BF16/FP16 compute with packed storage. Other NVFP4 layers use dynamic activation quantization and NVFP4 matmul on NVIDIA SM100+. Companion INT8 embeddings support scalar or per-row scales without expanding the full table. quanto-int8 @@ -373,7 +373,7 @@ For a pipeline whose primary DiT is named `transformer`, the shorter MiniMax-H3 supports this path while preserving its required FP32 patch, timestep, and output projections. See the -[MiniMax-H3 cookbook](/cookbook/diffusion/MiniMax/MiniMax-H3#7-feature-contracts-and-advanced-recipes) +[MiniMax-H3 cookbook](/cookbook/diffusion/MiniMax/MiniMax-H3#9-feature-contracts-and-advanced-recipes) for its distributed serving recipe. ### MXFP4 Online Quantization @@ -432,7 +432,7 @@ projections take that path. `kitchen_int8` is approximate and is not a consistency ground-truth mode. The BF16 path is unchanged when `comfy-kitchen` is not installed. See the -[MiniMax-H3 cookbook](/cookbook/diffusion/MiniMax/MiniMax-H3#7-feature-contracts-and-advanced-recipes) +[MiniMax-H3 cookbook](/cookbook/diffusion/MiniMax/MiniMax-H3#9-feature-contracts-and-advanced-recipes) for the 24 GB offload recipe, including why `vae` must stay out of `--layerwise-offload-components`. @@ -466,13 +466,15 @@ sglang generate \ Each pattern is matched against the full layer prefix (e.g. `layers.0.attention.to_q`). A layer is skipped and left unquantized if its prefix contains any of the given patterns. -## Transformers Component BnB4 +## Transformers-managed Quantized Components -Model components that already have a native Transformers loading path can load -serialized BitsAndBytes 4-bit checkpoints with a standard top-level -`quantization_config`. Plain checkpoints keep using an available native SGLang -implementation. For example, replace FLUX's T5 component with the official -Diffusers checkpoint: +Model components that already have a native Transformers loading path delegate +self-describing checkpoints whose standard top-level `quantization_config` is +recognized by the installed Transformers version. Successful loading still +depends on that backend's optional dependencies, platform, and model/checkpoint +compatibility. An available native SGLang implementation remains preferred when +it can restore the same format. For example, replace FLUX's T5 component with +the official BitsAndBytes checkpoint: ```bash Command sglang serve \ @@ -481,11 +483,12 @@ sglang serve \ diffusers/FLUX.1-dev-bnb-4bit/text_encoder_2 ``` -The quantized component must stay resident. SGLang rejects component or -layerwise offload, nonstandard metadata locations, and native-only component -fallbacks for this path instead of silently changing the checkpoint contract. -Diffusion DiT components declared under the Diffusers library use the separate -quantization backends documented above. +Transformers owns format validation and places the quantized component directly +on its resident device. SGLang rejects component/layerwise offload, FSDP, +nonstandard metadata locations, unsupported upstream formats, and native-only +fallbacks instead of silently changing the checkpoint contract. Diffusion DiT +components declared under the Diffusers library use the separate quantization +backends documented above. ## Validated ModelOpt Checkpoints @@ -771,6 +774,14 @@ sglang generate \ --save-output ``` +Qwen-Image 2.1 also supports independent NVFP4 DiT and native language-encoder +component directories through `--component-paths.transformer` and +`--component-paths.text_encoder`. Private max-calibrated exports passed B200 +generation, editing, transparent RGBA, offload, and TP2 checks. These checks +do not cover arbitrary exports or RTX 5090. See its +[cookbook](/cookbook/diffusion/Qwen-Image/Qwen-Image-2.1#nvfp4-components) for +component-loading instructions and hardware requirements. + ### Notes - Use `--transformer-path` for mixed ModelOpt NVFP4 transformer repos or local @@ -781,8 +792,9 @@ sglang generate \ primary `--transformer-path` override targets only `transformer`. Use a per-component override such as `--transformer-2-path` only when you intentionally want a non-default `transformer_2`. -- On Blackwell, the diffusion ModelOpt NVFP4 path defaults to FlashInfer - TensorRT-LLM FP4 GEMM (`flashinfer_trtllm`). +- On B200, the diffusion ModelOpt NVFP4 path defaults to FlashInfer + TensorRT-LLM FP4 GEMM (`flashinfer_trtllm`). On SM120, including RTX 5090, + the default is `auto`; TensorRT-LLM FP4 GEMM does not support that architecture. - The published Qwen Image NVFP4 exports keep the `img_mod`/`txt_mod` modulation projections and first/last transformer blocks in BF16. - Qwen-Image NVFP4 does not always improve latency at 1024x1024. On B200, the @@ -913,13 +925,21 @@ startup rejects a degree that cuts a row-parallel matrix inside a GGML block. | MiniMax-H3 `fl2va`, pruned AdaLN curve | [`unsloth/MiniMax-H3-GGUF`](https://huggingface.co/unsloth/MiniMax-H3-GGUF) `minimax_h3_fl2va_pruned-Q4_K.gguf` (10.7 GiB loaded DiT) | 1x H200 (141 GiB) | 2-step t2va 1344x768, 107 frames, H.264 + AAC | | MiniMax-H3 `fl2va`, pruned AdaLN curve | `minimax_h3_fl2va_pruned-Q4_K.gguf` | 1x GB300 (CUDA 13, PyTorch 2.13) | 50-step t2va 1344x768, 107 frames, H.264 + AAC; 105.38 s and 80.88 GB peak | | MiniMax-H3 `fl2va`, pruned AdaLN curve | `minimax_h3_fl2va_pruned-Q4_K.gguf` | 2x GB300, TP2 (CUDA 13, PyTorch 2.13) | 2-step t2va 1344x768, 107 frames, H.264 + AAC; 7.55 s and 51.90 GB peak per rank | +| Qwen-Image 2.1 | Private native-name Q4_0 exports; DiT 3.91 GiB, encoder 7.03 GiB | 1x B200; separate TP2 check | 1024px/40-step generation, editing, and transparent RGBA; each component and both together; combined offload matched resident pixels | -The DiT loads at 17.5 GiB against 61.7 GiB for the BF16 checkpoint. Weight +The unpruned MiniMax-H3 DiT loads at 17.5 GiB against 61.7 GiB for the BF16 checkpoint. Weight fidelity was checked tensor-by-tensor against the BF16 reference: cosine 1.00000 for the F32/BF16 tensors and 0.9973 for Q4_K/Q4_0. -Not validated in the measurements above: any other quantization type, the -`ref2va` partition, and a BF16-vs-GGUF output comparison. +The MiniMax-H3 measurements do not validate other quantization types, the +`ref2va` partition, or a BF16-vs-GGUF output comparison. + +Qwen-Image 2.1's Q4_0 exports use the base component configs and native tensor +names. Its [cookbook](/cookbook/diffusion/Qwen-Image/Qwen-Image-2.1#gguf-components) +describes how to load compatible component files and choose precision settings. +These private exports establish loading +and execution compatibility, not general output quality or compatibility with +other GGUF exports. ## Nunchaku (SVDQuant) @@ -1179,8 +1199,8 @@ sglang generate \ ``` The selected SRT backend must provide an MXFP8 kernel on NVIDIA or ROCm. On -Ascend, the hardware requirement remains A5 series or newer; -`npu_dynamic_mx_quant` is not available on A2/A3. +Ascend, the hardware requirement remains 950PR/DT Series or newer; +`npu_dynamic_mx_quant` is not available on A2/A3 Series. ## MXFP8 Offline Quantization (msmodelslim) @@ -1213,8 +1233,8 @@ sglang generate \ --save-output ``` -> **Hardware requirement:** Ascend A5 series or newer. `npu_dynamic_dual_level_mx_quant` -> and `npu_dual_level_quant_matmul` are not available on A2/A3. +> **Hardware requirement:** Ascend 950PR/DT Series or newer. `npu_dynamic_dual_level_mx_quant` +> and `npu_dual_level_quant_matmul` are not available on A2/A3 Series. > > **Note:** Online MXFP4 weight quantization is experimental. The offline msmodelslim > flow uses pre-quantized weights and may produce different numerical results. diff --git a/docs/docs/sglang-diffusion/realtime_models.mdx b/docs/docs/sglang-diffusion/realtime_models.mdx index 9e2fe41217dd..cbe90371354d 100644 --- a/docs/docs/sglang-diffusion/realtime_models.mdx +++ b/docs/docs/sglang-diffusion/realtime_models.mdx @@ -41,7 +41,7 @@ details: - [SANA-WM](/cookbook/diffusion/SANA-WM/SANA-WM) For the complete model list, see -[Supported Models and Optimization Compatibility](./compatibility_matrix). +[Supported Models](./compatibility_matrix). ## Causal Cache Controls diff --git a/docs/docs/sglang-diffusion/ring_sp_performance.mdx b/docs/docs/sglang-diffusion/ring_sp_performance.mdx index c81c025d0f23..a8dbce0d57ec 100644 --- a/docs/docs/sglang-diffusion/ring_sp_performance.mdx +++ b/docs/docs/sglang-diffusion/ring_sp_performance.mdx @@ -303,8 +303,9 @@ See [Encoder Parallelism](/docs/sglang-diffusion/encoder_parallel). Cross-node ring support is model-specific, not a property of the launch -flags alone — see [Supported Models and Optimization Compatibility](/docs/sglang-diffusion/compatibility_matrix) -for which models have it. Passing `--ring-degree > 1` for a model that only +flags alone. Confirm support in the model cookbook and see +[Parallelism](/docs/sglang-diffusion/parallelism) for topology requirements. +Passing `--ring-degree > 1` for a model that only has single-node Ulysses may either raise or, in some cases, silently compute incorrect output; check the model's cookbook page before assuming cross-node scaling is supported. diff --git a/docs/docs/sglang-diffusion/support_new_models.mdx b/docs/docs/sglang-diffusion/support_new_models.mdx index 11462336786b..5733f9cd788a 100644 --- a/docs/docs/sglang-diffusion/support_new_models.mdx +++ b/docs/docs/sglang-diffusion/support_new_models.mdx @@ -12,12 +12,11 @@ though the runtime is split into separate folders. The files are split by runtime responsibility. For a new model, read the request path first: -1. `registry.py` chooses the model family, sampling params, and pipeline config. -2. `configs/pipeline_configs/{model}.py` defines model-specific denoising and +1. `configs/pipeline_configs/{model}.py` defines model-specific denoising and decoding behavior. -3. `runtime/pipelines/{model}.py` wires modules into stages. -4. `runtime/pipelines_core/stages/` runs the shared stage logic. -5. `runtime/models/` contains native model components only when the architecture +2. `runtime/pipelines/{model}.py` wires modules into stages. +3. `runtime/pipelines_core/stages/` runs the shared stage logic. +4. `runtime/models/` contains native model components only when the architecture cannot be reused. That is the dependency direction. Avoid making a model PR that requires readers @@ -33,6 +32,30 @@ utilities, and common action-policy helpers. Model packages may call these helpers. Keep ownership in shared runtime folders unless the code is truly architecture-specific. +## Place helpers with their owners + +Use the narrowest existing owner before adding a utility module: + +| Scope | Location | +| --- | --- | +| Used by one file, or specific to one operation | A private helper in the consuming file; inline short one-off expressions | +| Shared within a domain | A purpose-named module in that domain, such as `runtime/realtime/video.py` or `runtime/layers/attention/mask_strategy.py` | +| Shared across domains, without model or pipeline semantics | An existing focused module under `runtime/utils/`, such as `argparse.py`, `process.py`, or `precision.py` | + +Do not create a top-level `utils/` package or grow a catch-all `utils.py` or +`common.py`. Split large mixed-responsibility files along ownership boundaries, +not arbitrary line counts. A helper folder is warranted only when several +cohesive modules need it, not for a single function or hypothetical reuse. + +Model code must not import pipeline stages. Put contracts shared by models and +stages in a lower-level domain module; for example, realtime cache keys belong +under `runtime/realtime/`. Keep GPU initialization, monkey patches, and model +loading out of generic utility imports. + +When moving internal helpers, update all callers, tests, and cookbook examples +together. Preserve documented registration and serving entry points; do not +add re-export chains just to retain obsolete internal utility paths. + ## Out-of-Tree Models and Pipelines An installed package can register native component models and a pipeline @@ -100,7 +123,6 @@ behavior. | Area | Add or edit when | Typical file | | --- | --- | --- | -| Registry | Always, unless extending an already registered family | `python/sglang/multimodal_gen/registry.py` | | Runtime parameters | The request schema differs from existing models | `configs/sample/{model}.py` | | Pipeline config | Denoising, decoding, precision, position encoding, or CFG hooks differ | `configs/pipeline_configs/{model}.py` | | Pipeline wiring | The model needs a new stage layout or module list | `runtime/pipelines/{model}.py` | @@ -113,11 +135,10 @@ behavior. For a new native architecture, the common minimum is: -1. `registry.py` -2. `configs/sample/{model}.py` -3. `configs/pipeline_configs/{model}.py` -4. `runtime/pipelines/{model}.py` -5. `runtime/models/dits/{model}.py` +1. `configs/sample/{model}.py` +2. `configs/pipeline_configs/{model}.py` +3. `runtime/pipelines/{model}.py` +4. `runtime/models/dits/{model}.py` Every extra file should map to model behavior that existing code cannot express clearly. @@ -359,6 +380,54 @@ class MyModelPipeline(LoRAPipeline, ComposedPipelineBase): EntryClass = [MyModelPipeline] ``` +#### Reuse component loaders + +Most components should keep the default loader for their role (transformer, +text encoder, VAE, scheduler, or tokenizer). For an auxiliary module that accepts +`model_cls(**config)` and loads an unchanged state dict, select +`PlainStateDictComponentLoader` instead of adding a model-specific loader class: + +```python +from sglang.multimodal_gen.runtime.loader.component_loaders.component_loader import ( + PlainStateDictComponentLoader, +) + + +class MyModelPipeline(ComposedPipelineBase): + _required_config_modules = ["transformer", "queryformer", "text_projection"] + component_loaders = { + "queryformer": PlainStateDictComponentLoader, + "text_projection": PlainStateDictComponentLoader, + } + + # Wire the stages as in the examples above. +``` + +The mapping uses exact pipeline component names, including aliases declared in +`_extra_config_module_map`. It is local to this pipeline; omitted components keep +their existing dispatch. An explicitly selected loader fails on errors instead +of falling back to a different implementation. + +Register each module through `EntryClass` in its model file, or +`ModelRegistry.register_model` for an out-of-tree package. The shared loader: + +- Resolves the registered class from `config.json`'s `_class_name`, falling back + to the architecture in `model_index.json`, and passes non-metadata config + fields to its constructor. +- Loads a single safetensors file or an indexed sharded checkpoint using the + shared checkpoint selector, with strict key and shape checks. +- Honors exact overrides such as `--component-weights-paths.queryformer PATH` + and `--component-precisions.queryformer bf16`. Precision defaults to the + pipeline's `dit_precision`. +- Leaves eval mode and final CPU/GPU placement to the shared loading lifecycle. + It does not implement TP/FSDP weight sharding, quantization, or direct GPU + loading. Explicit quantization and direct-GPU-loading overrides are rejected. + +Keep a specialized loader when the component needs a config-object constructor, +weight remapping or fusion, sharding, or a non-standard checkpoint format. Such +a loader can also be selected through `component_loaders`; registering a model +alone does not opt it into the plain state-dict protocol. + ### 4. Last-Resort Before-Denoising Stage A `BeforeDenoisingStage` is not a catch-all replacement for the native stages. @@ -427,19 +496,27 @@ native integration contract. ### 6. Registry -Register the family once the sampling params and pipeline config exist. +Define a `register()` function in `configs/pipeline_configs/{model}.py`. The +runtime auto-discovers it on startup and calls it to register the sampling +params and pipeline config. ```python -register_configs( - model_family="my_model", - sampling_param_cls=MyModelSamplingParams, - pipeline_config_cls=MyModelPipelineConfig, - hf_model_paths=["org/my-model"], -) +# python/sglang/multimodal_gen/configs/pipeline_configs/my_model.py +def register(): + from sglang.multimodal_gen.registry import register_configs + + register_configs( + sampling_param_cls=MyModelSamplingParams, + pipeline_config_cls=MyModelPipelineConfig, + hf_model_paths=["org/my-model"], + model_detectors=[lambda hf_id: "my-model" in hf_id.lower()], + ) ``` -The pipeline file is discovered through its `EntryClass`; do not add a second -pipeline registry unless the existing registry requires it. +`model_detectors` matches a model path or `model_index.json` `_class_name` when +the Hugging Face path varies; see `wan.py` or `qwen_image21.py` for real +examples. The pipeline file is discovered through its `EntryClass`; do not add +a second pipeline registry unless the existing registry requires it. ## Verify the Port @@ -456,8 +533,9 @@ implementation. baseline. 5. If the model supports LoRA, CFG parallelism, or disaggregation, test each feature explicitly. -6. Add or update docs, examples, or the compatibility matrix when users need a - new launch command. +6. Add or update the cookbook, examples, and + [Supported Models](./compatibility_matrix) catalog when users need a new + launch command. Common failure points: diff --git a/docs/docs/supported-models.mdx b/docs/docs/supported-models.mdx index bcfcf89a75ca..33ae19614b5b 100644 --- a/docs/docs/supported-models.mdx +++ b/docs/docs/supported-models.mdx @@ -84,4 +84,13 @@ SGLang supports model families across text generation, retrieval, and reward wor > RLHF and reward scoring pipelines optimized for production latency. + + Typed choice, score, and yes or no answers with a probability for every option. + diff --git a/docs/docs/supported-models/decision_models.mdx b/docs/docs/supported-models/decision_models.mdx new file mode 100644 index 000000000000..60200987c1f6 --- /dev/null +++ b/docs/docs/supported-models/decision_models.mdx @@ -0,0 +1,274 @@ +--- +title: Decision models +description: Answer typed choice, score, and yes or no questions with a probability for every option from a chat model, without generating text. +--- + +`/v1/decisions` turns a chat model into a decision model. You send an input and a list of typed questions, and each answer comes back with the probability of every option, read from the model's next-token scores at the answer position. No text is generated and no output is parsed. It needs no special checkpoint: any generation model served with a Jinja chat template can answer, as long as the checks below pass for it. + + + `/v1/decisions` is an SGLang extension under `/v1`, like `/v1/score` and `/v1/rerank`. It is not part of the OpenAI API, so call it over HTTP rather than through an OpenAI SDK method. Until a release contains it, install a [nightly build](/docs/get-started/install#nightly-builds), which is built from the main branch. + + +## Supported models + +| Model | Example HuggingFace identifier | Notes | +| --- | --- | --- | +| **Qwen3.8-27B** | `Qwen/Qwen3.8-27B` | Validated on one H200 in BF16, with and without NEXTN speculative decoding. See the [model page](/cookbook/autoregressive/Qwen/Qwen3.8-27B). | +| **Qwen3.5-35B-A3B** | `Qwen/Qwen3.5-35B-A3B` | Validated on one H200 in BF16. See the [model page](/cookbook/autoregressive/Qwen/Qwen3.5). | + +Other chat models are served when the answer labels are single tokens at the answer position and the answer does not start inside a reasoning block. The server checks these per request and returns a 400 that names the reason when one fails, see [Errors](#errors). It does not check how the chat template renders the question or where the template itself would put an answer, so inspect `prompt_token_ids` from `return_prompt_token_ids` before relying on a new model. + +## Launch command + +```bash Command +python -m sglang.launch_server \ + --model-path Qwen/Qwen3.8-27B \ + --host 127.0.0.1 --port 30000 +``` + +No extra flag is needed. The same server keeps serving `/v1/chat/completions` and `/generate`, and decisions can run alongside that traffic. A server launched with `--enable-mis` or `--dllm-algorithm`, or one that uses a built-in conversation template instead of the tokenizer's Jinja template, refuses decisions. + +## Example request + +```python Example +import requests + +URL = "http://127.0.0.1:30000" + +response = requests.post( + f"{URL}/v1/decisions", + json={ + "input": "I've been trying to connect my Stripe account for 3 days and the integration keeps failing. I'm losing sales.", + "questions": [ + { + "id": "team", + "type": "choice", + "question": "Which team should handle this ticket?", + "options": [ + {"name": "billing", "description": "Payment or subscription issues"}, + {"name": "technical", "description": "Bugs or integration problems"}, + {"name": "sales", "description": "Pricing or account questions"}, + ], + }, + { + "id": "frustration", + "type": "score", + "question": "How frustrated is the customer?", + "levels": ["Calm", "Frustrated but civil", "Very angry"], + }, + { + "id": "urgent", + "type": "yes_no", + "question": "The customer needs an answer today.", + }, + ], + }, + timeout=60, +) +response.raise_for_status() +for question_id, answer in response.json()["answers"].items(): + print(question_id, answer) +``` + +Each answer contains: + +- `type`: the question type, `choice`, `score`, or `yes_no`. +- `probabilities`: one value per option name, level index, or `yes` and `no`, summing to 1. For a yes or no question, `probabilities["yes"]` is the answer. +- `choice` for a choice question (the most probable option) or `score` for a score question (the probability-weighted mean level index). +- `label_mass`: the full-vocabulary probability of the answer labels at the answer position. A low value means the model puts most of its probability outside the offered answers. + +Questions are answered independently and scored together in one batch. To route the ticket in this example, send it to the chosen team, and hand it to a person when the top probability falls below a threshold you validate on your own labeled tickets. + +## How answers are computed + +The server renders each question as one user message with the model's chat template and thinking turned off, and labels the answers `A` to `Z` for options, `0` to `9` for levels, and `yes` and `no`. It checks that each label is one distinct token at the answer position, then runs one prefill pass per question through the scoring path of `/v1/score` and reads the next-token log-probabilities of the labels over the full vocabulary. For a yes or no question with log-probabilities `lp_yes` and `lp_no` and request temperature `T`: + +- `probabilities["yes"] = exp(lp_yes / T) / (exp(lp_yes / T) + exp(lp_no / T))`, which equals a softmax of the two label logits divided by `T`, because the vocabulary normalizer cancels. +- `label_mass = exp(lp_yes) + exp(lp_no)`, which does not depend on `T`. + +Choice and score questions work the same way over their labels. None of these values is a calibrated probability that the decision is correct. Validate any threshold on labeled data from your workload. + +## Request and response reference + +Request fields: + +| Field | Type | Meaning | +| --- | --- | --- | +| `input` | string, object, or array | What the questions are about. Objects and arrays are rendered as compact JSON. Must not be blank. | +| `questions` | list of questions | At least one. Each question has a unique `id`, which keys its answer. | +| `temperature` | number above 0, default 1 | Divides the label logits before the softmax over the labels. It does not change `label_mass`. | +| `chat_template_kwargs` | object, default empty | Extra chat template arguments, applied after `--default-chat-template-kwargs`. The thinking toggle that the chat template or its reasoning parser names stays off in either case. | +| `prompt_format_version` | integer, optional | Pins the prompt wording. A server that serves another version returns 400. | +| `return_prompt_token_ids` | boolean, default false | Adds `prompt_token_ids` and `label_token_ids` to each answer. | +| `model` | string, optional | Echoed in the response, as `default` when omitted. It does not select a LoRA adapter, and the `base:adapter` form returns 400. | + +Question fields: + +| Type | Fields | Labels | +| --- | --- | --- | +| `choice` | `id`, `question`, and `options`: 2 to 26 objects with a `name` and an optional `description` | `A` to `Z` in list order | +| `score` | `id`, `question`, and `levels`: 2 to 10 level descriptions, lowest first | `0` to `9` | +| `yes_no` | `id`, `question`, and optional `yes` and `no` descriptions | `yes` and `no` | + +`question`, descriptions, and levels can also be JSON objects or arrays. Each question becomes one user message: the input, a blank line, the question line, one line per option, level, or described yes or no answer, and a closing instruction to answer with one label only. An option renders as `A: billing` or `A: billing - `, a level as `0: `, and a described yes or no answer as `yes: `. + +The response has `object` set to `decisions`, `model`, `prompt_format_version`, `answers` keyed by question id, and `usage` with `prompt_tokens` for all questions and `completion_tokens` 0. + +## Errors + +The request fails with HTTP 400 for: + +- an unknown question type or an unknown field anywhere in the request +- option or level counts outside the ranges above +- a blank input, question, question id, or level +- a repeated question id, or option names that are blank, contain control or line break characters, or repeat another name after trimming and case folding +- a label that is not one distinct token at the answer position for the served tokenizer +- a prompt that does not fit the model's context length +- a chat template that always thinks before answering or starts every answer with a reasoning block, `chat_template_kwargs` that set the thinking toggle to anything but `false`, a rendered prompt that leaves a reasoning block open, or a model whose reasoning parser expects answers to start inside a reasoning block when the rendered prompt does not close one +- a `prompt_format_version` other than the served one, or a `model` that names a LoRA adapter +- a server launched with `--enable-mis` or `--dllm-algorithm`, or one that uses a built-in conversation template, whether named with `--chat-template`, loaded from a JSON template file, or inferred from the model path +- a model whose chat route uses a built-in encoder instead of a chat template, a tokenizer that does not encode the rendered chat text back to the same ids, or a model that is not a generation model +- a chat template that raises an error for the question message, or a server launched with `--skip-tokenizer-init` + +Errors about one question name its id or its position in `questions`. + +## Pin and replay the prompt + +The server owns the prompt wording of `/v1/decisions` and versions it. Every response carries `prompt_format_version`, and a change to the wording ships as a new version. Send `prompt_format_version` to fail loudly instead of receiving answers from different wording after a server upgrade. + +To keep an answer reproducible independently of the server's wording, ask for the scored ids and replay them through `/v1/score`, which only scores the ids you send: + +```python Example +import requests + +URL = "http://127.0.0.1:30000" + +body = { + "input": "The integration keeps failing.", + "questions": [{"id": "urgent", "type": "yes_no", "question": "The customer needs an answer today."}], + "prompt_format_version": 1, + "return_prompt_token_ids": True, +} +answer = requests.post(f"{URL}/v1/decisions", json=body, timeout=60).json()["answers"]["urgent"] +replay = requests.post( + f"{URL}/v1/score", + json={ + "query": [], + "items": [answer["prompt_token_ids"]], + "label_token_ids": [answer["label_token_ids"]], + "apply_softmax": True, + "return_token_logprobs": True, + }, + timeout=60, +).json() +print(answer["probabilities"], replay["scores"][0]) +``` + +The replayed scores equal the answer's probabilities when both requests use the same `temperature` and run in the same cache state. If the decision set `temperature`, send the same value in the `/v1/score` body. A replay right after the decision reuses its cached prefix, so it can differ slightly, as described in [Reproducibility](#reproducibility). + +## Thinking models + +Qwen3.8-27B and Qwen3.5-35B-A3B think by default. `/v1/decisions` turns thinking off for every question, refuses a request that turns it back on, and refuses a rendered prompt that leaves a reasoning block open, so the answer is never read inside the reasoning. Chat requests to the same server keep the model's default, and launching with `--reasoning-parser qwen3` affects chat only. + +## Reproducibility + +These models mix full and linear attention. Answer probabilities can move by up to several hundredths (0.07 in our checks), and `label_mass` by up to about 0.14, between cold and prefix-cached requests and across batch compositions, while the chosen option stayed the same in our checks. On Qwen3.5-35B-A3B, `--disable-radix-cache` gave identical values across sequential repeats, at the cost of prefix reuse. + +## System One compatible API + +`POST /v1/systemone` serves the same decisions in the request and response shape of the System One API, with a `state`, a map of `noul`, `choice`, and `score` questions keyed by your ids, and one answer per id. Clients written for that API, including the official TypeSafe SDKs, get decisions by pointing their base URL at the server, with the exceptions listed below. The route uses the same rendering, label checks, and scoring as `/v1/decisions`, and refuses the same servers, chat templates, and prompts, listed in [Errors](#errors). Its request fields are checked as described below. + +```bash Command +pip install typesafe-sdk +``` + +```python Example +from typesafe_sdk import Choice, Noul, Score, TypeSafeClient + +# A server started without --api-key accepts any key, so a placeholder works. +client = TypeSafeClient(base_url="http://127.0.0.1:30000", api_key="unused") +result = client.system_one( + "I've been trying to connect my Stripe account for 3 days and the integration keeps failing.", + { + "team": Choice( + instructions="Which team should handle this ticket?", + criteria={"billing": None, "technical": "Bugs or integration problems", "sales": None}, + ), + "urgent": Noul(instructions="The customer needs an answer today."), + "frustration": Score( + instructions="How frustrated is the customer?", + criteria=["Calm", "Frustrated but civil", "Very angry"], + ), + }, +) +for name, answer in result.answers.items(): + print(name, answer) +``` + +What to expect from this route: + +- Any `model` name is accepted, including the SDK default `jev-latest`, unless it names a LoRA adapter after a colon, as in `base:adapter`. The response `model` is always the served model. +- A `noul` answer is the probability of yes. A `choice` answer has the most probable option and a `score` answer the probability-weighted mean level, each with `probabilities` and `confidence`, and a score answer echoes its levels in `legend`. +- `confidence` follows the formulas TypeSafe publishes for its adapter. It is a statistic of the returned probabilities, which are renormalized over the answer labels and ignore probability outside them. It is not calibrated, it can read 0 for a score split between distant levels, and thresholds tuned on another provider may not transfer. +- Each answer also carries `x_label_mass`, the full-vocabulary probability of its labels, which `/v1/decisions` returns as `label_mass`. The Python SDK drops fields it does not define, so read it from `result.raw_http_response.json()`. +- A choice takes 1 to 255 options and a score 1 to 10 levels. Option names that are blank, contain control or line break characters, or repeat another name after trimming and case folding are refused, and so are blank or null levels, as on `/v1/decisions`. +- Beyond 26 options, every option gets a two-letter label (`AA`, `AB`, and so on) checked to be one token for the served tokenizer. These labels include common words and have unequal priors, so answers above 26 options can depend on option order. More than 26 options also need a Hugging Face tokenizer that reports its added tokens, and a chat template with an added token between the message and the answer position whose following text tokenizes the same on its own, as the Qwen models do with ``. Without them, a choice is limited to 26 options. +- `instructions` are optional. A choice or score question without them drops its question line, and a `noul` question is answered from its `true` and `false` descriptions. A `noul` question with neither, such as the SDK's bare `Noul()`, is refused. +- Unknown top-level fields are ignored. Unknown keys inside a question or inside `noul` criteria are refused, and so are the `/v1/decisions` fields `temperature`, `prompt_format_version`, and `return_prompt_token_ids` with any value other than null. `chat_template_kwargs` is accepted as an SGLang extension. +- Unlike `/v1/decisions`, an empty `state` and empty or blank question ids are accepted, as the System One schema allows. +- Invalid requests return 422 with the location of the error, other refusals return 400, and neither is retried by the SDKs. +- `usage.input_tokens` counts the prompt of every question, so the state is counted once per question, and `usage.output_tokens` is 0. +- The SDKs' `models.list()` and response `request_id` are not supported, because `GET /v1/models` keeps its OpenAI format and no request id header is sent. +- The SDKs time out after 10 seconds and retry, so raise their timeout for requests with many questions over a long state. A request whose client gives up still finishes its encoding and part of its prefill, so each retry repeats at least the encoding. + +## Build decisions without `/v1/decisions` + +When you need your own prompt or labels, build the prompt on the client and score it through `/v1/score`, which takes token ids and one label list per item. This needs `transformers` and `jinja2` on the client. Every label must be exactly one token at the answer position: + +```python Example +import requests +from transformers import AutoTokenizer + +MODEL = "Qwen/Qwen3.8-27B" +URL = "http://127.0.0.1:30000" +tokenizer = AutoTokenizer.from_pretrained(MODEL) + +options = ["billing", "technical", "sales"] +letters = ["A", "B", "C"] +prompt = ( + "I've been trying to connect my Stripe account for 3 days and the integration keeps failing.\n\n" + "Question: Which team should handle this ticket?\n" + + "\n".join(f"{letter}: {name}" for letter, name in zip(letters, options)) + + "\nAnswer with the letter of one option only." +) +text = tokenizer.apply_chat_template( + [{"role": "user", "content": prompt}], add_generation_prompt=True, enable_thinking=False, tokenize=False +) +input_ids = tokenizer.encode(text, add_special_tokens=False) +label_ids = [] +for letter in letters: + ids = tokenizer.encode(text + letter, add_special_tokens=False) + if ids[: len(input_ids)] != input_ids or len(ids) != len(input_ids) + 1: + raise ValueError(f"Label {letter} is not one token at the answer position") + label_ids.append(ids[-1]) + +scores = requests.post( + f"{URL}/v1/score", + json={"query": [], "items": [input_ids], "label_token_ids": [label_ids], "apply_softmax": True}, + timeout=60, +).json()["scores"][0] +print(dict(zip(options, scores))) +``` + +`enable_thinking=False` renders a closed think block, so the answer token follows it directly. Do not score a prompt that leaves the think block open, because the scored position then falls inside the reasoning. + +Structured output gives the same decision through generation. A one-token regex such as `(A|B|C)` with `temperature` 0 and logprobs returns probabilities renormalized over the labels, equal to the `/v1/score` values for one-character labels in the same cache state. A JSON schema with `enum` fields fills several fields in one response, but gives no clean per-option probability, because an enum value can span several tokens. See [Structured outputs](/docs/advanced_features/structured_outputs) for the request formats. + +## Limitations + +- Labels must be one token at the answer position. `/v1/decisions` assigns one-token labels for you, and a client-built prompt must do the same. +- For `yes_no`, `label_mass` counts only the lowercase `yes` and `no` tokens. The model also puts probability on `Yes` and `No`, so this value reads lower than for choice and score questions even on clear cases, while `probabilities["yes"]` is unaffected. +- The server tells from the chat template and its reasoning parser whether answers could start inside a reasoning block, using the parser's reasoning tags in the generation prompt and in the template's own rendering of a finished answer. Reasoning that shows neither is not detected, so check `prompt_token_ids` for such a model. Some templates of models that do not reason still contain reasoning tags without ever closing a block in the generation prompt, and those models are refused. +- A default temperature in `--preferred-sampling-params` can reach scoring requests and scale both the probabilities and `label_mass` when a decision shares a batch with generation or runs under speculative decoding. Leave it unset for decisions. +- The server tokenizes each question's full prompt once on the HTTP event loop, as `/v1/score` does for its items, and lets other requests run between questions. Tokenizers whose label check falls back to the full prompt encode it once more per label. +- A decision takes a single `input`, not a chat history. To decide about a conversation, pass the history as the input text or as a JSON array. diff --git a/docs/docs/supported-models/generative_models.mdx b/docs/docs/supported-models/generative_models.mdx index 5c7cc94e77b9..6811d2954c72 100644 --- a/docs/docs/supported-models/generative_models.mdx +++ b/docs/docs/supported-models/generative_models.mdx @@ -268,6 +268,11 @@ in the GitHub search bar. tiiuae/Falcon-H1-34B-Instruct TII's hybrid Mamba-Transformer architecture combining attention and state-space models for efficient long-context inference. + + Mamba-Codestral (7B, Mamba2) + mistralai/Mamba-Codestral-7B-v0.1 + Mistral AI's pure Mamba2 state-space model for code generation; uses selective state spaces instead of attention, so it holds no KV cache. + Hunyuan-Large (389B, MoE) tencent/Tencent-Hunyuan-Large @@ -303,5 +308,10 @@ in the GitHub search bar. JetBrains/Mellum2-12B-A2.5B-Base, JetBrains/Mellum2-12B-A2.5B-Thinking JetBrains' Qwen3-MoE-based code generation model with interleaved sliding-window/full attention, per-layer-type RoPE, and per-layer dense/sparse MLP routing. + + GigaChat 3.5 + ai-sage/GigaChat3.5-432B-A28B + GigaChat's hybrid MoE model: per-layer mix of DeepSeek-style MLA full attention and Qwen3 Gated-Delta-Net (GDN) linear attention, DeepSeek-style routed + shared experts, and multiple multi-token-prediction (MTP) heads for speculative decoding. Emits tool calls in GCML format. + diff --git a/docs/docs/supported-models/mindspore_models.mdx b/docs/docs/supported-models/mindspore_models.mdx index f47dc20dda4b..3e5115fc88c0 100644 --- a/docs/docs/supported-models/mindspore_models.mdx +++ b/docs/docs/supported-models/mindspore_models.mdx @@ -37,7 +37,7 @@ Current SGLang-MindSpore supports Qwen3 and DeepSeek V3/R1 models. This doc uses Use the following script for offline inference: -```python Offline Infer +```python Offline Inference import sglang as sgl # Initialize the engine with MindSpore backend diff --git a/docs/docs/supported-models/multimodal_language_models.mdx b/docs/docs/supported-models/multimodal_language_models.mdx index ac706a6c1410..f7e5010a3e79 100644 --- a/docs/docs/supported-models/multimodal_language_models.mdx +++ b/docs/docs/supported-models/multimodal_language_models.mdx @@ -171,6 +171,18 @@ in the GitHub search bar. NVIDIA Nemotron Nano v2 VL enables multi-image reasoning and video understanding, along with strong document intelligence, visual Q&A and summarization capabilities. It builds on Nemotron Nano V2, a hybrid Mamba-Transformer LLM, in order to achieve higher inference throughput in long document and video scenarios. Use --trust-remote-code. You may need to adjust --max-mamba-cache-size [default is 512] to fit memory constraints. + + NVIDIA Cosmos3 Reasoner (Nano) + nvidia/Cosmos3-Nano + The understanding tower (Qwen3-VL-based reasoner) of NVIDIA's Cosmos3 world model, served as a standalone VLM for image and video understanding from the unified diffusers-layout checkpoint. The generation (diffusion) tower is dropped at load time. + Launch with --model-type llm; see Cosmos3 LLM serving. Pass --language-model-only for text-only serving. + + + NVIDIA Cosmos3-Edge (4B) + nvidia/Cosmos3-Edge + The 4B dense understanding tower of Cosmos3-Edge: an Arcee-structure text model with a SigLIP2 vision tower and an Edge-specific spatial-merge projector, supporting image and video understanding. + Launch with --model-type llm; see Cosmos3 LLM serving. Pass --language-model-only for text-only serving. + Ernie4.5-VL baidu/ERNIE-4.5-VL-28B-A3B-PT @@ -289,6 +301,11 @@ SGLang supports video input for Vision-Language Models (VLMs), enabling temporal Qwen/Qwen3-VL-235B-A22B-Instruct The processor gathers video_data, runs Qwen's frame sampler, and merges the resulting features with text tokens before inference. + + Ling-VL (Ling-3.0-flash-VL) + inclusionAI/Ling-3.0-flash-VL + InclusionAI's KDA/MLA hybrid MoE VLM (125B total / ~5.1B active) built on the Ling-3.0-flash backbone with a Qwen3-ViT-family vision tower. Image and video inputs are expanded by the Bailing processor with centered mRoPE positions. Requires --trust-remote-code. Audio is not supported. + GLM-4v (4.5V, 4.1V, MOE) zai-org/GLM-4.5V @@ -349,16 +366,55 @@ response = requests.post(url, json=data) print(response.text) ``` +## Serving the LLM tower of Cosmos3 checkpoints + +`nvidia/Cosmos3-Nano` and `nvidia/Cosmos3-Edge` are unified world-model checkpoints in a diffusers layout, so `sglang serve` routes them to the diffusion runtime by default (see the [Cosmos3 cookbook page](/cookbook/diffusion/Cosmos/Cosmos3) for media generation). To serve the understanding tower as a vision-language model instead, force the LLM backend with `--model-type llm`: + +```bash Launch Server +sglang serve \ + --model-path nvidia/Cosmos3-Nano \ + --model-type llm \ + --host 0.0.0.0 --port 30000 +``` + +The same command serves `nvidia/Cosmos3-Edge` by swapping the model path. Both accept image and video inputs through the OpenAI-compatible API: + +```bash Example Request +curl http://localhost:30000/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "nvidia/Cosmos3-Nano", + "messages": [{ + "role": "user", + "content": [ + {"type": "text", "text": "Describe this image."}, + {"type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"}} + ] + }], + "max_tokens": 128 + }' +``` + +To serve only the text model (skipping the vision tower entirely), add `--language-model-only`. In this mode `/model_info` reports `has_image_understanding: false`, and requests containing image or video inputs are rejected: + +```bash Text-Only Launch +sglang serve \ + --model-path nvidia/Cosmos3-Nano \ + --model-type llm \ + --language-model-only \ + --host 0.0.0.0 --port 30000 +``` + ## Usage Notes ### Performance Optimization -For multimodal models, you can use the `--keep-mm-feature-on-device` flag to optimize for latency at the cost of increased GPU memory usage: +For multimodal models, you can use `--mm-feature-transport=cuda_ipc` to optimize for latency at the cost of increased GPU memory usage: - **Default behavior**: Multimodal feature tensors are moved to CPU after processing to save GPU memory -- **With `--keep-mm-feature-on-device`**: Feature tensors remain on GPU, reducing device-to-host copy overhead and improving latency, but consuming more GPU memory +- **With `--mm-feature-transport=cuda_ipc`**: Feature tensors stay in a bounded GPU pool shared over CUDA IPC, reducing device-to-host copy overhead and improving latency, but consuming more GPU memory -Use this flag when you have sufficient GPU memory and want to minimize latency for multimodal inference. +Use this option when you have sufficient GPU memory and want to minimize latency for multimodal inference. ### Multimodal Inputs Limitation diff --git a/docs/docs/supported-models/support_new_models.mdx b/docs/docs/supported-models/support_new_models.mdx index ba7836732787..bc2f7ac0ba37 100644 --- a/docs/docs/supported-models/support_new_models.mdx +++ b/docs/docs/supported-models/support_new_models.mdx @@ -61,7 +61,7 @@ should give the same text output and very similar prefill logits: ``` - Get the SGLang output: ```bash Command - python3 -m sglang.bench_one_batch --correct --model [new model] + python3 -m sglang.benchmark.one_batch --correct --model [new model] ``` ### Add the Model to the Test Suite diff --git a/docs/index.mdx b/docs/index.mdx index 92e3c916ef4b..e871f613c9b8 100644 --- a/docs/index.mdx +++ b/docs/index.mdx @@ -50,7 +50,7 @@ import { PopularModels } from "/src/snippets/_popular_models.jsx"; Native support across Hardware Platforms - including NVIDIA, AMD, Intel Xeon, Google TPU, and Ascend NPU accelerators. + including NVIDIA, AMD, Intel Xeon, Google TPU, Ascend NPU, and Moore Threads MUSA accelerators. @@ -90,7 +90,7 @@ It is designed to deliver low-latency and high-throughput inference across a wid }} > Qwen3.8-Flash-Next: Day-0 Support in SGLang - {"Qwen3.8-Flash-Next: Day-0 Support in SGLang"} + {"Accelerating Long-Context and Agentic Inference with NVFP4 KV Cache"}

- {"August 26, 2026"} + {"September 16, 2026"}

Fast Engine Recovery: Sub-Second Engine Restart for SGLang via Weight Cache Daemon - {"Fast Engine Recovery: Sub-Second Engine Restart for SGLang via Weight Cache Daemon"} + {"SGLang and Miles Add Day-0 Support for DeepSeek-V4.1"}

- {"August 21, 2026"} + {"September 10, 2026"}

Chasing the Batch-1 Floor: Ling-3.0-flash Speculative Decode on Blackwell - {"Chasing the Batch-1 Floor: Ling-3.0-flash Speculative Decode on Blackwell"} + {"Running DeepSeek-V4-Flash and Kimi-K3 on Consumer Hardware with SSD Expert Pack"}

- {"August 21, 2026"} + {"August 29, 2026"}

Mooncake for Miles: From Fragmented Rollout Data to Efficient Bulk I/O - {"Mooncake for Miles: From Fragmented Rollout Data to Efficient Bulk I/O"} + {"Infer-forge: Harness, Loop, and Graph Engineering Around SGLang"}

- {"August 20, 2026"} + {"August 28, 2026"}

Pushing the Limits of Serving DeepSeek-V4-Pro - {"Pushing the Limits of Serving DeepSeek-V4-Pro"} + {"MiniMax-H3 on 8\u00d7H200: 1.95\u00d7 Lossless, Up to 6.24\u00d7 at 0.76\u20130.91 SSIM"}

- {"August 19, 2026"} + {"August 27, 2026"}

Miles v0.1: Production-level Post-training - {"Miles v0.1: Production-level Post-training"} + {"Qwen3.8-Flash-Next: Day-0 Support in SGLang"}

- {"August 18, 2026"} + {"August 26, 2026"}

diff --git a/docs/scripts/check_cookbook_configs.mjs b/docs/scripts/check_cookbook_configs.mjs index ec77b6438697..9e661b138c8b 100755 --- a/docs/scripts/check_cookbook_configs.mjs +++ b/docs/scripts/check_cookbook_configs.mjs @@ -300,6 +300,33 @@ for (const path of walk(CONFIGS)) { checkH3("B200 1x8", { hw: "b200", nodes: 1, gpus_per_node: 8, placement: "resident" }, { tp_size: 1, ulysses_degree: 8, ring_degree: 1 }); checkH3("H100 1x4", { hw: "h100", nodes: 1, gpus_per_node: 4, placement: "resident" }, { tp_size: 2, ulysses_degree: 2, ring_degree: 1 }); checkH3("H200 2x8", { hw: "h200", nodes: 2, gpus_per_node: 8, placement: "resident" }, { tp_size: 1, ulysses_degree: 8, ring_degree: 2 }); + for (const hw of ["gb200", "gb300"]) { + checkH3(`${hw} 1x4`, { hw, nodes: 1, gpus_per_node: 4, placement: "resident" }, { tp_size: 1, ulysses_degree: 4, ring_degree: 1 }, hw === "gb300"); + checkH3(`${hw} 2x4`, { hw, nodes: 2, gpus_per_node: 4, placement: "resident" }, { tp_size: 1, ulysses_degree: 4, ring_degree: 2 }, false); + const selection = selectionOf({ hw, nodes: 2, gpus_per_node: 4 }); + const encoder = config.overlayDims.find((dim) => dim.id === "encoder").options.find((option) => option.id === "auto"); + if (!encoder.flags(selection).includes("--encoder-parallel replicate")) { + fail(where, `${hw} cross-node auto encoder must replicate`); + } + if (config.runModes(selection).includes("docker")) { + fail(where, `${hw} must not advertise an unvalidated Docker command`); + } + } + for (const extra of [ + { hw: "gb200" }, + { hw: "gb300", mode: "i2va" }, + { hw: "gb300", weights: "ref2va", mode: "v2v" }, + { hw: "gb300", quality: "extra-high" }, + { hw: "gb300", outputs: "2" }, + { hw: "gb300", precision: "fp8" }, + { hw: "gb300", attention: "sage" }, + ]) { + const selection = selectionOf({ nodes: 1, gpus_per_node: 4, placement: "resident", ...extra }); + const resolved = validateResolved(selection, "H3 Grace Blackwell coverage"); + if (resolved?.builder.verification?.request !== "unverified") { + fail(where, `H3 Grace Blackwell request is outside the measured scope: ${JSON.stringify(extra)}`); + } + } for (const hw of ["mi300x", "mi355x"]) { for (const count of [1, 2, 4, 8]) { checkH3(`${hw} 1x${count}`, { hw, nodes: 1, gpus_per_node: count, placement: "resident" }, { tp_size: 1, ulysses_degree: count, ring_degree: 1 }); diff --git a/docs/src/snippets/_deployment.jsx b/docs/src/snippets/_deployment.jsx index 58a95d828576..9ec2a3c61a5a 100644 --- a/docs/src/snippets/_deployment.jsx +++ b/docs/src/snippets/_deployment.jsx @@ -80,6 +80,7 @@ // Legacy "Mean" data is being re-measured to P50; drop once done // multiNodeHints optional — {[hwId]: string[]} prepended as `# ...` lines // dockerImages optional — `docker run` image, keyed by +// `hw|variant|quant` then `variant|quant` then // `hw|quant|strategy` then `hw|quant` then `hw`; // falls back to `lmsysorg/sglang:dev` // dockerHostNetworkWhen optional — `(selection, {flags, env}) => boolean` @@ -108,6 +109,11 @@ export const Deployment = ({ config, benchmarks }) => { // ==== 1. Hardware catalog (shared across cookbooks) ==== // VRAM is per-GPU on-chip memory, not per-module. + const AMD_RDMA_DOCKER_FLAGS = [ + "--device /dev/infiniband", "--cap-add IPC_LOCK", + "--ulimit memlock=-1", "--ulimit stack=67108864", + "--ulimit nofile=1048576:1048576", + ]; const HARDWARE_CATALOG = { blackwell: [ { id: "b300", label: "B300", vram: "288GB" }, @@ -127,16 +133,28 @@ export const Deployment = ({ config, benchmarks }) => { { id: "h20-3e", label: "H20-3e", vram: "141GB" }, { id: "h800", label: "H800", vram: "80GB" }, ], + // ROCm multi-node runs the RDMA NICs straight through: /dev/infiniband + // covers rdma_cm plus the per-NIC uverbsN nodes, IPC_LOCK + an unlimited + // memlock let the transport pin its registered buffers, and the stack / + // nofile raises are for the per-QP file descriptors a full 8-NIC mesh opens. amd: [ - { id: "mi300x", label: "MI300X", vram: "192GB" }, - { id: "mi325x", label: "MI325X", vram: "256GB" }, - { id: "mi350x", label: "MI350X", vram: "288GB" }, - { id: "mi355x", label: "MI355X", vram: "288GB" }, + { id: "mi300x", label: "MI300X", vram: "192GB", + multiNodeDockerFlags: [...AMD_RDMA_DOCKER_FLAGS] }, + { id: "mi325x", label: "MI325X", vram: "256GB", + multiNodeDockerFlags: [...AMD_RDMA_DOCKER_FLAGS] }, + { id: "mi350x", label: "MI350X", vram: "288GB", + multiNodeDockerFlags: [...AMD_RDMA_DOCKER_FLAGS] }, + { id: "mi355x", label: "MI355X", vram: "288GB", + multiNodeDockerFlags: [...AMD_RDMA_DOCKER_FLAGS] }, ], - // Atlas 800I A3 (910C): 1 card = 2 dies, so --tp-size is 2× the card - // count (32 cards -> --tp-size 64). + // Ascend device layout: one /dev/davinciN per core. An A3 Series card is + // the exception — 2 dies per card, so an 8-card node exposes 16 devices + // and --tp-size is twice the card count. A 950PR/DT Series card is a + // single core, so the device count and --tp-size follow the cards. Both + // counts feed the docker `--device` list (`npuDevices`). npu: [ - { id: "a3", label: "Atlas 800I A3", vram: "64GB/die" }, + { id: "a3", label: "A3 Series", vram: "64GB/die", npuDevices: 16 }, + { id: "a5", label: "950PR/DT Series", vram: "128GB", npuDevices: 8 }, ], }; @@ -409,13 +427,33 @@ export const Deployment = ({ config, benchmarks }) => { // grid (not ) — Mintlify wraps
with scroll wrappers. // gridTemplateColumns set inline (depends on measurements.length). + // Horizontal-scroll box for a table wider than the card. The class is + // what opts it out of the site-wide scrollbar suppression at the top of + // custom.css: without it the box scrolls but paints NO scrollbar, so the + // off-card columns look like they do not exist. + // marginTop lives here rather than on benchTable because overflow makes + // this box a BFC — a margin inside it would stop collapsing with the + // preceding sibling's margin-bottom and silently add 4px. + benchTableScroll: { + overflowX: "auto", + // Pin the block axis: a lone overflow-x would compute overflow-y to + // `auto` and clip anything painting outside the box. + overflowY: "hidden", + marginTop: "4px", + }, benchTable: { display: "grid", // columnGap 0 so cells' bottom borders form one continuous line. columnGap: 0, rowGap: "3px", - marginTop: "4px", alignItems: "baseline", + // Without this the grid box stays at the scroll container's width while + // the tracks overflow past it, and the sticky label column — which can + // only stick inside its containing block — slides away after that + // width. Growing the box to the tracks makes the whole scroll range its + // containing block. Inert when the table fits: max-content is then + // narrower than the card and `width: auto` still wins. + minWidth: "max-content", }, benchTableHead: { textAlign: "right", @@ -427,6 +465,16 @@ export const Deployment = ({ config, benchmarks }) => { }, benchTableCornerHead: { paddingBottom: "4px", + // Pinned with the label column below it; background must match + // benchBlock's so scrolled-under values do not show through. + position: "sticky", + left: 0, + background: isDark ? "#111827" : "#fafafa", + zIndex: 1, + // This cell is empty, and the grid aligns items to the baseline, so it + // would otherwise be only as tall as its padding (4px vs the header + // row's 23px) and the scrolled-under header would show above it. + alignSelf: "stretch", }, // Header underline — one div spanning all columns (continuous line). benchTableSeparator: { @@ -439,6 +487,13 @@ export const Deployment = ({ config, benchmarks }) => { textAlign: "left", fontSize: "12px", color: isDark ? "#9ca3af" : "#6b7280", whiteSpace: "nowrap", + // The table scrolls horizontally, so this column has to stay put: + // scrolled to the right end the rows are otherwise four unlabeled + // numbers in four different units. Inert when nothing overflows. + position: "sticky", + left: 0, + background: isDark ? "#111827" : "#fafafa", + zIndex: 1, }, benchTableValue: { textAlign: "right", fontSize: "12px", @@ -780,11 +835,15 @@ export const Deployment = ({ config, benchmarks }) => { let cmd; if (mode === "docker") { - // Image keyed by `hw|quant|strategy` (most specific), then `hw|quant`, - // then `hw`; `:dev` if unmapped. The strategy key covers a tier that - // needs its own build (e.g. a spec-decoding preview image). + // Image keyed by `hw|variant|quant` (most specific), then `variant|quant`, + // then `hw|quant|strategy`, `hw|quant`, `hw`; `:dev` if unmapped. The + // variant keys cover a checkpoint that needs its own build (e.g. a + // new-variant preview image); the strategy key covers a tier that needs + // one (e.g. a spec-decoding preview image). const di = config.dockerImages || {}; - const image = di[`${sel.hw}|${sel.quant}|${sel.strategy}`] + const image = di[`${sel.hw}|${sel.variant}|${sel.quant}`] + || di[`${sel.variant}|${sel.quant}`] + || di[`${sel.hw}|${sel.quant}|${sel.strategy}`] || di[`${sel.hw}|${sel.quant}`] || di[sel.hw] || "lmsysorg/sglang:dev"; const dockerRunCommand = typeof config.dockerRunCommand === "function" ? config.dockerRunCommand(sel) @@ -801,14 +860,31 @@ export const Deployment = ({ config, benchmarks }) => { return (extra && extra.vendor) || "nvidia"; }; // `config.hardware` overrides by id, as in buildHardwareGroups. - const fabricFlagsOf = (hwId) => { + const catalogEntryOf = (hwId) => { const extra = (config.hardware || []).find((h) => h.id === hwId); - if (extra) return extra.multiNodeDockerFlags || []; + if (extra) return extra; for (const list of Object.values(HARDWARE_CATALOG)) { const hit = list.find((h) => h.id === hwId); - if (hit) return hit.multiNodeDockerFlags || []; + if (hit) return hit; + } + return null; + }; + const fabricFlagsOf = (hwId) => + (catalogEntryOf(hwId) || {}).multiNodeDockerFlags || []; + // NPU cards are reached with --device, one per /dev/davinciN core; + // `npuDevices` carries the per-product-line count (16 on an A3 Series + // node, 8 on a 950PR/DT Series node), four devices per line as the host + // docs show. + const davinciLines = (devices) => { + const lines = []; + for (let i = 0; i < devices; i += 4) { + const group = []; + for (let k = i; k < Math.min(i + 4, devices); k++) { + group.push(`--device=/dev/davinci${k}`); + } + lines.push(" " + group.join(" ")); } - return []; + return lines; }; const gpuAccessLines = vendorOf(sel.hw) === "amd" ? [ @@ -820,14 +896,10 @@ export const Deployment = ({ config, benchmarks }) => { ] : vendorOf(sel.hw) === "npu" ? [ - // NPU: --privileged grants the davinci devices (16 dies on an - // 8-card Atlas 800I A3 node); the host CANN driver/firmware/state - // must be mounted in. + // NPU: --privileged grants the davinci devices; the host CANN + // driver/firmware/state must be mounted in. "docker run --privileged --shm-size=16g", - " --device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3", - " --device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7", - " --device=/dev/davinci8 --device=/dev/davinci9 --device=/dev/davinci10 --device=/dev/davinci11", - " --device=/dev/davinci12 --device=/dev/davinci13 --device=/dev/davinci14 --device=/dev/davinci15", + ...davinciLines((catalogEntryOf(sel.hw) || {}).npuDevices || 16), " --device=/dev/davinci_manager", " --device=/dev/hisi_hdc", " -v /usr/local/sbin:/usr/local/sbin", @@ -939,6 +1011,12 @@ export const Deployment = ({ config, benchmarks }) => { // Split workload fields into shared (uniform → context line) vs differing // (→ per-column header). max_concurrency is always per-column. const ALWAYS_PER_COLUMN = new Set(["max_concurrency"]); + // isl and osl print as one `in/out=I/O` token, so they have to be + // classified together. Classified apart — isl varying, osl uniform — the + // shared context line rendered the FIRST measurement's isl as if it held + // for every column: the GLM-5.3-Flash FP8 + TRT-LLM cell says + // "in/out=1000/1000" above a table that is half 8000/1000. + const ATOMIC_WORKLOAD_GROUPS = [["isl", "osl"]]; const partitionWorkload = (measurements) => { const shared = new Set(); const differing = new Set(); @@ -954,6 +1032,10 @@ export const Deployment = ({ config, benchmarks }) => { if (ALWAYS_PER_COLUMN.has(k) || seen.size > 1) differing.add(k); else shared.add(k); } + for (const group of ATOMIC_WORKLOAD_GROUPS) { + if (!group.some((k) => differing.has(k))) continue; + for (const k of group) if (shared.delete(k)) differing.add(k); + } return { shared, differing }; }; @@ -967,34 +1049,48 @@ export const Deployment = ({ config, benchmarks }) => { {sharedText && (
{sharedText}
)} + {/* tabIndex + role: a scroll box with no focusable child is + unreachable without a pointer (WCAG 2.1.1). A lone value column + is `max-content` + `1fr` and so can never exceed the card — no + scrolling, hence no tab stop. */}
1 ? { + tabIndex: 0, + role: "group", + "aria-label": sharedText ? `${title} — ${sharedText}` : title, + } : {})} > - {showColHeaders && ( -
- )} - {showColHeaders && colHeaders.map((h, i) => ( -
{h}
- ))} - {showColHeaders && ( -
- )} - {rows.map((r) => [ -
{r.label}
, - ...r.values.map((v, i) => ( -
- {v !== null ? v : "—"} -
- )), - ])} +
+ {showColHeaders && ( +
+ )} + {showColHeaders && colHeaders.map((h, i) => ( +
{h}
+ ))} + {showColHeaders && ( +
+ )} + {rows.map((r) => [ +
{r.label}
, + ...r.values.map((v, i) => ( +
+ {v !== null ? v : "—"} +
+ )), + ])} +
{legend && (
@@ -1259,6 +1355,7 @@ export const Deployment = ({ config, benchmarks }) => { }; const [sel, setSel] = useState(() => initialSelectionFromCells()); + const [selectionHydrated, setSelectionHydrated] = useState(false); const INTERNAL_HASH_STATE_KEY = "__sglangDeployInternalHash"; const DEPLOYMENT_COMPONENT_ID = "deployment-configurator"; useEffect(() => { @@ -1292,12 +1389,14 @@ export const Deployment = ({ config, benchmarks }) => { if (el) el.scrollIntoView({ behavior: "smooth", block: "start" }); }; hydrate(); + setSelectionHydrated(true); window.addEventListener("hashchange", hydrate); return () => window.removeEventListener("hashchange", hydrate); }, []); // history.replaceState does NOT fire hashchange — dispatch a custom event so // the Playground hears chip-click selection changes. useEffect(() => { + if (!selectionHydrated) return; const target = "#" + new URLSearchParams(sel).toString(); if (window.location.hash !== target) { const historyState = @@ -1311,7 +1410,7 @@ export const Deployment = ({ config, benchmarks }) => { ); } window.dispatchEvent(new CustomEvent("sglang-deploy-sel", { detail: sel })); - }, [sel]); + }, [sel, selectionHydrated]); const [modal, setModal] = useState(null); // 'curl' | 'env' | 'bench' | null useEffect(() => { @@ -1993,7 +2092,7 @@ export const Deployment = ({ config, benchmarks }) => { const options = visibleOptions(dim, sel); const currentOption = selectedOption(dim); return ( -
+
{direct ? dim.title : `${dim.title} options`} diff --git a/docs/src/snippets/_kimi_k3_mamba_ratio_calculator.jsx b/docs/src/snippets/_kimi_k3_mamba_ratio_calculator.jsx index 3a996b2ae496..97a0b2baf938 100644 --- a/docs/src/snippets/_kimi_k3_mamba_ratio_calculator.jsx +++ b/docs/src/snippets/_kimi_k3_mamba_ratio_calculator.jsx @@ -135,10 +135,18 @@ export const KimiK3MambaRatioCalculator = () => { const eff = derive(cfg.flags, cfg.env); const bs = derive(cfg.baseFlags.length ? cfg.baseFlags : cfg.flags, cfg.baseFlags.length ? cfg.baseEnv : cfg.env); - // A --max-mamba-cache-size cell sizes the pool explicitly — no ratio to - // compute or broadcast. - const explicitSizing = (cfg.baseFlags.length ? cfg.baseFlags : cfg.flags) - .some((f) => f.startsWith("--max-mamba-cache-size")); + // Recipes that size the dual pool without the ratio neither render one nor + // broadcast one: + // - a --max-mamba-cache-size cell pins the KDA slot count explicitly; + // - the Ascend NPU recipes size both pools internally and never set + // --mamba-full-memory-ratio (every NPU recipe carries --device npu). + const baseFlagList = cfg.baseFlags.length ? cfg.baseFlags : cfg.flags; + const explicitSizing = baseFlagList.some((f) => f.startsWith("--max-mamba-cache-size")); + const npuRecipe = baseFlagList.some((f) => { + const [head, ...rest] = f.trim().split(/[\s=]+/); + return head === "--device" && rest[0] === "npu"; + }); + const ratioNotApplicable = explicitSizing || npuRecipe; const { ratio, tp, dp, attnTp, dcp, kvDtype, ssmDtype, radixOff, strategy, skipLock, slots, specOn, replaySpec, block, pdRole } = eff; const valid = Number.isFinite(ratio) && ratio > 0 && length > 0 && 96 % attnTp === 0; const baseValid = Number.isFinite(bs.ratio) && bs.ratio > 0 && length > 0; @@ -155,18 +163,22 @@ export const KimiK3MambaRatioCalculator = () => { const cliFlag = valid ? `--mamba-full-memory-ratio ${result}` : ""; // Broadcast both results: the Deploy command takes the base-config value, - // the Playground's composed command takes the effective one. + // the Playground's composed command takes the effective one. On a recipe + // that does not use the ratio, broadcast nulls instead of skipping the + // dispatch — the panels must drop a ratio pinned for a recipe the reader + // has since left rather than keep injecting it. useEffect(() => { - if (explicitSizing) return; window.dispatchEvent( new CustomEvent("sglang-k3-mamba-ratio", { - detail: { - ratio: valid ? result : null, - baseRatio: baseValid ? baseResult : null, - }, + detail: ratioNotApplicable + ? { ratio: null, baseRatio: null } + : { + ratio: valid ? result : null, + baseRatio: baseValid ? baseResult : null, + }, }) ); - }, [result, valid, baseResult, baseValid, explicitSizing]); + }, [result, valid, baseResult, baseValid, ratioNotApplicable]); const copyFlag = () => { if (!cliFlag || typeof navigator === "undefined" || !navigator.clipboard) return; @@ -235,10 +247,12 @@ export const KimiK3MambaRatioCalculator = () => { specLabel, ].filter(Boolean); - if (explicitSizing) { + if (ratioNotApplicable) { return (
- This recipe sizes the KDA state pool explicitly with --max-mamba-cache-size, so the ratio calculator does not apply. + {explicitSizing + ? <>This recipe sizes the KDA state pool explicitly with --max-mamba-cache-size, so the ratio calculator does not apply. + : <>This recipe runs on Ascend NPUs, which size the KDA state pool and the MLA KV pool internally and never set --mamba-full-memory-ratio, so the ratio calculator does not apply.}
); } diff --git a/docs/src/snippets/_playground.jsx b/docs/src/snippets/_playground.jsx index b1b7ae9cc030..d8188aa95e03 100644 --- a/docs/src/snippets/_playground.jsx +++ b/docs/src/snippets/_playground.jsx @@ -225,6 +225,17 @@ export const Playground = ({ config }) => { Array.isArray(vs) && vs.includes(base[k])); }; + // The PD router is shared by every cell, so a platform whose prefill/decode + // pair wants a different routing policy declares `routerOverrides: [{when, + // port?, command}]` instead of rewriting the default out from under the rest. + // First match wins; fields the override omits fall back to the default. + const resolveRouter = (fc, sel) => { + if (!fc || !fc.router) return null; + const hit = (fc.routerOverrides || []).find( + (r) => r && matchConstraint(sel, r.when)); + return hit ? { ...fc.router, ...hit } : fc.router; + }; + // Normalize a chip entry into `{value, label?, hidden, disabled, // disableReason, ...rest}`. `value` resolves to `entry.id` (rich form) or // `entry.value` (wrapper form), or the entry itself for bare values. @@ -379,6 +390,23 @@ export const Playground = ({ config }) => { ANCHOR_NEAR_DPATTN, ANCHOR_NEAR_MOE, }; + // -------- HiCache flag family -------- + // Shared because two axes own it: `hicache` emits it, and `umbp` strips the + // whole family when it takes the tier over (the two are mutually exclusive). + const HICACHE_HEADS = [ + "--enable-hierarchical-cache", "--hicache-ratio", "--hicache-size", + "--hicache-write-policy", "--hicache-mem-layout", "--hicache-io-backend", + "--hicache-storage-backend", "--hicache-storage-prefetch-policy", + "--hicache-storage-backend-extra-config", + ]; + + // `umbp.roleOverrides: [{mode, when, enable, allowTp, env?, flags?, note?}]` — + // first entry whose PD role and `when` match the live selection. `enable` is + // the default when the reader has not toggled the card; `allowTp` lifts + // requiresDpAttention; env / flags join the recipe while it is on. + const umbpRoleOverride = (fc, sel, h) => (fc.roleOverrides || []).find((r) => + r && sel && r.mode === sel.pdMode && (!r.when || h.matchConstraint(sel, r.when))); + // -------- Prefill-CP flag family (shared by the attention axis) -------- // Every flag head that toggles/parameterizes prefill context parallelism: // the canonical pair plus all per-family legacy spellings. @@ -531,16 +559,44 @@ export const Playground = ({ config }) => { ]); } } - if (value.dpAttn !== null && value.dpAttn !== undefined + // `forceOff: [{when, stripEnv, reason}]` on a knob — the selection + // dictates the value, so neither the pick nor the base cell gets a say. + // Used where a PD role runs a different topology than the aggregated + // cell it derives from: the role is TP-only, but the cell is DP. + const dpForced = (knobEntry("dpAttn").forceOff || []).find( + (r) => r && h.matchConstraint(factsNow(), r.when)); + if (dpForced) { + flags = h.stripFlagsByFirstToken(flags, [ + ...h.DP_HEADS, "--enable-dp-attention", + "--enable-dp-attention-local-control-broadcast", + ]); + // The cell's DP-only env would otherwise outlive the flags it tunes. + const stripEnv = dpForced.stripEnv || []; + if (stripEnv.length) { + env = env.filter((e) => !stripEnv.includes(e.split("=")[0])); + } + } else if (value.dpAttn !== null && value.dpAttn !== undefined && !blocked("dpAttn", value.dpAttn)) { // Capture the spelling before stripping — the TP/EP handlers do the // same, and a lookup on the stripped array always hits the fallback. const dpHead = h.flagSpelling(flags, h.DP_HEADS, "--dp-size"); - flags = h.stripFlagsByFirstToken(flags, [...h.DP_HEADS, "--enable-dp-attention"]); + // The local-control-broadcast companion only means anything with DP + // attention on, so it has to go down with it — stripping just + // `--enable-dp-attention` would leave it orphaned in the command. + // apply re-seeds from the base cell, so this restores it when the + // cell had it and the user is only re-sizing DP rather than disabling. + const hadLocalBroadcast = + h.hasFlag(flags, "--enable-dp-attention-local-control-broadcast"); + flags = h.stripFlagsByFirstToken(flags, [ + ...h.DP_HEADS, "--enable-dp-attention", + "--enable-dp-attention-local-control-broadcast", + ]); if (typeof value.dpAttn === "number" && value.dpAttn > 0) { flags = h.insertAfter(flags, h.ANCHOR_NEAR_TP, [ `${dpHead} ${value.dpAttn}`, "--enable-dp-attention", + ...(hadLocalBroadcast + ? ["--enable-dp-attention-local-control-broadcast"] : []), ]); } } @@ -602,14 +658,18 @@ export const Playground = ({ config }) => { {knobs.map((knob) => { const kc = h.evaluateChip(knob, base); if (kc.hidden) return null; + // Mirror apply's forceOff: show the value the command actually + // gets, grayed, instead of a live-looking pick apply discards. + const forced = (knob.forceOff || []).find( + (r) => r && h.matchConstraint(base, r.when)); return ( {knob.label || knob.id.toUpperCase()} - {renderSelect(knobDisplay(knob), entriesFor(knob), + {renderSelect(forced ? false : knobDisplay(knob), entriesFor(knob), (nv) => setKnob(knob.id, nv), base, labelFor(knob), { hideValues: hideNullFor(knob), - disabled: kc.disabled, - disabledReason: kc.disableReason })} + disabled: kc.disabled || !!forced, + disabledReason: forced ? forced.reason : kc.disableReason })} ); })} @@ -719,7 +779,7 @@ export const Playground = ({ config }) => { && (!mmOpt.excludesStrategy || !mmOpt.excludesStrategy.includes(base.strategy)); const backendIsMega = slotDisplay("backend") === "megamoe"; // `ep.showWhen` (function of base) drops the whole EP select on bases - // where EP is not a supported lever (e.g. the single-shape A3 recipe). + // where EP is not a supported lever (e.g. the single-shape A3 Series recipe). const epShown = !!fc.ep && !(typeof fc.ep.showWhen === "function" && !fc.ep.showWhen(base)); return ( @@ -930,11 +990,18 @@ export const Playground = ({ config }) => { pdDisagg: { // The transport default is the config's first entry, so a model whose // recipes standardize on one backend does not silently start on another. - initState: (fc) => ({ - mode: "off", - transferBackend: (fc && (fc.transferBackends || [])[0] || {}).id || "mooncake", - ibDevice: "auto", - }), + // An entry with `defaultWhen` takes over as the default for the selections + // it matches (e.g. MORI and the rdmaN NIC list on ROCm). + initState: (fc, base) => { + const pick = (entries) => (entries || []).find((e) => + e && e.defaultWhen && base && matchConstraint(base, e.defaultWhen)); + const backends = (fc && fc.transferBackends) || []; + return { + mode: "off", + transferBackend: (pick(backends) || backends[0] || {}).id || "mooncake", + ibDevice: (pick(fc && fc.ibDevices) || {}).id || "auto", + }; + }, apply: ({ flags, env, value, sel, fc, h }) => { // The bootstrap port is the base cell's to choose — the router's @@ -985,10 +1052,30 @@ export const Playground = ({ config }) => { // render, so a base flag is never left over from an earlier selection // and must not be stripped when the role is Off. const modeMeta = (fc.modes || []).find((m) => m.id === mode); - if (modeMeta && modeMeta.flags && modeMeta.flags.length) { + // A role may scope its flags/env to a subset of hardware via `when` + // (same shape as a backend's `envWhen`), so a ROCm-only prefill recipe + // doesn't leak onto CUDA. Absent `when`, the role applies everywhere. + // + // `roleOverrides: [{when, mode, flags, env}]` then re-specifies a role + // for a narrower selection — a second operating point (low-latency vs + // high-throughput sizing) is a different flag set for the SAME role, + // so it belongs here rather than as a duplicate entry in the Mode + // select. First match wins and replaces the role's own flags/env. + const roleOverride = (fc.roleOverrides || []).find((r) => + r && r.mode === mode && r.when && h.matchConstraint(sel, r.when)); + const roleSpec = roleOverride || modeMeta; + const modeGate = roleOverride ? null : (modeMeta && modeMeta.when); + const modeOk = !modeGate || Object.keys(modeGate).every( + (k) => (modeGate[k] || []).includes(sel[k])); + if (modeOk && roleSpec && roleSpec.flags && roleSpec.flags.length) { flags = h.stripFlagsByFirstToken( - flags, modeMeta.flags.map((f) => f.split(/[\s=]/)[0])); - adds.push(...modeMeta.flags); + flags, roleSpec.flags.map((f) => f.split(/[\s=]/)[0])); + adds.push(...roleSpec.flags); + } + // `stripFlags` drops base-cell flags the role has no use for (an + // aggregated-serving knob) without re-emitting a value of its own. + if (modeOk && roleSpec && roleSpec.stripFlags && roleSpec.stripFlags.length) { + flags = h.stripFlagsByFirstToken(flags, roleSpec.stripFlags); } // Single-host needs no --dist-init-addr: prefill/decode derive their // ZMQ/dist ports from the role-specific --port (spaced 100 apart, see @@ -1015,8 +1102,8 @@ export const Playground = ({ config }) => { if (ok) env = [...env, ...meta.env.filter((e) => !env.includes(e))]; } // Same for env declared on the selected role. - if (modeMeta && modeMeta.env && modeMeta.env.length) { - env = [...env, ...modeMeta.env.filter((e) => !env.includes(e))]; + if (modeOk && roleSpec && roleSpec.env && roleSpec.env.length) { + env = [...env, ...roleSpec.env.filter((e) => !env.includes(e))]; } } return { flags, env }; @@ -1034,12 +1121,19 @@ export const Playground = ({ config }) => { return null; }, - render: ({ axisId, value, setValue, fc, base, s, renderSelect }) => { + render: ({ axisId, value, setValue, fc, base, s, h, renderSelect }) => { const setSlot = (k, v) => setValue({ ...value, [k]: v }); const showModes = (fc.modes || []).length > 0; const showBackends = (fc.transferBackends || []).length > 0; const showIb = (fc.ibDevices || []).length > 0; if (!showModes && !showBackends && !showIb) return null; + // `notes: [{when, mode, text}]` — a prerequisite the role needs from + // another card, surfaced only while the selection actually trips it. + // `when` matches against constraintBase, so it can key off cross-axis + // facts like dpAttnOn, not just the five cell dimensions. + const note = (fc.notes || []).find( + (n) => n && (!n.mode || [].concat(n.mode).includes(value.mode)) + && h.matchConstraint(base, n.when)); return (
@@ -1066,6 +1160,7 @@ export const Playground = ({ config }) => { )}
+ {note &&
{note.text}
}
); }, @@ -1168,10 +1263,7 @@ export const Playground = ({ config }) => { const backendOptions = fc.backends || []; const ownedHeads = [ - "--enable-hierarchical-cache", "--hicache-ratio", "--hicache-size", - "--hicache-write-policy", "--hicache-mem-layout", "--hicache-io-backend", - "--hicache-storage-backend", "--hicache-storage-prefetch-policy", - "--hicache-storage-backend-extra-config", + ...HICACHE_HEADS, ...((fc.requiredFlags || []).map((f) => f.split(/\s/)[0])), ...backendOptions.flatMap((o) => (o.flags || []).map((f) => f.split(/\s/)[0])), ]; @@ -1281,6 +1373,103 @@ export const Playground = ({ config }) => { }, }, + // ---- Axis: UMBP (unified cache external linker) ------------------------- + // A SIBLING of HiCache, not a tier inside it: the unified radix tree loads + // and offloads straight against an external store with no host cache tier, + // and sglang refuses the pair outright (arg_groups/hicache_hook.py raises on + // --enable-hierarchical-cache or --hicache-storage-backend alongside it). + // So enabling this strips the whole HiCache family instead of layering on + // it, and it is declared after hicache so that strip runs last. + umbp: { + initState: () => ({ enable: null, backend: null }), + + deriveFromBase: (cell, fc, h) => { + const flags = (cell && cell.flags) || []; + return { + enable: h.hasFlag(flags, "--enable-unified-cache-external-linker"), + backend: h.findFlagArg(flags, "--unified-cache-external-linker-backend"), + }; + }, + + apply: ({ flags, env, value, fc, sel, h, derived }) => { + const roleOverride = umbpRoleOverride(fc, sel, h); + const ownedHeads = [ + "--enable-unified-cache-external-linker", + "--unified-cache-external-linker-backend", + ...((fc.requiredFlags || []).map((f) => f.split(/\s/)[0])), + ]; + flags = h.stripFlagsByFirstToken(flags, ownedHeads); + const ownedEnv = [...(fc.requiredEnv || []), ...((roleOverride && roleOverride.env) || [])]; + if (ownedEnv.length) { + env = h.stripEnvByPrefix(env, ownedEnv.map((e) => e.split("=")[0])); + } + const enabled = value.enable !== null + ? value.enable + : (roleOverride ? !!roleOverride.enable : !!(derived && derived.enable)); + if (!enabled) return { flags, env }; + if (fc.onlyHw && sel && !fc.onlyHw.includes(sel.hw)) return { flags, env }; + // The linker keys by DP rank; under pure TP each rank opens its own + // keyspace and the store holds TP copies of the same tokens, so the + // recipe is only meaningful with DP attention on. Read it off the live + // flags rather than the Deploy dims — the attention axis runs first. + // A role override with `allowTp` is a TP-only shape validated as-is. + if (fc.requiresDpAttention && !(roleOverride && roleOverride.allowTp) + && !flags.some((f) => f.split(/[\s=]/)[0] === "--enable-dp-attention")) { + return { flags, env }; + } + flags = h.stripFlagsByFirstToken(flags, HICACHE_HEADS); + const backend = value.backend + || (derived && derived.backend) || fc.defaultBackend || "mori"; + flags = h.insertBeforeTail(flags, [ + "--enable-unified-cache-external-linker", + `--unified-cache-external-linker-backend ${backend}`, + ...(fc.requiredFlags || []), + ...((roleOverride && roleOverride.flags) || []), + ]); + const extraEnv = [...(fc.requiredEnv || []), ...((roleOverride && roleOverride.env) || [])]; + env = [...env, ...extraEnv.filter((e) => !env.includes(e))]; + return { flags, env }; + }, + + render: ({ axisId, value, setValue, fc, base, s, h, renderChip, renderSelect, derived }) => { + if (fc.onlyHw && !fc.onlyHw.includes(base.hw)) return null; + const setSlot = (k, v) => setValue({ ...value, [k]: v }); + const roleOverride = umbpRoleOverride(fc, base, h); + const enabled = value.enable !== null + ? value.enable + : (roleOverride ? !!roleOverride.enable : !!(derived && derived.enable)); + const needsDp = !!fc.requiresDpAttention && !base.dpAttnOn + && !(roleOverride && roleOverride.allowTp); + const backend = value.backend !== null + ? value.backend : ((derived && derived.backend) || fc.defaultBackend || "mori"); + return ( +
+
+ UMBP + + {renderChip("Enable", enabled, true, + () => setSlot("enable", !enabled), + { disabled: needsDp, + disabledReason: needsDp + ? "Needs DP Attention — the linker keyspace is per DP rank, so under pure TP the store holds one copy per TP rank." + : "" })} + + {(fc.backends || []).length > 0 && ( + + Store + {renderSelect(backend, fc.backends, + (v) => setSlot("backend", v), base)} + + )} +
+ {enabled && !needsDp && roleOverride && roleOverride.note && ( +
{roleOverride.note}
+ )} +
+ ); + }, + }, + // ---- Axis: Flag Selects (generic, config-declared) ---------------------- // A LIST of single-selects, each declared entirely in config: // { id, title, stripPrefixes: [...], stripEnv?: [...], @@ -1463,6 +1652,15 @@ export const Playground = ({ config }) => { let flags = [...baseFlags]; let env = [...(baseEnv || [])]; let pdMode = null; + // The selected PD role, resolved up front rather than from the loop's + // getRenderHints: axes that run BEFORE pdDisagg still need it, since a role + // can constrain what another axis is allowed to emit (a TP-only role has to + // force DP-Attention off in the attention axis, which composes earlier). + const pdFc = pgFeatures.pdDisagg; + const pdDelta = allDeltas.pdDisagg; + const pdRoleSel = (pdFc && (pdFc.modes || []).length && pdDelta) + ? pdDelta.mode + : ((sel && sel.pdMode) || "off"); for (const [axisId, handler] of Object.entries(AXIS_HANDLERS)) { const fc = pgFeatures[axisId]; if (!fc) continue; @@ -1471,7 +1669,7 @@ export const Playground = ({ config }) => { const derived = derivedMap ? derivedMap[axisId] : null; const specAlgorithm = (findFlagArg( flags, "--speculative-algorithm") || "").toUpperCase() || null; - const liveSel = { ...sel, specAlgorithm }; + const liveSel = { ...sel, specAlgorithm, pdMode: pdRoleSel }; const out = handler.apply({ flags, env, value, fc, sel: liveSel, h: helpers, derived }); flags = out.flags; env = out.env; @@ -1520,13 +1718,17 @@ export const Playground = ({ config }) => { } let cmd; if (mode === "docker") { - // Image keyed by `hw|quant|strategy` (most specific), then `hw|quant`, then - // `hw`; `:dev` if unmapped (matches _deployment.jsx). The strategy key covers - // a tier that needs its own build (e.g. a spec-decoding preview image), so the - // playground base must resolve it too or it hands back an image that cannot - // run the command. + // Image keyed by `hw|variant|quant` (most specific), then `variant|quant`, + // then `hw|quant|strategy`, `hw|quant`, `hw`; `:dev` if unmapped (matches + // _deployment.jsx). The variant keys cover a checkpoint that needs its own + // build (e.g. a new-variant preview image) and the strategy key a tier that + // needs one (e.g. a spec-decoding preview image), so the playground base + // must resolve them too or it hands back an image that cannot run the + // command. const di = config.dockerImages || {}; - const image = di[`${sel.hw}|${sel.quant}|${sel.strategy}`] + const image = di[`${sel.hw}|${sel.variant}|${sel.quant}`] + || di[`${sel.variant}|${sel.quant}`] + || di[`${sel.hw}|${sel.quant}|${sel.strategy}`] || di[`${sel.hw}|${sel.quant}`] || di[sel.hw] || "lmsysorg/sglang:dev"; const dockerRunCommand = typeof config.dockerRunCommand === "function" ? config.dockerRunCommand(sel) @@ -1537,18 +1739,74 @@ export const Playground = ({ config }) => { && config.dockerHostNetworkWhen(sel, { flags: f, env: cellEnv })); // Mirrors `multiNodeDockerFlags` on the _deployment.jsx HARDWARE_CATALOG // (Mintlify strips module state, so the engines cannot share it). + const AMD_RDMA_DOCKER_FLAGS = [ + "--device /dev/infiniband", "--cap-add IPC_LOCK", + "--ulimit memlock=-1", "--ulimit stack=67108864", + "--ulimit nofile=1048576:1048576", + ]; const HW_MULTINODE_DOCKER_FLAGS = { "dgx-spark": [ "--ulimit memlock=-1:-1", "--cap-add IPC_LOCK", "--device /dev/infiniband", ], + mi300x: AMD_RDMA_DOCKER_FLAGS, + mi325x: AMD_RDMA_DOCKER_FLAGS, + mi350x: AMD_RDMA_DOCKER_FLAGS, + mi355x: AMD_RDMA_DOCKER_FLAGS, }; const fabricFlags = HW_MULTINODE_DOCKER_FLAGS[sel.hw] || []; + // Mirrors the vendor branches in _deployment.jsx: ROCm reaches its GPUs + // through /dev/kfd + /dev/dri and the video group (not --gpus all), and + // Ascend NPUs are reached with --device, one per /dev/davinciN core (16 + // on an A3 Series node, 8 on a 950PR/DT Series node — the catalog's + // `npuDevices`). + const isAmdHw = /^mi\d/.test(sel.hw || ""); + const HW_NPU_DEVICES = { a3: 16, a5: 8 }; + const npuDevices = HW_NPU_DEVICES[sel.hw]; + const davinciLines = (devices) => { + const lines = []; + for (let i = 0; i < devices; i += 4) { + const group = []; + for (let k = i; k < Math.min(i + 4, devices); k++) { + group.push(`--device=/dev/davinci${k}`); + } + lines.push(" " + group.join(" ")); + } + return lines; + }; const dockerLines = [ - "docker run --gpus all", - " --shm-size 32g", + ...(isAmdHw + ? [ + "docker run", + " --device=/dev/kfd --device=/dev/dri", + " --group-add video", + " --cap-add=SYS_PTRACE --security-opt seccomp=unconfined", + " --shm-size 32g", + ] + : npuDevices + ? [ + // NPU: --privileged grants the davinci devices; the host CANN + // driver/firmware/state must be mounted in. + "docker run --privileged --shm-size=16g", + ...davinciLines(npuDevices), + " --device=/dev/davinci_manager", + " --device=/dev/hisi_hdc", + " -v /usr/local/sbin:/usr/local/sbin", + " -v /usr/local/Ascend/driver:/usr/local/Ascend/driver", + " -v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware", + " -v /etc/ascend_install.info:/etc/ascend_install.info", + " -v /var/queue_schedule:/var/queue_schedule", + " -v ~/.cache/:/root/.cache/", + ] + : [ + "docker run --gpus all", + " --shm-size 32g", + ]), hostNetwork ? " --network host" : ` -p ${servePort}:${servePort}`, - ...(multinode ? fabricFlags.map((x) => " " + x) : []), - " -v ~/.cache/huggingface:/root/.cache/huggingface", + // A PD pair is cross-host even when each role is a single-node cell, so + // the RDMA fabric flags are needed for `pdMode` too, not just multinode. + ...((multinode || pdMode) ? fabricFlags.map((x) => " " + x) : []), + // The NPU device block already mounts ~/.cache/. + ...(npuDevices ? [] : [" -v ~/.cache/huggingface:/root/.cache/huggingface"]), ...(config.dockerMounts || []).map((mount) => ` -v ${mount}`), ` --env "HF_TOKEN={{HF_TOKEN}}"`, ...cellEnv.map((e) => ` --env ${e}`), @@ -1578,9 +1836,8 @@ export const Playground = ({ config }) => { } if (pdMode === "prefill" || pdMode === "decode") { const sibling = pdMode === "prefill" ? "decode" : "prefill"; - const routerCfg = config.playgroundFeatures - && config.playgroundFeatures.pdDisagg - && config.playgroundFeatures.pdDisagg.router; + const routerCfg = resolveRouter(config.playgroundFeatures + && config.playgroundFeatures.pdDisagg, sel); const routerPort = (routerCfg && routerCfg.port) || 8000; const routerLine = routerCfg ? `# then front BOTH with the Router shown below.\n` @@ -2261,9 +2518,18 @@ export const Playground = ({ config }) => { ? matchedCell : null; const pgSpecAlgoFlag = pgFlagsLatest.find( (f) => f.split(/[\s=]/)[0] === "--speculative-algorithm"); + // Off PD, the hint is a "you forgot to set it" warning and disappears once + // the ceiling is explicit. A PD role always ships one, so that form would + // never fire there — but the role is exactly where the sizing rule bites, + // since the ceiling has to be set on both roles and the decode graphs sized + // to what it leaves per rank. Under a role the hint switches to that form and + // shows whether or not the flag is present. + const pgPdRole = pdMode === "prefill" || pdMode === "decode"; const pgSpecHint = !!pgSpecAlgoFlag && - !pgFlagsLatest.some((f) => f.split(/[\s=]/)[0] === "--max-running-requests"); + (pgPdRole + || !pgFlagsLatest.some( + (f) => f.split(/[\s=]/)[0] === "--max-running-requests")); const specAlgoLabels = { EAGLE: "MTP", EAGLE3: "MTP", FROZEN_KV_MTP: "MTP", DSPARK: "DSpark", DFLASH: "DFlash", NGRAM: "N-gram", @@ -2301,9 +2567,8 @@ export const Playground = ({ config }) => { // PD-Disagg router, if configured. When a PD role is active, cURL retargets // to the router port and a companion router block renders below the command. const pdRouter = (pdMode !== "off" - && config.playgroundFeatures - && config.playgroundFeatures.pdDisagg - && config.playgroundFeatures.pdDisagg.router) || null; + && resolveRouter(config.playgroundFeatures + && config.playgroundFeatures.pdDisagg, base)) || null; const curlEnv = (pdRouter && pdRouter.port != null) ? { ...env, CURL_PORT: String(pdRouter.port) } : env; @@ -2548,7 +2813,11 @@ export const Playground = ({ config }) => { {pgSpecHint && (
- ⚠️ Speculative decoding ({pgSpecAlgoName}) is on — SGLang resets --max-running-requests to 48 when it isn't set. Add --max-running-requests <N> sized for your target concurrency. + {pgPdRole ? ( + <>⚠️ Speculative decoding ({pgSpecAlgoName}) is on — for a target concurrency of N, set --max-running-requests <N*2> on both the prefill and decode roles. That ceiling is server-wide and floor-divided by attn_dp_size, so size the decode graphs to the per-rank batch it leaves: --cuda-graph-bs-decode up to N*2 / dp_size. + ) : ( + <>⚠️ Speculative decoding ({pgSpecAlgoName}) is on — SGLang resets --max-running-requests to 48 when it isn't set. Add --max-running-requests <N> sized for your target concurrency. + )}
)} {pgCpDpHint && ( diff --git a/docs/src/snippets/_qwen38_mamba_ratio_calculator.jsx b/docs/src/snippets/_qwen38_mamba_ratio_calculator.jsx index ba8dfb546485..6f87e956c71e 100644 --- a/docs/src/snippets/_qwen38_mamba_ratio_calculator.jsx +++ b/docs/src/snippets/_qwen38_mamba_ratio_calculator.jsx @@ -89,17 +89,19 @@ export const Qwen38MambaRatioCalculator = () => { // reported as out of range rather than silently mis-computed. const tp = Number(flagArg("--tp")) || Number(flagArg("--tp-size")) || 1; - // The NVFP4 checkpoint declares kv_cache_quant_algo: FP8, so the default - // --kv-cache-dtype auto lands on fp8_e4m3 there with no flag present; the - // BF16 / FP8 checkpoints keep a bf16 KV pool under the same default. An - // explicit flag always wins over the checkpoint's declaration. + // The two RadixArk NVFP4 exports declare kv_cache_quant_algo: FP8, so the + // default --kv-cache-dtype auto lands on fp8_e4m3 there with no flag + // present. The BF16 / FP8 checkpoints keep a bf16 KV pool under the same + // default, and so does NVIDIA's NVFP4 export — it is the same W4A4 body, + // but it ships no kv_cache_scheme, so this cannot key off the `nvfp4` + // prefix. An explicit flag always wins over the checkpoint's declaration. const kvFlag = flagArg("--kv-cache-dtype"); const kvDtype = kvFlag === "fp8_e4m3" ? "fp8_e4m3" : kvFlag === "bfloat16" || kvFlag === "bf16" ? "bfloat16" - : String(quant).startsWith("nvfp4") + : quant === "nvfp4-bf16-head" || quant === "nvfp4-fp4-head" ? "fp8_e4m3" : "bfloat16"; diff --git a/docs/src/snippets/autoregressive/deepseek-ocr-v2-deployment.jsx b/docs/src/snippets/autoregressive/deepseek-ocr-v2-deployment.jsx index a8bfbec9f37e..2e7cb06701d3 100644 --- a/docs/src/snippets/autoregressive/deepseek-ocr-v2-deployment.jsx +++ b/docs/src/snippets/autoregressive/deepseek-ocr-v2-deployment.jsx @@ -10,6 +10,7 @@ export const DeepSeekOCR2Deployment = () => { { id: 'mi325x', label: 'MI325X', default: false }, { id: 'mi355x', label: 'MI355X', default: false }, { id: 'xeon', label: 'XEON', default: false }, + { id: 'arc_b', label: 'BMG', default: false }, ] }, quantization: { @@ -25,8 +26,8 @@ export const DeepSeekOCR2Deployment = () => { type: 'checkbox', items: [ { id: 'tp', label: 'TP', subtitle: 'Tensor Parallel', default: true, required: true }, - { id: 'dp', label: 'DP', subtitle: 'Data Parallel', default: false, disabledWhen: (v) => v.hardware === 'xeon', disabledReason: 'Intel Xeon CPUs only support Tensor Parallel (TP)' }, - { id: 'ep', label: 'EP', subtitle: 'Expert Parallel', default: false, disabledWhen: (v) => v.hardware === 'xeon', disabledReason: 'Intel Xeon CPUs only support Tensor Parallel (TP)' } + { id: 'dp', label: 'DP', subtitle: 'Data Parallel', default: false, disabledWhen: (v) => v.hardware === 'xeon' || v.hardware === 'arc_b', disabledReason: 'Only Tensor Parallel (TP) is supported on this hardware' }, + { id: 'ep', label: 'EP', subtitle: 'Expert Parallel', default: false, disabledWhen: (v) => v.hardware === 'xeon' || v.hardware === 'arc_b', disabledReason: 'Only Tensor Parallel (TP) is supported on this hardware' } ] }, }; @@ -42,6 +43,8 @@ export const DeepSeekOCR2Deployment = () => { cmd += ` --model-path ${modelPath}`; if (hardware === 'xeon') { cmd += ` \\\n --device cpu \\\n --disable-overlap-schedule \\\n --trust-remote-code`; + } else if (hardware === 'arc_b') { + cmd += ` \\\n --device xpu`; } cmd += ` \\\n --enable-multimodal`; @@ -272,9 +275,8 @@ export const DeepSeekOCR2Deployment = () => { ) : option.type === 'checkbox' ? ( (option.items || []).map((item) => { const isChecked = (values[option.name] || []).includes(item.id); - const isDisabled = - item.required || - (typeof item.disabledWhen === 'function' && item.disabledWhen(values)); + const dynDisabled = typeof item.disabledWhen === 'function' && item.disabledWhen(values); + const isDisabled = item.required || dynDisabled; return (