diff --git a/docs/cookbook/autoregressive/Qwen/Qwen3-VL.mdx b/docs/cookbook/autoregressive/Qwen/Qwen3-VL.mdx index 67cd6b11f5bf..25fc03293b91 100644 --- a/docs/cookbook/autoregressive/Qwen/Qwen3-VL.mdx +++ b/docs/cookbook/autoregressive/Qwen/Qwen3-VL.mdx @@ -80,6 +80,161 @@ python -m sglang.launch_server \ * **Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings. +#### B300 BF16 mixed chunking for the 4B model + +For text-only traffic on one NVIDIA B300 (TP1), you can enable mixed chunking +with TRTLLM attention to tune time to first token (TTFT). The benefit depends on +concurrency and the runtime configuration; benchmark both settings for your +traffic. Model weights, KV cache, and attention queries all remain BF16. The recipe uses page size 32, +16384-token prefill chunks, and the default NHD KV-cache layout. + +Use a source build containing the BF16 mixed-batch attention and fused QK +normalization/mRoPE changes in +[PR #42913](https://github.com/sgl-project/sglang/pull/42913). +These measurements used those changes with Torch `2.14.1+cu130`, FlashInfer +`0.7.0.post1`, and Transformers `5.17.0` on Linux with CUDA 13.0; they are not +measurements of a published release. The pinned source and reproduction scripts +are in the [experiment record](https://github.com/zcnrex/sglang/tree/f907431182d8097b1317d3029f006fb55b8e23df/benchmark/qwen3_vl). + + + +```bash Command +CUDA_VISIBLE_DEVICES=0 sglang serve \ + --model-path Qwen/Qwen3-VL-4B-Instruct \ + --dtype bfloat16 \ + --kv-cache-dtype auto \ + --attention-backend trtllm_mha \ + --page-size 32 \ + --chunked-prefill-size 16384 \ + --enable-mixed-chunk \ + --cuda-graph-backend-prefill disabled \ + --enforce-disable-flashinfer-allreduce-fusion \ + --host 127.0.0.1 \ + --port 30000 +``` + +With `--dtype bfloat16`, `--kv-cache-dtype auto` also selects BF16. To reproduce +the comparison with mixed chunking disabled, remove only `--enable-mixed-chunk`. + + + + + +The workload used text-only random prompts with a nominal length of 8192 input +tokens before chat templating and retokenization, plus 1024 output tokens. Actual +model input lengths vary. Concurrency points 1/4/8/16/32/64/128 used +30/40/80/80/160/320/640 measured requests, respectively. Each point used +`max(64, concurrency)` warmup requests followed by a cache flush. Both sweeps +used the same source build and configuration, changing only mixed chunking. +Each completed all 1350 measured requests and the expected token counts. + +Throughput below counts **output tokens only** on one GPU. TTFT and time per +output token (TPOT) are medians; “off” and “on” refer to mixed chunking. + + + + + + + + + + + + +
ConcurrencyOutput tokens/s (off)Output tokens/s (on)TTFT ms (off)TTFT ms (on)TPOT ms (off)TPOT ms (on)
1383.49383.8175.0875.502.542.53
41229.771234.49187.29158.943.063.08
81886.311888.61384.52262.653.873.96
162547.932558.44566.15383.805.725.87
323118.623136.651080.92522.489.199.73
643488.073518.272158.99543.4716.2417.60
1283763.543801.874120.29577.6229.9732.95
+ +At concurrency 128, enabling mixed chunking reduced median TTFT from 4120 ms +to 578 ms (86.0%) while output throughput changed from 3764 to 3802 tokens/s +(+1.0%). Median TPOT increased from 29.97 to 32.95 ms (+10.0%). Use this recipe +when faster first-token delivery matters, and evaluate the inter-token latency +tradeoff for your traffic. These measurements describe the pinned build above; +later kernel, cache-layout, or graph settings can change the tradeoff. Compare +mean and tail latency as well as medians before choosing a setting. + +Text GSM8K scored 1216/1314 (92.54%) for the combined BF16 configuration and +1215/1314 (92.47%) for the BF16 reference. The evaluation used the first five +dataset rows as few-shot examples, then all 1314 remaining rows, temperature 0, +a 2048-token output limit, and concurrency 32. This single-run check does not +establish equivalent accuracy on other workloads. A one-image smoke test produced the same 32 generated tokens as the reference; +broad image and video accuracy were not evaluated. The experiment record includes the raw results, pinned +benchmark client, source hashes, and evaluation protocol. + + + + + +This separate text-only snapshot uses source commit +[`c76cfaba1a`](https://github.com/zcnrex/sglang/commit/c76cfaba1a1154a4a24b8694184cb8d3318fc358) +from PR #42913, with HND KV-cache layout and mixed chunking enabled. It is not +an off/on comparison and does not measure the causal benefit of mixed chunking +or HND. The historical NHD comparison above retains its original configuration. +Use a source build at this commit with Torch `2.14.1+cu130`, FlashInfer +`0.7.0.post1`, and Transformers `5.17.0`. The model revision is +`ebb281ec70b05090aa6165b016eac8ec08e71b17`. + +```bash Command +export MODEL_PATH="$(hf download Qwen/Qwen3-VL-4B-Instruct --revision ebb281ec70b05090aa6165b016eac8ec08e71b17)" +CUDA_VISIBLE_DEVICES=0 SGLANG_USE_HND_KVCACHE=1 sglang serve \ + --model-path "$MODEL_PATH" \ + --dtype bfloat16 --kv-cache-dtype bfloat16 --bf16-gemm-backend cutedsl \ + --attention-backend trtllm_mha --page-size 32 --enable-mixed-chunk \ + --chunked-prefill-size 16384 --cuda-graph-backend-prefill disabled \ + --cuda-graph-max-bs-decode 128 \ + --cuda-graph-bs-decode 1 2 4 8 12 16 24 32 40 48 56 64 72 80 88 96 104 112 120 128 \ + --max-total-tokens 1600000 --enforce-disable-flashinfer-allreduce-fusion \ + --random-seed 0 --host 127.0.0.1 --port 30000 +``` + +All points use TP1, BF16 weights, attention queries and KV cache, the same +20 decode graph buckets, and a server-verified capacity of 1,600,000 KV tokens. +The workload uses nominal 8192-token random inputs and 1024-token outputs, +with random range ratio 1 and client seed 42. Concurrency 1/4/8/16/32/64/128 +uses 30/40/80/80/160/320/640 measured requests and `max(64, concurrency)` warmup +requests, followed by a cache flush. Every run completed the exact requested +request, input-token and output-token counts. Commands, the pinned benchmark +client and raw measurements are retained in the +[snapshot evidence](https://github.com/zcnrex/sglang/tree/a2f3a4e381d359e3fb4f1e9984ceb9f94b6a3f2e/benchmark/qwen3_vl/results/final_qkv_full_sweep). + +Throughput is **output tokens/s on one GPU**, not combined input/output +throughput. TTFT and TPOT below are **medians in milliseconds**. + + + + + + + + + + + + + +
ConcurrencyOutput tokens/sMedian TTFT msMedian TPOT ms
1411.6385.582.35
41276.59254.642.89
81907.64380.053.83
162582.38635.045.59
323139.82808.909.36
643540.85687.2817.31
1283832.18599.6432.68
128, second GPU3845.56591.8032.53
+ +At concurrency 128, mean TTFT was 1475.70/1466.74 ms, p95 was +6803.68/6799.04 ms, and p99 was 8642.01/8627.08 ms for the two GPUs, +respectively. The median alone understates first-token latency for some requests. +The eight servers ran concurrently on separate B300 GPUs on one host. Normal +public startup autotuning used fresh per-worker caches; the two C128 runs +selected different M128 down-projection tactics and differed in throughput by +0.349%. GPU clocks and overlapping host work also varied. These are individual +configuration measurements, not an isolated same-device comparison or a +repeatability bound. + +Separate normal-startup GSM8K validation of the batch-one/two QKV extension +scored 1217/1314 in both arms at concurrency 1, and 1213/1314 with the extension +versus 1216/1314 without it at concurrency 2. The latter used matching startup +tactics; its three-question deficit remains an empirical limitation. A fixed-batch +diagnostic on four disagreement prompts then matched full logits through 512 +decode steps, but does not explain the full-serving differences or establish +broad accuracy equivalence. One-fixture image checks also passed. Detailed +protocols, per-question results and limitations are in the +[validation record](https://github.com/zcnrex/sglang/tree/a2f3a4e381d359e3fb4f1e9984ceb9f94b6a3f2e/benchmark/qwen3_vl). + + + ## 4. Model Invocation ### 4.1 Basic Usage