[Cookbook] Document BF16 mixed-chunk latency tuning for Qwen3-VL on B300

docs/cookbook/autoregressive/Qwen/Qwen3-VL.mdx

@@ -80,6 +80,161 @@ python -m sglang.launch_server \
 
 * **Xeon CPU service configuration:** Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
 
+#### B300 BF16 mixed chunking for the 4B model
+
+For text-only traffic on one NVIDIA B300 (TP1), you can enable mixed chunking
+with TRTLLM attention to tune time to first token (TTFT). The benefit depends on
+concurrency and the runtime configuration; benchmark both settings for your
+traffic. Model weights, KV cache, and attention queries all remain BF16. The recipe uses page size 32,
+16384-token prefill chunks, and the default NHD KV-cache layout.
+
+Use a source build containing the BF16 mixed-batch attention and fused QK
+normalization/mRoPE changes in
+[PR #42913](https://github.com/sgl-project/sglang/pull/42913).
+These measurements used those changes with Torch `2.14.1+cu130`, FlashInfer
+`0.7.0.post1`, and Transformers `5.17.0` on Linux with CUDA 13.0; they are not
+measurements of a published release. The pinned source and reproduction scripts
+are in the [experiment record](https://github.com/zcnrex/sglang/tree/f907431182d8097b1317d3029f006fb55b8e23df/benchmark/qwen3_vl).
+
+<Accordion title="Launch the B300 BF16 mixed-chunk configuration">
+
+```bash Command
+CUDA_VISIBLE_DEVICES=0 sglang serve \
+  --model-path Qwen/Qwen3-VL-4B-Instruct \
+  --dtype bfloat16 \
+  --kv-cache-dtype auto \
+  --attention-backend trtllm_mha \
+  --page-size 32 \
+  --chunked-prefill-size 16384 \
+  --enable-mixed-chunk \
+  --cuda-graph-backend-prefill disabled \
+  --enforce-disable-flashinfer-allreduce-fusion \
+  --host 127.0.0.1 \
+  --port 30000
+```
+
+With `--dtype bfloat16`, `--kv-cache-dtype auto` also selects BF16. To reproduce
+the comparison with mixed chunking disabled, remove only `--enable-mixed-chunk`.
+
+</Accordion>
+
+<Accordion title="Measured throughput, latency, and accuracy">
+
+The workload used text-only random prompts with a nominal length of 8192 input
+tokens before chat templating and retokenization, plus 1024 output tokens. Actual
+model input lengths vary. Concurrency points 1/4/8/16/32/64/128 used
+30/40/80/80/160/320/640 measured requests, respectively. Each point used
+`max(64, concurrency)` warmup requests followed by a cache flush. Both sweeps
+used the same source build and configuration, changing only mixed chunking.
+Each completed all 1350 measured requests and the expected token counts.
+
+Throughput below counts **output tokens only** on one GPU. TTFT and time per
+output token (TPOT) are medians; “off” and “on” refer to mixed chunking.
+
+<table>
+  <thead><tr><th>Concurrency</th><th>Output tokens/s (off)</th><th>Output tokens/s (on)</th><th>TTFT ms (off)</th><th>TTFT ms (on)</th><th>TPOT ms (off)</th><th>TPOT ms (on)</th></tr></thead>
+  <tbody>
+    <tr><td>1</td><td>383.49</td><td>383.81</td><td>75.08</td><td>75.50</td><td>2.54</td><td>2.53</td></tr>
+    <tr><td>4</td><td>1229.77</td><td>1234.49</td><td>187.29</td><td>158.94</td><td>3.06</td><td>3.08</td></tr>
+    <tr><td>8</td><td>1886.31</td><td>1888.61</td><td>384.52</td><td>262.65</td><td>3.87</td><td>3.96</td></tr>
+    <tr><td>16</td><td>2547.93</td><td>2558.44</td><td>566.15</td><td>383.80</td><td>5.72</td><td>5.87</td></tr>
+    <tr><td>32</td><td>3118.62</td><td>3136.65</td><td>1080.92</td><td>522.48</td><td>9.19</td><td>9.73</td></tr>
+    <tr><td>64</td><td>3488.07</td><td>3518.27</td><td>2158.99</td><td>543.47</td><td>16.24</td><td>17.60</td></tr>
+    <tr><td>128</td><td>3763.54</td><td>3801.87</td><td>4120.29</td><td>577.62</td><td>29.97</td><td>32.95</td></tr>
+  </tbody>
+</table>
+
+At concurrency 128, enabling mixed chunking reduced median TTFT from 4120 ms
+to 578 ms (86.0%) while output throughput changed from 3764 to 3802 tokens/s
+(+1.0%). Median TPOT increased from 29.97 to 32.95 ms (+10.0%). Use this recipe
+when faster first-token delivery matters, and evaluate the inter-token latency
+tradeoff for your traffic. These measurements describe the pinned build above;
+later kernel, cache-layout, or graph settings can change the tradeoff. Compare
+mean and tail latency as well as medians before choosing a setting.
+
+Text GSM8K scored 1216/1314 (92.54%) for the combined BF16 configuration and
+1215/1314 (92.47%) for the BF16 reference. The evaluation used the first five
+dataset rows as few-shot examples, then all 1314 remaining rows, temperature 0,
+a 2048-token output limit, and concurrency 32. This single-run check does not
+establish equivalent accuracy on other workloads. A one-image smoke test produced the same 32 generated tokens as the reference;
+broad image and video accuracy were not evaluated. The experiment record includes the raw results, pinned
+benchmark client, source hashes, and evaluation protocol.
+
+</Accordion>
+
+<Accordion title="Pinned B300 BF16 HND configuration and throughput snapshot">
+
+This separate text-only snapshot uses source commit
+[`c76cfaba1a`](https://github.com/zcnrex/sglang/commit/c76cfaba1a1154a4a24b8694184cb8d3318fc358)
+from PR #42913, with HND KV-cache layout and mixed chunking enabled. It is not
+an off/on comparison and does not measure the causal benefit of mixed chunking
+or HND. The historical NHD comparison above retains its original configuration.
+Use a source build at this commit with Torch `2.14.1+cu130`, FlashInfer
+`0.7.0.post1`, and Transformers `5.17.0`. The model revision is
+`ebb281ec70b05090aa6165b016eac8ec08e71b17`.
+
+```bash Command
+export MODEL_PATH="$(hf download Qwen/Qwen3-VL-4B-Instruct --revision ebb281ec70b05090aa6165b016eac8ec08e71b17)"
+CUDA_VISIBLE_DEVICES=0 SGLANG_USE_HND_KVCACHE=1 sglang serve \
+  --model-path "$MODEL_PATH" \
+  --dtype bfloat16 --kv-cache-dtype bfloat16 --bf16-gemm-backend cutedsl \
+  --attention-backend trtllm_mha --page-size 32 --enable-mixed-chunk \
+  --chunked-prefill-size 16384 --cuda-graph-backend-prefill disabled \
+  --cuda-graph-max-bs-decode 128 \
+  --cuda-graph-bs-decode 1 2 4 8 12 16 24 32 40 48 56 64 72 80 88 96 104 112 120 128 \
+  --max-total-tokens 1600000 --enforce-disable-flashinfer-allreduce-fusion \
+  --random-seed 0 --host 127.0.0.1 --port 30000
+```
+
+All points use TP1, BF16 weights, attention queries and KV cache, the same
+20 decode graph buckets, and a server-verified capacity of 1,600,000 KV tokens.
+The workload uses nominal 8192-token random inputs and 1024-token outputs,
+with random range ratio 1 and client seed 42. Concurrency 1/4/8/16/32/64/128
+uses 30/40/80/80/160/320/640 measured requests and `max(64, concurrency)` warmup
+requests, followed by a cache flush. Every run completed the exact requested
+request, input-token and output-token counts. Commands, the pinned benchmark
+client and raw measurements are retained in the
+[snapshot evidence](https://github.com/zcnrex/sglang/tree/a2f3a4e381d359e3fb4f1e9984ceb9f94b6a3f2e/benchmark/qwen3_vl/results/final_qkv_full_sweep).
+
+Throughput is **output tokens/s on one GPU**, not combined input/output
+throughput. TTFT and TPOT below are **medians in milliseconds**.
+
+<table>
+  <thead><tr><th>Concurrency</th><th>Output tokens/s</th><th>Median TTFT ms</th><th>Median TPOT ms</th></tr></thead>
+  <tbody>
+    <tr><td>1</td><td>411.63</td><td>85.58</td><td>2.35</td></tr>
+    <tr><td>4</td><td>1276.59</td><td>254.64</td><td>2.89</td></tr>
+    <tr><td>8</td><td>1907.64</td><td>380.05</td><td>3.83</td></tr>
+    <tr><td>16</td><td>2582.38</td><td>635.04</td><td>5.59</td></tr>
+    <tr><td>32</td><td>3139.82</td><td>808.90</td><td>9.36</td></tr>
+    <tr><td>64</td><td>3540.85</td><td>687.28</td><td>17.31</td></tr>
+    <tr><td>128</td><td>3832.18</td><td>599.64</td><td>32.68</td></tr>
+    <tr><td>128, second GPU</td><td>3845.56</td><td>591.80</td><td>32.53</td></tr>
+  </tbody>
+</table>
+
+At concurrency 128, mean TTFT was 1475.70/1466.74 ms, p95 was
+6803.68/6799.04 ms, and p99 was 8642.01/8627.08 ms for the two GPUs,
+respectively. The median alone understates first-token latency for some requests.
+The eight servers ran concurrently on separate B300 GPUs on one host. Normal
+public startup autotuning used fresh per-worker caches; the two C128 runs
+selected different M128 down-projection tactics and differed in throughput by
+0.349%. GPU clocks and overlapping host work also varied. These are individual
+configuration measurements, not an isolated same-device comparison or a
+repeatability bound.
+
+Separate normal-startup GSM8K validation of the batch-one/two QKV extension
+scored 1217/1314 in both arms at concurrency 1, and 1213/1314 with the extension
+versus 1216/1314 without it at concurrency 2. The latter used matching startup
+tactics; its three-question deficit remains an empirical limitation. A fixed-batch
+diagnostic on four disagreement prompts then matched full logits through 512
+decode steps, but does not explain the full-serving differences or establish
+broad accuracy equivalence. One-fixture image checks also passed. Detailed
+protocols, per-question results and limitations are in the
+[validation record](https://github.com/zcnrex/sglang/tree/a2f3a4e381d359e3fb4f1e9984ceb9f94b6a3f2e/benchmark/qwen3_vl).
+
+</Accordion>
+
 ## 4. Model Invocation
 
 ### 4.1 Basic Usage