[HiCache][perf] Queue and batch write-through backups off the request-finish path

[alphabetc1]

Motivation

With --enable-hierarchical-cache (write_through) the unified radix cache backs up every new tree node to host. After #39050 the D2H submits are merged per step, but the per-node bookkeeping that precedes a submit still runs inside cache_finished_req, on the scheduler thread, at the moment the node is inserted: build_backup_spec (SWA/Mamba transfer lists, torch.cat/clone), host pool alloc for KV + side pools (with SWA host eviction when that pool is full), resolve_host_transfers, commit_backup, inc_lock_ref, and the ack bookkeeping. On Inkling-Small-NVFP4 that is ~0.2 ms per node, and most of it is fixed per-call torch-op and Python overhead on small tensors, not per-token work.

Modifications

Write-through backups are now queued at insert and backed up in merged batches at flush. There is no flag. Write-back, the buffer pipeline and the external linker keep their existing paths.

Accuracy Tests

Speed Tests and Profiling

Base upstream/main 70f0b73 vs this PR, TP=4, Rust tree core (the default). Workload: 128 lanes x 16 timed turns, multi-turn (a 1k-token prefix shared by every 16 lanes, 4-5.6k-token first prompt, +256 tokens per turn, outputs 128-512 with ignore_eos). The working set stays on device, so L2 only adds the write-through cost (no host hits). Each round runs L1 (main), L2 (main) and L2 (PR) servers back to back in a rotated order; loss is relative to the same round's L1, and the PR column is paired against main L2 per round. The numbers were taken with the batched path enabled on the PR's earlier heads (e7fa154 / feae617), which run the same write-through code as this head.

host / model (recipe)	rounds	L2 loss vs L1, main	L2 loss vs L1, PR	PR vs main L2, paired
4x B300, Inkling-Small-NVFP4 (Long Context: MXFP8 KV)	5	1.63%	0.80%	+0.85% (t=3.1)
4x GB300, Inkling-Small-NVFP4 (Long Context: MXFP8 KV)	3	2.59%	1.33%	+1.29% (t=17.6)
4x H200, Inkling-Small-NVFP4 (H200 recipe; TP=4 instead of its TP=8)	3	0.68%	0.41%	+0.27% (t=3.4)
4x B300, DeepSeek-V4-Flash (B300 FP4 low-latency)	5	1.96%	0.93%	+1.06% (t=2.1)
4x GB300, DeepSeek-V4-Flash (GB300 FP4 low-latency)	6*	1.96%	1.22%	+0.42% (t=0.9)
4x H200, DeepSeek-V4-Flash-FP8 (H200 FP8 low-latency)	5	L2 1-2% faster than L1	same	-0.04% (t=-0.08)
1x H200, Qwen3-8B (server defaults, 64 lanes)	3	2.84%	2.69%	+0.15% (t=3.4)

* Round 1's L1 server ran before the FlashInfer autotune cache settled and is left out of the loss columns (n=5); the paired column uses all 6 rounds.

Inkling and DSV4 use --max-total-tokens 2097152 (Inkling also --max-mamba-cache-size 1024), DSV4 --hicache-ratio 2. The PR helps where the request-finish bookkeeping is on the critical path (Inkling on Blackwell) and is neutral elsewhere: DSV4 runs at ~3k tok/s with a run-to-run spread larger than its L2 cost, and the H200 Qwen3-8B L2 cost is dominated by the page-size-1 host copy path, which this PR does not touch. An earlier GB300 session with the same commits (7 vs 7 interleaved L2 servers) measured +1.25% (t=3.2); its per-step profile shows the backup path 0.50 -> 0.30-0.33 ms and process_batch_result -0.22 to -0.27 ms, and the largest remaining L2 cost is check_hicache_events (0.77-0.89 ms per step).

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

---

CI States

Latest PR Test (Base): ✅ Run #38109269825
Latest PR Test (Extra): ❌ Run #38109269607
Latest PR Test (AMD ROCm 10): ❌ Run #38109269809