[ziqifan617]

Stack and review base

Depends on #45647. Logical base: ziqifan617/kvcr-hicache-storage at 705e7a2913d1a6459ef3363f2b8f3e2753a27485.

GitHub base is main, so Files changed is cumulative until ancestors land. Upstream cannot target a base branch in the contributor fork. Exact incremental review: ziqifan617/sglang@705e7a2...142e34c

Do not merge before the parent. Rebase/revalidate after it lands. HiCache and linker are independent sibling modes.

Motivation

Keep the earlier HiCache performance work available as a separate review slice, without enlarging initial KVCR enablement or silently changing defaults.

Modifications

• Bounded delivery windows and configurable read/metadata-query batching.
• Opt-in native descriptor coalescing and UCX split sizing, requiring compatible KVCR support.
• Process-owned anonymous DRAM, optional huge-page advice and audited NUMA placement.
• Explicitly opt-in Python thread-switch tuning; preserve default behavior.
• Preserve source claims, sparse fallback, per-pool validation and terminal native lifetime.
• Preserve the pinned upstream positional argument ordering; append the new option separately.

The optional KVCR SDK tuning is a separate companion draft; basic HiCache enablement does not depend on it.

Accuracy Tests

Four full ten-prompt runs through Dynamo passed semantic/request/output/native-byte checks. All 160 measured outputs matched the previous reference; 40 measured restores each completed 30,400,315,392 bytes. 713 evidence files were checksum-verified.

Speed Tests and Profiling

Qwen3-32B-FP8, two same-node TP1 B200 replicas, 116K input/four output tokens, identical runtime and Dynamo source on each side. Old SGLang d037381bf7 vs this 142e34c04b:

Order	Old restore TTFT p50	New restore TTFT p50
Old then new	1523.47 ms	1239.99 ms
New then old	1265.01 ms	1252.53 ms

Both orders passed the prospective 5% median-regression screen. This is no observed regression for this fixed microbenchmark, not a reproducible 18.6% speedup or an isolated causal claim for this PR. Only two runs per revision; old results varied and the new stack includes an upstream rebase.

Both sides used the same private upstream-aligned runtime (FlashInfer 0.7.2, CUTLASS 4.8.0, DeepGEMM 0.2.3, sgl-kernel 0.4.10). This source-checkout comparison does not certify a new image, balanced-NUMA tuning, cross-host transfer or broader models.

Known merge gate

Upstream has advanced since the tested base. Fresh merge-tree comparison against a3d00eff58b6c22ea7539f882cc10c537634200b conflicts in python/sglang/srt/arg_groups/field_order.py. Keep this draft until the ordered-field contract is reconciled and the resulting source revalidated. The published head is the tested source, not an untested rebase.

Checklist

• Tests, opt-in documentation and matched model regression evidence.
• Scoped formatting checks recorded.
• Reconcile upstream conflict and rerun applicable checks.
• Upstream CI/review.

Full draft stack

Shared foundation #45646
├── HiCache storage #45647
│   └── HiCache performance #45649
└── Linker contracts #45648
    └── Direct linker #45650

• Foundation: [KVCR 1/5] Share rank-local endpoint contracts #45646
• HiCache storage: [Feat] hicache: Add KVCR storage backend #45647
• HiCache performance: [KVCR 3/5] Add opt-in HiCache delivery and placement controls #45649
• Linker contracts: [KVCR 4/5] Add request-scoped linker hooks and owned inventory #45648
• Direct linker: [Feat] cache: Add KVCR direct-device linker #45650
• KVCR optional SDK tuning: perf: add opt-in DRAM descriptor coalescing and UCX batching ai-dynamo/kvcr#113
• KVCR SGLang quick-start preview: docs: add SGLang HiCache quick start alongside vLLM ai-dynamo/kvcr#114

These are upstream draft review slices with logical dependencies, not branches based on upstream staging refs. Dependent Files changed views are cumulative; the incremental comparison above isolates this slice. Do not merge dependent drafts before their ancestors. The original #43635 and existing Dynamo PRs remain unchanged.

[mintlify]

Preview approval required

This pull request comes from a fork, so Mintlify won't build a preview until someone with write access approves commit 142e34c.

Approve preview