[Performance] Enable CUDA IPC pool handle cache by default

[mickqian]

Summary

• enable the existing CUDA IPC pool-handle cache by default
• gate it on the resolved cuda_ipc multimodal feature transport mode
• preserve SGLANG_USE_IPC_POOL_HANDLE_CACHE=0 as an explicit rollback
• log that the cache reuses the existing bounded pool and does not reserve another pool

This does not enable CUDA IPC feature transport by default. CPU transport remains the default. Users must still explicitly select --mm-feature-transport=cuda_ipc, which reserves the configured bounded pool and reduces KV-cache headroom.

Root cause

Without the handle cache, every transported multimodal feature reconstructs the same long-lived pool through torch.UntypedStorage._new_shared_cuda(). That repeatedly opens and closes the same CUDA IPC mapping on the scheduler critical path.

The pooled transport already carries a stable pool allocation and byte offset. Reusing one consumer mapping for the lifetime of that pool removes the redundant mapping lifecycle. Stale entries retain the existing invalidate-and-retry fallback.

In matched TP4 torch-profiler traces:

Scheduler operation	Cache off	Cache on
cudaIpcOpenMemHandle	32 calls / 292.94 ms	0
cudaIpcCloseMemHandle	32 calls / 219.99 ms	0
CUDA allocation calls	38 calls / 135.59 ms	38 calls / 7.14 ms
scheduler.process_input GPU annotation	682.04 ms	22.01 ms

The ViT and language-model prefill portions were stable in the same traces; the improvement comes from the feature-transport mapping lifecycle.

PR-before vs PR-after performance

Hardware/model: 4x NVIDIA H100 80GB, TP4, Qwen/Qwen3.6-35B-A3B-FP8.

Serving configuration: breakable prefill CUDA graph only (2048/4096/8192-token captures), full decode graph, FA3, multimodal DP encoder, mixed chunk, and a 512 MiB CUDA IPC feature pool. Each request used four seeded random JPEGs sized from 512x512 to 768x1024 and 32 text-input tokens.

Method: A/B/A server sequence (cache on / cache off / cache on), three formal repetitions per sequence after three warmup requests, with the radix cache flushed before every formal run. “After” is the median of all six cache-on repetitions; “before” is the median of the three cache-off repetitions. The source and all launch flags were otherwise byte-identical.

Workload	Metric	Before: cache off	After: cache on	Change
output=1, burst	Request throughput	7.55 req/s	20.43 req/s	+170.6%
	Mean TTFT	2808.30 ms	842.36 ms	-70.0%
	P99 TTFT	3142.78 ms	1145.97 ms	-63.5%
output=1, rate=4	Request throughput	6.11 req/s	6.59 req/s	+7.9% (arrival-limited)
	Mean TTFT	496.30 ms	97.70 ms	-80.3%
	P99 TTFT	843.24 ms	160.68 ms	-80.9%
output=64, burst	Request throughput	5.59 req/s	17.17 req/s	+207.2%
	Output throughput	191.43 tok/s	588.05 tok/s	+207.2%
	Mean TTFT	1761.37 ms	360.77 ms	-79.5%
	Mean E2E	2007.29 ms	560.14 ms	-72.1%
	Mean TPOT	7.67 ms	6.45 ms	-15.9%

Memory gate

Both sides used the same 512 MiB producer pool. The handle cache only retains a mapping to that allocation; it does not allocate a second pool. In the controlled B/A server restart, the four GPUs had identical memory use after the first three-request warmup and before the first formal workload:

69534 / 68076 / 68058 / 67578 MiB with cache off and with cache on.

Across all formal-run pre-measurements, per-GPU medians differed by less than 100 MiB between the two configurations. A launch with the environment variable completely unset also logged the default-on policy and started with 68966 / 68022 / 68022 / 67542 MiB, matching the fresh cache-off server.

The CUDA IPC transport itself still reserves its configured pool and remains opt-in because that reservation can reduce KV-cache capacity.

Validation

• test_mm_process_config.py -k MultimodalFeatureTransportRuntime: 3 passed on NVIDIA H100
• test_cuda_ipc_transport.py: cross-process pooled reconstruction passed on NVIDIA H100
• test_server_args.py -k MultimodalFeatureTransport: 6 passed on NVIDIA H100
• launch and 24-request random-image serving smoke with SGLANG_USE_IPC_POOL_HANDLE_CACHE unset: passed
• pre-commit on all changed files: passed

---

CI States

Latest PR Test (Base): ❌ Run #29585133436
Latest PR Test (Extra): ❌ Run #29585133351

[gemini-code-assist]

Code Review

This pull request enables CUDA IPC pool-handle caching by default (setting SGLANG_USE_IPC_POOL_HANDLE_CACHE to true) and refactors its resolution to be instance-specific rather than a module-level global variable. It also adds logging for the caching status and includes corresponding unit tests. The reviewer suggested defensively checking if self.cudaipc_mmfeature_pool is not None before accessing its _pool_ipc_handle attribute to prevent potential AttributeErrors.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

To prevent potential AttributeErrors, it is safer to defensively check if self.cudaipc_mmfeature_pool is not None before attempting to access its _pool_ipc_handle attribute.

Suggested change

	pool_ipc_handle=(
	self.cudaipc_mmfeature_pool._pool_ipc_handle
	if _IPC_POOL_HANDLE_CACHE
	if self.use_ipc_pool_handle_cache
	else None
	),
	pool_ipc_handle=(
	self.cudaipc_mmfeature_pool._pool_ipc_handle
	if (self.use_ipc_pool_handle_cache and self.cudaipc_mmfeature_pool is not None)
	else None
	),

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
lmsysorg	🟢 Ready	View Preview	Jul 17, 2026, 1:46 PM

💡 Tip: Enable Workflows to automatically generate PRs for you.

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
lmsysorg	🟡 Building	–	Jul 17, 2026, 1:44 PM

💡 Tip: Enable Workflows to automatically generate PRs for you.