[vasilevklart]

Every KV cache is shared over CUDA IPC, even when no other worker reads it. Once its manager is dropped, the cache sits in torch's IPC limbo: it stays allocated until exit, and at exit torch's cleanup reads freed memory and segfaults.

This uses the local-only transfer engine when the cache spans one worker (needs_remote_transfer False). Caches another instance reads (P/D) still go over IPC.

1×H100, build and drop 4 KV managers, 8 runs	main	this PR
memory left allocated	16 MiB	0
"shared CUDA tensors" warning at exit	8/8	0/8
segfault at exit (MALLOC_PERTURB_=165)	5/8	0/8

Tests: test_kv_ipc_transfer.py::test_a_cache_no_other_worker_reads_is_freed_with_its_manager fails on main and passes here. GPU KV modules (1×H100) and CPU Core fail the same tests as main, by name. ruff clean.

The mid-run heap corruption seen in the same runs is a separate bug, fixed in #976.

AI assistance (Claude) was used for this PR.

[NSagan271]

🤖 Automated live test, AI-generated (Claude Code). Not checked line by line by a human. Numbers come from one allocation, so treat small deltas as provisional.

Live test: Fix exit segfault from KV caches shared over CUDA IPC

Works. A cache no other worker reads no longer goes through CUDA IPC, so the "shared CUDA tensors" warning at exit goes away on every single-worker config. Caches another worker reads (P/D) still use IPC and still transfer correctly. One limit: the exit crash itself remains for caches that really are shared (below).

H200, SHM, mstar serve, Rust runtime · PR head 61bc461 vs its merge base 5522394 · MALLOC_PERTURB_=165, PYTHONFAULTHANDLER=1 on every process · FlashInfer 0.6.18

Served configs, 2 restarts per arm, SIGINT stop

Config	Workload	"shared CUDA tensors" at exit, main → PR	Requests
Qwen3.5-0.8B	14 greedy prompts + 128 at c=32	2/2 → 0/2	all complete
Whisper large-v3 (KV offload on)	96 at c=64	2/2 → 0/2	all complete
Orpheus colocated	24 speech at c=8	2/2 → 0/2	all complete
Orpheus P/D (KV to a second worker)	24 speech at c=8	2/2 → 2/2 (still IPC, as intended)	all complete
BAGEL P/D	text, image→text, first-token stops, t2i	2/2 → 2/2	outputs byte-identical to main, 0 errors
Qwen3-Omni P/D (3 GPUs)	24 t2s + 16 a2s at c=8	3 warnings → 2 (the caches not shared drop out)	all complete

• On either arm, none of the 24 boots logged a segfault, a heap error or the local-only engine's "Cross-worker KV migration … local-only resource" refusal. SIGINT was enough every time; no process had to be killed.
• The greedy Qwen3.5 outputs match main as often as main matches itself across restarts.

Tests

Check	main	PR
test_a_cache_no_other_worker_reads_is_freed_with_its_manager	absent	passes 3/3
GPU KV/IPC/capture modules (8 files, 104–105 tests), 3 runs	1 failure	1 failure

• The one failure, on both arms: test_kv_ipc_transfer.py::test_a_read_from_another_process_lands_the_named_pages_and_tokens, 1 run in 3 per arm (Fix heap corruption when sizing thrown-away CUDA graph captures #976's arm too). The reader subprocess prints "Producer process has been terminated before all shared CUDA tensors released" and exits with −11. That test shares its cache on purpose, so this looks like the same exit crash on the IPC path this PR keeps. It isn't new, but a P/D deployment can still segfault at exit. That fits the description's scope; worth a follow-up issue.

Not tested

• A multi-node P/D layout (RDMA/TCP transfer engines are untouched by the diff), replicas with prefix_cache_tp.

Commands

mstar serve x --config configs/<cfg>.yaml --gpus … · python -m benchmark.runner --profiling-type closed_loop --model {qwen3.5,whisper_large,orpheus,qwen3omni} … · stop: SIGINT to the server, then the server log is grepped for shared CUDA tensors, Segmentation fault|Fatal Python error, malloc|double free|corrupted, Cross-worker KV migration

[NSagan271]

LGTM, note that this doesn't fix the issue for KV caches that actually have to be shared: (according to claude), torch expects each IPC-shared tensor to be shared once, and the current code in the case of PD disaggregation / CFG parallel end up sharing it multiple times, so the tensor will still remain in limbo.