Skip to content

[HiCache] Fix sidecar pool life-time issue - #31668

Closed
stepinto wants to merge 8 commits into
sgl-project:mainfrom
stepinto:fix_mamba_lifecycle_0718
Closed

stepinto wants to merge 8 commits into
sgl-project:mainfrom
stepinto:fix_mamba_lifecycle_0718

Conversation

@stepinto

@stepinto stepinto commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

Motivation

This PR fixes #31445.

Modifications

There is a life-time issue for sidecar pools. If a prefetch request is aborted before prefetch reads in the IO thread, the host indices are freed, result into a use-after-free hazard.

There 3 changes:

a) Move the prefetch code KV-derived pool to CacheController.page_transfer, and use the same variable PrefetchOperation.completed_tokens to track its progress. Do not save hits to PrefetchOperation.extra_pool_hit_pages. Only non-KV derived pools (e.g. SWA) are performed in HybridCacheController.

b) Use pool_transfer_done as a completion mark for non-KV derived pools, e.g. SWA. Use that variable to determine whether the scheduler thread or the prefetch IO thread to release memory when the prefetch operation is terminated early, e.g. in best effort mode.

c) There protection in the original code, as introduced by #27550, does not cover best_effort code path. It is removed in this PR. The original production (by checking pool_transfers_done in can_terminate) introduced another issue: If the prefetch thread stucks at reading side-car pool, the prefetch of that request will not be timeout in scheduler thread.

Tests

SWE-Bench

Tested SWE-Bench on a single-box sglang server with best_effort configuration and assertion enabled in prefetch IO thread to detect use-after-free in host memory pool. Scores are good. No assertion failure occured.

DSv4 Flash

Cache hit: L1=8.2% L2=18.4% L3=38.7

Group Total Resolved
Baseline 500 356
This PR 500 364

(*) Testing scores may vary due to randomness and environment issues.

GLM-5.2

Cache hit: L1=3.7% L2=1.6% L3=14.1%

Group Total Resolved
Baseline 500 405
This PR 500 400

(*) Testing scores may vary due to randomness and environment issues.

Multi-turn benchmark

Tested multiturn_bench on DSv4 flash and GLM5.2. The L3 cahce hit ratio seems good. (The last two rounds should hit L3.)

DSv4 Flash

nohup env HF_HUB_OFFLINE=1 \
SGLANG_ENABLE_UNIFIED_RADIX_TREE=1 \
sglang serve \
    --model=deepseek-ai/DeepSeek-V4-Flash \
    --tp-size 8 \
    --max-total-tokens=$((256*1024)) \
    --moe-runner-backend flashinfer_mxfp4 \
    --disable-cuda-graph \
    --mem-fraction-static 0.85 \
    --reasoning-parser deepseek-v4 \
    --tool-call-parser deepseekv4 \
    --host 0.0.0.0 \
    --port 30000 \
    --max-total-tokens=500000 \
    --swa-full-tokens-ratio=0.4 \
    --enable-metrics \
    --enable-hierarchical-cache \
    --hicache-storage-backend=mooncake \
    --hicache-storage-backend-extra-config '{"master_server_address": "127.0.0.1:50051", "local_hostname": "localhost", "metadata_server": "http://127.0.0.1:8080/metadata", "global_segment_size": "512gb", "protocol": "tcp"}' \
    > sglang.log &

python benchmark/hicache/bench_multiturn.py  \
    --num-clients=128 \
    --num-rounds=10 \
    --disable-auto-run  \
    --model=deepseek-ai/DeepSeek-V4-Flash  \
    --enable-round-barrier \
    --request-rate 1



Performance metrics summary:
  Total requests: 1280 at 1.0 requests per second
  Average Prompt Length: 3104.00 tokens
  Average Output Length: 64.00 tokens
  P90 Prompt Length: 5696 tokens
  P99 Prompt Length: 5696 tokens
  P90 Output Length: 64 tokens
  P99 Output Length: 64 tokens
  Average TTFT: 0.36
  P90 TTFT: 0.45
  P99 TTFT: 0.53
  Median TTFT: 0.35
  Max TTFT: 0.91
  Average ITL: 0.1430
  P90 ITL: 0.2494
  P99 ITL: 0.3968
  Median ITL: 0.1239
  Max ITL: 0.7457
  Average latency: 9.37
  P90 latency: 10.14
  P99 latency: 11.19
  Median latency: 9.33
  Max latency: 11.69
  Input token throughput: 2485.35 tokens per second
  Output token throughput: 51.24 tokens per second
  Request Throughput: 0.80 requests per second
  Cache Hit Rate: 0.806765
Per-round metrics:
  Round 0: Average TTFT = 0.34s, Cache Hit Rate = 0.000000 (128 requests, 128 clients)
  Round 1: Average TTFT = 0.34s, Cache Hit Rate = 0.470588 (128 requests, 128 clients)
  Round 2: Average TTFT = 0.34s, Cache Hit Rate = 0.615385 (128 requests, 128 clients)
  Round 3: Average TTFT = 0.35s, Cache Hit Rate = 0.685714 (128 requests, 128 clients)
  Round 4: Average TTFT = 0.34s, Cache Hit Rate = 0.813210 (128 requests, 128 clients)
  Round 5: Average TTFT = 0.37s, Cache Hit Rate = 0.830189 (128 requests, 128 clients)
  Round 6: Average TTFT = 0.36s, Cache Hit Rate = 0.838710 (128 requests, 128 clients)
  Round 7: Average TTFT = 0.37s, Cache Hit Rate = 0.845070 (128 requests, 128 clients)
  Round 8: Average TTFT = 0.36s, Cache Hit Rate = 0.893750 (128 requests, 128 clients)
  Round 9: Average TTFT = 0.41s, Cache Hit Rate = 0.898876 (128 requests, 128 clients)

GLM-5.2

nohup env HF_HUB_OFFLINE=1 \
SGLANG_ENABLE_UNIFIED_RADIX_TREE=1 \
sglang serve \
  --model-path=PhalaCloud/GLM-5.2-W4AFP8 \
  --tp-size=8 \
  --max-total-tokens=$((256*1024)) \
  --enable-hierarchical-cache \
  --hicache-ratio=2 \
  --log-level=debug \
  --enable-metrics \
  --host=0.0.0.0 \
  --mem-fraction-static=0.75 \
  --hicache-ratio=2 \
  --reasoning-parser glm45 \
  --tool-call-parser glm47 \
  --hicache-storage-backend=mooncake \
  --hicache-storage-backend-extra-config '{"master_server_address": "127.0.0.1:50051", "local_hostname": "localhost", "metadata_server": "http://127.0.0.1:8080/metadata", "global_segment_size": "512gb", "protocol": "tcp"}' \
  > sglang.log &

python benchmark/hicache/bench_multiturn.py  \
    --num-clients=128 \
    --num-rounds=10 \
    --disable-auto-run  \
    --model=PhalaCloud/GLM-5.2-W4AFP8  \
    --enable-round-barrier \
    --request-rate 1

Performance metrics summary:
  Total requests: 1280 at 1.0 requests per second
  Average Prompt Length: 3104.00 tokens
  Average Output Length: 64.00 tokens
  P90 Prompt Length: 5696 tokens
  P99 Prompt Length: 5696 tokens
  P90 Output Length: 64 tokens
  P99 Output Length: 64 tokens
  Average TTFT: 0.36
  P90 TTFT: 0.60
  P99 TTFT: 0.98
  Median TTFT: 0.24
  Max TTFT: 1.30
  Average ITL: 0.0202
  P90 ITL: 0.0211
  P99 ITL: 0.2124
  Median ITL: 0.0146
  Max ITL: 0.9828
  Average latency: 1.63
  P90 latency: 2.27
  P99 latency: 3.21
  Median latency: 1.54
  Max latency: 4.48
  Input token throughput: 2870.08 tokens per second
  Output token throughput: 59.18 tokens per second
  Request Throughput: 0.92 requests per second
  Cache Hit Rate: 0.834729
Per-round metrics:
  Round 0: Average TTFT = 0.27s, Cache Hit Rate = 0.000000 (128 requests, 128 clients)
  Round 1: Average TTFT = 0.26s, Cache Hit Rate = 0.527574 (128 requests, 128 clients)
  Round 2: Average TTFT = 0.27s, Cache Hit Rate = 0.691707 (128 requests, 128 clients)
  Round 3: Average TTFT = 0.27s, Cache Hit Rate = 0.770982 (128 requests, 128 clients)
  Round 4: Average TTFT = 0.30s, Cache Hit Rate = 0.818182 (128 requests, 128 clients)
  Round 5: Average TTFT = 0.31s, Cache Hit Rate = 0.848467 (128 requests, 128 clients)
  Round 6: Average TTFT = 0.36s, Cache Hit Rate = 0.870716 (128 requests, 128 clients)
  Round 7: Average TTFT = 0.45s, Cache Hit Rate = 0.887104 (128 requests, 128 clients)
  Round 8: Average TTFT = 0.50s, Cache Hit Rate = 0.899707 (128 requests, 128 clients)
  Round 9: Average TTFT = 0.59s, Cache Hit Rate = 0.910025 (128 requests, 128 clients)

Reproduce case

Passed the reproduce case (#32197).

Checklist

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): ❌ Run #31145822964
Latest PR Test (Extra): ✅ Run #31145822817

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces synchronization and validation logic for hybrid prefetch results across attention groups to enforce an all-or-nothing policy. If any hybrid pool falls short, the prefetch result is discarded and resources are released. Feedback on the changes points out a potential race condition in hiradix_cache.py where extra_pools are released unconditionally even if the pool transfers are not yet complete, suggesting to only release them when operation.pool_transfers_done is true.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment on lines +1643 to +1646
self.cache_controller.append_host_mem_release(
host_indices=operation.host_indices[:completed_tokens],
extra_pools=pool_transfers,
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

In _sync_and_check_hybrid_prefetch_result, extra_pools is released unconditionally as pool_transfers. However, if the prefetch operation has not completed its pool transfers yet (operation.pool_transfers_done is False), the background IO thread might still be holding or writing to these transfer buffers. Releasing them prematurely can lead to a race condition or use-after-free/double-free issues.

To prevent this, we should only release pool_transfers if operation.pool_transfers_done is True, matching the logic used in unified_radix_cache.py.

Suggested change
self.cache_controller.append_host_mem_release(
host_indices=operation.host_indices[:completed_tokens],
extra_pools=pool_transfers,
)
self.cache_controller.append_host_mem_release(
host_indices=operation.host_indices[:completed_tokens],
extra_pools=pool_transfers if operation.pool_transfers_done else None,
)

@stepinto
stepinto force-pushed the fix_mamba_lifecycle_0718 branch from b7c5c2b to edb953b Compare July 18, 2026 12:56
@hzh0425 hzh0425 self-assigned this Jul 18, 2026
Comment thread python/sglang/srt/mem_cache/unified_radix_cache.py
@stepinto
stepinto force-pushed the fix_mamba_lifecycle_0718 branch from edb953b to 9b2a519 Compare July 23, 2026 11:48
@stepinto stepinto changed the title [WIP][HiCache] Fix sidecar pool life-time issue [HiCache] Fix sidecar pool life-time issue Jul 23, 2026
and getattr(operation, "pool_transfers", None)
and not getattr(operation, "pool_transfers_done", True)
):
can_terminate = False

operation_terminated = operation.is_terminated()

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But under can_terminate_prefetch, does wait_complete still need to check pool_transfers_done?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fxed.

@github-actions github-actions Bot added the hicache Hierarchical Caching for SGLang label Jul 24, 2026
@huangtingwei9988

Copy link
Copy Markdown
Collaborator

/tag-and-rerun-ci

@stepinto
stepinto force-pushed the fix_mamba_lifecycle_0718 branch 2 times, most recently from 7e6f321 to 6de9b08 Compare July 27, 2026 01:41
@stepinto

Copy link
Copy Markdown
Contributor Author

Rebase to latest code of main branch.

transfer.host_indices = operation.host_indices
if transfer.keys is None:
transfer.keys = operation.hash_value
pass

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this be passed?

for dsv4, some sidecar components may share indices with swa, not kv

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This else branch is transfer.indices_from_pool == PoolName.KV. The case you mentioned are in the previous if branch, not removed.

self._barrier_attn_groups()
self.dec_host_lock_ref(last_host_node_id, anchor_lock_params)
del self.ongoing_prefetch[rid]
pool_transfers = [x for xfers in comp_xfers.values() for x in xfers]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pls add some unit test for this case

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Added test_release_aborted_request_l3_prefetch_io_in_progress and test_release_aborted_request_l3_prefetch_io_done.

@hzh0425 hzh0425 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Almost lgtm, just a bit worried that this might break the dsv4 hicache

prefix_keys = operation.prefix_keys
kv_derived_transfers = [
transfer
for transfer in getattr(operation, "pool_transfers", [])

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For full-only model, this might throw error?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. Tested Qwen3-32B with L3.

@@ -28,6 +28,7 @@
HiCacheStorageExtraInfo,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

btw, We need to benchmark to see if the current segmented KV + sidecar mode causes any performance regression.
We can use moocnake to benchmark the latency under patterns like 256k and 512k

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. I have tested that for 256K. My box cannot test 512K.

@stepinto
stepinto force-pushed the fix_mamba_lifecycle_0718 branch from a25e5ea to b137657 Compare August 3, 2026 03:04
@stepinto

stepinto commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

I tested GLM 5.2 w4a8 on H20 box with 256K L3 prefetch.

Testing method:

# sglang
nohup env HF_HUB_OFFLINE=1 \
SGLANG_ENABLE_UNIFIED_RADIX_TREE=1 \
sglang serve \
  --model-path=PhalaCloud/GLM-5.2-W4AFP8 \
  --tp-size=8 \
  --kv-cache-dtype fp8_e4m3 \
  --enable-hierarchical-cache \
  --hicache-storage-prefetch-policy wait_complete \
  --hicache-ratio=2 \
  --enable-metrics \
  --host=0.0.0.0 \
  --mem-fraction-static=0.75 \
  --hicache-ratio=2 \
  --reasoning-parser glm45 \
  --tool-call-parser glm47 \
  --hicache-storage-backend=mooncake \
  --hicache-storage-backend-extra-config '{"master_server_address": "127.0.0.1:50051", "local_hostname": "localhost", "metadata_server": "http://127.0.0.1:8080/metadata", "global_segment_size": "512gb", "protocol": "tcp"}' \
  > sglang.log 2>&1 &

# Run this command for 11 times.  The first time will fill L3 cache.  The rest are prefetch from L3.
python benchmark/hicache/bench_multiturn.py  \
    --num-clients=1 \
    --num-rounds=1 \
    --disable-auto-run  \
    --model=PhalaCloud/GLM-5.2-W4AFP8  \
    --request-rate 1 \
    --request-length $((256*1024)) \
    --output-length 64

I ran this for baseline (131bd51) and this PR (b137657).

The results are very interesting. After this PR, TTFT is reduced ~10s -> ~9s.

image

The reason is, before this PR, there is a large sidecar read in the end of prefetch, which results into long-tail.

Here are a prefetch for baseline and PR. The first is baseline. There are large requests that other requests in the end.

prefetch_old prefetch_new

@stepinto
stepinto force-pushed the fix_mamba_lifecycle_0718 branch 2 times, most recently from 77e3feb to 079467d Compare August 6, 2026 07:24
@stepinto

stepinto commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

Rebased to latest code.

@stepinto
stepinto force-pushed the fix_mamba_lifecycle_0718 branch from 079467d to ee76c62 Compare August 6, 2026 08:24
@stepinto

stepinto commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

Replay the same benchmark as #30393.

Run sharegpt benchmark. Flush cache. Re-run sharegpt benchmark, loading from L3.

The accept len and accept rate seems similar before flush cache.

Before flush cache: accept len ≈ 3.58,accept rate ≈ 0.52
After flush cahce: accept len ≈ 3.54,accept rate ≈ 0.51

The results are similar to the results in #30393.

nohup env HF_HUB_OFFLINE=1 \
SGLANG_ENABLE_UNIFIED_RADIX_TREE=1 \
sglang serve \
    --model=deepseek-ai/DeepSeek-V4-Flash-0731 \
    --tp-size 4 \
    --max-total-tokens=$((256*1024)) \
    --moe-runner-backend flashinfer_mxfp4 \
    --speculative-algorithm DSPARK \
    --disable-cuda-graph \
    --mem-fraction-static 0.90 \
    --chunked-prefill-size 4096 \
    --reasoning-parser deepseek-v4 \
    --tool-call-parser deepseekv4 \
    --host 0.0.0.0 \
    --port 30000 \
    --max-total-tokens=500000 \
    --swa-full-tokens-ratio=0.1 \
    --enable-metrics \
    --enable-hierarchical-cache \
    --hicache-storage-backend=mooncake \
    --hicache-storage-backend-extra-config '{"master_server_address": "127.0.0.1:50051", "local_hostname": "localhost", "metadata_server": "http://127.0.0.1:8080/metadata", "global_segment_size": "512gb", "protocol": "tcp"}' \
    > sglang.log &


  python -m sglang.bench_serving \
    --backend sglang \
    --host 127.0.0.1 --port 30000 \
    --model deepseek-ai/DeepSeek-V4-Flash-0731 \
    --dataset-name sharegpt \
    --num-prompts 200

@stepinto

stepinto commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

Tested dsv4 flash 0731 with EAGLE3. Got similar results as in link. Need to patch #33344.

Before flush cache: 2.461 / 0.487
After flush cache: 2.340 / 0.447

nohup env HF_HUB_OFFLINE=1 \
SGLANG_ENABLE_UNIFIED_RADIX_TREE=1 \
sglang serve \
  --trust-remote-code \
  --mem-fraction-static 0.8 \
  --model-path deepseek-ai/DeepSeek-V4-Flash-0731 \
  --moe-runner-backend marlin \
  --speculative-draft-model-path AQ-MedAI/DeepSeek-V4-Flash-0731-eagle3 \
  --tp-size 8 \
  --speculative-algorithm EAGLE3 \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --speculative-draft-attention-backend fa3 \
  --host 0.0.0.0 \
  --port 30000 > sglang.log 2>&1 &

  python -m sglang.bench_serving \
    --backend sglang \
    --host 127.0.0.1 --port 30000 \
    --model deepseek-ai/DeepSeek-V4-Flash-0731 \
    --dataset-name sharegpt \
    --num-prompts 200

Leoyzen added a commit to Leoyzen/sglang that referenced this pull request Aug 8, 2026
…project#32035 sgl-project#33656 sgl-project#32183 sgl-project#33145)

Applied PRs (latest from GitHub):
  sgl-project#33288  Indexer logits OOM fix
  sgl-project#30393  HiCache packed/sidecar draft caches
  sgl-project#31170  DPA prefix_affinity load balancing
  sgl-project#33795  DSpark compact ragged-verify CUDA graph JIT race
  sgl-project#32467  C128 plan-kernel warp barrier
  sgl-project#33865  DSpark x prefill CP unblock
  sgl-project#30371  SWA state pool sizing (storage page)
  sgl-project#33358  FlashMLA norm-rope K-tokens-per-block ILP
  sgl-project#33872  num_draft_tokens clamp + extend_len==0 skip (supersede sgl-project#32183)
  sgl-project#34002  Sidecar backup vacuously-successful fix (replaces sgl-project#33656, with tests)
  sgl-project#33862  Reclaim redundant host mirrors after storage backup
  sgl-project#31315  Avoid repeated Mooncake gets after stale hits
  sgl-project#32327  Q8KV8 sparse MLA prefill backend (flashmla_sparse_q8)
  sgl-project#31668  Fix sidecar pool life-time (use-after-free on prefetch abort)
  sgl-project#31195  TP0 verify-token-budget broadcast (adapted to get_schedule() API)

Dropped (per user request or superseded):
  sgl-project#32771  IndexCache C4 top-k reuse — has bug
  sgl-project#32035  DSpark C128 online compressor — has bug
  sgl-project#33656  Superseded by sgl-project#34002 (same fix + unit tests)
  sgl-project#32183  Superseded by sgl-project#33872 (included in supersede PR)
  sgl-project#33145  Base f01f706 already has superior reasoning-effort profile system

Conflicts resolved:
  sgl-project#31195: adapted to base get_schedule().disable_overlap_schedule API
  sgl-project#32327: path remapped jit_kernel/ -> kernels/jit/ and kernels/ops/attention/
  sgl-project#31668: applied cleanly on top of sgl-project#30393+sgl-project#34002+sgl-project#33862 modifications
@stepinto

Copy link
Copy Markdown
Contributor Author

This PR is moved to #27010.

@stepinto

Copy link
Copy Markdown
Contributor Author

Close this issue, as #27010 has been merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

UnifiedRadixCache: extra-pool prefetch lifecycle missing on abort (segfault under timeout policy)

4 participants