Skip to content

[BugFix] Release HiCache prefetch resources on disagg-prefill bootstrap-queue abort - #30053

Merged
ispobock merged 2 commits into
sgl-project:mainfrom
kflansburg:fix/hicache-prefetch-bootstrap-abort-leak
Jul 5, 2026
Merged

[BugFix] Release HiCache prefetch resources on disagg-prefill bootstrap-queue abort#30053
ispobock merged 2 commits into
sgl-project:mainfrom
kflansburg:fix/hicache-prefetch-bootstrap-abort-leak

Conversation

@kflansburg

@kflansburg kflansburg commented Jul 3, 2026 •
edited by github-actions Bot
Loading

Copy link
Copy Markdown
Contributor

Motivation

When abort_request() processes requests still in disagg_prefill_bootstrap_queue, it aborts the KV sender but never calls release_aborted_request(). Since _add_request_to_queue() calls _prefetch_kvcache() (which increments prefetch_tokens_occupied and creates an ongoing_prefetch entry) before adding the request to the bootstrap queue, each aborted bootstrap-queue request permanently leaks len(prefetch_key) tokens from the counter. Once the counter exceeds prefetch_capacity_limit, prefetch_rate_limited() returns True for every subsequent request and storage prefetch is permanently disabled for the lifetime of the process.

The waiting-queue abort path already calls release_aborted_request() (line 3857); the bootstrap-queue path was never updated to match.

Introduced in v0.5.12 by #23631 (commit 2330482).

Relates to #26886, #27619.

Modifications

Call release_aborted_request() in the bootstrap-queue abort loop, gated on self.enable_hicache_storage, matching the waiting-queue pattern.

Checklist

  • Format: pre-commit run --all-files
  • The fix is a 3-line addition following the existing pattern

Motivation

Modifications

Accuracy Tests

Speed Tests and Profiling

Checklist

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

CI States

Latest PR Test (Base): ❌ Run #28727371683
Latest PR Test (Extra): ✅ Run #28727371619

agent
[BugFix] Release HiCache prefetch resources on disagg-prefill bootstr…
7d31c79 
…ap-queue abort

## Motivation

When `abort_request()` processes requests still in `disagg_prefill_bootstrap_queue`,
it aborts the KV sender but never calls `release_aborted_request()`. Since
`_add_request_to_queue()` calls `_prefetch_kvcache()` (which increments
`prefetch_tokens_occupied` and creates an `ongoing_prefetch` entry) before
adding the request to the bootstrap queue, each aborted bootstrap-queue request
permanently leaks `len(prefetch_key)` tokens from the counter. Once the counter
exceeds `prefetch_capacity_limit`, `prefetch_rate_limited()` returns `True` for
every subsequent request and storage prefetch is permanently disabled for the
lifetime of the process.

The waiting-queue abort path already calls `release_aborted_request()` (line
3857); the bootstrap-queue path was never updated to match.

Introduced in v0.5.12 by sgl-project#23631 (commit 2330482).

Relates to sgl-project#26886, sgl-project#27619.

## Modifications

Call `release_aborted_request()` in the bootstrap-queue abort loop, gated on
`self.enable_hicache_storage`, matching the waiting-queue pattern.

## Checklist

- [x] Format: `pre-commit run --all-files`
- [x] The fix is a 3-line addition following the existing pattern
@gemini-code-assist

gemini-code-assist Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@hzh0425 hzh0425 self-assigned this Jul 4, 2026
@ispobock
ispobock merged commit 48ba79c into sgl-project:main Jul 5, 2026
325 of 348 checks passed
This was referenced Jul 5, 2026
Open
Open
@oleksii-tumanov oleksii-tumanov mentioned this pull request Jul 6, 2026
Open
5 tasks
hzwzwzw added a commit to hzwzwzw/sglang that referenced this pull request Jul 13, 2026
@hzwzwzw @claude
hicache: port P1 upstream fixes — draft alloc + reset + SWA L3 + disa…
556d754 
…gg abort + double-free + DSv4 PP SWA + HiSparse tail

Seven upstream-merged bug/stability fixes for HiCache and DSv4 PP.
All adapted where our fork's structure differs (HostKVCache lives
inline in memory_pool_host.py rather than pool_host/base.py; some
fill_ids semantics differ).

- sgl-project#29823 (1e80d93) fix draft host pool allocator type
  kv_cache_builder.py: pass allocator_type=server_args.hicache_storage_backend
  into the draft host pool kwargs so the draft path uses the same
  allocator kind as target (was defaulting to None -> wrong allocator
  under storage-backed setups).

- sgl-project#26009 (f4029d0) clear storage reset state
  cache_controller.py reset(): also clear host_mem_release_queue and
  reset prefetch_tokens_occupied so no stale credit / stale slots survive
  a scheduler reset.

- sgl-project#27291 (b7ae714) fix SWA L3 cache miss due to prefetch/hit len mismatch
  scheduler.py: prefetch computes new_input_tokens up to
  _compute_max_prefix_len(len(fill_ids)) rather than the raw tail --
  the storage prefetch key was overshooting SWA-window match_prefix by
  up to page_size, causing systematic L3 misses on full-prefix reuse.
  Upstream uses full_untruncated_fill_ids; our fork uses fill_ids
  (equivalent surface, no _untruncated variant).

- sgl-project#30053 (48ba79c) release HiCache prefetch resources on disagg-prefill
  bootstrap-queue abort
  scheduler.py: when aborting a bootstrap-queued disagg-prefill req,
  also call tree_cache.release_aborted_request(req.rid) so its
  in-flight L2/L3 prefetch resources are released. Previously they
  leaked until process shutdown.

- sgl-project#29310 (bd2a5db) detect double-free / double-alloc in HostKVCache
  memory_pool_host.py HostKVCache: track a per-slot slot_used bool
  tensor; alloc asserts none of the returned slots is already used,
  free asserts every freed slot is currently used. Turns silent
  double-free (which corrupts the free_slots list and eventually
  causes phantom slot reuse) into a loud assertion at the call site.

- sgl-project#29106 (c1b5c7e) fix DSv4 PP HiCache SWA allocation and layer mapping
  deepseek_v4_memory_pool.py: size the SWA sub-pool by stage_layer_num
  (= len(stage_ratios)) instead of the model-global layer_num, matching
  the c4/c128 sub-pools which are already stage-local; add
  get_swa_raw_buffer(layer_id) helper that goes through _swa_local_layer_id
  for consistency.
  hybrid_pool_assembler.py: assert that swa_kv_pool.kv_buffer length
  matches transfer_layer_num (catches the mismatch that sgl-project#29106 fixes at
  the source); rename the misnamed c4_state_global_layers list --
  previously it held LOCAL layer ids but was later indexed with
  compress_state_pools[layer_id + start_layer]. Introduce a proper
  c4_state_local_layers (for c4_state_mapping keys) and a true
  c4_state_global_layers list, and index compress_state_pools /
  indexer_compress_state_pools by the real global id (no offset). Drop
  unused c128_state_mapping / c128_state_global_layers now that C128 is
  not registered with HiCache (upstream sgl-project#27293, already in base).

- sgl-project#30408 (1d8e3c2) fix DSV4 HiSparse SWA tail allocation forwarding
  disaggregation/decode.py DecodePreallocQueue:
  - Admission-control available_size for hisparse now uses
    logical_attn_allocator.full_available_size() when the SWA-tail
    prealloc path is active and that method exists, matching the
    allocation actually issued below.
  - _pre_alloc direct-to-host branch now routes through
    alloc_extend_swa_tail (with swa_tail_len and swa_evicted_seqlen
    bookkeeping) when the SWA-tail path is active; falls back to the
    original alloc_logical_only otherwise. Keeps our fork's
    mem_pool_host.alloc(fill_len) call (upstream uses
    alloc_paged_token_slots but our host pool exposes .alloc()).

Verified: AST parse OK, compileall OK across managers + mem_cache trees.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Chronostasys pushed a commit to MindLab-Research/sglang that referenced this pull request Aug 24, 2026
@kflansburg
[BugFix] Release HiCache prefetch resources on disagg-prefill bootstr…
38b2c69 
…ap-queue abort (sgl-project#30053)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ispobock ispobock ispobock approved these changes
@merrymercy merrymercy Awaiting requested review from merrymercy merrymercy is a code owner
@Ying1123 Ying1123 Awaiting requested review from Ying1123 Ying1123 is a code owner
@hnyls2002 hnyls2002 Awaiting requested review from hnyls2002 hnyls2002 is a code owner
@xiezhq-hermann xiezhq-hermann Awaiting requested review from xiezhq-hermann xiezhq-hermann is a code owner

Assignees

@hzh0425 hzh0425

Labels

bypass-fastfail hicache Hierarchical Caching for SGLang run-ci run-ci-extra

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@kflansburg @ispobock @hzh0425