[HiCache] Optimize HiCache host pool free-list release

[huangtingwei9988]

Motivation

HiCache host-pool free() currently appends released slots back to free_slots with an eager torch.cat:

self.free_slots = torch.cat([self.free_slots, indices_cpu])

For large host pools, free_slots can contain millions of entries. Even when only a small number of slots is released, this operation copies the entire free-list on every free() call. In decode, this shows up inside scheduler.process_batch_result after HiCache write-back / host eviction and can introduce multi-millisecond CPU spikes.

The fix is to make host-pool slot release lazy, matching the device allocator’s free_pages + release_pages pattern:

• free() only records released chunks in release_slots.
• available_size() includes both free_slots and pending released slots.
• alloc() continues to allocate from free_slots directly while possible.
• Only when free_slots is insufficient does alloc() merge release_slots back into free_slots once, then proceed with the normal allocation path.

This removes the eager large torch.cat from the hot free() path while preserving the existing free-list allocation semantics.

benchmark

In this run, the host KV cache capacity reported by metrics was 6,611,072 token slots

Validation on Ling v2.6, TP=4, batch=1, 2k input / 128 output, HiCache direct/page_first_direct/write_through

Profile result, pre-patch vs current lazy-merge patch:

first scheduler.process_batch_result:
  mean: 5.520 ms -> 2.050 ms
  max:  9.776 ms -> 2.776 ms

aten::cat inside the first process_batch_result:
  mean max: 2.914 ms -> 0.045 ms
  max:      6.207 ms -> 0.052 ms

total scheduler.process_batch_result:
  mean: 24.859 ms -> 12.260 ms

Speed Tests and Profiling

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review and Merge Process

1. Ping Merge Oncalls to start the process. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
   • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

---

CI States

Latest PR Test (Base): ❌ Run #29303604544
Latest PR Test (Extra): ✅ Run #29303604433

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[huangtingwei9988]

/tag-run-ci-label

[hzh0425]

LGTM，PLS Add this optimization for MambaPoolHost.free() and DeepSeekV4PagedHostPool.free()

[hzh0425]

/rerun-group hicache

[github-actions]

Results for /rerun-group hicache:

🚀 1-gpu-h100 (2 tests): ✅ View workflow run

cd test/ && python3 registered/hicache/test_hicache_spec_file_storage.py
cd test/ && python3 registered/hicache/test_hicache_variants.py

🚀 2-gpu-h100 (4 tests): ✅ View workflow run

cd test/ && python3 registered/hicache/test_hicache_spec_mooncake_storage.py
cd test/ && python3 registered/hicache/test_hicache_storage_file_backend.py
cd test/ && python3 registered/hicache/test_hicache_storage_mooncake_backend.py
cd test/ && python3 registered/hicache/test_hicache_storage_runtime_attach_detach.py

🚀 1-gpu-5090 (1 test): ✅ View workflow run

cd test/ && python3 registered/hicache/test_hicache_storage.py

🚀 4-gpu-h100 (2 tests): ✅ View workflow run

cd test/ && python3 registered/hicache/test_hicache_storage_3fs_backend.py
cd test/ && python3 registered/hicache/test_qwen35_hicache.py

[huangtingwei9988]

/rerun-failed-ci

[huangtingwei9988]

/rerun-failed-ci

[huangtingwei9988]

/rerun-failed-ci

[huangtingwei9988]

/rerun-failed-ci

[huangtingwei9988]

/rerun-failed-ci

[hzh0425]

The failed test is unrelated

[hzh0425]

should we still need flatten? @huangtingwei9988

[huangtingwei9988]

flatten() is redundant because indices is already 1-D