[mem_cache] skip duplicates host evict via environ

[Liwansi]

Motivation

When device-side memory is constrained, evicted cache lines are continuously offloaded to host memory. Once host memory reaches capacity, a host-side eviction policy must be enforced. Under a write-back strategy, cache lines that are redundantly held on both device and host are preferentially selected for eviction. Consequently, prefix cache lines that were previously offloaded from device to host may be re-transmitted, thereby degrading overall throughput during benchmarking.

Modifications

To mitigate the throughput degradation caused by redundant re-transmission of prefix cache lines, we introduce an environment variable SGLANG_HICACHE_SKIP_HOST_DUPLICATE_RECLAIM that bypasses the _reclaim_full_host_duplicates eviction path. When set to 1, the system skips the preferential eviction of device–host duplicate cache lines and proceeds directly to the general host eviction routine (drive_host_eviction), thereby preserving previously offloaded prefix entries on the host side and avoiding unnecessary re-offloading overhead.

Accuracy Tests

None

Speed Tests and Profiling

140k + 95%cache

Before:

After:

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review and Merge Process

1. Ping Merge Oncalls to start the process. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
   • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

---

CI States

Latest PR Test (Base): ✅ Run #34211872549
Latest PR Test (Extra): ❌ Run #34211872289
Latest PR Test (AMD ROCm 7.2): ❌ Run #34211872520

[iforgetmyname]

/tag-and-rerun-ci

[Liwansi]

/rerun-failed-ci

[alphabetc1]

Could you also implement this policy for the Rust TreeCore? Currently this environment variable is silently ignored when SGLANG_UNIFIED_RADIX_TREE_CORE_BACKEND=rust.
Please also provide benchmark results showing the transfer reduction and throughput impact.

[Liwansi]

Could you also implement this policy for the Rust TreeCore? Currently this environment variable is silently ignored when SGLANG_UNIFIED_RADIX_TREE_CORE_BACKEND=rust. Please also provide benchmark results showing the transfer reduction and throughput impact.

The profiling and benchmark results have been attached. However, I'm not familiar with the Rust-side implementation, so I'm unable to adapt it at this time. Apologies for that.

[Liwansi]

/rerun-failed-ci