Skip to content

[HiCache] Resolve write_back to write_through under buffer_only host memory - #42754

Merged
hanming-lu merged 1 commit into
mainfrom
oss-sync-e1e0d1bc99
Oct 6, 2026
Merged

hanming-lu merged 1 commit into
mainfrom
oss-sync-e1e0d1bc99

Conversation

@metamergebot

@metamergebot metamergebot commented Oct 6, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

With --hicache-host-memory-mode buffer_only the host pool is only a staging buffer: every cached page lives in the storage backend, so there is no retained host tier for --hicache-write-policy write_back to defer writes into. Argument resolution rejected the pair at startup. That turns a launch config which carries write_back as its general HiCache policy into a failed launch the moment it switches the host memory mode, even though the only policy buffer-only can run is already determined by the mode. The same file already resolves other incompatible HiCache pairs (layout vs. I/O backend, storage backend vs. layout) instead of rejecting them.

Modifications

  • validate_hicache_host_memory_mode resolves write_back to write_through under buffer_only through declare_resolution, with an info log naming the rewrite, instead of raising. The rewrite is that narrow: an explicit write_through_selective stays as given, and the cache mode keeps whatever policy it was given.
  • TestHiCacheArgs.test_buffer_only_resolves_write_back_to_write_through pins the resolved policy for each write policy under buffer_only and for write_back under cache. The write_back case fails on the previous code with the old ValueError.

The HiCache hook already runs before the optimistic-prefill gate in the resolution pipeline, so that gate reads the resolved write_through. No documentation describes the rejection, so none changes.

Ran locally: test/registered/unit/server_args/test_server_args.py (TestHiCacheArgs), test/registered/unit/server_args/test_resolution_declarations.py and test/registered/unit/mem_cache/test_decode_retraction_backup.py, all passing; the pinned pre-commit linters on both files.

Accuracy Tests

Not applicable: argument resolution only, no change to model forward or kernels.

Speed Tests and Profiling

Not applicable.

Checklist


CI States

Latest PR Test (Base): 🚫 Run #37452625943
Latest PR Test (Extra): ⚠️ Not enabled -- add run-ci-extra label to opt in.
Latest PR Test (AMD ROCm 10): ❌ Run #37452626037

…memory

With --hicache-host-memory-mode buffer_only the host pool is only a staging
buffer and every cached page lives in the storage backend, so there is no
retained host tier for --hicache-write-policy write_back to defer writes
into. Argument resolution rejected the pair at startup, which turned a
launch config that carries write_back as its general HiCache policy into a
failed launch the moment it switched the host memory mode, even though the
only policy buffer-only can run is already determined by the mode.

Resolve write_back to write_through under buffer_only through
declare_resolution, with an info log naming the rewrite, instead of
raising. The rewrite is that narrow: an explicit write_through_selective
stays as given, and the cache mode keeps whatever policy it was given.

A server-args unit test pins the resolved policy for each write policy
under buffer_only and for write_back under cache; the write_back case
fails on the previous code with the old ValueError.

Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com>
@metamergebot metamergebot added the run-ci CI: run the baseline test suite on this PR label Oct 6, 2026
@github-actions github-actions Bot added the hicache Hierarchical Caching for SGLang label Oct 6, 2026
@metamergebot metamergebot added the bypass-fail-fast CI: a failing job no longer aborts its siblings (lint still gates) label Oct 6, 2026
@hanming-lu
hanming-lu merged commit a34cba3 into main Oct 6, 2026
285 of 328 checks passed
@hanming-lu
hanming-lu deleted the oss-sync-e1e0d1bc99 branch October 6, 2026 18:02
maocheng23 added a commit to maocheng23/sglang that referenced this pull request Oct 7, 2026
Picks up sgl-project#42754, whose server-args test the Intel CPU image (built from
the PR head) otherwise runs against the pre-sgl-project#42754 HiCache hook.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cursor Bot pushed a commit to cipheraxat/sglang that referenced this pull request Oct 9, 2026
…memory (sgl-project#42754)

Co-authored-by: merrymercy <merrymercy@users.noreply.github.com>
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bypass-fail-fast CI: a failing job no longer aborts its siblings (lint still gates) hicache Hierarchical Caching for SGLang run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants