[ziqifan617]

Summary

Keep tenant isolation enabled without losing legitimate same-tenant KV reuse across workers.

The parent stack already includes cache_salt in HiCache storage keys. This PR fixes a compatibility gap in peer fetching: the router and storage can assign different IDs to the same salted cache page, but native peer-hint membership previously assumed those IDs were interchangeable.

This fixes a false cache miss for the same tenant. It is not a fix for a demonstrated cross-tenant KV leak, and it does not remove the salt or change persisted keys.

Example: reusing a tenant's document on another worker

Assume the same model and cache layout on two serving replicas, called worker 1 and worker 2.

1. Tenant A sends cache_salt="tenant-A" and a long document followed by "Summarize this." Worker 1 processes the document and stores its KV in KVCR.
2. Tenant A sends the same document followed by "What are the main risks?" The request reaches worker 2, which has no local copy.
3. Worker 2 should fetch the cached document prefix from worker 1, then process the new question.

The problem is that the same cached document page has two names:

Component	ID for that page (illustrative labels, not actual hashes)
Router event: "worker 1 has this page"	event-A-123
HiCache/KVCR storage: "read this object"	storage-A-789

Before this fix, worker 2 effectively asks: "Fetch storage-A-789; the router says worker 1 has event-A-123." Native membership compares the storage-page ID against the advertised event IDs, finds no match, and rejects the peer hint. The correct KV exists on worker 1, but worker 2 can miss that reuse opportunity and recompute the prefix.

After this fix, the framework reconstructs a request-local binding between the two IDs using the same tokens, salt, namespace and complete preceding prefix. The adapter translates the hint only when its advertised event ID covers the exact requested storage page. The actual salted storage key remains unchanged.

Tenant B, using cache_salt="tenant-B", produces different IDs. An announcement for tenant A's pages does not match B's binding and does not enable B to reuse A's KV.

Why does this appear with cache salt but not ordinary unsalted requests?

With neither cache_salt nor extra_key, both paths use the same original token-hash chain with no namespace seed. Their page IDs agree.

When namespacing is enabled, the recipes differ:

• Router events preserve the existing salt-only seed format.
• Storage uses a framed namespace containing both extra_key and cache_salt.

Consequently, supplying the same salt to both workers does not make the router ID equal the storage ID: the layers encode it differently. A non-null extra_key can also activate this mismatch without a salt.

The salt itself is not the problem. The bug is the missing translation between these two identity formats. "Legacy" in the source comments means the existing event format retained for compatibility; this PR does not deprecate it or cache salt.

Request	Before this PR	After this PR
No salt and no extra_key	IDs agree; no mismatch from this cause	Existing behavior preserved
Same tokens and same non-empty salt on another worker	Legitimate peer reuse can falsely miss	Covered peer hints translate to storage IDs
Same tokens but different tenant salts	Storage namespaces are separate	Separation preserved

Stack and review base

Direct stacked follow-up to sgl-project#45647. This is a core correctness fix and does not depend on the optional performance work in sgl-project#45649.

• Target branch: ziqifan617/kvcr-hicache-storage
• Exact parent: 705e7a2913d1a6459ef3363f2b8f3e2753a27485
• This PR is in the contributor fork so its base can be the actual head branch of [Feat] hicache: Add KVCR storage backend sgl-project/sglang#45647. It contains one salt-fix commit across 11 files, not the cumulative upstream stack. The three intervening [KVCR 3/5] Add opt-in HiCache delivery and placement controls sgl-project/sglang#45649 performance commits are excluded.
• Existing upstream PRs and their branches are unchanged. After this fix lands in the core branch, [KVCR 3/5] Add opt-in HiCache delivery and placement controls sgl-project/sglang#45649 should inherit it when rebased onto its updated parent. Do not merge this independently of the parent stack; upstream landing/rebase remains separate.

Modifications

• Derive a request-scoped binding from full storage-page hashes to router event IDs using complete token-prefix context; check the matched storage parent before accepting a binding.
• Carry bindings through ordinary/hybrid existence queries, batched reads, indexer/KDA sidecar reads, PP tickets/receivers and decode-side prefix probing.
• Translate only requested storage keys whose bound event IDs are actually covered by the advertised hint; preserve rank-aligned source selection.
• Invalid/incomplete bindings decline peer access rather than forwarding IDs from the wrong domain. Existing local lookup/recompute remains available.
• Preserve existing unsalted behavior, storage/event hash formats, native membership enforcement and operation/watchdog bounds. No opportunistic-query bypass.
• Add eight registered CPU regression cases across the existing adapter and PP fixtures; extend decode propagation expectations.
• Excludes the separate DRAM allocator/THP fix, Mooncake work, FP32 LM-head settings and unrelated performance changes.

Accuracy Tests

Checks on this rebased core-only PR source

• Eight portable helper checks passed using the actual SGLang C++ hash implementation compiled locally. The harness loads the relevant source definitions without GPU-only package imports; this is not a full Linux controller-suite or native-transport run.
• Removing the hint translation causes four of those eight checks to fail, including same-tenant payload membership and fail-closed hint cases.
• Scoped isort 7.0.0, Ruff 0.15.1 checks using the repository's selected rules (F401,F821,F823,UP037), formatting, AST parsing of all 11 changed Python files and git diff --check passed.
• The PR's eight added regression cases and extended decode test preserve their previous method bodies; the surrounding fixtures now use the core-only baseline, without performance-only imports/helpers/tests.
• Direct runs of the five focused registered suites below were attempted locally; each stopped at import because this local environment has no PyTorch. They are not reported as passing.
• The primary dirty checkout and unrelated edits were preserved.

Prior validation of the binding implementation, not a fresh run of this commit

On the earlier performance-stack runtime 142e34c04b (not this core-only base), the private binding backport passed 72 adapter CPU cases and a real GLM-5.3-Flash source-to-destination continuation canary:

• Two independent TP4/EP1 replicas on eight B200s; checkpoint zai-org/GLM-5.3-Flash@46a050bec13002ad29ee795c16e61cf4fa3c4a9c.
• Ordinary and Unicode salts each restored 8,192/8,193 prompt tokens into a fresh destination, then generated 64/64 token IDs identical to independent cold controls.
• 24 successful native source writes / 24 remote deliveries; 520 matching KV/indexer/KDA physical-key digest checks.
• A foreign tenant supplied the source tenant's hint: zero reused tokens and no native delivery.
• 210 exported artifacts checksum verified.

That earlier run also used a separate allocator-only overlay and the existing --enable-fp32-lm-head workaround, neither included here. Router IDs came from actual source events, but the source-selection envelope was assembled by the canary. Same-node TCP/shared-memory, not cross-host RDMA or a deployed Dynamo frontend. The 72-case count includes the separate allocator regression and is not claimed as the exact test count of this PR.

Before this rebase, a Linux-suite rerun was attempted through the retained Pod identity guard, but the named Pod was no longer present. The guard stopped before remote exec. No replacement allocation or GPU workload was started. The earlier native canary is evidence for the binding implementation, not fresh qualification of this rebased core-only commit. Complete Linux CPU suites and core-only native continuation revalidation remain merge gates.

Suggested focused Linux CPU checks (with the normal SGLang test dependencies installed):

CUDA_VISIBLE_DEVICES=99 PYTHONPATH=python python test/registered/unit/mem_cache/test_kvcr_storage.py -v
CUDA_VISIBLE_DEVICES=99 PYTHONPATH=python python test/registered/unit/mem_cache/test_pp_prefetch_ticket.py -v
CUDA_VISIBLE_DEVICES=99 PYTHONPATH=python python test/registered/unit/disaggregation/test_decode_hicache_tree_core.py -v
CUDA_VISIBLE_DEVICES=99 PYTHONPATH=python python test/registered/unit/mem_cache/test_mem_cache_utils.py -v
CUDA_VISIBLE_DEVICES=99 PYTHONPATH=python python test/registered/unit/mem_cache/test_buffer_mode_sidecar.py -v

Speed Tests and Profiling

No sustained-load, throughput, latency or speedup claim. This correctness fix does not relax native deadlines. Full runtime/performance qualification remains separate.

Security boundary

This preserves cache-reuse namespace separation under trusted, correctly assigned tenant salts. It is not an authentication/ACL, encryption or resource-quota feature. Empty salt intentionally aliases the unsalted namespace.

Checklist

• Salt fix rebased directly onto [Feat] hicache: Add KVCR storage backend sgl-project/sglang#45647 with [KVCR 3/5] Add opt-in HiCache delivery and placement controls sgl-project/sglang#45649's performance commits excluded.
• Scoped source/format/import checks.
• Portable helper checks and failure-injection evidence.
• Earlier native transfer/continuation evidence and limitations documented separately.
• Fresh complete Linux CPU suites / CI.
• Native continuation revalidation on the core-only source.
• Parent-stack landing, review and revalidation.