[DHX98]

Title: [Store] Prefetch SSD-only objects to DRAM on exist

Description

With SSD offload enabled, a key whose MEMORY replica has been evicted keeps serving reads from a LOCAL_DISK replica — paid synchronously on the get() critical path. For workloads that probe with exist before reading (prefix cache lookup, async scheduling), the probe is a strong, early "will read soon" signal.

This PR adds an opt-in, best-effort prefetch: is_exist / batch_is_exist with ExistOptions.prefetch_to_memory=true starts an asynchronous SSD→DRAM promotion for keys that turn out SSD-only, so the follow-up get() can be served from DRAM. Promotion happens inside the probe→get queueing window; if it does not finish in time, or fails, or DRAM is saturated, behavior degrades to exactly today's SSD read. No correctness contract changes anywhere.

Mechanism, briefly:

• The exist path does a per-key dedup lookup and a thread-pool enqueue only — no RPC added to the probe (a sharded PrefetchThrottle with TTL dedup and a DRAM-saturation cooldown).
• A bounded pool (4 threads) classifies candidates via read-only admin replica queries (no lease, no promotion-on-hit, no metrics side effects), then promotes local keys via a new RegisterPrefetchTask master RPC plus the same execution chain promotion-on-hit uses (PromotionAllocStart → AllocateBatch → BatchLoad → PromotionWrite → NotifyPromotionSuccess, extracted as a shared helper), or delegates to remote holders through a new additive prefetch_offload_object RPC.
• RegisterPrefetchTask is tenant-aware, holder-checked (holder_id == client_id), shares the promotion_in_flight_ cap and the per-key promotion_tasks entry with promotion-on-hit (double promotion is impossible), and reports PROMOTION_ALREADY_EXISTS distinctly so best-effort callers skip quietly. NotifyPromotionSuccess grants the usual read lease for prefetch promotions so the DRAM replica survives until the follow-up get().
• Any get (single or batched) headed for SSD fires one demand-side promotion attempt for its disk keys (ignore_cooldown bypasses only the client-side throttle backoff — never a master-side gate: the shared in-flight cap, holder check, and dedup TTL all still apply). The get never waits on promotions: it proceeds to the SSD read immediately, and a promoted replica benefits the next get.

Default configuration is a complete no-op: enable_ssd_prefetch=false.

Relationship to prior work

• RFC: [RFC]: Prefetch SSD-Only Objects to DRAM on Exist #2213 (@Pz1116). The API shape (ExistOptions struct rather than a bare bool) follows @LujhCoconut's suggestion on that RFC and the ReplicateConfig pattern.
• This PR builds on the design pioneered by [Feature]Prefetch SSD-Only Objects to DRAM on Exist #2646 (@huangdong2022) and presents a new implementation on current main, shaped by the community's feedback on that PR (core commits credit huangdong2022 as co-author). Beyond a clean rebase, this version:
  • shares the promotion execution chain with promotion-on-hit instead of duplicating it (one chain, one failure-handling posture);
  • makes zero wire-protocol changes to existing RPCs (read-only queries ride the pre-existing admin RPCs; new RPCs are purely additive);
  • addresses the review findings on [Feature]Prefetch SSD-Only Objects to DRAM on Exist #2646 (throttle entries expiring while in-flight; per-key get-side master polling; missing completion callbacks; tenant handling in task registration);
  • ships a test plan that covers the feature end to end: unit tests for the throttle state machine and the master RPC semantics, a Python integration test with a negative control (plain exist must not promote) wired into scripts/ci/run_ssd_offload_smoke.sh with its own master instance (promotion_on_hit=false so the control is meaningful), and a reproducible A/B benchmark methodology.
• RFC [RFC]: Explicit SSD→DRAM Prefetch Trigger for Mooncake Store #3417 (@LujhCoconut): see the alignment discussion there. This PR's mechanism layer (task registration, shared chain, dedup, lease, holder-local execution) is what [RFC]: Explicit SSD→DRAM Prefetch Trigger for Mooncake Store #3417's router-side explicit trigger would also build on; the exist-probe trigger is the RFC's "engine SDK without a router" path. We deliberately skip the frequency sketch (an explicit probe is the hotness signal) and the watermark gate in favor of a DRAM-saturation cooldown that yields to eviction/offload; promotion_in_flight quota is shared, not bypassed. A dedicated prefetch buffer / soft partition is left as follow-up.

Measured impact

DSv4-Flash + vllm-ascend on Ascend A2; 48 long prefixes (at the prefix-cache hit granularity); fill → overflow-to-SSD → settle → measure; promotion_on_hit=false; same dataset and seed across rounds. Absolute latency values are omitted; numbers are TTFT decrease relative to the offload-only baseline of the same round, computed as (baseline − prefetch) / baseline.

Concurrency 2 (store DRAM comfortably sized):

metric	TTFT decrease with prefetch
median	5.2%
mean	5.1%
p99	22.0%

Concurrency 4 (deliberately DRAM-constrained store, ~97.5% full — a
stress regime where eviction churns constantly):

metric	TTFT decrease with prefetch
median	85.2%
mean	81.4%
p99	44.9%

NVIDIA H20:

metric	TTFT decrease with prefetch
median	27.4%
mean	6.7%
p99	0.3%

Why the rounds differ so much: without prefetch, a queued request pays serial SSD reads on the critical path, so the offload-only baseline's TTFT degrades roughly linearly with queue depth, while the prefetch round hides those reads inside the queueing window and stays flat. The decrease scales with queue depth and with the SSD-read share of TTFT; an idle, serial workload sees ≈ 0 decrease by design.

Honest stress metric from the DRAM-constrained run: promotion completed 503, failed 860 (DRAM saturation), and the memory-pressure cooldown correctly yielded ~2.1k times — prefetch never starves eviction; it just loses.

Get-side decomposition (measured; drives one removal)

Earlier revisions also waited on in-flight promotions from the get path (ssd_get_wait_ms). Decomposition measurements — prefetch+kick without waiting vs with waiting (a one-line decoupled build, since the knob previously gated both) — showed the wait never turned a pending get into a DRAM read and added latency instead, so it was removed and the get path returned to the upstream structure plus the kick. Absolute latencies are omitted as above; percentages are relative within each measured pair:

comparison	setup	result
wait's isolated effect	store get median, 15 trials	+13% latency with wait (worse)
wait's isolated effect	engine TTFT median, c=4, DRAM-constrained	+28% TTFT with wait (worse)
kick vs promotion-on-hit	threshold=2, 15 trials	kick promotes on the first touch in 12/15 trials and puts the second touch at DRAM speed; promotion-on-hit does not promote within three touches in 13/15
kept config vs baseline	engine TTFT median, c=4, DRAM-constrained	−20% TTFT vs offload-only

The A/B numbers above were measured on a revision that included the wait; its isolated effect was negative in both tiers, so removing it does not degrade the reported gains. Zero LEASE_EXPIRED on every arm of the decomposition runs.

Design

Full component-level design with diagrams: docs/source/design/store/ssd-prefetch.md (in this PR) and design RFC #4271.

Commit-by-commit guide

1. [Store] Add read-only replica list queries on the client — registers the pre-existing admin read-only RPCs and adds client wrappers (no lease / no promotion / no metrics).
2. [Bugfix][Store] Recognize mounted segments in IsReplicaOnLocalMemory — independent local-memory detection fix (self-contained, could land alone).
3. [Store] Add PrefetchThrottle for best-effort SSD prefetch — sharded dedup state machine + cooldown; pure utility, unit-tested.
4. [Store] Add RegisterPrefetchTask master RPC and prefetch lease — master side: tenant-aware, holder-checked, quota-shared, distinct idempotency, from_prefetch lease. Unit-tested.
5. [Store] Add FileStorage::PrefetchKeys sharing the promotion execution chain — extracts PromoteOneKeyFromLocalDisk (promotion-on-hit behavior unchanged; its per-key loop body moves verbatim into the helper) and adds the best-effort batch driver.
6. [Store] Wire SSD prefetch into exist/get paths — SsdPrefetcher, exist trigger, get-side demand kick, holder RPC. (Updated after review: the get-side wait that this commit originally introduced was measured and removed — see "Get-side decomposition" below.)
7. [Store] Expose ExistOptions in C and Python bindings — store_c, store_py (keyword args), DummyClient (accepted, ignored with a log).
8. [Store][CI] Add prefetch-on-exist integration test — Python integration test wired into scripts/ci/run_ssd_offload_smoke.sh.
9. [Docs][Store] Expand SSD prefetch design doc with component-level design — full component-level design with diagrams (docs/source/design/store/ssd-prefetch.md).
10. [Docs][Store] Fix mermaid syntax in SSD prefetch design diagrams — diagram rendering fixes, docs only.
11. [Store] Apply clang-format and ruff-format to prefetch changes — mechanical formatting, no behavior change.
12. Merge branch 'main' — adapt to upstream [Store] Extract per-object metadata and runtime state headers from MasterService #4127/[Store] Centralize MasterService test access in a test-only peer #4135 (header split, MasterServiceTestPeer), [Store] Add ProbeKey/BatchProbeKey RPCs for lease-free existence probes #3801 (probeKey coexistence), [Store] Migrate lightweight Python utilities to unified package #3586 and [Doc] Group design documents by module in the docs sidebar #4196 (design doc moved to docs/source/design/store/ssd-prefetch.md); no feature changes.

Rolling-upgrade compatibility

• No existing RPC signature changes.
• New RPCs (RegisterPrefetchTask, prefetch_offload_object) are additive: old masters/peers reject them and prefetch degrades to a no-op.
• Read-only queries reuse the pre-existing admin RPC handlers.
• All knobs default off: enable_ssd_prefetch=false.

How Has This Been Tested?

Test commands:

ctest --test-dir build --output-on-failure -R \
  'prefetch_throttle_test|prefetch_task_master_test|client_readonly_query_test|promotion_on_hit_test|file_storage_promotion_test'
bash scripts/ci/run_ssd_offload_smoke.sh   # includes test_prefetch_on_exist.py

Manual end-to-end: A/B TTFT benchmark on vllm-ascend (fill → overflow → settle → measure; see "Measured impact").

Test results:

• Unit tests pass (new: prefetch_throttle_test 11 cases, prefetch_task_master_test 5 cases, client_readonly_query_test 4 cases)
• Integration tests pass (test_prefetch_on_exist incl. negative control: plain exist does not promote)
• Manual testing: A/B TTFT above; promotion-on-hit regression tests green (shared execution chain)

Checklist

• Self-review of my own code
• ./scripts/code_format.sh
• pre-commit on changed files passes
• Documentation updated (docs/source/design/store/ssd-prefetch.md)
• Tests added
• >500 LOC → RFC: [RFC]: Prefetch SSD-Only Objects to DRAM on Exist #2213; implementation design RFC: [RFC]: SSD Prefetch-on-Exist — Design and Implementation #4271 (see also [RFC]: Explicit SSD→DRAM Prefetch Trigger for Mooncake Store #3417 discussion)

Reviews welcome from @LujhCoconut @huangtingwei9988 @Pz1116 @huangdong2022 — the design RFC (#4271) has the component-level details, and #3417 has the alignment discussion.

AI Assistance Disclosure

• AI tools were used. The human submitter has reviewed and can defend every changed line.

[LujhCoconut]

Overall Comment

The mechanism layer is solid. One concern worth a second pair of eyes cc @fcczzz : the net benefit depends less on "DRAM utilization" than on whether the SSD-resident re-reference working set fits into DRAM headroom. PromotionAllocStart never evicts, so promotions only succeed on truly free DRAM;when the hot set is much larger than DRAM (plausibly the common case, since caches usually run full), successful promotions evict each other (thrash) while failed attempts keep costing master-side write-lock RPCs — and the only guard is a fixed-length backoff (5s by default) with no success-rate awareness. The benchmarks cover the favorable regime (48 long prefixes), not the adversarial one where the hot set exceeds DRAM.

That said, the feature is strictly opt-in (default off, complete no-op when off), so the blast radius is contained and I'm okay with it on that basis. I'd still like another pair of eyes on the saturated-regime question; if others share the concern, a follow-up with success-rate-aware backoff (or a configurable watermark gate) plus one "working set ≫ DRAM" benchmark point would close it.

Inline comments

Inline comments below — the three I care most about are the dead cross-process wait branch, the failure-path coupling with the on-hit retry pipeline (which also resets the execution-failure circuit breaker), and the missing early-exit in PrefetchKeys.

1. mooncake-store/src/ssd_prefetcher.cpp:366 — Dead branch: every client-visible replica query filters out non-COMPLETE replicas (TryGetReadableReplicaDescriptor, master_service.cpp:4166), so a PROCESSING memory replica can never be observed here and promotion_in_flight is always false. ssd_get_wait_ms thus only works for promotions started by this process; remote / other-process / on-hit promotions are invisible, contrary to the design doc. Either expose PROCESSING replicas via an admin variant or drop the branch and fix the docs.

2. mooncake-store/src/master_service.cpp:9762 — Two issues: (a) hardcoding execution_failures = 0 resets the kMaxPromotionExecutionFailures circuit breaker that TryPushPromotionQueue otherwise propagates (master_service.cpp:9650), so a persistently failing hot key can loop prefetch → fail → on-hit retry ×3 → prefetch forever when promotion_on_hit=true; (b) NotifyPromotionFailure re-records an on-hit candidate unconditionally (master_service.cpp:10129), and with promotion_on_hit=false the only sweeper early-returns (master_service.cpp:8759), so each prefetch failure permanently leaks a candidate entry (and pins the tenant state, since TenantState::Empty() includes candidates). Suggest skipping candidate re-record for from_prefetch and propagating the existing failure count.

3. mooncake-store/src/file_storage.cpp:1000 — PrefetchKeys should break on the first NO_AVAILABLE_HANDLE: today a saturated 128-key chunk burns ~384 doomed write-lock RPCs (register + alloc + notify per key) before the cooldown engages. Early-exit (releasing registered tasks) then enter cooldown.

Non-blocking

4. mooncake-store/src/real_client.cpp:7414 — [Non-blocking] ignore_cooldown=true bypasses the memory-pressure backoff exactly in the regime it exists for. Per-key dedup bounds the rate, but please document the interaction and consider a global cap on kick-driven attempts (e.g. after N consecutive failures, kicks honor the cooldown too).

5. mooncake-store/include/client_metric.h:561 — [Non-blocking] These counters are write-only: not ylt metrics, not in serialize(), nothing reads them. Wire them into the metrics output or drop them.

6. mooncake-store/include/prefetch_throttle.h:286 — [Non-blocking] With dedup_ttl=0 MaybeSweep never runs, so every distinct key accumulates forever. Skip inserting entries when dedup is disabled, or cap the table.

7. mooncake-store/src/file_storage.cpp:949 — [Non-blocking] Promotion-on-hit already stages through client_buffer_allocator_, but serially (at most one task per heartbeat on the single heartbeat thread); prefetch adds 4 concurrent workers drawing from the same pool that demand SSD reads (LoadBatch) depend on, so with large objects the staging allocations can starve demand gets (BUFFER_OVERFLOW). Worth a prefetch-specific share cap, or at least a doc note.

8. docs/source/design/store/ssd-prefetch.md:116 — [Minor] scripts/ssd-prefetch-ttft-ab/ is not in this PR

[fcczzz]

@zxpdemonio @alogfans Hi, could you help with the remaining code-owner reviews for this PR?

The mooncake-wheel changes include the Python prefetch integration tests and a BufferPool compatibility entrypoint. The Transfer Engine change adds ascendcl linkage to shared_segment_test when USE_ASCEND_DIRECT is enabled.

The Store-side reviews are complete. When you have a chance, could you review the changes in your areas and approve if everything looks good? Thanks!