[TianDi101]

Summary

Converges UMBP eviction onto one path. Whoever decides a key must go — the master, or the node itself — calls PeerPool::Evict, and the bytes are freed by that medium's single Evict() implementation. Nothing evicts inline on a put.

It also folds in two follow-ups:

• feat(umbp): byte-sized eviction rounds; demote asynchronously instead of inside EvictKey #742: sizes the master's rounds in bytes and makes demotion asynchronous.
• feat(umbp): peer-driven eviction with master-ranked victims (opt-in) #760: adds eviction modes. Nodes now evict locally by default, at their own watermarks. Two opt-in modes remain: master-ranked victims, and the master's own rounds.

Rebased on main (after #754). Please squash on merge: the branch adds the design doc in its first commit and deletes it again.

Why

A committed key could be freed five ways, and only the master's request went through PeerPool:

Path	Problem
Master EvictKey	Shipped keys only, dropping the tier the master measured. The peer freed by placement: on a non-on_evict tier it deleted the key from every backend, and with copy-mode promotion (key in DRAM + SSD) SSD pressure freed the DRAM copy and left SSD full.
PageBackend masterless round	A second copy of Evict's body, inline on commit. Ran before the pool's watermark check, so it deleted the cold keys watermark offload was about to move to SSD. Invisible to InstrumentedBackend.
PeerSsdManager round	Inline after every SSD write (plus evict-then-retry on a failed write), in every mode, under the pool lock; bypassed PeerPool.
SPDK daemon SpdkSsdTier::EvictLRU	Evicted on allocation failure without telling PeerSsdManager — it kept listing the keys, and the master kept routing reads to bytes that were gone.
Tier transitions	Fine — the primitive, used as intended.

The master's own rounds were also slow and coarse:

• Too small: 32 candidates per over-full (node, tier) every 5 s, about 4 MB/s at the 512 GiB deployment eviction_manager.cpp cites. Draining one 0.9 → 0.7 band took about 6.7 hours.
• Stale and summed: the master reacts to a heartbeat up to 5 s old, against one fixed 0.9/0.7 pair applied to each node's summed per-medium usage. It can't see one full backend inside a larger medium.

Changes

1. Per-tier evict requests (4254bd052)

• EvictionVictim{key, tier}: MasterEvictStrategy::SelectVictims and EvictKeyDispatcher carry the tier each victim was charged to.
• EvictKeyRequest gains repeated TierType tiers = 2, parallel to keys. Old peers ignore it; an old master sends none and the peer frees the copy that currently holds each key.
• PeerPool::Evict(PoolEvictRequest{key, tier, backend_id}) frees exactly one copy: demote it if it is the placement and its tier has a downstream (any trigger), else drop that copy only. Drops run outside the pool lock; lifecycle_mutex_ keeps ClearLocal out.

2. One eviction implementation per medium (5da726828, 3a0621f64)

• MediumBackend::Evict is the only code that frees a committed key for capacity. PageBackend::MaybeEvictToLowWatermark and PeerSsdManager::EvictToLowWatermark (with its watermarks and evict-and-retry) are removed; a write that does not fit fails.
• MediumBackend::EnableLocalEviction is replaced by ordering hooks that free nothing: TrackEvictionOrder() and EvictionCandidates(max) → EvictionOffer{key, bytes}, coldest first, skipping leased/pinned keys.
• The SPDK tier no longer auto-evicts on allocation failure.

3. Local watermark eviction through the same call (5da726828)

• PeerPool::EnableLocalEviction(backend, high, low) starts one background worker. Commits, NO_SPACE allocations and finished transitions only wake it.
• The worker takes offers until their bytes cover the overage and calls PeerPool::Evict({key, backend_id}, kReclaim), the same call EvictKey makes, so a tier with a downstream demotes instead of deleting. Lowest tier first, ≤64 keys per call, bounded by a byte budget as well as live usage.
• New metric mori_umbp_client_local_evict_total{event=round|key|no_candidate|stalled|master_key}; the SSD medium's eviction_round event is removed.

4. Byte-sized rounds and asynchronous demotion (d0faaa820, formerly #742)

• Master: evict_batch_size (32 rows) is replaced by max_evict_bytes_per_round (env UMBP_EVICT_MAX_BYTES_PER_ROUND, default 0 = the whole overage above the low watermark). The store returns every eligible row and the strategy trims to the byte budget.
• Peer: PeerPool::Evict queues a demotion on the transition worker instead of copying inside the call, so neither EvictKey nor the local worker waits on data movement. It returns PoolEvictResult{key, bytes_freed, outcome}, where the outcome is one of kFreed, kDemotionQueued, kInFlight or kNone.
• Worker: an eviction's demotion runs whatever the tier's own watermark says. If nothing downstream has room, the copy is dropped instead, unless another transition owns the key.
• Budget: queued and in-flight demotions count toward the local budget, so a pass never queues more than the overage.

5. Eviction modes (34e67ed26, 0af37b45f, formerly #760; master_ranked fixes from e2e testing 458e3e914, 3935422db, e8b950a44)

Each node chooses how it evicts when it has a master: PoolClientConfig::eviction_mode, env UMBP_EVICTION_MODE.

Mode	DRAM / HBM / SSD evicted by	Victims from	Master's automatic rounds for this node
local (default)	the node, at its own per-backend watermarks	the node's own LRU; it never asks the master	skipped
master_ranked	the node, at its own watermarks	the master's new GetEvictionCandidates RPC: this node's keys on that medium (optionally one logical tier), least recently accessed first, master-leased keys skipped. Falls back to local LRU.	skipped
master	the master's EvictKey rounds (SSD still local)	the master's strategy	run (the previous behaviour)

• Skip flag: a self-evicting node sends peer_driven_eviction on register and on every heartbeat, and the master's EvictionManager leaves it out of its rounds. The master keeps the flag in memory only, so there is no store schema change.
• EvictKey stays: every mode still serves it, so the master (or a future user-facing API) can evict on demand.
• Watermarks: self-evicting nodes use a tiered policy's per-tier watermarks when there is one; otherwise SSD's own pair, or the node pair for DRAM/HBM.
• Master-ranked details:
  • The node reports the keys it reads locally on its heartbeats (HeartbeatRequest.local_reads, deduped, at most 64k per beat), and the master records an access for each. Without this a local_first node's hits never reached the master's recency, so its hottest keys ranked coldest (458e3e914). master mode reports them too, since its rounds rank by the same recency.
  • The eviction pass never blocks on the ranking (3935422db): it waits at most 20 ms for its first query, and takes a slower answer when it arrives if it is under 3 s old. One query per backend is in flight, at most every 200 ms; the RPC deadline is 3 s. At e2e scale the in-memory master needs ~0.6 s per DRAM query and ~1.9 s per SSD query (it scans its whole index).
  • One pass can reclaim a whole watermark band (108k keys in the e2e), far more than one answer carries (≤4096 keys), so the pass asks again for what its budget still needs as each answer runs out, and evicts by local order in between (e8b950a44).
  • The master reports value sizes, so they're rounded up to the backend's pages to match the budget.
• Compatibility: an older master doesn't advertise support. A self-evicting node still evicts locally, logs a warning that the master may also evict it in its rounds, and master_ranked falls back to local LRU.

Behavior changes

Situation	Before	After
Node with a master, DRAM/HBM fills	waits for the master's next 5 s round on a heartbeat up to 5 s old	evicts at its own watermark immediately (local); set UMBP_EVICTION_MODE=master for the old behaviour
Master evicts a DRAM+SSD key, SSD over budget	DRAM copy freed (wrong medium) or both deleted	only the SSD copy
Master evicts from a watermark-trigger tier with a downstream	deleted from every backend	demoted
Master eviction round	32 keys per (node, tier) per 5 s	the whole overage, optionally capped by UMBP_EVICT_MAX_BYTES_PER_ROUND
EvictKey on a demoting tier	copied the bytes inside the RPC	queues the copy and returns
No master, tier has an offload target	local round deletes cold keys before offload runs	demoted
No master, DRAM fills	evicted synchronously on the committing thread	background worker; a burst faster than it can see NO_SPACE briefly
SSD write finds no room	evict a round, retry once	fails; the worker reclaims at the watermark
SPDK allocation fails	daemon silently drops its oldest keys	the write fails
Tiered policy, self-evicting node	SSD / node watermark pairs	the logical tier's pair

Test plan

• Full tests/cpp/umbp suite (53 tests) in rocm/mori-dev with RDMA: 51/53. The two failures are pre-existing on main, and both passed 3 of 3 run on their own:
  • umbp_cross_node_smoke: RDMA defaultContext assertion under ctest -j 8.
  • umbp_pool_client_ranges: the GPU gather intermittently reads back zeros.
• Repeats: PeerPoolLocalEviction.* 30/30, MasterPeerDrivenEviction.* 30/30, and the end-to-end eviction-mode tests 5/5.
• An earlier simulation squash-merged feat(umbp): converge eviction onto one path; nodes evict locally by default #741 into main at 097bb470f with the master-ranked commit on top: it applied cleanly and the suite passed 52/53. Not re-run after the eviction-modes commit.
• Coverage:
  • Tier-carrying victims, including through the master → pool chain.
  • Peer service with scoped, unscoped and mismatched tiers.
  • Scoped PeerPool::Evict: tier, backend, absent copy, watermark-tier demotion, full-downstream drop.
  • Local eviction drains exactly to the low watermark, honours recency, never offers leased keys, and demotes with a downstream. It counts held copies toward the budget, loses no key alongside watermark offload, and rejects invalid config.
  • Evict returns kDemotionQueued while the copy is still held, and a repeat call returns kInFlight.
  • A round frees the whole overage (77 keys, not 32), and the env limit throttles it to 10.
  • Master ranking by recency, byte budget, logical-tier filter and key cap. A self-evicting node gets no EvictKey rounds.
  • Ranked candidates go first and page-round to the exact overage; the medium covers the rest. A source slower than the pass waits for doesn't hold eviction up.
  • End to end (test_umbp_peer_driven_eviction, real master + DRAM node over gRPC):
    • local evicts within 3 s without asking the master;
    • master_ranked takes all 7 victims from the master, and keys the node read locally rank behind every other;
    • against an older master, master_ranked falls back to local;
    • master leaves DRAM to the master's rounds.
  • SsdBackend offers candidates and SSD drains through the pool; PeerSsdManager no longer evicts on writes.
• Tests that fill tiny DRAM tiers on purpose to exercise routing (umbp_cross_node_smoke, umbp_tier_benchmark) now run in master mode, the behaviour they were written against.
• SPDK (3a0621f64): only syntax-checked against 3rdparty/spdk/include; needs an SPDK-enabled build.
• Default full build (ops + Python bindings on): built with BUILD_OPS/PYBINDS=OFF; the bindings reference none of the changed APIs.
• E2E, local (default): the whole chain (feat(umbp): converge eviction onto one path; nodes evict locally by default #741 + fix(umbp): SSD staging no longer fails local loads; a local read holds its bytes until it is done #762 + fix(umbp): retry eviction demotions instead of dropping the only copy; count what is dropped #763 + feat(umbp): serve metrics from a masterless node, and summarise offload/load/evict/promote/demote at INFO #683) ran the InferenceY DSv4 agentic recipe on n06-21 with DRAM 300 GB → SSD 1.5 TB (test/umbp-evict-e2e-v2 @ 2e1663d91): 283,993 tok/s, 0 failed loads, 0 dropped demotions, all old-bug signatures at 0.
• E2E-scale checks of master_ranked (a master beside the standalone server; found the three fixes above):
  • Smoke (4 GB DRAM / 8 GB SSD, a hot set re-read throughout): before the fixes the hot set survived 0/512 and DRAM sat at 100% (3,519 NO_SPACE); after, 512/512 in DRAM, 0 NO_SPACE, 95% of victims master-ranked, and local is unchanged.
  • Full run on c160d07ef (fixes 1–2): completed, with 0 GPU faults, 0 dropped demotions and no put failures. But the master ranked 0 of 913k victims, which led to fix 3.
  • Full run on the final heads (b8d526f23): pending.
  • Two earlier attempts died to an sglang GPU fault in its own device memory (not UMBP).
• Throughput with a master: 127,665 vs 283,993 tok/s without one, with the same hit split. It's tracked in UMBP: master-backed standalone server halves e2e throughput vs masterless (DSv4 agentic, DRAM→SSD tiers) #767 and isn't caused by the ranking (that run ranked nothing).

Follow-ups (not in this PR)

• Make master_ranked cheap: a master-side per-(node, tier) LRU index. Today each GetEvictionCandidates scans the master's index (every key in the in-memory store), the same work as one master round.
• Measure the ranking: whether the master's ranking earns its RPC over local is still open. Its recency is per key, not per copy, so it can't tell which copy (DRAM or SSD) is the hot one.
• Remove the master's fixed 0.9/0.7: it's only used by master mode now.
• feat(umbp): serve metrics from a masterless node, and summarise offload/load/evict/promote/demote at INFO #683: its PageBackend local-evict counters conflict with this branch, which moves those counters to the pool.

🤖 Generated with Claude Code

[TianDi101]

Combined into #753 (with #742, #683 and the fixes from the DRAM→SSD end-to-end run). Leaving this open until #753 lands.

[TianDi101]

End-to-end validation of the eviction PR chain: passed.

Branch test/umbp-evict-e2e-v2 @ 2e1663d91 = main (097bb470f) + merges of #741, #762, #763 and #683, nothing else. (The #683 merge drops PageBackend's local-evict counters, which #741 removed, and adds #763's K dropped summary line.) Workload: InferenceY single-node agentic (DeepSeek-V4-Pro fp4, TP8 + DP attention, MTP, conc 256, CI run 29816's config) on n06-21, 3600 s profiling. UMBP is a standalone server with no master (so default local eviction), with a 300 GB DRAM tier demoting to a 1.5 TB SSD tier on 3 drives.

	earlier e2e on #753	this chain (exact PR heads)
Input throughput	281,015 tok/s	283,993 tok/s
Prefix hit (HBM + UMBP)	92.03%	91.99%
Failed loads / SSD read or write errors / sglang crashes	0 / 0 / 0	0 / 0 / 0
DRAM→SSD demotions / copies dropped	1.48M / not measured	1.47M / 0
Demotion failures retried until they landed (#763)	not measured	5,048 (SSD staging full), all landed
SSD keys deferred to a later round / staging released early (#762)	2.93M / 60.3M	2.83M / 60.3M
Local eviction rounds / stalls (#741)	19 / 0	19 / 0

The log signatures of the 7 bugs this e2e originally found all stay at 0: whole-batch staging failures, UMBP get failed, overdue local holds, SSD write failures, PrepareReadBatch read failed, dropped demotions, and any [error] line. An earlier pass on the pre-trim heads gave the same result (277,392 tok/s, 0 failures).

Notes:

• The harness sets UMBP_STANDALONE_STARTUP_TIMEOUT_MS=1800000. Since fix(umbp): keep the standalone server serving when its GPU-registered DRAM pool is migrated #756 the server reports ready while its 300 GB pool is still being GPU-registered, and under concurrent model loading sglang's default 30 s connect timed out. That is not part of this chain.
• One rerun was lost to an sglang GPU memory fault in one DP rank (its own kernel, no UMBP data being loaded at the time); the rerun passed.

🤖 Generated with Claude Code