From 444e511df87c7162b72e3f5aa420af162ff67228 Mon Sep 17 00:00:00 2001 From: TianDi101 Date: Sat, 10 Oct 2026 10:27:14 +0000 Subject: [PATCH 01/19] docs(umbp): design for converging eviction onto one path Eviction is decided in three places (master EvictKey, PageBackend's masterless round, PeerSsdManager's after-write round) and freed in two more ways than the medium's Evict (the page round's own copy of it, and the SPDK daemon's EvictLRU). The master's request also drops the tier it measured, so the peer frees by placement and can delete copies nobody asked about. The doc lays out the target: the master names (key, tier); PeerPool::Evict is the one routing entry point both the master and a background local eviction worker call; MediumBackend::Evict is the only code that frees a committed key; nothing evicts on the put path. Co-Authored-By: Claude Opus 5.5 --- src/umbp/doc/design-eviction-convergence.md | 307 ++++++++++++++++++++ 1 file changed, 307 insertions(+) create mode 100644 src/umbp/doc/design-eviction-convergence.md diff --git a/src/umbp/doc/design-eviction-convergence.md b/src/umbp/doc/design-eviction-convergence.md new file mode 100644 index 000000000..f41b3690c --- /dev/null +++ b/src/umbp/doc/design-eviction-convergence.md @@ -0,0 +1,307 @@ +# UMBP Eviction Convergence — Design + +**Goal:** one eviction path. Whoever decides that a key has to go — the master, +or a node that has no master — asks `PeerPool::Evict`, and the bytes are freed +by that medium's single `Evict` implementation. Nothing evicts inline on a put. + +**Scope:** `src/umbp/distributed/` (master eviction, peer service, `PeerPool`, +`PageBackend`, `SsdBackend` / `PeerSsdManager`) and the SPDK tier the SSD medium +reaches through the proxy daemon (`src/umbp/local/tiers/spdk_ssd_tier.cpp`). + +**The three requirements:** + +1. The master's eviction request names the medium it measured as over-full. +2. Each medium has exactly one eviction implementation, and every eviction goes + through it. +3. On a node with no master, watermark eviction calls the same function the + master's request does. + +--- + +## 1. Where eviction happened before + +Five places freed a committed key. Only the first two went through the +medium's `Evict`, and only the first went through `PeerPool`. + +| # | Site | Runs when | How it frees | Problem | +|---|---|---|---|---| +| 1 | Master `EvictionManager` → `EvictKey` RPC | Master timer; a (node, tier) is over the master's watermark | `PeerService::EvictKey` → `PeerPool::Evict(kReclaim)` → `MediumBackend::Evict` | The request carries **keys only**. The master picks victims per (node, tier) but `SelectVictims` returns node → keys and `EvictKeyRequest` is `repeated string keys`, so the tier is lost. The peer frees by the key's placement: on a tier without `on_evict` offload it deletes the key from **every** backend. With copy-mode promotion a key lives in DRAM *and* SSD, so SSD pressure can free DRAM instead, and DRAM pressure can delete the SSD copy too. | +| 2 | Tier transitions (offload / promotion) | After a key is copied | `source->Evict` (drop the moved-from copy), `target->Evict` (undo a failed target) | None — this is the primitive, used as intended. | +| 3 | `PageBackend::MaybeEvictToLowWatermark` | No master only; inline on commit and on a failed allocation | Its own copy of `PageBackend::Evict`'s body | A second implementation. It runs below `PeerPool`, so `on_evict` demotion never happens, the pool's placement goes stale, and `InstrumentedBackend` never sees it. It runs *inside* `PeerPool::BatchCommit`, before the pool's `watermark` offload check, so it frees the tier down to its low watermark first and the offload never fires: cold keys are deleted instead of moved to SSD. It also runs on the put path under the backend's exclusive lock. | +| 4 | `PeerSsdManager::EvictToLowWatermark` | Every mode, master or not; inline after every SSD write, plus evict-then-retry when a write fails | `PeerSsdManager::Evict(key)` (the right function) | Bypasses `PeerPool` and `InstrumentedBackend` (its victims and bytes are counted nowhere that is exported). Runs inside `SsdBackend::BatchCommit`, i.e. inside `PeerPool::BatchCommit` holding the pool lock. | +| 5 | `SpdkSsdTier::EvictLRU` in the SPDK proxy daemon | `ssd_backend=spdk`; inline when an allocation fails | Erases keys from the daemon's own map | Nobody is told. `PeerSsdManager` keeps listing the keys, no REMOVE reaches the master, and the master keeps routing reads to bytes that are gone. Fragmentation can trigger it below `PeerSsdManager`'s own watermark. | + +So "local eviction does not call `Evict()`" (row 3) was not the only gap: +there were three triggers, two layers, and a hidden fifth path under SSD. + +--- + +## 2. Target shape + +``` + policy: which keys, which medium routing: which copy, demote or drop freeing: one per medium + ─────────────────────────────────── ──────────────────────────────────── ─────────────────────── + master EvictionManager ─EvictKey(key, tier)─┐ + ├─▶ PeerPool::Evict(requests) ─┬─▶ MediumBackend::Evict(keys) + PeerPool local eviction worker ─────────────┘ │ ├ PageBackend::Evict + (no master: every medium; │ └ SsdBackend::Evict + with a master: SSD only) │ └ PeerSsdManager::Evict + └─ demote ─▶ TierTransitionExecutor + copy down, then source->Evict +``` + +Three rules hold after this change: + +1. **`MediumBackend::Evict` is the only code that frees a committed key for + capacity.** Freeing that is *not* eviction stays where it is: aborting or + reaping an uncommitted slot, discarding a stale or duplicate commit, and + `ClearLocal`. +2. **`PeerPool::Evict` is the only place that decides what happens to a copy** + — move it down a tier or drop it. Every eviction policy calls it. +3. **No eviction work on a put path.** A put may wake the eviction worker. It + never selects victims and never frees. + +--- + +## 3. Requirement 1 — per-tier eviction requests + +**Master.** A victim is now a key *and* the medium to free it from: + +```cpp +// types.h +struct EvictionVictim { + std::string key; + TierType tier = TierType::UNKNOWN; +}; +``` + +- `MasterEvictStrategy::SelectVictims` returns `node_id → vector`, + taking the tier from the candidate's `Location.tier` — the same tier whose + budget it charged. A key over budget in two tiers of one node appears twice, + once per tier, which is now meaningful. +- `EvictKeyDispatcher::DispatchEvictKey` takes victims; `MasterPeerStubPool` + fills both proto fields below. Chunking to the gRPC limit is unchanged. + +`MasterEvictStrategy` is a public, pluggable interface. Its return type changes; +there is no other implementation in tree and the Python bindings do not expose it. + +**Wire.** `EvictKeyRequest` gains a field parallel to `keys`: + +```proto +message EvictKeyRequest { + repeated string keys = 1; + // tiers[i] is the medium keys[i] should be freed from. Ignored unless it has + // exactly as many entries as keys. + repeated TierType tiers = 2; +} +``` + +| Master | Peer | Result | +|---|---|---| +| new | new | each key freed from the named medium | +| new | old | old peer ignores `tiers`: previous behavior | +| old | new | `tiers` empty: each key is freed from the medium that currently holds it (§4) | + +**Peer.** `PeerServiceServer::EvictKey` turns the request into +`PoolEvictRequest{key, tier}` and calls `PeerPool::Evict`. + +--- + +## 4. `PeerPool::Evict` — the one routing entry point + +```cpp +struct PoolEvictRequest { + std::string key; + // The medium to free the key from. UNKNOWN: the medium that holds the key's + // placement (what a request without a tier means). + TierType tier = TierType::UNKNOWN; + // A specific backend instead of a medium; the local eviction worker knows + // exactly which backend is over its watermark. Wins over `tier`. + uint32_t backend_id = BackendRegistry::kMaxBackends; +}; + +std::vector Evict(const std::vector& requests, PoolEvictMode mode); +// Unchanged convenience form: every key with no scope. +std::vector Evict(const std::vector& keys, PoolEvictMode mode); +``` + +**Which copy.** `backend_id` names it. Otherwise `tier` selects the backend of +that medium that holds the key, preferring the placement. With neither, it is +the placement (or, without one, the first owner in read order). No copy there → +0 bytes freed and nothing else happens. + +**`kReclaim`, per copy:** + +1. The key is mid-transition → 0 bytes freed; the caller retries next round + (the same contract as a read lease). +2. The copy is a *draining* source (left behind by an offload whose source + delete hit a read lease) → retry deleting that copy. +3. The copy is the key's placement and its logical tier has an offload target → + **demote**: copy it down, then `source->Evict`. If the target already holds + the key (a copy-mode promotion left one there) nothing is copied and only the + source is freed. +4. Otherwise, or if the demotion failed (no room downstream) → **drop** that copy + only: `Evict({key})` on that one backend. +5. If the freed copy was the placement, point the placement at the remaining + copy, or forget the key. + +`bytes_freed` is what the named copy released, so it is now accurate per tier. + +**`kDiscard`** is unchanged: delete from every backend, no demotion. + +**Decision — demote regardless of trigger.** Before, only an `on_evict` tier +demoted; evicting from a `watermark` tier dropped the key from every backend. +Now `offload_trigger` only says whether the pool *also* moves keys proactively at +the tier's watermark; any eviction of a copy whose tier has a downstream moves it +there. Two reasons: + +- The data stays reachable instead of being deleted while a lower tier had room. +- On a masterless node, local eviction and the pool's `watermark` offload now do + the *same* thing to the same tier. Before, the local round deleted exactly the + cold keys the offload was about to move (§1 row 3). + +The cost is that an `EvictKey` handled for a `watermark` tier now copies bytes, +as `on_evict` tiers already did inside the same handler. + +**Locking.** `Evict` holds `lifecycle_mutex_` shared for the whole call, so +`ClearLocal` cannot run underneath it. Demotions release `operation_mutex_` for +the byte copy as before (`RunTransition`), and the drops now also run outside +`operation_mutex_`: an SSD delete is device IO and must not stall every commit +on the node. Placement repair re-checks the backends under the lock afterwards, +so a key re-committed in that window keeps its placement. + +--- + +## 5. Requirement 2 — one eviction implementation per medium + +| Medium | The one implementation | Removed | +|---|---|---| +| DRAM / HBM (`PageBackend`) | `PageBackend::Evict` | `MaybeEvictToLowWatermark` and its four call sites (`Allocate`, `BatchAllocate`, `Commit`, `BatchCommit`), the round mutex and warning latch | +| SSD (`SsdBackend` → `PeerSsdManager`) | `PeerSsdManager::Evict(key)`, reached only through `SsdBackend::Evict` | `EvictToLowWatermark`, the check after every `Write` / `WriteBatch`, the evict-then-retry on a failed write, the manager's own watermarks and `eviction_mu_` | +| SSD via SPDK proxy (`SpdkSsdTier` in the daemon) | `SpdkSsdTier::Evict(key)`, driven by `PeerSsdManager` over the proxy | `EvictLRU` and the evict-on-allocation-failure retry: a full or fragmented device now fails the write. The daemon's only client is `PeerSsdManager`, which evicts through the pool. | + +`MediumBackend::EnableLocalEviction` is removed. In its place the medium offers +**ordering, not eviction**: + +```cpp +// Start keeping the order EvictionCandidates() reads. Off by default: a +// master-only node never asks for candidates. +virtual void TrackEvictionOrder() {} +// Up to max_count keys this medium could free right now, coldest first, +// skipping keys under a read lease, a pin, or an in-flight read. Frees nothing; +// Evict() re-checks every key. +virtual std::vector EvictionCandidates(size_t max_count) { return {}; } +``` + +`PageBackend` keeps its LRU (touched by commit and resolve) behind +`TrackEvictionOrder`; `PeerSsdManager` already keeps one unconditionally. + +**Why the order comes from the medium, not the pool.** `PeerPool` has an LRU of +its own, `access_order_`, but maintains it only for `watermark` offload, and +turning it on disables `BatchResolve`'s single-backend fast path — which is the +masterless 100%-hit restore path. The medium already sees every read, including +that fast path. + +--- + +## 6. Requirement 3 — masterless watermark eviction through the same call + +```cpp +// PeerPool +bool EnableLocalEviction(uint32_t backend_id, double high_watermark, double low_watermark); +``` + +The first call starts one eviction worker thread. The worker: + +- **Wakes** when a commit lands, when an allocation answers NO_SPACE, and when a + transition finishes; it also polls every 100 ms, and leaves at least 1 ms + between passes so a stream of commits cannot keep it spinning. +- **Orders** backends lowest tier first (SSD, then DRAM, then HBM), so a + demotion from an upper tier finds room downstream. +- For each backend at or above its high watermark: repeat + `EvictionCandidates(64)` → `PeerPool::Evict({key, backend_id}…, kReclaim)` — + **the same call the master's request makes** — until usage is at or below the + low watermark, the medium has no candidates, or a batch frees nothing (every + candidate was mid-transition or became busy). The next wake tries again. + +Batches of 64 keep each `Evict` call short, so no single call holds the pool +lock for a whole drain. + +**Which backends evict locally.** The policy is unchanged; only the mechanism moved. + +| Medium | No master | With a master | +|---|---|---| +| DRAM / HBM | `PoolClientConfig::local_evict_{high,low}_watermark` (from `UMBP_DRAM_{HIGH,LOW}_WM`) | none — the master decides | +| SSD | the SSD config's own watermarks (`UMBP_SSD_{HIGH,LOW}_WM`) | the same, alongside the master, as before | + +SSD keeps local eviction with a master because the master's loop is periodic +and SSD previously reclaimed right after each write; removing that would let SSD +fill between master rounds. Both paths emit REMOVE events, so the master's index +stays correct. + +--- + +## 7. Behavior changes + +| Situation | Before | After | +|---|---|---| +| Master evicts a key held in DRAM and SSD (copy promotion), DRAM over budget, DRAM tier not `on_evict` | DRAM and SSD copies deleted | DRAM copy demoted (SSD already has it, so only DRAM is freed) | +| Same key, SSD over budget | DRAM copy freed (wrong medium), or both copies deleted | only the SSD copy deleted | +| Master evicts from a `watermark`-trigger tier with an offload target | dropped from every backend | demoted | +| Request from an old master (no tiers) | demote-if-`on_evict` else delete everywhere | free the placement's copy: demote if its tier has a downstream, else drop that copy | +| No master, tier has an offload target | local round deletes cold keys before offload sees the tier full | local eviction demotes them | +| No master, DRAM fills | evicted synchronously on the committing thread | evicted by the background worker; a burst faster than the worker can see NO_SPACE until it catches up | +| SSD write fails for lack of space | evict a round, retry once | the write fails; the worker reclaims SSD at its watermark | +| SPDK daemon allocation fails | daemon silently evicts its own oldest keys | the write fails | + +A failed put in a KV cache becomes a later miss, not an error; the master path +already accepts NO_SPACE between its rounds. The headroom above the high +watermark is what absorbs a burst while the worker runs. + +--- + +## 8. Metrics + +- `InstrumentedBackend`'s eviction series (`op="evict"`, `status=ok|miss`, and + bytes) now see **every** eviction, local ones included, because every eviction + is a call through the interface. +- `PeerPool` adds the counters the backend cannot see: + `mori_umbp_client_local_evict_total{event=round|key|no_candidate|stalled}` — + passes that found a backend over its high watermark, keys freed, passes where + the medium offered no candidate, and passes where a batch freed nothing. +- Removed: the SSD medium's `event="eviction_round"`, which counted the deleted + inline round. + +--- + +## 9. Tests + +- **Master strategy:** victims carry the tier they were charged to; a key over + budget in two tiers appears once per tier. +- **Master → pool chain:** a dispatched (key, tier) frees only that medium. +- **Peer service:** `EvictKey` with `tiers` scopes each key; without, falls back + to unscoped. +- **`PeerPool::Evict`:** tier scope, backend scope and unscoped; a copy-promoted + key keeps the copy that was not named; a `watermark` tier demotes; demotion + falls back to a drop when the target is full; mid-transition keys are left + alone. +- **Local eviction worker:** a masterless page backend filled past its high + watermark drains to its low watermark without any put doing the work; leased + keys survive; a tier with an offload target demotes instead of dropping; an + SSD backend drains through the same worker. +- **Media:** `PageBackend` and `PeerSsdManager` no longer evict on their own; + `EvictionCandidates` is coldest-first and skips busy keys. +- The full `tests/cpp/umbp` suite. + +--- + +## 10. Not done here + +- **A DRAM backstop in master mode.** DRAM is still master-only when a master + exists. The worker supports it; turning it on is a config question + (an emergency watermark well above the master's) left for a follow-up. +- **One source of watermarks.** There are still four: the master's + `EvictionConfig`, `local_evict_*` for paged media, the SSD config, and each + logical tier's watermarks in the policy file. +- **The SPDK change is not compiled by the default dev image**, which has no + SPDK; it needs a build with SPDK enabled before it ships. From 4254bd052a344274617984ff7912446bf3e28eb9 Mon Sep 17 00:00:00 2001 From: TianDi101 Date: Sat, 10 Oct 2026 10:37:36 +0000 Subject: [PATCH 02/19] feat(umbp): evict per tier -- the master names the medium, the pool frees that copy The master picks victims per (node, tier) but shipped only keys, so the peer freed by placement: on a tier without on_evict offload it deleted the key from every backend, and with copy-mode promotion (key in DRAM and SSD) SSD pressure freed the DRAM copy and left SSD as full as before. - EvictionVictim{key, tier}: SelectVictims and EvictKeyDispatcher carry the tier each victim was charged to. - EvictKeyRequest gains `repeated TierType tiers = 2`, parallel to keys. An older peer ignores it; an older master sends none and the peer frees each key from the medium that currently holds it. - PeerPool::Evict takes PoolEvictRequest{key, tier, backend_id} and frees exactly one copy: demote it when it is the placement and its tier has a downstream (any trigger -- on_evict vs watermark now only says whether the pool also offloads proactively), otherwise drop that copy only. Drops run outside the pool lock; lifecycle_mutex_ keeps ClearLocal out. - A transition that lands on a parked (draining) copy promotes it to the durable one, so a later eviction cannot delete it as still-draining. Co-Authored-By: Claude Opus 5.5 --- .../distributed/master/evict_strategy.cpp | 11 +- .../distributed/master/eviction_manager.cpp | 32 ++- src/umbp/distributed/master/master_server.cpp | 13 +- src/umbp/distributed/peer/peer_service.cpp | 29 ++- src/umbp/distributed/pool/peer_pool.cpp | 212 +++++++++++++----- src/umbp/distributed/proto/umbp_peer.proto | 5 + .../umbp/distributed/master/evict_strategy.h | 18 +- .../distributed/master/eviction_manager.h | 6 +- .../include/umbp/distributed/pool/peer_pool.h | 39 +++- src/umbp/include/umbp/distributed/types.h | 13 ++ .../test_master_evict_strategy.cpp | 49 +++- .../test_master_eviction_chain.cpp | 104 ++++++++- tests/cpp/umbp/distributed/test_peer_pool.cpp | 157 +++++++++++++ .../test_peer_service_dispatch.cpp | 51 ++++- 14 files changed, 619 insertions(+), 120 deletions(-) diff --git a/src/umbp/distributed/master/evict_strategy.cpp b/src/umbp/distributed/master/evict_strategy.cpp index bd62987f5..424ce0bbb 100644 --- a/src/umbp/distributed/master/evict_strategy.cpp +++ b/src/umbp/distributed/master/evict_strategy.cpp @@ -26,7 +26,7 @@ namespace mori::umbp { -std::unordered_map> LruMasterEvictStrategy::SelectVictims( +std::unordered_map> LruMasterEvictStrategy::SelectVictims( std::vector candidates, std::unordered_map> bytes_to_free) { // Oldest-access first (LRU); no tiebreak (peers don't ship depth in KvEvent). @@ -36,16 +36,17 @@ std::unordered_map> LruMasterEvictStrategy }); // Walk oldest-first, taking a candidate while its (node, tier) has budget - // left. Group by node so each peer gets a single EvictKey keys[]. - std::unordered_map> per_node_keys; + // left. Group by node so each peer gets a single EvictKey. The victim keeps + // the tier it was charged to: that is the medium the peer must free. + std::unordered_map> per_node_victims; for (const auto& c : candidates) { auto& tier_budget = bytes_to_free[c.location.node_id]; auto it = tier_budget.find(c.location.tier); if (it == tier_budget.end() || it->second <= 0) continue; - per_node_keys[c.location.node_id].push_back(c.key); + per_node_victims[c.location.node_id].push_back(EvictionVictim{c.key, c.location.tier}); it->second -= static_cast(c.size); } - return per_node_keys; + return per_node_victims; } } // namespace mori::umbp diff --git a/src/umbp/distributed/master/eviction_manager.cpp b/src/umbp/distributed/master/eviction_manager.cpp index 7e732528d..2177a3332 100644 --- a/src/umbp/distributed/master/eviction_manager.cpp +++ b/src/umbp/distributed/master/eviction_manager.cpp @@ -139,7 +139,9 @@ void EvictionManager::RunOnce() { // Flatten the candidates and translate the per-(node, tier) byte budget into // the strategy's shape, then let the strategy pick victims (default LRU), - // grouped by node so each peer gets a single EvictKey keys[]. + // grouped by node so each peer gets a single EvictKey. Each victim keeps + // the tier it was charged to, so the peer frees exactly the copy that is + // over budget. std::vector candidates; for (auto& [bucket, rows] : candidates_by_bucket) { for (auto& c : rows) candidates.push_back(std::move(c)); @@ -149,15 +151,16 @@ void EvictionManager::RunOnce() { strategy_budget[ntk.node_id][ntk.tier] = static_cast(bytes); } - auto per_node_keys = strategy_->SelectVictims(std::move(candidates), std::move(strategy_budget)); + auto per_node_victims = + strategy_->SelectVictims(std::move(candidates), std::move(strategy_budget)); size_t selected = 0; - for (const auto& [node_id, keys] : per_node_keys) selected += keys.size(); + for (const auto& [node_id, victims] : per_node_victims) selected += victims.size(); if (selected == 0) return; MORI_UMBP_INFO("[EvictionManager] Selected {} victims across {} nodes", selected, - per_node_keys.size()); + per_node_victims.size()); // Look up peer addresses once per dispatch round. ClientRegistry // owns the (node_id -> peer_address) mapping; we can't ship an @@ -165,16 +168,16 @@ void EvictionManager::RunOnce() { std::unordered_map node_to_peer; for (const auto& client : clients) node_to_peer[client.node_id] = client.peer_address; - for (auto& [node_id, keys] : per_node_keys) { + for (auto& [node_id, victims] : per_node_victims) { auto it = node_to_peer.find(node_id); if (it == node_to_peer.end() || it->second.empty()) { MORI_UMBP_WARN("[EvictionManager] no peer_address for node={} — skipping {} keys", node_id, - keys.size()); + victims.size()); continue; } if (dispatcher_ == nullptr) { MORI_UMBP_DEBUG("[EvictionManager] dispatcher unset; would EvictKey on node={} ({} keys)", - node_id, keys.size()); + node_id, victims.size()); continue; } // Chunked to the shared gRPC message limit. An eviction round frees a @@ -189,19 +192,26 @@ void EvictionManager::RunOnce() { // next heartbeat are what shrink the index), and a chunk that fails is // retried next round. So N chunks reach the same state as one message // would have, and a partial round still makes progress. + // + // Sized on the keys alone: the parallel tier field is a packed enum of one + // byte per key, inside the per-item overhead GrpcMaxItemsPerBatch already + // reserves. + std::vector keys; + keys.reserve(victims.size()); + for (const auto& victim : victims) keys.push_back(victim.key); size_t sent = 0; size_t chunks = 0; - while (sent < keys.size()) { + while (sent < victims.size()) { const size_t take = GrpcMaxItemsPerBatch(keys, sent); - std::vector chunk(std::make_move_iterator(keys.begin() + sent), - std::make_move_iterator(keys.begin() + sent + take)); + std::vector chunk(std::make_move_iterator(victims.begin() + sent), + std::make_move_iterator(victims.begin() + sent + take)); sent += take; ++chunks; dispatcher_->DispatchEvictKey(node_id, it->second, std::move(chunk)); } if (chunks > 1) { MORI_UMBP_DEBUG("[EvictionManager] node={} {} keys dispatched in {} chunks", node_id, - keys.size(), chunks); + victims.size(), chunks); } } } diff --git a/src/umbp/distributed/master/master_server.cpp b/src/umbp/distributed/master/master_server.cpp index 89b53e9a9..77e40a613 100644 --- a/src/umbp/distributed/master/master_server.cpp +++ b/src/umbp/distributed/master/master_server.cpp @@ -128,14 +128,21 @@ int EvictKeyDeadlineMs() { class MasterPeerStubPool : public EvictKeyDispatcher { public: void DispatchEvictKey(const std::string& node_id, const std::string& peer_address, - std::vector keys) override { - if (keys.empty() || peer_address.empty()) return; + std::vector victims) override { + if (victims.empty() || peer_address.empty()) return; auto stub = GetOrCreateStub(node_id, peer_address); if (stub == nullptr) return; + // keys and tiers are parallel, so a peer that predates `tiers` still reads + // a complete key list. ::umbp::EvictKeyRequest req; - for (auto& k : keys) req.add_keys(std::move(k)); + req.mutable_keys()->Reserve(static_cast(victims.size())); + req.mutable_tiers()->Reserve(static_cast(victims.size())); + for (auto& victim : victims) { + req.add_keys(std::move(victim.key)); + req.add_tiers(static_cast<::umbp::TierType>(victim.tier)); + } ::umbp::EvictKeyResponse resp; grpc::ClientContext ctx; ctx.set_deadline(std::chrono::system_clock::now() + diff --git a/src/umbp/distributed/peer/peer_service.cpp b/src/umbp/distributed/peer/peer_service.cpp index 7e68b9813..78f766145 100644 --- a/src/umbp/distributed/peer/peer_service.cpp +++ b/src/umbp/distributed/peer/peer_service.cpp @@ -265,18 +265,25 @@ class PeerServiceServer::UMBPPeerServiceImpl final : public ::umbp::UMBPPeer::Se grpc::Status EvictKey(grpc::ServerContext* /*ctx*/, const ::umbp::EvictKeyRequest* request, ::umbp::EvictKeyResponse* response) override { - // Eviction carries no tier either. Master drives this to reclaim capacity - // it measured per (node, tier), so the pool is free to demote where a tier - // configures on_evict offload; otherwise a key mirrored across media is - // dropped from ALL of them. Either way the freed bytes are summed per key - // — master sizes its next eviction round off this total. - std::vector keys(request->keys().begin(), request->keys().end()); - if (keys.empty()) return grpc::Status::OK; - auto evicted = - pool_ == nullptr ? std::vector{} : pool_->Evict(keys, PoolEvictMode::kReclaim); - for (size_t i = 0; i < keys.size(); ++i) { + // Master reclaims capacity it measured per (node, tier), and names that + // tier per key: the pool frees exactly that medium's copy (demoting it + // where its tier has a downstream) and leaves any other copy alone. An + // older master sends no tiers; each key is then freed from the medium that + // currently holds it. bytes_freed is informational -- the master sizes its + // next round off the heartbeat's capacity, not off this reply. + const int n = request->keys_size(); + if (n == 0) return grpc::Status::OK; + const bool scoped = request->tiers_size() == n; + std::vector requests(static_cast(n)); + for (int i = 0; i < n; ++i) { + requests[i].key = request->keys(i); + if (scoped) requests[i].tier = static_cast(request->tiers(i)); + } + auto evicted = pool_ == nullptr ? std::vector{} + : pool_->Evict(requests, PoolEvictMode::kReclaim); + for (size_t i = 0; i < requests.size(); ++i) { auto* entry = response->add_evicted(); - entry->set_key(keys[i]); + entry->set_key(requests[i].key); entry->set_bytes_freed(i < evicted.size() ? evicted[i].bytes_freed : 0); } return grpc::Status::OK; diff --git a/src/umbp/distributed/pool/peer_pool.cpp b/src/umbp/distributed/pool/peer_pool.cpp index 50d6a5048..6418a178a 100644 --- a/src/umbp/distributed/pool/peer_pool.cpp +++ b/src/umbp/distributed/pool/peer_pool.cpp @@ -25,6 +25,7 @@ #include "umbp/distributed/pool/peer_pool.h" #include +#include #include #include #include @@ -251,6 +252,14 @@ void PeerPool::FinishTransitionLocked(const TransitionJob& job, if (draining->second.empty()) draining_sources_.erase(draining); } } + // The target may be a copy an earlier move left parked (it answered + // ALREADY_EXISTS, so nothing was copied). It is the durable copy now, and a + // later eviction must not delete it as though it were still draining. + auto draining = draining_sources_.find(job.key); + if (draining != draining_sources_.end()) { + draining->second.erase(result.target_backend_id); + if (draining->second.empty()) draining_sources_.erase(draining); + } } TierTransitionResult PeerPool::RunTransition(std::unique_lock& lock, @@ -835,87 +844,170 @@ std::vector PeerPool::BatchContains(const std::vector& keys) return out; } +uint32_t PeerPool::EvictSourceLocked(const PoolEvictRequest& request) const { + constexpr uint32_t kNone = BackendRegistry::kMaxBackends; + // A copy qualifies when its backend holds the key and, for a tier-scoped + // request, is that medium. + const auto holds = [&](uint32_t backend_id) { + if (backend_id >= kNone) return false; + auto* backend = backends_->Get(backend_id); + return backend != nullptr && + (request.tier == TierType::UNKNOWN || backend->Tier() == request.tier) && + backend->Contains(request.key); + }; + if (request.backend_id < kNone) return holds(request.backend_id) ? request.backend_id : kNone; + // The placement first: it is the copy reads are served from, and the only + // one a demotion can start from. + auto placement = placements_.find(request.key); + if (placement != placements_.end() && holds(placement->second)) return placement->second; + if (request.tier == TierType::UNKNOWN) return FindOwnerLocked(request.key); + for (auto* backend : backends_->All()) { + const uint32_t backend_id = backends_->BackendId(backend); + if (holds(backend_id)) return backend_id; + } + return kNone; +} + +void PeerPool::RepairAfterDropLocked(const std::string& key) { + auto draining = draining_sources_.find(key); + if (draining != draining_sources_.end()) { + for (auto it = draining->second.begin(); it != draining->second.end();) { + auto* source = backends_->Get(*it); + it = source == nullptr || !source->Contains(key) ? draining->second.erase(it) : std::next(it); + } + if (draining->second.empty()) draining_sources_.erase(draining); + } + auto placement = placements_.find(key); + if (placement != placements_.end()) { + auto* backend = backends_->Get(placement->second); + if (backend != nullptr && backend->Contains(key)) return; + } + const uint32_t owner = FindOwnerLocked(key); + if (owner != BackendRegistry::kMaxBackends) { + placements_[key] = owner; + return; + } + placements_.erase(key); + ForgetAccessLocked(key); + draining_sources_.erase(key); +} + std::vector PeerPool::Evict(const std::vector& keys, PoolEvictMode mode) { - std::unique_lock operation_lock(operation_mutex_); + std::vector requests(keys.size()); + for (size_t i = 0; i < keys.size(); ++i) requests[i].key = keys[i]; + return Evict(requests, mode); +} + +std::vector PeerPool::Evict(const std::vector& requests, + PoolEvictMode mode) { + // Shared for the whole call: the drops below run without operation_mutex_, + // and ClearLocal must not wipe the backends between a drop being decided and + // being carried out. + std::shared_lock lifecycle_lock(lifecycle_mutex_); std::vector out; - out.reserve(keys.size()); - for (const auto& key : keys) out.push_back(EvictResult{key, 0}); - if (backends_ == nullptr || keys.empty()) return out; + out.reserve(requests.size()); + for (const auto& request : requests) out.push_back(EvictResult{request.key, 0}); + if (backends_ == nullptr || requests.empty()) return out; - std::vector handled(keys.size(), false); - for (size_t i = 0; i < keys.size(); ++i) { + // A copy to free by deleting it: (request index, backend). Decided under + // the pool lock, carried out without it. + std::vector> drops; + // A copy to free by moving it down a tier: (request index, source backend). + std::vector> demotions; + + std::unique_lock operation_lock(operation_mutex_); + for (size_t i = 0; i < requests.size(); ++i) { + const auto& request = requests[i]; // A copy already owns this key. Freeing nothing makes the caller retry, // which is the same contract as a read lease delaying a source delete, and // it keeps a discard from racing a target commit that would resurrect it. - if (migrating_keys_.count(keys[i]) != 0) { - handled[i] = true; + if (migrating_keys_.count(request.key) != 0) continue; + // A discard has to leave no copy behind, so it ignores scope, skips + // demotion and drops the key everywhere. + if (mode == PoolEvictMode::kDiscard) { + for (auto* backend : backends_->All()) drops.emplace_back(i, backends_->BackendId(backend)); continue; } - // A discard has to leave no copy behind, so it skips demotion and the - // drain shortcut and goes straight to the delete pass below. - if (mode == PoolEvictMode::kDiscard) continue; - auto draining = draining_sources_.find(keys[i]); - if (draining != draining_sources_.end()) { - std::vector drained; - for (uint32_t source_id : draining->second) { - auto* source = backends_->Get(source_id); - if (source != nullptr) { - auto retried = source->Evict({keys[i]}); - if (!retried.empty()) out[i].bytes_freed += retried.front().bytes_freed; - } - if (source == nullptr || !source->Contains(keys[i])) { - drained.push_back(source_id); - } - } - for (uint32_t source_id : drained) draining->second.erase(source_id); - if (draining->second.empty()) draining_sources_.erase(draining); - handled[i] = true; + // A source left behind by an offload whose delete hit a read lease: the + // durable copy already lives downstream, so this copy is only ever deleted, + // never moved again. A request with no scope means "finish the drain", as + // it always has -- freeing the durable copy instead would undo the offload. + auto draining = draining_sources_.find(request.key); + const bool unscoped = + request.tier == TierType::UNKNOWN && request.backend_id >= BackendRegistry::kMaxBackends; + if (draining != draining_sources_.end() && unscoped) { + for (uint32_t source_id : draining->second) drops.emplace_back(i, source_id); continue; } - - auto placement = placements_.find(keys[i]); - const uint32_t source_id = - placement != placements_.end() ? placement->second : FindOwnerLocked(keys[i]); - if (tier_graph_ == nullptr) continue; - const auto tier_index = tier_graph_->TierIndexForBackendId(source_id); - if (!tier_index.has_value()) continue; - const auto& tier = tier_graph_->NodeAt(*tier_index); - if (tier.trigger != PoolOffloadTrigger::kOnEvict || tier.offload_to.empty()) { + const uint32_t source_id = EvictSourceLocked(request); + if (source_id == BackendRegistry::kMaxBackends) continue; + if (draining != draining_sources_.end() && draining->second.count(source_id) != 0) { + drops.emplace_back(i, source_id); continue; } - auto migrated = RunTransition(operation_lock, - {TierTransitionKind::kOffload, keys[i], source_id, *tier_index}); - handled[i] = migrated.success; - out[i].bytes_freed = migrated.bytes_freed; - } - std::vector delete_keys; - std::vector delete_indices; - for (size_t i = 0; i < keys.size(); ++i) { - if (!handled[i]) { - delete_keys.push_back(keys[i]); - delete_indices.push_back(i); + // Demote when this copy is the key's placement (the only copy a transition + // can start from) and its tier has somewhere to go. The trigger does not + // matter here: on_evict vs watermark only says whether the pool ALSO moves + // keys proactively, and either way dropping data a lower tier has room for + // would be the wrong answer to "free this tier". + bool demote = false; + if (tier_graph_ != nullptr) { + auto placement = placements_.find(request.key); + const auto tier_index = tier_graph_->TierIndexForBackendId(source_id); + demote = placement != placements_.end() && placement->second == source_id && + tier_index.has_value() && !tier_graph_->NodeAt(*tier_index).offload_to.empty(); + } + if (demote) { + demotions.emplace_back(i, source_id); + } else { + drops.emplace_back(i, source_id); } } - for (auto* backend : backends_->All()) { - auto results = backend->Evict(delete_keys); - for (size_t i = 0; i < results.size() && i < delete_indices.size(); ++i) { - out[delete_indices[i]].bytes_freed += results[i].bytes_freed; + + // One at a time: RunTransition releases the lock for the byte copy itself. + for (const auto& [index, source_id] : demotions) { + const auto tier_index = tier_graph_->TierIndexForBackendId(source_id); + if (!tier_index.has_value()) { + drops.emplace_back(index, source_id); + continue; } + auto migrated = RunTransition(operation_lock, {TierTransitionKind::kOffload, + requests[index].key, source_id, *tier_index}); + if (migrated.success) { + out[index].bytes_freed = migrated.bytes_freed; + continue; + } + // Nowhere downstream had room, or the copy moved meanwhile: free it by + // deleting it instead. The tier asked for bytes back either way. + drops.emplace_back(index, source_id); } + if (drops.empty()) return out; - for (size_t i = 0; i < keys.size(); ++i) { - if (handled[i]) continue; - const uint32_t owner = FindOwnerLocked(keys[i]); - if (owner != BackendRegistry::kMaxBackends) { - placements_[keys[i]] = owner; - } else { - placements_.erase(keys[i]); - ForgetAccessLocked(keys[i]); - draining_sources_.erase(keys[i]); + // The drops themselves run without the pool lock: deleting from SSD is + // device IO, and holding the lock across it would stall every allocate and + // commit on the peer. One Evict call per backend. + operation_lock.unlock(); + std::map> by_backend; + for (const auto& [index, backend_id] : drops) by_backend[backend_id].push_back(index); + for (const auto& [backend_id, indices] : by_backend) { + auto* backend = backends_->Get(backend_id); + if (backend == nullptr) continue; + std::vector keys; + keys.reserve(indices.size()); + for (size_t index : indices) keys.push_back(requests[index].key); + auto results = backend->Evict(keys); + for (size_t j = 0; j < indices.size() && j < results.size(); ++j) { + out[indices[j]].bytes_freed += results[j].bytes_freed; } } + operation_lock.lock(); + + std::unordered_set repaired; + for (const auto& [index, backend_id] : drops) { + if (repaired.insert(requests[index].key).second) RepairAfterDropLocked(requests[index].key); + } return out; } diff --git a/src/umbp/distributed/proto/umbp_peer.proto b/src/umbp/distributed/proto/umbp_peer.proto index 54680d9b3..d2d90a269 100644 --- a/src/umbp/distributed/proto/umbp_peer.proto +++ b/src/umbp/distributed/proto/umbp_peer.proto @@ -142,6 +142,11 @@ message EvictKeyEntry { } message EvictKeyRequest { repeated string keys = 1; + // tiers[i] is the medium keys[i] must be freed from -- the one whose budget + // the master charged it to. Honoured only when it has exactly as many + // entries as `keys`; otherwise (an older master) each key is freed from the + // medium that currently holds it. An older peer ignores this field. + repeated TierType tiers = 2; } message EvictKeyResponse { repeated EvictKeyEntry evicted = 1; diff --git a/src/umbp/include/umbp/distributed/master/evict_strategy.h b/src/umbp/include/umbp/distributed/master/evict_strategy.h index 4da9aaec9..105e8a3d1 100644 --- a/src/umbp/include/umbp/distributed/master/evict_strategy.h +++ b/src/umbp/include/umbp/distributed/master/evict_strategy.h @@ -27,23 +27,23 @@ #include #include -#include "umbp/distributed/types.h" // EvictionCandidate, TierType +#include "umbp/distributed/types.h" // EvictionCandidate, EvictionVictim, TierType namespace mori::umbp { -// Master-side victim selection for DRAM/HBM eviction. EvictionManager handles -// watermark detection, candidate gathering, and EvictKey dispatch; the strategy -// only ranks candidates and picks victims within the per-(node,tier) budget. -// (SSD eviction is peer-local; master never evicts SSD.) +// Master-side victim selection. EvictionManager handles watermark detection, +// candidate gathering, and EvictKey dispatch; the strategy only ranks +// candidates and picks victims within the per-(node,tier) budget. class MasterEvictStrategy { public: virtual ~MasterEvictStrategy() = default; // Pick victims from @p candidates within the per-(node,tier) byte budget // @p bytes_to_free. Both are by value so the impl may sort/decrement freely. - // Returns victims grouped by node_id (one keys[] per peer); empty groups - // omitted. - virtual std::unordered_map> SelectVictims( + // Returns victims grouped by node_id (one EvictKey per peer); empty groups + // omitted. Each victim carries the tier whose budget it was charged to, so + // the peer frees that medium's copy and no other. + virtual std::unordered_map> SelectVictims( std::vector candidates, std::unordered_map> bytes_to_free) = 0; }; @@ -51,7 +51,7 @@ class MasterEvictStrategy { // Default policy: pure LRU (evict oldest-access-first until each budget is met). class LruMasterEvictStrategy : public MasterEvictStrategy { public: - std::unordered_map> SelectVictims( + std::unordered_map> SelectVictims( std::vector candidates, std::unordered_map> bytes_to_free) override; }; diff --git a/src/umbp/include/umbp/distributed/master/eviction_manager.h b/src/umbp/include/umbp/distributed/master/eviction_manager.h index caa2737af..9b2f55002 100644 --- a/src/umbp/include/umbp/distributed/master/eviction_manager.h +++ b/src/umbp/include/umbp/distributed/master/eviction_manager.h @@ -30,6 +30,7 @@ #include #include "umbp/distributed/config.h" +#include "umbp/distributed/types.h" namespace mori::umbp { @@ -42,6 +43,9 @@ class MasterEvictStrategy; // REMOVE events arrive on the peer's next heartbeat and that's where // the index actually shrinks. // +// Each victim names the medium to free the key from: the peer frees that +// copy and leaves any other copy of the key alone. +// // The implementation lives in master_server.cpp (MasterPeerStubPool) // where the gRPC stub plumbing belongs. Tests can swap in a fake. class EvictKeyDispatcher { @@ -49,7 +53,7 @@ class EvictKeyDispatcher { virtual ~EvictKeyDispatcher() = default; virtual void DispatchEvictKey(const std::string& node_id, const std::string& peer_address, - std::vector keys) = 0; + std::vector victims) = 0; }; class EvictionManager { diff --git a/src/umbp/include/umbp/distributed/pool/peer_pool.h b/src/umbp/include/umbp/distributed/pool/peer_pool.h index c8e18ff74..22a52fed7 100644 --- a/src/umbp/include/umbp/distributed/pool/peer_pool.h +++ b/src/umbp/include/umbp/distributed/pool/peer_pool.h @@ -75,16 +75,31 @@ struct PoolResolvedEntry { // differently, so the caller has to say which one it means instead of leaving // it to the tier configuration. enum class PoolEvictMode { - // Free bytes on the tier that holds the key. Where a tier configures - // on_evict offload the key is demoted, not dropped: the master budgets per - // (node, tier), so moving the bytes downstream does relieve the pressure it - // measured, and the data stays reachable. + // Free the bytes of ONE copy of the key: the one the request names (see + // PoolEvictRequest). Where that copy's logical tier has an offload target the + // key is demoted, not dropped -- the pressure being relieved is on the named + // medium, so moving the bytes downstream relieves it and the data stays + // reachable. Any other copy of the key is left alone. kReclaim, // Drop the key from every backend on the peer regardless of tier - // configuration, for when the value itself must stop existing. + // configuration or scope, for when the value itself must stop existing. kDiscard, }; +// One key to evict, and which of its copies. A key can live on several media +// at once (a copy-mode promotion leaves the lower copy behind), so "evict this +// key" is ambiguous: the request says which medium is under pressure. +struct PoolEvictRequest { + std::string key; + // The medium to free the key from -- the tier a master charged it to. + // UNKNOWN means the medium that currently holds the key's placement, which + // is how a request with no tier (an older master) is read. + TierType tier = TierType::UNKNOWN; + // One specific backend instead of a medium. Local watermark eviction knows + // exactly which backend is over its watermark. Takes precedence over tier. + uint32_t backend_id = BackendRegistry::kMaxBackends; +}; + // The implicit peer-local default Pool. It owns logical key placement and // delegates backend choice to PoolPolicy; BackendRegistry continues to own the // physical storage instances. This keeps the first Pool layer behaviorally @@ -107,6 +122,13 @@ class PeerPool { // check discards. Takes no read lease and does not touch recency: asking // whether a key is here is not a read of it. std::vector BatchContains(const std::vector& keys); + // The one routing entry point for eviction. Every policy calls this -- the + // master's EvictKey RPC and this pool's own local watermark eviction alike -- + // and every byte it frees is freed by the medium's MediumBackend::Evict, + // directly or as the last step of a demotion. One result per request, in + // request order; bytes_freed is what the named copy released. + std::vector Evict(const std::vector& requests, PoolEvictMode mode); + // Every key unscoped: freed from the medium that currently holds it. std::vector Evict(const std::vector& keys, PoolEvictMode mode); void ClearLocal(); std::vector DrainPendingEvents(); @@ -156,6 +178,13 @@ class PeerPool { // is process-local and can be stale or empty (after a restart, or once // another path evicted the key), so the backends are the authority. uint32_t FindOwnerLocked(const std::string& key) const; + // The backend holding the copy an eviction request names, or kMaxBackends + // when there is no such copy (see PoolEvictRequest for how scope resolves). + uint32_t EvictSourceLocked(const PoolEvictRequest& request) const; + // After copies of `key` were dropped: point the placement at a copy that is + // still there, or forget the key if none is. Re-checks the backends rather + // than trusting what was dropped, because drops run without the pool lock. + void RepairAfterDropLocked(const std::string& key); // Queues at most `max_count` offload candidates for one tier, oldest first, // and reports how many it queued. Bounded on purpose: a peer can hold // millions of placements, so neither the scan nor the queue may be diff --git a/src/umbp/include/umbp/distributed/types.h b/src/umbp/include/umbp/distributed/types.h index 97ddf7227..9b6f532d2 100644 --- a/src/umbp/include/umbp/distributed/types.h +++ b/src/umbp/include/umbp/distributed/types.h @@ -160,6 +160,19 @@ struct EvictionCandidate { uint64_t size; }; +// One master eviction decision: free `key` from the `tier` medium of a node. +// The tier is the one whose budget the victim was charged to. A key over +// budget in two media of the same node is two victims, one per medium; without +// the tier the peer could only guess which copy the master meant. +struct EvictionVictim { + std::string key; + TierType tier = TierType::UNKNOWN; + + bool operator==(const EvictionVictim& other) const { + return key == other.key && tier == other.tier; + } +}; + // Ordering hint for IMasterMetadataStore::EnumerateEvictionCandidates. This is // a performance affordance, NOT eviction policy: it only tells the store what // order to return rows in so a backend with an index (e.g. a Redis ZSET keyed diff --git a/tests/cpp/umbp/distributed/test_master_evict_strategy.cpp b/tests/cpp/umbp/distributed/test_master_evict_strategy.cpp index a5bc7ac45..55cf1c0ee 100644 --- a/tests/cpp/umbp/distributed/test_master_evict_strategy.cpp +++ b/tests/cpp/umbp/distributed/test_master_evict_strategy.cpp @@ -25,6 +25,7 @@ // byte budget. #include +#include #include #include #include @@ -67,8 +68,48 @@ TEST(LruMasterEvictStrategy, PicksOldestFirstUntilBudgetMet) { ASSERT_EQ(victims.count("n1"), 1u); // Oldest-first: a, then b; c (newest) is spared once the 150-byte budget met. ASSERT_EQ(victims["n1"].size(), 2u); - EXPECT_EQ(victims["n1"][0], "a"); - EXPECT_EQ(victims["n1"][1], "b"); + EXPECT_EQ(victims["n1"][0].key, "a"); + EXPECT_EQ(victims["n1"][1].key, "b"); +} + +TEST(LruMasterEvictStrategy, VictimCarriesTheTierItWasChargedTo) { + LruMasterEvictStrategy strategy; + auto now = Clock::now(); + std::vector candidates = { + MakeCandidate("k", "n1", TierType::SSD, 100, now), + }; + std::unordered_map> budget; + budget["n1"][TierType::SSD] = 100; + + auto victims = strategy.SelectVictims(candidates, budget); + ASSERT_EQ(victims["n1"].size(), 1u); + // The peer frees exactly this medium's copy, so the tier must survive + // selection: without it an SSD-pressure victim could free the DRAM copy. + EXPECT_EQ(victims["n1"][0], (EvictionVictim{"k", TierType::SSD})); +} + +TEST(LruMasterEvictStrategy, KeyOverBudgetInTwoTiersIsOneVictimPerTier) { + // A copy-mode promotion leaves the same key in DRAM and SSD. When both + // media are over budget the master charges each copy to its own tier, and + // the peer must be told to free both -- one victim per (key, tier). + LruMasterEvictStrategy strategy; + auto now = Clock::now(); + std::vector candidates = { + MakeCandidate("k", "n1", TierType::DRAM, 100, now - std::chrono::seconds(1)), + MakeCandidate("k", "n1", TierType::SSD, 100, now - std::chrono::seconds(1)), + }; + std::unordered_map> budget; + budget["n1"][TierType::DRAM] = 100; + budget["n1"][TierType::SSD] = 100; + + auto victims = strategy.SelectVictims(candidates, budget); + ASSERT_EQ(victims["n1"].size(), 2u); + EXPECT_NE( + std::find(victims["n1"].begin(), victims["n1"].end(), EvictionVictim{"k", TierType::DRAM}), + victims["n1"].end()); + EXPECT_NE( + std::find(victims["n1"].begin(), victims["n1"].end(), EvictionVictim{"k", TierType::SSD}), + victims["n1"].end()); } TEST(LruMasterEvictStrategy, HonoursPerNodeTierBudgetIndependently) { @@ -85,9 +126,9 @@ TEST(LruMasterEvictStrategy, HonoursPerNodeTierBudgetIndependently) { auto victims = strategy.SelectVictims(candidates, budget); ASSERT_EQ(victims["n1"].size(), 1u); - EXPECT_EQ(victims["n1"][0], "n1-old"); + EXPECT_EQ(victims["n1"][0].key, "n1-old"); ASSERT_EQ(victims["n2"].size(), 1u); - EXPECT_EQ(victims["n2"][0], "n2-old"); + EXPECT_EQ(victims["n2"][0].key, "n2-old"); } TEST(LruMasterEvictStrategy, SkipsTiersWithNoBudget) { diff --git a/tests/cpp/umbp/distributed/test_master_eviction_chain.cpp b/tests/cpp/umbp/distributed/test_master_eviction_chain.cpp index ca5e0c70d..a85bd7df9 100644 --- a/tests/cpp/umbp/distributed/test_master_eviction_chain.cpp +++ b/tests/cpp/umbp/distributed/test_master_eviction_chain.cpp @@ -114,12 +114,18 @@ class PoolDispatcher final : public EvictKeyDispatcher { explicit PoolDispatcher(PeerPool* pool) : pool_(pool) {} void DispatchEvictKey(const std::string& node_id, const std::string& /*peer_address*/, - std::vector keys) override { - auto results = pool_->Evict(keys, PoolEvictMode::kReclaim); + std::vector victims) override { + std::vector requests(victims.size()); + for (size_t i = 0; i < victims.size(); ++i) { + requests[i].key = victims[i].key; + requests[i].tier = victims[i].tier; + } + auto results = pool_->Evict(requests, PoolEvictMode::kReclaim); std::lock_guard lock(mutex_); ++rounds_; - for (size_t i = 0; i < keys.size(); ++i) { - dispatched_.push_back(keys[i]); + for (size_t i = 0; i < victims.size(); ++i) { + dispatched_.push_back(victims[i].key); + victims_.push_back(victims[i]); if (i < results.size()) freed_bytes_ += results[i].bytes_freed; } node_ids_.push_back(node_id); @@ -129,6 +135,10 @@ class PoolDispatcher final : public EvictKeyDispatcher { std::lock_guard lock(mutex_); return dispatched_; } + std::vector Victims() const { + std::lock_guard lock(mutex_); + return victims_; + } size_t Rounds() const { std::lock_guard lock(mutex_); return rounds_; @@ -146,6 +156,7 @@ class PoolDispatcher final : public EvictKeyDispatcher { PeerPool* pool_; mutable std::mutex mutex_; std::vector dispatched_; + std::vector victims_; std::vector node_ids_; size_t rounds_ = 0; uint64_t freed_bytes_ = 0; @@ -351,8 +362,18 @@ TEST(MasterEvictionChain, MasterRoundReclaimsTheParkedSourceOfAMovePromotion) { EXPECT_TRUE(registry.Get("cold")->Contains("moved")) << "source was deleted during the promotion, so there is nothing to reclaim"; + // The parked source is what occupies the cold medium, so that is the bucket + // the master sees over its watermark; the hot medium is comfortably under. + // The victim therefore names SSD -- the copy to free -- and not the promoted + // DRAM copy, which a tier-less request used to reach first via the placement. InMemoryMasterMetadataStore store; - PublishNode(&store, {"moved"}, /*total_bytes=*/4 * kPageSize, /*used_bytes=*/4 * kPageSize); + const auto now = Clock::now(); + RegisterNode(&store, now); + std::map caps; + caps[TierType::SSD] = TierCapacity{4 * kPageSize, 0, kPageSize}; + caps[TierType::DRAM] = TierCapacity{4 * kPageSize, 3 * kPageSize, kPageSize}; + ReportHeartbeat(&store, /*seq=*/1, now, caps, TierType::SSD, "cold", {"moved"}, + /*full_sync=*/true); PoolDispatcher dispatcher(&pool); EvictionManager manager(store, OneSecondRounds(), &dispatcher); @@ -362,11 +383,84 @@ TEST(MasterEvictionChain, MasterRoundReclaimsTheParkedSourceOfAMovePromotion) { manager.Stop(); EXPECT_TRUE(reclaimed) << "the parked source outlived a master eviction round that named it"; + ASSERT_FALSE(dispatcher.Victims().empty()); + EXPECT_EQ(dispatcher.Victims().front(), (EvictionVictim{"moved", TierType::SSD})); // The promoted copy is the only one left, so the key is still servable. EXPECT_TRUE(registry.Get("hot")->Contains("moved")) << "the evict drained the promoted target instead of the parked source"; } +// A copy-mode promotion leaves a key in BOTH media, and the master charges +// each copy to its own tier. When only SSD is over budget the master must free +// the SSD copy -- and only that one. A tier-less request could not say so: the +// pool went by the placement, which a promotion points at the DRAM copy, so SSD +// pressure demoted the hot copy (cold already held it), freed DRAM instead, and +// left SSD exactly as full as before. +TEST(MasterEvictionChain, SsdPressureFreesOnlyTheSsdCopyOfAPromotedKey) { + constexpr auto kShortLease = std::chrono::milliseconds(50); + LocalCopyEngine engine; + BackendRegistry registry; + PageBackend::OwnershipConfig hot_ownership; + hot_ownership.buffer_sizes = {4 * kPageSize}; + auto hot = MakePageBackend(TierType::DRAM, kPageSize, hot_ownership, kNoExpiry, kShortLease); + ASSERT_TRUE(hot->Init(&engine)); + ASSERT_TRUE(registry.Register("hot", std::move(hot))); + PageBackend::OwnershipConfig cold_ownership; + cold_ownership.buffer_sizes = {4 * kPageSize}; + auto cold = MakePageBackend(TierType::SSD, kPageSize, cold_ownership, kNoExpiry, kShortLease); + ASSERT_TRUE(cold->Init(&engine)); + ASSERT_TRUE(registry.Register("cold", std::move(cold))); + + auto tiers = HotColdOnEvict(); + tiers.back().promote_trigger = PoolPromoteTrigger::kOnRead; + tiers.back().promote_mode = PoolTransitionMode::kCopy; + auto compiled = LogicalTierGraph::Compile(tiers, registry); + ASSERT_TRUE(compiled.ok()) << compiled.error; + PeerPool pool(®istry, MakeTieredPlacementPolicy(compiled.graph), &engine); + + PoolPlacementRequest request; + request.key = "dual"; + request.size = kPageSize; + request.tier = TierType::DRAM; + request.logical_tier = "cold"; + auto allocation = pool.BatchAllocate({request}).front(); + ASSERT_EQ(allocation.allocation.outcome, AllocateOutcome::kSuccessAllocated); + ASSERT_TRUE(pool.BatchCommit({PoolCommitRequest{ + {allocation.backend_id, allocation.allocation.slot_id}, "dual"}}) + .front() + .commit.success); + ASSERT_TRUE(pool.BatchResolve({"dual"}, false).front().resolved.found); + ASSERT_TRUE( + WaitFor([&] { return registry.Get("hot")->Contains("dual"); }, std::chrono::seconds(2))) + << "promotion never reached the hot tier"; + ASSERT_TRUE(registry.Get("cold")->Contains("dual")) << "copy promotion removed the source"; + // Let the promotion's read lease on the cold copy lapse. + std::this_thread::sleep_for(4 * kShortLease); + + InMemoryMasterMetadataStore store; + const auto now = Clock::now(); + RegisterNode(&store, now); + std::map caps; + caps[TierType::SSD] = TierCapacity{4 * kPageSize, 0, kPageSize}; + caps[TierType::DRAM] = TierCapacity{4 * kPageSize, 3 * kPageSize, kPageSize}; + ReportHeartbeat(&store, /*seq=*/1, now, caps, TierType::SSD, "cold", {"dual"}, + /*full_sync=*/true); + + PoolDispatcher dispatcher(&pool); + EvictionManager manager(store, OneSecondRounds(), &dispatcher); + manager.Start(); + const bool freed = + WaitFor([&] { return !registry.Get("cold")->Contains("dual"); }, std::chrono::seconds(8)); + manager.Stop(); + + ASSERT_TRUE(freed) << "SSD was over budget but its copy was never freed"; + EXPECT_EQ(dispatcher.Victims().front(), (EvictionVictim{"dual", TierType::SSD})); + EXPECT_TRUE(registry.Get("hot")->Contains("dual")) + << "SSD pressure freed the DRAM copy as well (or instead)"; + ASSERT_TRUE(pool.PlacementBackend("dual").has_value()); + EXPECT_EQ(*pool.PlacementBackend("dual"), registry.BackendId(registry.Get("hot"))); +} + // Master-side eviction reads one capacity per TierType, and a client with a // weighted multi-backend policy sums its backends into that one figure // (MasterClient::SnapshotAndCacheTierCapacities). So a hot tier sharing DRAM with diff --git a/tests/cpp/umbp/distributed/test_peer_pool.cpp b/tests/cpp/umbp/distributed/test_peer_pool.cpp index b181e6ec6..e0aa1d5b8 100644 --- a/tests/cpp/umbp/distributed/test_peer_pool.cpp +++ b/tests/cpp/umbp/distributed/test_peer_pool.cpp @@ -636,6 +636,163 @@ TEST(PeerPool, DiscardDeletesInsteadOfDemoting) { EXPECT_EQ(pool.PlacementCount(), 0u); } +// A copy-mode promotion leaves the key on both media; the placement follows the +// promoted (hot) copy. Returns once both copies exist and the promotion's read +// lease on the cold copy has lapsed. +void PromoteByCopy(PeerPool* pool, BackendRegistry* registry, const std::string& key) { + auto request = PutRequest(key); + request.logical_tier = "cold"; + auto allocation = pool->BatchAllocate({request}).front(); + ASSERT_EQ(allocation.allocation.outcome, AllocateOutcome::kSuccessAllocated); + ASSERT_TRUE(pool->BatchCommit({PoolCommitRequest{ + {allocation.backend_id, allocation.allocation.slot_id}, key}}) + .front() + .commit.success); + ASSERT_TRUE(pool->BatchResolve({key}, false).front().resolved.found); + ASSERT_TRUE(WaitForKey(registry, "hot", key)); + ASSERT_TRUE(registry->Get("cold")->Contains(key)); + std::this_thread::sleep_for(std::chrono::milliseconds(20)); +} + +std::shared_ptr CopyPromotingHotCold(BackendRegistry* registry) { + auto tiers = HotColdTiers(PoolOffloadTrigger::kOnEvict); + tiers.back().promote_trigger = PoolPromoteTrigger::kOnRead; + tiers.back().promote_mode = PoolTransitionMode::kCopy; + auto compiled = LogicalTierGraph::Compile(tiers, *registry); + EXPECT_TRUE(compiled.ok()) << compiled.error; + return compiled.graph; +} + +TEST(PeerPool, TierScopedEvictFreesOnlyThatMediumsCopy) { + constexpr uint64_t kPageSize = 64; + constexpr auto kShortLease = std::chrono::milliseconds(1); + LocalCopyEngine engine; + BackendRegistry registry; + RegisterPageBackend(®istry, &engine, "hot", TierType::DRAM, kPageSize, 4, kShortLease); + RegisterPageBackend(®istry, &engine, "cold", TierType::SSD, kPageSize, 4, kShortLease); + PeerPool pool(®istry, MakeTieredPlacementPolicy(CopyPromotingHotCold(®istry)), &engine); + PromoteByCopy(&pool, ®istry, "dual"); + + // SSD is the medium under pressure: its copy goes, the promoted DRAM copy + // stays and keeps serving reads. + auto evicted = pool.Evict({PoolEvictRequest{"dual", TierType::SSD}}, PoolEvictMode::kReclaim); + ASSERT_EQ(evicted.size(), 1u); + EXPECT_EQ(evicted.front().bytes_freed, 64u); + EXPECT_FALSE(registry.Get("cold")->Contains("dual")); + EXPECT_TRUE(registry.Get("hot")->Contains("dual")); + EXPECT_EQ(pool.PlacementBackend("dual"), + std::optional{registry.BackendId(registry.Get("hot"))}); +} + +TEST(PeerPool, TierScopedEvictOfThePromotedCopyKeepsTheLowerOne) { + constexpr uint64_t kPageSize = 64; + constexpr auto kShortLease = std::chrono::milliseconds(1); + LocalCopyEngine engine; + BackendRegistry registry; + RegisterPageBackend(®istry, &engine, "hot", TierType::DRAM, kPageSize, 4, kShortLease); + RegisterPageBackend(®istry, &engine, "cold", TierType::SSD, kPageSize, 4, kShortLease); + PeerPool pool(®istry, MakeTieredPlacementPolicy(CopyPromotingHotCold(®istry)), &engine); + PromoteByCopy(&pool, ®istry, "dual"); + + // DRAM under pressure: the hot copy demotes, finds cold already holding the + // key, and so frees DRAM without copying anything. The key stays readable. + auto evicted = pool.Evict({PoolEvictRequest{"dual", TierType::DRAM}}, PoolEvictMode::kReclaim); + EXPECT_EQ(evicted.front().bytes_freed, 64u); + EXPECT_FALSE(registry.Get("hot")->Contains("dual")); + EXPECT_TRUE(registry.Get("cold")->Contains("dual")); + EXPECT_EQ(pool.PlacementBackend("dual"), + std::optional{registry.BackendId(registry.Get("cold"))}); + EXPECT_TRUE(pool.BatchResolve({"dual"}, false).front().resolved.found); +} + +TEST(PeerPool, BackendScopedEvictNamesOneInstance) { + BackendRegistry registry; + ASSERT_TRUE(registry.Register("dram_a", std::make_unique(TierType::DRAM))); + ASSERT_TRUE(registry.Register("dram_b", std::make_unique(TierType::DRAM))); + PeerPool pool(®istry, MakeSingleBackendPolicy()); + // The same key on two instances of one medium: only a backend scope can tell + // them apart. + for (const char* name : {"dram_a", "dram_b"}) { + auto* backend = registry.Get(name); + auto allocated = backend->BatchAllocate({AllocateRequest{"k", 32}}).front(); + ASSERT_TRUE(backend->BatchCommit({CommitRequest{allocated.slot_id, "k"}}).front().success); + } + + PoolEvictRequest request; + request.key = "k"; + request.backend_id = registry.BackendId(registry.Get("dram_b")); + auto evicted = pool.Evict({request}, PoolEvictMode::kReclaim); + EXPECT_EQ(evicted.front().bytes_freed, 32u); + EXPECT_TRUE(registry.Get("dram_a")->Contains("k")); + EXPECT_FALSE(registry.Get("dram_b")->Contains("k")); +} + +TEST(PeerPool, EvictOfACopyThatIsNotThereFreesNothing) { + BackendRegistry registry; + ASSERT_TRUE(registry.Register("dram", std::make_unique(TierType::DRAM))); + PeerPool pool(®istry, MakeSingleBackendPolicy()); + auto* backend = registry.Get("dram"); + auto allocated = backend->BatchAllocate({AllocateRequest{"k", 32}}).front(); + ASSERT_TRUE(backend->BatchCommit({CommitRequest{allocated.slot_id, "k"}}).front().success); + + // Stale master metadata: it believes the key is on SSD. The DRAM copy is + // not the master's to free under an SSD budget. + auto evicted = pool.Evict({PoolEvictRequest{"k", TierType::SSD}}, PoolEvictMode::kReclaim); + EXPECT_EQ(evicted.front().bytes_freed, 0u); + EXPECT_TRUE(backend->Contains("k")); +} + +TEST(PeerPool, ReclaimDemotesOnAWatermarkTierToo) { + // The trigger only says whether the pool moves keys proactively. Asked to + // free a copy whose tier has a downstream, the answer is the same either way: + // move it, do not delete data a lower tier has room for. + constexpr uint64_t kPageSize = 64; + LocalCopyEngine engine; + BackendRegistry registry; + // Big enough that one key stays far under the 0.4 watermark, so the pool's + // own proactive offload never runs and only the explicit evict moves it. + RegisterPageBackend(®istry, &engine, "hot", TierType::DRAM, kPageSize, 16); + RegisterPageBackend(®istry, &engine, "cold", TierType::SSD, kPageSize, 16); + auto compiled = LogicalTierGraph::Compile(HotColdTiers(PoolOffloadTrigger::kWatermark), registry); + ASSERT_TRUE(compiled.ok()) << compiled.error; + PeerPool pool(®istry, MakeTieredPlacementPolicy(compiled.graph), &engine); + CommitKey(&pool, "k"); + + auto evicted = pool.Evict({PoolEvictRequest{"k", TierType::DRAM}}, PoolEvictMode::kReclaim); + EXPECT_EQ(evicted.front().bytes_freed, 64u); + EXPECT_FALSE(registry.Get("hot")->Contains("k")); + EXPECT_TRUE(registry.Get("cold")->Contains("k")) + << "a watermark tier deleted instead of demoting"; +} + +TEST(PeerPool, DemotionWithNoRoomDownstreamDropsTheCopy) { + // "Free this tier" still has to free it when the tier below is full. + // + // A buffer is sized by its mapping, which rounds up to the host page, so a + // tier is only genuinely one page long when the page IS the host page. + constexpr uint64_t kPageSize = 4096; + LocalCopyEngine engine; + BackendRegistry registry; + RegisterPageBackend(®istry, &engine, "hot", TierType::DRAM, kPageSize, 4); + RegisterPageBackend(®istry, &engine, "cold", TierType::SSD, kPageSize, 1); + auto compiled = LogicalTierGraph::Compile(HotColdTiers(PoolOffloadTrigger::kOnEvict), registry); + ASSERT_TRUE(compiled.ok()) << compiled.error; + PeerPool pool(®istry, MakeTieredPlacementPolicy(compiled.graph), &engine); + // Fill cold's only page directly. + auto* cold = registry.Get("cold"); + auto filler = cold->BatchAllocate({AllocateRequest{"filler", kPageSize}}).front(); + ASSERT_EQ(filler.outcome, AllocateOutcome::kSuccessAllocated); + ASSERT_TRUE(cold->BatchCommit({CommitRequest{filler.slot_id, "filler"}}).front().success); + CommitKey(&pool, "k"); + + auto evicted = pool.Evict({PoolEvictRequest{"k", TierType::DRAM}}, PoolEvictMode::kReclaim); + EXPECT_EQ(evicted.front().bytes_freed, 64u); + EXPECT_FALSE(registry.Get("hot")->Contains("k")); + EXPECT_FALSE(cold->Contains("k")); + EXPECT_TRUE(cold->Contains("filler")); + EXPECT_FALSE(pool.PlacementBackend("k").has_value()); +} + TEST(PeerPool, WatermarkMigratesCommittedKey) { // Unlike the other tier tests, this one asserts on utilization, so the owned // buffer must be a whole number of host allocation granules: a smaller diff --git a/tests/cpp/umbp/distributed/test_peer_service_dispatch.cpp b/tests/cpp/umbp/distributed/test_peer_service_dispatch.cpp index 80c33989d..e714d7693 100644 --- a/tests/cpp/umbp/distributed/test_peer_service_dispatch.cpp +++ b/tests/cpp/umbp/distributed/test_peer_service_dispatch.cpp @@ -420,22 +420,61 @@ TEST_F(PeerServiceDispatchTest, BatchResolveReportsTheServingTier) { EXPECT_EQ(resp.size(1), 55u); } -TEST_F(PeerServiceDispatchTest, EvictFansOutAcrossEveryMedium) { - // A key mirrored across media must disappear from ALL of them, and the freed - // bytes master sizes its next round from are the sum. +TEST_F(PeerServiceDispatchTest, EvictWithATierFreesOnlyThatMedium) { + // The master charged this victim to the second medium. A key mirrored + // across media loses exactly that copy; the other one is not the master's to + // free, because it was measured against a different budget. SeedKey(kFirstByTier, "mirrored", 111); SeedKey(kSecondByTier, "mirrored", 222); ::umbp::EvictKeyRequest req; req.add_keys("mirrored"); + req.add_tiers(Proto(kSecondByTier)); ::umbp::EvictKeyResponse resp; grpc::ClientContext ctx; ASSERT_TRUE(stub_->EvictKey(&ctx, req, &resp).ok()); ASSERT_EQ(resp.evicted_size(), 1); EXPECT_EQ(resp.evicted(0).key(), "mirrored"); - EXPECT_EQ(resp.evicted(0).bytes_freed(), 333u); - EXPECT_EQ(Backend(kFirstByTier)->OwnedKeyCount(), 0u); - EXPECT_EQ(Backend(kSecondByTier)->OwnedKeyCount(), 0u); + EXPECT_EQ(resp.evicted(0).bytes_freed(), 222u); + EXPECT_TRUE(Backend(kFirstByTier)->Contains("mirrored")); + EXPECT_FALSE(Backend(kSecondByTier)->Contains("mirrored")); +} + +TEST_F(PeerServiceDispatchTest, EvictWithoutTiersFreesTheCopyThatHoldsTheKey) { + // An older master sends no tiers. Each key is then freed from the medium + // that holds it -- the first owner in read order here, since the seed went + // around the pool and left no placement -- and never from every medium at + // once, which used to delete copies no budget had asked for. + SeedKey(kFirstByTier, "mirrored", 111); + SeedKey(kSecondByTier, "mirrored", 222); + + ::umbp::EvictKeyRequest req; + req.add_keys("mirrored"); + ::umbp::EvictKeyResponse resp; + grpc::ClientContext ctx; + ASSERT_TRUE(stub_->EvictKey(&ctx, req, &resp).ok()); + ASSERT_EQ(resp.evicted_size(), 1); + EXPECT_EQ(resp.evicted(0).bytes_freed(), 111u); + EXPECT_FALSE(Backend(kFirstByTier)->Contains("mirrored")); + EXPECT_TRUE(Backend(kSecondByTier)->Contains("mirrored")); +} + +TEST_F(PeerServiceDispatchTest, EvictWithMismatchedTierCountIsUnscoped) { + // `tiers` is parallel to `keys`; anything else cannot be paired up, so it is + // ignored rather than misapplied to the wrong keys. + SeedKey(kFirstByTier, "a", 10); + SeedKey(kSecondByTier, "b", 20); + + ::umbp::EvictKeyRequest req; + req.add_keys("a"); + req.add_keys("b"); + req.add_tiers(Proto(kSecondByTier)); + ::umbp::EvictKeyResponse resp; + grpc::ClientContext ctx; + ASSERT_TRUE(stub_->EvictKey(&ctx, req, &resp).ok()); + ASSERT_EQ(resp.evicted_size(), 2); + EXPECT_EQ(resp.evicted(0).bytes_freed(), 10u); + EXPECT_EQ(resp.evicted(1).bytes_freed(), 20u); } TEST_F(PeerServiceDispatchTest, EvictOfAnAbsentKeyReportsZeroBytes) { From 5da7268284997a27ed544c39f747315935c28a67 Mon Sep 17 00:00:00 2001 From: TianDi101 Date: Sat, 10 Oct 2026 10:54:48 +0000 Subject: [PATCH 03/19] feat(umbp): one eviction implementation per medium; local eviction through PeerPool::Evict Before this, a committed key could be freed four ways: the master's EvictKey (through PeerPool and the medium's Evict), PageBackend's masterless round (its own copy of Evict's body, inline on commit), PeerSsdManager's round (inline after every SSD write, plus evict-then-retry on a failed write), and tier transitions. The two inline rounds bypassed PeerPool -- no on_evict demotion, stale placement, invisible to InstrumentedBackend -- and the page round ran before the pool's watermark check, so on a masterless node it deleted the cold keys the watermark offload was about to move. Now: - MediumBackend::Evict is the only code that frees a committed key for capacity. PageBackend::MaybeEvictToLowWatermark and PeerSsdManager::EvictToLowWatermark (with its watermarks, after-write checks and evict-and-retry) are gone; a write that does not fit fails. - MediumBackend::EnableLocalEviction is replaced by ordering hooks that free nothing: TrackEvictionOrder() and EvictionCandidates(max) -> EvictionOffer{key, bytes}, coldest first, skipping leased/pinned keys. - PeerPool::EnableLocalEviction(backend, high, low) starts one background worker. Commits, NO_SPACE allocations and finished transitions only wake it; it takes offers until their bytes cover the overage and calls PeerPool::Evict({key, backend_id}, kReclaim) -- the call the master's EvictKey makes -- so a tier with a downstream demotes instead of deleting. Lowest tier first, 64 keys per call, with a fixed byte budget as well as the live usage check. - PoolClient enables it for every medium without a master (paged media at local_evict_*, SSD at its own watermarks) and for SSD with a master, as before. - A demotion that loses a race to another transition no longer falls back to deleting the source, which could abort that transition and lose the key. - mori_umbp_client_local_evict_total{event=round|key|no_candidate|stalled}; the SSD medium's eviction_round event is gone (rounds are the pool's, keys and bytes are InstrumentedBackend's). Co-Authored-By: Claude Opus 5.5 --- .../distributed/peer/backend/page_backend.cpp | 177 ++++---------- .../distributed/peer/backend/ssd_backend.cpp | 22 +- .../distributed/peer/ssd/peer_ssd_manager.cpp | 172 +++----------- src/umbp/distributed/pool/peer_pool.cpp | 219 +++++++++++++++++- src/umbp/distributed/pool_client.cpp | 54 +++-- src/umbp/doc/design-eviction-convergence.md | 27 ++- src/umbp/include/umbp/distributed/config.h | 14 +- .../umbp/distributed/master/master_metrics.h | 9 + .../peer/backend/instrumented_backend.h | 7 +- .../distributed/peer/backend/medium_backend.h | 37 ++- .../distributed/peer/backend/page_backend.h | 70 ++---- .../distributed/peer/backend/ssd_backend.h | 4 + .../distributed/peer/ssd/peer_ssd_manager.h | 77 +++--- .../include/umbp/distributed/pool/peer_pool.h | 69 ++++++ src/umbp/include/umbp/distributed/types.h | 14 ++ tests/cpp/umbp/distributed/test_peer_pool.cpp | 168 ++++++++++++++ .../distributed/test_peer_ssd_eviction.cpp | 121 +++++----- .../test_peer_ssd_single_flight.cpp | 2 +- .../cpp/umbp/distributed/test_ssd_backend.cpp | 66 ++++++ 19 files changed, 869 insertions(+), 460 deletions(-) diff --git a/src/umbp/distributed/peer/backend/page_backend.cpp b/src/umbp/distributed/peer/backend/page_backend.cpp index 479866e53..d32c7e1ef 100644 --- a/src/umbp/distributed/peer/backend/page_backend.cpp +++ b/src/umbp/distributed/peer/backend/page_backend.cpp @@ -359,17 +359,8 @@ void HostPageMemorySource::Release() { // --------------------------------------------------------------------------- AllocateResult PageBackend::Allocate(const std::string& key, uint64_t size) { - AllocateResult out; - { - std::unique_lock lock(mutex_); - out = AllocateLocked(key, size); - } - // A full pool is the one state the commit-time round cannot have prevented - // (one object larger than the headroom, or a burst that never reached - // commit). Evicting here does not rescue THIS allocation -- it is not - // retried -- but it stops the pool staying wedged for every put after it. - if (out.outcome == AllocateOutcome::kFailedNoSpace) MaybeEvictToLowWatermark(); - return out; + std::unique_lock lock(mutex_); + return AllocateLocked(key, size); } AllocateResult PageBackend::AllocateLocked(const std::string& key, uint64_t size, @@ -433,18 +424,13 @@ std::vector PageBackend::BatchAllocate( const std::vector& entries) { std::vector out(entries.size()); if (entries.empty()) return out; - bool no_space = false; - { - std::unique_lock lock(mutex_); - for (size_t i = 0; i < entries.size(); ++i) { - out[i] = AllocateLocked(entries[i].key, entries[i].size, entries[i].preferred_numa_node); - if (out[i].outcome == AllocateOutcome::kSuccessAllocated) { - out[i].descs = BuildBufferDescsLocked(out[i].pages); - } - no_space = no_space || out[i].outcome == AllocateOutcome::kFailedNoSpace; + std::unique_lock lock(mutex_); + for (size_t i = 0; i < entries.size(); ++i) { + out[i] = AllocateLocked(entries[i].key, entries[i].size, entries[i].preferred_numa_node); + if (out[i].outcome == AllocateOutcome::kSuccessAllocated) { + out[i].descs = BuildBufferDescsLocked(out[i].pages); } } - if (no_space) MaybeEvictToLowWatermark(); return out; } @@ -453,16 +439,8 @@ std::vector PageBackend::BatchAllocate( // --------------------------------------------------------------------------- bool PageBackend::Commit(uint64_t slot_id, const std::string& key, uint64_t& bytes_committed) { - bool ok = false; - { - std::unique_lock lock(mutex_); - ok = CommitLocked(slot_id, key, bytes_committed); - } - // Check-after-write, on the committing thread: there is no eviction thread, - // and a commit is the only event that grows usage. Outside the lock because - // a round takes mutex_ itself. No-op unless local eviction is enabled. - MaybeEvictToLowWatermark(); - return ok; + std::unique_lock lock(mutex_); + return CommitLocked(slot_id, key, bytes_committed); } bool PageBackend::CommitLocked(uint64_t slot_id, const std::string& key, @@ -510,15 +488,10 @@ bool PageBackend::CommitLocked(uint64_t slot_id, const std::string& key, std::vector PageBackend::BatchCommit(const std::vector& entries) { std::vector out(entries.size()); if (entries.empty()) return out; - { - std::unique_lock lock(mutex_); - for (size_t i = 0; i < entries.size(); ++i) { - out[i].success = CommitLocked(entries[i].slot_id, entries[i].key, out[i].bytes_committed); - } + std::unique_lock lock(mutex_); + for (size_t i = 0; i < entries.size(); ++i) { + out[i].success = CommitLocked(entries[i].slot_id, entries[i].key, out[i].bytes_committed); } - // One round for the whole batch, not one per key: a batch is a single burst - // of growth and a round already frees down to the low watermark. - MaybeEvictToLowWatermark(); return out; } @@ -583,7 +556,7 @@ std::vector PageBackend::BatchResolve(const std::vector> touched; - if (local_evict_enabled_) touched.reserve(keys.size()); + if (track_eviction_order_) touched.reserve(keys.size()); for (size_t i = 0; i < keys.size(); ++i) { auto it = owned_.find(keys[i]); if (it == owned_.end()) continue; @@ -595,7 +568,7 @@ std::vector PageBackend::BatchResolve(const std::vectorsecond.pages); RenewLeaseAtomic(it->second, lease_deadline); - if (local_evict_enabled_) touched.emplace_back(&keys[i], &it->second); + if (track_eviction_order_) touched.emplace_back(&keys[i], &it->second); } if (!touched.empty()) { std::lock_guard lru_lock(lru_mutex_); @@ -831,35 +804,45 @@ void PageBackend::SetEventPublishing(bool enabled) { } // --------------------------------------------------------------------------- -// Local eviction (no-master deployments; see EnableLocalEviction) +// Eviction order (the policy that consumes it lives in PeerPool) // --------------------------------------------------------------------------- -void PageBackend::EnableLocalEviction(double high_watermark, double low_watermark) { - if (!(high_watermark > 0.0 && high_watermark <= 1.0 && low_watermark > 0.0 && - low_watermark < high_watermark)) { - MORI_UMBP_WARN( - "[PageBackend] ignoring local eviction watermarks high={} low={} " - "(require 0 < low < high <= 1) — local eviction stays OFF", - high_watermark, low_watermark); - return; - } +void PageBackend::TrackEvictionOrder() { std::unique_lock lock(mutex_); - local_evict_enabled_ = true; - local_evict_high_wm_ = high_watermark; - local_evict_low_wm_ = low_watermark; - // Keys committed before this call are not in lru_ and would therefore be - // invisible to a round. Enabling before Init is the contract, but seed - // anyway so a late enable degrades to "wrong order", not "never evicted". - { - std::lock_guard lru_lock(lru_mutex_); - for (auto& [key, slot] : owned_) TouchLruLocked(key, slot); + if (track_eviction_order_) return; + track_eviction_order_ = true; + // Keys committed before this call are not in lru_ and would otherwise never + // be offered. They have no recorded access, so they go in as coldest. + std::lock_guard lru_lock(lru_mutex_); + for (auto& [key, slot] : owned_) TouchLruLocked(key, slot); +} + +std::vector PageBackend::EvictionCandidates(size_t max_count) { + std::vector out; + if (max_count == 0) return out; + // Shared: this only reads owned_ and pins_. The LRU has its own lock. + std::shared_lock lock(mutex_); + if (!track_eviction_order_) return out; + const auto now = std::chrono::steady_clock::now(); + std::lock_guard lru_lock(lru_mutex_); + for (auto it = lru_.rbegin(); it != lru_.rend() && out.size() < max_count; ++it) { + auto owned_it = owned_.find(*it); + if (owned_it == owned_.end()) continue; // defensive; lru_/owned_ stay in sync + // The two protections Evict() applies: an in-flight read owns the pages + // until its lease expires, a copy pin until the copy is done. Offering + // them would only make the caller's batch free less than it asked for. + if (HasActiveReadLeaseLocked(owned_it->second, now)) continue; + if (HasActivePinLocked(*it)) continue; + // Whole pages: that is what the allocator gives back, and what Capacity() + // counts against the watermark. + out.push_back( + EvictionOffer{*it, static_cast(owned_it->second.pages.size()) * page_size_}); } - MORI_UMBP_INFO("[PageBackend] local eviction ON tier={} high={} low={}", static_cast(tier_), - high_watermark, low_watermark); + return out; } void PageBackend::TouchLruLocked(const std::string& key, OwnedSlot& slot) { - if (!local_evict_enabled_) return; + if (!track_eviction_order_) return; if (slot.in_lru) { lru_.splice(lru_.begin(), lru_, slot.lru_it); return; @@ -876,7 +859,7 @@ void PageBackend::RemoveFromLruLocked(OwnedSlot& slot) { } void PageBackend::TouchLru(const std::string& key, OwnedSlot& slot) { - if (!local_evict_enabled_) return; + if (!track_eviction_order_) return; std::lock_guard lru_lock(lru_mutex_); TouchLruLocked(key, slot); } @@ -886,72 +869,6 @@ void PageBackend::RemoveFromLru(OwnedSlot& slot) { RemoveFromLruLocked(slot); } -size_t PageBackend::MaybeEvictToLowWatermark() { - // Only one round at a time. A second committing thread backs off rather - // than over-evicting, and try_lock keeps the put path non-blocking. - std::unique_lock round(local_evict_round_mutex_, std::try_to_lock); - if (!round.owns_lock()) return 0; - - std::unique_lock lock(mutex_); - if (!local_evict_enabled_ || !allocator_) return 0; - - const uint64_t total = allocator_->TotalBytes(); - if (total == 0) return 0; - const uint64_t used = allocator_->UsedBytes(); - if (static_cast(used) < local_evict_high_wm_ * static_cast(total)) return 0; - - const auto low_bytes = static_cast(local_evict_low_wm_ * static_cast(total)); - if (used <= low_bytes) return 0; - const uint64_t to_free = used - low_bytes; - - // Oldest first. Two passes so the list is not mutated under its own - // reverse iterator; the selection pass is where the skips happen. - const auto now = std::chrono::steady_clock::now(); - uint64_t selected_bytes = 0; - std::vector victims; - { - std::lock_guard lru_lock(lru_mutex_); - for (auto it = lru_.rbegin(); it != lru_.rend() && selected_bytes < to_free; ++it) { - auto owned_it = owned_.find(*it); - if (owned_it == owned_.end()) continue; // defensive; lru_/owned_ stay in sync - // Same two protections master-driven Evict honours: an in-flight read - // owns the pages until its lease expires, and a copy pin until the - // worker is done. A round that finds only protected keys frees what it - // can and stops — it must not spin waiting for a reader. - if (HasActiveReadLeaseLocked(owned_it->second, now)) continue; - if (HasActivePinLocked(*it)) continue; - victims.push_back(*it); - selected_bytes += static_cast(owned_it->second.pages.size()) * page_size_; - } - } - - size_t freed = 0; - for (const auto& key : victims) { - auto owned_it = owned_.find(key); - if (owned_it == owned_.end()) continue; - if (allocator_) allocator_->Deallocate(owned_it->second.pages); - // A REMOVE is queued for symmetry with master-driven Evict; with no master - // event publishing is off and this is a no-op, so nothing accumulates. - QueueEventLocked(KvEvent{KvEvent::Kind::REMOVE, key, tier_, 0}); - RemoveFromLru(owned_it->second); - owned_.erase(owned_it); - ++freed; - } - if (freed > 0) { - MORI_UMBP_DEBUG("[PageBackend] local eviction freed {} key(s) ({} bytes) tier={}", freed, - selected_bytes, static_cast(tier_)); - local_evict_warned_ = false; - } else if (!local_evict_warned_) { - // Latched: a wedged pool would otherwise log this on every commit. - local_evict_warned_ = true; - MORI_UMBP_WARN( - "[PageBackend] local eviction found no reclaimable key at {}/{} bytes used " - "(all leased or pinned) tier={}", - used, total, static_cast(tier_)); - } - return freed; -} - void PageBackend::SetAutoFlushHook(size_t threshold, std::function cb) { std::unique_lock lock(mutex_); auto_flush_threshold_ = threshold; diff --git a/src/umbp/distributed/peer/backend/ssd_backend.cpp b/src/umbp/distributed/peer/backend/ssd_backend.cpp index 6e14857e8..39880a256 100644 --- a/src/umbp/distributed/peer/backend/ssd_backend.cpp +++ b/src/umbp/distributed/peer/backend/ssd_backend.cpp @@ -718,6 +718,20 @@ std::vector SsdBackend::Evict(const std::vector& keys) return results; } +std::vector SsdBackend::EvictionCandidates(size_t max_count) { + if (ssd_ == nullptr || max_count == 0) return {}; + auto candidates = ssd_->EvictionCandidates(max_count); + std::lock_guard lock(mutex_); + if (read_leases_.empty() && migration_reads_.empty()) return candidates; + candidates.erase(std::remove_if(candidates.begin(), candidates.end(), + [&](const EvictionOffer& offer) { + return read_leases_.count(offer.key) != 0 || + migration_reads_.count(offer.key) != 0; + }), + candidates.end()); + return candidates; +} + // --------------------------------------------------------------------------- // Bootstrap // --------------------------------------------------------------------------- @@ -862,10 +876,10 @@ std::vector SsdBackend::SampleMetrics() const { event("single_flight_dup", ssd_->ReadDup()); event("single_flight_merged", ssd_->ReadMerged()); - // Local high-watermark eviction. The victims and the freed bytes are the - // decorator's (Evict is an interface call); the ROUNDS are not — master never - // asked for them — and neither is a backend delete that refused. - event("eviction_round", ssd_->EvictionRounds()); + // Every eviction -- master-driven or the pool's local watermark eviction -- + // arrives through Evict(), so the keys and bytes are the decorator's and the + // rounds are the pool's (mori_umbp_client_local_evict_total). What neither + // can see is a delete the drive itself refused. event("eviction_backend_failed", ssd_->EvictionBackendFailures()); // Staging pressure. slot_full_reject is the one to watch: it counts resolves diff --git a/src/umbp/distributed/peer/ssd/peer_ssd_manager.cpp b/src/umbp/distributed/peer/ssd/peer_ssd_manager.cpp index b6e3f823a..c37c0941b 100644 --- a/src/umbp/distributed/peer/ssd/peer_ssd_manager.cpp +++ b/src/umbp/distributed/peer/ssd/peer_ssd_manager.cpp @@ -38,14 +38,6 @@ namespace mori::umbp { -namespace { -// Watermarks must satisfy 0 < low < high <= 1. We fail fast on a bad value -// rather than silently clamping, so a misconfigured env surfaces immediately. -bool WatermarksValid(double high, double low) { - return high > 0.0 && high <= 1.0 && low > 0.0 && low < high; -} -} // namespace - // --------------------------------------------------------------------------- // Construction // --------------------------------------------------------------------------- @@ -56,14 +48,8 @@ PeerSsdManager::PeerSsdManager(const PeerSsdConfig& cfg) { return; } const auto& ssd = cfg.ssd; - - if (!WatermarksValid(ssd.high_watermark, ssd.low_watermark)) { - throw std::runtime_error( - "[PeerSsdManager] invalid SSD watermarks (require 0 < low_watermark < " - "high_watermark <= 1)"); - } - high_watermark_ = ssd.high_watermark; - low_watermark_ = ssd.low_watermark; + // The SSD watermarks in `ssd` are consumed by PeerPool's local eviction, + // which PoolClient enables for this medium -- not here. single_flight_ = ssd.single_flight_reads; // Explicit backend selection — an unknown value is a configuration error, not @@ -100,18 +86,8 @@ PeerSsdManager::PeerSsdManager(const PeerSsdConfig& cfg) { "' (expected one of: file, spdk, spdk_proxy)"); } -PeerSsdManager::PeerSsdManager(std::unique_ptr backend, double high_watermark, - double low_watermark, bool single_flight) - : backend_(std::move(backend)), - high_watermark_(high_watermark), - low_watermark_(low_watermark), - single_flight_(single_flight) { - if (!WatermarksValid(high_watermark_, low_watermark_)) { - throw std::runtime_error( - "[PeerSsdManager] invalid SSD watermarks (require 0 < low_watermark < " - "high_watermark <= 1)"); - } -} +PeerSsdManager::PeerSsdManager(std::unique_ptr backend, bool single_flight) + : backend_(std::move(backend)), single_flight_(single_flight) {} PeerSsdManager::~PeerSsdManager() = default; @@ -215,14 +191,11 @@ bool PeerSsdManager::Write(const std::string& key, } // Backend IO outside our mutex_ — SSDTier is internally synchronized. A - // failure may be ENOSPC: run one eviction round to reclaim space and retry - // once before giving up (best-effort, no event/record on final failure). + // failure (ENOSPC included) records nothing and queues nothing. Making room + // is the pool's local eviction, on its own thread, never this write. if (!backend_->Write(key, data, total_size)) { - EvictToLowWatermark(); - if (!backend_->Write(key, data, total_size)) { - MORI_UMBP_WARN("[PeerSsdManager] backend Write failed key={} size={}", key, total_size); - return false; - } + MORI_UMBP_WARN("[PeerSsdManager] backend Write failed key={} size={}", key, total_size); + return false; } // Bytes physically written to the SSD device (write IO bandwidth source). @@ -244,19 +217,7 @@ bool PeerSsdManager::Write(const std::string& key, pending_events_.push_back(KvEvent{KvEvent::Kind::ADD, key, TierType::SSD, total_size}); } } - // Keep the just-written key out of this write's own watermark victim set. - // A successful Write must imply the key still exists when it returns. - auto& slot = inflight_reads_[key]; - if (!slot) slot = std::make_unique(); - ++slot->refs; - } - - // Check-after-write trigger, on this copy worker (no dedicated thread). - auto [used, total] = Capacity(); - if (total > 0 && static_cast(used) >= high_watermark_ * static_cast(total)) { - EvictToLowWatermark(); } - UnpinForMigration(key); return true; } @@ -274,7 +235,7 @@ std::vector PeerSsdManager::WriteBatch(const std::vector& key // Stage timers for the [SsdPerf/peer] PUT breakdown (no-ops unless // UMBP_SSD_TIMING is set). const auto t_begin = ssdperf::Now(); - double dedup_ms = 0.0, backend_ms = 0.0, retry_ms = 0.0, record_ms = 0.0, evict_ms = 0.0; + double dedup_ms = 0.0, backend_ms = 0.0, record_ms = 0.0; // Pass 1 — dedup under one lock. Already-owned keys succeed with no IO (same // rationale as Write: a sequential re-put must not repeat the device write). @@ -296,46 +257,25 @@ std::vector PeerSsdManager::WriteBatch(const std::vector& key const size_t dedup_hits = keys.size() - todo.size(); if (todo.empty()) return results; - auto gather = [&](const std::vector& idxs) { - std::vector k; - std::vector d; - std::vector z; - k.reserve(idxs.size()); - d.reserve(idxs.size()); - z.reserve(idxs.size()); - for (size_t i : idxs) { - k.push_back(keys[i]); - d.push_back(srcs[i]); - z.push_back(sizes[i]); - } - return std::make_tuple(std::move(k), std::move(d), std::move(z)); - }; - - // Pass 2 — one batched backend write outside the lock. Failures are most - // likely ENOSPC, so retry exactly those once after a single eviction round - // (mirrors Write's evict-and-retry, amortized over the whole batch). + // Pass 2 — one batched backend write outside the lock. A key that fails + // (ENOSPC included) is reported as failed and recorded nowhere; making room + // is the pool's local eviction, on its own thread, never this write. const auto t_backend = ssdperf::Now(); - auto [bk, bd, bz] = gather(todo); - auto ok = backend_->BatchWrite(bk, bd, bz); + std::vector batch_keys; + std::vector batch_srcs; + std::vector batch_sizes; + batch_keys.reserve(todo.size()); + batch_srcs.reserve(todo.size()); + batch_sizes.reserve(todo.size()); + for (size_t i : todo) { + batch_keys.push_back(keys[i]); + batch_srcs.push_back(srcs[i]); + batch_sizes.push_back(sizes[i]); + } + auto ok = backend_->BatchWrite(batch_keys, batch_srcs, batch_sizes); ok.resize(todo.size(), false); backend_ms = ssdperf::MsSince(t_backend); - std::vector retry; - for (size_t j = 0; j < todo.size(); ++j) { - if (!ok[j]) retry.push_back(todo[j]); - } - if (!retry.empty()) { - const auto t_retry = ssdperf::Now(); - EvictToLowWatermark(); - auto [rk, rd, rz] = gather(retry); - auto retry_ok = backend_->BatchWrite(rk, rd, rz); - retry_ok.resize(retry.size(), false); - for (size_t j = 0, r = 0; j < todo.size(); ++j) { - if (!ok[j]) ok[j] = retry_ok[r++]; - } - retry_ms = ssdperf::MsSince(t_retry); - } - // Pass 3 — record locations + ADD events under one lock. The owned_ re-check // matches Write: a concurrent writer may have landed the same content-addressed // bytes, and only the first recorder emits ADD SSD. @@ -369,29 +309,21 @@ std::vector PeerSsdManager::WriteBatch(const std::vector& key } } - // One check-after-write for the whole batch instead of one per key. - const auto t_evict = ssdperf::Now(); - auto [used, total] = Capacity(); - if (total > 0 && static_cast(used) >= high_watermark_ * static_cast(total)) { - EvictToLowWatermark(); - } - evict_ms = ssdperf::MsSince(t_evict); - if (ssdperf::Enabled()) { const double total_ms = ssdperf::MsSince(t_begin); + const auto [used, total] = Capacity(); MORI_UMBP_INFO( "[SsdPerf/peer] PUT keys={} dedup_hits={} written={} bytes={} total_ms={:.3f} GB_s={:.2f} " - "| dedup={:.3f}ms backend={:.3f}ms ({:.0f}%) evict_retry={:.3f}ms record={:.3f}ms " - "evict_check={:.3f}ms | used={}/{}B", + "| dedup={:.3f}ms backend={:.3f}ms ({:.0f}%) record={:.3f}ms | used={}/{}B", keys.size(), dedup_hits, todo.size(), written_bytes, total_ms, ssdperf::GbPerSec(written_bytes, total_ms), dedup_ms, backend_ms, - ssdperf::Pct(backend_ms, total_ms), retry_ms, record_ms, evict_ms, used, total); + ssdperf::Pct(backend_ms, total_ms), record_ms, used, total); } return results; } // --------------------------------------------------------------------------- -// Eviction (local, read-priority) +// Eviction (the medium's one implementation; read-priority) // --------------------------------------------------------------------------- bool PeerSsdManager::Evict(const std::string& key) { @@ -431,50 +363,21 @@ bool PeerSsdManager::Evict(const std::string& key) { return true; } -std::vector PeerSsdManager::SelectVictims(size_t bytes_to_free) { +std::vector PeerSsdManager::EvictionCandidates(size_t max_count) { std::lock_guard lock(mutex_); - std::vector victims; - if (bytes_to_free == 0) return victims; + std::vector candidates; + if (max_count == 0) return candidates; // Oldest first (LRU tail -> MRU front), skipping keys being read or evicted. - size_t freed = 0; - for (auto it = lru_.rbegin(); it != lru_.rend(); ++it) { + for (auto it = lru_.rbegin(); it != lru_.rend() && candidates.size() < max_count; ++it) { const std::string& key = *it; if (inflight_reads_.count(key) != 0) continue; if (evicting_.count(key) != 0) continue; auto owned_it = owned_.find(key); if (owned_it == owned_.end()) continue; // defensive; lru_/owned_ stay in sync - victims.push_back(key); - freed += owned_it->second.size; - if (freed >= bytes_to_free) break; - } - return victims; -} - -void PeerSsdManager::EvictToLowWatermark() { - if (!backend_) return; - - // Only one eviction round at a time: a second concurrent worker (or a worker - // racing ClearLocal) backs off instead of over-evicting. try_lock keeps the - // copy path non-blocking. - std::unique_lock round(eviction_mu_, std::try_to_lock); - if (!round.owns_lock()) return; - - auto [used, total] = Capacity(); - if (total == 0) return; - double low_bytes = low_watermark_ * static_cast(total); - if (static_cast(used) <= low_bytes) return; - size_t bytes_to_free = used - static_cast(low_bytes); - - // A real round is about to run (we own the round lock and are over the low - // watermark). Count it before selecting victims. - metrics_.evict_rounds.fetch_add(1, std::memory_order_relaxed); - - // Single pass, no retry loop: if everything reclaimable is in use we free - // what we can and stop, so a fully-pinned tier cannot starve the worker. - for (const auto& key : SelectVictims(bytes_to_free)) { - Evict(key); + candidates.push_back(EvictionOffer{key, owned_it->second.size}); } + return candidates; } // --------------------------------------------------------------------------- @@ -487,9 +390,10 @@ void PeerSsdManager::ClearLocal() { // re-add owned_/ADD SSD after backend->Clear() wipes its bytes. // PoolClient::Clear() Quiesce()s first; new callers must too. // - // Exclude eviction rounds for the whole operation (lock order: eviction_mu_ - // before mutex_, matching EvictToLowWatermark). - std::lock_guard round(eviction_mu_); + // No eviction is in flight: the only caller of Evict is PeerPool::Evict, + // which holds the lifecycle lock PeerPool::ClearLocal takes exclusively + // before it gets here. An Evict that did overlap would still be safe -- it + // finds its key gone from owned_ and records nothing -- just pointless. { std::unique_lock lock(mutex_); owned_.clear(); diff --git a/src/umbp/distributed/pool/peer_pool.cpp b/src/umbp/distributed/pool/peer_pool.cpp index 6418a178a..c5c5d3c20 100644 --- a/src/umbp/distributed/pool/peer_pool.cpp +++ b/src/umbp/distributed/pool/peer_pool.cpp @@ -45,6 +45,22 @@ constexpr size_t kWatermarkCandidateBatch = 64; // watermark would re-walk the placements. constexpr auto kWatermarkScanInterval = std::chrono::milliseconds(5); +// Local eviction frees at most this many keys per Evict() call, so no single +// call holds the pool lock for a whole drain; the pass keeps calling until the +// backend is back under its low watermark. +constexpr size_t kLocalEvictBatch = 64; +// Wakes are the primary signal; this catches growth no wake reported. +constexpr auto kLocalEvictPollInterval = std::chrono::milliseconds(100); +// Floor between passes, so commits arriving back to back above the high +// watermark cannot keep the worker re-reading capacity in a tight loop. +constexpr auto kLocalEvictMinGap = std::chrono::milliseconds(1); + +uint64_t UsedBytes(const TierCapacity& capacity) { + return capacity.total_bytes > capacity.available_bytes + ? capacity.total_bytes - capacity.available_bytes + : 0; +} + } // namespace PeerPool::PeerPool(BackendRegistry* backends, std::unique_ptr policy, @@ -76,7 +92,12 @@ PeerPool::PeerPool(BackendRegistry* backends, std::unique_ptr policy } } -PeerPool::~PeerPool() { StopTransitionWorker(); } +PeerPool::~PeerPool() { + // Eviction first: a pass may be inside RunTransition, which the transition + // machinery below must still be alive for. + StopLocalEviction(); + StopTransitionWorker(); +} void PeerPool::EnqueueTransition(TransitionJob job) { std::lock_guard lock(transition_mutex_); @@ -231,6 +252,8 @@ void PeerPool::FinishTransitionLocked(const TransitionJob& job, return; } ++transition_metrics_.succeeded; + // The target medium just grew without passing through BatchCommit. + WakeLocalEviction(); if (job.kind == TierTransitionKind::kPromotion) { transition_metrics_.promoted_bytes += result.bytes_moved; } else { @@ -451,6 +474,10 @@ std::vector PeerPool::BatchAllocate( pending_keys_.erase(requests[index].key); break; case AllocateOutcome::kFailedNoSpace: + // A full medium: the local eviction worker should not wait for its + // poll to notice. This put still fails over to the next candidate + // (or fails); eviction never runs on the put itself. + WakeLocalEviction(); advance(index); break; } @@ -532,6 +559,9 @@ std::vector PeerPool::BatchCommit( } } MaybeEnqueueWatermarkOffloadLocked(); + // A commit is what grows usage. Waking is all the put path does; the worker + // reads capacity and evicts on its own thread. + WakeLocalEviction(); return out; } @@ -979,6 +1009,11 @@ std::vector PeerPool::Evict(const std::vector& re out[index].bytes_freed = migrated.bytes_freed; continue; } + // Another transition got to this key first (an earlier RunTransition in + // this loop released the lock): it is moving the copy, which is the better + // outcome. Deleting the source now could land before that copy pins it + // and lose the key outright, so leave it -- 0 freed, the caller retries. + if (migrating_keys_.count(requests[index].key) != 0) continue; // Nowhere downstream had room, or the copy moved meanwhile: free it by // deleting it instead. The tier asked for bytes back either way. drops.emplace_back(index, source_id); @@ -1011,6 +1046,188 @@ std::vector PeerPool::Evict(const std::vector& re return out; } +// --------------------------------------------------------------------------- +// Local watermark eviction +// +// The policy half of eviction on a node that has to evict for itself. It +// decides when (a backend at its high watermark) and what (the medium's own +// coldest candidates), and then asks Evict() above -- the same call the +// master's EvictKey makes. It never frees a byte itself, and it runs on its +// own thread: a put only wakes it. +// --------------------------------------------------------------------------- + +bool PeerPool::EnableLocalEviction(uint32_t backend_id, double high_watermark, + double low_watermark) { + if (!(high_watermark > 0.0 && high_watermark <= 1.0 && low_watermark > 0.0 && + low_watermark < high_watermark)) { + MORI_UMBP_WARN( + "[PeerPool] ignoring local eviction watermarks high={} low={} for backend {} " + "(require 0 < low < high <= 1)", + high_watermark, low_watermark, backend_id); + return false; + } + auto* backend = backends_ == nullptr ? nullptr : backends_->Get(backend_id); + if (backend == nullptr) return false; + // Before the worker can ask for candidates. + backend->TrackEvictionOrder(); + + std::lock_guard lock(local_evict_mutex_); + if (stop_local_evict_) return false; + local_evict_[backend_id] = LocalEvictionWatermarks{high_watermark, low_watermark}; + if (!local_evict_worker_.joinable()) { + local_evict_worker_ = std::thread(&PeerPool::LocalEvictionLoop, this); + } + const auto* entry = backends_->GetEntry(backend_id); + MORI_UMBP_INFO("[PeerPool] local eviction ON backend={} tier={} high={} low={}", + entry == nullptr ? std::string(backend->Name()) : entry->name, + TierTypeName(backend->Tier()), high_watermark, low_watermark); + return true; +} + +LocalEvictionMetrics PeerPool::LocalEvictionStats() const { + LocalEvictionMetrics metrics; + metrics.rounds = local_evict_rounds_.load(std::memory_order_relaxed); + metrics.keys = local_evict_keys_.load(std::memory_order_relaxed); + metrics.no_candidate = local_evict_no_candidate_.load(std::memory_order_relaxed); + metrics.stalled = local_evict_stalled_.load(std::memory_order_relaxed); + return metrics; +} + +void PeerPool::WakeLocalEviction() { + // A wake already pending covers this one. With local eviction off nothing + // ever clears the flag, so every call after the first returns right here. + if (local_evict_wake_.exchange(true, std::memory_order_acq_rel)) return; + // Notify under the lock the worker waits with, or a notify landing between + // its predicate check and its wait would be lost until the next poll. + std::lock_guard lock(local_evict_mutex_); + local_evict_cv_.notify_one(); +} + +void PeerPool::StopLocalEviction() { + { + std::lock_guard lock(local_evict_mutex_); + stop_local_evict_ = true; + } + local_evict_cv_.notify_all(); + if (local_evict_worker_.joinable()) local_evict_worker_.join(); +} + +void PeerPool::LocalEvictionLoop() { + for (;;) { + { + std::unique_lock lock(local_evict_mutex_); + local_evict_cv_.wait_for(lock, kLocalEvictPollInterval, [&] { + return stop_local_evict_ || local_evict_wake_.load(std::memory_order_acquire); + }); + if (stop_local_evict_) return; + } + // Cleared before the pass, so a commit that lands during it schedules the + // next one rather than being absorbed by this. + local_evict_wake_.store(false, std::memory_order_release); + LocalEvictionPass(); + std::this_thread::sleep_for(kLocalEvictMinGap); + } +} + +size_t PeerPool::LocalEvictionPass() { + std::lock_guard pass(local_evict_pass_mutex_); + std::vector> targets; + { + std::lock_guard lock(local_evict_mutex_); + targets.assign(local_evict_.begin(), local_evict_.end()); + } + if (targets.empty() || backends_ == nullptr) return 0; + + // Lowest tier first -- SSD, then DRAM, then HBM. Evicting from an upper + // tier demotes into the one below, so freeing the lower one first is what + // lets that demotion find room instead of falling back to a drop. + const auto depth = [this](uint32_t backend_id) { + auto* backend = backends_->Get(backend_id); + return backend == nullptr ? 0 : static_cast(backend->Tier()); + }; + std::stable_sort(targets.begin(), targets.end(), + [&](const auto& a, const auto& b) { return depth(a.first) > depth(b.first); }); + + size_t freed = 0; + for (const auto& [backend_id, watermarks] : targets) { + freed += ReclaimBackend(backend_id, watermarks); + } + return freed; +} + +size_t PeerPool::ReclaimBackend(uint32_t backend_id, const LocalEvictionWatermarks& watermarks) { + auto* backend = backends_->Get(backend_id); + if (backend == nullptr) return 0; + const TierCapacity capacity = backend->Capacity(); + if (capacity.total_bytes == 0) return 0; + const double total = static_cast(capacity.total_bytes); + const uint64_t used = UsedBytes(capacity); + if (static_cast(used) < watermarks.high * total) return 0; + const auto low_bytes = static_cast(watermarks.low * total); + if (used <= low_bytes) return 0; + local_evict_rounds_.fetch_add(1, std::memory_order_relaxed); + + // Free what was over the low watermark at the start -- counted in the bytes + // each offer said it gives back, which is what Capacity() counts -- or until + // usage is back at the low watermark, whichever comes first. The budget is + // the guard against a medium whose reported usage lags its deletes, which + // re-reading usage alone would keep draining. + const uint64_t budget = used - low_bytes; + const TierType tier = backend->Tier(); + uint64_t freed_bytes = 0; + size_t freed_keys = 0; + while (freed_bytes < budget) { + { + std::lock_guard lock(local_evict_mutex_); + if (stop_local_evict_) break; + } + auto offers = backend->EvictionCandidates(kLocalEvictBatch); + if (offers.empty()) { + local_evict_no_candidate_.fetch_add(1, std::memory_order_relaxed); + MORI_UMBP_DEBUG( + "[PeerPool] local eviction: backend {} over its watermark but offers no " + "candidate (all leased or pinned)", + backend_id); + break; + } + // Only as many of the coldest as the remaining budget needs (at least one), + // so a batch cannot carry the drain past the low watermark. + std::vector requests; + std::vector offered_bytes; + uint64_t taken = 0; + for (auto& offer : offers) { + if (!requests.empty() && freed_bytes + taken >= budget) break; + taken += offer.bytes; + offered_bytes.push_back(offer.bytes); + PoolEvictRequest request; + request.key = std::move(offer.key); + request.tier = tier; + request.backend_id = backend_id; + requests.push_back(std::move(request)); + } + // The same entry point the master's EvictKey uses: a candidate whose tier + // has a downstream is demoted, anything else is dropped from this backend. + const auto results = Evict(requests, PoolEvictMode::kReclaim); + size_t batch_keys = 0; + for (size_t i = 0; i < results.size() && i < offered_bytes.size(); ++i) { + if (results[i].bytes_freed == 0) continue; + ++batch_keys; + freed_bytes += offered_bytes[i]; + } + if (batch_keys == 0) { + // Every candidate was mid-transition or became busy since it was + // offered. Spinning would only re-offer the same keys; the next wake or + // poll tries again. + local_evict_stalled_.fetch_add(1, std::memory_order_relaxed); + break; + } + freed_keys += batch_keys; + if (UsedBytes(backend->Capacity()) <= low_bytes) break; + } + local_evict_keys_.fetch_add(freed_keys, std::memory_order_relaxed); + return freed_keys; +} + void PeerPool::ClearLocal() { std::unique_lock lifecycle_lock(lifecycle_mutex_); std::unique_lock operation_lock(operation_mutex_); diff --git a/src/umbp/distributed/pool_client.cpp b/src/umbp/distributed/pool_client.cpp index 266f8d84b..da90954ea 100644 --- a/src/umbp/distributed/pool_client.cpp +++ b/src/umbp/distributed/pool_client.cpp @@ -1063,19 +1063,10 @@ bool PoolClient::Init() { // every named backend, including multiple instances of the same medium. backend = MakeInstrumentedBackend(std::move(backend)); - // A node with no master owns its whole storage policy, and both halves of - // that follow from the same fact: nobody will ever call Evict() on this - // backend, and nobody will ever drain its heartbeat outbox. So the backend - // evicts for itself, and stops recording events addressed to a master that - // does not exist. Both are no-ops in a cluster -- master-driven eviction - // stays the only policy there, and the outbox keeps feeding the heartbeat. - // - // Before Init(), which is the contract EnableLocalEviction states. - if (!master_client_) { - backend->SetEventPublishing(false); - backend->EnableLocalEviction(config_.local_evict_high_watermark, - config_.local_evict_low_watermark); - } + // A node with no master has nobody to drain its heartbeat outbox, so the + // backend stops recording events addressed to a master that does not + // exist. (Its eviction is the pool's job; see EnableLocalEviction below.) + if (!master_client_) backend->SetEventPublishing(false); // Narrowed to MemoryRegistrar: a backend publishes endpoints, it does not // move bytes, and that is a compile-time fact (design doc §5 Rule C). @@ -1102,6 +1093,33 @@ bool PoolClient::Init() { } default_pool_ = std::make_unique(®istry_, std::move(placement_policy), transfer_engine_.get()); + + // Local watermark eviction (design-eviction-convergence.md §6). It goes + // through PeerPool::Evict, the same call a master's EvictKey makes, on the + // pool's own thread -- never on a put. Which media get it is unchanged: + // * no master: every medium, since nothing else would ever free a page; + // * with a master: SSD only, which has always reclaimed for itself between + // the master's periodic rounds. DRAM/HBM stay master-driven there. + // SSD uses the watermarks in its own config, paged media the node-wide pair. + for (const auto& backend_config : backend_configs) { + auto* backend = registry_.Get(backend_config.name); + if (backend == nullptr) continue; + const uint32_t backend_id = registry_.BackendId(backend); + if (backend_config.tier == TierType::SSD) { + if (!default_pool_->EnableLocalEviction(backend_id, backend_config.ssd.ssd.high_watermark, + backend_config.ssd.ssd.low_watermark)) { + MORI_UMBP_ERROR("[PoolClient] backend '{}': invalid SSD watermarks high={} low={}", + backend_config.name, backend_config.ssd.ssd.high_watermark, + backend_config.ssd.ssd.low_watermark); + Shutdown(); + return false; + } + } else if (!master_client_) { + default_pool_->EnableLocalEviction(backend_id, config_.local_evict_high_watermark, + config_.local_evict_low_watermark); + } + } + const bool weighted_placement = config_.placement_policy == PoolPlacementPolicy::WEIGHTED || config_.placement_policy == PoolPlacementPolicy::TIERED; if (master_client_) { @@ -5079,6 +5097,16 @@ void PoolClient::PublishComponentMetrics() { sample(MORI_UMBP_METRIC_CLIENT_TIER_READ_HITS, MORI_UMBP_METRIC_CLIENT_TIER_READ_HITS_HELP, {{"tier", tier}}, hits); } + + const LocalEvictionMetrics local = default_pool_->LocalEvictionStats(); + sample(MORI_UMBP_METRIC_CLIENT_LOCAL_EVICT, MORI_UMBP_METRIC_CLIENT_LOCAL_EVICT_HELP, + {{"event", "round"}}, local.rounds); + sample(MORI_UMBP_METRIC_CLIENT_LOCAL_EVICT, MORI_UMBP_METRIC_CLIENT_LOCAL_EVICT_HELP, + {{"event", "key"}}, local.keys); + sample(MORI_UMBP_METRIC_CLIENT_LOCAL_EVICT, MORI_UMBP_METRIC_CLIENT_LOCAL_EVICT_HELP, + {{"event", "no_candidate"}}, local.no_candidate); + sample(MORI_UMBP_METRIC_CLIENT_LOCAL_EVICT, MORI_UMBP_METRIC_CLIENT_LOCAL_EVICT_HELP, + {{"event", "stalled"}}, local.stalled); metric_publisher_.Publish("pool", {}, samples, sink); } } diff --git a/src/umbp/doc/design-eviction-convergence.md b/src/umbp/doc/design-eviction-convergence.md index f41b3690c..35df7cd8f 100644 --- a/src/umbp/doc/design-eviction-convergence.md +++ b/src/umbp/doc/design-eviction-convergence.md @@ -141,7 +141,10 @@ the placement (or, without one, the first owner in read order). No copy there the key (a copy-mode promotion left one there) nothing is copied and only the source is freed. 4. Otherwise, or if the demotion failed (no room downstream) → **drop** that copy - only: `Evict({key})` on that one backend. + only: `Evict({key})` on that one backend. One exception: if the demotion + failed because *another* transition started moving the key meanwhile, leave + it (0 freed). Deleting the source then could land before that transition pins + it, and the key would be lost instead of moved. 5. If the freed copy was the placement, point the placement at the remaining copy, or forget the key. @@ -184,18 +187,28 @@ so a key re-committed in that window keeps its placement. **ordering, not eviction**: ```cpp +// types.h: a key, and the capacity freeing it gives back (page-rounded where +// the medium allocates in pages). +struct EvictionOffer { std::string key; uint64_t bytes; }; + // Start keeping the order EvictionCandidates() reads. Off by default: a // master-only node never asks for candidates. virtual void TrackEvictionOrder() {} // Up to max_count keys this medium could free right now, coldest first, // skipping keys under a read lease, a pin, or an in-flight read. Frees nothing; // Evict() re-checks every key. -virtual std::vector EvictionCandidates(size_t max_count) { return {}; } +virtual std::vector EvictionCandidates(size_t max_count) { return {}; } ``` `PageBackend` keeps its LRU (touched by commit and resolve) behind `TrackEvictionOrder`; `PeerSsdManager` already keeps one unconditionally. +**Why an offer carries its size.** The worker asks for a batch of 64 but takes +only as many offers as the remaining budget needs. Without sizes it would evict +the whole batch before re-reading usage, and on a small or nearly-drained medium +one batch overshoots the low watermark by far -- the old in-medium rounds avoided +that by summing sizes while selecting, and so must this. + **Why the order comes from the medium, not the pool.** `PeerPool` has an LRU of its own, `access_order_`, but maintains it only for `watermark` offload, and turning it on disables `BatchResolve`'s single-backend fast path — which is the @@ -219,10 +232,14 @@ The first call starts one eviction worker thread. The worker: - **Orders** backends lowest tier first (SSD, then DRAM, then HBM), so a demotion from an upper tier finds room downstream. - For each backend at or above its high watermark: repeat - `EvictionCandidates(64)` → `PeerPool::Evict({key, backend_id}…, kReclaim)` — - **the same call the master's request makes** — until usage is at or below the - low watermark, the medium has no candidates, or a batch frees nothing (every + `EvictionCandidates(64)` → take offers until their bytes cover what is left of + the budget (usage minus the low watermark, fixed at the start) → + `PeerPool::Evict({key, backend_id}…, kReclaim)` — **the same call the master's + request makes** — until usage is at or below the low watermark, the budget is + covered, the medium has no candidates, or a batch frees nothing (every candidate was mid-transition or became busy). The next wake tries again. + The fixed budget is what stops a medium whose reported usage lags its deletes + from being drained to empty. Batches of 64 keep each `Evict` call short, so no single call holds the pool lock for a whole drain. diff --git a/src/umbp/include/umbp/distributed/config.h b/src/umbp/include/umbp/distributed/config.h index 351d18b24..d5ff0bc0c 100644 --- a/src/umbp/include/umbp/distributed/config.h +++ b/src/umbp/include/umbp/distributed/config.h @@ -303,13 +303,13 @@ struct PoolClientConfig { // (2 MiB by default). Set to an explicit byte count to override. uint64_t dram_page_size = 0; - // Watermarks for the paged medium's own eviction, used ONLY when this node - // has no master (PoolClient::Init enables it exactly then). With a master, - // eviction is a cluster-wide decision shipped as Evict RPCs and these are - // ignored; without one, nothing else would ever free a page. Lowered from - // UMBPDramConfig::high_watermark / low_watermark, which is where the - // deleted local backend read the same policy from. The SSD medium has its - // own pair inside UMBPSsdConfig and does not consult these. + // Watermarks for local eviction of the paged media (DRAM/HBM), used ONLY + // when this node has no master (PoolClient::Init enables PeerPool's local + // eviction for them exactly then). With a master, their eviction is a + // cluster-wide decision shipped as EvictKey RPCs and these are ignored; + // without one, nothing else would ever free a page. Lowered from + // UMBPDramConfig::high_watermark / low_watermark. The SSD medium uses its + // own pair inside UMBPSsdConfig, with or without a master. double local_evict_high_watermark = 0.9; double local_evict_low_watermark = 0.7; diff --git a/src/umbp/include/umbp/distributed/master/master_metrics.h b/src/umbp/include/umbp/distributed/master/master_metrics.h index cd4f7b2dc..448d26e51 100644 --- a/src/umbp/include/umbp/distributed/master/master_metrics.h +++ b/src/umbp/include/umbp/distributed/master/master_metrics.h @@ -181,6 +181,15 @@ #define MORI_UMBP_METRIC_CLIENT_TIER_READ_HITS_HELP \ "Reads served by each logical tier of this client's pool" +// --- Local watermark eviction ---------------------------------------------- +// The keys and bytes freed are already in the backend evict series (every +// eviction is an Evict() call there); these are what only the pool's policy +// sees. event=round|key|no_candidate|stalled -- see LocalEvictionMetrics. + +#define MORI_UMBP_METRIC_CLIENT_LOCAL_EVICT "mori_umbp_client_local_evict_total" +#define MORI_UMBP_METRIC_CLIENT_LOCAL_EVICT_HELP \ + "Local watermark eviction on this client's pool, by event" + // --- Per-client RPC call counters ------------------------------------------ // Full name: prefix + sanitized_node_id diff --git a/src/umbp/include/umbp/distributed/peer/backend/instrumented_backend.h b/src/umbp/include/umbp/distributed/peer/backend/instrumented_backend.h index 05eafbd77..1e1e36ef1 100644 --- a/src/umbp/include/umbp/distributed/peer/backend/instrumented_backend.h +++ b/src/umbp/include/umbp/distributed/peer/backend/instrumented_backend.h @@ -91,8 +91,11 @@ class InstrumentedBackend final : public MediumBackend { void ClearFullSyncAcked() override { inner_->ClearFullSyncAcked(); } bool IsClearFullSyncPending() const override { return inner_->IsClearFullSyncPending(); } void SetEventPublishing(bool enabled) override { inner_->SetEventPublishing(enabled); } - void EnableLocalEviction(double high_watermark, double low_watermark) override { - inner_->EnableLocalEviction(high_watermark, low_watermark); + // Ordering only, not an operation: nothing is freed, so nothing is counted. + // The eviction that follows is an Evict() call and is measured there. + void TrackEvictionOrder() override { inner_->TrackEvictionOrder(); } + std::vector EvictionCandidates(size_t max_count) override { + return inner_->EvictionCandidates(max_count); } void SetAutoFlushHook(size_t threshold, std::function cb) override { diff --git a/src/umbp/include/umbp/distributed/peer/backend/medium_backend.h b/src/umbp/include/umbp/distributed/peer/backend/medium_backend.h index 401d64ab1..b9cfc80f0 100644 --- a/src/umbp/include/umbp/distributed/peer/backend/medium_backend.h +++ b/src/umbp/include/umbp/distributed/peer/backend/medium_backend.h @@ -276,15 +276,27 @@ class MediumBackend : public MetricSource { // call (the default) simply has no outbox worth gating. virtual void SetEventPublishing(bool /*enabled*/) {} - // Turn on the backend's OWN eviction, for a node with no master. + // ---- eviction ordering ---- // - // Evict(keys) above is a master decision: it picks victims with a - // cluster-wide view of who else holds the key. Nothing in the system calls - // it otherwise, so a masterless node needs some medium-local policy or it - // simply fills up. A backend that already self-evicts (SsdBackend, whose - // PeerSsdManager runs its own watermark loop from UMBPSsdConfig) ignores - // this; the default is to ignore it. - virtual void EnableLocalEviction(double /*high_watermark*/, double /*low_watermark*/) {} + // A backend never decides to evict. The master (EvictKey) or the pool's own + // local watermark eviction decides when and how much, both through + // PeerPool::Evict, and Evict() below is the one place a medium frees a + // committed key. What only the medium can supply is ORDER: it sees every + // read, including the resolve paths that never touch the pool's metadata. + // So it offers candidates, and nothing else (design-eviction-convergence.md). + + // Start keeping the recency order EvictionCandidates() reads. Off by + // default: a node whose eviction is all master-driven never asks for + // candidates, and keeping the order costs every read a touch. Idempotent, + // and safe after Init: keys already owned are seeded into the order. + virtual void TrackEvictionOrder() {} + + // Up to `max_count` keys this medium could free right now, coldest first, + // each with the capacity freeing it gives back, skipping keys that a read + // lease, pin or in-flight read protects. Frees nothing: the caller passes + // the keys back through Evict(), which re-checks every one. A medium that + // keeps no order returns nothing and is therefore never evicted locally. + virtual std::vector EvictionCandidates(size_t /*max_count*/) { return {}; } // ---- observability ---- // @@ -363,9 +375,12 @@ class MediumBackend : public MetricSource { return out; } - // Master-driven eviction. Idempotent; see EvictResult::bytes_freed. One - // result per key, in request order — the peer service relies on that to sum - // freed bytes for a key mirrored across media. + // The medium's ONE eviction implementation: free these keys' bytes on this + // medium, skipping any a read lease or pin protects, and queue a REMOVE for + // each key actually freed. Every eviction reaches here through + // PeerPool::Evict -- master-driven or local -- and nothing else in a backend + // may free a committed key for capacity. Idempotent; see + // EvictResult::bytes_freed. One result per key, in request order. virtual std::vector Evict(const std::vector& keys) = 0; // ---- bootstrap (GetPeerInfo) ---- diff --git a/src/umbp/include/umbp/distributed/peer/backend/page_backend.h b/src/umbp/include/umbp/distributed/peer/backend/page_backend.h index f337f66fb..a3a61d925 100644 --- a/src/umbp/include/umbp/distributed/peer/backend/page_backend.h +++ b/src/umbp/include/umbp/distributed/peer/backend/page_backend.h @@ -331,38 +331,22 @@ class PageBackend : public MediumBackend { // keeping one is a leak of one event per put. void SetEventPublishing(bool enabled) override; - // ==================== Local (masterless) eviction ==================== + // ==================== Eviction ordering ==================== // - // Eviction here is normally the MASTER's policy: it chooses victims with a - // cluster-wide view and calls Evict(keys) through the peer service. Nothing - // else ever calls Evict, so a node with no master fills its pool and - // Allocate starts answering NO_SPACE forever — where the deleted local - // backend's LocalStorageManager would have evicted on a watermark. This is - // that watermark loop, restored on the backend that now serves embedded - // deployments. + // This backend never decides to evict. The master, or the pool's local + // watermark eviction on a node without one, decides and calls Evict() -- + // the only code here that frees a committed key -- through PeerPool::Evict. + // What this backend supplies is the order: an LRU of owned keys, inserted by + // Commit and renewed by Resolve. // - // Opt-in so the cluster path is bit-for-bit unchanged: with a master, a - // second local policy would free keys the master still indexes and hands out - // routes for. PoolClient::Init enables it exactly when it builds no - // MasterClient. - // - // Enabling it turns on an LRU ordering (inserted by Commit, renewed by - // Resolve) that is not maintained otherwise, so the read path pays nothing - // in a cluster. A round runs on the committing thread — there is no - // eviction thread — and frees oldest-first down to `low` once usage reaches - // `high`. Leased and copy-pinned keys are skipped, exactly as master-driven - // Evict skips them; a round that can only find protected keys frees what it - // can and stops rather than spinning. - // - // Must be called before Init(). Watermarks must satisfy - // 0 < low < high <= 1; anything else is rejected with a warning and leaves - // local eviction off. - void EnableLocalEviction(double high_watermark, double low_watermark) override; + // The LRU is maintained only once TrackEvictionOrder() is called, so a node + // whose eviction is all master-driven pays nothing for it on the read path. + void TrackEvictionOrder() override; - // Test seam: run one local-eviction round synchronously. Returns the number - // of keys freed (0 when local eviction is off or usage is below the high - // watermark). - size_t RunLocalEvictionOnceForTest() { return MaybeEvictToLowWatermark(); } + // Coldest first, skipping keys under a read lease or a copy pin -- the same + // two protections Evict() applies, so a candidate is one Evict() would free + // unless it is touched in between. Empty until TrackEvictionOrder(). + std::vector EvictionCandidates(size_t max_count) override; // ==================== Reaper ==================== @@ -398,7 +382,7 @@ class PageBackend : public MediumBackend { // (RenewLeaseAtomic) so a slower thread can never shorten a lease another // thread already extended. std::atomic read_lease_ns{0}; - // Position in lru_, valid only while in_lru is true (local eviction on). + // Position in lru_, valid only while in_lru is true (order tracked). // Held here for the same reason as the lease: the alternative is a second // map keyed by the same string. // @@ -487,10 +471,10 @@ class PageBackend : public MediumBackend { void ReaperLoop(); void ReaperSweep(); - // ---- local eviction (no-master deployments; see EnableLocalEviction) ---- + // ---- eviction order (see TrackEvictionOrder) ---- // Caller MUST hold `lru_mutex_`. Move `key` to the MRU end, inserting it if - // it is not yet tracked. No-op when local eviction is off. + // it is not yet tracked. No-op until the order is tracked. void TouchLruLocked(const std::string& key, OwnedSlot& slot); // Caller MUST hold `lru_mutex_`. Drop `key` from the LRU if tracked. @@ -501,11 +485,6 @@ class PageBackend : public MediumBackend { void TouchLru(const std::string& key, OwnedSlot& slot); void RemoveFromLru(OwnedSlot& slot); - // Run one round if local eviction is on and usage has reached the high - // watermark. Takes `mutex_` itself, so callers must NOT hold it. Returns - // the number of keys freed. - size_t MaybeEvictToLowWatermark(); - // Owned pages held back at ClearLocal() because of an active read lease. // Released by ReaperSweep() when release_at <= now. struct DeferredFree { @@ -564,25 +543,18 @@ class PageBackend : public MediumBackend { std::vector deferred_frees_; - // ---- local eviction state (see EnableLocalEviction) ---- + // ---- eviction order (see TrackEvictionOrder) ---- // Front = most recently committed/resolved, back = eviction candidate. - // Empty and unmaintained unless local_evict_enabled_. + // Empty and unmaintained until track_eviction_order_. std::list lru_; // Guards lru_ and the per-slot lru_it/in_lru fields. Separate from mutex_ // so a resolve can record an access while holding only a shared lock; the // critical section is a list splice, not a hash lookup plus a page-vector // copy. Always taken after mutex_ when both are held. mutable std::mutex lru_mutex_; - bool local_evict_enabled_ = false; - double local_evict_high_wm_ = 0.9; - double local_evict_low_wm_ = 0.7; - // Serializes rounds without holding mutex_ across one: a second committing - // thread backs off instead of over-evicting (same shape as PeerSsdManager). - std::mutex local_evict_round_mutex_; - // Latches the "nothing reclaimable" warning so a wedged pool reports once - // per episode rather than once per commit. Cleared by the next round that - // frees something. - bool local_evict_warned_ = false; + // Written under mutex_ held exclusively, read under it held shared or + // exclusively -- every LRU touch already holds one or the other. + bool track_eviction_order_ = false; // False stops QueueEventLocked from recording anything (no master to ship // to). True is the cluster default. diff --git a/src/umbp/include/umbp/distributed/peer/backend/ssd_backend.h b/src/umbp/include/umbp/distributed/peer/backend/ssd_backend.h index 75c3e7fd1..35adb13b2 100644 --- a/src/umbp/include/umbp/distributed/peer/backend/ssd_backend.h +++ b/src/umbp/include/umbp/distributed/peer/backend/ssd_backend.h @@ -167,6 +167,10 @@ class SsdBackend : public MediumBackend { void ReleaseMigrationRead(const std::string& key) override; bool Contains(const std::string& key) const override; std::vector Evict(const std::vector& keys) override; + // The manager's LRU, minus keys a staged read lease or migration read holds + // here -- Evict() would refuse those. The manager keeps its order always, so + // TrackEvictionOrder() needs nothing (the default). + std::vector EvictionCandidates(size_t max_count) override; void SetAutoFlushHook(size_t threshold, std::function cb) override; diff --git a/src/umbp/include/umbp/distributed/peer/ssd/peer_ssd_manager.h b/src/umbp/include/umbp/distributed/peer/ssd/peer_ssd_manager.h index 3810c4d25..b10116c37 100644 --- a/src/umbp/include/umbp/distributed/peer/ssd/peer_ssd_manager.h +++ b/src/umbp/include/umbp/distributed/peer/ssd/peer_ssd_manager.h @@ -52,9 +52,14 @@ struct SsdReadOutcome { }; // Peer-side owner of the local SSD tier: one SSD TierBackend + the -// key->SSD-location map, capacity, the owned-location event outbox, and the -// read-prepare and local-eviction paths. Reached from the distributed data -// plane through SsdBackend : MediumBackend, which forwards to it. +// key->SSD-location map, capacity, the owned-location event outbox, the +// read-prepare path, and the SSD medium's one eviction implementation +// (Evict). Reached from the distributed data plane through +// SsdBackend : MediumBackend, which forwards to it. +// +// It decides nothing about eviction. When to evict and how much is +// PeerPool's local watermark eviction (or the master), which reaches Evict() +// through SsdBackend::Evict; this class only offers its LRU as candidates. // // It reuses ONLY the low-level TierBackend (SSDTier). Peer DRAM is owned by // PageBackend, and a second thing here that also held DRAM and demoted between @@ -64,11 +69,10 @@ class PeerSsdManager { public: explicit PeerSsdManager(const PeerSsdConfig& cfg); - // Test-only: inject a ready-made backend and explicit watermarks so unit - // tests can drive eviction with a controllable (e.g. blocking) fake backend. - // Production code must use the config constructor. - PeerSsdManager(std::unique_ptr backend, double high_watermark, double low_watermark, - bool single_flight = true); + // Test-only: inject a ready-made backend so unit tests can drive the manager + // with a controllable (e.g. blocking) fake. Production code must use the + // config constructor. + explicit PeerSsdManager(std::unique_ptr backend, bool single_flight = true); ~PeerSsdManager(); @@ -86,8 +90,8 @@ class PeerSsdManager { // `owned_` has carried this all along; only the accessor was missing. uint64_t SizeOf(const std::string& key) const; - // Hold a key against local watermark eviction across a peer-local migration. - // Uses the same refcount as active reads so eviction has one protection rule. + // Hold a key against eviction across a peer-local migration. Uses the same + // refcount as active reads so eviction has one protection rule. bool PinForMigration(const std::string& key); void UnpinForMigration(const std::string& key); @@ -111,44 +115,38 @@ class PeerSsdManager { // drive but one. // // Result length and order match @p keys. An already-owned key reports true - // without any device IO (LRU refreshed). On a partial failure the failed keys - // are retried once after a single eviction round, so a batch that trips the - // high watermark mid-way still lands. + // without any device IO (LRU refreshed). A key the device has no room for + // fails: this never evicts to make room -- that is the pool's local + // eviction, on its own thread, at the SSD's watermark. std::vector WriteBatch(const std::vector& keys, const std::vector& srcs, const std::vector& sizes); - // Local eviction of a single key. Read priority: a key with an - // in-flight PrepareRead (inflight_reads_ > 0) is NOT evicted (returns false). + // The SSD medium's ONE eviction implementation: free a single key. Read + // priority: a key with an in-flight PrepareRead or migration pin + // (inflight_reads_ > 0) is NOT evicted (returns false). // Concurrency: marks the key in evicting_ under the lock, runs the backend // evict outside the lock, and only on backend success removes owned_/lru_ and // queues a REMOVE SSD event (so REMOVE is never emitted while the bytes still - // exist, and two workers cannot double-evict the same victim). + // exist, and two callers cannot double-evict the same key). // - // Does NOT itself hold eviction_mu_ (EvictToLowWatermark already holds it - // while looping over victims, so taking it here would deadlock). The normal - // caller is EvictToLowWatermark; a direct caller must not run concurrently - // with ClearLocal (production relies on PoolClient::Clear quiescing the copy - // pipeline first, so no eviction is in flight during a Clear). + // Reached only through SsdBackend::Evict, which PeerPool::Evict calls under + // its lifecycle lock -- the one ClearLocal takes exclusively -- so an + // eviction is never in flight during a Clear. bool Evict(const std::string& key); - // Local LRU victim selection (oldest first), skipping keys that are - // being read (inflight_reads_ > 0) or already being evicted (evicting_). - // Accumulates sizes until >= bytes_to_free; returns fewer if not enough - // free-able keys exist (never blocks). - // - // Not pluggable yet (only master-side eviction is). A real SSD plugin must - // own the per-algorithm state that lives here in lru_/owned_, not just the - // selection step; target design is a stateful policy with - // OnAdd/OnTouch/OnRemove/Clear hooks plus a SelectVictims called under mutex_. - std::vector SelectVictims(size_t bytes_to_free); + // Up to @p max_count keys, coldest first (LRU tail -> MRU front), skipping + // keys that are being read or pinned (inflight_reads_) or already being + // evicted (evicting_). Ordering only: frees nothing, and Evict() re-checks + // every key. Never blocks on IO. + std::vector EvictionCandidates(size_t max_count); // Distributed Clear: drop the logical owned-location map + undrained events, // then delete the physical SSD bytes (a user Clear means the cache is no // longer wanted). Read priority: clears the logical map first (so new reads // immediately miss with kNotFound), waits for any in-flight PrepareRead to // finish (SSD reads cannot be safely aborted), and only then wipes the - // backend. Serializes against eviction rounds via eviction_mu_. + // backend. No eviction runs concurrently: see Evict. // Precondition: callers (PoolClient::Clear) MUST quiesce the SSD copy // pipeline first so no in-flight copy re-populates owned_ right after this // returns. Crash-restart leftover (metadata gone, files remain) is a @@ -217,7 +215,6 @@ class PeerSsdManager { // Byte counters for SSD IO bandwidth (rate() in Grafana = bytes/s). uint64_t CopyBytes() const { return metrics_.copy_bytes.load(std::memory_order_relaxed); } uint64_t ReadBytes() const { return metrics_.read_bytes.load(std::memory_order_relaxed); } - uint64_t EvictionRounds() const { return metrics_.evict_rounds.load(std::memory_order_relaxed); } uint64_t EvictionVictims() const { return metrics_.evict_victims.load(std::memory_order_relaxed); } @@ -243,19 +240,8 @@ class PeerSsdManager { // Build the full ADD list for every owned SSD key. Caller MUST hold mutex_. std::vector SnapshotOwnedKeysLocked() const; - // Evict oldest keys until used <= low_watermark * total. Runs the backend - // IO outside mutex_ (via Evict); serialized by eviction_mu_ so concurrent - // copy workers do not run overlapping eviction rounds (and never over-evict). - void EvictToLowWatermark(); - - // Serializes eviction rounds (EvictToLowWatermark) and excludes ClearLocal. - // Always acquired BEFORE mutex_ to keep a single lock order. - std::mutex eviction_mu_; - mutable std::mutex mutex_; std::unique_ptr backend_; // null when cfg.enabled == false - double high_watermark_ = 0.9; - double low_watermark_ = 0.7; // key -> {size, lru position}. The authoritative owned-location map. std::unordered_map owned_; @@ -311,7 +297,7 @@ class PeerSsdManager { // inflight_reads_: key -> InflightRead (entry exists only while refs > 0). // Eviction skips keys with a live read. // evicting_: keys currently inside Evict's backend-evict window; new reads - // of these miss (kNotFound) and SelectVictims skips them. + // of these miss (kNotFound) and EvictionCandidates skips them. std::unordered_map> inflight_reads_; std::unordered_set evicting_; std::condition_variable reads_drained_cv_; // notified when inflight_reads_ empties @@ -348,7 +334,6 @@ class PeerSsdManager { std::atomic read_merged{0}; std::atomic copy_bytes{0}; // bytes written to SSD (write IO) std::atomic read_bytes{0}; // bytes read from SSD (read IO) - std::atomic evict_rounds{0}; std::atomic evict_victims{0}; std::atomic evict_bytes_freed{0}; std::atomic evict_backend_failures{0}; diff --git a/src/umbp/include/umbp/distributed/pool/peer_pool.h b/src/umbp/include/umbp/distributed/pool/peer_pool.h index 22a52fed7..25dd41ac8 100644 --- a/src/umbp/include/umbp/distributed/pool/peer_pool.h +++ b/src/umbp/include/umbp/distributed/pool/peer_pool.h @@ -24,6 +24,7 @@ // MIT License #pragma once +#include #include #include #include @@ -71,6 +72,23 @@ struct PoolResolvedEntry { ResolvedEntry resolved; }; +// Local watermark eviction counters (see PeerPool::EnableLocalEviction). The +// keys and bytes actually freed are not here: every eviction is an Evict() +// call on a backend, and InstrumentedBackend already counts those. These are +// what only the policy can see. +struct LocalEvictionMetrics { + // Passes that found a backend at or above its high watermark. + uint64_t rounds = 0; + // Keys the passes freed (demoted or dropped). + uint64_t keys = 0; + // Passes the medium offered no candidate for: everything it holds is under + // a read lease or pin, or it keeps no eviction order at all. + uint64_t no_candidate = 0; + // Batches in which every candidate freed nothing (mid-transition, or busy by + // the time Evict reached it). The pass stops and the next wake retries. + uint64_t stalled = 0; +}; + // Why a key is being removed. A tiered pool answers the two intents // differently, so the caller has to say which one it means instead of leaving // it to the tier configuration. @@ -143,6 +161,24 @@ class PeerPool { // a tier policy is actually keeping hot data where it was supposed to. std::map TierReadHits() const; + // ---- local watermark eviction ---- + // + // Turn on watermark eviction for one backend: once its usage reaches `high`, + // the pool frees its coldest keys (MediumBackend::EvictionCandidates) down to + // `low` by calling Evict() above -- the same call the master's EvictKey + // makes, so a key whose tier has a downstream is demoted, not deleted. It + // runs on one background thread, started by the first call; a put only wakes + // it and never evicts itself. PoolClient enables it for every backend on a + // node with no master, and for SSD always. + // + // Returns false and changes nothing for an unknown backend or watermarks + // outside 0 < low < high <= 1. + bool EnableLocalEviction(uint32_t backend_id, double high_watermark, double low_watermark); + LocalEvictionMetrics LocalEvictionStats() const; + // Test seam: one synchronous pass over every enabled backend, as the worker + // runs it. Returns the number of keys freed. + size_t RunLocalEvictionOnceForTest() { return LocalEvictionPass(); } + private: struct PendingPlacement { uint32_t backend_id = BackendRegistry::kMaxBackends; @@ -168,6 +204,22 @@ class PeerPool { void EnqueueTransition(TransitionJob job); void TransitionWorkerLoop(); void StopTransitionWorker(); + + struct LocalEvictionWatermarks { + double high = 0.0; + double low = 0.0; + }; + // Cheap enough for the put path: one atomic exchange, and a notify only when + // no wake is already pending. Never reads capacity. + void WakeLocalEviction(); + void LocalEvictionLoop(); + void StopLocalEviction(); + // One pass: every enabled backend, lowest tier first. Returns keys freed. + size_t LocalEvictionPass(); + // Frees one backend down to its low watermark if it is at or above its high + // one. Returns keys freed. + size_t ReclaimBackend(uint32_t backend_id, const LocalEvictionWatermarks& watermarks); + void TouchLocked(const std::string& key); void ForgetAccessLocked(const std::string& key); // Whether this read should promote the key out of `source_tier`, per that @@ -259,6 +311,23 @@ class PeerPool { std::vector queued_by_tier_; bool stop_transition_worker_ = false; std::thread transition_worker_; + + // ---- local watermark eviction (see EnableLocalEviction) ---- + // Guards local_evict_, stop_local_evict_ and the worker's start. + mutable std::mutex local_evict_mutex_; + std::condition_variable local_evict_cv_; + std::map local_evict_; + bool stop_local_evict_ = false; + // Set by a wake, cleared by the worker at the start of a pass. Atomic so a + // commit that finds a wake already pending touches no lock at all. + std::atomic local_evict_wake_{false}; + std::thread local_evict_worker_; + // One pass at a time: the worker and the test seam share LocalEvictionPass. + std::mutex local_evict_pass_mutex_; + std::atomic local_evict_rounds_{0}; + std::atomic local_evict_keys_{0}; + std::atomic local_evict_no_candidate_{0}; + std::atomic local_evict_stalled_{0}; }; } // namespace mori::umbp diff --git a/src/umbp/include/umbp/distributed/types.h b/src/umbp/include/umbp/distributed/types.h index 9b6f532d2..01f351c67 100644 --- a/src/umbp/include/umbp/distributed/types.h +++ b/src/umbp/include/umbp/distributed/types.h @@ -173,6 +173,20 @@ struct EvictionVictim { } }; +// One key a medium offers for local eviction (MediumBackend:: +// EvictionCandidates), with the bytes freeing it gives back to that medium's +// capacity -- page-rounded where the medium allocates in pages. The size is +// what lets the caller take exactly as many offers as it needs instead of a +// whole batch. +struct EvictionOffer { + std::string key; + uint64_t bytes = 0; + + bool operator==(const EvictionOffer& other) const { + return key == other.key && bytes == other.bytes; + } +}; + // Ordering hint for IMasterMetadataStore::EnumerateEvictionCandidates. This is // a performance affordance, NOT eviction policy: it only tells the store what // order to return rows in so a backend with an index (e.g. a Redis ZSET keyed diff --git a/tests/cpp/umbp/distributed/test_peer_pool.cpp b/tests/cpp/umbp/distributed/test_peer_pool.cpp index e0aa1d5b8..7ccbf5089 100644 --- a/tests/cpp/umbp/distributed/test_peer_pool.cpp +++ b/tests/cpp/umbp/distributed/test_peer_pool.cpp @@ -27,6 +27,7 @@ #include #include #include +#include #include #include #include @@ -1091,5 +1092,172 @@ TEST(PeerPool, ReadPromotesWithMoveDrainsSourceAndKeepsBytes) { payload); } +// --------------------------------------------------------------------------- +// Local watermark eviction +// +// The pool's own policy on a node that evicts for itself. It runs on a +// background thread, so these observe rather than await; and it goes through +// Evict() -- the master's entry point -- so what it frees and how (demote vs +// drop) is the same contract the tests above pin down. +// --------------------------------------------------------------------------- + +bool WaitUntil(const std::function& done, + std::chrono::milliseconds budget = std::chrono::seconds(5)) { + const auto deadline = std::chrono::steady_clock::now() + budget; + while (std::chrono::steady_clock::now() < deadline) { + if (done()) return true; + std::this_thread::sleep_for(std::chrono::milliseconds(2)); + } + return done(); +} + +// A buffer is sized by its mapping, so a tier is exactly N pages long only when +// the page is the host page. Every key below is one page. +constexpr uint64_t kHostPage = 4096; + +TEST(PeerPoolLocalEviction, DrainsAFullBackendToItsLowWatermark) { + LocalCopyEngine engine; + BackendRegistry registry; + RegisterPageBackend(®istry, &engine, "dram", TierType::DRAM, kHostPage, 10); + PeerPool pool(®istry, MakeSingleBackendPolicy(), &engine); + ASSERT_TRUE(pool.EnableLocalEviction(registry.BackendId(registry.Get("dram")), 0.9, 0.5)); + + for (int i = 0; i < 9; ++i) CommitKey(&pool, "k" + std::to_string(i)); + // 9 of 10 pages is the high watermark; the worker frees down to 5, and no + // further -- the offers carry their size, so a batch cannot overshoot. + auto* dram = registry.Get("dram"); + ASSERT_TRUE(WaitUntil([&] { return dram->OwnedKeyCount() <= 5; })); + std::this_thread::sleep_for(std::chrono::milliseconds(20)); + EXPECT_EQ(dram->OwnedKeyCount(), 5u); + // Coldest first. + for (int i = 0; i < 4; ++i) EXPECT_FALSE(dram->Contains("k" + std::to_string(i))) << i; + for (int i = 4; i < 9; ++i) EXPECT_TRUE(dram->Contains("k" + std::to_string(i))) << i; + EXPECT_EQ(pool.PlacementCount(), 5u); + const auto stats = pool.LocalEvictionStats(); + EXPECT_GE(stats.rounds, 1u); + EXPECT_EQ(stats.keys, 4u); +} + +TEST(PeerPoolLocalEviction, ReadsKeepAKeyWarm) { + LocalCopyEngine engine; + BackendRegistry registry; + // A lease short enough to have lapsed before the worker looks. + RegisterPageBackend(®istry, &engine, "dram", TierType::DRAM, kHostPage, 10, + std::chrono::milliseconds(1)); + PeerPool pool(®istry, MakeSingleBackendPolicy(), &engine); + // Before any traffic, as PoolClient does: the medium keeps its order from the + // first commit. 8 of 10 pages stays under the high watermark. + ASSERT_TRUE(pool.EnableLocalEviction(registry.BackendId(registry.Get("dram")), 0.9, 0.5)); + + for (int i = 0; i < 8; ++i) CommitKey(&pool, "k" + std::to_string(i)); + // The oldest key is read, which renews its recency in the medium's order. + ASSERT_TRUE(pool.BatchResolve({"k0"}, false).front().resolved.found); + std::this_thread::sleep_for(std::chrono::milliseconds(5)); + CommitKey(&pool, "k8"); + + auto* dram = registry.Get("dram"); + ASSERT_TRUE(WaitUntil([&] { return dram->OwnedKeyCount() <= 5; })); + EXPECT_TRUE(dram->Contains("k0")) << "a just-read key was evicted ahead of colder ones"; + EXPECT_FALSE(dram->Contains("k1")); +} + +TEST(PeerPoolLocalEviction, KeysUnderAReadLeaseAreNeverOffered) { + LocalCopyEngine engine; + BackendRegistry registry; + RegisterPageBackend(®istry, &engine, "dram", TierType::DRAM, kHostPage, 10); + PeerPool pool(®istry, MakeSingleBackendPolicy(), &engine); + std::vector keys; + for (int i = 0; i < 9; ++i) { + keys.push_back("k" + std::to_string(i)); + CommitKey(&pool, keys.back()); + } + // Every key is being read (the default lease outlives the test). + for (const auto& entry : pool.BatchResolve(keys, false)) ASSERT_TRUE(entry.resolved.found); + + ASSERT_TRUE(pool.EnableLocalEviction(registry.BackendId(registry.Get("dram")), 0.9, 0.5)); + EXPECT_EQ(pool.RunLocalEvictionOnceForTest(), 0u); + EXPECT_EQ(registry.Get("dram")->OwnedKeyCount(), 9u); + EXPECT_GE(pool.LocalEvictionStats().no_candidate, 1u); +} + +TEST(PeerPoolLocalEviction, ATierWithADownstreamDemotesInsteadOfDropping) { + // The same call the master makes, so the same answer: a key whose tier has + // somewhere to go is moved there, not deleted. + LocalCopyEngine engine; + BackendRegistry registry; + RegisterPageBackend(®istry, &engine, "hot", TierType::DRAM, kHostPage, 10); + RegisterPageBackend(®istry, &engine, "cold", TierType::SSD, kHostPage, 20); + auto compiled = LogicalTierGraph::Compile(HotColdTiers(PoolOffloadTrigger::kOnEvict), registry); + ASSERT_TRUE(compiled.ok()) << compiled.error; + PeerPool pool(®istry, MakeTieredPlacementPolicy(compiled.graph), &engine); + ASSERT_TRUE(pool.EnableLocalEviction(registry.BackendId(registry.Get("hot")), 0.9, 0.5)); + + for (int i = 0; i < 9; ++i) CommitKey(&pool, "k" + std::to_string(i)); + auto* hot = registry.Get("hot"); + auto* cold = registry.Get("cold"); + ASSERT_TRUE(WaitUntil([&] { return hot->OwnedKeyCount() <= 5; })); + for (int i = 0; i < 4; ++i) { + const std::string key = "k" + std::to_string(i); + EXPECT_FALSE(hot->Contains(key)) << key; + EXPECT_TRUE(cold->Contains(key)) << key << " was dropped, not demoted"; + EXPECT_TRUE(pool.BatchResolve({key}, false).front().resolved.found) << key; + } +} + +TEST(PeerPoolLocalEviction, NeverDeletesWhatAWatermarkOffloadWouldMove) { + // The masterless failure this replaces: the backend's own round ran inside + // the commit, emptied the tier below the pool's watermark check, and so + // deleted the cold keys the watermark offload was about to move. Now both + // the offload and local eviction move them, so every key survives somewhere. + LocalCopyEngine engine; + BackendRegistry registry; + RegisterPageBackend(®istry, &engine, "hot", TierType::DRAM, kHostPage, 10); + RegisterPageBackend(®istry, &engine, "cold", TierType::SSD, kHostPage, 40); + auto tiers = HotColdTiers(PoolOffloadTrigger::kWatermark); + tiers.front().high_watermark = 0.9; + tiers.front().low_watermark = 0.5; + auto compiled = LogicalTierGraph::Compile(tiers, registry); + ASSERT_TRUE(compiled.ok()) << compiled.error; + PeerPool pool(®istry, MakeTieredPlacementPolicy(compiled.graph), &engine); + // Deliberately the SAME watermarks as the offload: the case that used to lose. + ASSERT_TRUE(pool.EnableLocalEviction(registry.BackendId(registry.Get("hot")), 0.9, 0.5)); + + std::vector keys; + for (int i = 0; i < 20; ++i) { + keys.push_back("k" + std::to_string(i)); + CommitKey(&pool, keys.back()); + // Let the background workers keep up, as a real put rate would. + ASSERT_TRUE(WaitUntil([&] { return registry.Get("hot")->OwnedKeyCount() < 9; })); + } + for (const auto& key : keys) { + EXPECT_TRUE(registry.Get("hot")->Contains(key) || registry.Get("cold")->Contains(key)) + << key << " was deleted although the cold tier had room for it"; + } +} + +TEST(PeerPoolLocalEviction, RejectsBadWatermarksAndUnknownBackends) { + BackendRegistry registry; + ASSERT_TRUE(registry.Register("dram", std::make_unique(TierType::DRAM))); + PeerPool pool(®istry, MakeSingleBackendPolicy()); + const uint32_t id = registry.BackendId(registry.Get("dram")); + EXPECT_FALSE(pool.EnableLocalEviction(id, 0.5, 0.7)); // low above high + EXPECT_FALSE(pool.EnableLocalEviction(id, 1.5, 0.7)); // high above 1 + EXPECT_FALSE(pool.EnableLocalEviction(id, 0.9, 0.0)); // low not positive + EXPECT_FALSE(pool.EnableLocalEviction(BackendRegistry::kMaxBackends - 1, 0.9, 0.7)); + EXPECT_TRUE(pool.EnableLocalEviction(id, 0.9, 0.7)); +} + +TEST(PeerPoolLocalEviction, AMediumThatKeepsNoOrderIsNeverEvicted) { + // MockBackend offers no candidates (the MediumBackend default). Local + // eviction must leave it alone rather than guess at victims. + BackendRegistry registry; + ASSERT_TRUE(registry.Register("dram", std::make_unique(TierType::DRAM))); + PeerPool pool(®istry, MakeSingleBackendPolicy()); + CommitKey(&pool, "k"); + ASSERT_TRUE(pool.EnableLocalEviction(registry.BackendId(registry.Get("dram")), 0.9, 0.5)); + pool.RunLocalEvictionOnceForTest(); + EXPECT_TRUE(registry.Get("dram")->Contains("k")); +} + } // namespace } // namespace mori::umbp diff --git a/tests/cpp/umbp/distributed/test_peer_ssd_eviction.cpp b/tests/cpp/umbp/distributed/test_peer_ssd_eviction.cpp index be19c8f93..a5e8005b2 100644 --- a/tests/cpp/umbp/distributed/test_peer_ssd_eviction.cpp +++ b/tests/cpp/umbp/distributed/test_peer_ssd_eviction.cpp @@ -20,11 +20,15 @@ // OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE // SOFTWARE. // -// SSD local capacity management + eviction. Drives PeerSsdManager -// through a controllable in-memory TierBackend (the test-only constructor) so -// LRU ordering, watermark eviction, the in-flight-read guard, idempotent Write, +// The SSD medium's eviction primitive and its candidate order. Drives +// PeerSsdManager through a controllable in-memory TierBackend (the test-only +// constructor) so LRU ordering, the in-flight-read guard, idempotent Write, // backend-evict failure, concurrent eviction, and physical Clear are all // deterministic without real disk IO. +// +// What is deliberately NOT here: watermark eviction. The manager never decides +// to evict -- not after a write, not when a write runs out of room. That is +// PeerPool's local eviction, tested in test_peer_pool against an SsdBackend. #include #include @@ -158,10 +162,10 @@ struct Harness { std::unique_ptr mgr; }; -Harness MakeHarness(size_t capacity, double high = 0.9, double low = 0.7) { +Harness MakeHarness(size_t capacity) { auto be = std::make_unique(capacity); FakeBackend* raw = be.get(); - return Harness{raw, std::make_unique(std::move(be), high, low)}; + return Harness{raw, std::make_unique(std::move(be))}; } int CountKind(const std::vector& events, KvEvent::Kind kind) { @@ -172,6 +176,12 @@ int CountKind(const std::vector& events, KvEvent::Kind kind) { return n; } +std::vector Keys(const std::vector& offers) { + std::vector keys; + for (const auto& offer : offers) keys.push_back(offer.key); + return keys; +} + bool HasRemove(const std::vector& events, const std::string& key) { for (const auto& e : events) { if (e.kind == KvEvent::Kind::REMOVE && e.tier == TierType::SSD && e.key == key) return true; @@ -188,62 +198,66 @@ TEST(PeerSsdEviction, WriteAndPrepareReadRefreshLru) { ASSERT_TRUE(h.mgr->Write("C", OneSeg("cccc"), 4)); // LRU now (oldest->newest): A, B, C. Reading A promotes it to MRU, so the - // oldest becomes B and SelectVictims must pick B first (not the just-read A). + // oldest becomes B and the first candidate must be B (not the just-read A). std::vector buf(4); auto out = h.mgr->PrepareRead("A", buf.data(), buf.size()); ASSERT_EQ(out.status, SsdReadStatus::kOk); EXPECT_EQ(std::string(buf.data(), out.size), "aaaa"); - auto victims = h.mgr->SelectVictims(/*bytes_to_free=*/1); - ASSERT_FALSE(victims.empty()); - EXPECT_EQ(victims.front(), "B"); - EXPECT_NE(victims.front(), "A"); + auto candidates = h.mgr->EvictionCandidates(/*max_count=*/1); + ASSERT_EQ(candidates.size(), 1u); + EXPECT_EQ(candidates.front().key, "B"); + EXPECT_EQ(candidates.front().bytes, 4u); +} + +TEST(PeerSsdEviction, CandidatesAreColdestFirstAndCapped) { + auto h = MakeHarness(/*capacity=*/1'000'000); + for (int i = 1; i <= 5; ++i) { + ASSERT_TRUE(h.mgr->Write("k" + std::to_string(i), OneSeg("data"), 4)); + } + EXPECT_EQ(Keys(h.mgr->EvictionCandidates(3)), (std::vector{"k1", "k2", "k3"})); + EXPECT_EQ(h.mgr->EvictionCandidates(100).size(), 5u); + // Ordering only: asking freed nothing. + for (int i = 1; i <= 5; ++i) EXPECT_TRUE(h.mgr->Exists("k" + std::to_string(i))); } -TEST(PeerSsdEviction, WatermarkTriggersEvictionDownToLow) { - // capacity 1000, high 0.9 (=>900), low 0.7 (=>700); 100-byte values. - auto h = MakeHarness(/*capacity=*/1000, /*high=*/0.9, /*low=*/0.7); +TEST(PeerSsdEviction, WritesNeverEvictEvenWhenNearlyFull) { + // 900 of 1000 bytes used is where the manager's own round used to fire. It + // has none now: the pool's local eviction decides, on its own thread, so a + // write must leave every earlier key exactly where it was. + auto h = MakeHarness(/*capacity=*/1000); std::string val(100, 'x'); for (int i = 1; i <= 9; ++i) { ASSERT_TRUE(h.mgr->Write("k" + std::to_string(i), OneSeg(val), val.size())); } - // After k9: used hit 900 >= high -> evict oldest down to <= 700. auto [used, total] = h.mgr->Capacity(); + EXPECT_EQ(used, 900u); EXPECT_EQ(total, 1000u); - EXPECT_LE(used, 700u); - - // Oldest (k1, k2) evicted first; newest still present. - EXPECT_FALSE(h.mgr->Exists("k1")); - EXPECT_FALSE(h.mgr->Exists("k2")); - EXPECT_TRUE(h.mgr->Exists("k9")); - - auto events = h.mgr->DrainPendingEvents(); - EXPECT_TRUE(HasRemove(events, "k1")); - EXPECT_TRUE(HasRemove(events, "k2")); - EXPECT_EQ(CountKind(events, KvEvent::Kind::REMOVE), 2); -} - -TEST(PeerSsdEviction, SuccessfulWriteDoesNotEvictItself) { - auto h = MakeHarness(/*capacity=*/100, /*high=*/0.9, /*low=*/0.7); - std::string value(90, 'x'); - ASSERT_TRUE(h.mgr->Write("only", OneSeg(value), value.size())); - EXPECT_TRUE(h.mgr->Exists("only")); + for (int i = 1; i <= 9; ++i) EXPECT_TRUE(h.mgr->Exists("k" + std::to_string(i))); + EXPECT_EQ(CountKind(h.mgr->DrainPendingEvents(), KvEvent::Kind::REMOVE), 0); } -TEST(PeerSsdEviction, EnospcTriggersEvictThenRetry) { - // Fill to 800/1000 (below the 0.9 high watermark, so no watermark eviction - // fires during the fill), then write a 300-byte value that overflows the - // device: backend Write -> ENOSPC -> one evict round (frees the oldest down - // to the 0.5 low watermark) -> retry succeeds. After the retry used is 800, - // still below high, so no second round disturbs the just-written key. - auto h = MakeHarness(/*capacity=*/1000, /*high=*/0.9, /*low=*/0.5); +TEST(PeerSsdEviction, WriteThatDoesNotFitFailsWithoutEvicting) { + // 800 of 1000 used, then a 300-byte value: the device answers ENOSPC. The + // write fails and records nothing; it does not evict older keys to make room + // -- that would be eviction work on the put path. + auto h = MakeHarness(/*capacity=*/1000); for (int i = 1; i <= 8; ++i) { ASSERT_TRUE(h.mgr->Write("k" + std::to_string(i), OneSeg(std::string(100, 'a')), 100)); } - ASSERT_TRUE(h.mgr->Write("big", OneSeg(std::string(300, 'c')), 300)); - EXPECT_TRUE(h.mgr->Exists("big")); - EXPECT_FALSE(h.mgr->Exists("k1")); // oldest reclaimed to make room - EXPECT_LE(h.mgr->Capacity().first, 1000u); + h.mgr->DrainPendingEvents(); + EXPECT_FALSE(h.mgr->Write("big", OneSeg(std::string(300, 'c')), 300)); + EXPECT_FALSE(h.mgr->Exists("big")); + EXPECT_TRUE(h.mgr->Exists("k1")); + EXPECT_TRUE(h.mgr->DrainPendingEvents().empty()); + + // The batched path behaves the same, per key. + const std::string big(300, 'c'); + const std::string small(100, 's'); + auto ok = + h.mgr->WriteBatch({"big", "small"}, {big.data(), small.data()}, {big.size(), small.size()}); + EXPECT_EQ(ok, (std::vector{false, true})); + EXPECT_TRUE(h.mgr->Exists("k1")); } TEST(PeerSsdEviction, InFlightReadIsNotEvicted) { @@ -257,9 +271,9 @@ TEST(PeerSsdEviction, InFlightReadIsNotEvicted) { std::thread reader([&] { out = h.mgr->PrepareRead("K", buf.data(), buf.size()); }); h.backend->WaitReadsStarted(1); // PrepareRead has marked K in-flight and is blocked in the read - // Eviction must skip a key that is being read. + // Eviction must skip a key that is being read, and must not offer it. EXPECT_FALSE(h.mgr->Evict("K")); - EXPECT_TRUE(h.mgr->SelectVictims(1'000'000).empty()); + EXPECT_TRUE(h.mgr->EvictionCandidates(100).empty()); EXPECT_TRUE(h.mgr->Exists("K")); h.backend->UnblockReads(); @@ -370,26 +384,19 @@ TEST(PeerSsdEviction, ClearLocalWaitsForInFlightRead) { EXPECT_FALSE(h.mgr->Exists("K")); } -TEST(PeerSsdEviction, InvalidWatermarksThrow) { - // low >= high, and high > 1 are both rejected (fail-fast, no silent clamp). - EXPECT_THROW(PeerSsdManager(std::make_unique(1000), 0.5, 0.7), std::runtime_error); - EXPECT_THROW(PeerSsdManager(std::make_unique(1000), 1.5, 0.7), std::runtime_error); - EXPECT_THROW(PeerSsdManager(std::make_unique(1000), 0.9, 0.0), std::runtime_error); -} - -TEST(PeerSsdEviction, SelectVictimsBoundaries) { +TEST(PeerSsdEviction, EvictionCandidatesBoundaries) { auto h = MakeHarness(/*capacity=*/1'000'000); ASSERT_TRUE(h.mgr->Write("K", OneSeg("data"), 4)); - EXPECT_TRUE(h.mgr->SelectVictims(0).empty()); // nothing to free + EXPECT_TRUE(h.mgr->EvictionCandidates(0).empty()); // nothing asked for - // All candidates in flight -> no victim, no spin. + // All candidates in flight -> none offered, no spin. h.backend->BlockReads(); std::vector buf(4); SsdReadOutcome out{}; std::thread reader([&] { out = h.mgr->PrepareRead("K", buf.data(), buf.size()); }); h.backend->WaitReadsStarted(1); - EXPECT_TRUE(h.mgr->SelectVictims(1'000'000).empty()); + EXPECT_TRUE(h.mgr->EvictionCandidates(100).empty()); h.backend->UnblockReads(); reader.join(); EXPECT_EQ(out.status, SsdReadStatus::kOk); @@ -402,7 +409,7 @@ TEST(PeerSsdEviction, DisabledManagerIsInert) { EXPECT_FALSE(mgr.Write("K", OneSeg("data"), 4)); EXPECT_FALSE(mgr.Evict("K")); - EXPECT_TRUE(mgr.SelectVictims(100).empty()); + EXPECT_TRUE(mgr.EvictionCandidates(100).empty()); std::vector buf(4); EXPECT_EQ(mgr.PrepareRead("K", buf.data(), buf.size()).status, SsdReadStatus::kNotFound); mgr.ClearLocal(); // no backend -> no crash, no-op diff --git a/tests/cpp/umbp/distributed/test_peer_ssd_single_flight.cpp b/tests/cpp/umbp/distributed/test_peer_ssd_single_flight.cpp index 20e382115..dca5075df 100644 --- a/tests/cpp/umbp/distributed/test_peer_ssd_single_flight.cpp +++ b/tests/cpp/umbp/distributed/test_peer_ssd_single_flight.cpp @@ -147,7 +147,7 @@ struct Harness { Harness MakeHarness(bool single_flight = true) { auto be = std::make_unique(1'000'000); GatedBackend* raw = be.get(); - return Harness{raw, std::make_unique(std::move(be), 0.9, 0.7, single_flight)}; + return Harness{raw, std::make_unique(std::move(be), single_flight)}; } std::vector> OneSeg(const std::string& s) { diff --git a/tests/cpp/umbp/distributed/test_ssd_backend.cpp b/tests/cpp/umbp/distributed/test_ssd_backend.cpp index f91edca33..24095a7ec 100644 --- a/tests/cpp/umbp/distributed/test_ssd_backend.cpp +++ b/tests/cpp/umbp/distributed/test_ssd_backend.cpp @@ -46,6 +46,7 @@ #include "umbp/common/fd_limit.h" #include "umbp/distributed/peer/backend/ssd_backend.h" +#include "umbp/distributed/pool/peer_pool.h" #include "umbp/distributed/transfer/composite_transfer_engine.h" #include "umbp/distributed/transfer/local_copy_engine.h" #ifdef UMBP_ENABLE_GDS @@ -556,6 +557,71 @@ TEST_F(SsdBackendTest, EvictResultsAreOnePerKeyInRequestOrder) { EXPECT_EQ(evicted[1].bytes_freed, 1024u); } +// --------------------------------------------------------------------------- +// Local eviction: the backend offers order, the pool decides +// --------------------------------------------------------------------------- + +TEST_F(SsdBackendTest, CandidatesAreColdestFirstAndSkipALeasedKey) { + SsdBackend* backend = Start(); + ASSERT_TRUE(Put(backend, "a", Payload(1024, 1))); + ASSERT_TRUE(Put(backend, "b", Payload(2048, 2))); + ASSERT_TRUE(Put(backend, "c", Payload(1024, 3))); + + auto offers = backend->EvictionCandidates(10); + ASSERT_EQ(offers.size(), 3u); + EXPECT_EQ(offers[0], (EvictionOffer{"a", 1024})); + EXPECT_EQ(offers[1], (EvictionOffer{"b", 2048})); + EXPECT_EQ(offers[2], (EvictionOffer{"c", 1024})); + + // A staged read holds "b" for its lease, and Evict would refuse it, so it is + // not offered at all. + ASSERT_TRUE(backend->BatchResolve({"b"}, false)[0].found); + offers = backend->EvictionCandidates(10); + for (const auto& offer : offers) EXPECT_NE(offer.key, "b"); + EXPECT_EQ(offers.size(), 2u); +} + +TEST_F(SsdBackendTest, PoolLocalEvictionDrainsTheSsdMediumThroughEvict) { + // SSD's watermark eviction used to run inside every write. It now lives in + // the pool, on the pool's thread, and reaches this medium through Evict() + // like any master-driven eviction does. + constexpr uint64_t kCapacity = 1024 * 1024; + SsdBackend::Config cfg; + cfg.page_size = kPageSize; + cfg.staging_pages = 8; + cfg.read_lease_ttl = std::chrono::milliseconds{500}; + cfg.ssd.enabled = true; + cfg.ssd.ssd.enabled = true; + cfg.ssd.ssd.storage_dir = dir_.string(); + cfg.ssd.ssd.capacity_bytes = kCapacity; + cfg.ssd.ssd.io.backend = UMBPIoBackend::Posix; + auto owned = std::make_unique(std::move(cfg)); + ASSERT_TRUE(owned->Init(®istrar_)); + SsdBackend* ssd = owned.get(); + BackendRegistry registry; + ASSERT_TRUE(registry.Register("ssd", std::move(owned))); + PeerPool pool(®istry, MakeSingleBackendPolicy()); + + // Fill past the high watermark. Record overheads make the exact count a + // property of the segment format, so fill by measurement, not arithmetic. + std::vector keys; + const auto used = [&] { return ssd->Capacity().total_bytes - ssd->Capacity().available_bytes; }; + while (used() < kCapacity * 9 / 10) { + keys.push_back("k" + std::to_string(keys.size())); + ASSERT_TRUE(Put(ssd, keys.back(), Payload(32 * 1024, static_cast(keys.size())))) + << "filled to " << used() << " of " << kCapacity; + } + ASSERT_GE(keys.size(), 8u); + + ASSERT_TRUE(pool.EnableLocalEviction(registry.BackendId(ssd), 0.9, 0.5)); + pool.RunLocalEvictionOnceForTest(); + + EXPECT_LE(used(), kCapacity / 2) << "not drained to the low watermark"; + EXPECT_FALSE(ssd->Contains(keys.front())) << "the coldest key survived"; + EXPECT_TRUE(ssd->Contains(keys.back())) << "drained past the low watermark"; + EXPECT_GE(pool.LocalEvictionStats().keys, 1u); +} + TEST_F(SsdBackendTest, ClearLocalDropsEverythingAndGatesAllocation) { SsdBackend* backend = Start(); ASSERT_TRUE(Put(backend, "key-a", Payload(1024, 1))); From 3a0621f6469b2b3446d991985de21b99274ffc2e Mon Sep 17 00:00:00 2001 From: TianDi101 Date: Sat, 10 Oct 2026 10:56:18 +0000 Subject: [PATCH 04/19] fix(umbp): SPDK tier no longer evicts behind its owner's back With ssd_backend=spdk, PeerSsdManager reaches the device through the spdk_proxy daemon, whose SpdkSsdTier retried a failed allocation by dropping its own oldest keys (EvictLRU). Nothing told PeerSsdManager: it kept listing those keys, no REMOVE reached the master, and the master kept routing reads to bytes that were gone. Fragmentation could trigger it well below the SSD watermark. A write that does not fit now fails, as it does on the file tier, and freeing space is the owner's local eviction, which reaches the tier through Evict(). PeerSsdManager is the daemon's only client. Not built by the default dev image (no SPDK); checked with a syntax-only compile against 3rdparty/spdk/include. Co-Authored-By: Claude Opus 5.5 --- .../include/umbp/local/tiers/spdk_ssd_tier.h | 6 +- src/umbp/local/tiers/spdk_ssd_tier.cpp | 87 ++++--------------- 2 files changed, 21 insertions(+), 72 deletions(-) diff --git a/src/umbp/include/umbp/local/tiers/spdk_ssd_tier.h b/src/umbp/include/umbp/local/tiers/spdk_ssd_tier.h index 6b1725ce0..27a1cf9e8 100644 --- a/src/umbp/include/umbp/local/tiers/spdk_ssd_tier.h +++ b/src/umbp/include/umbp/local/tiers/spdk_ssd_tier.h @@ -25,7 +25,8 @@ // Metadata management follows mooncake-store's OffsetAllocatorStorageBackend: // - Sharded map (kNumShards) with std::shared_mutex for concurrent reads // - RefCounted allocation handles (AllocationPtr) for safe concurrent access -// - Auto LRU eviction on allocation failure +// - NO eviction on allocation failure: a write that does not fit fails, and +// freeing space is the owner's decision (PeerSsdManager calls Evict()). #pragma once #include @@ -207,9 +208,6 @@ class SpdkSsdTier : public TierBackend { const std::vector& sizes, std::vector& results); - // -- LRU eviction -------------------------------------------------------- - size_t EvictLRU(size_t needed); - // -- Member data --------------------------------------------------------- bool initialized_ = false; std::shared_ptr<::umbp::offset_allocator::OffsetAllocator> allocator_; diff --git a/src/umbp/local/tiers/spdk_ssd_tier.cpp b/src/umbp/local/tiers/spdk_ssd_tier.cpp index 558e6734e..3fd7d4992 100644 --- a/src/umbp/local/tiers/spdk_ssd_tier.cpp +++ b/src/umbp/local/tiers/spdk_ssd_tier.cpp @@ -242,51 +242,18 @@ std::vector SpdkSsdTier::GetLRUCandidates(size_t max_candidates) co return result; } -// =========================================================================== -// LRU eviction — evicts entries until `needed` bytes are freed. -// Lock ordering: lru_mu_ → shard.mutex (consistent with all other methods). -// Returns actual bytes freed (may be 0 if LRU is empty). -// =========================================================================== -size_t SpdkSsdTier::EvictLRU(size_t needed) { - size_t freed = 0; - int evicted = 0; - while (freed < needed) { - std::lock_guard lru_lk(lru_mu_); - if (lru_list_.empty()) break; - - std::string key = lru_list_.back(); - auto& shard = shards_[ShardForKey(key)]; - std::unique_lock slk(shard.mutex); - auto it = shard.map.find(key); - if (it != shard.map.end()) { - size_t entry_size = AlignUp(it->second.data_size); - bool immediate = (it->second.allocation.use_count() == 1); - freed += entry_size; - evicted_bytes_.fetch_add(entry_size, std::memory_order_relaxed); - shard.map.erase(it); - if (!immediate) { - MORI_UMBP_WARN( - "EvictLRU: key '{}' ({}KB) has in-flight readers, " - "space reclaim deferred", - key, entry_size / 1024); - } - } - lru_list_.pop_back(); - ++evicted; - } - if (evicted > 0) { - MORI_UMBP_INFO("EvictLRU: evicted {} entries, freed {}MB (requested {}MB)", evicted, - freed / (1024 * 1024), needed / (1024 * 1024)); - } - return freed; -} - // =========================================================================== // PrepareWriteAlloc — common Phase 1 for all BatchWrite* variants. // // 1) Checks existing keys (shard shared lock + LRU update) // 2) Batch allocates space for new keys -// 3) On allocation failure, evicts LRU entries and retries +// +// A key that does not fit FAILS. The tier never evicts to make room: the +// keys it would pick belong to whoever owns this tier (PeerSsdManager, over +// the proxy), which tracks them and reports their REMOVE to the master. An +// eviction here would be invisible to both -- the owner would keep listing a +// key whose bytes are gone. Making room is the owner's local eviction, which +// reaches this tier through Evict(). // =========================================================================== std::vector SpdkSsdTier::PrepareWriteAlloc( const std::vector& keys, const std::vector& sizes, @@ -322,34 +289,18 @@ std::vector SpdkSsdTier::PrepareWriteAlloc( if (alloc_sizes.empty()) return pending; - // Allocate with auto-eviction retry - size_t success_count = 0; - constexpr int kMaxEvictRetries = 3; - - for (int retry = 0; retry <= kMaxEvictRetries; ++retry) { - std::vector remaining(alloc_sizes.begin() + static_cast(success_count), - alloc_sizes.end()); - - auto handles = allocator_->batch_allocate(remaining); - - for (size_t j = 0; j < handles.size(); ++j) { - if (!handles[j].has_value()) break; - PendingWrite pw; - pw.idx = new_indices[success_count + j]; - pw.aligned_size = alloc_sizes[success_count + j]; - pw.allocation = std::make_shared(std::move(handles[j].value())); - pending.push_back(std::move(pw)); - } - - success_count = pending.size(); - if (success_count >= new_indices.size()) break; - - size_t needed = 0; - for (size_t k = success_count; k < alloc_sizes.size(); ++k) needed += alloc_sizes[k]; - - size_t freed = EvictLRU(needed); - if (freed == 0) break; + // One attempt. The allocator stops at the first request it cannot place, + // so everything from there on fails with it. + auto handles = allocator_->batch_allocate(alloc_sizes); + for (size_t j = 0; j < handles.size(); ++j) { + if (!handles[j].has_value()) break; + PendingWrite pw; + pw.idx = new_indices[j]; + pw.aligned_size = alloc_sizes[j]; + pw.allocation = std::make_shared(std::move(handles[j].value())); + pending.push_back(std::move(pw)); } + const size_t success_count = pending.size(); if (success_count < new_indices.size()) { size_t failed = new_indices.size() - success_count; @@ -439,7 +390,7 @@ std::vector SpdkSsdTier::PrepareReadLookup( // =========================================================================== // BatchWrite — deep-queue NVMe write pipeline // -// Phase 1: PrepareWriteAlloc (check existing, allocate with auto-eviction) +// Phase 1: PrepareWriteAlloc (check existing, allocate; no eviction) // Phase 2: memcpy + submit + drain pipeline on calling thread // Phase 3: CommitWriteEntries (update sharded map + LRU) // =========================================================================== From 1eaaab2fb1750800668be87295e7aedce22feebb Mon Sep 17 00:00:00 2001 From: TianDi101 Date: Mon, 12 Oct 2026 14:47:59 +0000 Subject: [PATCH 05/19] docs(umbp): drop the eviction-convergence design doc The PR description carries the design; the two code comments that pointed at the doc already explain themselves. Co-Authored-By: Claude Opus 5.5 --- src/umbp/distributed/pool_client.cpp | 6 +- src/umbp/doc/design-eviction-convergence.md | 324 ------------------ .../distributed/peer/backend/medium_backend.h | 2 +- 3 files changed, 4 insertions(+), 328 deletions(-) delete mode 100644 src/umbp/doc/design-eviction-convergence.md diff --git a/src/umbp/distributed/pool_client.cpp b/src/umbp/distributed/pool_client.cpp index da90954ea..6b6121b8d 100644 --- a/src/umbp/distributed/pool_client.cpp +++ b/src/umbp/distributed/pool_client.cpp @@ -1094,9 +1094,9 @@ bool PoolClient::Init() { default_pool_ = std::make_unique(®istry_, std::move(placement_policy), transfer_engine_.get()); - // Local watermark eviction (design-eviction-convergence.md §6). It goes - // through PeerPool::Evict, the same call a master's EvictKey makes, on the - // pool's own thread -- never on a put. Which media get it is unchanged: + // Local watermark eviction. It goes through PeerPool::Evict, the same call + // a master's EvictKey makes, on the pool's own thread -- never on a put. + // Which media get it is unchanged: // * no master: every medium, since nothing else would ever free a page; // * with a master: SSD only, which has always reclaimed for itself between // the master's periodic rounds. DRAM/HBM stay master-driven there. diff --git a/src/umbp/doc/design-eviction-convergence.md b/src/umbp/doc/design-eviction-convergence.md deleted file mode 100644 index 35df7cd8f..000000000 --- a/src/umbp/doc/design-eviction-convergence.md +++ /dev/null @@ -1,324 +0,0 @@ -# UMBP Eviction Convergence — Design - -**Goal:** one eviction path. Whoever decides that a key has to go — the master, -or a node that has no master — asks `PeerPool::Evict`, and the bytes are freed -by that medium's single `Evict` implementation. Nothing evicts inline on a put. - -**Scope:** `src/umbp/distributed/` (master eviction, peer service, `PeerPool`, -`PageBackend`, `SsdBackend` / `PeerSsdManager`) and the SPDK tier the SSD medium -reaches through the proxy daemon (`src/umbp/local/tiers/spdk_ssd_tier.cpp`). - -**The three requirements:** - -1. The master's eviction request names the medium it measured as over-full. -2. Each medium has exactly one eviction implementation, and every eviction goes - through it. -3. On a node with no master, watermark eviction calls the same function the - master's request does. - ---- - -## 1. Where eviction happened before - -Five places freed a committed key. Only the first two went through the -medium's `Evict`, and only the first went through `PeerPool`. - -| # | Site | Runs when | How it frees | Problem | -|---|---|---|---|---| -| 1 | Master `EvictionManager` → `EvictKey` RPC | Master timer; a (node, tier) is over the master's watermark | `PeerService::EvictKey` → `PeerPool::Evict(kReclaim)` → `MediumBackend::Evict` | The request carries **keys only**. The master picks victims per (node, tier) but `SelectVictims` returns node → keys and `EvictKeyRequest` is `repeated string keys`, so the tier is lost. The peer frees by the key's placement: on a tier without `on_evict` offload it deletes the key from **every** backend. With copy-mode promotion a key lives in DRAM *and* SSD, so SSD pressure can free DRAM instead, and DRAM pressure can delete the SSD copy too. | -| 2 | Tier transitions (offload / promotion) | After a key is copied | `source->Evict` (drop the moved-from copy), `target->Evict` (undo a failed target) | None — this is the primitive, used as intended. | -| 3 | `PageBackend::MaybeEvictToLowWatermark` | No master only; inline on commit and on a failed allocation | Its own copy of `PageBackend::Evict`'s body | A second implementation. It runs below `PeerPool`, so `on_evict` demotion never happens, the pool's placement goes stale, and `InstrumentedBackend` never sees it. It runs *inside* `PeerPool::BatchCommit`, before the pool's `watermark` offload check, so it frees the tier down to its low watermark first and the offload never fires: cold keys are deleted instead of moved to SSD. It also runs on the put path under the backend's exclusive lock. | -| 4 | `PeerSsdManager::EvictToLowWatermark` | Every mode, master or not; inline after every SSD write, plus evict-then-retry when a write fails | `PeerSsdManager::Evict(key)` (the right function) | Bypasses `PeerPool` and `InstrumentedBackend` (its victims and bytes are counted nowhere that is exported). Runs inside `SsdBackend::BatchCommit`, i.e. inside `PeerPool::BatchCommit` holding the pool lock. | -| 5 | `SpdkSsdTier::EvictLRU` in the SPDK proxy daemon | `ssd_backend=spdk`; inline when an allocation fails | Erases keys from the daemon's own map | Nobody is told. `PeerSsdManager` keeps listing the keys, no REMOVE reaches the master, and the master keeps routing reads to bytes that are gone. Fragmentation can trigger it below `PeerSsdManager`'s own watermark. | - -So "local eviction does not call `Evict()`" (row 3) was not the only gap: -there were three triggers, two layers, and a hidden fifth path under SSD. - ---- - -## 2. Target shape - -``` - policy: which keys, which medium routing: which copy, demote or drop freeing: one per medium - ─────────────────────────────────── ──────────────────────────────────── ─────────────────────── - master EvictionManager ─EvictKey(key, tier)─┐ - ├─▶ PeerPool::Evict(requests) ─┬─▶ MediumBackend::Evict(keys) - PeerPool local eviction worker ─────────────┘ │ ├ PageBackend::Evict - (no master: every medium; │ └ SsdBackend::Evict - with a master: SSD only) │ └ PeerSsdManager::Evict - └─ demote ─▶ TierTransitionExecutor - copy down, then source->Evict -``` - -Three rules hold after this change: - -1. **`MediumBackend::Evict` is the only code that frees a committed key for - capacity.** Freeing that is *not* eviction stays where it is: aborting or - reaping an uncommitted slot, discarding a stale or duplicate commit, and - `ClearLocal`. -2. **`PeerPool::Evict` is the only place that decides what happens to a copy** - — move it down a tier or drop it. Every eviction policy calls it. -3. **No eviction work on a put path.** A put may wake the eviction worker. It - never selects victims and never frees. - ---- - -## 3. Requirement 1 — per-tier eviction requests - -**Master.** A victim is now a key *and* the medium to free it from: - -```cpp -// types.h -struct EvictionVictim { - std::string key; - TierType tier = TierType::UNKNOWN; -}; -``` - -- `MasterEvictStrategy::SelectVictims` returns `node_id → vector`, - taking the tier from the candidate's `Location.tier` — the same tier whose - budget it charged. A key over budget in two tiers of one node appears twice, - once per tier, which is now meaningful. -- `EvictKeyDispatcher::DispatchEvictKey` takes victims; `MasterPeerStubPool` - fills both proto fields below. Chunking to the gRPC limit is unchanged. - -`MasterEvictStrategy` is a public, pluggable interface. Its return type changes; -there is no other implementation in tree and the Python bindings do not expose it. - -**Wire.** `EvictKeyRequest` gains a field parallel to `keys`: - -```proto -message EvictKeyRequest { - repeated string keys = 1; - // tiers[i] is the medium keys[i] should be freed from. Ignored unless it has - // exactly as many entries as keys. - repeated TierType tiers = 2; -} -``` - -| Master | Peer | Result | -|---|---|---| -| new | new | each key freed from the named medium | -| new | old | old peer ignores `tiers`: previous behavior | -| old | new | `tiers` empty: each key is freed from the medium that currently holds it (§4) | - -**Peer.** `PeerServiceServer::EvictKey` turns the request into -`PoolEvictRequest{key, tier}` and calls `PeerPool::Evict`. - ---- - -## 4. `PeerPool::Evict` — the one routing entry point - -```cpp -struct PoolEvictRequest { - std::string key; - // The medium to free the key from. UNKNOWN: the medium that holds the key's - // placement (what a request without a tier means). - TierType tier = TierType::UNKNOWN; - // A specific backend instead of a medium; the local eviction worker knows - // exactly which backend is over its watermark. Wins over `tier`. - uint32_t backend_id = BackendRegistry::kMaxBackends; -}; - -std::vector Evict(const std::vector& requests, PoolEvictMode mode); -// Unchanged convenience form: every key with no scope. -std::vector Evict(const std::vector& keys, PoolEvictMode mode); -``` - -**Which copy.** `backend_id` names it. Otherwise `tier` selects the backend of -that medium that holds the key, preferring the placement. With neither, it is -the placement (or, without one, the first owner in read order). No copy there → -0 bytes freed and nothing else happens. - -**`kReclaim`, per copy:** - -1. The key is mid-transition → 0 bytes freed; the caller retries next round - (the same contract as a read lease). -2. The copy is a *draining* source (left behind by an offload whose source - delete hit a read lease) → retry deleting that copy. -3. The copy is the key's placement and its logical tier has an offload target → - **demote**: copy it down, then `source->Evict`. If the target already holds - the key (a copy-mode promotion left one there) nothing is copied and only the - source is freed. -4. Otherwise, or if the demotion failed (no room downstream) → **drop** that copy - only: `Evict({key})` on that one backend. One exception: if the demotion - failed because *another* transition started moving the key meanwhile, leave - it (0 freed). Deleting the source then could land before that transition pins - it, and the key would be lost instead of moved. -5. If the freed copy was the placement, point the placement at the remaining - copy, or forget the key. - -`bytes_freed` is what the named copy released, so it is now accurate per tier. - -**`kDiscard`** is unchanged: delete from every backend, no demotion. - -**Decision — demote regardless of trigger.** Before, only an `on_evict` tier -demoted; evicting from a `watermark` tier dropped the key from every backend. -Now `offload_trigger` only says whether the pool *also* moves keys proactively at -the tier's watermark; any eviction of a copy whose tier has a downstream moves it -there. Two reasons: - -- The data stays reachable instead of being deleted while a lower tier had room. -- On a masterless node, local eviction and the pool's `watermark` offload now do - the *same* thing to the same tier. Before, the local round deleted exactly the - cold keys the offload was about to move (§1 row 3). - -The cost is that an `EvictKey` handled for a `watermark` tier now copies bytes, -as `on_evict` tiers already did inside the same handler. - -**Locking.** `Evict` holds `lifecycle_mutex_` shared for the whole call, so -`ClearLocal` cannot run underneath it. Demotions release `operation_mutex_` for -the byte copy as before (`RunTransition`), and the drops now also run outside -`operation_mutex_`: an SSD delete is device IO and must not stall every commit -on the node. Placement repair re-checks the backends under the lock afterwards, -so a key re-committed in that window keeps its placement. - ---- - -## 5. Requirement 2 — one eviction implementation per medium - -| Medium | The one implementation | Removed | -|---|---|---| -| DRAM / HBM (`PageBackend`) | `PageBackend::Evict` | `MaybeEvictToLowWatermark` and its four call sites (`Allocate`, `BatchAllocate`, `Commit`, `BatchCommit`), the round mutex and warning latch | -| SSD (`SsdBackend` → `PeerSsdManager`) | `PeerSsdManager::Evict(key)`, reached only through `SsdBackend::Evict` | `EvictToLowWatermark`, the check after every `Write` / `WriteBatch`, the evict-then-retry on a failed write, the manager's own watermarks and `eviction_mu_` | -| SSD via SPDK proxy (`SpdkSsdTier` in the daemon) | `SpdkSsdTier::Evict(key)`, driven by `PeerSsdManager` over the proxy | `EvictLRU` and the evict-on-allocation-failure retry: a full or fragmented device now fails the write. The daemon's only client is `PeerSsdManager`, which evicts through the pool. | - -`MediumBackend::EnableLocalEviction` is removed. In its place the medium offers -**ordering, not eviction**: - -```cpp -// types.h: a key, and the capacity freeing it gives back (page-rounded where -// the medium allocates in pages). -struct EvictionOffer { std::string key; uint64_t bytes; }; - -// Start keeping the order EvictionCandidates() reads. Off by default: a -// master-only node never asks for candidates. -virtual void TrackEvictionOrder() {} -// Up to max_count keys this medium could free right now, coldest first, -// skipping keys under a read lease, a pin, or an in-flight read. Frees nothing; -// Evict() re-checks every key. -virtual std::vector EvictionCandidates(size_t max_count) { return {}; } -``` - -`PageBackend` keeps its LRU (touched by commit and resolve) behind -`TrackEvictionOrder`; `PeerSsdManager` already keeps one unconditionally. - -**Why an offer carries its size.** The worker asks for a batch of 64 but takes -only as many offers as the remaining budget needs. Without sizes it would evict -the whole batch before re-reading usage, and on a small or nearly-drained medium -one batch overshoots the low watermark by far -- the old in-medium rounds avoided -that by summing sizes while selecting, and so must this. - -**Why the order comes from the medium, not the pool.** `PeerPool` has an LRU of -its own, `access_order_`, but maintains it only for `watermark` offload, and -turning it on disables `BatchResolve`'s single-backend fast path — which is the -masterless 100%-hit restore path. The medium already sees every read, including -that fast path. - ---- - -## 6. Requirement 3 — masterless watermark eviction through the same call - -```cpp -// PeerPool -bool EnableLocalEviction(uint32_t backend_id, double high_watermark, double low_watermark); -``` - -The first call starts one eviction worker thread. The worker: - -- **Wakes** when a commit lands, when an allocation answers NO_SPACE, and when a - transition finishes; it also polls every 100 ms, and leaves at least 1 ms - between passes so a stream of commits cannot keep it spinning. -- **Orders** backends lowest tier first (SSD, then DRAM, then HBM), so a - demotion from an upper tier finds room downstream. -- For each backend at or above its high watermark: repeat - `EvictionCandidates(64)` → take offers until their bytes cover what is left of - the budget (usage minus the low watermark, fixed at the start) → - `PeerPool::Evict({key, backend_id}…, kReclaim)` — **the same call the master's - request makes** — until usage is at or below the low watermark, the budget is - covered, the medium has no candidates, or a batch frees nothing (every - candidate was mid-transition or became busy). The next wake tries again. - The fixed budget is what stops a medium whose reported usage lags its deletes - from being drained to empty. - -Batches of 64 keep each `Evict` call short, so no single call holds the pool -lock for a whole drain. - -**Which backends evict locally.** The policy is unchanged; only the mechanism moved. - -| Medium | No master | With a master | -|---|---|---| -| DRAM / HBM | `PoolClientConfig::local_evict_{high,low}_watermark` (from `UMBP_DRAM_{HIGH,LOW}_WM`) | none — the master decides | -| SSD | the SSD config's own watermarks (`UMBP_SSD_{HIGH,LOW}_WM`) | the same, alongside the master, as before | - -SSD keeps local eviction with a master because the master's loop is periodic -and SSD previously reclaimed right after each write; removing that would let SSD -fill between master rounds. Both paths emit REMOVE events, so the master's index -stays correct. - ---- - -## 7. Behavior changes - -| Situation | Before | After | -|---|---|---| -| Master evicts a key held in DRAM and SSD (copy promotion), DRAM over budget, DRAM tier not `on_evict` | DRAM and SSD copies deleted | DRAM copy demoted (SSD already has it, so only DRAM is freed) | -| Same key, SSD over budget | DRAM copy freed (wrong medium), or both copies deleted | only the SSD copy deleted | -| Master evicts from a `watermark`-trigger tier with an offload target | dropped from every backend | demoted | -| Request from an old master (no tiers) | demote-if-`on_evict` else delete everywhere | free the placement's copy: demote if its tier has a downstream, else drop that copy | -| No master, tier has an offload target | local round deletes cold keys before offload sees the tier full | local eviction demotes them | -| No master, DRAM fills | evicted synchronously on the committing thread | evicted by the background worker; a burst faster than the worker can see NO_SPACE until it catches up | -| SSD write fails for lack of space | evict a round, retry once | the write fails; the worker reclaims SSD at its watermark | -| SPDK daemon allocation fails | daemon silently evicts its own oldest keys | the write fails | - -A failed put in a KV cache becomes a later miss, not an error; the master path -already accepts NO_SPACE between its rounds. The headroom above the high -watermark is what absorbs a burst while the worker runs. - ---- - -## 8. Metrics - -- `InstrumentedBackend`'s eviction series (`op="evict"`, `status=ok|miss`, and - bytes) now see **every** eviction, local ones included, because every eviction - is a call through the interface. -- `PeerPool` adds the counters the backend cannot see: - `mori_umbp_client_local_evict_total{event=round|key|no_candidate|stalled}` — - passes that found a backend over its high watermark, keys freed, passes where - the medium offered no candidate, and passes where a batch freed nothing. -- Removed: the SSD medium's `event="eviction_round"`, which counted the deleted - inline round. - ---- - -## 9. Tests - -- **Master strategy:** victims carry the tier they were charged to; a key over - budget in two tiers appears once per tier. -- **Master → pool chain:** a dispatched (key, tier) frees only that medium. -- **Peer service:** `EvictKey` with `tiers` scopes each key; without, falls back - to unscoped. -- **`PeerPool::Evict`:** tier scope, backend scope and unscoped; a copy-promoted - key keeps the copy that was not named; a `watermark` tier demotes; demotion - falls back to a drop when the target is full; mid-transition keys are left - alone. -- **Local eviction worker:** a masterless page backend filled past its high - watermark drains to its low watermark without any put doing the work; leased - keys survive; a tier with an offload target demotes instead of dropping; an - SSD backend drains through the same worker. -- **Media:** `PageBackend` and `PeerSsdManager` no longer evict on their own; - `EvictionCandidates` is coldest-first and skips busy keys. -- The full `tests/cpp/umbp` suite. - ---- - -## 10. Not done here - -- **A DRAM backstop in master mode.** DRAM is still master-only when a master - exists. The worker supports it; turning it on is a config question - (an emergency watermark well above the master's) left for a follow-up. -- **One source of watermarks.** There are still four: the master's - `EvictionConfig`, `local_evict_*` for paged media, the SSD config, and each - logical tier's watermarks in the policy file. -- **The SPDK change is not compiled by the default dev image**, which has no - SPDK; it needs a build with SPDK enabled before it ships. diff --git a/src/umbp/include/umbp/distributed/peer/backend/medium_backend.h b/src/umbp/include/umbp/distributed/peer/backend/medium_backend.h index b9cfc80f0..b410cec4e 100644 --- a/src/umbp/include/umbp/distributed/peer/backend/medium_backend.h +++ b/src/umbp/include/umbp/distributed/peer/backend/medium_backend.h @@ -283,7 +283,7 @@ class MediumBackend : public MetricSource { // PeerPool::Evict, and Evict() below is the one place a medium frees a // committed key. What only the medium can supply is ORDER: it sees every // read, including the resolve paths that never touch the pool's metadata. - // So it offers candidates, and nothing else (design-eviction-convergence.md). + // So it offers candidates, and nothing else. // Start keeping the recency order EvictionCandidates() reads. Off by // default: a node whose eviction is all master-driven never asks for From cdfd43414f6a04e89875ffa57a1f3602fc7db32e Mon Sep 17 00:00:00 2001 From: TianDi101 Date: Mon, 12 Oct 2026 14:55:51 +0000 Subject: [PATCH 06/19] test(umbp): fold overlapping eviction tests and tighten their comments Same coverage, fewer lines: - the SSD- and DRAM-scoped evicts of a promoted key share one test, as do the stale-tier and backend-scoped evicts; - the read-recency check rides on the drain test, and the bad-config check on the unordered-medium one; - the mismatched-tier-count case rides on the unscoped EvictKey test; - the two "writes never evict" cases share one fill; - dropped VictimCarriesTheTierItWasChargedTo (KeyOverBudgetInTwoTiers checks the tier on both victims) and the master-chain SSD-pressure test (the parked-source chain test checks the SSD victim end to end, and TierScopedEvictFreesOnlyThatMediumsCopy what the pool does with it); - removed the now-unused Keys() and HasRemove() helpers. Co-Authored-By: Claude Opus 5.5 --- .../test_master_evict_strategy.cpp | 21 +-- .../test_master_eviction_chain.cpp | 77 +------- tests/cpp/umbp/distributed/test_peer_pool.cpp | 165 ++++++------------ .../test_peer_service_dispatch.cpp | 42 ++--- .../distributed/test_peer_ssd_eviction.cpp | 73 ++------ .../cpp/umbp/distributed/test_ssd_backend.cpp | 12 +- 6 files changed, 89 insertions(+), 301 deletions(-) diff --git a/tests/cpp/umbp/distributed/test_master_evict_strategy.cpp b/tests/cpp/umbp/distributed/test_master_evict_strategy.cpp index 55cf1c0ee..af8107523 100644 --- a/tests/cpp/umbp/distributed/test_master_evict_strategy.cpp +++ b/tests/cpp/umbp/distributed/test_master_evict_strategy.cpp @@ -72,26 +72,9 @@ TEST(LruMasterEvictStrategy, PicksOldestFirstUntilBudgetMet) { EXPECT_EQ(victims["n1"][1].key, "b"); } -TEST(LruMasterEvictStrategy, VictimCarriesTheTierItWasChargedTo) { - LruMasterEvictStrategy strategy; - auto now = Clock::now(); - std::vector candidates = { - MakeCandidate("k", "n1", TierType::SSD, 100, now), - }; - std::unordered_map> budget; - budget["n1"][TierType::SSD] = 100; - - auto victims = strategy.SelectVictims(candidates, budget); - ASSERT_EQ(victims["n1"].size(), 1u); - // The peer frees exactly this medium's copy, so the tier must survive - // selection: without it an SSD-pressure victim could free the DRAM copy. - EXPECT_EQ(victims["n1"][0], (EvictionVictim{"k", TierType::SSD})); -} - TEST(LruMasterEvictStrategy, KeyOverBudgetInTwoTiersIsOneVictimPerTier) { - // A copy-mode promotion leaves the same key in DRAM and SSD. When both - // media are over budget the master charges each copy to its own tier, and - // the peer must be told to free both -- one victim per (key, tier). + // A key in DRAM and SSD, both over budget: one victim per (key, tier), each + // carrying the tier its copy was charged to. LruMasterEvictStrategy strategy; auto now = Clock::now(); std::vector candidates = { diff --git a/tests/cpp/umbp/distributed/test_master_eviction_chain.cpp b/tests/cpp/umbp/distributed/test_master_eviction_chain.cpp index a85bd7df9..71e6bf5fd 100644 --- a/tests/cpp/umbp/distributed/test_master_eviction_chain.cpp +++ b/tests/cpp/umbp/distributed/test_master_eviction_chain.cpp @@ -362,10 +362,8 @@ TEST(MasterEvictionChain, MasterRoundReclaimsTheParkedSourceOfAMovePromotion) { EXPECT_TRUE(registry.Get("cold")->Contains("moved")) << "source was deleted during the promotion, so there is nothing to reclaim"; - // The parked source is what occupies the cold medium, so that is the bucket - // the master sees over its watermark; the hot medium is comfortably under. - // The victim therefore names SSD -- the copy to free -- and not the promoted - // DRAM copy, which a tier-less request used to reach first via the placement. + // Only SSD (the parked source) is over its watermark, so the victim must name + // SSD, not the promoted DRAM copy the placement points at. InMemoryMasterMetadataStore store; const auto now = Clock::now(); RegisterNode(&store, now); @@ -390,77 +388,6 @@ TEST(MasterEvictionChain, MasterRoundReclaimsTheParkedSourceOfAMovePromotion) { << "the evict drained the promoted target instead of the parked source"; } -// A copy-mode promotion leaves a key in BOTH media, and the master charges -// each copy to its own tier. When only SSD is over budget the master must free -// the SSD copy -- and only that one. A tier-less request could not say so: the -// pool went by the placement, which a promotion points at the DRAM copy, so SSD -// pressure demoted the hot copy (cold already held it), freed DRAM instead, and -// left SSD exactly as full as before. -TEST(MasterEvictionChain, SsdPressureFreesOnlyTheSsdCopyOfAPromotedKey) { - constexpr auto kShortLease = std::chrono::milliseconds(50); - LocalCopyEngine engine; - BackendRegistry registry; - PageBackend::OwnershipConfig hot_ownership; - hot_ownership.buffer_sizes = {4 * kPageSize}; - auto hot = MakePageBackend(TierType::DRAM, kPageSize, hot_ownership, kNoExpiry, kShortLease); - ASSERT_TRUE(hot->Init(&engine)); - ASSERT_TRUE(registry.Register("hot", std::move(hot))); - PageBackend::OwnershipConfig cold_ownership; - cold_ownership.buffer_sizes = {4 * kPageSize}; - auto cold = MakePageBackend(TierType::SSD, kPageSize, cold_ownership, kNoExpiry, kShortLease); - ASSERT_TRUE(cold->Init(&engine)); - ASSERT_TRUE(registry.Register("cold", std::move(cold))); - - auto tiers = HotColdOnEvict(); - tiers.back().promote_trigger = PoolPromoteTrigger::kOnRead; - tiers.back().promote_mode = PoolTransitionMode::kCopy; - auto compiled = LogicalTierGraph::Compile(tiers, registry); - ASSERT_TRUE(compiled.ok()) << compiled.error; - PeerPool pool(®istry, MakeTieredPlacementPolicy(compiled.graph), &engine); - - PoolPlacementRequest request; - request.key = "dual"; - request.size = kPageSize; - request.tier = TierType::DRAM; - request.logical_tier = "cold"; - auto allocation = pool.BatchAllocate({request}).front(); - ASSERT_EQ(allocation.allocation.outcome, AllocateOutcome::kSuccessAllocated); - ASSERT_TRUE(pool.BatchCommit({PoolCommitRequest{ - {allocation.backend_id, allocation.allocation.slot_id}, "dual"}}) - .front() - .commit.success); - ASSERT_TRUE(pool.BatchResolve({"dual"}, false).front().resolved.found); - ASSERT_TRUE( - WaitFor([&] { return registry.Get("hot")->Contains("dual"); }, std::chrono::seconds(2))) - << "promotion never reached the hot tier"; - ASSERT_TRUE(registry.Get("cold")->Contains("dual")) << "copy promotion removed the source"; - // Let the promotion's read lease on the cold copy lapse. - std::this_thread::sleep_for(4 * kShortLease); - - InMemoryMasterMetadataStore store; - const auto now = Clock::now(); - RegisterNode(&store, now); - std::map caps; - caps[TierType::SSD] = TierCapacity{4 * kPageSize, 0, kPageSize}; - caps[TierType::DRAM] = TierCapacity{4 * kPageSize, 3 * kPageSize, kPageSize}; - ReportHeartbeat(&store, /*seq=*/1, now, caps, TierType::SSD, "cold", {"dual"}, - /*full_sync=*/true); - - PoolDispatcher dispatcher(&pool); - EvictionManager manager(store, OneSecondRounds(), &dispatcher); - manager.Start(); - const bool freed = - WaitFor([&] { return !registry.Get("cold")->Contains("dual"); }, std::chrono::seconds(8)); - manager.Stop(); - - ASSERT_TRUE(freed) << "SSD was over budget but its copy was never freed"; - EXPECT_EQ(dispatcher.Victims().front(), (EvictionVictim{"dual", TierType::SSD})); - EXPECT_TRUE(registry.Get("hot")->Contains("dual")) - << "SSD pressure freed the DRAM copy as well (or instead)"; - ASSERT_TRUE(pool.PlacementBackend("dual").has_value()); - EXPECT_EQ(*pool.PlacementBackend("dual"), registry.BackendId(registry.Get("hot"))); -} - // Master-side eviction reads one capacity per TierType, and a client with a // weighted multi-backend policy sums its backends into that one figure // (MasterClient::SnapshotAndCacheTierCapacities). So a hot tier sharing DRAM with diff --git a/tests/cpp/umbp/distributed/test_peer_pool.cpp b/tests/cpp/umbp/distributed/test_peer_pool.cpp index 7ccbf5089..bb010a6cd 100644 --- a/tests/cpp/umbp/distributed/test_peer_pool.cpp +++ b/tests/cpp/umbp/distributed/test_peer_pool.cpp @@ -637,9 +637,8 @@ TEST(PeerPool, DiscardDeletesInsteadOfDemoting) { EXPECT_EQ(pool.PlacementCount(), 0u); } -// A copy-mode promotion leaves the key on both media; the placement follows the -// promoted (hot) copy. Returns once both copies exist and the promotion's read -// lease on the cold copy has lapsed. +// Copy-promotes `key`: both media hold it, the placement is the hot copy, and +// the promotion's read lease on the cold copy has lapsed. void PromoteByCopy(PeerPool* pool, BackendRegistry* registry, const std::string& key) { auto request = PutRequest(key); request.logical_tier = "cold"; @@ -672,86 +671,58 @@ TEST(PeerPool, TierScopedEvictFreesOnlyThatMediumsCopy) { RegisterPageBackend(®istry, &engine, "hot", TierType::DRAM, kPageSize, 4, kShortLease); RegisterPageBackend(®istry, &engine, "cold", TierType::SSD, kPageSize, 4, kShortLease); PeerPool pool(®istry, MakeTieredPlacementPolicy(CopyPromotingHotCold(®istry)), &engine); - PromoteByCopy(&pool, ®istry, "dual"); + const uint32_t hot = registry.BackendId(registry.Get("hot")); + const uint32_t cold = registry.BackendId(registry.Get("cold")); + PromoteByCopy(&pool, ®istry, "a"); + PromoteByCopy(&pool, ®istry, "b"); - // SSD is the medium under pressure: its copy goes, the promoted DRAM copy - // stays and keeps serving reads. - auto evicted = pool.Evict({PoolEvictRequest{"dual", TierType::SSD}}, PoolEvictMode::kReclaim); - ASSERT_EQ(evicted.size(), 1u); + // SSD scope: only the SSD copy goes; the promoted DRAM copy keeps serving. + auto evicted = pool.Evict({PoolEvictRequest{"a", TierType::SSD}}, PoolEvictMode::kReclaim); EXPECT_EQ(evicted.front().bytes_freed, 64u); - EXPECT_FALSE(registry.Get("cold")->Contains("dual")); - EXPECT_TRUE(registry.Get("hot")->Contains("dual")); - EXPECT_EQ(pool.PlacementBackend("dual"), - std::optional{registry.BackendId(registry.Get("hot"))}); -} - -TEST(PeerPool, TierScopedEvictOfThePromotedCopyKeepsTheLowerOne) { - constexpr uint64_t kPageSize = 64; - constexpr auto kShortLease = std::chrono::milliseconds(1); - LocalCopyEngine engine; - BackendRegistry registry; - RegisterPageBackend(®istry, &engine, "hot", TierType::DRAM, kPageSize, 4, kShortLease); - RegisterPageBackend(®istry, &engine, "cold", TierType::SSD, kPageSize, 4, kShortLease); - PeerPool pool(®istry, MakeTieredPlacementPolicy(CopyPromotingHotCold(®istry)), &engine); - PromoteByCopy(&pool, ®istry, "dual"); + EXPECT_FALSE(registry.Get("cold")->Contains("a")); + EXPECT_TRUE(registry.Get("hot")->Contains("a")); + EXPECT_EQ(pool.PlacementBackend("a"), std::optional{hot}); - // DRAM under pressure: the hot copy demotes, finds cold already holding the - // key, and so frees DRAM without copying anything. The key stays readable. - auto evicted = pool.Evict({PoolEvictRequest{"dual", TierType::DRAM}}, PoolEvictMode::kReclaim); + // DRAM scope: cold already holds the key, so DRAM frees without a copy. + evicted = pool.Evict({PoolEvictRequest{"b", TierType::DRAM}}, PoolEvictMode::kReclaim); EXPECT_EQ(evicted.front().bytes_freed, 64u); - EXPECT_FALSE(registry.Get("hot")->Contains("dual")); - EXPECT_TRUE(registry.Get("cold")->Contains("dual")); - EXPECT_EQ(pool.PlacementBackend("dual"), - std::optional{registry.BackendId(registry.Get("cold"))}); - EXPECT_TRUE(pool.BatchResolve({"dual"}, false).front().resolved.found); + EXPECT_FALSE(registry.Get("hot")->Contains("b")); + EXPECT_EQ(pool.PlacementBackend("b"), std::optional{cold}); + EXPECT_TRUE(pool.BatchResolve({"b"}, false).front().resolved.found); } -TEST(PeerPool, BackendScopedEvictNamesOneInstance) { +TEST(PeerPool, ScopedEvictFreesOnlyTheNamedCopy) { + // The same key on two DRAM instances. BackendRegistry registry; ASSERT_TRUE(registry.Register("dram_a", std::make_unique(TierType::DRAM))); ASSERT_TRUE(registry.Register("dram_b", std::make_unique(TierType::DRAM))); PeerPool pool(®istry, MakeSingleBackendPolicy()); - // The same key on two instances of one medium: only a backend scope can tell - // them apart. for (const char* name : {"dram_a", "dram_b"}) { auto* backend = registry.Get(name); auto allocated = backend->BatchAllocate({AllocateRequest{"k", 32}}).front(); ASSERT_TRUE(backend->BatchCommit({CommitRequest{allocated.slot_id, "k"}}).front().success); } + // A tier with no copy (stale master metadata) frees nothing. + auto evicted = pool.Evict({PoolEvictRequest{"k", TierType::SSD}}, PoolEvictMode::kReclaim); + EXPECT_EQ(evicted.front().bytes_freed, 0u); + + // A backend scope frees only that instance. PoolEvictRequest request; request.key = "k"; request.backend_id = registry.BackendId(registry.Get("dram_b")); - auto evicted = pool.Evict({request}, PoolEvictMode::kReclaim); + evicted = pool.Evict({request}, PoolEvictMode::kReclaim); EXPECT_EQ(evicted.front().bytes_freed, 32u); EXPECT_TRUE(registry.Get("dram_a")->Contains("k")); EXPECT_FALSE(registry.Get("dram_b")->Contains("k")); } -TEST(PeerPool, EvictOfACopyThatIsNotThereFreesNothing) { - BackendRegistry registry; - ASSERT_TRUE(registry.Register("dram", std::make_unique(TierType::DRAM))); - PeerPool pool(®istry, MakeSingleBackendPolicy()); - auto* backend = registry.Get("dram"); - auto allocated = backend->BatchAllocate({AllocateRequest{"k", 32}}).front(); - ASSERT_TRUE(backend->BatchCommit({CommitRequest{allocated.slot_id, "k"}}).front().success); - - // Stale master metadata: it believes the key is on SSD. The DRAM copy is - // not the master's to free under an SSD budget. - auto evicted = pool.Evict({PoolEvictRequest{"k", TierType::SSD}}, PoolEvictMode::kReclaim); - EXPECT_EQ(evicted.front().bytes_freed, 0u); - EXPECT_TRUE(backend->Contains("k")); -} - TEST(PeerPool, ReclaimDemotesOnAWatermarkTierToo) { - // The trigger only says whether the pool moves keys proactively. Asked to - // free a copy whose tier has a downstream, the answer is the same either way: - // move it, do not delete data a lower tier has room for. + // Reclaim demotes whenever the tier has a downstream, whatever its trigger. constexpr uint64_t kPageSize = 64; LocalCopyEngine engine; BackendRegistry registry; - // Big enough that one key stays far under the 0.4 watermark, so the pool's - // own proactive offload never runs and only the explicit evict moves it. + // One key stays far under the watermark, so only the explicit evict moves it. RegisterPageBackend(®istry, &engine, "hot", TierType::DRAM, kPageSize, 16); RegisterPageBackend(®istry, &engine, "cold", TierType::SSD, kPageSize, 16); auto compiled = LogicalTierGraph::Compile(HotColdTiers(PoolOffloadTrigger::kWatermark), registry); @@ -767,10 +738,8 @@ TEST(PeerPool, ReclaimDemotesOnAWatermarkTierToo) { } TEST(PeerPool, DemotionWithNoRoomDownstreamDropsTheCopy) { - // "Free this tier" still has to free it when the tier below is full. - // - // A buffer is sized by its mapping, which rounds up to the host page, so a - // tier is only genuinely one page long when the page IS the host page. + // Reclaim still frees the tier when the tier below is full. Host-page-sized + // pages, so `cold` really is one page long. constexpr uint64_t kPageSize = 4096; LocalCopyEngine engine; BackendRegistry registry; @@ -1093,12 +1062,8 @@ TEST(PeerPool, ReadPromotesWithMoveDrainsSourceAndKeepsBytes) { } // --------------------------------------------------------------------------- -// Local watermark eviction -// -// The pool's own policy on a node that evicts for itself. It runs on a -// background thread, so these observe rather than await; and it goes through -// Evict() -- the master's entry point -- so what it frees and how (demote vs -// drop) is the same contract the tests above pin down. +// Local watermark eviction: a background worker that frees through Evict(), +// so it follows the same demote-or-drop contract as the tests above. // --------------------------------------------------------------------------- bool WaitUntil(const std::function& done, @@ -1111,54 +1076,36 @@ bool WaitUntil(const std::function& done, return done(); } -// A buffer is sized by its mapping, so a tier is exactly N pages long only when -// the page is the host page. Every key below is one page. +// Host-page-sized pages, so a tier is exactly N pages; every key is one page. constexpr uint64_t kHostPage = 4096; -TEST(PeerPoolLocalEviction, DrainsAFullBackendToItsLowWatermark) { +TEST(PeerPoolLocalEviction, DrainsAFullBackendColdestFirstToItsLowWatermark) { LocalCopyEngine engine; BackendRegistry registry; - RegisterPageBackend(®istry, &engine, "dram", TierType::DRAM, kHostPage, 10); - PeerPool pool(®istry, MakeSingleBackendPolicy(), &engine); - ASSERT_TRUE(pool.EnableLocalEviction(registry.BackendId(registry.Get("dram")), 0.9, 0.5)); - - for (int i = 0; i < 9; ++i) CommitKey(&pool, "k" + std::to_string(i)); - // 9 of 10 pages is the high watermark; the worker frees down to 5, and no - // further -- the offers carry their size, so a batch cannot overshoot. - auto* dram = registry.Get("dram"); - ASSERT_TRUE(WaitUntil([&] { return dram->OwnedKeyCount() <= 5; })); - std::this_thread::sleep_for(std::chrono::milliseconds(20)); - EXPECT_EQ(dram->OwnedKeyCount(), 5u); - // Coldest first. - for (int i = 0; i < 4; ++i) EXPECT_FALSE(dram->Contains("k" + std::to_string(i))) << i; - for (int i = 4; i < 9; ++i) EXPECT_TRUE(dram->Contains("k" + std::to_string(i))) << i; - EXPECT_EQ(pool.PlacementCount(), 5u); - const auto stats = pool.LocalEvictionStats(); - EXPECT_GE(stats.rounds, 1u); - EXPECT_EQ(stats.keys, 4u); -} - -TEST(PeerPoolLocalEviction, ReadsKeepAKeyWarm) { - LocalCopyEngine engine; - BackendRegistry registry; - // A lease short enough to have lapsed before the worker looks. + // 1 ms lease: lapsed by the time the worker looks. RegisterPageBackend(®istry, &engine, "dram", TierType::DRAM, kHostPage, 10, std::chrono::milliseconds(1)); PeerPool pool(®istry, MakeSingleBackendPolicy(), &engine); - // Before any traffic, as PoolClient does: the medium keeps its order from the - // first commit. 8 of 10 pages stays under the high watermark. + // Enabled before any commit, as PoolClient does. ASSERT_TRUE(pool.EnableLocalEviction(registry.BackendId(registry.Get("dram")), 0.9, 0.5)); for (int i = 0; i < 8; ++i) CommitKey(&pool, "k" + std::to_string(i)); - // The oldest key is read, which renews its recency in the medium's order. + // Reading the oldest key renews its recency. ASSERT_TRUE(pool.BatchResolve({"k0"}, false).front().resolved.found); std::this_thread::sleep_for(std::chrono::milliseconds(5)); CommitKey(&pool, "k8"); + // 9 of 10 pages hits the high watermark; the worker frees exactly down to 5. auto* dram = registry.Get("dram"); ASSERT_TRUE(WaitUntil([&] { return dram->OwnedKeyCount() <= 5; })); + std::this_thread::sleep_for(std::chrono::milliseconds(20)); + EXPECT_EQ(dram->OwnedKeyCount(), 5u); EXPECT_TRUE(dram->Contains("k0")) << "a just-read key was evicted ahead of colder ones"; - EXPECT_FALSE(dram->Contains("k1")); + for (int i = 1; i <= 4; ++i) EXPECT_FALSE(dram->Contains("k" + std::to_string(i))) << i; + for (int i = 5; i <= 8; ++i) EXPECT_TRUE(dram->Contains("k" + std::to_string(i))) << i; + EXPECT_EQ(pool.PlacementCount(), 5u); + EXPECT_GE(pool.LocalEvictionStats().rounds, 1u); + EXPECT_EQ(pool.LocalEvictionStats().keys, 4u); } TEST(PeerPoolLocalEviction, KeysUnderAReadLeaseAreNeverOffered) { @@ -1181,8 +1128,7 @@ TEST(PeerPoolLocalEviction, KeysUnderAReadLeaseAreNeverOffered) { } TEST(PeerPoolLocalEviction, ATierWithADownstreamDemotesInsteadOfDropping) { - // The same call the master makes, so the same answer: a key whose tier has - // somewhere to go is moved there, not deleted. + // Same call as a master evict: a tier with a downstream demotes, never drops. LocalCopyEngine engine; BackendRegistry registry; RegisterPageBackend(®istry, &engine, "hot", TierType::DRAM, kHostPage, 10); @@ -1205,10 +1151,8 @@ TEST(PeerPoolLocalEviction, ATierWithADownstreamDemotesInsteadOfDropping) { } TEST(PeerPoolLocalEviction, NeverDeletesWhatAWatermarkOffloadWouldMove) { - // The masterless failure this replaces: the backend's own round ran inside - // the commit, emptied the tier below the pool's watermark check, and so - // deleted the cold keys the watermark offload was about to move. Now both - // the offload and local eviction move them, so every key survives somewhere. + // Regression: the old in-commit round deleted the cold keys a watermark + // offload was about to move. Now both move them, so every key survives. LocalCopyEngine engine; BackendRegistry registry; RegisterPageBackend(®istry, &engine, "hot", TierType::DRAM, kHostPage, 10); @@ -1219,14 +1163,14 @@ TEST(PeerPoolLocalEviction, NeverDeletesWhatAWatermarkOffloadWouldMove) { auto compiled = LogicalTierGraph::Compile(tiers, registry); ASSERT_TRUE(compiled.ok()) << compiled.error; PeerPool pool(®istry, MakeTieredPlacementPolicy(compiled.graph), &engine); - // Deliberately the SAME watermarks as the offload: the case that used to lose. + // The same watermarks as the offload: the case that used to lose keys. ASSERT_TRUE(pool.EnableLocalEviction(registry.BackendId(registry.Get("hot")), 0.9, 0.5)); std::vector keys; for (int i = 0; i < 20; ++i) { keys.push_back("k" + std::to_string(i)); CommitKey(&pool, keys.back()); - // Let the background workers keep up, as a real put rate would. + // Let the workers keep up, as a real put rate would. ASSERT_TRUE(WaitUntil([&] { return registry.Get("hot")->OwnedKeyCount() < 9; })); } for (const auto& key : keys) { @@ -1235,7 +1179,7 @@ TEST(PeerPoolLocalEviction, NeverDeletesWhatAWatermarkOffloadWouldMove) { } } -TEST(PeerPoolLocalEviction, RejectsBadWatermarksAndUnknownBackends) { +TEST(PeerPoolLocalEviction, RejectsBadConfigAndLeavesAnUnorderedMediumAlone) { BackendRegistry registry; ASSERT_TRUE(registry.Register("dram", std::make_unique(TierType::DRAM))); PeerPool pool(®istry, MakeSingleBackendPolicy()); @@ -1244,17 +1188,10 @@ TEST(PeerPoolLocalEviction, RejectsBadWatermarksAndUnknownBackends) { EXPECT_FALSE(pool.EnableLocalEviction(id, 1.5, 0.7)); // high above 1 EXPECT_FALSE(pool.EnableLocalEviction(id, 0.9, 0.0)); // low not positive EXPECT_FALSE(pool.EnableLocalEviction(BackendRegistry::kMaxBackends - 1, 0.9, 0.7)); - EXPECT_TRUE(pool.EnableLocalEviction(id, 0.9, 0.7)); -} -TEST(PeerPoolLocalEviction, AMediumThatKeepsNoOrderIsNeverEvicted) { - // MockBackend offers no candidates (the MediumBackend default). Local - // eviction must leave it alone rather than guess at victims. - BackendRegistry registry; - ASSERT_TRUE(registry.Register("dram", std::make_unique(TierType::DRAM))); - PeerPool pool(®istry, MakeSingleBackendPolicy()); + // MockBackend offers no candidates, so it must be left alone. CommitKey(&pool, "k"); - ASSERT_TRUE(pool.EnableLocalEviction(registry.BackendId(registry.Get("dram")), 0.9, 0.5)); + ASSERT_TRUE(pool.EnableLocalEviction(id, 0.9, 0.5)); pool.RunLocalEvictionOnceForTest(); EXPECT_TRUE(registry.Get("dram")->Contains("k")); } diff --git a/tests/cpp/umbp/distributed/test_peer_service_dispatch.cpp b/tests/cpp/umbp/distributed/test_peer_service_dispatch.cpp index e714d7693..078c874b5 100644 --- a/tests/cpp/umbp/distributed/test_peer_service_dispatch.cpp +++ b/tests/cpp/umbp/distributed/test_peer_service_dispatch.cpp @@ -421,9 +421,7 @@ TEST_F(PeerServiceDispatchTest, BatchResolveReportsTheServingTier) { } TEST_F(PeerServiceDispatchTest, EvictWithATierFreesOnlyThatMedium) { - // The master charged this victim to the second medium. A key mirrored - // across media loses exactly that copy; the other one is not the master's to - // free, because it was measured against a different budget. + // A mirrored key charged to the second medium loses only that copy. SeedKey(kFirstByTier, "mirrored", 111); SeedKey(kSecondByTier, "mirrored", 222); @@ -440,13 +438,12 @@ TEST_F(PeerServiceDispatchTest, EvictWithATierFreesOnlyThatMedium) { EXPECT_FALSE(Backend(kSecondByTier)->Contains("mirrored")); } -TEST_F(PeerServiceDispatchTest, EvictWithoutTiersFreesTheCopyThatHoldsTheKey) { - // An older master sends no tiers. Each key is then freed from the medium - // that holds it -- the first owner in read order here, since the seed went - // around the pool and left no placement -- and never from every medium at - // once, which used to delete copies no budget had asked for. +TEST_F(PeerServiceDispatchTest, EvictWithoutUsableTiersFreesTheCopyThatHoldsTheKey) { + // No tiers (an older master), or a count that does not match `keys`: each + // key is freed from the one medium that holds it, never from every medium. SeedKey(kFirstByTier, "mirrored", 111); SeedKey(kSecondByTier, "mirrored", 222); + SeedKey(kSecondByTier, "b", 20); ::umbp::EvictKeyRequest req; req.add_keys("mirrored"); @@ -454,27 +451,20 @@ TEST_F(PeerServiceDispatchTest, EvictWithoutTiersFreesTheCopyThatHoldsTheKey) { grpc::ClientContext ctx; ASSERT_TRUE(stub_->EvictKey(&ctx, req, &resp).ok()); ASSERT_EQ(resp.evicted_size(), 1); - EXPECT_EQ(resp.evicted(0).bytes_freed(), 111u); + EXPECT_EQ(resp.evicted(0).bytes_freed(), 111u); // first owner in read order EXPECT_FALSE(Backend(kFirstByTier)->Contains("mirrored")); EXPECT_TRUE(Backend(kSecondByTier)->Contains("mirrored")); -} -TEST_F(PeerServiceDispatchTest, EvictWithMismatchedTierCountIsUnscoped) { - // `tiers` is parallel to `keys`; anything else cannot be paired up, so it is - // ignored rather than misapplied to the wrong keys. - SeedKey(kFirstByTier, "a", 10); - SeedKey(kSecondByTier, "b", 20); - - ::umbp::EvictKeyRequest req; - req.add_keys("a"); - req.add_keys("b"); - req.add_tiers(Proto(kSecondByTier)); - ::umbp::EvictKeyResponse resp; - grpc::ClientContext ctx; - ASSERT_TRUE(stub_->EvictKey(&ctx, req, &resp).ok()); - ASSERT_EQ(resp.evicted_size(), 2); - EXPECT_EQ(resp.evicted(0).bytes_freed(), 10u); - EXPECT_EQ(resp.evicted(1).bytes_freed(), 20u); + ::umbp::EvictKeyRequest mismatched; + mismatched.add_keys("mirrored"); + mismatched.add_keys("b"); + mismatched.add_tiers(Proto(kFirstByTier)); + ::umbp::EvictKeyResponse resp2; + grpc::ClientContext ctx2; + ASSERT_TRUE(stub_->EvictKey(&ctx2, mismatched, &resp2).ok()); + ASSERT_EQ(resp2.evicted_size(), 2); + EXPECT_EQ(resp2.evicted(0).bytes_freed(), 222u); // the tier was not misapplied + EXPECT_EQ(resp2.evicted(1).bytes_freed(), 20u); } TEST_F(PeerServiceDispatchTest, EvictOfAnAbsentKeyReportsZeroBytes) { diff --git a/tests/cpp/umbp/distributed/test_peer_ssd_eviction.cpp b/tests/cpp/umbp/distributed/test_peer_ssd_eviction.cpp index a5e8005b2..a0762a5ae 100644 --- a/tests/cpp/umbp/distributed/test_peer_ssd_eviction.cpp +++ b/tests/cpp/umbp/distributed/test_peer_ssd_eviction.cpp @@ -20,15 +20,10 @@ // OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE // SOFTWARE. // -// The SSD medium's eviction primitive and its candidate order. Drives -// PeerSsdManager through a controllable in-memory TierBackend (the test-only -// constructor) so LRU ordering, the in-flight-read guard, idempotent Write, -// backend-evict failure, concurrent eviction, and physical Clear are all -// deterministic without real disk IO. -// -// What is deliberately NOT here: watermark eviction. The manager never decides -// to evict -- not after a write, not when a write runs out of room. That is -// PeerPool's local eviction, tested in test_peer_pool against an SsdBackend. +// The SSD medium's eviction primitive and candidate order, driven through an +// in-memory TierBackend (the test-only constructor) so LRU order, the +// in-flight-read guard, idempotent Write, evict failure, concurrency and Clear +// are deterministic. Watermark eviction is PeerPool's, tested there. #include #include @@ -176,19 +171,6 @@ int CountKind(const std::vector& events, KvEvent::Kind kind) { return n; } -std::vector Keys(const std::vector& offers) { - std::vector keys; - for (const auto& offer : offers) keys.push_back(offer.key); - return keys; -} - -bool HasRemove(const std::vector& events, const std::string& key) { - for (const auto& e : events) { - if (e.kind == KvEvent::Kind::REMOVE && e.tier == TierType::SSD && e.key == key) return true; - } - return false; -} - // --------------------------------------------------------------------------- TEST(PeerSsdEviction, WriteAndPrepareReadRefreshLru) { @@ -197,8 +179,7 @@ TEST(PeerSsdEviction, WriteAndPrepareReadRefreshLru) { ASSERT_TRUE(h.mgr->Write("B", OneSeg("bbbb"), 4)); ASSERT_TRUE(h.mgr->Write("C", OneSeg("cccc"), 4)); - // LRU now (oldest->newest): A, B, C. Reading A promotes it to MRU, so the - // oldest becomes B and the first candidate must be B (not the just-read A). + // LRU is A, B, C; reading A makes B the coldest. std::vector buf(4); auto out = h.mgr->PrepareRead("A", buf.data(), buf.size()); ASSERT_EQ(out.status, SsdReadStatus::kOk); @@ -208,56 +189,30 @@ TEST(PeerSsdEviction, WriteAndPrepareReadRefreshLru) { ASSERT_EQ(candidates.size(), 1u); EXPECT_EQ(candidates.front().key, "B"); EXPECT_EQ(candidates.front().bytes, 4u); + EXPECT_EQ(h.mgr->EvictionCandidates(100).size(), 3u); + EXPECT_TRUE(h.mgr->Exists("B")) << "asking for candidates freed one"; } -TEST(PeerSsdEviction, CandidatesAreColdestFirstAndCapped) { - auto h = MakeHarness(/*capacity=*/1'000'000); - for (int i = 1; i <= 5; ++i) { - ASSERT_TRUE(h.mgr->Write("k" + std::to_string(i), OneSeg("data"), 4)); - } - EXPECT_EQ(Keys(h.mgr->EvictionCandidates(3)), (std::vector{"k1", "k2", "k3"})); - EXPECT_EQ(h.mgr->EvictionCandidates(100).size(), 5u); - // Ordering only: asking freed nothing. - for (int i = 1; i <= 5; ++i) EXPECT_TRUE(h.mgr->Exists("k" + std::to_string(i))); -} - -TEST(PeerSsdEviction, WritesNeverEvictEvenWhenNearlyFull) { - // 900 of 1000 bytes used is where the manager's own round used to fire. It - // has none now: the pool's local eviction decides, on its own thread, so a - // write must leave every earlier key exactly where it was. +TEST(PeerSsdEviction, WritesNeverEvict) { + // At 900 of 1000 (the old watermark round) and on ENOSPC alike, a write + // leaves every earlier key in place: eviction is the pool's job. auto h = MakeHarness(/*capacity=*/1000); std::string val(100, 'x'); for (int i = 1; i <= 9; ++i) { ASSERT_TRUE(h.mgr->Write("k" + std::to_string(i), OneSeg(val), val.size())); } - auto [used, total] = h.mgr->Capacity(); - EXPECT_EQ(used, 900u); - EXPECT_EQ(total, 1000u); - for (int i = 1; i <= 9; ++i) EXPECT_TRUE(h.mgr->Exists("k" + std::to_string(i))); - EXPECT_EQ(CountKind(h.mgr->DrainPendingEvents(), KvEvent::Kind::REMOVE), 0); -} - -TEST(PeerSsdEviction, WriteThatDoesNotFitFailsWithoutEvicting) { - // 800 of 1000 used, then a 300-byte value: the device answers ENOSPC. The - // write fails and records nothing; it does not evict older keys to make room - // -- that would be eviction work on the put path. - auto h = MakeHarness(/*capacity=*/1000); - for (int i = 1; i <= 8; ++i) { - ASSERT_TRUE(h.mgr->Write("k" + std::to_string(i), OneSeg(std::string(100, 'a')), 100)); - } - h.mgr->DrainPendingEvents(); + EXPECT_EQ(h.mgr->Capacity().first, 900u); EXPECT_FALSE(h.mgr->Write("big", OneSeg(std::string(300, 'c')), 300)); EXPECT_FALSE(h.mgr->Exists("big")); - EXPECT_TRUE(h.mgr->Exists("k1")); - EXPECT_TRUE(h.mgr->DrainPendingEvents().empty()); - // The batched path behaves the same, per key. + // The batched path fails the oversized key alone. const std::string big(300, 'c'); const std::string small(100, 's'); auto ok = h.mgr->WriteBatch({"big", "small"}, {big.data(), small.data()}, {big.size(), small.size()}); EXPECT_EQ(ok, (std::vector{false, true})); - EXPECT_TRUE(h.mgr->Exists("k1")); + for (int i = 1; i <= 9; ++i) EXPECT_TRUE(h.mgr->Exists("k" + std::to_string(i))); + EXPECT_EQ(CountKind(h.mgr->DrainPendingEvents(), KvEvent::Kind::REMOVE), 0); } TEST(PeerSsdEviction, InFlightReadIsNotEvicted) { diff --git a/tests/cpp/umbp/distributed/test_ssd_backend.cpp b/tests/cpp/umbp/distributed/test_ssd_backend.cpp index 24095a7ec..ad870e942 100644 --- a/tests/cpp/umbp/distributed/test_ssd_backend.cpp +++ b/tests/cpp/umbp/distributed/test_ssd_backend.cpp @@ -558,7 +558,7 @@ TEST_F(SsdBackendTest, EvictResultsAreOnePerKeyInRequestOrder) { } // --------------------------------------------------------------------------- -// Local eviction: the backend offers order, the pool decides +// Local eviction: the backend offers order; the pool decides // --------------------------------------------------------------------------- TEST_F(SsdBackendTest, CandidatesAreColdestFirstAndSkipALeasedKey) { @@ -573,8 +573,7 @@ TEST_F(SsdBackendTest, CandidatesAreColdestFirstAndSkipALeasedKey) { EXPECT_EQ(offers[1], (EvictionOffer{"b", 2048})); EXPECT_EQ(offers[2], (EvictionOffer{"c", 1024})); - // A staged read holds "b" for its lease, and Evict would refuse it, so it is - // not offered at all. + // A staged read's lease makes "b" unevictable, so it is not offered. ASSERT_TRUE(backend->BatchResolve({"b"}, false)[0].found); offers = backend->EvictionCandidates(10); for (const auto& offer : offers) EXPECT_NE(offer.key, "b"); @@ -582,9 +581,7 @@ TEST_F(SsdBackendTest, CandidatesAreColdestFirstAndSkipALeasedKey) { } TEST_F(SsdBackendTest, PoolLocalEvictionDrainsTheSsdMediumThroughEvict) { - // SSD's watermark eviction used to run inside every write. It now lives in - // the pool, on the pool's thread, and reaches this medium through Evict() - // like any master-driven eviction does. + // SSD's watermark eviction now runs in the pool and reaches it via Evict(). constexpr uint64_t kCapacity = 1024 * 1024; SsdBackend::Config cfg; cfg.page_size = kPageSize; @@ -602,8 +599,7 @@ TEST_F(SsdBackendTest, PoolLocalEvictionDrainsTheSsdMediumThroughEvict) { ASSERT_TRUE(registry.Register("ssd", std::move(owned))); PeerPool pool(®istry, MakeSingleBackendPolicy()); - // Fill past the high watermark. Record overheads make the exact count a - // property of the segment format, so fill by measurement, not arithmetic. + // Fill past the high watermark by measurement (record overhead varies). std::vector keys; const auto used = [&] { return ssd->Capacity().total_bytes - ssd->Capacity().available_bytes; }; while (used() < kCapacity * 9 / 10) { From d0faaa820fe8cc58de5a525c9220a7da93948ac6 Mon Sep 17 00:00:00 2001 From: TianDi101 Date: Tue, 13 Oct 2026 05:14:36 +0000 Subject: [PATCH 07/19] feat(umbp): size eviction rounds in bytes; queue demotions on the peer Folds #742 into this PR. - Master: a round frees each (node, tier)'s overage in bytes, optionally capped by UMBP_EVICT_MAX_BYTES_PER_ROUND, instead of 32 candidates per bucket -- which freed a few MB per round and took hours to drain a 512 GiB tier's 0.9 -> 0.7 band. The store returns every eligible row and the strategy trims to the budget. - Peer: PeerPool::Evict queues a demotion on the transition worker instead of running the copy inside the EvictKey RPC, so the call never waits on bytes. Each result carries an outcome (freed, demotion queued, in flight). The worker runs an eviction's demotion whatever the tier's own watermark says, and drops the copy if nothing downstream can take it. - Local eviction counts queued demotions toward its budget (a migration pin no longer hides a key from EvictionCandidates), so a pass never queues more than the overage while the copies run. Co-Authored-By: Claude Opus 5.5 --- .../distributed/master/eviction_manager.cpp | 37 +++- .../distributed/peer/backend/page_backend.cpp | 7 +- src/umbp/distributed/peer/peer_service.cpp | 9 +- src/umbp/distributed/pool/peer_pool.cpp | 154 +++++++++++------ src/umbp/doc/runtime-env-vars.md | 1 + src/umbp/include/umbp/common/env_time.h | 12 ++ src/umbp/include/umbp/distributed/config.h | 14 +- .../distributed/peer/backend/page_backend.h | 4 + .../include/umbp/distributed/pool/peer_pool.h | 55 ++++-- tests/cpp/umbp/distributed/test_env_time.cpp | 16 ++ .../test_master_evict_strategy.cpp | 18 ++ .../test_master_eviction_chain.cpp | 103 ++++++++++- tests/cpp/umbp/distributed/test_peer_pool.cpp | 161 ++++++++++++++++-- 13 files changed, 497 insertions(+), 94 deletions(-) diff --git a/src/umbp/distributed/master/eviction_manager.cpp b/src/umbp/distributed/master/eviction_manager.cpp index 2177a3332..d97bb52be 100644 --- a/src/umbp/distributed/master/eviction_manager.cpp +++ b/src/umbp/distributed/master/eviction_manager.cpp @@ -21,6 +21,7 @@ // SOFTWARE. #include "umbp/distributed/master/eviction_manager.h" +#include #include #include #include @@ -53,8 +54,10 @@ void EvictionManager::Start() { if (running_.load(std::memory_order_relaxed)) return; running_.store(true, std::memory_order_relaxed); thread_ = std::thread(&EvictionManager::EvictionLoop, this); - MORI_UMBP_INFO("[EvictionManager] Started (interval={}s, high={}, low={})", - config_.check_interval.count(), config_.high_watermark, config_.low_watermark); + MORI_UMBP_INFO( + "[EvictionManager] Started (interval={}s, high={}, low={}, max_bytes_per_round={})", + config_.check_interval.count(), config_.high_watermark, config_.low_watermark, + config_.max_evict_bytes_per_round); } void EvictionManager::Stop() { @@ -118,20 +121,35 @@ void EvictionManager::RunOnce() { } if (bytes_to_free.empty()) return; - MORI_UMBP_INFO("[EvictionManager] {} overloaded node-tiers detected", bytes_to_free.size()); + + // A round's size is a byte budget: the overage itself, optionally capped by + // max_evict_bytes_per_round. It used to be a row count (32 candidates per + // bucket), which freed a few MB per round however far over the tier was -- + // a 512 GiB tier's 0.9 -> 0.7 band took hours to drain -- while every round + // still paid the store's full scan. + if (config_.max_evict_bytes_per_round > 0) { + for (auto& [ntk, bytes] : bytes_to_free) { + bytes = std::min(bytes, config_.max_evict_bytes_per_round); + } + } + uint64_t budget_total = 0; + for (const auto& [ntk, bytes] : bytes_to_free) budget_total += bytes; + MORI_UMBP_INFO("[EvictionManager] {} overloaded node-tiers detected, {} bytes to free", + bytes_to_free.size(), budget_total); // Ask the store for eviction-eligible candidates in the overloaded buckets. // The store is policy-neutral: it returns rows ordered by the hint (LRU here, // so an indexed backend can push the ordering down) but makes no eviction - // decision and never sees the byte budget. Limit each bucket to the configured - // round size; a still-overloaded bucket is revisited on the next interval. + // decision and never sees the byte budget -- the strategy trims to it. No + // row cap: both stores read every eligible row of a bucket before capping, + // so a cap would only discard candidates the budget may need. std::vector buckets; buckets.reserve(bytes_to_free.size()); for (const auto& [ntk, _bytes] : bytes_to_free) buckets.push_back(ntk); - auto candidates_by_bucket = store_.EnumerateEvictionCandidates( - buckets, EvictionOrder::kLeastRecentlyAccessed, config_.evict_batch_size, - std::chrono::system_clock::now()); + auto candidates_by_bucket = + store_.EnumerateEvictionCandidates(buckets, EvictionOrder::kLeastRecentlyAccessed, + /*max_per_bucket=*/0, std::chrono::system_clock::now()); if (candidates_by_bucket.empty()) { MORI_UMBP_DEBUG("[EvictionManager] No eviction candidates found"); return; @@ -195,7 +213,8 @@ void EvictionManager::RunOnce() { // // Sized on the keys alone: the parallel tier field is a packed enum of one // byte per key, inside the per-item overhead GrpcMaxItemsPerBatch already - // reserves. + // reserves. Nothing else bounds a chunk: the peer queues any demotion and + // returns, so a call's length does not grow with the bytes it names. std::vector keys; keys.reserve(victims.size()); for (const auto& victim : victims) keys.push_back(victim.key); diff --git a/src/umbp/distributed/peer/backend/page_backend.cpp b/src/umbp/distributed/peer/backend/page_backend.cpp index d32c7e1ef..dbaeec665 100644 --- a/src/umbp/distributed/peer/backend/page_backend.cpp +++ b/src/umbp/distributed/peer/backend/page_backend.cpp @@ -587,7 +587,7 @@ bool PageBackend::AcquireMigrationRead(const std::string& key, ResolvedEntry* re resolved->pages = owned->second.pages; resolved->size = owned->second.size; resolved->page_size = page_size_; - pins_[key] = PinState{next_pin_token_++, std::chrono::steady_clock::now()}; + pins_[key] = PinState{next_pin_token_++, std::chrono::steady_clock::now(), /*migration=*/true}; return true; } @@ -831,8 +831,11 @@ std::vector PageBackend::EvictionCandidates(size_t max_count) { // The two protections Evict() applies: an in-flight read owns the pages // until its lease expires, a copy pin until the copy is done. Offering // them would only make the caller's batch free less than it asked for. + // A migration pin is the exception: the key is already on its way out, + // and the caller has to see it to count its bytes as in flight rather + // than reach past it for one more victim. if (HasActiveReadLeaseLocked(owned_it->second, now)) continue; - if (HasActivePinLocked(*it)) continue; + if (auto pin = pins_.find(*it); pin != pins_.end() && !pin->second.migration) continue; // Whole pages: that is what the allocator gives back, and what Capacity() // counts against the watermark. out.push_back( diff --git a/src/umbp/distributed/peer/peer_service.cpp b/src/umbp/distributed/peer/peer_service.cpp index 78f766145..a51e10b02 100644 --- a/src/umbp/distributed/peer/peer_service.cpp +++ b/src/umbp/distributed/peer/peer_service.cpp @@ -269,8 +269,11 @@ class PeerServiceServer::UMBPPeerServiceImpl final : public ::umbp::UMBPPeer::Se // tier per key: the pool frees exactly that medium's copy (demoting it // where its tier has a downstream) and leaves any other copy alone. An // older master sends no tiers; each key is then freed from the medium that - // currently holds it. bytes_freed is informational -- the master sizes its - // next round off the heartbeat's capacity, not off this reply. + // currently holds it. A demotion is only queued here -- the pool's + // transition worker copies the bytes -- so this returns without waiting on + // any copy, and a queued key reports 0 bytes freed. bytes_freed is + // informational: the master sizes its next round off the heartbeat's + // capacity, not off this reply. const int n = request->keys_size(); if (n == 0) return grpc::Status::OK; const bool scoped = request->tiers_size() == n; @@ -279,7 +282,7 @@ class PeerServiceServer::UMBPPeerServiceImpl final : public ::umbp::UMBPPeer::Se requests[i].key = request->keys(i); if (scoped) requests[i].tier = static_cast(request->tiers(i)); } - auto evicted = pool_ == nullptr ? std::vector{} + auto evicted = pool_ == nullptr ? std::vector{} : pool_->Evict(requests, PoolEvictMode::kReclaim); for (size_t i = 0; i < requests.size(); ++i) { auto* entry = response->add_evicted(); diff --git a/src/umbp/distributed/pool/peer_pool.cpp b/src/umbp/distributed/pool/peer_pool.cpp index c5c5d3c20..d7c82220c 100644 --- a/src/umbp/distributed/pool/peer_pool.cpp +++ b/src/umbp/distributed/pool/peer_pool.cpp @@ -99,12 +99,29 @@ PeerPool::~PeerPool() { StopTransitionWorker(); } -void PeerPool::EnqueueTransition(TransitionJob job) { +bool PeerPool::EnqueueTransition(TransitionJob job) { std::lock_guard lock(transition_mutex_); - if (stop_transition_worker_ || !queued_transition_keys_.insert(job.key).second) return; + if (stop_transition_worker_ || !queued_transition_keys_.insert(job.key).second) return false; if (job.source_tier < queued_by_tier_.size()) ++queued_by_tier_[job.source_tier]; transition_queue_.push_back(std::move(job)); transition_cv_.notify_one(); + return true; +} + +bool PeerPool::TransitionQueued(const std::string& key) { + // A key stays in queued_transition_keys_ until the worker has finished its + // job, so this covers a running job as well as a waiting one. + std::lock_guard lock(transition_mutex_); + return queued_transition_keys_.count(key) != 0; +} + +void PeerPool::DropEvictedCopyLocked(const TransitionJob& job) { + if (migrating_keys_.count(job.key) != 0) return; + auto* source = backends_ == nullptr ? nullptr : backends_->Get(job.source_backend_id); + if (source == nullptr) return; + // One key, under the pool lock: the cost a synchronous drop always had. + source->Evict({job.key}); + RepairAfterDropLocked(job.key); } void PeerPool::StopTransitionWorker() { @@ -220,7 +237,9 @@ void PeerPool::MaybeEnqueueWatermarkOffloadLocked() { } bool PeerPool::WatermarkDrivenLocked(const TransitionJob& job) const { - return job.kind == TierTransitionKind::kOffload && tier_graph_ != nullptr && + // An eviction's demotion was asked for, so the tier's own watermark has no + // say in whether it runs. + return !job.evict && job.kind == TierTransitionKind::kOffload && tier_graph_ != nullptr && tier_graph_->NodeAt(job.source_tier).trigger == PoolOffloadTrigger::kWatermark; } @@ -318,6 +337,7 @@ void PeerPool::TransitionWorkerLoop() { const bool watermark_driven = WatermarkDrivenLocked(job); if (!watermark_driven || tier_graph_->AtOrAbove(job.source_tier, tier.low_watermark)) { const auto result = RunTransition(lock, job); + if (job.evict && !result.success) DropEvictedCopyLocked(job); const bool still_pressured = watermark_driven && tier_graph_->AtOrAbove(job.source_tier, tier.low_watermark); retry = still_pressured && !result.success && job.attempts < 3; @@ -922,21 +942,22 @@ void PeerPool::RepairAfterDropLocked(const std::string& key) { draining_sources_.erase(key); } -std::vector PeerPool::Evict(const std::vector& keys, PoolEvictMode mode) { +std::vector PeerPool::Evict(const std::vector& keys, + PoolEvictMode mode) { std::vector requests(keys.size()); for (size_t i = 0; i < keys.size(); ++i) requests[i].key = keys[i]; return Evict(requests, mode); } -std::vector PeerPool::Evict(const std::vector& requests, - PoolEvictMode mode) { +std::vector PeerPool::Evict(const std::vector& requests, + PoolEvictMode mode) { // Shared for the whole call: the drops below run without operation_mutex_, // and ClearLocal must not wipe the backends between a drop being decided and // being carried out. std::shared_lock lifecycle_lock(lifecycle_mutex_); - std::vector out; + std::vector out; out.reserve(requests.size()); - for (const auto& request : requests) out.push_back(EvictResult{request.key, 0}); + for (const auto& request : requests) out.push_back(PoolEvictResult{request.key}); if (backends_ == nullptr || requests.empty()) return out; // A copy to free by deleting it: (request index, backend). Decided under @@ -951,7 +972,10 @@ std::vector PeerPool::Evict(const std::vector& re // A copy already owns this key. Freeing nothing makes the caller retry, // which is the same contract as a read lease delaying a source delete, and // it keeps a discard from racing a target commit that would resurrect it. - if (migrating_keys_.count(request.key) != 0) continue; + if (migrating_keys_.count(request.key) != 0) { + if (mode == PoolEvictMode::kReclaim) out[i].outcome = PoolEvictOutcome::kInFlight; + continue; + } // A discard has to leave no copy behind, so it ignores scope, skips // demotion and drops the key everywhere. if (mode == PoolEvictMode::kDiscard) { @@ -996,27 +1020,27 @@ std::vector PeerPool::Evict(const std::vector& re } } - // One at a time: RunTransition releases the lock for the byte copy itself. + // Queued, never run here: the copy is the transition worker's job, so an + // EvictKey RPC returns without waiting on it however many bytes it names. + // The worker runs an eviction's demotion regardless of the tier's own + // watermark, frees the source when the copy lands, and drops the copy + // instead if nothing downstream can take it (DropEvictedCopyLocked). for (const auto& [index, source_id] : demotions) { const auto tier_index = tier_graph_->TierIndexForBackendId(source_id); if (!tier_index.has_value()) { drops.emplace_back(index, source_id); continue; } - auto migrated = RunTransition(operation_lock, {TierTransitionKind::kOffload, - requests[index].key, source_id, *tier_index}); - if (migrated.success) { - out[index].bytes_freed = migrated.bytes_freed; - continue; + TransitionJob job{TierTransitionKind::kOffload, requests[index].key, source_id, *tier_index}; + job.evict = true; + if (EnqueueTransition(std::move(job))) { + out[index].outcome = PoolEvictOutcome::kDemotionQueued; + } else if (TransitionQueued(requests[index].key)) { + // Another job already owns the key -- a watermark offload, or this + // eviction's own earlier request. It will move or free the copy; + // deleting the source under it could lose the key instead. + out[index].outcome = PoolEvictOutcome::kInFlight; } - // Another transition got to this key first (an earlier RunTransition in - // this loop released the lock): it is moving the copy, which is the better - // outcome. Deleting the source now could land before that copy pins it - // and lose the key outright, so leave it -- 0 freed, the caller retries. - if (migrating_keys_.count(requests[index].key) != 0) continue; - // Nowhere downstream had room, or the copy moved meanwhile: free it by - // deleting it instead. The tier asked for bytes back either way. - drops.emplace_back(index, source_id); } if (drops.empty()) return out; @@ -1035,6 +1059,7 @@ std::vector PeerPool::Evict(const std::vector& re auto results = backend->Evict(keys); for (size_t j = 0; j < indices.size() && j < results.size(); ++j) { out[indices[j]].bytes_freed += results[j].bytes_freed; + if (results[j].bytes_freed > 0) out[indices[j]].outcome = PoolEvictOutcome::kFreed; } } operation_lock.lock(); @@ -1167,36 +1192,56 @@ size_t PeerPool::ReclaimBackend(uint32_t backend_id, const LocalEvictionWatermar if (used <= low_bytes) return 0; local_evict_rounds_.fetch_add(1, std::memory_order_relaxed); - // Free what was over the low watermark at the start -- counted in the bytes - // each offer said it gives back, which is what Capacity() counts -- or until - // usage is back at the low watermark, whichever comes first. The budget is - // the guard against a medium whose reported usage lags its deletes, which + // Cover what was over the low watermark at the start -- counted in the bytes + // each offer said it gives back, which is what Capacity() counts -- or stop + // once usage is back at the low watermark, whichever comes first. The budget + // is the guard against a medium whose reported usage lags its deletes, which // re-reading usage alone would keep draining. + // + // "Covered" includes bytes on their way out: a demotion queued by this pass + // or an earlier one frees its source only once the copy lands, and until + // then the key still sits at the cold end of the medium's order. Counting + // it keeps a pass from queueing more than the overage while the copies run. const uint64_t budget = used - low_bytes; const TierType tier = backend->Tier(); - uint64_t freed_bytes = 0; - size_t freed_keys = 0; - while (freed_bytes < budget) { + uint64_t covered = 0; + size_t evicted_keys = 0; + // Keys this pass already settled, which the medium keeps offering until + // they leave it. The window grows past them instead of re-offering them. + std::unordered_set settled; + size_t window = kLocalEvictBatch; + while (covered < budget) { { std::lock_guard lock(local_evict_mutex_); if (stop_local_evict_) break; } - auto offers = backend->EvictionCandidates(kLocalEvictBatch); + auto offers = backend->EvictionCandidates(window); + const bool exhausted = offers.size() < window; + offers.erase(std::remove_if(offers.begin(), offers.end(), + [&](const EvictionOffer& o) { return settled.count(o.key) != 0; }), + offers.end()); if (offers.empty()) { - local_evict_no_candidate_.fetch_add(1, std::memory_order_relaxed); - MORI_UMBP_DEBUG( - "[PeerPool] local eviction: backend {} over its watermark but offers no " - "candidate (all leased or pinned)", - backend_id); + if (!exhausted) { + window *= 2; // everything offered so far is settled: look further + continue; + } + if (settled.empty()) { + local_evict_no_candidate_.fetch_add(1, std::memory_order_relaxed); + MORI_UMBP_DEBUG( + "[PeerPool] local eviction: backend {} over its watermark but offers no " + "candidate (all leased or pinned)", + backend_id); + } break; } // Only as many of the coldest as the remaining budget needs (at least one), - // so a batch cannot carry the drain past the low watermark. + // and no more than one batch per Evict call. std::vector requests; std::vector offered_bytes; uint64_t taken = 0; for (auto& offer : offers) { - if (!requests.empty() && freed_bytes + taken >= budget) break; + if (requests.size() >= kLocalEvictBatch) break; + if (!requests.empty() && covered + taken >= budget) break; taken += offer.bytes; offered_bytes.push_back(offer.bytes); PoolEvictRequest request; @@ -1206,26 +1251,35 @@ size_t PeerPool::ReclaimBackend(uint32_t backend_id, const LocalEvictionWatermar requests.push_back(std::move(request)); } // The same entry point the master's EvictKey uses: a candidate whose tier - // has a downstream is demoted, anything else is dropped from this backend. + // has a downstream is queued for demotion, anything else is dropped from + // this backend. const auto results = Evict(requests, PoolEvictMode::kReclaim); - size_t batch_keys = 0; + bool progressed = false; for (size_t i = 0; i < results.size() && i < offered_bytes.size(); ++i) { - if (results[i].bytes_freed == 0) continue; - ++batch_keys; - freed_bytes += offered_bytes[i]; + settled.insert(results[i].key); + switch (results[i].outcome) { + case PoolEvictOutcome::kFreed: + case PoolEvictOutcome::kDemotionQueued: + ++evicted_keys; + [[fallthrough]]; + case PoolEvictOutcome::kInFlight: + covered += offered_bytes[i]; + progressed = true; + break; + case PoolEvictOutcome::kNone: + break; + } } - if (batch_keys == 0) { - // Every candidate was mid-transition or became busy since it was - // offered. Spinning would only re-offer the same keys; the next wake or - // poll tries again. + if (!progressed) { + // Every candidate became busy since it was offered (a read lease or pin + // landed in between). The next wake or poll tries again. local_evict_stalled_.fetch_add(1, std::memory_order_relaxed); break; } - freed_keys += batch_keys; if (UsedBytes(backend->Capacity()) <= low_bytes) break; } - local_evict_keys_.fetch_add(freed_keys, std::memory_order_relaxed); - return freed_keys; + local_evict_keys_.fetch_add(evicted_keys, std::memory_order_relaxed); + return evicted_keys; } void PeerPool::ClearLocal() { diff --git a/src/umbp/doc/runtime-env-vars.md b/src/umbp/doc/runtime-env-vars.md index 384a58eb6..5126c6830 100644 --- a/src/umbp/doc/runtime-env-vars.md +++ b/src/umbp/doc/runtime-env-vars.md @@ -60,6 +60,7 @@ Read by the **master process** (`bin/master_main.cpp` via | `UMBP_LEASE_DURATION_SEC` | `2` | sec | Master-side read-lease length granted by `Router::RouteGet`: `IsLeased()` keys are skipped by the eviction scan, keeping a key alive from the moment the master returns its location until the reader connects to the owning peer. Only needs to cover the master→reader gRPC round trip + reach the peer (the actual RDMA transfer is covered peer-side by `UMBP_DRAM_READ_LEASE_MS`), so seconds is already generous; larger values pin actively-read (hot) keys against eviction. | | `UMBP_HEARTBEAT_INTERVAL_DIVISOR` | `2` | count | Recommended client heartbeat interval = `heartbeat_ttl / divisor`. `min_allowed=1` guards against div-by-zero. Read by the master and echoed in `RegisterClientResponse.heartbeat_interval_ms`. | | `UMBP_EVICTKEY_DEADLINE_MS` | `1000` | ms | Per-call gRPC deadline applied to outbound `EvictKey` RPCs from `MasterPeerStubPool`. | +| `UMBP_EVICT_MAX_BYTES_PER_ROUND` | `0` | bytes | Most one eviction round may free from one `(node, tier)`. `0` = the whole overage (usage above the low watermark), at most `(1 − low_watermark)` of the tier. Set it to throttle eviction work, e.g. to bound how much a demoting tier copies downstream per round. Replaces the former fixed 32-candidate row cap. | | `UMBP_HIT_INDEX_TTL_SEC` | `7200` | sec | External KV hit-count entry TTL. A hash with no counted match for longer than this is removed from the hit index. | | `UMBP_HIT_INDEX_GC_INTERVAL_SEC` | `60` | sec | External KV hit-count GC sweep interval. | | `UMBP_HIT_QUERY_MAX_BATCH` | `4096` | count | Maximum hashes accepted by one `GetExternalKvHitCounts` request. Oversized requests return gRPC `INVALID_ARGUMENT`; the server does not truncate. | diff --git a/src/umbp/include/umbp/common/env_time.h b/src/umbp/include/umbp/common/env_time.h index 7d17a1b81..87b5abd0c 100644 --- a/src/umbp/include/umbp/common/env_time.h +++ b/src/umbp/include/umbp/common/env_time.h @@ -146,6 +146,18 @@ inline uint32_t GetEnvUint32(const char* name, uint32_t def, uint32_t min_allowe return static_cast(v); } +// Byte counts and other values past uint32. The parser is signed, so the +// ceiling is INT64_MAX -- far beyond any byte count this reads. +inline uint64_t GetEnvUint64(const char* name, uint64_t def, uint64_t min_allowed = 0) { + const int64_t v = env_time_detail::ResolveEnvInt(name, static_cast(def), + static_cast(min_allowed)); + if (v < 0) { + env_time_detail::WarnOnce(name, "negative", std::getenv(name)); + return def; + } + return static_cast(v); +} + // Resolve a string-enum env var against a fixed set of allowed values. // - Unset / empty / whitespace-only -> return `def` silently. // - After trimming leading/trailing whitespace, exact (case-sensitive, diff --git a/src/umbp/include/umbp/distributed/config.h b/src/umbp/include/umbp/distributed/config.h index d5ff0bc0c..22b420ef1 100644 --- a/src/umbp/include/umbp/distributed/config.h +++ b/src/umbp/include/umbp/distributed/config.h @@ -99,16 +99,24 @@ struct EvictionConfig { double low_watermark = 0.7; std::chrono::seconds check_interval{5}; std::chrono::seconds lease_duration{2}; - size_t evict_batch_size = 32; - // Only timing fields are env-overridable here; watermarks and batch size - // have dedicated tuning paths and are intentionally excluded. + // A round frees one (node, tier)'s overage -- usage above its low watermark + // -- in BYTES. This caps what one round may evict from one (node, tier); + // 0 means the whole overage, which is already at most (1 - low_watermark) + // of the tier. Set it to throttle eviction work, e.g. to bound how much a + // demoting tier copies downstream per round. + uint64_t max_evict_bytes_per_round = 0; + + // Watermarks are not env-overridable (they have dedicated tuning paths); + // timing and the round's byte limit are. static EvictionConfig FromEnvironment() { EvictionConfig cfg; cfg.check_interval = GetEnvSeconds("UMBP_EVICTION_CHECK_INTERVAL_SEC", cfg.check_interval, /*min_allowed=*/1); cfg.lease_duration = GetEnvSeconds("UMBP_LEASE_DURATION_SEC", cfg.lease_duration, /*min_allowed=*/1); + cfg.max_evict_bytes_per_round = + GetEnvUint64("UMBP_EVICT_MAX_BYTES_PER_ROUND", cfg.max_evict_bytes_per_round); return cfg; } }; diff --git a/src/umbp/include/umbp/distributed/peer/backend/page_backend.h b/src/umbp/include/umbp/distributed/peer/backend/page_backend.h index a3a61d925..cf1ca1f50 100644 --- a/src/umbp/include/umbp/distributed/peer/backend/page_backend.h +++ b/src/umbp/include/umbp/distributed/peer/backend/page_backend.h @@ -537,6 +537,10 @@ class PageBackend : public MediumBackend { struct PinState { uint64_t token = 0; std::chrono::steady_clock::time_point acquired_at; // for long-running warning only + // Taken by AcquireMigrationRead: the pool is moving this key off the + // medium, so it still counts as an eviction candidate (see + // EvictionCandidates). + bool migration = false; }; std::unordered_map pins_; uint64_t next_pin_token_ = 1; diff --git a/src/umbp/include/umbp/distributed/pool/peer_pool.h b/src/umbp/include/umbp/distributed/pool/peer_pool.h index 25dd41ac8..b2aa93717 100644 --- a/src/umbp/include/umbp/distributed/pool/peer_pool.h +++ b/src/umbp/include/umbp/distributed/pool/peer_pool.h @@ -72,6 +72,24 @@ struct PoolResolvedEntry { ResolvedEntry resolved; }; +// What PeerPool::Evict did with one request. A demotion is never run inside +// the call: it is queued to the pool's transition worker, so an EvictKey RPC +// returns without waiting on a copy. +enum class PoolEvictOutcome { + kNone, // nothing freed: no such copy, or a read lease / pin holds it + kFreed, // the copy was dropped; bytes_freed says how much + kDemotionQueued, // this call queued the copy's demotion; it frees the source when done + kInFlight, // a transition already owned the key; it will move or free it +}; + +struct PoolEvictResult { + std::string key; + // Bytes freed by the time Evict returned. 0 for a queued demotion, whose + // bytes come back when the transition completes. + uint64_t bytes_freed = 0; + PoolEvictOutcome outcome = PoolEvictOutcome::kNone; +}; + // Local watermark eviction counters (see PeerPool::EnableLocalEviction). The // keys and bytes actually freed are not here: every eviction is an Evict() // call on a backend, and InstrumentedBackend already counts those. These are @@ -79,7 +97,7 @@ struct PoolResolvedEntry { struct LocalEvictionMetrics { // Passes that found a backend at or above its high watermark. uint64_t rounds = 0; - // Keys the passes freed (demoted or dropped). + // Keys the passes evicted: dropped, or queued for demotion. uint64_t keys = 0; // Passes the medium offered no candidate for: everything it holds is under // a read lease or pin, or it keeps no eviction order at all. @@ -143,11 +161,14 @@ class PeerPool { // The one routing entry point for eviction. Every policy calls this -- the // master's EvictKey RPC and this pool's own local watermark eviction alike -- // and every byte it frees is freed by the medium's MediumBackend::Evict, - // directly or as the last step of a demotion. One result per request, in - // request order; bytes_freed is what the named copy released. - std::vector Evict(const std::vector& requests, PoolEvictMode mode); + // directly or as the last step of a demotion. A copy that drops is freed + // before this returns; a copy that demotes is only QUEUED (see + // PoolEvictOutcome), so the call never waits on a byte copy. One result per + // request, in request order. + std::vector Evict(const std::vector& requests, + PoolEvictMode mode); // Every key unscoped: freed from the medium that currently holds it. - std::vector Evict(const std::vector& keys, PoolEvictMode mode); + std::vector Evict(const std::vector& keys, PoolEvictMode mode); void ClearLocal(); std::vector DrainPendingEvents(); std::vector SnapshotOwnedKeysForFullSync(); @@ -176,7 +197,8 @@ class PeerPool { bool EnableLocalEviction(uint32_t backend_id, double high_watermark, double low_watermark); LocalEvictionMetrics LocalEvictionStats() const; // Test seam: one synchronous pass over every enabled backend, as the worker - // runs it. Returns the number of keys freed. + // runs it. Returns the number of keys evicted: dropped, or queued for + // demotion (which completes on the transition worker afterwards). size_t RunLocalEvictionOnceForTest() { return LocalEvictionPass(); } private: @@ -192,6 +214,11 @@ class PeerPool { uint32_t source_backend_id = BackendRegistry::kMaxBackends; LogicalTierGraph::TierIndex source_tier = 0; uint8_t attempts = 0; + // Queued by Evict(): a demotion someone asked for -- the master's EvictKey + // or local eviction -- not one the tier's own watermark started. It runs + // whatever the tier's usage is by then, and if nothing downstream can take + // the key the copy is dropped instead: the eviction still has to free it. + bool evict = false; }; // What the copy phase needs, resolved while the pool lock is still held. @@ -201,9 +228,17 @@ class PeerPool { std::vector targets; }; - void EnqueueTransition(TransitionJob job); + // False when the key already has a job (queued or running) or the worker is + // stopping; the job is not queued then. + bool EnqueueTransition(TransitionJob job); + // Whether a transition job owns `key` right now, queued or running. + bool TransitionQueued(const std::string& key); void TransitionWorkerLoop(); void StopTransitionWorker(); + // An eviction's demotion could not move the copy (no room downstream, or the + // copy moved meanwhile): drop it, as a synchronous eviction would have -- + // unless another transition owns the key, which would rather move it. + void DropEvictedCopyLocked(const TransitionJob& job); struct LocalEvictionWatermarks { double high = 0.0; @@ -214,10 +249,10 @@ class PeerPool { void WakeLocalEviction(); void LocalEvictionLoop(); void StopLocalEviction(); - // One pass: every enabled backend, lowest tier first. Returns keys freed. + // One pass: every enabled backend, lowest tier first. Returns keys evicted. size_t LocalEvictionPass(); - // Frees one backend down to its low watermark if it is at or above its high - // one. Returns keys freed. + // Brings one backend down to its low watermark if it is at or above its + // high one -- counting demotions still in flight -- and returns keys evicted. size_t ReclaimBackend(uint32_t backend_id, const LocalEvictionWatermarks& watermarks); void TouchLocked(const std::string& key); diff --git a/tests/cpp/umbp/distributed/test_env_time.cpp b/tests/cpp/umbp/distributed/test_env_time.cpp index 622a74ab4..630b1c137 100644 --- a/tests/cpp/umbp/distributed/test_env_time.cpp +++ b/tests/cpp/umbp/distributed/test_env_time.cpp @@ -40,6 +40,7 @@ using mori::umbp::GetEnvMicroseconds; using mori::umbp::GetEnvMilliseconds; using mori::umbp::GetEnvSeconds; using mori::umbp::GetEnvUint32; +using mori::umbp::GetEnvUint64; using mori::umbp::ResetEnvWarnStateForTesting; constexpr const char* kName = "UMBP_TEST_ENV_TIME_XYZ"; @@ -107,6 +108,21 @@ TEST_F(EnvTimeTest, ZeroIsAllowedWhenMinIsZero) { EXPECT_EQ(GetEnvUint32(kName, 7), 0u); } +TEST_F(EnvTimeTest, Uint64ParsesByteCountsPastUint32) { + // 8 GiB: the eviction byte limits need values a uint32 cannot hold. + ::setenv(kName, "8589934592", 1); + EXPECT_EQ(GetEnvUint64(kName, 1), 8589934592ULL); + ::unsetenv(kName); + EXPECT_EQ(GetEnvUint64(kName, 1ULL << 30), 1ULL << 30); +} + +TEST_F(EnvTimeTest, Uint64RejectsNegativesAndGarbage) { + ::setenv(kName, "-1", 1); + EXPECT_EQ(GetEnvUint64(kName, 5), 5u); + ::setenv(kName, "1G", 1); + EXPECT_EQ(GetEnvUint64(kName, 6), 6u); +} + TEST_F(EnvTimeTest, FallsBackWhenAboveUint32Max) { // A value that exceeds uint32 range must fall back to default. ::setenv(kName, "99999999999", 1); diff --git a/tests/cpp/umbp/distributed/test_master_evict_strategy.cpp b/tests/cpp/umbp/distributed/test_master_evict_strategy.cpp index af8107523..a1534d400 100644 --- a/tests/cpp/umbp/distributed/test_master_evict_strategy.cpp +++ b/tests/cpp/umbp/distributed/test_master_evict_strategy.cpp @@ -72,6 +72,24 @@ TEST(LruMasterEvictStrategy, PicksOldestFirstUntilBudgetMet) { EXPECT_EQ(victims["n1"][1].key, "b"); } +TEST(LruMasterEvictStrategy, ABudgetLargerThanAnyRowCapIsMetInOneRound) { + // The round is sized in bytes now: 1,000 candidates against a budget that + // covers 600 of them yields 600 victims, not the first 32. + LruMasterEvictStrategy strategy; + auto now = Clock::now(); + std::vector candidates; + for (int i = 0; i < 1000; ++i) { + candidates.push_back(MakeCandidate("k" + std::to_string(i), "n1", TierType::DRAM, 64, + now - std::chrono::seconds(1000 - i))); + } + std::unordered_map> budget; + budget["n1"][TierType::DRAM] = 600 * 64; + + auto victims = strategy.SelectVictims(candidates, budget); + ASSERT_EQ(victims["n1"].size(), 600u); + EXPECT_EQ(victims["n1"].front().key, "k0"); // oldest first +} + TEST(LruMasterEvictStrategy, KeyOverBudgetInTwoTiersIsOneVictimPerTier) { // A key in DRAM and SSD, both over budget: one victim per (key, tier), each // carrying the tier its copy was charged to. diff --git a/tests/cpp/umbp/distributed/test_master_eviction_chain.cpp b/tests/cpp/umbp/distributed/test_master_eviction_chain.cpp index 71e6bf5fd..fd45a7f35 100644 --- a/tests/cpp/umbp/distributed/test_master_eviction_chain.cpp +++ b/tests/cpp/umbp/distributed/test_master_eviction_chain.cpp @@ -38,6 +38,7 @@ #include #include +#include #include #include #include @@ -258,13 +259,22 @@ TEST(MasterEvictionChain, OverWatermarkTierDemotesIntoTheColdTier) { EXPECT_EQ(dispatcher.NodeIds().front(), kNodeId); // Every key the master named must have moved down a tier rather than vanished. + // The peer only queues the demotion -- the EvictKey call does not wait on the + // copy -- so the move is observed after the fact. const auto evicted = dispatcher.Dispatched(); ASSERT_FALSE(evicted.empty()); + const bool moved = WaitFor( + [&] { + for (const auto& key : evicted) { + if (registry.Get("hot")->Contains(key)) return false; + } + return true; + }, + std::chrono::seconds(5)); + ASSERT_TRUE(moved) << "a queued demotion never completed"; for (const auto& key : evicted) { - EXPECT_FALSE(registry.Get("hot")->Contains(key)) << key << " still in hot"; EXPECT_TRUE(registry.Get("cold")->Contains(key)) << key << " was dropped, not demoted"; } - EXPECT_GT(dispatcher.FreedBytes(), 0u); // Keys the master did not name are untouched: eviction is bounded by the // budget, so a round must not drain the whole tier. @@ -532,5 +542,94 @@ TEST(MasterEvictionChain, ParkedMoveSourceIsNamedLastAmongColderKeys) { EXPECT_TRUE(registry.Get("hot")->Contains("moved")); } +// --------------------------------------------------------------------------- +// Round size in bytes +// +// A round frees the overage, measured in bytes, rather than a fixed number of +// candidates per bucket. These drive the real EvictionManager against the +// real store and record what it dispatches; nothing is executed. +// --------------------------------------------------------------------------- + +// Records every EvictKey the manager sends, one entry per RPC. +class RecordingDispatcher final : public EvictKeyDispatcher { + public: + void DispatchEvictKey(const std::string& /*node_id*/, const std::string& /*peer_address*/, + std::vector victims) override { + std::lock_guard lock(mutex_); + calls_.push_back(std::move(victims)); + } + std::vector> Calls() const { + std::lock_guard lock(mutex_); + return calls_; + } + size_t Keys() const { + std::lock_guard lock(mutex_); + size_t n = 0; + for (const auto& call : calls_) n += call.size(); + return n; + } + + private: + mutable std::mutex mutex_; + std::vector> calls_; +}; + +std::vector NumberedKeys(size_t n) { + std::vector keys; + for (size_t i = 0; i < n; ++i) keys.push_back("k" + std::to_string(i)); + return keys; +} + +// 256 one-page keys filling a 256-page DRAM bucket: the overage above the 0.7 +// low watermark is 4,916 bytes, i.e. 77 keys -- more than the 32-row cap a +// round used to stop at. +constexpr size_t kFullBucketKeys = 256; +const uint64_t kFullBucketBytes = kFullBucketKeys * kPageSize; +const size_t kOverageKeys = + (kFullBucketBytes - static_cast(kFullBucketBytes * 0.7) + kPageSize - 1) / kPageSize; + +TEST(MasterEvictionBytes, ARoundFreesTheWholeOverageNotThirtyTwoKeys) { + InMemoryMasterMetadataStore store; + PublishNode(&store, NumberedKeys(kFullBucketKeys), kFullBucketBytes, kFullBucketBytes); + + RecordingDispatcher dispatcher; + EvictionManager manager(store, OneSecondRounds(), &dispatcher); + manager.Start(); + ASSERT_TRUE(WaitFor([&] { return !dispatcher.Calls().empty(); }, std::chrono::seconds(8))); + manager.Stop(); + + // A round's victims go out in calls bounded only by gRPC message size, so + // these few short keys are one call. + const auto first = dispatcher.Calls().front(); + ASSERT_GT(kOverageKeys, 32u); + EXPECT_EQ(first.size(), kOverageKeys) << "the round did not cover its byte budget"; + for (const auto& victim : first) EXPECT_EQ(victim.tier, TierType::DRAM); +} + +TEST(MasterEvictionBytes, PerRoundByteLimitThrottlesTheRound) { + InMemoryMasterMetadataStore store; + PublishNode(&store, NumberedKeys(kFullBucketKeys), kFullBucketBytes, kFullBucketBytes); + + auto config = OneSecondRounds(); + config.max_evict_bytes_per_round = 10 * kPageSize; + RecordingDispatcher dispatcher; + EvictionManager manager(store, config, &dispatcher); + manager.Start(); + ASSERT_TRUE(WaitFor([&] { return !dispatcher.Calls().empty(); }, std::chrono::seconds(8))); + manager.Stop(); + + EXPECT_EQ(dispatcher.Calls().front().size(), 10u); +} + +TEST(MasterEvictionBytes, PerRoundByteLimitIsReadFromTheEnvironment) { + ::setenv("UMBP_EVICT_MAX_BYTES_PER_ROUND", "8589934592", 1); + const auto config = EvictionConfig::FromEnvironment(); + ::unsetenv("UMBP_EVICT_MAX_BYTES_PER_ROUND"); + EXPECT_EQ(config.max_evict_bytes_per_round, 8589934592ULL); + + const auto defaults = EvictionConfig::FromEnvironment(); + EXPECT_EQ(defaults.max_evict_bytes_per_round, 0u); +} + } // namespace } // namespace mori::umbp diff --git a/tests/cpp/umbp/distributed/test_peer_pool.cpp b/tests/cpp/umbp/distributed/test_peer_pool.cpp index bb010a6cd..2f3007b24 100644 --- a/tests/cpp/umbp/distributed/test_peer_pool.cpp +++ b/tests/cpp/umbp/distributed/test_peer_pool.cpp @@ -112,6 +112,70 @@ bool WaitForKey(BackendRegistry* registry, const std::string& backend, const std return registry->Get(backend)->Contains(key); } +// The mirror image: an eviction's demotion runs on the transition worker, so +// the source copy leaving is observed rather than awaited. +bool WaitForKeyGone(BackendRegistry* registry, const std::string& backend, const std::string& key) { + for (int attempt = 0; attempt < 500 && registry->Get(backend)->Contains(key); ++attempt) { + std::this_thread::sleep_for(std::chrono::milliseconds(2)); + } + return !registry->Get(backend)->Contains(key); +} + +// A LocalCopyEngine that holds every copy until Release(), so a test can see +// what an eviction does while its demotion is still running. +class GatedEngine final : public TransferEngine { + public: + ~GatedEngine() override { Release(); } + + const char* Name() const override { return "gated"; } + TransferRef RegisterMemory(void* base, size_t size, mori::io::MemoryLocationType loc, + int device) override { + return inner_.RegisterMemory(base, size, loc, device); + } + void Deregister(const TransferRef& ref) override { inner_.Deregister(ref); } + bool CanHandle(const TransferRef& src, const TransferRef& dst) const override { + return inner_.CanHandle(src, dst); + } + TransferPlanSet Plan(const std::vector& items) const override { + return inner_.Plan(items); + } + std::unique_ptr Submit(std::vector plans) override { + { + std::unique_lock lock(mutex_); + ++held_; + cv_.notify_all(); + cv_.wait(lock, [&] { return open_; }); + } + return inner_.Submit(std::move(plans)); + } + + // Lets every held copy (and every later one) through. + void Release() { + std::lock_guard lock(mutex_); + open_ = true; + cv_.notify_all(); + } + // True once at least `n` copies have reached Submit and are being held. + bool WaitForHeld(int n) { + std::unique_lock lock(mutex_); + return cv_.wait_for(lock, std::chrono::seconds(5), [&] { return held_ >= n; }); + } + + private: + LocalCopyEngine inner_; + std::mutex mutex_; + std::condition_variable cv_; + bool open_ = false; + int held_ = 0; +}; + +// Releases the gate on scope exit, so a failed assertion cannot leave the +// transition worker blocked while the pool it belongs to is torn down. +struct ReleaseOnExit { + GatedEngine* engine; + ~ReleaseOnExit() { engine->Release(); } +}; + class NoSpaceBackend final : public MockBackend { public: explicit NoSpaceBackend(TierType tier) : MockBackend(tier) {} @@ -572,10 +636,12 @@ TEST(PeerPool, OnEvictMigratesBytesToConfiguredBackend) { .front() .commit.success); + // The demotion is queued, not run inside the call: nothing is freed yet. auto evicted = pool.Evict({"move"}, PoolEvictMode::kReclaim); ASSERT_EQ(evicted.size(), 1u); - EXPECT_EQ(evicted.front().bytes_freed, payload.size()); - EXPECT_FALSE(registry.Get("hot")->Contains("move")); + EXPECT_EQ(evicted.front().outcome, PoolEvictOutcome::kDemotionQueued); + EXPECT_EQ(evicted.front().bytes_freed, 0u); + ASSERT_TRUE(WaitForKeyGone(®istry, "hot", "move")); EXPECT_TRUE(registry.Get("cold")->Contains("move")); EXPECT_EQ(pool.PlacementBackend("move"), std::optional{registry.BackendId(registry.Get("cold"))}); @@ -592,11 +658,13 @@ TEST(PeerPool, OnEvictMigratesBytesToConfiguredBackend) { TEST(PeerPool, EvictionRetryDrainsLeasedSourceWithoutDeletingTarget) { constexpr uint64_t kPageSize = 64; - constexpr auto kShortLease = std::chrono::milliseconds(1); + // Long enough to still hold when the queued demotion runs on the worker, so + // its source delete is refused and the source parks as draining. + constexpr auto kLease = std::chrono::milliseconds(300); LocalCopyEngine engine; BackendRegistry registry; - RegisterPageBackend(®istry, &engine, "hot", TierType::DRAM, kPageSize, 2, kShortLease); - RegisterPageBackend(®istry, &engine, "cold", TierType::SSD, kPageSize, 2, kShortLease); + RegisterPageBackend(®istry, &engine, "hot", TierType::DRAM, kPageSize, 2, kLease); + RegisterPageBackend(®istry, &engine, "cold", TierType::SSD, kPageSize, 2, kLease); auto compiled = LogicalTierGraph::Compile(HotColdTiers(PoolOffloadTrigger::kOnEvict), registry); ASSERT_TRUE(compiled.ok()) << compiled.error; PeerPool pool(®istry, MakeTieredPlacementPolicy(compiled.graph), &engine); @@ -606,10 +674,10 @@ TEST(PeerPool, EvictionRetryDrainsLeasedSourceWithoutDeletingTarget) { auto first = pool.Evict({"leased"}, PoolEvictMode::kReclaim); ASSERT_EQ(first.front().bytes_freed, 0u); - EXPECT_TRUE(registry.Get("hot")->Contains("leased")); - EXPECT_TRUE(registry.Get("cold")->Contains("leased")); + ASSERT_TRUE(WaitForKey(®istry, "cold", "leased")); + EXPECT_TRUE(registry.Get("hot")->Contains("leased")) << "the lease did not hold the source"; - std::this_thread::sleep_for(std::chrono::milliseconds(5)); + std::this_thread::sleep_for(kLease + std::chrono::milliseconds(50)); auto retry = pool.Evict({"leased"}, PoolEvictMode::kReclaim); EXPECT_EQ(retry.front().bytes_freed, 64u); EXPECT_FALSE(registry.Get("hot")->Contains("leased")); @@ -683,10 +751,11 @@ TEST(PeerPool, TierScopedEvictFreesOnlyThatMediumsCopy) { EXPECT_TRUE(registry.Get("hot")->Contains("a")); EXPECT_EQ(pool.PlacementBackend("a"), std::optional{hot}); - // DRAM scope: cold already holds the key, so DRAM frees without a copy. + // DRAM scope: the queued demotion finds cold already holding the key, so it + // frees DRAM without a copy. evicted = pool.Evict({PoolEvictRequest{"b", TierType::DRAM}}, PoolEvictMode::kReclaim); - EXPECT_EQ(evicted.front().bytes_freed, 64u); - EXPECT_FALSE(registry.Get("hot")->Contains("b")); + EXPECT_EQ(evicted.front().outcome, PoolEvictOutcome::kDemotionQueued); + ASSERT_TRUE(WaitForKeyGone(®istry, "hot", "b")); EXPECT_EQ(pool.PlacementBackend("b"), std::optional{cold}); EXPECT_TRUE(pool.BatchResolve({"b"}, false).front().resolved.found); } @@ -731,8 +800,10 @@ TEST(PeerPool, ReclaimDemotesOnAWatermarkTierToo) { CommitKey(&pool, "k"); auto evicted = pool.Evict({PoolEvictRequest{"k", TierType::DRAM}}, PoolEvictMode::kReclaim); - EXPECT_EQ(evicted.front().bytes_freed, 64u); - EXPECT_FALSE(registry.Get("hot")->Contains("k")); + EXPECT_EQ(evicted.front().outcome, PoolEvictOutcome::kDemotionQueued); + // An eviction's demotion runs even though the tier is far under its own + // watermark -- the watermark governs proactive offload, not this. + ASSERT_TRUE(WaitForKeyGone(®istry, "hot", "k")); EXPECT_TRUE(registry.Get("cold")->Contains("k")) << "a watermark tier deleted instead of demoting"; } @@ -755,14 +826,43 @@ TEST(PeerPool, DemotionWithNoRoomDownstreamDropsTheCopy) { ASSERT_TRUE(cold->BatchCommit({CommitRequest{filler.slot_id, "filler"}}).front().success); CommitKey(&pool, "k"); + // Queued like any demotion; the worker finds no room and drops the copy. auto evicted = pool.Evict({PoolEvictRequest{"k", TierType::DRAM}}, PoolEvictMode::kReclaim); - EXPECT_EQ(evicted.front().bytes_freed, 64u); - EXPECT_FALSE(registry.Get("hot")->Contains("k")); + EXPECT_EQ(evicted.front().outcome, PoolEvictOutcome::kDemotionQueued); + ASSERT_TRUE(WaitForKeyGone(®istry, "hot", "k")) << "a failed demotion left the copy in place"; EXPECT_FALSE(cold->Contains("k")); EXPECT_TRUE(cold->Contains("filler")); EXPECT_FALSE(pool.PlacementBackend("k").has_value()); } +TEST(PeerPool, EvictReturnsBeforeTheDemotionCopyRuns) { + // The point of queueing: an EvictKey RPC must not wait on a byte copy, + // however large. The engine holds the copy; Evict has already returned. + constexpr uint64_t kPageSize = 64; + GatedEngine engine; + BackendRegistry registry; + RegisterPageBackend(®istry, &engine, "hot", TierType::DRAM, kPageSize, 4); + RegisterPageBackend(®istry, &engine, "cold", TierType::SSD, kPageSize, 4); + auto compiled = LogicalTierGraph::Compile(HotColdTiers(PoolOffloadTrigger::kOnEvict), registry); + ASSERT_TRUE(compiled.ok()) << compiled.error; + PeerPool pool(®istry, MakeTieredPlacementPolicy(compiled.graph), &engine); + ReleaseOnExit release{&engine}; + CommitKey(&pool, "k"); + + auto first = pool.Evict({PoolEvictRequest{"k", TierType::DRAM}}, PoolEvictMode::kReclaim); + EXPECT_EQ(first.front().outcome, PoolEvictOutcome::kDemotionQueued); + ASSERT_TRUE(engine.WaitForHeld(1)) << "the demotion never reached the copy"; + // Mid-copy: the source is untouched, and asking again does not queue twice. + EXPECT_TRUE(registry.Get("hot")->Contains("k")); + EXPECT_FALSE(registry.Get("cold")->Contains("k")); + auto again = pool.Evict({PoolEvictRequest{"k", TierType::DRAM}}, PoolEvictMode::kReclaim); + EXPECT_EQ(again.front().outcome, PoolEvictOutcome::kInFlight); + + engine.Release(); + ASSERT_TRUE(WaitForKeyGone(®istry, "hot", "k")); + EXPECT_TRUE(registry.Get("cold")->Contains("k")); +} + TEST(PeerPool, WatermarkMigratesCommittedKey) { // Unlike the other tier tests, this one asserts on utilization, so the owned // buffer must be a whole number of host allocation granules: a smaller @@ -1150,6 +1250,37 @@ TEST(PeerPoolLocalEviction, ATierWithADownstreamDemotesInsteadOfDropping) { } } +TEST(PeerPoolLocalEviction, QueuedDemotionsCountTowardTheBudget) { + // A queued demotion frees its source only when the copy lands, so until then + // the tier still reads as full. The worker must count those bytes as on + // their way out -- otherwise every pass would queue the next coldest keys + // until the whole tier was in flight. + GatedEngine engine; + BackendRegistry registry; + RegisterPageBackend(®istry, &engine, "hot", TierType::DRAM, kHostPage, 10); + RegisterPageBackend(®istry, &engine, "cold", TierType::SSD, kHostPage, 20); + auto compiled = LogicalTierGraph::Compile(HotColdTiers(PoolOffloadTrigger::kOnEvict), registry); + ASSERT_TRUE(compiled.ok()) << compiled.error; + PeerPool pool(®istry, MakeTieredPlacementPolicy(compiled.graph), &engine); + ReleaseOnExit release{&engine}; + ASSERT_TRUE(pool.EnableLocalEviction(registry.BackendId(registry.Get("hot")), 0.9, 0.5)); + + for (int i = 0; i < 9; ++i) CommitKey(&pool, "k" + std::to_string(i)); + ASSERT_TRUE(engine.WaitForHeld(1)) << "no demotion was queued"; + // Several more passes while every copy is held: still exactly the overage. + for (int i = 0; i < 3; ++i) pool.RunLocalEvictionOnceForTest(); + std::this_thread::sleep_for(std::chrono::milliseconds(250)); // let polls run too + EXPECT_EQ(pool.LocalEvictionStats().keys, 4u) << "queued past the overage"; + EXPECT_EQ(registry.Get("hot")->OwnedKeyCount(), 9u) << "a copy completed while held"; + + engine.Release(); + auto* hot = registry.Get("hot"); + ASSERT_TRUE(WaitUntil([&] { return hot->OwnedKeyCount() <= 5; })); + for (int i = 0; i < 4; ++i) { + EXPECT_TRUE(registry.Get("cold")->Contains("k" + std::to_string(i))) << i; + } +} + TEST(PeerPoolLocalEviction, NeverDeletesWhatAWatermarkOffloadWouldMove) { // Regression: the old in-commit round deleted the cold keys a watermark // offload was about to move. Now both move them, so every key survives. From f19c52b4812d60d9ff00cb473443fbc0ec9eceb2 Mon Sep 17 00:00:00 2001 From: TianDi101 Date: Tue, 13 Oct 2026 05:19:16 +0000 Subject: [PATCH 08/19] test(umbp): tighten the byte-budget/async-demotion comments; fold its tests Comments: drop history and restated rationale (the commit message has it). Tests, same coverage with fewer of them: - The two GetEnvUint64 tests become one. - Drop LruMasterEvictStrategy.ABudgetLargerThanAnyRowCapIsMetInOneRound: the strategy never had a row cap (the manager did), and it is unchanged. - The three MasterEvictionBytes tests become one, which also reads the limit through UMBP_EVICT_MAX_BYTES_PER_ROUND. - EvictReturnsBeforeTheDemotionCopyRuns folds into OnEvictMigratesBytesToConfiguredBackend, and QueuedDemotionsCountTowardTheBudget into the local-eviction demotion test (now ATierWithADownstreamDemotesJustTheOverage). - GatedEngine's hold times out after 10 s, so it needs no ReleaseOnExit guard to keep a failed test from hanging teardown. Co-Authored-By: Claude Opus 5.5 --- .../distributed/master/eviction_manager.cpp | 14 +-- .../distributed/peer/backend/page_backend.cpp | 5 +- src/umbp/distributed/peer/peer_service.cpp | 8 +- src/umbp/distributed/pool/peer_pool.cpp | 36 ++---- src/umbp/doc/runtime-env-vars.md | 2 +- src/umbp/include/umbp/distributed/config.h | 10 +- .../distributed/peer/backend/page_backend.h | 4 +- .../include/umbp/distributed/pool/peer_pool.h | 45 +++---- tests/cpp/umbp/distributed/test_env_time.cpp | 10 +- .../test_master_evict_strategy.cpp | 18 --- .../test_master_eviction_chain.cpp | 80 +++---------- tests/cpp/umbp/distributed/test_peer_pool.cpp | 112 +++++------------- 12 files changed, 92 insertions(+), 252 deletions(-) diff --git a/src/umbp/distributed/master/eviction_manager.cpp b/src/umbp/distributed/master/eviction_manager.cpp index d97bb52be..968e64cb7 100644 --- a/src/umbp/distributed/master/eviction_manager.cpp +++ b/src/umbp/distributed/master/eviction_manager.cpp @@ -122,11 +122,7 @@ void EvictionManager::RunOnce() { if (bytes_to_free.empty()) return; - // A round's size is a byte budget: the overage itself, optionally capped by - // max_evict_bytes_per_round. It used to be a row count (32 candidates per - // bucket), which freed a few MB per round however far over the tier was -- - // a 512 GiB tier's 0.9 -> 0.7 band took hours to drain -- while every round - // still paid the store's full scan. + // A round frees each bucket's overage in bytes, optionally capped per round. if (config_.max_evict_bytes_per_round > 0) { for (auto& [ntk, bytes] : bytes_to_free) { bytes = std::min(bytes, config_.max_evict_bytes_per_round); @@ -140,9 +136,8 @@ void EvictionManager::RunOnce() { // Ask the store for eviction-eligible candidates in the overloaded buckets. // The store is policy-neutral: it returns rows ordered by the hint (LRU here, // so an indexed backend can push the ordering down) but makes no eviction - // decision and never sees the byte budget -- the strategy trims to it. No - // row cap: both stores read every eligible row of a bucket before capping, - // so a cap would only discard candidates the budget may need. + // decision and never sees the byte budget; the strategy trims to it, so + // there is no row cap. std::vector buckets; buckets.reserve(bytes_to_free.size()); for (const auto& [ntk, _bytes] : bytes_to_free) buckets.push_back(ntk); @@ -213,8 +208,7 @@ void EvictionManager::RunOnce() { // // Sized on the keys alone: the parallel tier field is a packed enum of one // byte per key, inside the per-item overhead GrpcMaxItemsPerBatch already - // reserves. Nothing else bounds a chunk: the peer queues any demotion and - // returns, so a call's length does not grow with the bytes it names. + // reserves. Bytes don't lengthen a call: the peer only queues demotions. std::vector keys; keys.reserve(victims.size()); for (const auto& victim : victims) keys.push_back(victim.key); diff --git a/src/umbp/distributed/peer/backend/page_backend.cpp b/src/umbp/distributed/peer/backend/page_backend.cpp index dbaeec665..edf2ea65e 100644 --- a/src/umbp/distributed/peer/backend/page_backend.cpp +++ b/src/umbp/distributed/peer/backend/page_backend.cpp @@ -831,9 +831,8 @@ std::vector PageBackend::EvictionCandidates(size_t max_count) { // The two protections Evict() applies: an in-flight read owns the pages // until its lease expires, a copy pin until the copy is done. Offering // them would only make the caller's batch free less than it asked for. - // A migration pin is the exception: the key is already on its way out, - // and the caller has to see it to count its bytes as in flight rather - // than reach past it for one more victim. + // Except a migration pin: the key is already leaving, and the caller + // counts its bytes as in flight. if (HasActiveReadLeaseLocked(owned_it->second, now)) continue; if (auto pin = pins_.find(*it); pin != pins_.end() && !pin->second.migration) continue; // Whole pages: that is what the allocator gives back, and what Capacity() diff --git a/src/umbp/distributed/peer/peer_service.cpp b/src/umbp/distributed/peer/peer_service.cpp index a51e10b02..00b6fe708 100644 --- a/src/umbp/distributed/peer/peer_service.cpp +++ b/src/umbp/distributed/peer/peer_service.cpp @@ -269,11 +269,9 @@ class PeerServiceServer::UMBPPeerServiceImpl final : public ::umbp::UMBPPeer::Se // tier per key: the pool frees exactly that medium's copy (demoting it // where its tier has a downstream) and leaves any other copy alone. An // older master sends no tiers; each key is then freed from the medium that - // currently holds it. A demotion is only queued here -- the pool's - // transition worker copies the bytes -- so this returns without waiting on - // any copy, and a queued key reports 0 bytes freed. bytes_freed is - // informational: the master sizes its next round off the heartbeat's - // capacity, not off this reply. + // currently holds it. Demotions are only queued, so a demoted key reports + // 0 bytes freed; bytes_freed is informational anyway -- the master sizes + // its next round off the heartbeat's capacity, not off this reply. const int n = request->keys_size(); if (n == 0) return grpc::Status::OK; const bool scoped = request->tiers_size() == n; diff --git a/src/umbp/distributed/pool/peer_pool.cpp b/src/umbp/distributed/pool/peer_pool.cpp index d7c82220c..45efe1771 100644 --- a/src/umbp/distributed/pool/peer_pool.cpp +++ b/src/umbp/distributed/pool/peer_pool.cpp @@ -109,8 +109,7 @@ bool PeerPool::EnqueueTransition(TransitionJob job) { } bool PeerPool::TransitionQueued(const std::string& key) { - // A key stays in queued_transition_keys_ until the worker has finished its - // job, so this covers a running job as well as a waiting one. + // Keys leave queued_transition_keys_ only when their job finishes. std::lock_guard lock(transition_mutex_); return queued_transition_keys_.count(key) != 0; } @@ -119,7 +118,6 @@ void PeerPool::DropEvictedCopyLocked(const TransitionJob& job) { if (migrating_keys_.count(job.key) != 0) return; auto* source = backends_ == nullptr ? nullptr : backends_->Get(job.source_backend_id); if (source == nullptr) return; - // One key, under the pool lock: the cost a synchronous drop always had. source->Evict({job.key}); RepairAfterDropLocked(job.key); } @@ -237,8 +235,7 @@ void PeerPool::MaybeEnqueueWatermarkOffloadLocked() { } bool PeerPool::WatermarkDrivenLocked(const TransitionJob& job) const { - // An eviction's demotion was asked for, so the tier's own watermark has no - // say in whether it runs. + // An eviction's demotion runs whatever the tier's own watermark says. return !job.evict && job.kind == TierTransitionKind::kOffload && tier_graph_ != nullptr && tier_graph_->NodeAt(job.source_tier).trigger == PoolOffloadTrigger::kWatermark; } @@ -1020,11 +1017,9 @@ std::vector PeerPool::Evict(const std::vector } } - // Queued, never run here: the copy is the transition worker's job, so an - // EvictKey RPC returns without waiting on it however many bytes it names. - // The worker runs an eviction's demotion regardless of the tier's own - // watermark, frees the source when the copy lands, and drops the copy - // instead if nothing downstream can take it (DropEvictedCopyLocked). + // Queued, never copied here, so EvictKey never waits on bytes. The worker + // frees the source when the copy lands, or drops it if nothing downstream + // can take it (DropEvictedCopyLocked). for (const auto& [index, source_id] : demotions) { const auto tier_index = tier_graph_->TierIndexForBackendId(source_id); if (!tier_index.has_value()) { @@ -1036,9 +1031,8 @@ std::vector PeerPool::Evict(const std::vector if (EnqueueTransition(std::move(job))) { out[index].outcome = PoolEvictOutcome::kDemotionQueued; } else if (TransitionQueued(requests[index].key)) { - // Another job already owns the key -- a watermark offload, or this - // eviction's own earlier request. It will move or free the copy; - // deleting the source under it could lose the key instead. + // Another job owns the key and will move or free it; deleting the + // source under it could lose the key. out[index].outcome = PoolEvictOutcome::kInFlight; } } @@ -1197,17 +1191,13 @@ size_t PeerPool::ReclaimBackend(uint32_t backend_id, const LocalEvictionWatermar // once usage is back at the low watermark, whichever comes first. The budget // is the guard against a medium whose reported usage lags its deletes, which // re-reading usage alone would keep draining. - // - // "Covered" includes bytes on their way out: a demotion queued by this pass - // or an earlier one frees its source only once the copy lands, and until - // then the key still sits at the cold end of the medium's order. Counting - // it keeps a pass from queueing more than the overage while the copies run. + // Queued demotions count as covered: their source stays until the copy + // lands, and counting it stops a pass from queueing past the overage. const uint64_t budget = used - low_bytes; const TierType tier = backend->Tier(); uint64_t covered = 0; size_t evicted_keys = 0; - // Keys this pass already settled, which the medium keeps offering until - // they leave it. The window grows past them instead of re-offering them. + // The medium keeps offering settled keys until they leave; look past them. std::unordered_set settled; size_t window = kLocalEvictBatch; while (covered < budget) { @@ -1222,7 +1212,7 @@ size_t PeerPool::ReclaimBackend(uint32_t backend_id, const LocalEvictionWatermar offers.end()); if (offers.empty()) { if (!exhausted) { - window *= 2; // everything offered so far is settled: look further + window *= 2; continue; } if (settled.empty()) { @@ -1271,8 +1261,8 @@ size_t PeerPool::ReclaimBackend(uint32_t backend_id, const LocalEvictionWatermar } } if (!progressed) { - // Every candidate became busy since it was offered (a read lease or pin - // landed in between). The next wake or poll tries again. + // Every candidate got leased or pinned since it was offered; the next + // wake or poll tries again. local_evict_stalled_.fetch_add(1, std::memory_order_relaxed); break; } diff --git a/src/umbp/doc/runtime-env-vars.md b/src/umbp/doc/runtime-env-vars.md index 5126c6830..8e7a8ed65 100644 --- a/src/umbp/doc/runtime-env-vars.md +++ b/src/umbp/doc/runtime-env-vars.md @@ -60,7 +60,7 @@ Read by the **master process** (`bin/master_main.cpp` via | `UMBP_LEASE_DURATION_SEC` | `2` | sec | Master-side read-lease length granted by `Router::RouteGet`: `IsLeased()` keys are skipped by the eviction scan, keeping a key alive from the moment the master returns its location until the reader connects to the owning peer. Only needs to cover the master→reader gRPC round trip + reach the peer (the actual RDMA transfer is covered peer-side by `UMBP_DRAM_READ_LEASE_MS`), so seconds is already generous; larger values pin actively-read (hot) keys against eviction. | | `UMBP_HEARTBEAT_INTERVAL_DIVISOR` | `2` | count | Recommended client heartbeat interval = `heartbeat_ttl / divisor`. `min_allowed=1` guards against div-by-zero. Read by the master and echoed in `RegisterClientResponse.heartbeat_interval_ms`. | | `UMBP_EVICTKEY_DEADLINE_MS` | `1000` | ms | Per-call gRPC deadline applied to outbound `EvictKey` RPCs from `MasterPeerStubPool`. | -| `UMBP_EVICT_MAX_BYTES_PER_ROUND` | `0` | bytes | Most one eviction round may free from one `(node, tier)`. `0` = the whole overage (usage above the low watermark), at most `(1 − low_watermark)` of the tier. Set it to throttle eviction work, e.g. to bound how much a demoting tier copies downstream per round. Replaces the former fixed 32-candidate row cap. | +| `UMBP_EVICT_MAX_BYTES_PER_ROUND` | `0` | bytes | Most bytes one eviction round frees from one `(node, tier)`. `0` = its whole overage above the low watermark. Set it to throttle eviction, e.g. how much a demoting tier copies downstream per round. | | `UMBP_HIT_INDEX_TTL_SEC` | `7200` | sec | External KV hit-count entry TTL. A hash with no counted match for longer than this is removed from the hit index. | | `UMBP_HIT_INDEX_GC_INTERVAL_SEC` | `60` | sec | External KV hit-count GC sweep interval. | | `UMBP_HIT_QUERY_MAX_BATCH` | `4096` | count | Maximum hashes accepted by one `GetExternalKvHitCounts` request. Oversized requests return gRPC `INVALID_ARGUMENT`; the server does not truncate. | diff --git a/src/umbp/include/umbp/distributed/config.h b/src/umbp/include/umbp/distributed/config.h index 22b420ef1..462b16034 100644 --- a/src/umbp/include/umbp/distributed/config.h +++ b/src/umbp/include/umbp/distributed/config.h @@ -100,15 +100,11 @@ struct EvictionConfig { std::chrono::seconds check_interval{5}; std::chrono::seconds lease_duration{2}; - // A round frees one (node, tier)'s overage -- usage above its low watermark - // -- in BYTES. This caps what one round may evict from one (node, tier); - // 0 means the whole overage, which is already at most (1 - low_watermark) - // of the tier. Set it to throttle eviction work, e.g. to bound how much a - // demoting tier copies downstream per round. + // Most bytes one round evicts from one (node, tier). 0 = its whole overage + // above the low watermark. uint64_t max_evict_bytes_per_round = 0; - // Watermarks are not env-overridable (they have dedicated tuning paths); - // timing and the round's byte limit are. + // Watermarks have dedicated tuning paths and are not env-overridable. static EvictionConfig FromEnvironment() { EvictionConfig cfg; cfg.check_interval = diff --git a/src/umbp/include/umbp/distributed/peer/backend/page_backend.h b/src/umbp/include/umbp/distributed/peer/backend/page_backend.h index cf1ca1f50..8dd522c94 100644 --- a/src/umbp/include/umbp/distributed/peer/backend/page_backend.h +++ b/src/umbp/include/umbp/distributed/peer/backend/page_backend.h @@ -537,9 +537,7 @@ class PageBackend : public MediumBackend { struct PinState { uint64_t token = 0; std::chrono::steady_clock::time_point acquired_at; // for long-running warning only - // Taken by AcquireMigrationRead: the pool is moving this key off the - // medium, so it still counts as an eviction candidate (see - // EvictionCandidates). + // Set by AcquireMigrationRead; the key stays an eviction candidate. bool migration = false; }; std::unordered_map pins_; diff --git a/src/umbp/include/umbp/distributed/pool/peer_pool.h b/src/umbp/include/umbp/distributed/pool/peer_pool.h index b2aa93717..ef179ab7a 100644 --- a/src/umbp/include/umbp/distributed/pool/peer_pool.h +++ b/src/umbp/include/umbp/distributed/pool/peer_pool.h @@ -72,21 +72,17 @@ struct PoolResolvedEntry { ResolvedEntry resolved; }; -// What PeerPool::Evict did with one request. A demotion is never run inside -// the call: it is queued to the pool's transition worker, so an EvictKey RPC -// returns without waiting on a copy. +// What PeerPool::Evict did with one request. enum class PoolEvictOutcome { kNone, // nothing freed: no such copy, or a read lease / pin holds it - kFreed, // the copy was dropped; bytes_freed says how much - kDemotionQueued, // this call queued the copy's demotion; it frees the source when done - kInFlight, // a transition already owned the key; it will move or free it + kFreed, // the copy was dropped + kDemotionQueued, // queued for the transition worker, which frees the source + kInFlight, // another transition owns the key and will move or free it }; struct PoolEvictResult { std::string key; - // Bytes freed by the time Evict returned. 0 for a queued demotion, whose - // bytes come back when the transition completes. - uint64_t bytes_freed = 0; + uint64_t bytes_freed = 0; // by the time Evict returned; 0 for a queued demotion PoolEvictOutcome outcome = PoolEvictOutcome::kNone; }; @@ -161,10 +157,9 @@ class PeerPool { // The one routing entry point for eviction. Every policy calls this -- the // master's EvictKey RPC and this pool's own local watermark eviction alike -- // and every byte it frees is freed by the medium's MediumBackend::Evict, - // directly or as the last step of a demotion. A copy that drops is freed - // before this returns; a copy that demotes is only QUEUED (see - // PoolEvictOutcome), so the call never waits on a byte copy. One result per - // request, in request order. + // directly or as the last step of a demotion. Drops happen before this + // returns; demotions are only queued, so it never waits on a byte copy. One + // result per request, in request order. std::vector Evict(const std::vector& requests, PoolEvictMode mode); // Every key unscoped: freed from the medium that currently holds it. @@ -197,8 +192,7 @@ class PeerPool { bool EnableLocalEviction(uint32_t backend_id, double high_watermark, double low_watermark); LocalEvictionMetrics LocalEvictionStats() const; // Test seam: one synchronous pass over every enabled backend, as the worker - // runs it. Returns the number of keys evicted: dropped, or queued for - // demotion (which completes on the transition worker afterwards). + // runs it. Returns the number of keys dropped or queued for demotion. size_t RunLocalEvictionOnceForTest() { return LocalEvictionPass(); } private: @@ -214,10 +208,8 @@ class PeerPool { uint32_t source_backend_id = BackendRegistry::kMaxBackends; LogicalTierGraph::TierIndex source_tier = 0; uint8_t attempts = 0; - // Queued by Evict(): a demotion someone asked for -- the master's EvictKey - // or local eviction -- not one the tier's own watermark started. It runs - // whatever the tier's usage is by then, and if nothing downstream can take - // the key the copy is dropped instead: the eviction still has to free it. + // Queued by Evict(): runs regardless of the tier's watermark, and drops + // the copy if nothing downstream can take it. bool evict = false; }; @@ -228,16 +220,15 @@ class PeerPool { std::vector targets; }; - // False when the key already has a job (queued or running) or the worker is - // stopping; the job is not queued then. + // False if the key already has a job (queued or running) or the worker is + // stopping. bool EnqueueTransition(TransitionJob job); - // Whether a transition job owns `key` right now, queued or running. + // Whether a job owns `key`, queued or running. bool TransitionQueued(const std::string& key); void TransitionWorkerLoop(); void StopTransitionWorker(); - // An eviction's demotion could not move the copy (no room downstream, or the - // copy moved meanwhile): drop it, as a synchronous eviction would have -- - // unless another transition owns the key, which would rather move it. + // A failed eviction demotion drops the copy instead, unless another + // transition owns the key. void DropEvictedCopyLocked(const TransitionJob& job); struct LocalEvictionWatermarks { @@ -251,8 +242,8 @@ class PeerPool { void StopLocalEviction(); // One pass: every enabled backend, lowest tier first. Returns keys evicted. size_t LocalEvictionPass(); - // Brings one backend down to its low watermark if it is at or above its - // high one -- counting demotions still in flight -- and returns keys evicted. + // Brings one backend at or above its high watermark down to its low one, + // counting in-flight demotions. Returns keys evicted. size_t ReclaimBackend(uint32_t backend_id, const LocalEvictionWatermarks& watermarks); void TouchLocked(const std::string& key); diff --git a/tests/cpp/umbp/distributed/test_env_time.cpp b/tests/cpp/umbp/distributed/test_env_time.cpp index 630b1c137..9e5b6f8d4 100644 --- a/tests/cpp/umbp/distributed/test_env_time.cpp +++ b/tests/cpp/umbp/distributed/test_env_time.cpp @@ -108,15 +108,9 @@ TEST_F(EnvTimeTest, ZeroIsAllowedWhenMinIsZero) { EXPECT_EQ(GetEnvUint32(kName, 7), 0u); } -TEST_F(EnvTimeTest, Uint64ParsesByteCountsPastUint32) { - // 8 GiB: the eviction byte limits need values a uint32 cannot hold. - ::setenv(kName, "8589934592", 1); +TEST_F(EnvTimeTest, Uint64ParsesPastUint32AndRejectsGarbage) { + ::setenv(kName, "8589934592", 1); // 8 GiB EXPECT_EQ(GetEnvUint64(kName, 1), 8589934592ULL); - ::unsetenv(kName); - EXPECT_EQ(GetEnvUint64(kName, 1ULL << 30), 1ULL << 30); -} - -TEST_F(EnvTimeTest, Uint64RejectsNegativesAndGarbage) { ::setenv(kName, "-1", 1); EXPECT_EQ(GetEnvUint64(kName, 5), 5u); ::setenv(kName, "1G", 1); diff --git a/tests/cpp/umbp/distributed/test_master_evict_strategy.cpp b/tests/cpp/umbp/distributed/test_master_evict_strategy.cpp index a1534d400..af8107523 100644 --- a/tests/cpp/umbp/distributed/test_master_evict_strategy.cpp +++ b/tests/cpp/umbp/distributed/test_master_evict_strategy.cpp @@ -72,24 +72,6 @@ TEST(LruMasterEvictStrategy, PicksOldestFirstUntilBudgetMet) { EXPECT_EQ(victims["n1"][1].key, "b"); } -TEST(LruMasterEvictStrategy, ABudgetLargerThanAnyRowCapIsMetInOneRound) { - // The round is sized in bytes now: 1,000 candidates against a budget that - // covers 600 of them yields 600 victims, not the first 32. - LruMasterEvictStrategy strategy; - auto now = Clock::now(); - std::vector candidates; - for (int i = 0; i < 1000; ++i) { - candidates.push_back(MakeCandidate("k" + std::to_string(i), "n1", TierType::DRAM, 64, - now - std::chrono::seconds(1000 - i))); - } - std::unordered_map> budget; - budget["n1"][TierType::DRAM] = 600 * 64; - - auto victims = strategy.SelectVictims(candidates, budget); - ASSERT_EQ(victims["n1"].size(), 600u); - EXPECT_EQ(victims["n1"].front().key, "k0"); // oldest first -} - TEST(LruMasterEvictStrategy, KeyOverBudgetInTwoTiersIsOneVictimPerTier) { // A key in DRAM and SSD, both over budget: one victim per (key, tier), each // carrying the tier its copy was charged to. diff --git a/tests/cpp/umbp/distributed/test_master_eviction_chain.cpp b/tests/cpp/umbp/distributed/test_master_eviction_chain.cpp index fd45a7f35..61c503f89 100644 --- a/tests/cpp/umbp/distributed/test_master_eviction_chain.cpp +++ b/tests/cpp/umbp/distributed/test_master_eviction_chain.cpp @@ -259,8 +259,7 @@ TEST(MasterEvictionChain, OverWatermarkTierDemotesIntoTheColdTier) { EXPECT_EQ(dispatcher.NodeIds().front(), kNodeId); // Every key the master named must have moved down a tier rather than vanished. - // The peer only queues the demotion -- the EvictKey call does not wait on the - // copy -- so the move is observed after the fact. + // The peer only queues the demotion, so wait for the move. const auto evicted = dispatcher.Dispatched(); ASSERT_FALSE(evicted.empty()); const bool moved = WaitFor( @@ -542,15 +541,7 @@ TEST(MasterEvictionChain, ParkedMoveSourceIsNamedLastAmongColderKeys) { EXPECT_TRUE(registry.Get("hot")->Contains("moved")); } -// --------------------------------------------------------------------------- -// Round size in bytes -// -// A round frees the overage, measured in bytes, rather than a fixed number of -// candidates per bucket. These drive the real EvictionManager against the -// real store and record what it dispatches; nothing is executed. -// --------------------------------------------------------------------------- - -// Records every EvictKey the manager sends, one entry per RPC. +// Records what the manager sends, one entry per EvictKey; executes nothing. class RecordingDispatcher final : public EvictKeyDispatcher { public: void DispatchEvictKey(const std::string& /*node_id*/, const std::string& /*peer_address*/, @@ -562,73 +553,36 @@ class RecordingDispatcher final : public EvictKeyDispatcher { std::lock_guard lock(mutex_); return calls_; } - size_t Keys() const { - std::lock_guard lock(mutex_); - size_t n = 0; - for (const auto& call : calls_) n += call.size(); - return n; - } private: mutable std::mutex mutex_; std::vector> calls_; }; -std::vector NumberedKeys(size_t n) { +// Keys in the first EvictKey for a full 256-page DRAM bucket of one-page keys. +size_t FirstRoundVictims(const EvictionConfig& config) { std::vector keys; - for (size_t i = 0; i < n; ++i) keys.push_back("k" + std::to_string(i)); - return keys; -} - -// 256 one-page keys filling a 256-page DRAM bucket: the overage above the 0.7 -// low watermark is 4,916 bytes, i.e. 77 keys -- more than the 32-row cap a -// round used to stop at. -constexpr size_t kFullBucketKeys = 256; -const uint64_t kFullBucketBytes = kFullBucketKeys * kPageSize; -const size_t kOverageKeys = - (kFullBucketBytes - static_cast(kFullBucketBytes * 0.7) + kPageSize - 1) / kPageSize; - -TEST(MasterEvictionBytes, ARoundFreesTheWholeOverageNotThirtyTwoKeys) { - InMemoryMasterMetadataStore store; - PublishNode(&store, NumberedKeys(kFullBucketKeys), kFullBucketBytes, kFullBucketBytes); - - RecordingDispatcher dispatcher; - EvictionManager manager(store, OneSecondRounds(), &dispatcher); - manager.Start(); - ASSERT_TRUE(WaitFor([&] { return !dispatcher.Calls().empty(); }, std::chrono::seconds(8))); - manager.Stop(); - - // A round's victims go out in calls bounded only by gRPC message size, so - // these few short keys are one call. - const auto first = dispatcher.Calls().front(); - ASSERT_GT(kOverageKeys, 32u); - EXPECT_EQ(first.size(), kOverageKeys) << "the round did not cover its byte budget"; - for (const auto& victim : first) EXPECT_EQ(victim.tier, TierType::DRAM); -} - -TEST(MasterEvictionBytes, PerRoundByteLimitThrottlesTheRound) { + for (int i = 0; i < 256; ++i) keys.push_back("k" + std::to_string(i)); InMemoryMasterMetadataStore store; - PublishNode(&store, NumberedKeys(kFullBucketKeys), kFullBucketBytes, kFullBucketBytes); - - auto config = OneSecondRounds(); - config.max_evict_bytes_per_round = 10 * kPageSize; + PublishNode(&store, keys, 256 * kPageSize, 256 * kPageSize); RecordingDispatcher dispatcher; EvictionManager manager(store, config, &dispatcher); manager.Start(); - ASSERT_TRUE(WaitFor([&] { return !dispatcher.Calls().empty(); }, std::chrono::seconds(8))); + const bool sent = WaitFor([&] { return !dispatcher.Calls().empty(); }, std::chrono::seconds(8)); manager.Stop(); - - EXPECT_EQ(dispatcher.Calls().front().size(), 10u); + return sent ? dispatcher.Calls().front().size() : 0; } -TEST(MasterEvictionBytes, PerRoundByteLimitIsReadFromTheEnvironment) { - ::setenv("UMBP_EVICT_MAX_BYTES_PER_ROUND", "8589934592", 1); - const auto config = EvictionConfig::FromEnvironment(); - ::unsetenv("UMBP_EVICT_MAX_BYTES_PER_ROUND"); - EXPECT_EQ(config.max_evict_bytes_per_round, 8589934592ULL); +TEST(MasterEvictionBytes, ARoundFreesTheOverageUpToTheByteLimit) { + // The overage above the 0.7 low watermark is 77 keys, more than the 32 a + // round used to stop at. + EXPECT_EQ(FirstRoundVictims(OneSecondRounds()), 77u); - const auto defaults = EvictionConfig::FromEnvironment(); - EXPECT_EQ(defaults.max_evict_bytes_per_round, 0u); + ::setenv("UMBP_EVICT_MAX_BYTES_PER_ROUND", std::to_string(10 * kPageSize).c_str(), 1); + auto limited = OneSecondRounds(); + limited.max_evict_bytes_per_round = EvictionConfig::FromEnvironment().max_evict_bytes_per_round; + ::unsetenv("UMBP_EVICT_MAX_BYTES_PER_ROUND"); + EXPECT_EQ(FirstRoundVictims(limited), 10u); } } // namespace diff --git a/tests/cpp/umbp/distributed/test_peer_pool.cpp b/tests/cpp/umbp/distributed/test_peer_pool.cpp index 2f3007b24..87a8109b5 100644 --- a/tests/cpp/umbp/distributed/test_peer_pool.cpp +++ b/tests/cpp/umbp/distributed/test_peer_pool.cpp @@ -112,8 +112,7 @@ bool WaitForKey(BackendRegistry* registry, const std::string& backend, const std return registry->Get(backend)->Contains(key); } -// The mirror image: an eviction's demotion runs on the transition worker, so -// the source copy leaving is observed rather than awaited. +// Eviction demotions run on the transition worker, so wait for the source. bool WaitForKeyGone(BackendRegistry* registry, const std::string& backend, const std::string& key) { for (int attempt = 0; attempt < 500 && registry->Get(backend)->Contains(key); ++attempt) { std::this_thread::sleep_for(std::chrono::milliseconds(2)); @@ -121,12 +120,10 @@ bool WaitForKeyGone(BackendRegistry* registry, const std::string& backend, const return !registry->Get(backend)->Contains(key); } -// A LocalCopyEngine that holds every copy until Release(), so a test can see -// what an eviction does while its demotion is still running. +// A LocalCopyEngine that holds every copy until Release() -- or 10 s, so a +// failed test cannot hang the pool's teardown. class GatedEngine final : public TransferEngine { public: - ~GatedEngine() override { Release(); } - const char* Name() const override { return "gated"; } TransferRef RegisterMemory(void* base, size_t size, mori::io::MemoryLocationType loc, int device) override { @@ -144,18 +141,17 @@ class GatedEngine final : public TransferEngine { std::unique_lock lock(mutex_); ++held_; cv_.notify_all(); - cv_.wait(lock, [&] { return open_; }); + cv_.wait_for(lock, std::chrono::seconds(10), [&] { return open_; }); } return inner_.Submit(std::move(plans)); } - // Lets every held copy (and every later one) through. void Release() { std::lock_guard lock(mutex_); open_ = true; cv_.notify_all(); } - // True once at least `n` copies have reached Submit and are being held. + // True once `n` copies are being held. bool WaitForHeld(int n) { std::unique_lock lock(mutex_); return cv_.wait_for(lock, std::chrono::seconds(5), [&] { return held_ >= n; }); @@ -169,13 +165,6 @@ class GatedEngine final : public TransferEngine { int held_ = 0; }; -// Releases the gate on scope exit, so a failed assertion cannot leave the -// transition worker blocked while the pool it belongs to is torn down. -struct ReleaseOnExit { - GatedEngine* engine; - ~ReleaseOnExit() { engine->Release(); } -}; - class NoSpaceBackend final : public MockBackend { public: explicit NoSpaceBackend(TierType tier) : MockBackend(tier) {} @@ -615,7 +604,7 @@ TEST(TieredPlacementPolicy, HonorsExplicitLogicalTierAndTagsEvents) { TEST(PeerPool, OnEvictMigratesBytesToConfiguredBackend) { constexpr uint64_t kPageSize = 64; - LocalCopyEngine engine; + GatedEngine engine; BackendRegistry registry; RegisterPageBackend(®istry, &engine, "hot", TierType::DRAM, kPageSize, 2); RegisterPageBackend(®istry, &engine, "cold", TierType::SSD, kPageSize, 2); @@ -636,11 +625,18 @@ TEST(PeerPool, OnEvictMigratesBytesToConfiguredBackend) { .front() .commit.success); - // The demotion is queued, not run inside the call: nothing is freed yet. + // Evict only queues the demotion: it returns while the copy is held, and + // asking again does not queue it twice. auto evicted = pool.Evict({"move"}, PoolEvictMode::kReclaim); ASSERT_EQ(evicted.size(), 1u); EXPECT_EQ(evicted.front().outcome, PoolEvictOutcome::kDemotionQueued); EXPECT_EQ(evicted.front().bytes_freed, 0u); + ASSERT_TRUE(engine.WaitForHeld(1)) << "the demotion never reached the copy"; + EXPECT_TRUE(registry.Get("hot")->Contains("move")); + EXPECT_EQ(pool.Evict({"move"}, PoolEvictMode::kReclaim).front().outcome, + PoolEvictOutcome::kInFlight); + + engine.Release(); ASSERT_TRUE(WaitForKeyGone(®istry, "hot", "move")); EXPECT_TRUE(registry.Get("cold")->Contains("move")); EXPECT_EQ(pool.PlacementBackend("move"), @@ -658,8 +654,7 @@ TEST(PeerPool, OnEvictMigratesBytesToConfiguredBackend) { TEST(PeerPool, EvictionRetryDrainsLeasedSourceWithoutDeletingTarget) { constexpr uint64_t kPageSize = 64; - // Long enough to still hold when the queued demotion runs on the worker, so - // its source delete is refused and the source parks as draining. + // Outlives the queued demotion, so its source delete is refused. constexpr auto kLease = std::chrono::milliseconds(300); LocalCopyEngine engine; BackendRegistry registry; @@ -751,8 +746,8 @@ TEST(PeerPool, TierScopedEvictFreesOnlyThatMediumsCopy) { EXPECT_TRUE(registry.Get("hot")->Contains("a")); EXPECT_EQ(pool.PlacementBackend("a"), std::optional{hot}); - // DRAM scope: the queued demotion finds cold already holding the key, so it - // frees DRAM without a copy. + // DRAM scope: cold already holds the key, so the demotion frees DRAM without + // a copy. evicted = pool.Evict({PoolEvictRequest{"b", TierType::DRAM}}, PoolEvictMode::kReclaim); EXPECT_EQ(evicted.front().outcome, PoolEvictOutcome::kDemotionQueued); ASSERT_TRUE(WaitForKeyGone(®istry, "hot", "b")); @@ -801,8 +796,7 @@ TEST(PeerPool, ReclaimDemotesOnAWatermarkTierToo) { auto evicted = pool.Evict({PoolEvictRequest{"k", TierType::DRAM}}, PoolEvictMode::kReclaim); EXPECT_EQ(evicted.front().outcome, PoolEvictOutcome::kDemotionQueued); - // An eviction's demotion runs even though the tier is far under its own - // watermark -- the watermark governs proactive offload, not this. + // Runs although the tier is far under its own watermark. ASSERT_TRUE(WaitForKeyGone(®istry, "hot", "k")); EXPECT_TRUE(registry.Get("cold")->Contains("k")) << "a watermark tier deleted instead of demoting"; @@ -826,7 +820,7 @@ TEST(PeerPool, DemotionWithNoRoomDownstreamDropsTheCopy) { ASSERT_TRUE(cold->BatchCommit({CommitRequest{filler.slot_id, "filler"}}).front().success); CommitKey(&pool, "k"); - // Queued like any demotion; the worker finds no room and drops the copy. + // The worker finds no room downstream and drops the copy. auto evicted = pool.Evict({PoolEvictRequest{"k", TierType::DRAM}}, PoolEvictMode::kReclaim); EXPECT_EQ(evicted.front().outcome, PoolEvictOutcome::kDemotionQueued); ASSERT_TRUE(WaitForKeyGone(®istry, "hot", "k")) << "a failed demotion left the copy in place"; @@ -835,34 +829,6 @@ TEST(PeerPool, DemotionWithNoRoomDownstreamDropsTheCopy) { EXPECT_FALSE(pool.PlacementBackend("k").has_value()); } -TEST(PeerPool, EvictReturnsBeforeTheDemotionCopyRuns) { - // The point of queueing: an EvictKey RPC must not wait on a byte copy, - // however large. The engine holds the copy; Evict has already returned. - constexpr uint64_t kPageSize = 64; - GatedEngine engine; - BackendRegistry registry; - RegisterPageBackend(®istry, &engine, "hot", TierType::DRAM, kPageSize, 4); - RegisterPageBackend(®istry, &engine, "cold", TierType::SSD, kPageSize, 4); - auto compiled = LogicalTierGraph::Compile(HotColdTiers(PoolOffloadTrigger::kOnEvict), registry); - ASSERT_TRUE(compiled.ok()) << compiled.error; - PeerPool pool(®istry, MakeTieredPlacementPolicy(compiled.graph), &engine); - ReleaseOnExit release{&engine}; - CommitKey(&pool, "k"); - - auto first = pool.Evict({PoolEvictRequest{"k", TierType::DRAM}}, PoolEvictMode::kReclaim); - EXPECT_EQ(first.front().outcome, PoolEvictOutcome::kDemotionQueued); - ASSERT_TRUE(engine.WaitForHeld(1)) << "the demotion never reached the copy"; - // Mid-copy: the source is untouched, and asking again does not queue twice. - EXPECT_TRUE(registry.Get("hot")->Contains("k")); - EXPECT_FALSE(registry.Get("cold")->Contains("k")); - auto again = pool.Evict({PoolEvictRequest{"k", TierType::DRAM}}, PoolEvictMode::kReclaim); - EXPECT_EQ(again.front().outcome, PoolEvictOutcome::kInFlight); - - engine.Release(); - ASSERT_TRUE(WaitForKeyGone(®istry, "hot", "k")); - EXPECT_TRUE(registry.Get("cold")->Contains("k")); -} - TEST(PeerPool, WatermarkMigratesCommittedKey) { // Unlike the other tier tests, this one asserts on utilization, so the owned // buffer must be a whole number of host allocation granules: a smaller @@ -1227,9 +1193,11 @@ TEST(PeerPoolLocalEviction, KeysUnderAReadLeaseAreNeverOffered) { EXPECT_GE(pool.LocalEvictionStats().no_candidate, 1u); } -TEST(PeerPoolLocalEviction, ATierWithADownstreamDemotesInsteadOfDropping) { +TEST(PeerPoolLocalEviction, ATierWithADownstreamDemotesJustTheOverage) { // Same call as a master evict: a tier with a downstream demotes, never drops. - LocalCopyEngine engine; + // A queued demotion frees its source only when the copy lands, so passes run + // meanwhile must count it, or they would queue the whole tier. + GatedEngine engine; BackendRegistry registry; RegisterPageBackend(®istry, &engine, "hot", TierType::DRAM, kHostPage, 10); RegisterPageBackend(®istry, &engine, "cold", TierType::SSD, kHostPage, 20); @@ -1241,43 +1209,19 @@ TEST(PeerPoolLocalEviction, ATierWithADownstreamDemotesInsteadOfDropping) { for (int i = 0; i < 9; ++i) CommitKey(&pool, "k" + std::to_string(i)); auto* hot = registry.Get("hot"); auto* cold = registry.Get("cold"); - ASSERT_TRUE(WaitUntil([&] { return hot->OwnedKeyCount() <= 5; })); - for (int i = 0; i < 4; ++i) { - const std::string key = "k" + std::to_string(i); - EXPECT_FALSE(hot->Contains(key)) << key; - EXPECT_TRUE(cold->Contains(key)) << key << " was dropped, not demoted"; - EXPECT_TRUE(pool.BatchResolve({key}, false).front().resolved.found) << key; - } -} - -TEST(PeerPoolLocalEviction, QueuedDemotionsCountTowardTheBudget) { - // A queued demotion frees its source only when the copy lands, so until then - // the tier still reads as full. The worker must count those bytes as on - // their way out -- otherwise every pass would queue the next coldest keys - // until the whole tier was in flight. - GatedEngine engine; - BackendRegistry registry; - RegisterPageBackend(®istry, &engine, "hot", TierType::DRAM, kHostPage, 10); - RegisterPageBackend(®istry, &engine, "cold", TierType::SSD, kHostPage, 20); - auto compiled = LogicalTierGraph::Compile(HotColdTiers(PoolOffloadTrigger::kOnEvict), registry); - ASSERT_TRUE(compiled.ok()) << compiled.error; - PeerPool pool(®istry, MakeTieredPlacementPolicy(compiled.graph), &engine); - ReleaseOnExit release{&engine}; - ASSERT_TRUE(pool.EnableLocalEviction(registry.BackendId(registry.Get("hot")), 0.9, 0.5)); - - for (int i = 0; i < 9; ++i) CommitKey(&pool, "k" + std::to_string(i)); ASSERT_TRUE(engine.WaitForHeld(1)) << "no demotion was queued"; - // Several more passes while every copy is held: still exactly the overage. for (int i = 0; i < 3; ++i) pool.RunLocalEvictionOnceForTest(); std::this_thread::sleep_for(std::chrono::milliseconds(250)); // let polls run too EXPECT_EQ(pool.LocalEvictionStats().keys, 4u) << "queued past the overage"; - EXPECT_EQ(registry.Get("hot")->OwnedKeyCount(), 9u) << "a copy completed while held"; + EXPECT_EQ(hot->OwnedKeyCount(), 9u) << "a copy completed while held"; engine.Release(); - auto* hot = registry.Get("hot"); ASSERT_TRUE(WaitUntil([&] { return hot->OwnedKeyCount() <= 5; })); for (int i = 0; i < 4; ++i) { - EXPECT_TRUE(registry.Get("cold")->Contains("k" + std::to_string(i))) << i; + const std::string key = "k" + std::to_string(i); + EXPECT_FALSE(hot->Contains(key)) << key; + EXPECT_TRUE(cold->Contains(key)) << key << " was dropped, not demoted"; + EXPECT_TRUE(pool.BatchResolve({key}, false).front().resolved.found) << key; } } From 158ef808ec61fa25e561cbb8304b2ce708ac2a74 Mon Sep 17 00:00:00 2001 From: TianDi101 Date: Tue, 13 Oct 2026 07:02:04 +0000 Subject: [PATCH 09/19] refactor(umbp): cut the eviction comments down to the non-obvious Comment-only. Drops narrative and rationale that restates the code, and the same "the pool decides, the medium only supplies order" paragraph that was repeated across several headers; field and helper docs become trailing one-liners. Added comment lines across the PR: ~400 -> 195. Co-Authored-By: Claude Opus 5.5 --- .../distributed/master/eviction_manager.cpp | 10 +- .../distributed/peer/backend/page_backend.cpp | 14 +-- .../distributed/peer/backend/ssd_backend.cpp | 6 +- src/umbp/distributed/peer/peer_service.cpp | 10 +- .../distributed/peer/ssd/peer_ssd_manager.cpp | 18 +-- src/umbp/distributed/pool/peer_pool.cpp | 107 +++++------------- src/umbp/distributed/pool_client.cpp | 13 +-- src/umbp/distributed/proto/umbp_peer.proto | 6 +- src/umbp/include/umbp/distributed/config.h | 9 +- .../umbp/distributed/master/evict_strategy.h | 9 +- .../distributed/master/eviction_manager.h | 3 +- .../umbp/distributed/master/master_metrics.h | 5 +- .../distributed/peer/backend/medium_backend.h | 32 ++---- .../distributed/peer/backend/page_backend.h | 15 +-- .../distributed/peer/backend/ssd_backend.h | 4 +- .../distributed/peer/ssd/peer_ssd_manager.h | 39 ++----- .../include/umbp/distributed/pool/peer_pool.h | 107 +++++------------- src/umbp/include/umbp/distributed/types.h | 11 +- src/umbp/local/tiers/spdk_ssd_tier.cpp | 8 +- .../test_master_eviction_chain.cpp | 3 +- tests/cpp/umbp/distributed/test_peer_pool.cpp | 25 ++-- .../distributed/test_peer_ssd_eviction.cpp | 9 +- .../cpp/umbp/distributed/test_ssd_backend.cpp | 4 +- 23 files changed, 126 insertions(+), 341 deletions(-) diff --git a/src/umbp/distributed/master/eviction_manager.cpp b/src/umbp/distributed/master/eviction_manager.cpp index 968e64cb7..15938a0bf 100644 --- a/src/umbp/distributed/master/eviction_manager.cpp +++ b/src/umbp/distributed/master/eviction_manager.cpp @@ -152,9 +152,8 @@ void EvictionManager::RunOnce() { // Flatten the candidates and translate the per-(node, tier) byte budget into // the strategy's shape, then let the strategy pick victims (default LRU), - // grouped by node so each peer gets a single EvictKey. Each victim keeps - // the tier it was charged to, so the peer frees exactly the copy that is - // over budget. + // grouped by node so each peer gets a single EvictKey. Victims keep their + // tier, so the peer frees the copy that is over budget. std::vector candidates; for (auto& [bucket, rows] : candidates_by_bucket) { for (auto& c : rows) candidates.push_back(std::move(c)); @@ -205,10 +204,7 @@ void EvictionManager::RunOnce() { // next heartbeat are what shrink the index), and a chunk that fails is // retried next round. So N chunks reach the same state as one message // would have, and a partial round still makes progress. - // - // Sized on the keys alone: the parallel tier field is a packed enum of one - // byte per key, inside the per-item overhead GrpcMaxItemsPerBatch already - // reserves. Bytes don't lengthen a call: the peer only queues demotions. + // The one-byte tier per key fits in GrpcMaxItemsPerBatch's item overhead. std::vector keys; keys.reserve(victims.size()); for (const auto& victim : victims) keys.push_back(victim.key); diff --git a/src/umbp/distributed/peer/backend/page_backend.cpp b/src/umbp/distributed/peer/backend/page_backend.cpp index edf2ea65e..8b858a54b 100644 --- a/src/umbp/distributed/peer/backend/page_backend.cpp +++ b/src/umbp/distributed/peer/backend/page_backend.cpp @@ -811,8 +811,7 @@ void PageBackend::TrackEvictionOrder() { std::unique_lock lock(mutex_); if (track_eviction_order_) return; track_eviction_order_ = true; - // Keys committed before this call are not in lru_ and would otherwise never - // be offered. They have no recorded access, so they go in as coldest. + // Seed keys committed before this call, as coldest. std::lock_guard lru_lock(lru_mutex_); for (auto& [key, slot] : owned_) TouchLruLocked(key, slot); } @@ -820,7 +819,6 @@ void PageBackend::TrackEvictionOrder() { std::vector PageBackend::EvictionCandidates(size_t max_count) { std::vector out; if (max_count == 0) return out; - // Shared: this only reads owned_ and pins_. The LRU has its own lock. std::shared_lock lock(mutex_); if (!track_eviction_order_) return out; const auto now = std::chrono::steady_clock::now(); @@ -828,15 +826,11 @@ std::vector PageBackend::EvictionCandidates(size_t max_count) { for (auto it = lru_.rbegin(); it != lru_.rend() && out.size() < max_count; ++it) { auto owned_it = owned_.find(*it); if (owned_it == owned_.end()) continue; // defensive; lru_/owned_ stay in sync - // The two protections Evict() applies: an in-flight read owns the pages - // until its lease expires, a copy pin until the copy is done. Offering - // them would only make the caller's batch free less than it asked for. - // Except a migration pin: the key is already leaving, and the caller - // counts its bytes as in flight. + // Skip what Evict() refuses (read lease, copy pin); a migration pin is + // offered so the caller can count it as in flight. if (HasActiveReadLeaseLocked(owned_it->second, now)) continue; if (auto pin = pins_.find(*it); pin != pins_.end() && !pin->second.migration) continue; - // Whole pages: that is what the allocator gives back, and what Capacity() - // counts against the watermark. + // Whole pages, as Capacity() counts them. out.push_back( EvictionOffer{*it, static_cast(owned_it->second.pages.size()) * page_size_}); } diff --git a/src/umbp/distributed/peer/backend/ssd_backend.cpp b/src/umbp/distributed/peer/backend/ssd_backend.cpp index 39880a256..07b3058bf 100644 --- a/src/umbp/distributed/peer/backend/ssd_backend.cpp +++ b/src/umbp/distributed/peer/backend/ssd_backend.cpp @@ -876,10 +876,8 @@ std::vector SsdBackend::SampleMetrics() const { event("single_flight_dup", ssd_->ReadDup()); event("single_flight_merged", ssd_->ReadMerged()); - // Every eviction -- master-driven or the pool's local watermark eviction -- - // arrives through Evict(), so the keys and bytes are the decorator's and the - // rounds are the pool's (mori_umbp_client_local_evict_total). What neither - // can see is a delete the drive itself refused. + // Evictions are counted by the decorator and the pool; only a delete the + // drive refused is counted here. event("eviction_backend_failed", ssd_->EvictionBackendFailures()); // Staging pressure. slot_full_reject is the one to watch: it counts resolves diff --git a/src/umbp/distributed/peer/peer_service.cpp b/src/umbp/distributed/peer/peer_service.cpp index 00b6fe708..c2188c04d 100644 --- a/src/umbp/distributed/peer/peer_service.cpp +++ b/src/umbp/distributed/peer/peer_service.cpp @@ -265,13 +265,9 @@ class PeerServiceServer::UMBPPeerServiceImpl final : public ::umbp::UMBPPeer::Se grpc::Status EvictKey(grpc::ServerContext* /*ctx*/, const ::umbp::EvictKeyRequest* request, ::umbp::EvictKeyResponse* response) override { - // Master reclaims capacity it measured per (node, tier), and names that - // tier per key: the pool frees exactly that medium's copy (demoting it - // where its tier has a downstream) and leaves any other copy alone. An - // older master sends no tiers; each key is then freed from the medium that - // currently holds it. Demotions are only queued, so a demoted key reports - // 0 bytes freed; bytes_freed is informational anyway -- the master sizes - // its next round off the heartbeat's capacity, not off this reply. + // Free each key's copy on the tier the master named (an older master names + // none: the medium holding it). bytes_freed is informational; a queued + // demotion reports 0. const int n = request->keys_size(); if (n == 0) return grpc::Status::OK; const bool scoped = request->tiers_size() == n; diff --git a/src/umbp/distributed/peer/ssd/peer_ssd_manager.cpp b/src/umbp/distributed/peer/ssd/peer_ssd_manager.cpp index c37c0941b..57a27b50f 100644 --- a/src/umbp/distributed/peer/ssd/peer_ssd_manager.cpp +++ b/src/umbp/distributed/peer/ssd/peer_ssd_manager.cpp @@ -48,8 +48,7 @@ PeerSsdManager::PeerSsdManager(const PeerSsdConfig& cfg) { return; } const auto& ssd = cfg.ssd; - // The SSD watermarks in `ssd` are consumed by PeerPool's local eviction, - // which PoolClient enables for this medium -- not here. + // `ssd`'s watermarks drive PeerPool's local eviction, not this class. single_flight_ = ssd.single_flight_reads; // Explicit backend selection — an unknown value is a configuration error, not @@ -191,8 +190,7 @@ bool PeerSsdManager::Write(const std::string& key, } // Backend IO outside our mutex_ — SSDTier is internally synchronized. A - // failure (ENOSPC included) records nothing and queues nothing. Making room - // is the pool's local eviction, on its own thread, never this write. + // failure (ENOSPC included) records nothing; this never evicts. if (!backend_->Write(key, data, total_size)) { MORI_UMBP_WARN("[PeerSsdManager] backend Write failed key={} size={}", key, total_size); return false; @@ -257,9 +255,8 @@ std::vector PeerSsdManager::WriteBatch(const std::vector& key const size_t dedup_hits = keys.size() - todo.size(); if (todo.empty()) return results; - // Pass 2 — one batched backend write outside the lock. A key that fails - // (ENOSPC included) is reported as failed and recorded nowhere; making room - // is the pool's local eviction, on its own thread, never this write. + // Pass 2 — one batched backend write outside the lock. A failed key + // (ENOSPC included) is recorded nowhere; this never evicts. const auto t_backend = ssdperf::Now(); std::vector batch_keys; std::vector batch_srcs; @@ -389,11 +386,8 @@ void PeerSsdManager::ClearLocal() { // in-flight reads (below) but not in-flight Writes, so a racing Write could // re-add owned_/ADD SSD after backend->Clear() wipes its bytes. // PoolClient::Clear() Quiesce()s first; new callers must too. - // - // No eviction is in flight: the only caller of Evict is PeerPool::Evict, - // which holds the lifecycle lock PeerPool::ClearLocal takes exclusively - // before it gets here. An Evict that did overlap would still be safe -- it - // finds its key gone from owned_ and records nothing -- just pointless. + // No Evict overlaps this: PeerPool::Evict holds the lifecycle lock that + // PeerPool::ClearLocal takes exclusively. { std::unique_lock lock(mutex_); owned_.clear(); diff --git a/src/umbp/distributed/pool/peer_pool.cpp b/src/umbp/distributed/pool/peer_pool.cpp index 45efe1771..7c6650870 100644 --- a/src/umbp/distributed/pool/peer_pool.cpp +++ b/src/umbp/distributed/pool/peer_pool.cpp @@ -45,15 +45,11 @@ constexpr size_t kWatermarkCandidateBatch = 64; // watermark would re-walk the placements. constexpr auto kWatermarkScanInterval = std::chrono::milliseconds(5); -// Local eviction frees at most this many keys per Evict() call, so no single -// call holds the pool lock for a whole drain; the pass keeps calling until the -// backend is back under its low watermark. +// Keys per Evict() call, so one call never holds the pool lock for a whole drain. constexpr size_t kLocalEvictBatch = 64; -// Wakes are the primary signal; this catches growth no wake reported. +// Wakes are the main signal; the poll catches growth no wake reported. constexpr auto kLocalEvictPollInterval = std::chrono::milliseconds(100); -// Floor between passes, so commits arriving back to back above the high -// watermark cannot keep the worker re-reading capacity in a tight loop. -constexpr auto kLocalEvictMinGap = std::chrono::milliseconds(1); +constexpr auto kLocalEvictMinGap = std::chrono::milliseconds(1); // between passes uint64_t UsedBytes(const TierCapacity& capacity) { return capacity.total_bytes > capacity.available_bytes @@ -93,9 +89,7 @@ PeerPool::PeerPool(BackendRegistry* backends, std::unique_ptr policy } PeerPool::~PeerPool() { - // Eviction first: a pass may be inside RunTransition, which the transition - // machinery below must still be alive for. - StopLocalEviction(); + StopLocalEviction(); // first: a pass may still be using the transition worker StopTransitionWorker(); } @@ -109,7 +103,6 @@ bool PeerPool::EnqueueTransition(TransitionJob job) { } bool PeerPool::TransitionQueued(const std::string& key) { - // Keys leave queued_transition_keys_ only when their job finishes. std::lock_guard lock(transition_mutex_); return queued_transition_keys_.count(key) != 0; } @@ -291,9 +284,7 @@ void PeerPool::FinishTransitionLocked(const TransitionJob& job, if (draining->second.empty()) draining_sources_.erase(draining); } } - // The target may be a copy an earlier move left parked (it answered - // ALREADY_EXISTS, so nothing was copied). It is the durable copy now, and a - // later eviction must not delete it as though it were still draining. + // A parked copy reused as the target (ALREADY_EXISTS) is durable again. auto draining = draining_sources_.find(job.key); if (draining != draining_sources_.end()) { draining->second.erase(result.target_backend_id); @@ -491,10 +482,7 @@ std::vector PeerPool::BatchAllocate( pending_keys_.erase(requests[index].key); break; case AllocateOutcome::kFailedNoSpace: - // A full medium: the local eviction worker should not wait for its - // poll to notice. This put still fails over to the next candidate - // (or fails); eviction never runs on the put itself. - WakeLocalEviction(); + WakeLocalEviction(); // don't wait for the poll; the put never evicts advance(index); break; } @@ -576,8 +564,6 @@ std::vector PeerPool::BatchCommit( } } MaybeEnqueueWatermarkOffloadLocked(); - // A commit is what grows usage. Waking is all the put path does; the worker - // reads capacity and evicts on its own thread. WakeLocalEviction(); return out; } @@ -893,8 +879,6 @@ std::vector PeerPool::BatchContains(const std::vector& keys) uint32_t PeerPool::EvictSourceLocked(const PoolEvictRequest& request) const { constexpr uint32_t kNone = BackendRegistry::kMaxBackends; - // A copy qualifies when its backend holds the key and, for a tier-scoped - // request, is that medium. const auto holds = [&](uint32_t backend_id) { if (backend_id >= kNone) return false; auto* backend = backends_->Get(backend_id); @@ -903,8 +887,7 @@ uint32_t PeerPool::EvictSourceLocked(const PoolEvictRequest& request) const { backend->Contains(request.key); }; if (request.backend_id < kNone) return holds(request.backend_id) ? request.backend_id : kNone; - // The placement first: it is the copy reads are served from, and the only - // one a demotion can start from. + // Prefer the placement: the only copy a demotion can start from. auto placement = placements_.find(request.key); if (placement != placements_.end() && holds(placement->second)) return placement->second; if (request.tier == TierType::UNKNOWN) return FindOwnerLocked(request.key); @@ -948,19 +931,15 @@ std::vector PeerPool::Evict(const std::vector& key std::vector PeerPool::Evict(const std::vector& requests, PoolEvictMode mode) { - // Shared for the whole call: the drops below run without operation_mutex_, - // and ClearLocal must not wipe the backends between a drop being decided and - // being carried out. + // Drops run without operation_mutex_; this keeps ClearLocal out meanwhile. std::shared_lock lifecycle_lock(lifecycle_mutex_); std::vector out; out.reserve(requests.size()); for (const auto& request : requests) out.push_back(PoolEvictResult{request.key}); if (backends_ == nullptr || requests.empty()) return out; - // A copy to free by deleting it: (request index, backend). Decided under - // the pool lock, carried out without it. + // (request index, backend), decided under the pool lock. std::vector> drops; - // A copy to free by moving it down a tier: (request index, source backend). std::vector> demotions; std::unique_lock operation_lock(operation_mutex_); @@ -973,17 +952,14 @@ std::vector PeerPool::Evict(const std::vector if (mode == PoolEvictMode::kReclaim) out[i].outcome = PoolEvictOutcome::kInFlight; continue; } - // A discard has to leave no copy behind, so it ignores scope, skips - // demotion and drops the key everywhere. + // A discard drops every copy, ignoring scope. if (mode == PoolEvictMode::kDiscard) { for (auto* backend : backends_->All()) drops.emplace_back(i, backends_->BackendId(backend)); continue; } - // A source left behind by an offload whose delete hit a read lease: the - // durable copy already lives downstream, so this copy is only ever deleted, - // never moved again. A request with no scope means "finish the drain", as - // it always has -- freeing the durable copy instead would undo the offload. + // A source parked by a lease-blocked offload is only ever deleted; an + // unscoped request means "finish the drain". auto draining = draining_sources_.find(request.key); const bool unscoped = request.tier == TierType::UNKNOWN && request.backend_id >= BackendRegistry::kMaxBackends; @@ -998,11 +974,8 @@ std::vector PeerPool::Evict(const std::vector continue; } - // Demote when this copy is the key's placement (the only copy a transition - // can start from) and its tier has somewhere to go. The trigger does not - // matter here: on_evict vs watermark only says whether the pool ALSO moves - // keys proactively, and either way dropping data a lower tier has room for - // would be the wrong answer to "free this tier". + // Demote the placement copy if its tier has an offload target, whatever + // the trigger. bool demote = false; if (tier_graph_ != nullptr) { auto placement = placements_.find(request.key); @@ -1017,9 +990,7 @@ std::vector PeerPool::Evict(const std::vector } } - // Queued, never copied here, so EvictKey never waits on bytes. The worker - // frees the source when the copy lands, or drops it if nothing downstream - // can take it (DropEvictedCopyLocked). + // Queued, so EvictKey never waits on a copy. for (const auto& [index, source_id] : demotions) { const auto tier_index = tier_graph_->TierIndexForBackendId(source_id); if (!tier_index.has_value()) { @@ -1031,16 +1002,13 @@ std::vector PeerPool::Evict(const std::vector if (EnqueueTransition(std::move(job))) { out[index].outcome = PoolEvictOutcome::kDemotionQueued; } else if (TransitionQueued(requests[index].key)) { - // Another job owns the key and will move or free it; deleting the - // source under it could lose the key. + // Another job will move it; deleting the source could lose the key. out[index].outcome = PoolEvictOutcome::kInFlight; } } if (drops.empty()) return out; - // The drops themselves run without the pool lock: deleting from SSD is - // device IO, and holding the lock across it would stall every allocate and - // commit on the peer. One Evict call per backend. + // Unlocked: an SSD delete is device IO. One Evict call per backend. operation_lock.unlock(); std::map> by_backend; for (const auto& [index, backend_id] : drops) by_backend[backend_id].push_back(index); @@ -1065,15 +1033,7 @@ std::vector PeerPool::Evict(const std::vector return out; } -// --------------------------------------------------------------------------- -// Local watermark eviction -// -// The policy half of eviction on a node that has to evict for itself. It -// decides when (a backend at its high watermark) and what (the medium's own -// coldest candidates), and then asks Evict() above -- the same call the -// master's EvictKey makes. It never frees a byte itself, and it runs on its -// own thread: a put only wakes it. -// --------------------------------------------------------------------------- +// ---- Local watermark eviction: picks when and what, then calls Evict(). ---- bool PeerPool::EnableLocalEviction(uint32_t backend_id, double high_watermark, double low_watermark) { @@ -1113,11 +1073,9 @@ LocalEvictionMetrics PeerPool::LocalEvictionStats() const { } void PeerPool::WakeLocalEviction() { - // A wake already pending covers this one. With local eviction off nothing - // ever clears the flag, so every call after the first returns right here. + // A pending wake covers this one (with eviction off, nothing clears it). if (local_evict_wake_.exchange(true, std::memory_order_acq_rel)) return; - // Notify under the lock the worker waits with, or a notify landing between - // its predicate check and its wait would be lost until the next poll. + // Under the worker's lock, so the notify can't slip between check and wait. std::lock_guard lock(local_evict_mutex_); local_evict_cv_.notify_one(); } @@ -1140,8 +1098,7 @@ void PeerPool::LocalEvictionLoop() { }); if (stop_local_evict_) return; } - // Cleared before the pass, so a commit that lands during it schedules the - // next one rather than being absorbed by this. + // Cleared first, so a commit during the pass schedules another. local_evict_wake_.store(false, std::memory_order_release); LocalEvictionPass(); std::this_thread::sleep_for(kLocalEvictMinGap); @@ -1157,9 +1114,7 @@ size_t PeerPool::LocalEvictionPass() { } if (targets.empty() || backends_ == nullptr) return 0; - // Lowest tier first -- SSD, then DRAM, then HBM. Evicting from an upper - // tier demotes into the one below, so freeing the lower one first is what - // lets that demotion find room instead of falling back to a drop. + // Lowest tier first, so upper tiers' demotions find room below. const auto depth = [this](uint32_t backend_id) { auto* backend = backends_->Get(backend_id); return backend == nullptr ? 0 : static_cast(backend->Tier()); @@ -1186,13 +1141,8 @@ size_t PeerPool::ReclaimBackend(uint32_t backend_id, const LocalEvictionWatermar if (used <= low_bytes) return 0; local_evict_rounds_.fetch_add(1, std::memory_order_relaxed); - // Cover what was over the low watermark at the start -- counted in the bytes - // each offer said it gives back, which is what Capacity() counts -- or stop - // once usage is back at the low watermark, whichever comes first. The budget - // is the guard against a medium whose reported usage lags its deletes, which - // re-reading usage alone would keep draining. - // Queued demotions count as covered: their source stays until the copy - // lands, and counting it stops a pass from queueing past the overage. + // Stop once offers cover the starting overage (usage can lag deletes) or + // usage reaches the low watermark. Queued demotions count as covered. const uint64_t budget = used - low_bytes; const TierType tier = backend->Tier(); uint64_t covered = 0; @@ -1224,8 +1174,7 @@ size_t PeerPool::ReclaimBackend(uint32_t backend_id, const LocalEvictionWatermar } break; } - // Only as many of the coldest as the remaining budget needs (at least one), - // and no more than one batch per Evict call. + // The coldest the budget still needs, at most one batch. std::vector requests; std::vector offered_bytes; uint64_t taken = 0; @@ -1240,9 +1189,6 @@ size_t PeerPool::ReclaimBackend(uint32_t backend_id, const LocalEvictionWatermar request.backend_id = backend_id; requests.push_back(std::move(request)); } - // The same entry point the master's EvictKey uses: a candidate whose tier - // has a downstream is queued for demotion, anything else is dropped from - // this backend. const auto results = Evict(requests, PoolEvictMode::kReclaim); bool progressed = false; for (size_t i = 0; i < results.size() && i < offered_bytes.size(); ++i) { @@ -1261,8 +1207,7 @@ size_t PeerPool::ReclaimBackend(uint32_t backend_id, const LocalEvictionWatermar } } if (!progressed) { - // Every candidate got leased or pinned since it was offered; the next - // wake or poll tries again. + // All got leased or pinned since offered; retry on the next wake. local_evict_stalled_.fetch_add(1, std::memory_order_relaxed); break; } diff --git a/src/umbp/distributed/pool_client.cpp b/src/umbp/distributed/pool_client.cpp index 6b6121b8d..93c5d920e 100644 --- a/src/umbp/distributed/pool_client.cpp +++ b/src/umbp/distributed/pool_client.cpp @@ -1063,9 +1063,7 @@ bool PoolClient::Init() { // every named backend, including multiple instances of the same medium. backend = MakeInstrumentedBackend(std::move(backend)); - // A node with no master has nobody to drain its heartbeat outbox, so the - // backend stops recording events addressed to a master that does not - // exist. (Its eviction is the pool's job; see EnableLocalEviction below.) + // With no master, nothing drains the heartbeat outbox. if (!master_client_) backend->SetEventPublishing(false); // Narrowed to MemoryRegistrar: a backend publishes endpoints, it does not @@ -1094,13 +1092,8 @@ bool PoolClient::Init() { default_pool_ = std::make_unique(®istry_, std::move(placement_policy), transfer_engine_.get()); - // Local watermark eviction. It goes through PeerPool::Evict, the same call - // a master's EvictKey makes, on the pool's own thread -- never on a put. - // Which media get it is unchanged: - // * no master: every medium, since nothing else would ever free a page; - // * with a master: SSD only, which has always reclaimed for itself between - // the master's periodic rounds. DRAM/HBM stay master-driven there. - // SSD uses the watermarks in its own config, paged media the node-wide pair. + // Local watermark eviction: every medium without a master, SSD only with + // one (as before). SSD uses its own watermarks, paged media the node pair. for (const auto& backend_config : backend_configs) { auto* backend = registry_.Get(backend_config.name); if (backend == nullptr) continue; diff --git a/src/umbp/distributed/proto/umbp_peer.proto b/src/umbp/distributed/proto/umbp_peer.proto index d2d90a269..a4d3cdaf9 100644 --- a/src/umbp/distributed/proto/umbp_peer.proto +++ b/src/umbp/distributed/proto/umbp_peer.proto @@ -142,10 +142,8 @@ message EvictKeyEntry { } message EvictKeyRequest { repeated string keys = 1; - // tiers[i] is the medium keys[i] must be freed from -- the one whose budget - // the master charged it to. Honoured only when it has exactly as many - // entries as `keys`; otherwise (an older master) each key is freed from the - // medium that currently holds it. An older peer ignores this field. + // The medium to free keys[i] from. Used only if sized like `keys`; older + // masters send none. repeated TierType tiers = 2; } message EvictKeyResponse { diff --git a/src/umbp/include/umbp/distributed/config.h b/src/umbp/include/umbp/distributed/config.h index 462b16034..db229a883 100644 --- a/src/umbp/include/umbp/distributed/config.h +++ b/src/umbp/include/umbp/distributed/config.h @@ -307,13 +307,8 @@ struct PoolClientConfig { // (2 MiB by default). Set to an explicit byte count to override. uint64_t dram_page_size = 0; - // Watermarks for local eviction of the paged media (DRAM/HBM), used ONLY - // when this node has no master (PoolClient::Init enables PeerPool's local - // eviction for them exactly then). With a master, their eviction is a - // cluster-wide decision shipped as EvictKey RPCs and these are ignored; - // without one, nothing else would ever free a page. Lowered from - // UMBPDramConfig::high_watermark / low_watermark. The SSD medium uses its - // own pair inside UMBPSsdConfig, with or without a master. + // Local eviction watermarks for DRAM/HBM, used only without a master (SSD + // has its own pair). Lowered from UMBPDramConfig. double local_evict_high_watermark = 0.9; double local_evict_low_watermark = 0.7; diff --git a/src/umbp/include/umbp/distributed/master/evict_strategy.h b/src/umbp/include/umbp/distributed/master/evict_strategy.h index 105e8a3d1..c0aacf5eb 100644 --- a/src/umbp/include/umbp/distributed/master/evict_strategy.h +++ b/src/umbp/include/umbp/distributed/master/evict_strategy.h @@ -31,18 +31,15 @@ namespace mori::umbp { -// Master-side victim selection. EvictionManager handles watermark detection, -// candidate gathering, and EvictKey dispatch; the strategy only ranks -// candidates and picks victims within the per-(node,tier) budget. +// Master-side victim selection within each (node, tier) budget. class MasterEvictStrategy { public: virtual ~MasterEvictStrategy() = default; // Pick victims from @p candidates within the per-(node,tier) byte budget // @p bytes_to_free. Both are by value so the impl may sort/decrement freely. - // Returns victims grouped by node_id (one EvictKey per peer); empty groups - // omitted. Each victim carries the tier whose budget it was charged to, so - // the peer frees that medium's copy and no other. + // Victims grouped by node_id (empty groups omitted), each with the tier it + // was charged to. virtual std::unordered_map> SelectVictims( std::vector candidates, std::unordered_map> bytes_to_free) = 0; diff --git a/src/umbp/include/umbp/distributed/master/eviction_manager.h b/src/umbp/include/umbp/distributed/master/eviction_manager.h index 9b2f55002..d03a14a02 100644 --- a/src/umbp/include/umbp/distributed/master/eviction_manager.h +++ b/src/umbp/include/umbp/distributed/master/eviction_manager.h @@ -43,8 +43,7 @@ class MasterEvictStrategy; // REMOVE events arrive on the peer's next heartbeat and that's where // the index actually shrinks. // -// Each victim names the medium to free the key from: the peer frees that -// copy and leaves any other copy of the key alone. +// Each victim names the medium to free the key from. // // The implementation lives in master_server.cpp (MasterPeerStubPool) // where the gRPC stub plumbing belongs. Tests can swap in a fake. diff --git a/src/umbp/include/umbp/distributed/master/master_metrics.h b/src/umbp/include/umbp/distributed/master/master_metrics.h index 448d26e51..23322a83e 100644 --- a/src/umbp/include/umbp/distributed/master/master_metrics.h +++ b/src/umbp/include/umbp/distributed/master/master_metrics.h @@ -182,9 +182,8 @@ "Reads served by each logical tier of this client's pool" // --- Local watermark eviction ---------------------------------------------- -// The keys and bytes freed are already in the backend evict series (every -// eviction is an Evict() call there); these are what only the pool's policy -// sees. event=round|key|no_candidate|stalled -- see LocalEvictionMetrics. +// event=round|key|no_candidate|stalled (LocalEvictionMetrics); bytes freed are +// in the backend evict series. #define MORI_UMBP_METRIC_CLIENT_LOCAL_EVICT "mori_umbp_client_local_evict_total" #define MORI_UMBP_METRIC_CLIENT_LOCAL_EVICT_HELP \ diff --git a/src/umbp/include/umbp/distributed/peer/backend/medium_backend.h b/src/umbp/include/umbp/distributed/peer/backend/medium_backend.h index b410cec4e..3a9bdd7e4 100644 --- a/src/umbp/include/umbp/distributed/peer/backend/medium_backend.h +++ b/src/umbp/include/umbp/distributed/peer/backend/medium_backend.h @@ -276,26 +276,14 @@ class MediumBackend : public MetricSource { // call (the default) simply has no outbox worth gating. virtual void SetEventPublishing(bool /*enabled*/) {} - // ---- eviction ordering ---- - // - // A backend never decides to evict. The master (EvictKey) or the pool's own - // local watermark eviction decides when and how much, both through - // PeerPool::Evict, and Evict() below is the one place a medium frees a - // committed key. What only the medium can supply is ORDER: it sees every - // read, including the resolve paths that never touch the pool's metadata. - // So it offers candidates, and nothing else. - - // Start keeping the recency order EvictionCandidates() reads. Off by - // default: a node whose eviction is all master-driven never asks for - // candidates, and keeping the order costs every read a touch. Idempotent, - // and safe after Init: keys already owned are seeded into the order. + // ---- eviction ordering: the pool decides; a medium only supplies order ---- + + // Start the recency order EvictionCandidates() reads. Off by default (it + // costs every read a touch). Idempotent; seeds keys already owned. virtual void TrackEvictionOrder() {} - // Up to `max_count` keys this medium could free right now, coldest first, - // each with the capacity freeing it gives back, skipping keys that a read - // lease, pin or in-flight read protects. Frees nothing: the caller passes - // the keys back through Evict(), which re-checks every one. A medium that - // keeps no order returns nothing and is therefore never evicted locally. + // Up to `max_count` unprotected keys, coldest first, with the bytes each + // frees. Frees nothing; Evict() re-checks. Empty if no order is kept. virtual std::vector EvictionCandidates(size_t /*max_count*/) { return {}; } // ---- observability ---- @@ -375,12 +363,8 @@ class MediumBackend : public MetricSource { return out; } - // The medium's ONE eviction implementation: free these keys' bytes on this - // medium, skipping any a read lease or pin protects, and queue a REMOVE for - // each key actually freed. Every eviction reaches here through - // PeerPool::Evict -- master-driven or local -- and nothing else in a backend - // may free a committed key for capacity. Idempotent; see - // EvictResult::bytes_freed. One result per key, in request order. + // The medium's only eviction path, reached via PeerPool::Evict: frees keys a + // lease or pin doesn't protect and queues a REMOVE for each. Idempotent. virtual std::vector Evict(const std::vector& keys) = 0; // ---- bootstrap (GetPeerInfo) ---- diff --git a/src/umbp/include/umbp/distributed/peer/backend/page_backend.h b/src/umbp/include/umbp/distributed/peer/backend/page_backend.h index 8dd522c94..190763a57 100644 --- a/src/umbp/include/umbp/distributed/peer/backend/page_backend.h +++ b/src/umbp/include/umbp/distributed/peer/backend/page_backend.h @@ -332,20 +332,9 @@ class PageBackend : public MediumBackend { void SetEventPublishing(bool enabled) override; // ==================== Eviction ordering ==================== - // - // This backend never decides to evict. The master, or the pool's local - // watermark eviction on a node without one, decides and calls Evict() -- - // the only code here that frees a committed key -- through PeerPool::Evict. - // What this backend supplies is the order: an LRU of owned keys, inserted by - // Commit and renewed by Resolve. - // - // The LRU is maintained only once TrackEvictionOrder() is called, so a node - // whose eviction is all master-driven pays nothing for it on the read path. + // An LRU of owned keys (Commit inserts, Resolve renews), kept only after + // TrackEvictionOrder(). Candidates skip what Evict() would refuse. void TrackEvictionOrder() override; - - // Coldest first, skipping keys under a read lease or a copy pin -- the same - // two protections Evict() applies, so a candidate is one Evict() would free - // unless it is touched in between. Empty until TrackEvictionOrder(). std::vector EvictionCandidates(size_t max_count) override; // ==================== Reaper ==================== diff --git a/src/umbp/include/umbp/distributed/peer/backend/ssd_backend.h b/src/umbp/include/umbp/distributed/peer/backend/ssd_backend.h index 35adb13b2..ebc1fc16a 100644 --- a/src/umbp/include/umbp/distributed/peer/backend/ssd_backend.h +++ b/src/umbp/include/umbp/distributed/peer/backend/ssd_backend.h @@ -167,9 +167,7 @@ class SsdBackend : public MediumBackend { void ReleaseMigrationRead(const std::string& key) override; bool Contains(const std::string& key) const override; std::vector Evict(const std::vector& keys) override; - // The manager's LRU, minus keys a staged read lease or migration read holds - // here -- Evict() would refuse those. The manager keeps its order always, so - // TrackEvictionOrder() needs nothing (the default). + // The manager's LRU (always kept) minus keys a staged or migration read holds. std::vector EvictionCandidates(size_t max_count) override; void SetAutoFlushHook(size_t threshold, std::function cb) override; diff --git a/src/umbp/include/umbp/distributed/peer/ssd/peer_ssd_manager.h b/src/umbp/include/umbp/distributed/peer/ssd/peer_ssd_manager.h index b10116c37..f2d1da996 100644 --- a/src/umbp/include/umbp/distributed/peer/ssd/peer_ssd_manager.h +++ b/src/umbp/include/umbp/distributed/peer/ssd/peer_ssd_manager.h @@ -53,13 +53,8 @@ struct SsdReadOutcome { // Peer-side owner of the local SSD tier: one SSD TierBackend + the // key->SSD-location map, capacity, the owned-location event outbox, the -// read-prepare path, and the SSD medium's one eviction implementation -// (Evict). Reached from the distributed data plane through -// SsdBackend : MediumBackend, which forwards to it. -// -// It decides nothing about eviction. When to evict and how much is -// PeerPool's local watermark eviction (or the master), which reaches Evict() -// through SsdBackend::Evict; this class only offers its LRU as candidates. +// read-prepare path, and Evict(). Reached through SsdBackend. It never +// decides to evict; it only offers its LRU as candidates. // // It reuses ONLY the low-level TierBackend (SSDTier). Peer DRAM is owned by // PageBackend, and a second thing here that also held DRAM and demoted between @@ -69,9 +64,7 @@ class PeerSsdManager { public: explicit PeerSsdManager(const PeerSsdConfig& cfg); - // Test-only: inject a ready-made backend so unit tests can drive the manager - // with a controllable (e.g. blocking) fake. Production code must use the - // config constructor. + // Test-only: inject a fake backend. explicit PeerSsdManager(std::unique_ptr backend, bool single_flight = true); ~PeerSsdManager(); @@ -115,30 +108,20 @@ class PeerSsdManager { // drive but one. // // Result length and order match @p keys. An already-owned key reports true - // without any device IO (LRU refreshed). A key the device has no room for - // fails: this never evicts to make room -- that is the pool's local - // eviction, on its own thread, at the SSD's watermark. + // without any device IO (LRU refreshed). A key with no room fails; this + // never evicts to make room. std::vector WriteBatch(const std::vector& keys, const std::vector& srcs, const std::vector& sizes); - // The SSD medium's ONE eviction implementation: free a single key. Read - // priority: a key with an in-flight PrepareRead or migration pin - // (inflight_reads_ > 0) is NOT evicted (returns false). - // Concurrency: marks the key in evicting_ under the lock, runs the backend - // evict outside the lock, and only on backend success removes owned_/lru_ and - // queues a REMOVE SSD event (so REMOVE is never emitted while the bytes still - // exist, and two callers cannot double-evict the same key). - // - // Reached only through SsdBackend::Evict, which PeerPool::Evict calls under - // its lifecycle lock -- the one ClearLocal takes exclusively -- so an - // eviction is never in flight during a Clear. + // Frees one key; false if it is being read or pinned (inflight_reads_ > 0). + // Marks the key in evicting_ under the lock, evicts outside it, and only on + // success removes owned_/lru_ and queues a REMOVE (never while the bytes + // still exist; no double evict). PeerPool::Evict holds the lifecycle lock, + // so this never overlaps a Clear. bool Evict(const std::string& key); - // Up to @p max_count keys, coldest first (LRU tail -> MRU front), skipping - // keys that are being read or pinned (inflight_reads_) or already being - // evicted (evicting_). Ordering only: frees nothing, and Evict() re-checks - // every key. Never blocks on IO. + // Coldest first, skipping keys being read, pinned or evicted. Frees nothing. std::vector EvictionCandidates(size_t max_count); // Distributed Clear: drop the logical owned-location map + undrained events, diff --git a/src/umbp/include/umbp/distributed/pool/peer_pool.h b/src/umbp/include/umbp/distributed/pool/peer_pool.h index ef179ab7a..6adac00f8 100644 --- a/src/umbp/include/umbp/distributed/pool/peer_pool.h +++ b/src/umbp/include/umbp/distributed/pool/peer_pool.h @@ -86,50 +86,32 @@ struct PoolEvictResult { PoolEvictOutcome outcome = PoolEvictOutcome::kNone; }; -// Local watermark eviction counters (see PeerPool::EnableLocalEviction). The -// keys and bytes actually freed are not here: every eviction is an Evict() -// call on a backend, and InstrumentedBackend already counts those. These are -// what only the policy can see. +// Local watermark eviction counters; bytes freed are counted by +// InstrumentedBackend. struct LocalEvictionMetrics { - // Passes that found a backend at or above its high watermark. - uint64_t rounds = 0; - // Keys the passes evicted: dropped, or queued for demotion. - uint64_t keys = 0; - // Passes the medium offered no candidate for: everything it holds is under - // a read lease or pin, or it keeps no eviction order at all. - uint64_t no_candidate = 0; - // Batches in which every candidate freed nothing (mid-transition, or busy by - // the time Evict reached it). The pass stops and the next wake retries. - uint64_t stalled = 0; + uint64_t rounds = 0; // passes that found a backend over its high watermark + uint64_t keys = 0; // keys dropped or queued for demotion + uint64_t no_candidate = 0; // passes where every key was leased or pinned + uint64_t stalled = 0; // batches that freed nothing; the next wake retries }; // Why a key is being removed. A tiered pool answers the two intents // differently, so the caller has to say which one it means instead of leaving // it to the tier configuration. enum class PoolEvictMode { - // Free the bytes of ONE copy of the key: the one the request names (see - // PoolEvictRequest). Where that copy's logical tier has an offload target the - // key is demoted, not dropped -- the pressure being relieved is on the named - // medium, so moving the bytes downstream relieves it and the data stays - // reachable. Any other copy of the key is left alone. + // Free the one copy the request names: demoted if its tier has an offload + // target, else dropped. Other copies are left alone. kReclaim, // Drop the key from every backend on the peer regardless of tier // configuration or scope, for when the value itself must stop existing. kDiscard, }; -// One key to evict, and which of its copies. A key can live on several media -// at once (a copy-mode promotion leaves the lower copy behind), so "evict this -// key" is ambiguous: the request says which medium is under pressure. +// One key to evict, and which of its copies (a key can live on several media). struct PoolEvictRequest { std::string key; - // The medium to free the key from -- the tier a master charged it to. - // UNKNOWN means the medium that currently holds the key's placement, which - // is how a request with no tier (an older master) is read. - TierType tier = TierType::UNKNOWN; - // One specific backend instead of a medium. Local watermark eviction knows - // exactly which backend is over its watermark. Takes precedence over tier. - uint32_t backend_id = BackendRegistry::kMaxBackends; + TierType tier = TierType::UNKNOWN; // UNKNOWN: the medium holding the placement + uint32_t backend_id = BackendRegistry::kMaxBackends; // overrides tier when set }; // The implicit peer-local default Pool. It owns logical key placement and @@ -154,12 +136,9 @@ class PeerPool { // check discards. Takes no read lease and does not touch recency: asking // whether a key is here is not a read of it. std::vector BatchContains(const std::vector& keys); - // The one routing entry point for eviction. Every policy calls this -- the - // master's EvictKey RPC and this pool's own local watermark eviction alike -- - // and every byte it frees is freed by the medium's MediumBackend::Evict, - // directly or as the last step of a demotion. Drops happen before this - // returns; demotions are only queued, so it never waits on a byte copy. One - // result per request, in request order. + // The single eviction entry point (master EvictKey and local eviction). + // Drops happen before it returns; demotions are only queued. One result per + // request, in order. std::vector Evict(const std::vector& requests, PoolEvictMode mode); // Every key unscoped: freed from the medium that currently holds it. @@ -177,22 +156,12 @@ class PeerPool { // a tier policy is actually keeping hot data where it was supposed to. std::map TierReadHits() const; - // ---- local watermark eviction ---- - // - // Turn on watermark eviction for one backend: once its usage reaches `high`, - // the pool frees its coldest keys (MediumBackend::EvictionCandidates) down to - // `low` by calling Evict() above -- the same call the master's EvictKey - // makes, so a key whose tier has a downstream is demoted, not deleted. It - // runs on one background thread, started by the first call; a put only wakes - // it and never evicts itself. PoolClient enables it for every backend on a - // node with no master, and for SSD always. - // - // Returns false and changes nothing for an unknown backend or watermarks - // outside 0 < low < high <= 1. + // Background watermark eviction for one backend: at `high` usage, Evict() + // its coldest keys down to `low`. Puts only wake the worker. False for an + // unknown backend or watermarks outside 0 < low < high <= 1. bool EnableLocalEviction(uint32_t backend_id, double high_watermark, double low_watermark); LocalEvictionMetrics LocalEvictionStats() const; - // Test seam: one synchronous pass over every enabled backend, as the worker - // runs it. Returns the number of keys dropped or queued for demotion. + // Test seam: one synchronous pass. Returns keys dropped or queued. size_t RunLocalEvictionOnceForTest() { return LocalEvictionPass(); } private: @@ -208,9 +177,7 @@ class PeerPool { uint32_t source_backend_id = BackendRegistry::kMaxBackends; LogicalTierGraph::TierIndex source_tier = 0; uint8_t attempts = 0; - // Queued by Evict(): runs regardless of the tier's watermark, and drops - // the copy if nothing downstream can take it. - bool evict = false; + bool evict = false; // from Evict(): ignores watermarks; drops on failure }; // What the copy phase needs, resolved while the pool lock is still held. @@ -220,30 +187,21 @@ class PeerPool { std::vector targets; }; - // False if the key already has a job (queued or running) or the worker is - // stopping. + // False if the key already has a job or the worker is stopping. bool EnqueueTransition(TransitionJob job); - // Whether a job owns `key`, queued or running. - bool TransitionQueued(const std::string& key); + bool TransitionQueued(const std::string& key); // queued or running void TransitionWorkerLoop(); void StopTransitionWorker(); - // A failed eviction demotion drops the copy instead, unless another - // transition owns the key. void DropEvictedCopyLocked(const TransitionJob& job); struct LocalEvictionWatermarks { double high = 0.0; double low = 0.0; }; - // Cheap enough for the put path: one atomic exchange, and a notify only when - // no wake is already pending. Never reads capacity. - void WakeLocalEviction(); + void WakeLocalEviction(); // put path: one atomic exchange, no capacity read void LocalEvictionLoop(); void StopLocalEviction(); - // One pass: every enabled backend, lowest tier first. Returns keys evicted. - size_t LocalEvictionPass(); - // Brings one backend at or above its high watermark down to its low one, - // counting in-flight demotions. Returns keys evicted. + size_t LocalEvictionPass(); // lowest tier first; returns keys evicted size_t ReclaimBackend(uint32_t backend_id, const LocalEvictionWatermarks& watermarks); void TouchLocked(const std::string& key); @@ -256,12 +214,10 @@ class PeerPool { // is process-local and can be stale or empty (after a restart, or once // another path evicted the key), so the backends are the authority. uint32_t FindOwnerLocked(const std::string& key) const; - // The backend holding the copy an eviction request names, or kMaxBackends - // when there is no such copy (see PoolEvictRequest for how scope resolves). + // The backend holding the copy a request names, or kMaxBackends. uint32_t EvictSourceLocked(const PoolEvictRequest& request) const; - // After copies of `key` were dropped: point the placement at a copy that is - // still there, or forget the key if none is. Re-checks the backends rather - // than trusting what was dropped, because drops run without the pool lock. + // Re-points the placement at a surviving copy (or forgets the key); drops + // run unlocked, so it re-checks the backends. void RepairAfterDropLocked(const std::string& key); // Queues at most `max_count` offload candidates for one tier, oldest first, // and reports how many it queued. Bounded on purpose: a peer can hold @@ -338,18 +294,15 @@ class PeerPool { bool stop_transition_worker_ = false; std::thread transition_worker_; - // ---- local watermark eviction (see EnableLocalEviction) ---- - // Guards local_evict_, stop_local_evict_ and the worker's start. + // Local watermark eviction. local_evict_mutex_ guards local_evict_, + // stop_local_evict_ and the worker's start. mutable std::mutex local_evict_mutex_; std::condition_variable local_evict_cv_; std::map local_evict_; bool stop_local_evict_ = false; - // Set by a wake, cleared by the worker at the start of a pass. Atomic so a - // commit that finds a wake already pending touches no lock at all. - std::atomic local_evict_wake_{false}; + std::atomic local_evict_wake_{false}; // lets a put skip the lock std::thread local_evict_worker_; - // One pass at a time: the worker and the test seam share LocalEvictionPass. - std::mutex local_evict_pass_mutex_; + std::mutex local_evict_pass_mutex_; // one pass at a time (worker vs test seam) std::atomic local_evict_rounds_{0}; std::atomic local_evict_keys_{0}; std::atomic local_evict_no_candidate_{0}; diff --git a/src/umbp/include/umbp/distributed/types.h b/src/umbp/include/umbp/distributed/types.h index 01f351c67..163884786 100644 --- a/src/umbp/include/umbp/distributed/types.h +++ b/src/umbp/include/umbp/distributed/types.h @@ -160,10 +160,7 @@ struct EvictionCandidate { uint64_t size; }; -// One master eviction decision: free `key` from the `tier` medium of a node. -// The tier is the one whose budget the victim was charged to. A key over -// budget in two media of the same node is two victims, one per medium; without -// the tier the peer could only guess which copy the master meant. +// One master eviction decision: free `key`'s copy on the `tier` medium. struct EvictionVictim { std::string key; TierType tier = TierType::UNKNOWN; @@ -173,11 +170,7 @@ struct EvictionVictim { } }; -// One key a medium offers for local eviction (MediumBackend:: -// EvictionCandidates), with the bytes freeing it gives back to that medium's -// capacity -- page-rounded where the medium allocates in pages. The size is -// what lets the caller take exactly as many offers as it needs instead of a -// whole batch. +// A key offered for local eviction, with the (page-rounded) bytes it frees. struct EvictionOffer { std::string key; uint64_t bytes = 0; diff --git a/src/umbp/local/tiers/spdk_ssd_tier.cpp b/src/umbp/local/tiers/spdk_ssd_tier.cpp index 3fd7d4992..f717ae295 100644 --- a/src/umbp/local/tiers/spdk_ssd_tier.cpp +++ b/src/umbp/local/tiers/spdk_ssd_tier.cpp @@ -248,12 +248,8 @@ std::vector SpdkSsdTier::GetLRUCandidates(size_t max_candidates) co // 1) Checks existing keys (shard shared lock + LRU update) // 2) Batch allocates space for new keys // -// A key that does not fit FAILS. The tier never evicts to make room: the -// keys it would pick belong to whoever owns this tier (PeerSsdManager, over -// the proxy), which tracks them and reports their REMOVE to the master. An -// eviction here would be invisible to both -- the owner would keep listing a -// key whose bytes are gone. Making room is the owner's local eviction, which -// reaches this tier through Evict(). +// A key that does not fit fails. The tier never evicts on its own: its owner +// (PeerSsdManager) tracks the keys and would keep listing ones it dropped. // =========================================================================== std::vector SpdkSsdTier::PrepareWriteAlloc( const std::vector& keys, const std::vector& sizes, diff --git a/tests/cpp/umbp/distributed/test_master_eviction_chain.cpp b/tests/cpp/umbp/distributed/test_master_eviction_chain.cpp index 61c503f89..dd5b74fe7 100644 --- a/tests/cpp/umbp/distributed/test_master_eviction_chain.cpp +++ b/tests/cpp/umbp/distributed/test_master_eviction_chain.cpp @@ -574,8 +574,7 @@ size_t FirstRoundVictims(const EvictionConfig& config) { } TEST(MasterEvictionBytes, ARoundFreesTheOverageUpToTheByteLimit) { - // The overage above the 0.7 low watermark is 77 keys, more than the 32 a - // round used to stop at. + // 77 keys over the 0.7 low watermark; rounds used to stop at 32. EXPECT_EQ(FirstRoundVictims(OneSecondRounds()), 77u); ::setenv("UMBP_EVICT_MAX_BYTES_PER_ROUND", std::to_string(10 * kPageSize).c_str(), 1); diff --git a/tests/cpp/umbp/distributed/test_peer_pool.cpp b/tests/cpp/umbp/distributed/test_peer_pool.cpp index 87a8109b5..a8f63883a 100644 --- a/tests/cpp/umbp/distributed/test_peer_pool.cpp +++ b/tests/cpp/umbp/distributed/test_peer_pool.cpp @@ -120,8 +120,7 @@ bool WaitForKeyGone(BackendRegistry* registry, const std::string& backend, const return !registry->Get(backend)->Contains(key); } -// A LocalCopyEngine that holds every copy until Release() -- or 10 s, so a -// failed test cannot hang the pool's teardown. +// Holds every copy until Release(), or 10 s so a failed test can't hang. class GatedEngine final : public TransferEngine { public: const char* Name() const override { return "gated"; } @@ -625,8 +624,7 @@ TEST(PeerPool, OnEvictMigratesBytesToConfiguredBackend) { .front() .commit.success); - // Evict only queues the demotion: it returns while the copy is held, and - // asking again does not queue it twice. + // Evict returns while the copy is held, and does not queue it twice. auto evicted = pool.Evict({"move"}, PoolEvictMode::kReclaim); ASSERT_EQ(evicted.size(), 1u); EXPECT_EQ(evicted.front().outcome, PoolEvictOutcome::kDemotionQueued); @@ -746,8 +744,7 @@ TEST(PeerPool, TierScopedEvictFreesOnlyThatMediumsCopy) { EXPECT_TRUE(registry.Get("hot")->Contains("a")); EXPECT_EQ(pool.PlacementBackend("a"), std::optional{hot}); - // DRAM scope: cold already holds the key, so the demotion frees DRAM without - // a copy. + // DRAM scope: cold already holds the key, so DRAM frees without a copy. evicted = pool.Evict({PoolEvictRequest{"b", TierType::DRAM}}, PoolEvictMode::kReclaim); EXPECT_EQ(evicted.front().outcome, PoolEvictOutcome::kDemotionQueued); ASSERT_TRUE(WaitForKeyGone(®istry, "hot", "b")); @@ -803,8 +800,7 @@ TEST(PeerPool, ReclaimDemotesOnAWatermarkTierToo) { } TEST(PeerPool, DemotionWithNoRoomDownstreamDropsTheCopy) { - // Reclaim still frees the tier when the tier below is full. Host-page-sized - // pages, so `cold` really is one page long. + // Reclaim still frees the tier when the tier below is full (cold: one page). constexpr uint64_t kPageSize = 4096; LocalCopyEngine engine; BackendRegistry registry; @@ -1127,10 +1123,7 @@ TEST(PeerPool, ReadPromotesWithMoveDrainsSourceAndKeepsBytes) { payload); } -// --------------------------------------------------------------------------- -// Local watermark eviction: a background worker that frees through Evict(), -// so it follows the same demote-or-drop contract as the tests above. -// --------------------------------------------------------------------------- +// ---- Local watermark eviction (frees through Evict(), as above) ---- bool WaitUntil(const std::function& done, std::chrono::milliseconds budget = std::chrono::seconds(5)) { @@ -1194,9 +1187,8 @@ TEST(PeerPoolLocalEviction, KeysUnderAReadLeaseAreNeverOffered) { } TEST(PeerPoolLocalEviction, ATierWithADownstreamDemotesJustTheOverage) { - // Same call as a master evict: a tier with a downstream demotes, never drops. - // A queued demotion frees its source only when the copy lands, so passes run - // meanwhile must count it, or they would queue the whole tier. + // A tier with a downstream demotes, and passes that run while copies are + // held count them rather than queueing the whole tier. GatedEngine engine; BackendRegistry registry; RegisterPageBackend(®istry, &engine, "hot", TierType::DRAM, kHostPage, 10); @@ -1226,8 +1218,7 @@ TEST(PeerPoolLocalEviction, ATierWithADownstreamDemotesJustTheOverage) { } TEST(PeerPoolLocalEviction, NeverDeletesWhatAWatermarkOffloadWouldMove) { - // Regression: the old in-commit round deleted the cold keys a watermark - // offload was about to move. Now both move them, so every key survives. + // Regression: the old in-commit round deleted keys offload was about to move. LocalCopyEngine engine; BackendRegistry registry; RegisterPageBackend(®istry, &engine, "hot", TierType::DRAM, kHostPage, 10); diff --git a/tests/cpp/umbp/distributed/test_peer_ssd_eviction.cpp b/tests/cpp/umbp/distributed/test_peer_ssd_eviction.cpp index a0762a5ae..d1e1a6c87 100644 --- a/tests/cpp/umbp/distributed/test_peer_ssd_eviction.cpp +++ b/tests/cpp/umbp/distributed/test_peer_ssd_eviction.cpp @@ -20,10 +20,8 @@ // OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE // SOFTWARE. // -// The SSD medium's eviction primitive and candidate order, driven through an -// in-memory TierBackend (the test-only constructor) so LRU order, the -// in-flight-read guard, idempotent Write, evict failure, concurrency and Clear -// are deterministic. Watermark eviction is PeerPool's, tested there. +// The SSD medium's Evict() and candidate order, over an in-memory TierBackend. +// Watermark eviction is PeerPool's, tested there. #include #include @@ -194,8 +192,7 @@ TEST(PeerSsdEviction, WriteAndPrepareReadRefreshLru) { } TEST(PeerSsdEviction, WritesNeverEvict) { - // At 900 of 1000 (the old watermark round) and on ENOSPC alike, a write - // leaves every earlier key in place: eviction is the pool's job. + // Neither 90% usage nor ENOSPC makes a write evict earlier keys. auto h = MakeHarness(/*capacity=*/1000); std::string val(100, 'x'); for (int i = 1; i <= 9; ++i) { diff --git a/tests/cpp/umbp/distributed/test_ssd_backend.cpp b/tests/cpp/umbp/distributed/test_ssd_backend.cpp index ad870e942..8397c11ca 100644 --- a/tests/cpp/umbp/distributed/test_ssd_backend.cpp +++ b/tests/cpp/umbp/distributed/test_ssd_backend.cpp @@ -557,9 +557,7 @@ TEST_F(SsdBackendTest, EvictResultsAreOnePerKeyInRequestOrder) { EXPECT_EQ(evicted[1].bytes_freed, 1024u); } -// --------------------------------------------------------------------------- -// Local eviction: the backend offers order; the pool decides -// --------------------------------------------------------------------------- +// ---- Local eviction: the backend offers order; the pool decides ---- TEST_F(SsdBackendTest, CandidatesAreColdestFirstAndSkipALeasedKey) { SsdBackend* backend = Start(); From 34e67ed2623383e67e9d03651dd8ca6413723df9 Mon Sep 17 00:00:00 2001 From: TianDi101 Date: Tue, 13 Oct 2026 08:24:23 +0000 Subject: [PATCH 10/19] feat(umbp): peer-driven eviction with master-ranked victims (opt-in) With a master, a node can evict at its own watermarks instead of waiting for the master's EvictKey rounds, while still taking its victims from the master's cluster-wide recency. - Master: new GetEvictionCandidates RPC returns a node's keys on one medium (optionally one logical tier), least recently accessed first, until their sizes cover the requested bytes; master-leased keys are skipped. RegisterClient and every Heartbeat carry peer_driven_eviction, and EvictionManager leaves such nodes out of its rounds (in memory only). ClientRegistryConfig::advertise_peer_driven_eviction gates both. - Peer: local eviction asks a candidate source first (at most once per 200 ms per backend), rounds its value sizes up to the backend's pages, evicts through PeerPool::Evict, and covers the rest from the medium's own LRU. In peer-driven mode PoolClient runs local eviction for every medium, at a tiered policy's per-tier watermarks when there is one. - Opt-in: PoolClientConfig::peer_driven_eviction or UMBP_PEER_DRIVEN_EVICTION=1, and only once the master confirms support; otherwise behaviour is unchanged. New metric event master_key. Co-Authored-By: Claude Opus 5.5 --- .../distributed/master/eviction_manager.cpp | 46 +++++ src/umbp/distributed/master/master_client.cpp | 37 ++++ src/umbp/distributed/master/master_server.cpp | 39 +++++ src/umbp/distributed/pool/peer_pool.cpp | 123 ++++++++++---- src/umbp/distributed/pool_client.cpp | 64 +++++-- src/umbp/distributed/proto/umbp.proto | 24 +++ src/umbp/doc/runtime-env-vars.md | 1 + src/umbp/include/umbp/distributed/config.h | 14 +- .../distributed/master/eviction_manager.h | 25 +++ .../umbp/distributed/master/master_client.h | 17 ++ .../umbp/distributed/master/master_metrics.h | 4 +- .../include/umbp/distributed/pool/peer_pool.h | 22 ++- .../include/umbp/distributed/pool_client.h | 1 + src/umbp/include/umbp/distributed/types.h | 10 ++ tests/cpp/umbp/distributed/CMakeLists.txt | 13 ++ .../test_master_eviction_chain.cpp | 36 ++++ .../distributed/test_peer_driven_eviction.cpp | 160 ++++++++++++++++++ tests/cpp/umbp/distributed/test_peer_pool.cpp | 32 ++++ 18 files changed, 607 insertions(+), 61 deletions(-) create mode 100644 tests/cpp/umbp/distributed/test_peer_driven_eviction.cpp diff --git a/src/umbp/distributed/master/eviction_manager.cpp b/src/umbp/distributed/master/eviction_manager.cpp index 15938a0bf..c96311eb0 100644 --- a/src/umbp/distributed/master/eviction_manager.cpp +++ b/src/umbp/distributed/master/eviction_manager.cpp @@ -50,6 +50,51 @@ EvictionManager::EvictionManager(IMasterMetadataStore& store, const EvictionConf EvictionManager::~EvictionManager() { Stop(); } +void EvictionManager::SetPeerDriven(const std::string& node_id, bool peer_driven) { + std::lock_guard lock(peer_driven_mutex_); + if (peer_driven) { + if (peer_driven_.insert(node_id).second) { + MORI_UMBP_INFO("[EvictionManager] node {} evicts for itself; no EvictKey rounds", node_id); + } + } else { + peer_driven_.erase(node_id); + } +} + +bool EvictionManager::PeerDriven(const std::string& node_id) const { + std::lock_guard lock(peer_driven_mutex_); + return peer_driven_.count(node_id) != 0; +} + +std::vector SelectPeerEvictionCandidates( + const IMasterMetadataStore& store, const std::string& node_id, TierType tier, + const std::string& logical_tier, uint64_t bytes, size_t max_keys, + std::chrono::system_clock::time_point now) { + if (max_keys == 0 || max_keys > kMaxPeerEvictionCandidates) { + max_keys = kMaxPeerEvictionCandidates; + } + const NodeTierKey bucket{node_id, tier}; + // A logical-tier filter drops rows after the store returns them, so it + // cannot cap. + auto by_bucket = store.EnumerateEvictionCandidates( + {bucket}, EvictionOrder::kLeastRecentlyAccessed, logical_tier.empty() ? max_keys : 0, now); + auto it = by_bucket.find(bucket); + if (it == by_bucket.end()) return {}; + auto& rows = it->second; + // The order is only a hint to the store. + std::sort(rows.begin(), rows.end(), + [](const auto& a, const auto& b) { return a.last_accessed_at < b.last_accessed_at; }); + std::vector out; + uint64_t covered = 0; + for (auto& row : rows) { + if (out.size() >= max_keys || covered >= bytes) break; + if (!logical_tier.empty() && row.location.logical_tier != logical_tier) continue; + covered += row.size; + out.push_back(std::move(row)); + } + return out; +} + void EvictionManager::Start() { if (running_.load(std::memory_order_relaxed)) return; running_.store(true, std::memory_order_relaxed); @@ -100,6 +145,7 @@ void EvictionManager::RunOnce() { std::map bytes_to_free; for (const auto& client : clients) { + if (PeerDriven(client.node_id)) continue; for (const auto& [tier, cap] : client.tier_capacities) { // No tier is special-cased here any more (backend-agnostic refactor // Phase 4). The old `if (tier == SSD) continue` existed because EvictKey diff --git a/src/umbp/distributed/master/master_client.cpp b/src/umbp/distributed/master/master_client.cpp index f15b5112d..a85d9ddb2 100644 --- a/src/umbp/distributed/master/master_client.cpp +++ b/src/umbp/distributed/master/master_client.cpp @@ -204,6 +204,7 @@ grpc::Status MasterClient::RegisterSelf( FillTierCapacities(req.mutable_tier_capacities(), tier_capacities); FillLogicalTierCapacities(req.mutable_logical_tier_capacities(), logical_tier_capacities); for (const auto& tag : config_.tags) req.add_tags(tag); + req.set_peer_driven_eviction(peer_driven_requested_); ::umbp::RegisterClientResponse resp; grpc::ClientContext ctx; @@ -217,6 +218,7 @@ grpc::Status MasterClient::RegisterSelf( if (resp.heartbeat_interval_ms() > 0) heartbeat_interval_ms_ = resp.heartbeat_interval_ms(); supports_max_allocatable_capacity_ = resp.supports_max_allocatable_bytes(); supports_logical_tiers_ = resp.supports_logical_tiers(); + supports_peer_driven_eviction_ = resp.supports_peer_driven_eviction(); aggregate_backend_capacities_ = aggregate_backend_capacities_requested_ && supports_max_allocatable_capacity_; { @@ -445,6 +447,38 @@ grpc::Status MasterClient::BatchLookup(const std::vector& keys, return grpc::Status::OK; } +std::optional> MasterClient::GetEvictionCandidates( + TierType tier, const std::string& logical_tier, uint64_t bytes, uint32_t max_keys) { + ScopedRpcTimer _rpc_timer(this, "GetEvictionCandidates"); + ::umbp::GetEvictionCandidatesRequest req; + req.set_node_id(config_.node_id); + req.set_tier(ToProtoTier(tier)); + req.set_logical_tier(logical_tier); + req.set_bytes(bytes); + req.set_max_keys(max_keys); + + ::umbp::GetEvictionCandidatesResponse resp; + grpc::ClientContext ctx; + // Eviction falls back to the medium's own order rather than wait long. + ctx.set_deadline(std::chrono::system_clock::now() + std::chrono::seconds(1)); + auto status = GetStub(stub_.get())->GetEvictionCandidates(&ctx, req, &resp); + _rpc_timer.SetStatus(status); + if (!status.ok()) { + static std::atomic failures{0}; + const uint64_t n = failures.fetch_add(1, std::memory_order_relaxed) + 1; + if (n <= 8 || n % 1000 == 0) { + MORI_UMBP_WARN("[Client] GetEvictionCandidates failed ({} so far): {}", n, + status.error_message()); + } + return std::nullopt; + } + const int n = std::min(resp.keys_size(), resp.sizes_size()); + std::vector out; + out.reserve(static_cast(n)); + for (int i = 0; i < n; ++i) out.push_back(EvictionOffer{resp.keys(i), resp.sizes(i)}); + return out; +} + namespace { // Auto-flush the heartbeat once this many KvEvents accumulate at the peer, so a // completed batch of puts becomes visible at the master without the application @@ -742,6 +776,7 @@ bool MasterClient::SendDeltaHeartbeatLocked(const std::mapLogicalTierCapacities()); } for (const auto& tag : config_.tags) re_req.add_tags(tag); + re_req.set_peer_driven_eviction(peer_driven_requested_); ::umbp::RegisterClientResponse re_resp; grpc::ClientContext re_ctx; grpc::Status re_status; @@ -754,6 +789,7 @@ bool MasterClient::SendDeltaHeartbeatLocked(const std::mapnode_id(), request->peer_driven_eviction()); UpdateClientCountMetric(); UpdateClientCapacityMetrics(request->node_id(), caps); UpdateClientLogicalTierMetrics(request->node_id(), registration.logical_tier_capacities); @@ -301,6 +302,7 @@ class MasterServer::UMBPMasterServiceImpl final : public ::umbp::UMBPMaster::Ser response->set_ack_seq(0); response->set_supports_max_allocatable_bytes(config_.advertise_max_allocatable_bytes); response->set_supports_logical_tiers(config_.advertise_logical_tiers); + response->set_supports_peer_driven_eviction(config_.advertise_peer_driven_eviction); return grpc::Status::OK; }); } @@ -310,6 +312,7 @@ class MasterServer::UMBPMasterServiceImpl final : public ::umbp::UMBPMaster::Ser ::umbp::UnregisterClientResponse* /*response*/) override { return GuardStore("UnregisterClient", [&]() -> grpc::Status { store_.UnregisterClient(request->node_id()); + NotePeerDriven(request->node_id(), false); UpdateClientCountMetric(); return grpc::Status::OK; }); @@ -320,6 +323,7 @@ class MasterServer::UMBPMasterServiceImpl final : public ::umbp::UMBPMaster::Ser grpc::Status Heartbeat(grpc::ServerContext* /*ctx*/, const ::umbp::HeartbeatRequest* request, ::umbp::HeartbeatResponse* response) override { return GuardStore("Heartbeat", [&]() -> grpc::Status { + NotePeerDriven(request->node_id(), request->peer_driven_eviction()); std::map caps; for (const auto& tc : request->tier_capacities()) { TierCapacity c; @@ -609,6 +613,30 @@ class MasterServer::UMBPMasterServiceImpl final : public ::umbp::UMBPMaster::Ser }); } + grpc::Status GetEvictionCandidates(grpc::ServerContext* /*ctx*/, + const ::umbp::GetEvictionCandidatesRequest* request, + ::umbp::GetEvictionCandidatesResponse* response) override { + if (!config_.advertise_peer_driven_eviction) { + return grpc::Status(grpc::StatusCode::UNIMPLEMENTED, "peer-driven eviction is disabled"); + } + return GuardStore("GetEvictionCandidates", [&]() -> grpc::Status { + if (request->node_id().empty()) { + return grpc::Status(grpc::StatusCode::INVALID_ARGUMENT, "node_id must not be empty"); + } + const auto rows = SelectPeerEvictionCandidates( + store_, request->node_id(), static_cast(request->tier()), + request->logical_tier(), request->bytes(), request->max_keys(), + std::chrono::system_clock::now()); + response->mutable_keys()->Reserve(static_cast(rows.size())); + response->mutable_sizes()->Reserve(static_cast(rows.size())); + for (const auto& row : rows) { + response->add_keys(row.key); + response->add_sizes(row.size); + } + return grpc::Status::OK; + }); + } + // -------- External KV mutation/query -------- grpc::Status ReportExternalKvBlocks( @@ -831,8 +859,17 @@ class MasterServer::UMBPMasterServiceImpl final : public ::umbp::UMBPMaster::Ser } void SetMetrics(mori::metrics::MetricsServer* metrics) { metrics_ = metrics; } + void SetEvictionManager(EvictionManager* eviction_manager) { + eviction_manager_ = eviction_manager; + } private: + void NotePeerDriven(const std::string& node_id, bool peer_driven) { + if (eviction_manager_ == nullptr || node_id.empty()) return; + eviction_manager_->SetPeerDriven(node_id, + peer_driven && config_.advertise_peer_driven_eviction); + } + void UpdateClientCountMetric() { if (!metrics_) return; metrics_->setGauge(MORI_UMBP_METRIC_CLIENT_COUNT, MORI_UMBP_METRIC_CLIENT_COUNT_HELP, @@ -903,6 +940,7 @@ class MasterServer::UMBPMasterServiceImpl final : public ::umbp::UMBPMaster::Ser Router& router_; ClientRegistryConfig config_; mori::metrics::MetricsServer* metrics_ = nullptr; + EvictionManager* eviction_manager_ = nullptr; // non-owning; set before Run() }; // --------------------------------------------------------------------------- @@ -919,6 +957,7 @@ MasterServer::MasterServer(MasterServerConfig config) peer_stub_pool_.get(), std::move(config_.evict_strategy))) { router_.SetLeaseDuration(config_.eviction_config.lease_duration); + service_->SetEvictionManager(eviction_manager_.get()); } MasterServer::~MasterServer() { diff --git a/src/umbp/distributed/pool/peer_pool.cpp b/src/umbp/distributed/pool/peer_pool.cpp index 7c6650870..12b28c3ba 100644 --- a/src/umbp/distributed/pool/peer_pool.cpp +++ b/src/umbp/distributed/pool/peer_pool.cpp @@ -50,6 +50,8 @@ constexpr size_t kLocalEvictBatch = 64; // Wakes are the main signal; the poll catches growth no wake reported. constexpr auto kLocalEvictPollInterval = std::chrono::milliseconds(100); constexpr auto kLocalEvictMinGap = std::chrono::milliseconds(1); // between passes +// Per backend, between candidate-source queries. +constexpr auto kRankedCandidatesMinInterval = std::chrono::milliseconds(200); uint64_t UsedBytes(const TierCapacity& capacity) { return capacity.total_bytes > capacity.available_bytes @@ -1069,6 +1071,7 @@ LocalEvictionMetrics PeerPool::LocalEvictionStats() const { metrics.keys = local_evict_keys_.load(std::memory_order_relaxed); metrics.no_candidate = local_evict_no_candidate_.load(std::memory_order_relaxed); metrics.stalled = local_evict_stalled_.load(std::memory_order_relaxed); + metrics.master_keys = local_evict_master_keys_.load(std::memory_order_relaxed); return metrics; } @@ -1149,45 +1152,26 @@ size_t PeerPool::ReclaimBackend(uint32_t backend_id, const LocalEvictionWatermar size_t evicted_keys = 0; // The medium keeps offering settled keys until they leave; look past them. std::unordered_set settled; - size_t window = kLocalEvictBatch; - while (covered < budget) { - { - std::lock_guard lock(local_evict_mutex_); - if (stop_local_evict_) break; - } - auto offers = backend->EvictionCandidates(window); - const bool exhausted = offers.size() < window; - offers.erase(std::remove_if(offers.begin(), offers.end(), - [&](const EvictionOffer& o) { return settled.count(o.key) != 0; }), - offers.end()); - if (offers.empty()) { - if (!exhausted) { - window *= 2; - continue; - } - if (settled.empty()) { - local_evict_no_candidate_.fetch_add(1, std::memory_order_relaxed); - MORI_UMBP_DEBUG( - "[PeerPool] local eviction: backend {} over its watermark but offers no " - "candidate (all leased or pinned)", - backend_id); - } - break; - } - // The coldest the budget still needs, at most one batch. + const auto stopping = [this] { + std::lock_guard lock(local_evict_mutex_); + return stop_local_evict_; + }; + const auto drained = [&] { return UsedBytes(backend->Capacity()) <= low_bytes; }; + + // Evicts the coldest of offers[*next...] the budget still needs, at most one + // batch. False if none of them freed or moved anything. + const auto evict_some = [&](const std::vector& offers, size_t* next) { std::vector requests; std::vector offered_bytes; uint64_t taken = 0; - for (auto& offer : offers) { + for (; *next < offers.size(); ++*next) { + const auto& offer = offers[*next]; if (requests.size() >= kLocalEvictBatch) break; if (!requests.empty() && covered + taken >= budget) break; + if (settled.count(offer.key) != 0) continue; taken += offer.bytes; offered_bytes.push_back(offer.bytes); - PoolEvictRequest request; - request.key = std::move(offer.key); - request.tier = tier; - request.backend_id = backend_id; - requests.push_back(std::move(request)); + requests.push_back(PoolEvictRequest{offer.key, tier, backend_id}); } const auto results = Evict(requests, PoolEvictMode::kReclaim); bool progressed = false; @@ -1199,24 +1183,93 @@ size_t PeerPool::ReclaimBackend(uint32_t backend_id, const LocalEvictionWatermar ++evicted_keys; [[fallthrough]]; case PoolEvictOutcome::kInFlight: - covered += offered_bytes[i]; + covered += offered_bytes[i]; // capacity units, as offered progressed = true; break; case PoolEvictOutcome::kNone: break; } } - if (!progressed) { + return progressed; + }; + + // The master's cluster-wide ranking first (at most one query per interval); + // the medium's own order covers whatever it leaves. + const auto ranked = RankedCandidates(backend_id, tier, budget); + if (ranked.has_value()) { + const size_t before = evicted_keys; + size_t next = 0; + while (covered < budget && next < ranked->size() && !stopping() && !drained()) { + evict_some(*ranked, &next); // stale entries just yield kNone + } + local_evict_master_keys_.fetch_add(evicted_keys - before, std::memory_order_relaxed); + } + + size_t window = kLocalEvictBatch; + while (covered < budget && !drained()) { + if (stopping()) break; + auto offers = backend->EvictionCandidates(window); + const bool exhausted = offers.size() < window; + offers.erase(std::remove_if(offers.begin(), offers.end(), + [&](const EvictionOffer& o) { return settled.count(o.key) != 0; }), + offers.end()); + if (offers.empty()) { + if (!exhausted) { + window *= 2; + continue; + } + if (settled.empty()) { + local_evict_no_candidate_.fetch_add(1, std::memory_order_relaxed); + MORI_UMBP_DEBUG( + "[PeerPool] local eviction: backend {} over its watermark but offers no " + "candidate (all leased or pinned)", + backend_id); + } + break; + } + size_t next = 0; + if (!evict_some(offers, &next)) { // All got leased or pinned since offered; retry on the next wake. local_evict_stalled_.fetch_add(1, std::memory_order_relaxed); break; } - if (UsedBytes(backend->Capacity()) <= low_bytes) break; } local_evict_keys_.fetch_add(evicted_keys, std::memory_order_relaxed); return evicted_keys; } +std::optional> PeerPool::RankedCandidates(uint32_t backend_id, + TierType tier, + uint64_t bytes) { + EvictionCandidateSource source; + { + std::lock_guard lock(local_evict_mutex_); + source = candidate_source_; + } + if (!source) return std::nullopt; + // Only the pass thread touches this (local_evict_pass_mutex_). A backend + // stuck over its watermark would otherwise query on every wake. + const auto now = std::chrono::steady_clock::now(); + auto& last = last_ranked_query_[backend_id]; + if (now - last < kRankedCandidatesMinInterval) return std::nullopt; + last = now; + const std::string logical = tier_graph_ == nullptr ? "" : tier_graph_->NameForBackend(backend_id); + auto offers = source(tier, logical, bytes); + // The source reports value sizes; the budget counts whole pages. + auto* backend = backends_->Get(backend_id); + const uint64_t page = backend == nullptr ? 0 : backend->PageSize(); + if (offers.has_value() && page > 0) { + for (auto& offer : *offers) + offer.bytes = std::max(1, (offer.bytes + page - 1) / page) * page; + } + return offers; +} + +void PeerPool::SetEvictionCandidateSource(EvictionCandidateSource source) { + std::lock_guard lock(local_evict_mutex_); + candidate_source_ = std::move(source); +} + void PeerPool::ClearLocal() { std::unique_lock lifecycle_lock(lifecycle_mutex_); std::unique_lock operation_lock(operation_mutex_); diff --git a/src/umbp/distributed/pool_client.cpp b/src/umbp/distributed/pool_client.cpp index 93c5d920e..bf9b3a70d 100644 --- a/src/umbp/distributed/pool_client.cpp +++ b/src/umbp/distributed/pool_client.cpp @@ -1092,24 +1092,41 @@ bool PoolClient::Init() { default_pool_ = std::make_unique(®istry_, std::move(placement_policy), transfer_engine_.get()); - // Local watermark eviction: every medium without a master, SSD only with - // one (as before). SSD uses its own watermarks, paged media the node pair. + // Local watermark eviction: SSD's own pair or the node pair; with + // peer_driven_eviction, a tiered policy's per-tier pair instead. SSD and + // every medium without a master start now; paged media under a master wait + // for it to confirm peer-driven eviction (below). + if (const char* env = std::getenv("UMBP_PEER_DRIVEN_EVICTION")) { + config_.peer_driven_eviction = std::string_view(env) != "0"; + } + struct LocalEvictTarget { + uint32_t backend_id; + double high; + double low; + }; + std::vector deferred_local_evict; for (const auto& backend_config : backend_configs) { auto* backend = registry_.Get(backend_config.name); if (backend == nullptr) continue; const uint32_t backend_id = registry_.BackendId(backend); - if (backend_config.tier == TierType::SSD) { - if (!default_pool_->EnableLocalEviction(backend_id, backend_config.ssd.ssd.high_watermark, - backend_config.ssd.ssd.low_watermark)) { - MORI_UMBP_ERROR("[PoolClient] backend '{}': invalid SSD watermarks high={} low={}", - backend_config.name, backend_config.ssd.ssd.high_watermark, - backend_config.ssd.ssd.low_watermark); - Shutdown(); - return false; + const bool ssd = backend_config.tier == TierType::SSD; + LocalEvictTarget target{ + backend_id, + ssd ? backend_config.ssd.ssd.high_watermark : config_.local_evict_high_watermark, + ssd ? backend_config.ssd.ssd.low_watermark : config_.local_evict_low_watermark}; + if (config_.peer_driven_eviction && tier_graph != nullptr) { + if (const auto index = tier_graph->TierIndexForBackendId(backend_id)) { + target.high = tier_graph->NodeAt(*index).high_watermark; + target.low = tier_graph->NodeAt(*index).low_watermark; } - } else if (!master_client_) { - default_pool_->EnableLocalEviction(backend_id, config_.local_evict_high_watermark, - config_.local_evict_low_watermark); + } + if (!ssd && master_client_) { + deferred_local_evict.push_back(target); + } else if (!default_pool_->EnableLocalEviction(backend_id, target.high, target.low) && ssd) { + MORI_UMBP_ERROR("[PoolClient] backend '{}': invalid SSD watermarks high={} low={}", + backend_config.name, target.high, target.low); + Shutdown(); + return false; } } @@ -1217,6 +1234,7 @@ bool PoolClient::Init() { const auto logical_caps = default_pool_ == nullptr ? std::map{} : default_pool_->LogicalTierCapacities(); + master_client_->RequestPeerDrivenEviction(config_.peer_driven_eviction); auto status = master_client_->RegisterSelf(tier_caps, peer_address, engine_desc_bytes, logical_caps); if (!status.ok()) { @@ -1224,6 +1242,19 @@ bool PoolClient::Init() { Shutdown(); return false; } + // Peer-driven: this node evicts every medium at its own watermarks, with + // victims ranked by the master, and the master stops sending EvictKey. + if (master_client_->PeerDrivenEviction() && default_pool_ != nullptr) { + for (const auto& target : deferred_local_evict) { + default_pool_->EnableLocalEviction(target.backend_id, target.high, target.low); + } + default_pool_->SetEvictionCandidateSource( + [client = master_client_.get()](TierType tier, const std::string& logical_tier, + uint64_t bytes) { + return client->GetEvictionCandidates(tier, logical_tier, bytes, /*max_keys=*/0); + }); + MORI_UMBP_INFO("[PoolClient] peer-driven eviction ON (victims ranked by the master)"); + } if (config_.master_config.auto_heartbeat) master_client_->StartHeartbeat(); if (weighted_placement && !master_client_->SupportsMaxAllocatableCapacity()) { MORI_UMBP_WARN( @@ -1298,6 +1329,7 @@ void PoolClient::Shutdown() { if (!status.ok()) { MORI_UMBP_WARN("[PoolClient] UnregisterSelf failed: {}", status.error_message()); } + if (default_pool_) default_pool_->SetEvictionCandidateSource(nullptr); } { @@ -1382,6 +1414,10 @@ TierTransitionMetrics PoolClient::TransitionMetrics() const { return default_pool_ == nullptr ? TierTransitionMetrics{} : default_pool_->TransitionMetrics(); } +LocalEvictionMetrics PoolClient::LocalEvictionStats() const { + return default_pool_ == nullptr ? LocalEvictionMetrics{} : default_pool_->LocalEvictionStats(); +} + // --------------------------------------------------------------------------- // Memory registration // --------------------------------------------------------------------------- @@ -5100,6 +5136,8 @@ void PoolClient::PublishComponentMetrics() { {{"event", "no_candidate"}}, local.no_candidate); sample(MORI_UMBP_METRIC_CLIENT_LOCAL_EVICT, MORI_UMBP_METRIC_CLIENT_LOCAL_EVICT_HELP, {{"event", "stalled"}}, local.stalled); + sample(MORI_UMBP_METRIC_CLIENT_LOCAL_EVICT, MORI_UMBP_METRIC_CLIENT_LOCAL_EVICT_HELP, + {{"event", "master_key"}}, local.master_keys); metric_publisher_.Publish("pool", {}, samples, sink); } } diff --git a/src/umbp/distributed/proto/umbp.proto b/src/umbp/distributed/proto/umbp.proto index aceb5d5ff..28c7d65a2 100644 --- a/src/umbp/distributed/proto/umbp.proto +++ b/src/umbp/distributed/proto/umbp.proto @@ -89,12 +89,16 @@ message RegisterClientRequest { bytes engine_desc = 5; // packed EngineDesc repeated string tags = 13; // opaque key=value labels (e.g. "sgl_role=prefill") repeated LogicalTierCapacity logical_tier_capacities = 14; + // The peer evicts at its own watermarks (GetEvictionCandidates); the master + // stops pushing EvictKey to it. Re-sent on every heartbeat. + bool peer_driven_eviction = 15; } message RegisterClientResponse { uint64 heartbeat_interval_ms = 1; // Recommended heartbeat interval uint64 ack_seq = 2; // Initial heartbeat seq peer should send (= 0) bool supports_max_allocatable_bytes = 3; bool supports_logical_tiers = 4; + bool supports_peer_driven_eviction = 5; } message UnregisterClientRequest { @@ -142,6 +146,22 @@ message HeartbeatRequest { uint64 delta_seq_baseline = 6; // valid for full-sync; master adopts this cursor repeated TierKvCount tier_kv_counts = 7; // live owned-key count per tier (client-reported) repeated LogicalTierCapacity logical_tier_capacities = 8; + bool peer_driven_eviction = 9; // as in RegisterClientRequest +} + +// A peer over its watermark asks for its own keys on one medium, coldest first +// by the master's cluster-wide recency, until their sizes cover `bytes`. +// Master-leased keys are skipped. +message GetEvictionCandidatesRequest { + string node_id = 1; + TierType tier = 2; + string logical_tier = 3; // empty: any logical tier on that medium + uint64 bytes = 4; + uint32 max_keys = 5; // 0: server cap +} +message GetEvictionCandidatesResponse { + repeated string keys = 1; + repeated uint64 sizes = 2; // parallel to keys } message HeartbeatResponse { ClientStatus status = 1; @@ -371,6 +391,10 @@ service UMBPMaster { rpc UnregisterClient(UnregisterClientRequest) returns (UnregisterClientResponse); rpc Heartbeat (HeartbeatRequest) returns (HeartbeatResponse); + // Peer-driven eviction: ranked victims for a peer over its watermark + rpc GetEvictionCandidates(GetEvictionCandidatesRequest) + returns (GetEvictionCandidatesResponse); + // Router (read-only) rpc RouteGet (RouteGetRequest) returns (RouteGetResponse); rpc RoutePut (RoutePutRequest) returns (RoutePutResponse); diff --git a/src/umbp/doc/runtime-env-vars.md b/src/umbp/doc/runtime-env-vars.md index 8e7a8ed65..93583b993 100644 --- a/src/umbp/doc/runtime-env-vars.md +++ b/src/umbp/doc/runtime-env-vars.md @@ -75,6 +75,7 @@ that has loaded `libmori_pybinds.so`). | Env var | Default | Unit | Description | |---|---|---|---| +| `UMBP_PEER_DRIVEN_EVICTION` | `0` | bool | `1`: with a master that supports it, evict every medium at this node's own watermarks (a tiered policy's per-tier pair, else the SSD / DRAM config pair), with victims ranked by the master (`GetEvictionCandidates`), and stop the master's `EvictKey` rounds for this node. `0`: master-driven eviction, as before. | | `UMBP_DRAM_READ_LEASE_MS` | `500` | ms | Peer-side DRAM/HBM read lease: how long a single `PeerDramAllocator::Resolve` protects its key's pages from concurrent local `Evict`, covering one RDMA read of those pages. Only needs to exceed one DRAM RDMA round trip (sub-ms), so 500 ms is ~100x margin. Read once at `PoolClient::Init`; `min_allowed=1`. | | `UMBP_SSD_READ_LEASE_MS` | `3000` | ms | Peer-side SSD read-staging slot lease: how long a claimed staging slot is reserved before the peer reclaims it by TTL (the fallback when the reader's best-effort `ReleaseSsdLease` is lost), and, echoed back in `PrepareSsdReadResponse.lease_ttl_ms`, the reader's validity window anchored at `t_send`. Must exceed one SSD read + RDMA (slower than DRAM), but too long pins a configured staging slot on a lost release. Also the fallback for the `PrepareSsdRead` RPC deadline when `UMBP_SSD_PREPARE_TIMEOUT_MS` is unset. Read once at `PoolClient::Init`; `min_allowed=1`. Also the read lease of a policy-declared `SsdBackend`, where it governs how long a contiguous staging span stays claimed and so sets that backend's read concurrency against a fixed arena — see `staging_slots`. | | `UMBP_RESOLVE_BUSY_TIMEOUT_MS` | `30000` | ms | Overall deadline for retrying a peer-local or remote batch resolve that returned `BUSY` because SSD staging was temporarily full. Retries discard the whole response and use exponential backoff capped at 50 ms. A batch whose own working set exceeds the arena returns a permanent failure immediately and is not retried. Values above 300000 ms are clamped. | diff --git a/src/umbp/include/umbp/distributed/config.h b/src/umbp/include/umbp/distributed/config.h index db229a883..15c87ea30 100644 --- a/src/umbp/include/umbp/distributed/config.h +++ b/src/umbp/include/umbp/distributed/config.h @@ -92,6 +92,9 @@ struct ClientRegistryConfig { // single backend can hold. Tests set this false to emulate that Master. bool advertise_max_allocatable_bytes = true; bool advertise_logical_tiers = true; + // Serve GetEvictionCandidates and stop EvictKey rounds for peers that evict + // for themselves. False emulates an older Master. + bool advertise_peer_driven_eviction = true; }; struct EvictionConfig { @@ -307,11 +310,18 @@ struct PoolClientConfig { // (2 MiB by default). Set to an explicit byte count to override. uint64_t dram_page_size = 0; - // Local eviction watermarks for DRAM/HBM, used only without a master (SSD - // has its own pair). Lowered from UMBPDramConfig. + // Local eviction watermarks for DRAM/HBM (SSD has its own pair), used + // without a master or in peer-driven mode; with peer_driven_eviction a + // tiered policy's per-tier pair takes precedence. From UMBPDramConfig. double local_evict_high_watermark = 0.9; double local_evict_low_watermark = 0.7; + // With a master: evict every medium at this node's own watermarks, taking + // victims from the master's ranking (GetEvictionCandidates), instead of + // waiting for EvictKey. Needs a master that supports it. Opt-in for now; + // env UMBP_PEER_DRIVEN_EVICTION overrides (0 / 1). + bool peer_driven_eviction = false; + UMBPCopyPipelineConfig copy_pipeline = [] { UMBPCopyPipelineConfig c; c.worker_threads = 1; diff --git a/src/umbp/include/umbp/distributed/master/eviction_manager.h b/src/umbp/include/umbp/distributed/master/eviction_manager.h index d03a14a02..a135d1b4f 100644 --- a/src/umbp/include/umbp/distributed/master/eviction_manager.h +++ b/src/umbp/include/umbp/distributed/master/eviction_manager.h @@ -22,11 +22,14 @@ #pragma once #include +#include #include +#include #include #include #include #include +#include #include #include "umbp/distributed/config.h" @@ -75,6 +78,15 @@ class EvictionManager { void Start(); void Stop(); + // A peer-driven node evicts at its own watermarks and asks for candidates + // (SelectPeerEvictionCandidates); rounds skip it. In memory only: peers + // re-assert it on every heartbeat. + void SetPeerDriven(const std::string& node_id, bool peer_driven); + bool PeerDriven(const std::string& node_id) const; + + // Test seam: one synchronous round. + void RunOnceForTest() { RunOnce(); } + private: void EvictionLoop(); @@ -91,6 +103,19 @@ class EvictionManager { std::atomic running_{false}; std::mutex cv_mutex_; std::condition_variable cv_; + mutable std::mutex peer_driven_mutex_; + std::unordered_set peer_driven_; }; +// Server cap on one GetEvictionCandidates reply. +constexpr size_t kMaxPeerEvictionCandidates = 4096; + +// `node_id`'s keys on `tier` (and `logical_tier`, if non-empty), least +// recently accessed first, until their sizes cover `bytes` or `max_keys` is +// reached (0: kMaxPeerEvictionCandidates). Master-leased keys are skipped. +std::vector SelectPeerEvictionCandidates( + const IMasterMetadataStore& store, const std::string& node_id, TierType tier, + const std::string& logical_tier, uint64_t bytes, size_t max_keys, + std::chrono::system_clock::time_point now); + } // namespace mori::umbp diff --git a/src/umbp/include/umbp/distributed/master/master_client.h b/src/umbp/include/umbp/distributed/master/master_client.h index 0b5eb3d7d..3de2379b2 100644 --- a/src/umbp/include/umbp/distributed/master/master_client.h +++ b/src/umbp/include/umbp/distributed/master/master_client.h @@ -122,6 +122,20 @@ class MasterClient { // wants to know "is the key resident?" and is not about to RDMA-read. grpc::Status BatchLookup(const std::vector& keys, std::vector* out); + // --- Peer-driven eviction --- + // Ask the master to stop EvictKey rounds for this node; set before + // RegisterSelf. In effect only once the master confirms support. + void RequestPeerDrivenEviction(bool enabled) { peer_driven_requested_ = enabled; } + bool PeerDrivenEviction() const { + return peer_driven_requested_ && supports_peer_driven_eviction_.load(); + } + // This node's keys on `tier`, coldest first by the master's recency, until + // their sizes cover `bytes`. nullopt when the RPC fails. + std::optional> GetEvictionCandidates(TierType tier, + const std::string& logical_tier, + uint64_t bytes, + uint32_t max_keys); + // --- Heartbeat (event-driven) --- // Bind every storage medium live on this peer. The heartbeat drains each // backend's outbox and concatenates them into ONE bundle under a single @@ -243,6 +257,9 @@ class MasterClient { bool aggregate_backend_capacities_ = false; bool supports_max_allocatable_capacity_ = false; bool supports_logical_tiers_ = false; + bool peer_driven_requested_ = false; + // Rewritten on re-registration (heartbeat thread), read by eviction. + std::atomic supports_peer_driven_eviction_{false}; // Initial conservative registration payload, retained so an UNKNOWN response // can rebuild the full peer record after Master state loss. std::map registration_capacities_; diff --git a/src/umbp/include/umbp/distributed/master/master_metrics.h b/src/umbp/include/umbp/distributed/master/master_metrics.h index 23322a83e..1660e6272 100644 --- a/src/umbp/include/umbp/distributed/master/master_metrics.h +++ b/src/umbp/include/umbp/distributed/master/master_metrics.h @@ -182,8 +182,8 @@ "Reads served by each logical tier of this client's pool" // --- Local watermark eviction ---------------------------------------------- -// event=round|key|no_candidate|stalled (LocalEvictionMetrics); bytes freed are -// in the backend evict series. +// event=round|key|no_candidate|stalled|master_key (LocalEvictionMetrics); bytes +// freed are in the backend evict series. #define MORI_UMBP_METRIC_CLIENT_LOCAL_EVICT "mori_umbp_client_local_evict_total" #define MORI_UMBP_METRIC_CLIENT_LOCAL_EVICT_HELP \ diff --git a/src/umbp/include/umbp/distributed/pool/peer_pool.h b/src/umbp/include/umbp/distributed/pool/peer_pool.h index 6adac00f8..e5facb6f3 100644 --- a/src/umbp/include/umbp/distributed/pool/peer_pool.h +++ b/src/umbp/include/umbp/distributed/pool/peer_pool.h @@ -29,6 +29,7 @@ #include #include #include +#include #include #include #include @@ -86,15 +87,6 @@ struct PoolEvictResult { PoolEvictOutcome outcome = PoolEvictOutcome::kNone; }; -// Local watermark eviction counters; bytes freed are counted by -// InstrumentedBackend. -struct LocalEvictionMetrics { - uint64_t rounds = 0; // passes that found a backend over its high watermark - uint64_t keys = 0; // keys dropped or queued for demotion - uint64_t no_candidate = 0; // passes where every key was leased or pinned - uint64_t stalled = 0; // batches that freed nothing; the next wake retries -}; - // Why a key is being removed. A tiered pool answers the two intents // differently, so the caller has to say which one it means instead of leaving // it to the tier configuration. @@ -164,6 +156,13 @@ class PeerPool { // Test seam: one synchronous pass. Returns keys dropped or queued. size_t RunLocalEvictionOnceForTest() { return LocalEvictionPass(); } + // Victims for a backend over its watermark, coldest first, before the + // medium's own order: (tier, logical tier, bytes) -> offers, or nullopt when + // unavailable. The master's GetEvictionCandidates in peer-driven mode. + using EvictionCandidateSource = std::function>( + TierType tier, const std::string& logical_tier, uint64_t bytes)>; + void SetEvictionCandidateSource(EvictionCandidateSource source); + private: struct PendingPlacement { uint32_t backend_id = BackendRegistry::kMaxBackends; @@ -203,6 +202,8 @@ class PeerPool { void StopLocalEviction(); size_t LocalEvictionPass(); // lowest tier first; returns keys evicted size_t ReclaimBackend(uint32_t backend_id, const LocalEvictionWatermarks& watermarks); + std::optional> RankedCandidates(uint32_t backend_id, TierType tier, + uint64_t bytes); void TouchLocked(const std::string& key); void ForgetAccessLocked(const std::string& key); @@ -307,6 +308,9 @@ class PeerPool { std::atomic local_evict_keys_{0}; std::atomic local_evict_no_candidate_{0}; std::atomic local_evict_stalled_{0}; + std::atomic local_evict_master_keys_{0}; + EvictionCandidateSource candidate_source_; // guarded by local_evict_mutex_ + std::map last_ranked_query_; // pass thread }; } // namespace mori::umbp diff --git a/src/umbp/include/umbp/distributed/pool_client.h b/src/umbp/include/umbp/distributed/pool_client.h index 12abc2037..14880566b 100644 --- a/src/umbp/include/umbp/distributed/pool_client.h +++ b/src/umbp/include/umbp/distributed/pool_client.h @@ -203,6 +203,7 @@ class PoolClient { // MediumBackend — no concrete backend type is named outside PoolClient::Init. BackendRegistry& Backends(); TierTransitionMetrics TransitionMetrics() const; + LocalEvictionMetrics LocalEvictionStats() const; std::map TierReadHits() const; std::string LogicalTierForBackend(uint32_t backend_id) const; diff --git a/src/umbp/include/umbp/distributed/types.h b/src/umbp/include/umbp/distributed/types.h index 163884786..bc3e95b44 100644 --- a/src/umbp/include/umbp/distributed/types.h +++ b/src/umbp/include/umbp/distributed/types.h @@ -84,6 +84,16 @@ struct TierTransitionMetrics { uint64_t promoted_bytes = 0; }; +// Local watermark eviction counters; bytes freed are counted by +// InstrumentedBackend. +struct LocalEvictionMetrics { + uint64_t rounds = 0; // passes that found a backend over its high watermark + uint64_t keys = 0; // keys dropped or queued for demotion + uint64_t no_candidate = 0; // passes where every key was leased or pinned + uint64_t stalled = 0; // batches that freed nothing; the next wake retries + uint64_t master_keys = 0; // of `keys`, those the candidate source ranked +}; + struct ExternalKvHitCountEntry { std::string hash; uint64_t hit_count_total = 0; diff --git a/tests/cpp/umbp/distributed/CMakeLists.txt b/tests/cpp/umbp/distributed/CMakeLists.txt index 452a2466d..2c6c76acf 100644 --- a/tests/cpp/umbp/distributed/CMakeLists.txt +++ b/tests/cpp/umbp/distributed/CMakeLists.txt @@ -277,6 +277,19 @@ target_link_libraries( gtest_main) add_test(NAME umbp_local_first COMMAND test_umbp_local_first) +# test_umbp_peer_driven_eviction — a real master and DRAM node: the node evicts +# at its own watermark with master-ranked victims, or not at all unless both +# sides opt in. +add_executable(test_umbp_peer_driven_eviction test_peer_driven_eviction.cpp) +if(CMAKE_CXX_COMPILER_ID MATCHES "GNU|Clang") + target_link_options(test_umbp_peer_driven_eviction PRIVATE -Wl,--no-as-needed) +endif() +target_link_libraries( + test_umbp_peer_driven_eviction + PRIVATE umbp_core umbp_common ${_TEST_PROTOBUF_LIB} ${_TEST_GRPCPP_LIB} + gtest_main) +add_test(NAME umbp_peer_driven_eviction COMMAND test_umbp_peer_driven_eviction) + # test_umbp_no_master — the distributed client with master_address empty: same # data plane, no routing/registration/heartbeat, a local miss is final. No # master is started in that file, so reaching for one hangs or crashes rather diff --git a/tests/cpp/umbp/distributed/test_master_eviction_chain.cpp b/tests/cpp/umbp/distributed/test_master_eviction_chain.cpp index dd5b74fe7..1f6f976d5 100644 --- a/tests/cpp/umbp/distributed/test_master_eviction_chain.cpp +++ b/tests/cpp/umbp/distributed/test_master_eviction_chain.cpp @@ -584,5 +584,41 @@ TEST(MasterEvictionBytes, ARoundFreesTheOverageUpToTheByteLimit) { EXPECT_EQ(FirstRoundVictims(limited), 10u); } +// Peer-driven eviction: the master ranks a node's keys on request and leaves +// the node out of its own rounds. +TEST(MasterPeerDrivenEviction, RanksANodesKeysAndSkipsItInRounds) { + InMemoryMasterMetadataStore store; + const auto t0 = Clock::now(); + RegisterNode(&store, t0); + std::map full; + full[TierType::DRAM] = TierCapacity{6 * kPageSize, 0, kPageSize}; + // k5 is the oldest; k0..k3 are "hot", k4/k5 "warm". + for (int i = 0; i < 6; ++i) { + ReportHeartbeat(&store, /*seq=*/i + 1, t0 + std::chrono::seconds(6 - i), full, TierType::DRAM, + i < 4 ? "hot" : "warm", {"k" + std::to_string(i)}, /*full_sync=*/i == 0); + } + const auto now = t0 + std::chrono::seconds(10); + const auto keys = [&](const std::string& logical, uint64_t pages, size_t max_keys) { + std::vector out; + for (const auto& row : SelectPeerEvictionCandidates(store, kNodeId, TierType::DRAM, logical, + pages * kPageSize, max_keys, now)) { + out.push_back(row.key); + } + return out; + }; + EXPECT_EQ(keys("", 3, 0), (std::vector{"k5", "k4", "k3"})); + EXPECT_EQ(keys("hot", 2, 0), (std::vector{"k3", "k2"})); + EXPECT_EQ(keys("", 100, 2), (std::vector{"k5", "k4"})); + + RecordingDispatcher dispatcher; + EvictionManager manager(store, OneSecondRounds(), &dispatcher); + manager.SetPeerDriven(kNodeId, true); + manager.RunOnceForTest(); + EXPECT_TRUE(dispatcher.Calls().empty()) << "a peer-driven node still got EvictKey"; + manager.SetPeerDriven(kNodeId, false); + manager.RunOnceForTest(); + EXPECT_FALSE(dispatcher.Calls().empty()); +} + } // namespace } // namespace mori::umbp diff --git a/tests/cpp/umbp/distributed/test_peer_driven_eviction.cpp b/tests/cpp/umbp/distributed/test_peer_driven_eviction.cpp new file mode 100644 index 000000000..563169adc --- /dev/null +++ b/tests/cpp/umbp/distributed/test_peer_driven_eviction.cpp @@ -0,0 +1,160 @@ +// Copyright © Advanced Micro Devices, Inc. All rights reserved. +// +// MIT License +// +// Permission is hereby granted, free of charge, to any person obtaining a copy +// of this software and associated documentation files (the "Software"), to deal +// in the Software without restriction, including without limitation the rights +// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +// copies of the Software, and to permit persons to whom the Software is +// furnished to do so, subject to the following conditions: +// +// The above copyright notice and this permission notice shall be included in all +// copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +// SOFTWARE. + +// Peer-driven eviction end to end: a real master and a DRAM PoolClient over +// gRPC. The master's own rounds run every 5 s, so anything evicted well inside +// that window was evicted by the node itself. + +#include + +#include +#include +#include +#include +#include +#include +#include +#include + +#include "umbp/distributed/config.h" +#include "umbp/distributed/master/master_server.h" +#include "umbp/distributed/pool_client.h" + +namespace mori::umbp { +namespace { + +constexpr size_t kPage = 4096; +constexpr size_t kPages = 16; + +std::string Key(int i) { return "pde-" + std::to_string(i); } + +bool WaitFor(const std::function& done, std::chrono::milliseconds budget) { + const auto deadline = std::chrono::steady_clock::now() + budget; + while (std::chrono::steady_clock::now() < deadline) { + if (done()) return true; + std::this_thread::sleep_for(std::chrono::milliseconds(10)); + } + return done(); +} + +class PeerDrivenEvictionTest : public ::testing::Test { + protected: + void TearDown() override { Stop(); } + + void Stop() { + if (client_) client_->Shutdown(); + client_.reset(); + if (master_) master_->Shutdown(); + if (server_thread_.joinable()) server_thread_.join(); + master_.reset(); + } + + // A master (advertising peer-driven eviction or not) and one 16-page DRAM + // node with 0.9 / 0.5 watermarks. + PoolClient* Start(bool master_supports, bool node_requests) { + MasterServerConfig master_cfg; + master_cfg.listen_address = "0.0.0.0:0"; + master_cfg.registry_config.advertise_peer_driven_eviction = master_supports; + master_ = std::make_unique(std::move(master_cfg)); + server_thread_ = std::thread([this] { master_->Run(); }); + if (!WaitFor([&] { return master_->GetBoundPort() != 0; }, std::chrono::seconds(5))) { + return nullptr; + } + + PoolClientConfig cfg; + cfg.master_config.node_id = "peer-driven"; + cfg.master_config.node_address = "127.0.0.1"; + cfg.master_config.master_address = "localhost:" + std::to_string(master_->GetBoundPort()); + cfg.io_engine.host = "0.0.0.0"; + cfg.io_engine.port = 0; + cfg.auto_peer_service_port = true; + cfg.dram_page_size = kPage; + cfg.medium = TierType::DRAM; + cfg.dram.buffer_sizes = {kPages * kPage}; + cfg.cache_remote_fetches = false; + cfg.local_evict_high_watermark = 0.9; + cfg.local_evict_low_watermark = 0.5; + cfg.peer_driven_eviction = node_requests; + client_ = std::make_unique(std::move(cfg)); + if (!client_->Init()) client_.reset(); + return client_.get(); + } + + void PutKeys(int first, int count) { + const std::vector value(kPage, 'x'); + for (int i = first; i < first + count; ++i) { + ASSERT_TRUE(client_->Put(Key(i), value.data(), value.size())) << i; + } + } + + MediumBackend* Dram() { return client_->Backends().Get(TierType::DRAM); } + + std::unique_ptr master_; + std::thread server_thread_; + std::unique_ptr client_; +}; + +TEST_F(PeerDrivenEvictionTest, NodeEvictsAtItsOwnWatermarkWithMasterRankedVictims) { + auto* node = Start(/*master_supports=*/true, /*node_requests=*/true); + ASSERT_NE(node, nullptr); + ASSERT_TRUE(node->Master().PeerDrivenEviction()); + + // Half full, and known to the master, before crossing the watermark. + PutKeys(0, 8); + std::vector known; + for (int i = 0; i < 8; ++i) known.push_back(Key(i)); + node->Master().FlushHeartbeat(); + ASSERT_TRUE(WaitFor( + [&] { + std::vector found; + return node->Master().BatchLookup(known, &found).ok() && + std::all_of(found.begin(), found.end(), [](bool f) { return f; }); + }, + std::chrono::seconds(5))); + + PutKeys(8, 7); // 15 of 16 pages + ASSERT_TRUE( + WaitFor([&] { return Dram()->OwnedKeyCount() <= kPages / 2; }, std::chrono::seconds(3))) + << "the node did not evict for itself"; + EXPECT_EQ(Dram()->OwnedKeyCount(), kPages / 2); + EXPECT_EQ(node->LocalEvictionStats().master_keys, 7u) << "victims did not come from the master"; + for (int i = 8; i < 15; ++i) EXPECT_TRUE(Dram()->Contains(Key(i))) << i; +} + +TEST_F(PeerDrivenEvictionTest, DramWaitsForTheMasterWithoutBothSidesOptingIn) { + const std::vector> cases = {{false, true}, {true, false}}; + for (const auto& [master_supports, node_requests] : cases) { + SCOPED_TRACE(::testing::Message() + << "master_supports=" << master_supports << " node_requests=" << node_requests); + auto* node = Start(master_supports, node_requests); + ASSERT_NE(node, nullptr); + EXPECT_FALSE(node->Master().PeerDrivenEviction()); + PutKeys(0, 15); + std::this_thread::sleep_for(std::chrono::milliseconds(1500)); + EXPECT_EQ(Dram()->OwnedKeyCount(), 15u); + EXPECT_EQ(node->LocalEvictionStats().rounds, 0u); + Stop(); + } +} + +} // namespace +} // namespace mori::umbp diff --git a/tests/cpp/umbp/distributed/test_peer_pool.cpp b/tests/cpp/umbp/distributed/test_peer_pool.cpp index a8f63883a..51a8e694f 100644 --- a/tests/cpp/umbp/distributed/test_peer_pool.cpp +++ b/tests/cpp/umbp/distributed/test_peer_pool.cpp @@ -24,6 +24,7 @@ // MIT License #include +#include #include #include #include @@ -1167,6 +1168,37 @@ TEST(PeerPoolLocalEviction, DrainsAFullBackendColdestFirstToItsLowWatermark) { EXPECT_EQ(pool.LocalEvictionStats().keys, 4u); } +TEST(PeerPoolLocalEviction, RankedCandidatesGoFirstAndTheMediumCoversTheRest) { + // The source (the master, in peer-driven mode) ranks differently from the + // medium's LRU, names a key this backend no longer holds, and reports value + // sizes rather than the whole pages the budget counts. + LocalCopyEngine engine; + BackendRegistry registry; + RegisterPageBackend(®istry, &engine, "dram", TierType::DRAM, kHostPage, 10, + std::chrono::milliseconds(1)); + PeerPool pool(®istry, MakeSingleBackendPolicy(), &engine); + std::atomic queries{0}; + std::atomic asked_bytes{0}; + pool.SetEvictionCandidateSource([&](TierType tier, const std::string&, uint64_t bytes) { + ++queries; + asked_bytes = tier == TierType::DRAM ? bytes : 0; + return std::optional>({{"k8", 64}, {"gone", 64}, {"k7", 64}}); + }); + ASSERT_TRUE(pool.EnableLocalEviction(registry.BackendId(registry.Get("dram")), 0.9, 0.5)); + + for (int i = 0; i < 9; ++i) CommitKey(&pool, "k" + std::to_string(i)); + auto* dram = registry.Get("dram"); + ASSERT_TRUE(WaitUntil([&] { return dram->OwnedKeyCount() <= 5; })); + std::this_thread::sleep_for(std::chrono::milliseconds(20)); + EXPECT_EQ(dram->OwnedKeyCount(), 5u) << "evicted past the overage"; + EXPECT_EQ(queries.load(), 1); + EXPECT_EQ(asked_bytes.load(), 4 * kHostPage); // 9 pages down to 5 + // Ranked k8, k7 first; then the medium's coldest, k0 and k1. + for (const char* gone : {"k8", "k7", "k0", "k1"}) EXPECT_FALSE(dram->Contains(gone)) << gone; + for (int i = 2; i <= 6; ++i) EXPECT_TRUE(dram->Contains("k" + std::to_string(i))) << i; + EXPECT_EQ(pool.LocalEvictionStats().master_keys, 2u); +} + TEST(PeerPoolLocalEviction, KeysUnderAReadLeaseAreNeverOffered) { LocalCopyEngine engine; BackendRegistry registry; From 0af37b45fc0d1ad8cefda3db3871603d40653712 Mon Sep 17 00:00:00 2001 From: TianDi101 Date: Tue, 13 Oct 2026 14:22:18 +0000 Subject: [PATCH 11/19] feat(umbp): eviction modes; nodes evict locally by default Replaces the peer_driven_eviction switch with PoolClientConfig::eviction_mode (env UMBP_EVICTION_MODE), which says how a node with a master evicts: - local (default): every medium at the node's own watermarks, by its own LRU, without asking the master. - master_ranked: the same, with victims ranked by the master (GetEvictionCandidates), falling back to local LRU. - master: DRAM/HBM wait for the master's EvictKey rounds; SSD still evicts locally (the previous behaviour). A self-evicting node (local or master_ranked) asks the master to leave it out of its rounds; every mode still serves EvictKey, so the master can evict on demand. A node without a master evicts locally as before. Self-evicting nodes use a tiered policy's per-tier watermarks when there is one. Tests that fill tiny tiers on purpose (cross-node smoke, tier benchmark) now run in master mode, the behaviour they were written against. Co-Authored-By: Claude Opus 5.5 --- src/umbp/distributed/pool_client.cpp | 67 ++++++++++--------- src/umbp/doc/runtime-env-vars.md | 2 +- src/umbp/include/umbp/distributed/config.h | 21 +++--- tests/cpp/umbp/distributed/CMakeLists.txt | 5 +- .../distributed/test_cross_node_smoke.cpp | 7 ++ .../distributed/test_peer_driven_eviction.cpp | 49 +++++++++----- .../distributed/test_umbp_tier_benchmark.cpp | 2 + 7 files changed, 90 insertions(+), 63 deletions(-) diff --git a/src/umbp/distributed/pool_client.cpp b/src/umbp/distributed/pool_client.cpp index bf9b3a70d..07bd6f444 100644 --- a/src/umbp/distributed/pool_client.cpp +++ b/src/umbp/distributed/pool_client.cpp @@ -1092,39 +1092,37 @@ bool PoolClient::Init() { default_pool_ = std::make_unique(®istry_, std::move(placement_policy), transfer_engine_.get()); - // Local watermark eviction: SSD's own pair or the node pair; with - // peer_driven_eviction, a tiered policy's per-tier pair instead. SSD and - // every medium without a master start now; paged media under a master wait - // for it to confirm peer-driven eviction (below). - if (const char* env = std::getenv("UMBP_PEER_DRIVEN_EVICTION")) { - config_.peer_driven_eviction = std::string_view(env) != "0"; - } - struct LocalEvictTarget { - uint32_t backend_id; - double high; - double low; - }; - std::vector deferred_local_evict; + // Local watermark eviction, for every medium unless a master's rounds own + // DRAM/HBM (eviction_mode kMaster; SSD always evicts locally). A node that + // evicts for itself uses a tiered policy's per-tier watermarks when there + // is one; otherwise SSD's own pair, or the node pair for paged media. + { + static constexpr const char* kModes[] = {"local", "master_ranked", "master"}; + const std::string mode = + GetEnvEnum("UMBP_EVICTION_MODE", kModes[static_cast(config_.eviction_mode)], + {kModes[0], kModes[1], kModes[2]}); + config_.eviction_mode = mode == "master" ? PeerEvictionMode::kMaster + : mode == "master_ranked" ? PeerEvictionMode::kMasterRanked + : PeerEvictionMode::kLocal; + } + const bool self_evicting = !master_client_ || config_.eviction_mode != PeerEvictionMode::kMaster; for (const auto& backend_config : backend_configs) { auto* backend = registry_.Get(backend_config.name); if (backend == nullptr) continue; - const uint32_t backend_id = registry_.BackendId(backend); const bool ssd = backend_config.tier == TierType::SSD; - LocalEvictTarget target{ - backend_id, - ssd ? backend_config.ssd.ssd.high_watermark : config_.local_evict_high_watermark, - ssd ? backend_config.ssd.ssd.low_watermark : config_.local_evict_low_watermark}; - if (config_.peer_driven_eviction && tier_graph != nullptr) { + if (!ssd && !self_evicting) continue; + const uint32_t backend_id = registry_.BackendId(backend); + double high = ssd ? backend_config.ssd.ssd.high_watermark : config_.local_evict_high_watermark; + double low = ssd ? backend_config.ssd.ssd.low_watermark : config_.local_evict_low_watermark; + if (self_evicting && tier_graph != nullptr) { if (const auto index = tier_graph->TierIndexForBackendId(backend_id)) { - target.high = tier_graph->NodeAt(*index).high_watermark; - target.low = tier_graph->NodeAt(*index).low_watermark; + high = tier_graph->NodeAt(*index).high_watermark; + low = tier_graph->NodeAt(*index).low_watermark; } } - if (!ssd && master_client_) { - deferred_local_evict.push_back(target); - } else if (!default_pool_->EnableLocalEviction(backend_id, target.high, target.low) && ssd) { + if (!default_pool_->EnableLocalEviction(backend_id, high, low) && ssd) { MORI_UMBP_ERROR("[PoolClient] backend '{}': invalid SSD watermarks high={} low={}", - backend_config.name, target.high, target.low); + backend_config.name, high, low); Shutdown(); return false; } @@ -1234,7 +1232,8 @@ bool PoolClient::Init() { const auto logical_caps = default_pool_ == nullptr ? std::map{} : default_pool_->LogicalTierCapacities(); - master_client_->RequestPeerDrivenEviction(config_.peer_driven_eviction); + // A self-evicting node asks the master to leave it out of its rounds. + master_client_->RequestPeerDrivenEviction(self_evicting); auto status = master_client_->RegisterSelf(tier_caps, peer_address, engine_desc_bytes, logical_caps); if (!status.ok()) { @@ -1242,19 +1241,21 @@ bool PoolClient::Init() { Shutdown(); return false; } - // Peer-driven: this node evicts every medium at its own watermarks, with - // victims ranked by the master, and the master stops sending EvictKey. - if (master_client_->PeerDrivenEviction() && default_pool_ != nullptr) { - for (const auto& target : deferred_local_evict) { - default_pool_->EnableLocalEviction(target.backend_id, target.high, target.low); - } + if (self_evicting && !master_client_->PeerDrivenEviction()) { + MORI_UMBP_WARN( + "[PoolClient] master predates peer-driven eviction: it may also evict this node in its " + "rounds, and master_ranked falls back to local LRU"); + } else if (config_.eviction_mode == PeerEvictionMode::kMasterRanked && + default_pool_ != nullptr) { default_pool_->SetEvictionCandidateSource( [client = master_client_.get()](TierType tier, const std::string& logical_tier, uint64_t bytes) { return client->GetEvictionCandidates(tier, logical_tier, bytes, /*max_keys=*/0); }); - MORI_UMBP_INFO("[PoolClient] peer-driven eviction ON (victims ranked by the master)"); } + static constexpr const char* kModeNames[] = {"local", "master_ranked", "master"}; + MORI_UMBP_INFO("[PoolClient] eviction mode: {}", + kModeNames[static_cast(config_.eviction_mode)]); if (config_.master_config.auto_heartbeat) master_client_->StartHeartbeat(); if (weighted_placement && !master_client_->SupportsMaxAllocatableCapacity()) { MORI_UMBP_WARN( diff --git a/src/umbp/doc/runtime-env-vars.md b/src/umbp/doc/runtime-env-vars.md index 93583b993..ebc19e2b1 100644 --- a/src/umbp/doc/runtime-env-vars.md +++ b/src/umbp/doc/runtime-env-vars.md @@ -75,7 +75,7 @@ that has loaded `libmori_pybinds.so`). | Env var | Default | Unit | Description | |---|---|---|---| -| `UMBP_PEER_DRIVEN_EVICTION` | `0` | bool | `1`: with a master that supports it, evict every medium at this node's own watermarks (a tiered policy's per-tier pair, else the SSD / DRAM config pair), with victims ranked by the master (`GetEvictionCandidates`), and stop the master's `EvictKey` rounds for this node. `0`: master-driven eviction, as before. | +| `UMBP_EVICTION_MODE` | `local` | enum | How a node with a master evicts. `local`: every medium at this node's own watermarks (a tiered policy's per-tier pair, else the SSD / DRAM config pair), by its own LRU, without asking the master; the master leaves it out of its `EvictKey` rounds. `master_ranked`: the same, with victims ranked by the master (`GetEvictionCandidates`), falling back to local LRU. `master`: DRAM/HBM wait for the master's rounds (SSD still evicts locally). A node without a master always evicts locally, and every mode still serves `EvictKey`. | | `UMBP_DRAM_READ_LEASE_MS` | `500` | ms | Peer-side DRAM/HBM read lease: how long a single `PeerDramAllocator::Resolve` protects its key's pages from concurrent local `Evict`, covering one RDMA read of those pages. Only needs to exceed one DRAM RDMA round trip (sub-ms), so 500 ms is ~100x margin. Read once at `PoolClient::Init`; `min_allowed=1`. | | `UMBP_SSD_READ_LEASE_MS` | `3000` | ms | Peer-side SSD read-staging slot lease: how long a claimed staging slot is reserved before the peer reclaims it by TTL (the fallback when the reader's best-effort `ReleaseSsdLease` is lost), and, echoed back in `PrepareSsdReadResponse.lease_ttl_ms`, the reader's validity window anchored at `t_send`. Must exceed one SSD read + RDMA (slower than DRAM), but too long pins a configured staging slot on a lost release. Also the fallback for the `PrepareSsdRead` RPC deadline when `UMBP_SSD_PREPARE_TIMEOUT_MS` is unset. Read once at `PoolClient::Init`; `min_allowed=1`. Also the read lease of a policy-declared `SsdBackend`, where it governs how long a contiguous staging span stays claimed and so sets that backend's read concurrency against a fixed arena — see `staging_slots`. | | `UMBP_RESOLVE_BUSY_TIMEOUT_MS` | `30000` | ms | Overall deadline for retrying a peer-local or remote batch resolve that returned `BUSY` because SSD staging was temporarily full. Retries discard the whole response and use exponential backoff capped at 50 ms. A batch whose own working set exceeds the arena returns a permanent failure immediately and is not retried. Values above 300000 ms are clamped. | diff --git a/src/umbp/include/umbp/distributed/config.h b/src/umbp/include/umbp/distributed/config.h index 15c87ea30..da394954c 100644 --- a/src/umbp/include/umbp/distributed/config.h +++ b/src/umbp/include/umbp/distributed/config.h @@ -233,6 +233,12 @@ struct BackendInstanceConfig { uint32_t placement_weight = 1; }; +enum class PeerEvictionMode { + kLocal, // at its own watermarks, by its own LRU; never asks the master + kMasterRanked, // at its own watermarks, victims ranked by the master + kMaster, // the master's EvictKey rounds (SSD still evicts locally) +}; + struct PoolClientConfig { UMBPMasterClientConfig master_config; UMBPIoEngineConfig io_engine; @@ -310,17 +316,16 @@ struct PoolClientConfig { // (2 MiB by default). Set to an explicit byte count to override. uint64_t dram_page_size = 0; - // Local eviction watermarks for DRAM/HBM (SSD has its own pair), used - // without a master or in peer-driven mode; with peer_driven_eviction a - // tiered policy's per-tier pair takes precedence. From UMBPDramConfig. + // Local eviction watermarks for DRAM/HBM (SSD has its own pair). Unless + // eviction_mode is kMaster, a tiered policy's per-tier pair takes + // precedence. From UMBPDramConfig. double local_evict_high_watermark = 0.9; double local_evict_low_watermark = 0.7; - // With a master: evict every medium at this node's own watermarks, taking - // victims from the master's ranking (GetEvictionCandidates), instead of - // waiting for EvictKey. Needs a master that supports it. Opt-in for now; - // env UMBP_PEER_DRIVEN_EVICTION overrides (0 / 1). - bool peer_driven_eviction = false; + // How this node evicts when it has a master (env UMBP_EVICTION_MODE = + // local | master_ranked | master). Without a master it always evicts + // locally. In every mode it still serves the master's EvictKey. + PeerEvictionMode eviction_mode = PeerEvictionMode::kLocal; UMBPCopyPipelineConfig copy_pipeline = [] { UMBPCopyPipelineConfig c; diff --git a/tests/cpp/umbp/distributed/CMakeLists.txt b/tests/cpp/umbp/distributed/CMakeLists.txt index 2c6c76acf..9f5e35c3f 100644 --- a/tests/cpp/umbp/distributed/CMakeLists.txt +++ b/tests/cpp/umbp/distributed/CMakeLists.txt @@ -277,9 +277,8 @@ target_link_libraries( gtest_main) add_test(NAME umbp_local_first COMMAND test_umbp_local_first) -# test_umbp_peer_driven_eviction — a real master and DRAM node: the node evicts -# at its own watermark with master-ranked victims, or not at all unless both -# sides opt in. +# test_umbp_peer_driven_eviction — a real master and DRAM node in each eviction +# mode: local (default), master_ranked, and master (the master's rounds). add_executable(test_umbp_peer_driven_eviction test_peer_driven_eviction.cpp) if(CMAKE_CXX_COMPILER_ID MATCHES "GNU|Clang") target_link_options(test_umbp_peer_driven_eviction PRIVATE -Wl,--no-as-needed) diff --git a/tests/cpp/umbp/distributed/test_cross_node_smoke.cpp b/tests/cpp/umbp/distributed/test_cross_node_smoke.cpp index 253ce9364..645cf732f 100644 --- a/tests/cpp/umbp/distributed/test_cross_node_smoke.cpp +++ b/tests/cpp/umbp/distributed/test_cross_node_smoke.cpp @@ -98,6 +98,8 @@ class CrossNodeSmoke : public ::testing::Test { cfg_a.auto_peer_service_port = true; cfg_a.dram.buffer_sizes = {kBlockSize}; cfg_a.dram_page_size = kBlockSize; + // These tiers are filled on purpose; leave eviction to the master's rounds. + cfg_a.eviction_mode = PeerEvictionMode::kMaster; client_a_ = std::make_unique(std::move(cfg_a)); ASSERT_TRUE(client_a_->Init()); @@ -110,6 +112,7 @@ class CrossNodeSmoke : public ::testing::Test { cfg_b.auto_peer_service_port = true; cfg_b.dram.buffer_sizes = {kBufSize}; cfg_b.dram_page_size = kBlockSize; + cfg_b.eviction_mode = PeerEvictionMode::kMaster; client_b_ = std::make_unique(std::move(cfg_b)); ASSERT_TRUE(client_b_->Init()); @@ -242,6 +245,8 @@ class CrossNodeMultiPage : public ::testing::Test { cfg.auto_peer_service_port = true; cfg.dram_page_size = kPageSize; cfg.dram.buffer_sizes = setup.buffer_sizes; + // These tiers are filled on purpose; leave eviction to the master's rounds. + cfg.eviction_mode = PeerEvictionMode::kMaster; auto cli = std::make_unique(std::move(cfg)); EXPECT_TRUE(cli->Init()); return cli; @@ -542,6 +547,8 @@ class CrossNodeOverlap : public ::testing::Test { cfg.dram_page_size = kPageSize; if (staging_buffer_size > 0) cfg.staging_buffer_size = staging_buffer_size; cfg.dram.buffer_sizes = buffer_sizes; + // These tiers are filled on purpose; leave eviction to the master's rounds. + cfg.eviction_mode = PeerEvictionMode::kMaster; auto cli = std::make_unique(std::move(cfg)); EXPECT_TRUE(cli->Init()); clients_.push_back(std::move(cli)); diff --git a/tests/cpp/umbp/distributed/test_peer_driven_eviction.cpp b/tests/cpp/umbp/distributed/test_peer_driven_eviction.cpp index 563169adc..9a70c7568 100644 --- a/tests/cpp/umbp/distributed/test_peer_driven_eviction.cpp +++ b/tests/cpp/umbp/distributed/test_peer_driven_eviction.cpp @@ -20,9 +20,9 @@ // OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE // SOFTWARE. -// Peer-driven eviction end to end: a real master and a DRAM PoolClient over -// gRPC. The master's own rounds run every 5 s, so anything evicted well inside -// that window was evicted by the node itself. +// Eviction modes end to end: a real master and a DRAM PoolClient over gRPC. +// The master's own rounds run every 5 s, so anything evicted well inside that +// window was evicted by the node itself. #include @@ -70,7 +70,7 @@ class PeerDrivenEvictionTest : public ::testing::Test { // A master (advertising peer-driven eviction or not) and one 16-page DRAM // node with 0.9 / 0.5 watermarks. - PoolClient* Start(bool master_supports, bool node_requests) { + PoolClient* Start(bool master_supports, PeerEvictionMode mode) { MasterServerConfig master_cfg; master_cfg.listen_address = "0.0.0.0:0"; master_cfg.registry_config.advertise_peer_driven_eviction = master_supports; @@ -93,7 +93,7 @@ class PeerDrivenEvictionTest : public ::testing::Test { cfg.cache_remote_fetches = false; cfg.local_evict_high_watermark = 0.9; cfg.local_evict_low_watermark = 0.5; - cfg.peer_driven_eviction = node_requests; + cfg.eviction_mode = mode; client_ = std::make_unique(std::move(cfg)); if (!client_->Init()) client_.reset(); return client_.get(); @@ -113,8 +113,8 @@ class PeerDrivenEvictionTest : public ::testing::Test { std::unique_ptr client_; }; -TEST_F(PeerDrivenEvictionTest, NodeEvictsAtItsOwnWatermarkWithMasterRankedVictims) { - auto* node = Start(/*master_supports=*/true, /*node_requests=*/true); +TEST_F(PeerDrivenEvictionTest, MasterRankedNodeEvictsTheMastersVictims) { + auto* node = Start(/*master_supports=*/true, PeerEvictionMode::kMasterRanked); ASSERT_NE(node, nullptr); ASSERT_TRUE(node->Master().PeerDrivenEviction()); @@ -140,18 +140,31 @@ TEST_F(PeerDrivenEvictionTest, NodeEvictsAtItsOwnWatermarkWithMasterRankedVictim for (int i = 8; i < 15; ++i) EXPECT_TRUE(Dram()->Contains(Key(i))) << i; } -TEST_F(PeerDrivenEvictionTest, DramWaitsForTheMasterWithoutBothSidesOptingIn) { - const std::vector> cases = {{false, true}, {true, false}}; - for (const auto& [master_supports, node_requests] : cases) { - SCOPED_TRACE(::testing::Message() - << "master_supports=" << master_supports << " node_requests=" << node_requests); - auto* node = Start(master_supports, node_requests); +TEST_F(PeerDrivenEvictionTest, EvictsLocallyUnlessTheModeIsMaster) { + struct Case { + bool master_supports; + PeerEvictionMode mode; + bool self_evicts; + }; + // The default; master_ranked against an older master (falls back to local + // LRU); and master mode, which leaves DRAM to the master's rounds. + const std::vector cases = {{true, PeerEvictionMode::kLocal, true}, + {false, PeerEvictionMode::kMasterRanked, true}, + {true, PeerEvictionMode::kMaster, false}}; + for (const auto& c : cases) { + SCOPED_TRACE(::testing::Message() << "master_supports=" << c.master_supports + << " mode=" << static_cast(c.mode)); + auto* node = Start(c.master_supports, c.mode); ASSERT_NE(node, nullptr); - EXPECT_FALSE(node->Master().PeerDrivenEviction()); - PutKeys(0, 15); - std::this_thread::sleep_for(std::chrono::milliseconds(1500)); - EXPECT_EQ(Dram()->OwnedKeyCount(), 15u); - EXPECT_EQ(node->LocalEvictionStats().rounds, 0u); + PutKeys(0, 15); // 15 of 16 pages + if (c.self_evicts) { + EXPECT_TRUE( + WaitFor([&] { return Dram()->OwnedKeyCount() <= kPages / 2; }, std::chrono::seconds(3))); + } else { + std::this_thread::sleep_for(std::chrono::milliseconds(1500)); + EXPECT_EQ(Dram()->OwnedKeyCount(), 15u); + } + EXPECT_EQ(node->LocalEvictionStats().master_keys, 0u) << "asked the master"; Stop(); } } diff --git a/tests/cpp/umbp/distributed/test_umbp_tier_benchmark.cpp b/tests/cpp/umbp/distributed/test_umbp_tier_benchmark.cpp index 5f1794219..c122b6647 100644 --- a/tests/cpp/umbp/distributed/test_umbp_tier_benchmark.cpp +++ b/tests/cpp/umbp/distributed/test_umbp_tier_benchmark.cpp @@ -199,6 +199,8 @@ class InProcessClusterTest : public ::testing::Test { config.cache_remote_fetches = false; config.placement_policy = PoolPlacementPolicy::WEIGHTED; config.backends = backends; + // Cases fill the pool on purpose; leave eviction to the master's rounds. + config.eviction_mode = PeerEvictionMode::kMaster; auto client = std::make_unique(std::move(config)); ASSERT_TRUE(client->Init()) << "failed to initialize PoolClient " << node_id; From cafbbed3a28fee6232a6be203a72d2d56e4ab901 Mon Sep 17 00:00:00 2001 From: TianDi101 Date: Mon, 12 Oct 2026 06:18:01 +0000 Subject: [PATCH 12/19] fix(umbp): retry an eviction demotion that failed on the way down The transition worker dropped an evicted copy on ANY failed demotion. That is right when nothing downstream has room -- the tier must still free -- but a copy, commit or read-back failure on a target that had room is a fault on the way, and the evicted copy is the only one: dropping it turned a burst of 3,934 transient SSD write failures into 3,934 lost keys in the e2e run. TierTransitionResult::retryable marks those failures; the worker requeues such an eviction demotion (up to 3 more attempts, at the back of the queue, so it runs after the backlog) and drops the copy only after that, or when no target had room. Co-Authored-By: Claude Opus 5.5 --- src/umbp/distributed/pool/peer_pool.cpp | 18 ++++- src/umbp/distributed/pool/tier_transition.cpp | 4 + .../umbp/distributed/pool/tier_transition.h | 5 ++ tests/cpp/umbp/distributed/test_peer_pool.cpp | 75 +++++++++++++++++++ 4 files changed, 100 insertions(+), 2 deletions(-) diff --git a/src/umbp/distributed/pool/peer_pool.cpp b/src/umbp/distributed/pool/peer_pool.cpp index 12b28c3ba..e016d1aeb 100644 --- a/src/umbp/distributed/pool/peer_pool.cpp +++ b/src/umbp/distributed/pool/peer_pool.cpp @@ -308,6 +308,13 @@ TierTransitionResult PeerPool::RunTransition(std::unique_lock& lock, return result; } +namespace { +// Further attempts an eviction demotion gets after a retryable failure. Each +// retry goes to the back of the queue, so with a backlog it runs well after the +// fault that failed it. +constexpr uint32_t kEvictDemotionRetries = 3; +} // namespace + void PeerPool::TransitionWorkerLoop() { for (;;) { TransitionJob job; @@ -327,10 +334,17 @@ void PeerPool::TransitionWorkerLoop() { const bool watermark_driven = WatermarkDrivenLocked(job); if (!watermark_driven || tier_graph_->AtOrAbove(job.source_tier, tier.low_watermark)) { const auto result = RunTransition(lock, job); - if (job.evict && !result.success) DropEvictedCopyLocked(job); + // An eviction's demotion that failed on the way (a write error, not a + // full downstream) is tried again before its copy is given up: the + // copy is the only one left, and dropping it turns a transient fault + // into a lost key. Only after the retries, or when nothing below has + // room, is the evicted copy dropped so the tier still frees. + const bool evict_retry = job.evict && !result.success && result.retryable && + job.attempts < kEvictDemotionRetries; + if (job.evict && !result.success && !evict_retry) DropEvictedCopyLocked(job); const bool still_pressured = watermark_driven && tier_graph_->AtOrAbove(job.source_tier, tier.low_watermark); - retry = still_pressured && !result.success && job.attempts < 3; + retry = (still_pressured && !result.success && job.attempts < 3) || evict_retry; // Keep the drain going without waiting for the next commit batch to // notice the tier is still above its low watermark. The scan skips // whatever is still queued, so this tops the batch back up rather diff --git a/src/umbp/distributed/pool/tier_transition.cpp b/src/umbp/distributed/pool/tier_transition.cpp index 9350de8cf..73808a753 100644 --- a/src/umbp/distributed/pool/tier_transition.cpp +++ b/src/umbp/distributed/pool/tier_transition.cpp @@ -99,11 +99,13 @@ TierTransitionResult TierTransitionExecutor::Execute( if (!BuildTransfers(source, resolved, target, allocation, &transfers) || !transfer_engine_->Transfer(transfers, nullptr)) { target->BatchAbort({allocation.slot_id}); + result.retryable = true; continue; } auto committed = target->BatchCommit({CommitRequest{allocation.slot_id, key}}); if (committed.empty() || !committed.front().success) { target->BatchAbort({allocation.slot_id}); + result.retryable = true; continue; } } else if (allocation.outcome != AllocateOutcome::kSuccessAlreadyExists) { @@ -114,10 +116,12 @@ TierTransitionResult TierTransitionExecutor::Execute( if (!target->AcquireMigrationRead(key, &target_read) || target_read.size != resolved.size) { if (target_read.found) target->ReleaseMigrationRead(key); if (allocation.outcome == AllocateOutcome::kSuccessAllocated) target->Evict({key}); + result.retryable = true; continue; } result.success = true; + result.retryable = false; result.target_backend_id = target_backend_id; result.bytes_moved = resolved.size; source->ReleaseMigrationRead(key); diff --git a/src/umbp/include/umbp/distributed/pool/tier_transition.h b/src/umbp/include/umbp/distributed/pool/tier_transition.h index dd36e0de9..39b6df6ef 100644 --- a/src/umbp/include/umbp/distributed/pool/tier_transition.h +++ b/src/umbp/include/umbp/distributed/pool/tier_transition.h @@ -43,6 +43,11 @@ struct TierTransitionResult { uint64_t bytes_moved = 0; uint64_t bytes_freed = 0; bool source_draining = false; + // A target had room but the copy, its commit, or the read-back check failed. + // That is a fault on the way (a device write error, a transfer failure), not + // a verdict that nothing downstream can take the key, so trying again later + // may succeed. Always false on success and when no target had room. + bool retryable = false; }; // Executes one peer-local copy or move. Placement policy and scheduling remain diff --git a/tests/cpp/umbp/distributed/test_peer_pool.cpp b/tests/cpp/umbp/distributed/test_peer_pool.cpp index 51a8e694f..e9fb57e5c 100644 --- a/tests/cpp/umbp/distributed/test_peer_pool.cpp +++ b/tests/cpp/umbp/distributed/test_peer_pool.cpp @@ -176,6 +176,39 @@ class NoSpaceBackend final : public MockBackend { } }; +// Moves bytes like LocalCopyEngine, except that its first `failures` copies +// fail -- the transient fault (a device or transport error) a demotion can hit +// on the way down although the target has room. +class FailingEngine final : public TransferEngine { + public: + explicit FailingEngine(int failures) : failures_(failures) {} + void FailNext(int n) { failures_.store(n); } + + const char* Name() const override { return "failing"; } + TransferRef RegisterMemory(void* base, size_t size, mori::io::MemoryLocationType loc, + int device) override { + return inner_.RegisterMemory(base, size, loc, device); + } + void Deregister(const TransferRef& ref) override { inner_.Deregister(ref); } + bool CanHandle(const TransferRef& src, const TransferRef& dst) const override { + return inner_.CanHandle(src, dst); + } + TransferPlanSet Plan(const std::vector& items) const override { + return inner_.Plan(items); + } + std::unique_ptr Submit(std::vector plans) override { + if (failures_.load() > 0) { + failures_.fetch_sub(1); + return nullptr; // the copy never happened + } + return inner_.Submit(std::move(plans)); + } + + private: + LocalCopyEngine inner_; + std::atomic failures_; +}; + class BusyResolveBackend final : public MockBackend { public: explicit BusyResolveBackend(TierType tier) : MockBackend(tier) {} @@ -826,6 +859,48 @@ TEST(PeerPool, DemotionWithNoRoomDownstreamDropsTheCopy) { EXPECT_FALSE(pool.PlacementBackend("k").has_value()); } +// A demotion that fails on the way down (here: the target's commit) is tried +// again, not abandoned -- the evicted copy is the only one, so giving up turns a +// transient write error into a lost key. +TEST(PeerPool, ATransientDemotionFailureIsRetriedNotDropped) { + constexpr uint64_t kPageSize = 4096; + FailingEngine engine(/*failures=*/0); + BackendRegistry registry; + RegisterPageBackend(®istry, &engine, "hot", TierType::DRAM, kPageSize, 4); + RegisterPageBackend(®istry, &engine, "cold", TierType::SSD, kPageSize, 4); + auto compiled = LogicalTierGraph::Compile(HotColdTiers(PoolOffloadTrigger::kOnEvict), registry); + ASSERT_TRUE(compiled.ok()) << compiled.error; + PeerPool pool(®istry, MakeTieredPlacementPolicy(compiled.graph), &engine); + CommitKey(&pool, "k"); + + engine.FailNext(2); // the first two demotion copies fail; the third lands + auto evicted = pool.Evict({PoolEvictRequest{"k", TierType::DRAM}}, PoolEvictMode::kReclaim); + EXPECT_EQ(evicted.front().outcome, PoolEvictOutcome::kDemotionQueued); + ASSERT_TRUE(WaitForKeyGone(®istry, "hot", "k")); + EXPECT_TRUE(registry.Get("cold")->Contains("k")) << "the key was dropped, not demoted"; + EXPECT_TRUE(pool.PlacementBackend("k").has_value()); +} + +// ...but a target that never accepts it does not pin the source tier forever: +// after the retries the evicted copy is dropped, so eviction still frees. +TEST(PeerPool, APersistentDemotionFailureStillFreesTheTier) { + constexpr uint64_t kPageSize = 4096; + FailingEngine engine(/*failures=*/0); + BackendRegistry registry; + RegisterPageBackend(®istry, &engine, "hot", TierType::DRAM, kPageSize, 4); + RegisterPageBackend(®istry, &engine, "cold", TierType::SSD, kPageSize, 4); + auto compiled = LogicalTierGraph::Compile(HotColdTiers(PoolOffloadTrigger::kOnEvict), registry); + ASSERT_TRUE(compiled.ok()) << compiled.error; + PeerPool pool(®istry, MakeTieredPlacementPolicy(compiled.graph), &engine); + CommitKey(&pool, "k"); + + engine.FailNext(1000); // every demotion copy fails + pool.Evict({PoolEvictRequest{"k", TierType::DRAM}}, PoolEvictMode::kReclaim); + ASSERT_TRUE(WaitForKeyGone(®istry, "hot", "k")); + EXPECT_FALSE(registry.Get("cold")->Contains("k")); + EXPECT_FALSE(pool.PlacementBackend("k").has_value()); +} + TEST(PeerPool, WatermarkMigratesCommittedKey) { // Unlike the other tier tests, this one asserts on utilization, so the owned // buffer must be a whole number of host allocation granules: a smaller From ece6dbbc61581c2c40608a9cbdbf65dd28209d59 Mon Sep 17 00:00:00 2001 From: TianDi101 Date: Mon, 12 Oct 2026 07:50:51 +0000 Subject: [PATCH 13/19] fix(umbp): an eviction demotion waits out a busy target instead of dropping When a demotion's target answered "no space" the transition worker dropped the evicted copy at once -- the only copy. For an SSD target "no space" only ever means its staging arena is momentarily full of in-flight reads (BatchAllocate does not check device capacity), so under heavy read load the e2e run lost 137 keys in one 30 s window, each of which a reader had just been told was present: its next get failed. No space at a target is now retryable, and a retryable eviction demotion is retried for a wall-clock budget (UMBP_EVICT_DEMOTION_RETRY_MS, default 30 s, from its first attempt) instead of a fixed count, which a short queue burned in milliseconds. A retry pauses 20 ms only when nothing else is queued, so it neither spins nor holds up other transitions. The copy is still dropped once the budget is spent, or at once when the target can never take the key, so a source tier always frees. Co-Authored-By: Claude Opus 5.5 --- src/umbp/distributed/pool/peer_pool.cpp | 57 +++++++++++++------ src/umbp/distributed/pool/tier_transition.cpp | 4 ++ src/umbp/doc/runtime-env-vars.md | 1 + .../include/umbp/distributed/pool/peer_pool.h | 11 +++- .../umbp/distributed/pool/tier_transition.h | 9 +-- tests/cpp/umbp/distributed/test_peer_pool.cpp | 31 +++++++++- 6 files changed, 91 insertions(+), 22 deletions(-) diff --git a/src/umbp/distributed/pool/peer_pool.cpp b/src/umbp/distributed/pool/peer_pool.cpp index e016d1aeb..f4eb21704 100644 --- a/src/umbp/distributed/pool/peer_pool.cpp +++ b/src/umbp/distributed/pool/peer_pool.cpp @@ -34,7 +34,25 @@ #include #include +#include "umbp/common/env_time.h" + namespace mori::umbp { + +namespace { +// An eviction demotion that failed retryably goes to the back of the queue and +// runs again until this long has passed since its first attempt; only then is +// its copy dropped. A time budget rather than a count, because the faults it +// rides out -- an SSD staging arena full of reads, a write-error burst -- last +// a wall-clock interval, and a short queue would burn a count in milliseconds. +std::chrono::milliseconds EvictDemotionRetryBudget() { + static const auto v = GetEnvMilliseconds("UMBP_EVICT_DEMOTION_RETRY_MS", + std::chrono::milliseconds(30000), /*min_allowed=*/0); + return v; +} +// Pause before re-running a retry when nothing else is queued, so a lone job +// waiting out pressure does not spin the worker. +constexpr std::chrono::milliseconds kEvictRetryIdlePause{20}; +} // namespace namespace { // One watermark scan queues at most this many keys; the worker tops the queue @@ -86,6 +104,7 @@ PeerPool::PeerPool(BackendRegistry* backends, std::unique_ptr policy break; } } + evict_demotion_retry_budget_ = EvictDemotionRetryBudget(); transition_worker_ = std::thread(&PeerPool::TransitionWorkerLoop, this); } } @@ -308,13 +327,6 @@ TierTransitionResult PeerPool::RunTransition(std::unique_lock& lock, return result; } -namespace { -// Further attempts an eviction demotion gets after a retryable failure. Each -// retry goes to the back of the queue, so with a backlog it runs well after the -// fault that failed it. -constexpr uint32_t kEvictDemotionRetries = 3; -} // namespace - void PeerPool::TransitionWorkerLoop() { for (;;) { TransitionJob job; @@ -327,20 +339,26 @@ void PeerPool::TransitionWorkerLoop() { transition_queue_.pop_front(); } bool retry = false; + bool evict_retry = false; { std::unique_lock lock(operation_mutex_); if (tier_graph_ != nullptr) { const auto& tier = tier_graph_->NodeAt(job.source_tier); const bool watermark_driven = WatermarkDrivenLocked(job); if (!watermark_driven || tier_graph_->AtOrAbove(job.source_tier, tier.low_watermark)) { + if (job.evict && job.first_attempt == std::chrono::steady_clock::time_point{}) { + job.first_attempt = std::chrono::steady_clock::now(); + } const auto result = RunTransition(lock, job); - // An eviction's demotion that failed on the way (a write error, not a - // full downstream) is tried again before its copy is given up: the - // copy is the only one left, and dropping it turns a transient fault - // into a lost key. Only after the retries, or when nothing below has - // room, is the evicted copy dropped so the tier still frees. - const bool evict_retry = job.evict && !result.success && result.retryable && - job.attempts < kEvictDemotionRetries; + // An eviction's demotion that failed in a way that can clear (no space + // at the target, a write error) is tried again before its copy is + // given up: the copy is the only one left, and a reader may already + // have been told the key is here. Only once the retry budget is spent, + // or the target can never take the key, is the evicted copy dropped, + // so the tier still frees. + evict_retry = + job.evict && !result.success && result.retryable && + std::chrono::steady_clock::now() - job.first_attempt < evict_demotion_retry_budget_; if (job.evict && !result.success && !evict_retry) DropEvictedCopyLocked(job); const bool still_pressured = watermark_driven && tier_graph_->AtOrAbove(job.source_tier, tier.low_watermark); @@ -356,7 +374,14 @@ void PeerPool::TransitionWorkerLoop() { } } } - if (retry) { + if (evict_retry) { + bool idle = false; + { + std::lock_guard lock(transition_mutex_); + idle = transition_queue_.empty(); + } + if (idle) std::this_thread::sleep_for(kEvictRetryIdlePause); + } else if (retry) { std::this_thread::sleep_for(std::chrono::milliseconds(uint64_t{1} << job.attempts)); } { @@ -364,7 +389,7 @@ void PeerPool::TransitionWorkerLoop() { queued_transition_keys_.erase(job.key); const size_t source_tier = job.source_tier; if (retry && !stop_transition_worker_) { - ++job.attempts; + if (job.attempts < UINT8_MAX) ++job.attempts; queued_transition_keys_.insert(job.key); transition_queue_.push_back(std::move(job)); transition_cv_.notify_one(); diff --git a/src/umbp/distributed/pool/tier_transition.cpp b/src/umbp/distributed/pool/tier_transition.cpp index 73808a753..b8c7f6fd6 100644 --- a/src/umbp/distributed/pool/tier_transition.cpp +++ b/src/umbp/distributed/pool/tier_transition.cpp @@ -109,6 +109,10 @@ TierTransitionResult TierTransitionExecutor::Execute( continue; } } else if (allocation.outcome != AllocateOutcome::kSuccessAlreadyExists) { + // No space is pressure, not a verdict: an SSD reports it while its staging + // arena is full of in-flight reads, a page tier while its eviction catches + // up. A shape failure (kFailed) can never succeed and stays final. + if (allocation.outcome == AllocateOutcome::kFailedNoSpace) result.retryable = true; continue; } diff --git a/src/umbp/doc/runtime-env-vars.md b/src/umbp/doc/runtime-env-vars.md index ebc19e2b1..1f3cf662d 100644 --- a/src/umbp/doc/runtime-env-vars.md +++ b/src/umbp/doc/runtime-env-vars.md @@ -61,6 +61,7 @@ Read by the **master process** (`bin/master_main.cpp` via | `UMBP_HEARTBEAT_INTERVAL_DIVISOR` | `2` | count | Recommended client heartbeat interval = `heartbeat_ttl / divisor`. `min_allowed=1` guards against div-by-zero. Read by the master and echoed in `RegisterClientResponse.heartbeat_interval_ms`. | | `UMBP_EVICTKEY_DEADLINE_MS` | `1000` | ms | Per-call gRPC deadline applied to outbound `EvictKey` RPCs from `MasterPeerStubPool`. | | `UMBP_EVICT_MAX_BYTES_PER_ROUND` | `0` | bytes | Most bytes one eviction round frees from one `(node, tier)`. `0` = its whole overage above the low watermark. Set it to throttle eviction, e.g. how much a demoting tier copies downstream per round. | +| `UMBP_EVICT_DEMOTION_RETRY_MS` | `30000` | ms | How long a peer retries an eviction demotion that failed in a way that can clear (no space at the target, e.g. an SSD staging arena full of reads, or a copy/commit error) before it drops the evicted copy so the tier still frees. Counted from the first attempt. `0` = drop on the first such failure. | | `UMBP_HIT_INDEX_TTL_SEC` | `7200` | sec | External KV hit-count entry TTL. A hash with no counted match for longer than this is removed from the hit index. | | `UMBP_HIT_INDEX_GC_INTERVAL_SEC` | `60` | sec | External KV hit-count GC sweep interval. | | `UMBP_HIT_QUERY_MAX_BATCH` | `4096` | count | Maximum hashes accepted by one `GetExternalKvHitCounts` request. Oversized requests return gRPC `INVALID_ARGUMENT`; the server does not truncate. | diff --git a/src/umbp/include/umbp/distributed/pool/peer_pool.h b/src/umbp/include/umbp/distributed/pool/peer_pool.h index e5facb6f3..b1c784bac 100644 --- a/src/umbp/include/umbp/distributed/pool/peer_pool.h +++ b/src/umbp/include/umbp/distributed/pool/peer_pool.h @@ -155,6 +155,11 @@ class PeerPool { LocalEvictionMetrics LocalEvictionStats() const; // Test seam: one synchronous pass. Returns keys dropped or queued. size_t RunLocalEvictionOnceForTest() { return LocalEvictionPass(); } + // How long an eviction demotion keeps retrying before its copy is dropped + // (UMBP_EVICT_DEMOTION_RETRY_MS, default 30 s). Tests shorten it. + void SetEvictDemotionRetryBudgetForTest(std::chrono::milliseconds budget) { + evict_demotion_retry_budget_ = budget; + } // Victims for a backend over its watermark, coldest first, before the // medium's own order: (tier, logical tier, bytes) -> offers, or nullopt when @@ -176,7 +181,9 @@ class PeerPool { uint32_t source_backend_id = BackendRegistry::kMaxBackends; LogicalTierGraph::TierIndex source_tier = 0; uint8_t attempts = 0; - bool evict = false; // from Evict(): ignores watermarks; drops on failure + // An eviction demotion's retries are bounded by time from its first attempt. + std::chrono::steady_clock::time_point first_attempt{}; + bool evict = false; // from Evict(): ignores watermarks; drops once retries run out }; // What the copy phase needs, resolved while the pool lock is still held. @@ -294,6 +301,8 @@ class PeerPool { std::vector queued_by_tier_; bool stop_transition_worker_ = false; std::thread transition_worker_; + // Read by the transition worker only (and the test seam before it starts). + std::chrono::milliseconds evict_demotion_retry_budget_{30000}; // Local watermark eviction. local_evict_mutex_ guards local_evict_, // stop_local_evict_ and the worker's start. diff --git a/src/umbp/include/umbp/distributed/pool/tier_transition.h b/src/umbp/include/umbp/distributed/pool/tier_transition.h index 39b6df6ef..f81c484bd 100644 --- a/src/umbp/include/umbp/distributed/pool/tier_transition.h +++ b/src/umbp/include/umbp/distributed/pool/tier_transition.h @@ -43,10 +43,11 @@ struct TierTransitionResult { uint64_t bytes_moved = 0; uint64_t bytes_freed = 0; bool source_draining = false; - // A target had room but the copy, its commit, or the read-back check failed. - // That is a fault on the way (a device write error, a transfer failure), not - // a verdict that nothing downstream can take the key, so trying again later - // may succeed. Always false on success and when no target had room. + // Every target refused or failed in a way that can clear: it reported no + // space (an SSD's staging arena under read load, or a tier its own eviction + // is about to free), or it had room but the copy, its commit, or the + // read-back check failed. Trying again later may succeed. False on success + // and when the target can never take the key (a shape failure). bool retryable = false; }; diff --git a/tests/cpp/umbp/distributed/test_peer_pool.cpp b/tests/cpp/umbp/distributed/test_peer_pool.cpp index e9fb57e5c..10a5bfaaf 100644 --- a/tests/cpp/umbp/distributed/test_peer_pool.cpp +++ b/tests/cpp/umbp/distributed/test_peer_pool.cpp @@ -843,6 +843,7 @@ TEST(PeerPool, DemotionWithNoRoomDownstreamDropsTheCopy) { auto compiled = LogicalTierGraph::Compile(HotColdTiers(PoolOffloadTrigger::kOnEvict), registry); ASSERT_TRUE(compiled.ok()) << compiled.error; PeerPool pool(®istry, MakeTieredPlacementPolicy(compiled.graph), &engine); + pool.SetEvictDemotionRetryBudgetForTest(std::chrono::milliseconds{50}); // Fill cold's only page directly. auto* cold = registry.Get("cold"); auto filler = cold->BatchAllocate({AllocateRequest{"filler", kPageSize}}).front(); @@ -894,13 +895,41 @@ TEST(PeerPool, APersistentDemotionFailureStillFreesTheTier) { PeerPool pool(®istry, MakeTieredPlacementPolicy(compiled.graph), &engine); CommitKey(&pool, "k"); - engine.FailNext(1000); // every demotion copy fails + pool.SetEvictDemotionRetryBudgetForTest(std::chrono::milliseconds{50}); + engine.FailNext(1000000); // every demotion copy fails pool.Evict({PoolEvictRequest{"k", TierType::DRAM}}, PoolEvictMode::kReclaim); ASSERT_TRUE(WaitForKeyGone(®istry, "hot", "k")); EXPECT_FALSE(registry.Get("cold")->Contains("k")); EXPECT_FALSE(pool.PlacementBackend("k").has_value()); } +// No space at the target is pressure, not a verdict: an SSD reports it while +// its staging arena is full of reads. The demotion waits it out instead of +// dropping the only copy -- dropping it lost keys a reader had just been told +// were there. +TEST(PeerPool, ADemotionWaitsOutATargetWithNoRoomForNow) { + constexpr uint64_t kPageSize = 4096; + LocalCopyEngine engine; + BackendRegistry registry; + RegisterPageBackend(®istry, &engine, "hot", TierType::DRAM, kPageSize, 4); + RegisterPageBackend(®istry, &engine, "cold", TierType::SSD, kPageSize, 1); + auto compiled = LogicalTierGraph::Compile(HotColdTiers(PoolOffloadTrigger::kOnEvict), registry); + ASSERT_TRUE(compiled.ok()) << compiled.error; + PeerPool pool(®istry, MakeTieredPlacementPolicy(compiled.graph), &engine); + auto* cold = registry.Get("cold"); + auto filler = cold->BatchAllocate({AllocateRequest{"filler", kPageSize}}).front(); + ASSERT_EQ(filler.outcome, AllocateOutcome::kSuccessAllocated); // cold's only page, pending + CommitKey(&pool, "k"); + + pool.Evict({PoolEvictRequest{"k", TierType::DRAM}}, PoolEvictMode::kReclaim); + std::this_thread::sleep_for(std::chrono::milliseconds{100}); + EXPECT_TRUE(registry.Get("hot")->Contains("k")) << "dropped while the target was only busy"; + + cold->BatchAbort({filler.slot_id}); // the pressure clears + ASSERT_TRUE(WaitForKeyGone(®istry, "hot", "k")); + EXPECT_TRUE(cold->Contains("k")); +} + TEST(PeerPool, WatermarkMigratesCommittedKey) { // Unlike the other tier tests, this one asserts on utilization, so the owned // buffer must be a whole number of host allocation granules: a smaller From 74544f8d461583f580888d910cc68db30bcac933 Mon Sep 17 00:00:00 2001 From: TianDi101 Date: Mon, 12 Oct 2026 08:49:44 +0000 Subject: [PATCH 14/19] feat(umbp): count evicted copies the pool gave up on `failed` counts every failed transition attempt, including ones a retry later rescues, so it cannot tell a pressure burst from lost keys. The pool now counts each evicted copy it actually drops -- the demotion could not land within its retry budget, or never could -- as TierTransitionMetrics::dropped, exported as mori_umbp_client_tier_transitions_total{outcome="dropped"}. The activity summary's "K dropped" comes with #683, which adds that summary. Co-Authored-By: Claude Opus 5.5 --- src/umbp/distributed/pool/peer_pool.cpp | 3 ++- src/umbp/distributed/pool_client.cpp | 2 ++ src/umbp/include/umbp/distributed/types.h | 4 ++++ tests/cpp/umbp/distributed/test_peer_pool.cpp | 5 +++++ 4 files changed, 13 insertions(+), 1 deletion(-) diff --git a/src/umbp/distributed/pool/peer_pool.cpp b/src/umbp/distributed/pool/peer_pool.cpp index f4eb21704..adcf81c0c 100644 --- a/src/umbp/distributed/pool/peer_pool.cpp +++ b/src/umbp/distributed/pool/peer_pool.cpp @@ -132,7 +132,8 @@ void PeerPool::DropEvictedCopyLocked(const TransitionJob& job) { if (migrating_keys_.count(job.key) != 0) return; auto* source = backends_ == nullptr ? nullptr : backends_->Get(job.source_backend_id); if (source == nullptr) return; - source->Evict({job.key}); + const auto evicted = source->Evict({job.key}); + if (!evicted.empty() && evicted.front().bytes_freed > 0) ++transition_metrics_.dropped; RepairAfterDropLocked(job.key); } diff --git a/src/umbp/distributed/pool_client.cpp b/src/umbp/distributed/pool_client.cpp index 07bd6f444..659f623de 100644 --- a/src/umbp/distributed/pool_client.cpp +++ b/src/umbp/distributed/pool_client.cpp @@ -5118,6 +5118,8 @@ void PoolClient::PublishComponentMetrics() { {{"outcome", "succeeded"}}, tiers.succeeded); sample(MORI_UMBP_METRIC_CLIENT_TIER_TRANSITIONS, MORI_UMBP_METRIC_CLIENT_TIER_TRANSITIONS_HELP, {{"outcome", "failed"}}, tiers.failed); + sample(MORI_UMBP_METRIC_CLIENT_TIER_TRANSITIONS, MORI_UMBP_METRIC_CLIENT_TIER_TRANSITIONS_HELP, + {{"outcome", "dropped"}}, tiers.dropped); sample(MORI_UMBP_METRIC_CLIENT_TIER_OFFLOADED_BYTES, MORI_UMBP_METRIC_CLIENT_TIER_OFFLOADED_BYTES_HELP, {}, tiers.offloaded_bytes); sample(MORI_UMBP_METRIC_CLIENT_TIER_PROMOTED_BYTES, diff --git a/src/umbp/include/umbp/distributed/types.h b/src/umbp/include/umbp/distributed/types.h index bc3e95b44..4b83814af 100644 --- a/src/umbp/include/umbp/distributed/types.h +++ b/src/umbp/include/umbp/distributed/types.h @@ -80,6 +80,10 @@ struct TierTransitionMetrics { uint64_t attempted = 0; uint64_t succeeded = 0; uint64_t failed = 0; + // Evicted copies given up on: their demotion could not land within the retry + // budget (or never could), so the key left this node. `failed` counts every + // failed attempt, including ones a retry later rescued; this counts loss. + uint64_t dropped = 0; uint64_t offloaded_bytes = 0; uint64_t promoted_bytes = 0; }; diff --git a/tests/cpp/umbp/distributed/test_peer_pool.cpp b/tests/cpp/umbp/distributed/test_peer_pool.cpp index 10a5bfaaf..a78af1002 100644 --- a/tests/cpp/umbp/distributed/test_peer_pool.cpp +++ b/tests/cpp/umbp/distributed/test_peer_pool.cpp @@ -858,6 +858,7 @@ TEST(PeerPool, DemotionWithNoRoomDownstreamDropsTheCopy) { EXPECT_FALSE(cold->Contains("k")); EXPECT_TRUE(cold->Contains("filler")); EXPECT_FALSE(pool.PlacementBackend("k").has_value()); + EXPECT_EQ(pool.TransitionMetrics().dropped, 1u); } // A demotion that fails on the way down (here: the target's commit) is tried @@ -880,6 +881,8 @@ TEST(PeerPool, ATransientDemotionFailureIsRetriedNotDropped) { ASSERT_TRUE(WaitForKeyGone(®istry, "hot", "k")); EXPECT_TRUE(registry.Get("cold")->Contains("k")) << "the key was dropped, not demoted"; EXPECT_TRUE(pool.PlacementBackend("k").has_value()); + EXPECT_EQ(pool.TransitionMetrics().dropped, 0u); + EXPECT_GE(pool.TransitionMetrics().failed, 2u) << "the failed attempts still count"; } // ...but a target that never accepts it does not pin the source tier forever: @@ -901,6 +904,7 @@ TEST(PeerPool, APersistentDemotionFailureStillFreesTheTier) { ASSERT_TRUE(WaitForKeyGone(®istry, "hot", "k")); EXPECT_FALSE(registry.Get("cold")->Contains("k")); EXPECT_FALSE(pool.PlacementBackend("k").has_value()); + EXPECT_EQ(pool.TransitionMetrics().dropped, 1u); } // No space at the target is pressure, not a verdict: an SSD reports it while @@ -928,6 +932,7 @@ TEST(PeerPool, ADemotionWaitsOutATargetWithNoRoomForNow) { cold->BatchAbort({filler.slot_id}); // the pressure clears ASSERT_TRUE(WaitForKeyGone(®istry, "hot", "k")); EXPECT_TRUE(cold->Contains("k")); + EXPECT_EQ(pool.TransitionMetrics().dropped, 0u); } TEST(PeerPool, WatermarkMigratesCommittedKey) { From 9d2fe968649cc0b3218d601df1655675e1e51932 Mon Sep 17 00:00:00 2001 From: TianDi101 Date: Tue, 13 Oct 2026 15:34:36 +0000 Subject: [PATCH 15/19] refactor(umbp): trim the demotion-retry comments; fold its tests Comment-only, plus one anonymous namespace instead of two back to back. The transient- and persistent-failure tests become one (retried until it lands; dropped once the budget runs out). Co-Authored-By: Claude Opus 5.5 --- src/umbp/distributed/pool/peer_pool.cpp | 22 +++----- src/umbp/distributed/pool/tier_transition.cpp | 5 +- .../include/umbp/distributed/pool/peer_pool.h | 6 +-- .../umbp/distributed/pool/tier_transition.h | 7 +-- src/umbp/include/umbp/distributed/types.h | 5 +- tests/cpp/umbp/distributed/test_peer_pool.cpp | 53 ++++++------------- 6 files changed, 31 insertions(+), 67 deletions(-) diff --git a/src/umbp/distributed/pool/peer_pool.cpp b/src/umbp/distributed/pool/peer_pool.cpp index adcf81c0c..162bc588e 100644 --- a/src/umbp/distributed/pool/peer_pool.cpp +++ b/src/umbp/distributed/pool/peer_pool.cpp @@ -39,21 +39,17 @@ namespace mori::umbp { namespace { -// An eviction demotion that failed retryably goes to the back of the queue and -// runs again until this long has passed since its first attempt; only then is -// its copy dropped. A time budget rather than a count, because the faults it -// rides out -- an SSD staging arena full of reads, a write-error burst -- last -// a wall-clock interval, and a short queue would burn a count in milliseconds. + +// How long a retryable eviction demotion keeps retrying before its copy is +// dropped. Time, not a count: the faults it rides out last a wall-clock +// interval, and a short queue would burn a count in milliseconds. std::chrono::milliseconds EvictDemotionRetryBudget() { static const auto v = GetEnvMilliseconds("UMBP_EVICT_DEMOTION_RETRY_MS", std::chrono::milliseconds(30000), /*min_allowed=*/0); return v; } -// Pause before re-running a retry when nothing else is queued, so a lone job -// waiting out pressure does not spin the worker. +// Pause before a lone retry, so waiting out pressure does not spin the worker. constexpr std::chrono::milliseconds kEvictRetryIdlePause{20}; -} // namespace -namespace { // One watermark scan queues at most this many keys; the worker tops the queue // up after each completed offload, so a tier still drains all the way down to @@ -351,12 +347,8 @@ void PeerPool::TransitionWorkerLoop() { job.first_attempt = std::chrono::steady_clock::now(); } const auto result = RunTransition(lock, job); - // An eviction's demotion that failed in a way that can clear (no space - // at the target, a write error) is tried again before its copy is - // given up: the copy is the only one left, and a reader may already - // have been told the key is here. Only once the retry budget is spent, - // or the target can never take the key, is the evicted copy dropped, - // so the tier still frees. + // The evicted copy is the only one: retry what can clear, and drop it + // only once the budget is spent or the target can never take it. evict_retry = job.evict && !result.success && result.retryable && std::chrono::steady_clock::now() - job.first_attempt < evict_demotion_retry_budget_; diff --git a/src/umbp/distributed/pool/tier_transition.cpp b/src/umbp/distributed/pool/tier_transition.cpp index b8c7f6fd6..cf926160b 100644 --- a/src/umbp/distributed/pool/tier_transition.cpp +++ b/src/umbp/distributed/pool/tier_transition.cpp @@ -109,9 +109,8 @@ TierTransitionResult TierTransitionExecutor::Execute( continue; } } else if (allocation.outcome != AllocateOutcome::kSuccessAlreadyExists) { - // No space is pressure, not a verdict: an SSD reports it while its staging - // arena is full of in-flight reads, a page tier while its eviction catches - // up. A shape failure (kFailed) can never succeed and stays final. + // No space is pressure (staging full of reads, eviction catching up); + // a shape failure (kFailed) stays final. if (allocation.outcome == AllocateOutcome::kFailedNoSpace) result.retryable = true; continue; } diff --git a/src/umbp/include/umbp/distributed/pool/peer_pool.h b/src/umbp/include/umbp/distributed/pool/peer_pool.h index b1c784bac..42d429c62 100644 --- a/src/umbp/include/umbp/distributed/pool/peer_pool.h +++ b/src/umbp/include/umbp/distributed/pool/peer_pool.h @@ -155,8 +155,7 @@ class PeerPool { LocalEvictionMetrics LocalEvictionStats() const; // Test seam: one synchronous pass. Returns keys dropped or queued. size_t RunLocalEvictionOnceForTest() { return LocalEvictionPass(); } - // How long an eviction demotion keeps retrying before its copy is dropped - // (UMBP_EVICT_DEMOTION_RETRY_MS, default 30 s). Tests shorten it. + // UMBP_EVICT_DEMOTION_RETRY_MS override; set it before queueing work. void SetEvictDemotionRetryBudgetForTest(std::chrono::milliseconds budget) { evict_demotion_retry_budget_ = budget; } @@ -301,8 +300,7 @@ class PeerPool { std::vector queued_by_tier_; bool stop_transition_worker_ = false; std::thread transition_worker_; - // Read by the transition worker only (and the test seam before it starts). - std::chrono::milliseconds evict_demotion_retry_budget_{30000}; + std::chrono::milliseconds evict_demotion_retry_budget_{30000}; // transition worker only // Local watermark eviction. local_evict_mutex_ guards local_evict_, // stop_local_evict_ and the worker's start. diff --git a/src/umbp/include/umbp/distributed/pool/tier_transition.h b/src/umbp/include/umbp/distributed/pool/tier_transition.h index f81c484bd..93e2ccfde 100644 --- a/src/umbp/include/umbp/distributed/pool/tier_transition.h +++ b/src/umbp/include/umbp/distributed/pool/tier_transition.h @@ -43,11 +43,8 @@ struct TierTransitionResult { uint64_t bytes_moved = 0; uint64_t bytes_freed = 0; bool source_draining = false; - // Every target refused or failed in a way that can clear: it reported no - // space (an SSD's staging arena under read load, or a tier its own eviction - // is about to free), or it had room but the copy, its commit, or the - // read-back check failed. Trying again later may succeed. False on success - // and when the target can never take the key (a shape failure). + // Failed in a way that can clear (no space, or a copy/commit/read-back + // error), so a later attempt may succeed. False for a shape failure. bool retryable = false; }; diff --git a/src/umbp/include/umbp/distributed/types.h b/src/umbp/include/umbp/distributed/types.h index 4b83814af..dd8533752 100644 --- a/src/umbp/include/umbp/distributed/types.h +++ b/src/umbp/include/umbp/distributed/types.h @@ -80,10 +80,7 @@ struct TierTransitionMetrics { uint64_t attempted = 0; uint64_t succeeded = 0; uint64_t failed = 0; - // Evicted copies given up on: their demotion could not land within the retry - // budget (or never could), so the key left this node. `failed` counts every - // failed attempt, including ones a retry later rescued; this counts loss. - uint64_t dropped = 0; + uint64_t dropped = 0; // evicted copies whose demotion never landed (lost keys) uint64_t offloaded_bytes = 0; uint64_t promoted_bytes = 0; }; diff --git a/tests/cpp/umbp/distributed/test_peer_pool.cpp b/tests/cpp/umbp/distributed/test_peer_pool.cpp index a78af1002..8c07de9f9 100644 --- a/tests/cpp/umbp/distributed/test_peer_pool.cpp +++ b/tests/cpp/umbp/distributed/test_peer_pool.cpp @@ -176,9 +176,7 @@ class NoSpaceBackend final : public MockBackend { } }; -// Moves bytes like LocalCopyEngine, except that its first `failures` copies -// fail -- the transient fault (a device or transport error) a demotion can hit -// on the way down although the target has room. +// LocalCopyEngine whose next `failures` copies fail. class FailingEngine final : public TransferEngine { public: explicit FailingEngine(int failures) : failures_(failures) {} @@ -861,10 +859,9 @@ TEST(PeerPool, DemotionWithNoRoomDownstreamDropsTheCopy) { EXPECT_EQ(pool.TransitionMetrics().dropped, 1u); } -// A demotion that fails on the way down (here: the target's commit) is tried -// again, not abandoned -- the evicted copy is the only one, so giving up turns a -// transient write error into a lost key. -TEST(PeerPool, ATransientDemotionFailureIsRetriedNotDropped) { +// A demotion copy that fails is retried, since the evicted copy is the only +// one; it is dropped only once the retry budget runs out, so the tier frees. +TEST(PeerPool, AFailedDemotionIsRetriedUntilItsBudgetRunsOut) { constexpr uint64_t kPageSize = 4096; FailingEngine engine(/*failures=*/0); BackendRegistry registry; @@ -873,44 +870,28 @@ TEST(PeerPool, ATransientDemotionFailureIsRetriedNotDropped) { auto compiled = LogicalTierGraph::Compile(HotColdTiers(PoolOffloadTrigger::kOnEvict), registry); ASSERT_TRUE(compiled.ok()) << compiled.error; PeerPool pool(®istry, MakeTieredPlacementPolicy(compiled.graph), &engine); - CommitKey(&pool, "k"); + CommitKey(&pool, "a"); + CommitKey(&pool, "b"); - engine.FailNext(2); // the first two demotion copies fail; the third lands - auto evicted = pool.Evict({PoolEvictRequest{"k", TierType::DRAM}}, PoolEvictMode::kReclaim); + engine.FailNext(2); // the first two copies fail; the third lands + auto evicted = pool.Evict({PoolEvictRequest{"a", TierType::DRAM}}, PoolEvictMode::kReclaim); EXPECT_EQ(evicted.front().outcome, PoolEvictOutcome::kDemotionQueued); - ASSERT_TRUE(WaitForKeyGone(®istry, "hot", "k")); - EXPECT_TRUE(registry.Get("cold")->Contains("k")) << "the key was dropped, not demoted"; - EXPECT_TRUE(pool.PlacementBackend("k").has_value()); + ASSERT_TRUE(WaitForKeyGone(®istry, "hot", "a")); + EXPECT_TRUE(registry.Get("cold")->Contains("a")) << "dropped, not demoted"; EXPECT_EQ(pool.TransitionMetrics().dropped, 0u); EXPECT_GE(pool.TransitionMetrics().failed, 2u) << "the failed attempts still count"; -} - -// ...but a target that never accepts it does not pin the source tier forever: -// after the retries the evicted copy is dropped, so eviction still frees. -TEST(PeerPool, APersistentDemotionFailureStillFreesTheTier) { - constexpr uint64_t kPageSize = 4096; - FailingEngine engine(/*failures=*/0); - BackendRegistry registry; - RegisterPageBackend(®istry, &engine, "hot", TierType::DRAM, kPageSize, 4); - RegisterPageBackend(®istry, &engine, "cold", TierType::SSD, kPageSize, 4); - auto compiled = LogicalTierGraph::Compile(HotColdTiers(PoolOffloadTrigger::kOnEvict), registry); - ASSERT_TRUE(compiled.ok()) << compiled.error; - PeerPool pool(®istry, MakeTieredPlacementPolicy(compiled.graph), &engine); - CommitKey(&pool, "k"); pool.SetEvictDemotionRetryBudgetForTest(std::chrono::milliseconds{50}); - engine.FailNext(1000000); // every demotion copy fails - pool.Evict({PoolEvictRequest{"k", TierType::DRAM}}, PoolEvictMode::kReclaim); - ASSERT_TRUE(WaitForKeyGone(®istry, "hot", "k")); - EXPECT_FALSE(registry.Get("cold")->Contains("k")); - EXPECT_FALSE(pool.PlacementBackend("k").has_value()); + engine.FailNext(1000000); // every copy fails + pool.Evict({PoolEvictRequest{"b", TierType::DRAM}}, PoolEvictMode::kReclaim); + ASSERT_TRUE(WaitForKeyGone(®istry, "hot", "b")); + EXPECT_FALSE(registry.Get("cold")->Contains("b")); + EXPECT_FALSE(pool.PlacementBackend("b").has_value()); EXPECT_EQ(pool.TransitionMetrics().dropped, 1u); } -// No space at the target is pressure, not a verdict: an SSD reports it while -// its staging arena is full of reads. The demotion waits it out instead of -// dropping the only copy -- dropping it lost keys a reader had just been told -// were there. +// No space at the target (an SSD whose staging is full of reads) is waited +// out, not a reason to drop the only copy. TEST(PeerPool, ADemotionWaitsOutATargetWithNoRoomForNow) { constexpr uint64_t kPageSize = 4096; LocalCopyEngine engine; From 458e3e91469dcd2d0a11c67414853d2a4a168d44 Mon Sep 17 00:00:00 2001 From: TianDi101 Date: Wed, 14 Oct 2026 05:27:21 +0000 Subject: [PATCH 16/19] fix(umbp): report local reads to the master for master-ranked eviction The master ranks a node's keys by last_accessed_at, which only RouteGet updates. A local_first node serves its hits without the master, so under master_ranked (and master) the hottest keys looked the coldest and went first: in a 4 GB DRAM / 8 GB SSD smoke a hot set re-read throughout the fill survived 0/512 under master_ranked vs 512/512 under local, and the hot reads then served from SSD slowed demotion until DRAM sat at 100%. A node whose eviction follows the master's recency now ships the keys it reads locally (deduped, bounded) on its heartbeats, and the master records an access for each. Local mode sends nothing. Co-Authored-By: Claude Opus 5.5 --- .../in_memory_master_metadata_store.cpp | 17 +++++++++++ src/umbp/distributed/master/master_client.cpp | 25 +++++++++++++++++ src/umbp/distributed/master/master_server.cpp | 6 ++++ .../master/redis_master_metadata_store.cpp | 17 +++++++++++ src/umbp/distributed/pool_client.cpp | 7 +++++ src/umbp/distributed/proto/umbp.proto | 4 +++ src/umbp/doc/runtime-env-vars.md | 2 +- .../master/in_memory_master_metadata_store.h | 2 ++ .../umbp/distributed/master/master_client.h | 11 ++++++++ .../master/master_metadata_store.h | 6 ++++ .../distributed/master/redis/lua_scripts.h | 15 ++++++++++ .../master/redis_master_metadata_store.h | 2 ++ .../distributed/test_peer_driven_eviction.cpp | 28 +++++++++++++++---- 13 files changed, 136 insertions(+), 6 deletions(-) diff --git a/src/umbp/distributed/master/in_memory_master_metadata_store.cpp b/src/umbp/distributed/master/in_memory_master_metadata_store.cpp index d51bd85a2..e439b9cd1 100644 --- a/src/umbp/distributed/master/in_memory_master_metadata_store.cpp +++ b/src/umbp/distributed/master/in_memory_master_metadata_store.cpp @@ -546,6 +546,23 @@ std::vector> InMemoryMasterMetadataStore::BatchLookupBlock return out; } +void InMemoryMasterMetadataStore::RecordLocalReads(const std::vector& keys, + std::chrono::system_clock::time_point now) { + if (keys.empty()) return; + // An access bump is atomic, so the shard's shared lock is enough. + std::vector> idx_by_shard(num_shards_); + for (size_t i = 0; i < keys.size(); ++i) idx_by_shard[shard_index(keys[i])].push_back(i); + for (size_t si = 0; si < num_shards_; ++si) { + if (idx_by_shard[si].empty()) continue; + auto& sh = shard_at(si); + std::shared_lock lock(sh.mutex); + for (size_t i : idx_by_shard[si]) { + auto it = sh.entries.find(keys[i]); + if (it != sh.entries.end()) it->second.RecordAccessAtomic(now); + } + } +} + std::vector InMemoryMasterMetadataStore::BatchExistsBlock( const std::vector& keys) const { std::vector results(keys.size(), false); diff --git a/src/umbp/distributed/master/master_client.cpp b/src/umbp/distributed/master/master_client.cpp index a85d9ddb2..2a52b2057 100644 --- a/src/umbp/distributed/master/master_client.cpp +++ b/src/umbp/distributed/master/master_client.cpp @@ -479,6 +479,24 @@ std::optional> MasterClient::GetEvictionCandidates( return out; } +void MasterClient::NoteLocalReads(const std::vector& keys) { + // Past kFlushAt distinct keys, ship them now rather than wait out the + // interval; past kMax (master unreachable) the rest are dropped. + constexpr size_t kFlushAt = 4096; + constexpr size_t kMax = 65536; + if (keys.empty() || !ReportsLocalReads()) return; + bool flush = false; + { + std::lock_guard lock(local_reads_mutex_); + for (const auto& key : keys) { + if (local_reads_.size() >= kMax) break; + local_reads_.insert(key); + } + flush = local_reads_.size() >= kFlushAt; + } + if (flush) FlushHeartbeat(); +} + namespace { // Auto-flush the heartbeat once this many KvEvents accumulate at the peer, so a // completed batch of puts becomes visible at the master without the application @@ -817,6 +835,13 @@ bool MasterClient::SendDeltaHeartbeatLocked(const std::map reads; + { + std::lock_guard lock(local_reads_mutex_); + reads.swap(local_reads_); + } + req.mutable_local_reads()->Reserve(static_cast(reads.size())); + while (!reads.empty()) req.add_local_reads(std::move(reads.extract(reads.begin()).value())); grpc::ClientContext ctx; ctx.set_deadline(std::chrono::system_clock::now() + std::chrono::milliseconds(HeartbeatRpcTimeoutMs())); diff --git a/src/umbp/distributed/master/master_server.cpp b/src/umbp/distributed/master/master_server.cpp index 46ccd7ed6..3da880c16 100644 --- a/src/umbp/distributed/master/master_server.cpp +++ b/src/umbp/distributed/master/master_server.cpp @@ -408,6 +408,12 @@ class MasterServer::UMBPMasterServiceImpl final : public ::umbp::UMBPMaster::Ser } } + // After the bundles, so a key added by this heartbeat is already there. + if (client_status != ClientStatus::UNKNOWN && request->local_reads_size() > 0) { + store_.RecordLocalReads({request->local_reads().begin(), request->local_reads().end()}, + now); + } + response->set_status(static_cast<::umbp::ClientStatus>(client_status)); response->set_acked_seq(acked_seq); response->set_request_full_sync(request_full_sync); diff --git a/src/umbp/distributed/master/redis_master_metadata_store.cpp b/src/umbp/distributed/master/redis_master_metadata_store.cpp index 6ed7ebd7b..67e2a76ec 100644 --- a/src/umbp/distributed/master/redis_master_metadata_store.cpp +++ b/src/umbp/distributed/master/redis_master_metadata_store.cpp @@ -799,6 +799,23 @@ std::vector> RedisMasterMetadataStore::BatchLookupBlockFor return out; } +void RedisMasterMetadataStore::RecordLocalReads(const std::vector& keys, + std::chrono::system_clock::time_point now) { + if (keys.empty()) return; + ScopedStoreOp _op(metrics_, "RecordLocalReads"); + const redis::ShardedBatch batch = redis::GroupKeysByShard(keys_, keys); + try { + redis::RunShardedRead( + batch, redis::kRecordLocalReadsLua, {std::to_string(ToEpochMs(now))}, "RecordLocalReads", + fanout_pool_.get(), /*tolerate_shard_failures=*/mode_ != Mode::kSingle, metrics_, + [&](size_t group) { return &client_for_shard(batch.shard_of_group[group]); }, + [](size_t, const RespValue&) {}); + } catch (const std::exception& e) { + // Recency is a hint; losing it must not fail the heartbeat carrying it. + MORI_UMBP_WARN("[RedisStore] RecordLocalReads failed: {}", e.what()); + } +} + std::vector RedisMasterMetadataStore::BatchExistsBlock( const std::vector& keys) const { std::vector results(keys.size(), false); diff --git a/src/umbp/distributed/pool_client.cpp b/src/umbp/distributed/pool_client.cpp index 07bd6f444..6b72c52d4 100644 --- a/src/umbp/distributed/pool_client.cpp +++ b/src/umbp/distributed/pool_client.cpp @@ -1234,6 +1234,9 @@ bool PoolClient::Init() { : default_pool_->LogicalTierCapacities(); // A self-evicting node asks the master to leave it out of its rounds. master_client_->RequestPeerDrivenEviction(self_evicting); + // Unless it evicts by its own LRU, its victims follow the master's + // recency, which must then hear about the reads served here. + master_client_->EnableLocalReadReports(config_.eviction_mode != PeerEvictionMode::kLocal); auto status = master_client_->RegisterSelf(tier_caps, peer_address, engine_desc_bytes, logical_caps); if (!status.ok()) { @@ -1760,15 +1763,19 @@ void PoolClient::ResolveLocalBatch(const std::vector& keys, } } } + const bool report = master_client_ != nullptr && master_client_->ReportsLocalReads(); + std::vector read_keys; for (size_t j = 0; j < candidates.size() && j < found.size(); ++j) { if (!found[j].resolved.found) continue; auto* backend = registry_.Get(found[j].backend_id); if (backend == nullptr) continue; (*holders)[candidates[j]] = backend; + if (report) read_keys.push_back((*batch)[j]); if (resolutions != nullptr) { (*resolutions)[candidates[j]] = std::move(found[j].resolved); } } + if (report) master_client_->NoteLocalReads(read_keys); } // --------------------------------------------------------------------------- diff --git a/src/umbp/distributed/proto/umbp.proto b/src/umbp/distributed/proto/umbp.proto index 28c7d65a2..115c5bd11 100644 --- a/src/umbp/distributed/proto/umbp.proto +++ b/src/umbp/distributed/proto/umbp.proto @@ -147,6 +147,10 @@ message HeartbeatRequest { repeated TierKvCount tier_kv_counts = 7; // live owned-key count per tier (client-reported) repeated LogicalTierCapacity logical_tier_capacities = 8; bool peer_driven_eviction = 9; // as in RegisterClientRequest + // Keys this node read locally since its last heartbeat (best effort, deduped). + // Sent only by a node whose eviction follows the master's recency, which + // local reads would otherwise never reach. + repeated string local_reads = 10; } // A peer over its watermark asks for its own keys on one medium, coldest first diff --git a/src/umbp/doc/runtime-env-vars.md b/src/umbp/doc/runtime-env-vars.md index ebc19e2b1..ca10751e7 100644 --- a/src/umbp/doc/runtime-env-vars.md +++ b/src/umbp/doc/runtime-env-vars.md @@ -75,7 +75,7 @@ that has loaded `libmori_pybinds.so`). | Env var | Default | Unit | Description | |---|---|---|---| -| `UMBP_EVICTION_MODE` | `local` | enum | How a node with a master evicts. `local`: every medium at this node's own watermarks (a tiered policy's per-tier pair, else the SSD / DRAM config pair), by its own LRU, without asking the master; the master leaves it out of its `EvictKey` rounds. `master_ranked`: the same, with victims ranked by the master (`GetEvictionCandidates`), falling back to local LRU. `master`: DRAM/HBM wait for the master's rounds (SSD still evicts locally). A node without a master always evicts locally, and every mode still serves `EvictKey`. | +| `UMBP_EVICTION_MODE` | `local` | enum | How a node with a master evicts. `local`: every medium at this node's own watermarks (a tiered policy's per-tier pair, else the SSD / DRAM config pair), by its own LRU, without asking the master; the master leaves it out of its `EvictKey` rounds. `master_ranked`: the same, with victims ranked by the master (`GetEvictionCandidates`), falling back to local LRU. `master`: DRAM/HBM wait for the master's rounds (SSD still evicts locally). In both master modes the node reports the keys it reads locally on its heartbeats, since the master's recency otherwise sees only `RouteGet`. A node without a master always evicts locally, and every mode still serves `EvictKey`. | | `UMBP_DRAM_READ_LEASE_MS` | `500` | ms | Peer-side DRAM/HBM read lease: how long a single `PeerDramAllocator::Resolve` protects its key's pages from concurrent local `Evict`, covering one RDMA read of those pages. Only needs to exceed one DRAM RDMA round trip (sub-ms), so 500 ms is ~100x margin. Read once at `PoolClient::Init`; `min_allowed=1`. | | `UMBP_SSD_READ_LEASE_MS` | `3000` | ms | Peer-side SSD read-staging slot lease: how long a claimed staging slot is reserved before the peer reclaims it by TTL (the fallback when the reader's best-effort `ReleaseSsdLease` is lost), and, echoed back in `PrepareSsdReadResponse.lease_ttl_ms`, the reader's validity window anchored at `t_send`. Must exceed one SSD read + RDMA (slower than DRAM), but too long pins a configured staging slot on a lost release. Also the fallback for the `PrepareSsdRead` RPC deadline when `UMBP_SSD_PREPARE_TIMEOUT_MS` is unset. Read once at `PoolClient::Init`; `min_allowed=1`. Also the read lease of a policy-declared `SsdBackend`, where it governs how long a contiguous staging span stays claimed and so sets that backend's read concurrency against a fixed arena — see `staging_slots`. | | `UMBP_RESOLVE_BUSY_TIMEOUT_MS` | `30000` | ms | Overall deadline for retrying a peer-local or remote batch resolve that returned `BUSY` because SSD staging was temporarily full. Retries discard the whole response and use exponential backoff capped at 50 ms. A batch whose own working set exceeds the arena returns a permanent failure immediately and is not retried. Values above 300000 ms are clamped. | diff --git a/src/umbp/include/umbp/distributed/master/in_memory_master_metadata_store.h b/src/umbp/include/umbp/distributed/master/in_memory_master_metadata_store.h index 95b7f2199..8298daf28 100644 --- a/src/umbp/include/umbp/distributed/master/in_memory_master_metadata_store.h +++ b/src/umbp/include/umbp/distributed/master/in_memory_master_metadata_store.h @@ -112,6 +112,8 @@ class InMemoryMasterMetadataStore : public IMasterMetadataStore { const std::vector& keys, const std::unordered_set& exclude_nodes, std::chrono::system_clock::time_point now, std::chrono::system_clock::duration lease_duration) override; + void RecordLocalReads(const std::vector& keys, + std::chrono::system_clock::time_point now) override; std::vector BatchExistsBlock(const std::vector& keys) const override; std::map> EnumerateEvictionCandidates( const std::vector& buckets, EvictionOrder order, size_t max_per_bucket, diff --git a/src/umbp/include/umbp/distributed/master/master_client.h b/src/umbp/include/umbp/distributed/master/master_client.h index 3de2379b2..f3c669c31 100644 --- a/src/umbp/include/umbp/distributed/master/master_client.h +++ b/src/umbp/include/umbp/distributed/master/master_client.h @@ -135,6 +135,14 @@ class MasterClient { const std::string& logical_tier, uint64_t bytes, uint32_t max_keys); + // Keys read locally, shipped (deduped, best effort) on the next heartbeat: + // the master's recency otherwise sees only RouteGet. For a node whose + // eviction follows that recency; a no-op until enabled and supported. + void EnableLocalReadReports(bool enabled) { local_read_reports_requested_ = enabled; } + bool ReportsLocalReads() const { + return local_read_reports_requested_ && supports_peer_driven_eviction_.load(); + } + void NoteLocalReads(const std::vector& keys); // --- Heartbeat (event-driven) --- // Bind every storage medium live on this peer. The heartbeat drains each @@ -260,6 +268,9 @@ class MasterClient { bool peer_driven_requested_ = false; // Rewritten on re-registration (heartbeat thread), read by eviction. std::atomic supports_peer_driven_eviction_{false}; + std::atomic local_read_reports_requested_{false}; + std::mutex local_reads_mutex_; + std::unordered_set local_reads_; // Initial conservative registration payload, retained so an UNKNOWN response // can rebuild the full peer record after Master state loss. std::map registration_capacities_; diff --git a/src/umbp/include/umbp/distributed/master/master_metadata_store.h b/src/umbp/include/umbp/distributed/master/master_metadata_store.h index 8ea1ff32d..ec80be17f 100644 --- a/src/umbp/include/umbp/distributed/master/master_metadata_store.h +++ b/src/umbp/include/umbp/distributed/master/master_metadata_store.h @@ -415,6 +415,12 @@ class IMasterMetadataStore { std::chrono::system_clock::time_point now, std::chrono::system_clock::duration lease_duration) = 0; + // Records an access (last_accessed_at = now, access_count + 1) for every key + // that still exists: reads a peer served locally (heartbeat local_reads), + // which RouteGet never sees. No lease. One round trip. + virtual void RecordLocalReads(const std::vector& keys, + std::chrono::system_clock::time_point now) = 0; + // Batched existence — pure read, no lease grant, no access record. // Used by BatchRoutePut for dedup (router.cpp:112): a writer // landing on an existing key isn't a "read" and must not extend diff --git a/src/umbp/include/umbp/distributed/master/redis/lua_scripts.h b/src/umbp/include/umbp/distributed/master/redis/lua_scripts.h index 07cd9d653..65139580f 100644 --- a/src/umbp/include/umbp/distributed/master/redis/lua_scripts.h +++ b/src/umbp/include/umbp/distributed/master/redis/lua_scripts.h @@ -266,6 +266,21 @@ end return out )LUA"; +// record_local_reads: +// KEYS[1..n] = block keys (single-slot per invocation, grouped by shard, as +// route_get_batch). ARGV = [now_ms]. +// An access bump without a lease for every key that still exists. +inline const std::string kRecordLocalReadsLua = R"LUA( +local now = ARGV[1] +for i = 1, #KEYS do + if redis.call('EXISTS', KEYS[i]) == 1 then + redis.call('HSET', KEYS[i], '_lacc', now) + redis.call('HINCRBY', KEYS[i], '_acnt', 1) + end +end +return 0 +)LUA"; + // exists_batch: // KEYS[1..n] = block keys (single-slot per invocation, grouped by shard by // the caller, same as route_get_batch). diff --git a/src/umbp/include/umbp/distributed/master/redis_master_metadata_store.h b/src/umbp/include/umbp/distributed/master/redis_master_metadata_store.h index c93a0922e..b651f563b 100644 --- a/src/umbp/include/umbp/distributed/master/redis_master_metadata_store.h +++ b/src/umbp/include/umbp/distributed/master/redis_master_metadata_store.h @@ -125,6 +125,8 @@ class RedisMasterMetadataStore : public IMasterMetadataStore { const std::vector& keys, const std::unordered_set& exclude_nodes, std::chrono::system_clock::time_point now, std::chrono::system_clock::duration lease_duration) override; + void RecordLocalReads(const std::vector& keys, + std::chrono::system_clock::time_point now) override; std::vector BatchExistsBlock(const std::vector& keys) const override; std::map> EnumerateEvictionCandidates( const std::vector& buckets, EvictionOrder order, size_t max_per_bucket, diff --git a/tests/cpp/umbp/distributed/test_peer_driven_eviction.cpp b/tests/cpp/umbp/distributed/test_peer_driven_eviction.cpp index 9a70c7568..a80da154f 100644 --- a/tests/cpp/umbp/distributed/test_peer_driven_eviction.cpp +++ b/tests/cpp/umbp/distributed/test_peer_driven_eviction.cpp @@ -118,10 +118,10 @@ TEST_F(PeerDrivenEvictionTest, MasterRankedNodeEvictsTheMastersVictims) { ASSERT_NE(node, nullptr); ASSERT_TRUE(node->Master().PeerDrivenEviction()); - // Half full, and known to the master, before crossing the watermark. - PutKeys(0, 8); + // Just under the watermark, and known to the master. + PutKeys(0, 14); std::vector known; - for (int i = 0; i < 8; ++i) known.push_back(Key(i)); + for (int i = 0; i < 14; ++i) known.push_back(Key(i)); node->Master().FlushHeartbeat(); ASSERT_TRUE(WaitFor( [&] { @@ -131,13 +131,31 @@ TEST_F(PeerDrivenEvictionTest, MasterRankedNodeEvictsTheMastersVictims) { }, std::chrono::seconds(5))); - PutKeys(8, 7); // 15 of 16 pages + // Local reads, which the master sees only through the heartbeat, must move + // these keys behind every other in its ranking. + const std::vector read = {Key(0), Key(1), Key(2)}; + std::vector buf(kPage); + for (const auto& key : read) ASSERT_TRUE(node->Get(key, buf.data(), buf.size())) << key; + ASSERT_TRUE(WaitFor( + [&] { + node->Master().FlushHeartbeat(); + auto ranked = node->Master().GetEvictionCandidates(TierType::DRAM, "", 14 * kPage, 0); + if (!ranked.has_value() || ranked->size() != 14) return false; + std::vector last; + for (size_t i = 11; i < 14; ++i) last.push_back((*ranked)[i].key); + std::sort(last.begin(), last.end()); + return last == read; + }, + std::chrono::seconds(5))) + << "local reads did not reach the master's ranking"; + + PutKeys(14, 1); // 15 of 16 pages ASSERT_TRUE( WaitFor([&] { return Dram()->OwnedKeyCount() <= kPages / 2; }, std::chrono::seconds(3))) << "the node did not evict for itself"; EXPECT_EQ(Dram()->OwnedKeyCount(), kPages / 2); EXPECT_EQ(node->LocalEvictionStats().master_keys, 7u) << "victims did not come from the master"; - for (int i = 8; i < 15; ++i) EXPECT_TRUE(Dram()->Contains(Key(i))) << i; + for (const auto& key : read) EXPECT_TRUE(Dram()->Contains(key)) << key; } TEST_F(PeerDrivenEvictionTest, EvictsLocallyUnlessTheModeIsMaster) { From 3935422db838dfb6902bc41c94f27fd90543d864 Mon Sep 17 00:00:00 2001 From: TianDi101 Date: Wed, 14 Oct 2026 05:53:59 +0000 Subject: [PATCH 17/19] fix(umbp): don't stall local eviction on the master's ranking master_ranked asked GetEvictionCandidates synchronously inside the eviction pass, at most every 200 ms per backend. The in-memory master answers by scanning its whole index; at e2e scale (500k DRAM + 2.7M SSD keys on one node) that is 0.66 s per DRAM query and 1.9 s per SSD query, so a DRAM pass stalled most of every second and every SSD query hit the 1 s deadline and was thrown away after the master had done the work. The pass now waits at most 20 ms for the query it sends (a small index still answers the same pass); a slower answer serves a later pass unless it is over 3 s old, and the medium's own order evicts meanwhile. One query per backend is in flight at a time. The deadline goes to 3 s to match. Co-Authored-By: Claude Opus 5.5 --- src/umbp/distributed/master/master_client.cpp | 5 +-- src/umbp/distributed/pool/peer_pool.cpp | 34 ++++++++++++++----- .../include/umbp/distributed/pool/peer_pool.h | 9 ++++- tests/cpp/umbp/distributed/test_peer_pool.cpp | 14 ++++++++ 4 files changed, 51 insertions(+), 11 deletions(-) diff --git a/src/umbp/distributed/master/master_client.cpp b/src/umbp/distributed/master/master_client.cpp index 2a52b2057..aa83899d1 100644 --- a/src/umbp/distributed/master/master_client.cpp +++ b/src/umbp/distributed/master/master_client.cpp @@ -459,8 +459,9 @@ std::optional> MasterClient::GetEvictionCandidates( ::umbp::GetEvictionCandidatesResponse resp; grpc::ClientContext ctx; - // Eviction falls back to the medium's own order rather than wait long. - ctx.set_deadline(std::chrono::system_clock::now() + std::chrono::seconds(1)); + // Eviction does not wait on this (PeerPool::RankedCandidates); past the + // age a peer still uses, an answer is wasted work. + ctx.set_deadline(std::chrono::system_clock::now() + std::chrono::seconds(3)); auto status = GetStub(stub_.get())->GetEvictionCandidates(&ctx, req, &resp); _rpc_timer.SetStatus(status); if (!status.ok()) { diff --git a/src/umbp/distributed/pool/peer_pool.cpp b/src/umbp/distributed/pool/peer_pool.cpp index 12b28c3ba..e2163166c 100644 --- a/src/umbp/distributed/pool/peer_pool.cpp +++ b/src/umbp/distributed/pool/peer_pool.cpp @@ -52,6 +52,10 @@ constexpr auto kLocalEvictPollInterval = std::chrono::milliseconds(100); constexpr auto kLocalEvictMinGap = std::chrono::milliseconds(1); // between passes // Per backend, between candidate-source queries. constexpr auto kRankedCandidatesMinInterval = std::chrono::milliseconds(200); +// How long a pass waits on the query it just sent (a small index answers in +// time); a slower answer serves a later pass, unless older than the max age. +constexpr auto kRankedCandidatesWait = std::chrono::milliseconds(20); +constexpr auto kRankedCandidatesMaxAge = std::chrono::seconds(3); uint64_t UsedBytes(const TierCapacity& capacity) { return capacity.total_bytes > capacity.available_bytes @@ -1247,14 +1251,28 @@ std::optional> PeerPool::RankedCandidates(uint32_t ba source = candidate_source_; } if (!source) return std::nullopt; - // Only the pass thread touches this (local_evict_pass_mutex_). A backend - // stuck over its watermark would otherwise query on every wake. - const auto now = std::chrono::steady_clock::now(); - auto& last = last_ranked_query_[backend_id]; - if (now - last < kRankedCandidatesMinInterval) return std::nullopt; - last = now; - const std::string logical = tier_graph_ == nullptr ? "" : tier_graph_->NameForBackend(backend_id); - auto offers = source(tier, logical, bytes); + // Only the pass thread touches this (local_evict_pass_mutex_). The master + // answers in time proportional to its index, so the pass never waits long: + // it evicts by the medium's order meanwhile. The interval keeps a backend + // stuck over its watermark from querying on every wake. + auto& query = ranked_queries_[backend_id]; + auto wait = std::chrono::milliseconds(0); + if (!query.answer.valid()) { + if (std::chrono::steady_clock::now() - query.started < kRankedCandidatesMinInterval) { + return std::nullopt; + } + query.started = std::chrono::steady_clock::now(); + const std::string logical = + tier_graph_ == nullptr ? "" : tier_graph_->NameForBackend(backend_id); + query.answer = std::async(std::launch::async, [source = std::move(source), tier, logical, + bytes] { return source(tier, logical, bytes); }); + wait = kRankedCandidatesWait; + } + if (query.answer.wait_for(wait) != std::future_status::ready) return std::nullopt; + auto offers = query.answer.get(); + if (std::chrono::steady_clock::now() - query.started > kRankedCandidatesMaxAge) { + return std::nullopt; // the index has moved on since + } // The source reports value sizes; the budget counts whole pages. auto* backend = backends_->Get(backend_id); const uint64_t page = backend == nullptr ? 0 : backend->PageSize(); diff --git a/src/umbp/include/umbp/distributed/pool/peer_pool.h b/src/umbp/include/umbp/distributed/pool/peer_pool.h index e5facb6f3..f69d1f7ac 100644 --- a/src/umbp/include/umbp/distributed/pool/peer_pool.h +++ b/src/umbp/include/umbp/distributed/pool/peer_pool.h @@ -30,6 +30,7 @@ #include #include #include +#include #include #include #include @@ -310,7 +311,13 @@ class PeerPool { std::atomic local_evict_stalled_{0}; std::atomic local_evict_master_keys_{0}; EvictionCandidateSource candidate_source_; // guarded by local_evict_mutex_ - std::map last_ranked_query_; // pass thread + // At most one source query in flight per backend; pass thread only. Last + // member, so a pending query is joined before anything else goes away. + struct RankedQuery { + std::chrono::steady_clock::time_point started{}; + std::future>> answer; + }; + std::map ranked_queries_; }; } // namespace mori::umbp diff --git a/tests/cpp/umbp/distributed/test_peer_pool.cpp b/tests/cpp/umbp/distributed/test_peer_pool.cpp index 51a8e694f..8882d0fce 100644 --- a/tests/cpp/umbp/distributed/test_peer_pool.cpp +++ b/tests/cpp/umbp/distributed/test_peer_pool.cpp @@ -1197,6 +1197,20 @@ TEST(PeerPoolLocalEviction, RankedCandidatesGoFirstAndTheMediumCoversTheRest) { for (const char* gone : {"k8", "k7", "k0", "k1"}) EXPECT_FALSE(dram->Contains(gone)) << gone; for (int i = 2; i <= 6; ++i) EXPECT_TRUE(dram->Contains("k" + std::to_string(i))) << i; EXPECT_EQ(pool.LocalEvictionStats().master_keys, 2u); + + // A source slower than a pass waits for (a master with a large index): the + // medium's order covers the overage meanwhile. + std::atomic answered{false}; + pool.SetEvictionCandidateSource([&](TierType, const std::string&, uint64_t) { + std::this_thread::sleep_for(std::chrono::milliseconds(500)); + answered = true; + return std::optional>({{"k2", 64}}); + }); + std::this_thread::sleep_for(std::chrono::milliseconds(250)); // past the query interval + for (int i = 9; i < 13; ++i) CommitKey(&pool, "k" + std::to_string(i)); + ASSERT_TRUE(WaitUntil([&] { return dram->OwnedKeyCount() <= 5; })); + EXPECT_FALSE(answered.load()) << "the pass waited on the source"; + EXPECT_EQ(pool.LocalEvictionStats().master_keys, 2u); } TEST(PeerPoolLocalEviction, KeysUnderAReadLeaseAreNeverOffered) { From e8b950a44a7787f608622fcd2c3683453a16f93f Mon Sep 17 00:00:00 2001 From: TianDi101 Date: Wed, 14 Oct 2026 07:07:26 +0000 Subject: [PATCH 18/19] fix(umbp): use the master's ranking for the whole eviction pass One pass reclaims a backend's whole watermark band: in the e2e a single DRAM pass covered 108k keys, while one GetEvictionCandidates answer carries at most 4096 and arrives after the pass has moved on. The pass asked once, at its start, so master_ranked ranked 0 of those 108k victims (at most 4% even when it used to wait for the answer). The pass now takes the master's answer whenever it is ready, and asks again for what the budget still needs as each answer runs out (still at most one query per 200 ms per backend); the medium's own order evicts in between. Co-Authored-By: Claude Opus 5.5 --- src/umbp/distributed/pool/peer_pool.cpp | 49 ++++++++++--------- .../include/umbp/distributed/pool/peer_pool.h | 3 +- 2 files changed, 29 insertions(+), 23 deletions(-) diff --git a/src/umbp/distributed/pool/peer_pool.cpp b/src/umbp/distributed/pool/peer_pool.cpp index e2163166c..9127f5ef6 100644 --- a/src/umbp/distributed/pool/peer_pool.cpp +++ b/src/umbp/distributed/pool/peer_pool.cpp @@ -52,8 +52,8 @@ constexpr auto kLocalEvictPollInterval = std::chrono::milliseconds(100); constexpr auto kLocalEvictMinGap = std::chrono::milliseconds(1); // between passes // Per backend, between candidate-source queries. constexpr auto kRankedCandidatesMinInterval = std::chrono::milliseconds(200); -// How long a pass waits on the query it just sent (a small index answers in -// time); a slower answer serves a later pass, unless older than the max age. +// How long a pass waits on its first query (a small index answers in time); a +// slower answer is taken when ready, unless older than the max age. constexpr auto kRankedCandidatesWait = std::chrono::milliseconds(20); constexpr auto kRankedCandidatesMaxAge = std::chrono::seconds(3); @@ -1197,21 +1197,26 @@ size_t PeerPool::ReclaimBackend(uint32_t backend_id, const LocalEvictionWatermar return progressed; }; - // The master's cluster-wide ranking first (at most one query per interval); - // the medium's own order covers whatever it leaves. - const auto ranked = RankedCandidates(backend_id, tier, budget); - if (ranked.has_value()) { - const size_t before = evicted_keys; - size_t next = 0; - while (covered < budget && next < ranked->size() && !stopping() && !drained()) { - evict_some(*ranked, &next); // stale entries just yield kNone - } - local_evict_master_keys_.fetch_add(evicted_keys - before, std::memory_order_relaxed); - } - + // The master's ranking whenever an answer is at hand, asked again for what + // the budget still needs as each one runs out; the medium's own order + // covers the rest meanwhile. A pass can span a whole band, far more keys + // than one answer carries. + auto ranked = RankedCandidates(backend_id, tier, budget, kRankedCandidatesWait); + size_t ranked_next = 0; + size_t master_keys = 0; size_t window = kLocalEvictBatch; while (covered < budget && !drained()) { if (stopping()) break; + if (!ranked.has_value() || ranked_next >= ranked->size()) { + ranked = RankedCandidates(backend_id, tier, budget - covered, std::chrono::milliseconds(0)); + ranked_next = 0; + } + if (ranked.has_value() && ranked_next < ranked->size()) { + const size_t before = evicted_keys; + evict_some(*ranked, &ranked_next); // stale entries just yield kNone + master_keys += evicted_keys - before; + continue; + } auto offers = backend->EvictionCandidates(window); const bool exhausted = offers.size() < window; offers.erase(std::remove_if(offers.begin(), offers.end(), @@ -1238,13 +1243,13 @@ size_t PeerPool::ReclaimBackend(uint32_t backend_id, const LocalEvictionWatermar break; } } + local_evict_master_keys_.fetch_add(master_keys, std::memory_order_relaxed); local_evict_keys_.fetch_add(evicted_keys, std::memory_order_relaxed); return evicted_keys; } -std::optional> PeerPool::RankedCandidates(uint32_t backend_id, - TierType tier, - uint64_t bytes) { +std::optional> PeerPool::RankedCandidates( + uint32_t backend_id, TierType tier, uint64_t bytes, std::chrono::milliseconds wait) { EvictionCandidateSource source; { std::lock_guard lock(local_evict_mutex_); @@ -1252,11 +1257,10 @@ std::optional> PeerPool::RankedCandidates(uint32_t ba } if (!source) return std::nullopt; // Only the pass thread touches this (local_evict_pass_mutex_). The master - // answers in time proportional to its index, so the pass never waits long: - // it evicts by the medium's order meanwhile. The interval keeps a backend - // stuck over its watermark from querying on every wake. + // answers in time proportional to its index, so the pass never waits long + // (`wait` only on a query it just sent): it evicts by the medium's order + // meanwhile. The interval keeps the master from being asked continuously. auto& query = ranked_queries_[backend_id]; - auto wait = std::chrono::milliseconds(0); if (!query.answer.valid()) { if (std::chrono::steady_clock::now() - query.started < kRankedCandidatesMinInterval) { return std::nullopt; @@ -1266,7 +1270,8 @@ std::optional> PeerPool::RankedCandidates(uint32_t ba tier_graph_ == nullptr ? "" : tier_graph_->NameForBackend(backend_id); query.answer = std::async(std::launch::async, [source = std::move(source), tier, logical, bytes] { return source(tier, logical, bytes); }); - wait = kRankedCandidatesWait; + } else { + wait = std::chrono::milliseconds(0); } if (query.answer.wait_for(wait) != std::future_status::ready) return std::nullopt; auto offers = query.answer.get(); diff --git a/src/umbp/include/umbp/distributed/pool/peer_pool.h b/src/umbp/include/umbp/distributed/pool/peer_pool.h index f69d1f7ac..689082484 100644 --- a/src/umbp/include/umbp/distributed/pool/peer_pool.h +++ b/src/umbp/include/umbp/distributed/pool/peer_pool.h @@ -204,7 +204,8 @@ class PeerPool { size_t LocalEvictionPass(); // lowest tier first; returns keys evicted size_t ReclaimBackend(uint32_t backend_id, const LocalEvictionWatermarks& watermarks); std::optional> RankedCandidates(uint32_t backend_id, TierType tier, - uint64_t bytes); + uint64_t bytes, + std::chrono::milliseconds wait); void TouchLocked(const std::string& key); void ForgetAccessLocked(const std::string& key); From 958b70d95b5ad793ecf3cffa0e63a749e4bf5776 Mon Sep 17 00:00:00 2001 From: TianDi101 Date: Thu, 15 Oct 2026 09:49:02 +0000 Subject: [PATCH 19/19] feat(umbp): count demotion retries; show retried and dropped in the summary With #683 in main, the activity summary can carry this PR's numbers: "(N moved, M failed, R retried, K dropped)". retried counts eviction demotions re-queued after a retryable failure, so a run shows whether the retry path is doing anything; it is also exported as mori_umbp_client_tier_transitions_total{outcome="retried"}. Co-Authored-By: Claude Opus 5.5 --- .../distributed/metrics/activity_summary.cpp | 20 ++++++++++++------- src/umbp/distributed/pool/peer_pool.cpp | 1 + src/umbp/distributed/pool_client.cpp | 2 ++ src/umbp/include/umbp/distributed/types.h | 1 + .../distributed/test_activity_summary.cpp | 4 +++- tests/cpp/umbp/distributed/test_peer_pool.cpp | 1 + 6 files changed, 21 insertions(+), 8 deletions(-) diff --git a/src/umbp/distributed/metrics/activity_summary.cpp b/src/umbp/distributed/metrics/activity_summary.cpp index 4bc44bfe2..746b52111 100644 --- a/src/umbp/distributed/metrics/activity_summary.cpp +++ b/src/umbp/distributed/metrics/activity_summary.cpp @@ -133,6 +133,8 @@ std::vector ActivitySummary::Render(const ActivitySnapshot& now, Delta(now.transitions.offloaded_bytes, before.transitions.offloaded_bytes); const uint64_t moves_ok = Delta(now.transitions.succeeded, before.transitions.succeeded); const uint64_t moves_failed = Delta(now.transitions.failed, before.transitions.failed); + const uint64_t moves_retried = Delta(now.transitions.retried, before.transitions.retried); + const uint64_t moves_dropped = Delta(now.transitions.dropped, before.transitions.dropped); const ReCacheActivity& rc_now = now.recache; const ReCacheActivity& rc_old = before.recache; @@ -157,8 +159,9 @@ std::vector ActivitySummary::Render(const ActivitySnapshot& now, std::snprintf(window.data(), window.size(), "%.1fs", window_seconds); const bool anything = Moved(total) || promoted != 0 || demoted != 0 || moves_failed != 0 || - rc_admitted != 0 || rc_dropped != 0 || rc_prefetched != 0 || - rc_fragmented != 0 || rc_already != 0; + moves_retried != 0 || moves_dropped != 0 || rc_admitted != 0 || + rc_dropped != 0 || rc_prefetched != 0 || rc_fragmented != 0 || + rc_already != 0; std::vector lines; @@ -205,11 +208,14 @@ std::vector ActivitySummary::Render(const ActivitySnapshot& now, // they are printed as a pair even when one of them is zero: "promote 0 B" // next to a large demote is the shape of a tier graph that only sheds. head += " · promote " + FormatBytes(promoted) + " · demote " + FormatBytes(demoted); - if (moves_ok != 0 || moves_failed != 0) { - std::array moves{}; - std::snprintf(moves.data(), moves.size(), " (%llu moved, %llu failed)", - static_cast(moves_ok), - static_cast(moves_failed)); + if (moves_ok != 0 || moves_failed != 0 || moves_retried != 0 || moves_dropped != 0) { + // dropped = evicted copies lost for good; retried = demotions re-queued. + std::array moves{}; + std::snprintf( + moves.data(), moves.size(), " (%llu moved, %llu failed, %llu retried, %llu dropped)", + static_cast(moves_ok), static_cast(moves_failed), + static_cast(moves_retried), + static_cast(moves_dropped)); head += moves.data(); } lines.push_back(std::move(head)); diff --git a/src/umbp/distributed/pool/peer_pool.cpp b/src/umbp/distributed/pool/peer_pool.cpp index 4c0fa1a84..b8eb8ec98 100644 --- a/src/umbp/distributed/pool/peer_pool.cpp +++ b/src/umbp/distributed/pool/peer_pool.cpp @@ -356,6 +356,7 @@ void PeerPool::TransitionWorkerLoop() { evict_retry = job.evict && !result.success && result.retryable && std::chrono::steady_clock::now() - job.first_attempt < evict_demotion_retry_budget_; + if (evict_retry) ++transition_metrics_.retried; if (job.evict && !result.success && !evict_retry) DropEvictedCopyLocked(job); const bool still_pressured = watermark_driven && tier_graph_->AtOrAbove(job.source_tier, tier.low_watermark); diff --git a/src/umbp/distributed/pool_client.cpp b/src/umbp/distributed/pool_client.cpp index fe943a16c..5605fb010 100644 --- a/src/umbp/distributed/pool_client.cpp +++ b/src/umbp/distributed/pool_client.cpp @@ -5173,6 +5173,8 @@ void PoolClient::PublishComponentMetrics() { {{"outcome", "failed"}}, tiers.failed); sample(MORI_UMBP_METRIC_CLIENT_TIER_TRANSITIONS, MORI_UMBP_METRIC_CLIENT_TIER_TRANSITIONS_HELP, {{"outcome", "dropped"}}, tiers.dropped); + sample(MORI_UMBP_METRIC_CLIENT_TIER_TRANSITIONS, MORI_UMBP_METRIC_CLIENT_TIER_TRANSITIONS_HELP, + {{"outcome", "retried"}}, tiers.retried); sample(MORI_UMBP_METRIC_CLIENT_TIER_OFFLOADED_BYTES, MORI_UMBP_METRIC_CLIENT_TIER_OFFLOADED_BYTES_HELP, {}, tiers.offloaded_bytes); sample(MORI_UMBP_METRIC_CLIENT_TIER_PROMOTED_BYTES, diff --git a/src/umbp/include/umbp/distributed/types.h b/src/umbp/include/umbp/distributed/types.h index dd8533752..b76991b2c 100644 --- a/src/umbp/include/umbp/distributed/types.h +++ b/src/umbp/include/umbp/distributed/types.h @@ -81,6 +81,7 @@ struct TierTransitionMetrics { uint64_t succeeded = 0; uint64_t failed = 0; uint64_t dropped = 0; // evicted copies whose demotion never landed (lost keys) + uint64_t retried = 0; // eviction demotions re-queued after a retryable failure uint64_t offloaded_bytes = 0; uint64_t promoted_bytes = 0; }; diff --git a/tests/cpp/umbp/distributed/test_activity_summary.cpp b/tests/cpp/umbp/distributed/test_activity_summary.cpp index 2d701a189..420759c6d 100644 --- a/tests/cpp/umbp/distributed/test_activity_summary.cpp +++ b/tests/cpp/umbp/distributed/test_activity_summary.cpp @@ -135,10 +135,12 @@ TEST(ActivitySummary, ReportsPromotionAndDemotionAsAPairEvenWhenOneIsZero) { second.transitions.offloaded_bytes = 8ULL * 1024 * 1024; second.transitions.succeeded = 4; second.transitions.failed = 1; + second.transitions.retried = 3; + second.transitions.dropped = 2; const std::string all = Joined(summary.Render(second, 60.0)); EXPECT_NE(all.find("promote 0 B"), std::string::npos) << all; EXPECT_NE(all.find("demote 8.00 MiB"), std::string::npos) << all; - EXPECT_NE(all.find("1 failed"), std::string::npos) << all; + EXPECT_NE(all.find("1 failed, 3 retried, 2 dropped"), std::string::npos) << all; } TEST(ActivitySummary, SurfacesLoadMissesWhichBytesAloneHide) { diff --git a/tests/cpp/umbp/distributed/test_peer_pool.cpp b/tests/cpp/umbp/distributed/test_peer_pool.cpp index 16f5d1881..04f670015 100644 --- a/tests/cpp/umbp/distributed/test_peer_pool.cpp +++ b/tests/cpp/umbp/distributed/test_peer_pool.cpp @@ -880,6 +880,7 @@ TEST(PeerPool, AFailedDemotionIsRetriedUntilItsBudgetRunsOut) { EXPECT_TRUE(registry.Get("cold")->Contains("a")) << "dropped, not demoted"; EXPECT_EQ(pool.TransitionMetrics().dropped, 0u); EXPECT_GE(pool.TransitionMetrics().failed, 2u) << "the failed attempts still count"; + EXPECT_GE(pool.TransitionMetrics().retried, 2u); pool.SetEvictDemotionRetryBudgetForTest(std::chrono::milliseconds{50}); engine.FailNext(1000000); // every copy fails