[TianDi101]

Stacked on #741 (feat/umbp-evict-converge). Review 3df1165a9, acc4b97ee, 466570e5b and 3c6dd4771; 2c0f8349f only merges #741's doc and test cleanup in. This PR retargets to main once #741 merges. (acc4b97ee is titled WIP; 466570e5b completes it. Squash on merge.)

Summary

Two changes to how eviction work is sized and executed:

1. Master rounds are sized in bytes, not rows. The round was capped at evict_batch_size = 32 candidates per over-full (node, tier) every 5 s. A round now frees the whole overage (usage above the low watermark), optionally throttled by UMBP_EVICT_MAX_BYTES_PER_ROUND.
2. Demotion is asynchronous on the peer. PeerPool::Evict no longer copies a victim downstream inside the call. It queues the demotion to the pool's transition worker and returns, so an EvictKey RPC never blocks on data movement and needs no per-RPC byte limit.

Why

• 32 rows per 5 s was too slow to matter. That is ~6 keys/s per tier. At the ~0.7 MB keys of the deployment eviction_manager.cpp cites (512 GiB, 155,157 keys per 0.9→0.7 band), that is ~4 MB/s, and draining one band took ~6.7 hours while puts hit NO_SPACE.
• The cap saved nothing. Both metadata stores read every eligible row of a bucket before capping, then sort and truncate in C++. Each round paid a full scan to free 32 keys.
• Nothing could tune it. No flag, env var or config path set evict_batch_size. The cap only became effective in refactor(umbp): make distributed mode backend- and transport-agnostic #540; before that, rounds were uncapped.
• Sized rounds plus synchronous demotion would stall the RPC. With the row cap gone, one EvictKey can name gigabytes on a demoting tier. Copying all of it inside the call blows UMBP_EVICTKEY_DEADLINE_MS (1 s). Queuing the copy keeps the RPC short without capping it in bytes.

Changes

Master

• EvictionConfig::evict_batch_size → max_evict_bytes_per_round (env UMBP_EVICT_MAX_BYTES_PER_ROUND, default 0 = whole overage). The overage is already at most (1 − low_watermark) of the tier; set this to throttle.
• RunOnce asks the store for every eligible row again (max_per_bucket = 0), and the strategy trims to the byte budget. EvictKey RPCs are split by gRPC message size only, as before.
• GetEnvUint64 added to env_time.h. runtime-env-vars.md documents the variable.

Peer

• PeerPool::Evict returns one PoolEvictResult {key, bytes_freed, outcome} per request. PoolEvictOutcome is one of:
  • kFreed: the copy was dropped.
  • kDemotionQueued: a demotion job was enqueued. bytes_freed is 0, because the source is freed only when the copy lands.
  • kInFlight: a transition already owns the key.
  • kNone: nothing freed (leased, pinned, or gone).
• Eviction demotions are TransitionJobs with evict = true. The worker runs them regardless of the tier's own watermark. If nothing downstream has room, it drops the evicted copy instead, unless another transition owns the key.
• Local (masterless) reclaim counts queued and in-flight demotions as covered bytes, so a pass does not queue more than the overage while copies run.
• A key mid-demotion holds a migration pin. PageBackend::EvictionCandidates still offers it, so the reclaimer can count it as in flight. It still skips SSD copy-through pins, and backend Evict still refuses every pinned key.

Does the master get slower? No added lock time. Both stores apply the cap after the scan, outside their locks. Removing it changes only post-scan work on the eviction thread: a full sort instead of a partial one, and more EvictKey RPCs. It also cuts scans, since a drain is one round instead of one round per 32 keys. The scan's own cost is unchanged and is the follow-up below.

Test plan

• Master: a round frees the whole overage (77 keys, not 32); max_evict_bytes_per_round throttles a round to 10 keys; a 600-key budget is met in one strategy pass; victims keep their tier; the env var is read with the default when unset; GetEnvUint64 parses values above 4 GiB and rejects negatives and garbage.
• Peer: EvictReturnsBeforeTheDemotionCopyRuns holds the copy behind a gated transfer engine and checks that Evict returns kDemotionQueued first. QueuedDemotionsCountTowardTheBudget checks that a pass queues exactly the overage (4 keys) while copies are held. The DRAM→SSD chain and peer-service tests now wait for the asynchronous move.
• Full tests/cpp/umbp suite in rocm/mori-dev with RDMA: 50/52 under ctest -j 8. umbp_cross_node_smoke and umbp_pool_client_ranges abort on the RDMA device-open assertion under parallel load. Serially, umbp_cross_node_smoke passes. umbp_pool_client_ranges hits its GPU read-back flakes (PoolClientNuma.*), which also fail on unmodified main: an interleaved A/B gave main 2/4 failures and this branch 3/4.
• Not measured: eviction throughput or master scan time at scale (1M–10M keys).

Follow-ups (not in this PR)

• Eviction efficiency: a master-maintained per-(node, tier) LRU, updated off the lookup path with deferred touches, so a round walks the cold end instead of scanning every key. Then trigger rounds from heartbeats instead of only the 5 s timer.
• Dispatch to nodes in parallel; rounds are still sequential across nodes.

🤖 Generated with Claude Code

[TianDi101]

Combined into #753 (with #741, #683 and the fixes from the DRAM→SSD end-to-end run). The WIP commit and its completion are squashed there. Leaving this open until #753 lands.

[TianDi101]

Folded into #741: this PR's change is d0faaa820 there, and f19c52b48 tightens its comments and folds its tests (same coverage). #741 is rebased on main. Closing this PR; please review the change in #741.