[TianDi101]

An SSD tier needed a hand-tuned staging_slots, or local loads failed. This makes the defaults work.

Stacked on #762. The changes are 0000dd8c7 and b6ad087fd; the rest of the diff is #762. I'll retarget to main once #762 lands.

Problem

An SsdBackend's staging arena is staging_slots × page_size, and an omitted staging_slots fell back to 16 pages. That is 1 MiB at the 64 KiB page a masterless node picks, or about one DSv4-Pro object (0.55 MiB) shared by 8 ranks. Any object over 1 MiB could never be read. The fallback comes from the old ssd_staging_buffer_slots = 16, which used to divide a 256 MiB buffer into slots. ssd_staging_buffer_size no longer sizes anything, so the 16 now means pages.

A starved arena surfaces as BUSY resolves. After UMBP_RESOLVE_BUSY_TIMEOUT_MS (30 s) that is a failed load, and sglang's linker raises on any failed load after a hit, which kills the scheduler. Our e2e policy has carried "staging_slots": 65536 (4 GiB) for that reason.

Even at 4 GiB, a full arena was handled badly: when a later key could not get a span, the call rolled back every reservation it already had and the caller retried the whole batch. The DSv4 e2e on #762 logged 1,752 of these, up to 741 reservations rolled back at once.

Change

• Default arena from the tier. With no staging_slots, the arena is 1/64 of the SSD tier's capacity, clamped to [64 MiB, 4 GiB] (SsdStagingPages). A 1.5 TB or 600 GB tier gets the 4 GiB the e2e was validated with, and a test's small tier gets 64 MiB. An explicit staging_slots, or UMBP_DISTRIBUTED_SSD_STAGING_BUFFER_SLOTS, still wins. 0 now means "size it from the tier", where the standalone server used to reject it.

• A full arena defers instead of rolling back, for local readers. A local call that already served something keeps its reservations and returns the rest as kDeferred. Since fix(umbp): SSD staging no longer fails local loads; a local read holds its bytes until it is done #762, the caller's next round reads those keys after this round's copies release their staging. kBusy now only means the call got nothing. Remote readers keep the old all-or-nothing rollback, because a deferral is a failure on the wire.

• Reads leave the last eighth of the arena to writes (b6ad087fd). SSD writes stage through the same arena, and a demotion that cannot stage is retried and then dropped (fix(umbp): retry eviction demotions instead of dropping the only copy; count what is dropped #763). In the first e2e below, loads of 17–21 GiB/s from SSD kept the arena full. 18,169 DRAM→SSD demotions got kFailedNoSpace from BatchAllocate, and 1,628 used up their 30 s retry budget and were dropped, with the SSD tier only 74–84% full. Readers (resolves and migration reads) now stop at seven eighths, while writes may use all of it. The largest object either side accepts is that seven eighths, so nothing is written that cannot be read back. Arenas under 8 pages keep no reserve.

Test plan

• UMBP C++ suite on the e2e branch (main + fix(umbp): SSD staging no longer fails local loads; a local read holds its bytes until it is done #762 + fix(umbp): retry eviction demotions instead of dropping the only copy; count what is dropped #763 + feat(umbp): serve metrics from a masterless node, and summarise offload/load/evict/promote/demote at INFO #683 + perf(umbp): build LocalCopyEngine transfer plans incrementally #765 + this): 55/55. umbp_pool_client_ranges (RDMA device contention) and test_prometheus_metric_sink (port race) fail under ctest -j 8 and pass serially.
• SsdBackendTest.ArenaFullMidBatchRollsBackRemoteAndDefersLocal: same setup, remote gets BUSY and keeps nothing; local keeps its span and defers the rest. ABusyResolveDropsTheHoldersItTook now uses a remote reader, since a local one defers.
• BackendPolicyConfig.LowersSsdStagingSlotsAndDefaultsWhenAbsent checks SsdStagingPages at 1 GiB, 128 GiB and 1.5 TB tiers, and 64 KiB and 2 MiB pages.
• Standalone server with a policy giving only capacities and paths: staging_pages=4768 arena_bytes=312475648 for a 20 GB tier (it was staging_pages=16 arena_bytes=1048576).
• SsdBackendTest.ReadsLeaveTheLastEighthToWrites: with an 8-page arena, reads hold 7 pages, the next read is BUSY, and a write still allocates.
• DSv4-Pro agentic e2e on n06-21 (test/umbp-evict-e2e-v3 @ 0bc1b7f7e, i.e. 0000dd8c7 only). DRAM 100 GB + SSD 600 GB, no UMBP tuning (the policy gives only capacities and paths), 1 h at conc 256.
  • Passed: 7,529 requests, 0 errors, 215,306 input tok/s, 0 failed loads, 0 [error] lines.
  • The SSD tier served 78% of loaded bytes (24.5 TiB in the hour) and evicted 445 GiB in two watermark passes.
  • 4.88 M resolves were deferred. All 452 staging arena busy events rolled back 0 reservations (the 300 GB run on fix(umbp): SSD staging no longer fails local loads; a local read holds its bytes until it is done #762 alone logged 1,752 that rolled back up to 741).
  • 1,628 demotions were dropped, which is what b6ad087fd fixes.
• The same e2e with b6ad087fd (test/umbp-evict-e2e-v3 @ 8613bddbe).
  • Passed: 7,690 requests, 0 errors, 223,843 input tok/s, 0 failed loads, 0 [error] lines.
  • 0 dropped demotions, against 1,628 without the reserve. Demotion failures (all retried) fell from 18,169 to 212, of 1.93 M.
  • The SSD tier served 78.6% of loaded bytes (25.7 TiB) and evicted 557 GiB.
  • All 1,124 staging arena busy events rolled back 0 reservations. There are more of them because readers now stop at seven eighths, but there were 0 BUSY timeouts.

🤖 Generated with Claude Code

[TianDi101]

Superseded by #776. Its local-reader deferral is carried over there. The 1/64-of-tier sizing is replaced by a fixed 4 GiB preset capped at the tier. The 7/8 reader cap is dropped, because demotions and promotions no longer use the staging arena.

🤖 Generated with Claude Code