Skip to content

fix(umbp): stop false put-failure noise (sampled allocation WARN; in-flight key counts as stored) - #777

Merged
TianDi101 merged 2 commits into
mainfrom
fix/umbp-page-alloc-warn-ratelimit
Oct 16, 2026
Merged

TianDi101 merged 2 commits into
mainfrom
fix/umbp-page-alloc-warn-ratelimit

Conversation

@TianDi101

@TianDi101 TianDi101 commented Oct 15, 2026 •

Copy link
Copy Markdown
Collaborator

Two fixes for put failures that look like errors but aren't, both found in the e2e of #762/#763/#774/#775/#776 (DRAM 300 GB + SSD 600 GB).

1. Sample PageBackend's allocation-failure WARN

PageBackend::AllocateLocked logged one WARN per failed key, under the backend's exclusive lock. When a DRAM tier fills, every put and every promotion fails there.

Seen in: test/umbp-e2e-v4, promote_trigger: on_read.

Change: log the first 16 failures and then every 10,000th, with the running count. The counter is guarded by the lock the function already holds.

2. A put whose key is already being stored is not a failure

When two writers put the same key at once, PeerPool refuses the second because the first holds the reservation. Local puts reported that as a failure, although the first writer stores the same bytes.

Seen in: test/umbp-e2e-v5, during a read burst. Four DP ranks offloaded the same shared-prefix pages together, and sglang's linker logged UMBP offload failed: ... success=1/31.

  • The linker also stops the rest of that offload batch on any false, so a harmless race cost other pages their offload.
  • Nothing failed in UMBP: every allocation committed, with 0 aborts and 0 failed transfers.
  • It predates this chain: 10 of 33 earlier runs show it, 8–28 times per 2 h run.

Change: PeerPool flags that refusal as in_flight. Its outcome stays kFailed, so other callers (remote writers in master mode) are unchanged and nothing reads uncommitted bytes. The three local put paths treat in_flight like "already exists". No sglang change: it only logs when UMBP returns false.

Testing

  • Part 1: test_page_backend, umbp_peer_pool, test_ssd_backend, umbp_medium_selection and umbp_pool_client_batch_put pass. No test asserts on the WARN line.
  • Part 2: new assertions in PeerPool.ReservesDuplicateKeysUntilCommitOrAbort, and BatchPutWarnTest.KeyAlreadyBeingStoredIsNotAFailure (same key twice in one batch). The unit suite run is pending; a free node opens when the current e2e ends.

🤖 Generated with Claude Code

TianDi101 and others added 2 commits October 15, 2026 11:07
AllocateLocked logged one WARN per failed key, under the backend's
exclusive lock.  When a DRAM tier fills, every put and every promotion
lands there: in the e2e of #762/#763/#774/#775/#776 (DRAM 300 GB + SSD
600 GB, promote on_read) a 2.5-minute DRAM-full burst logged 640,868 of
them, holding the lock that reads, commits and eviction need while the
tier was trying to recover.

Log the first 16 and then every 10,000th, with the running count.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lure

When two writers put the same key at once, PeerPool refuses the second
because the first holds the reservation. Local puts reported that as a
failure, although the first writer stores the same bytes. sglang's linker
then logged "UMBP offload failed" and dropped the rest of its offload batch.

PeerPool now flags that refusal as in_flight (still kFailed for other
callers, so nothing reads uncommitted bytes). Local puts treat it like
"already exists".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@TianDi101 TianDi101 changed the title fix(umbp): sample PageBackend's allocation-failure WARN fix(umbp): stop false put-failure noise (sampled allocation WARN; in-flight key counts as stored) Oct 15, 2026
@TianDi101
TianDi101 merged commit e0a5e13 into main Oct 16, 2026
16 of 17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant