Repository navigation
fix(umbp): stop false put-failure noise (sampled allocation WARN; in-flight key counts as stored) - #777
Merged
Conversation
AllocateLocked logged one WARN per failed key, under the backend's exclusive lock. When a DRAM tier fills, every put and every promotion lands there: in the e2e of #762/#763/#774/#775/#776 (DRAM 300 GB + SSD 600 GB, promote on_read) a 2.5-minute DRAM-full burst logged 640,868 of them, holding the lock that reads, commits and eviction need while the tier was trying to recover. Log the first 16 and then every 10,000th, with the running count. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lure When two writers put the same key at once, PeerPool refuses the second because the first holds the reservation. Local puts reported that as a failure, although the first writer stores the same bytes. sglang's linker then logged "UMBP offload failed" and dropped the rest of its offload batch. PeerPool now flags that refusal as in_flight (still kFailed for other callers, so nothing reads uncommitted bytes). Local puts treat it like "already exists". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TianDi101
added a commit
that referenced
this pull request
Oct 15, 2026
3 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two fixes for put failures that look like errors but aren't, both found in the e2e of #762/#763/#774/#775/#776 (DRAM 300 GB + SSD 600 GB).
1. Sample
PageBackend's allocation-failure WARNPageBackend::AllocateLockedlogged one WARN per failed key, under the backend's exclusive lock. When a DRAM tier fills, every put and every promotion fails there.Seen in:
test/umbp-e2e-v4,promote_trigger: on_read.Allocate reason=NO_SPACElines, while the tier needed that same lock to serve reads, commit and evict.Change: log the first 16 failures and then every 10,000th, with the running count. The counter is guarded by the lock the function already holds.
2. A put whose key is already being stored is not a failure
When two writers put the same key at once,
PeerPoolrefuses the second because the first holds the reservation. Local puts reported that as a failure, although the first writer stores the same bytes.Seen in:
test/umbp-e2e-v5, during a read burst. Four DP ranks offloaded the same shared-prefix pages together, and sglang's linker loggedUMBP offload failed: ... success=1/31.false, so a harmless race cost other pages their offload.Change:
PeerPoolflags that refusal asin_flight. Its outcome stayskFailed, so other callers (remote writers in master mode) are unchanged and nothing reads uncommitted bytes. The three local put paths treatin_flightlike "already exists". No sglang change: it only logs when UMBP returnsfalse.Testing
test_page_backend,umbp_peer_pool,test_ssd_backend,umbp_medium_selectionandumbp_pool_client_batch_putpass. No test asserts on the WARN line.PeerPool.ReservesDuplicateKeysUntilCommitOrAbort, andBatchPutWarnTest.KeyAlreadyBeingStoredIsNotAFailure(same key twice in one batch). The unit suite run is pending; a free node opens when the current e2e ends.🤖 Generated with Claude Code