misc: update doc by zhyncs · Pull Request #715 · sgl-project/sglang

zhyncs · 2024-07-24T17:43:31Z

Thank you for your contribution, we really appreciate it. The following instructions will help improve your pull request and make it easier to receive feedback. If there are any items you don't understand, don't worry. Just submit the pull request and ask the maintainers for help.

Motivation

Please explain the motivation behind this PR and the goal you aim to achieve with it.

Modification

Briefly describe the changes made in this PR.

Checklist

Ensure pre-commit pre-commit run --all-files or other linting tools are used to fix potential lint issues.
Confirm that modifications are covered by complete unit tests. If not, please add more unit tests for correctness.
Modify documentation as needed, such as docstrings or example tutorials.

## Motivation  ## Modifications  ## Accuracy Tests  ## Benchmarking and Profiling  ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.ai/developer_guide/contribution_guide.html#format-code-with-pre-commit). - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.ai/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [x] Update documentation according to [Write documentations](https://docs.sglang.ai/developer_guide/contribution_guide.html#write-documentations). - [x] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.ai/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.ai/developer_guide/contribution_guide.html#benchmark-the-speed).

… explain Measured 2026-08-17 02:18:09, all three ranks, inside the OOM this reporter (sgl-project#695) was written to diagnose: RADIX SHAPE: walk failed after 3 nodes (RuntimeError('Boolean value of Tensor with more than one value is ambiguous')). Partial: tokens=1, locked_nodes=1. Cause, and it is mine: `len(getattr(node, "value", ()) or ())`. node.value is a TENSOR, so `tensor or ()` evaluates bool(tensor), which raises for any tensor with more than one element. The walk therefore survived only empty or single-element nodes -- an empty tree -- and fell over on the first real one. It has presumably never worked on a populated tree since it was written. A diagnostic that only works when there is nothing to diagnose is not a diagnostic, and this one failed at precisely the moment its output was the evidence needed: the tree shape at the crash would have shown whether the 147,456 tokens the counter certified were reachable from the unlocked leaf frontier the actuator walks. That question is still open BECAUSE of this bug. Length is now asked for directly and never via truthiness. Tests, hermetic (CUDA_VISIBLE_DEVICES=""): test_multi_element_tensor_value_does_not_break_the_walk .. the specimen shape test_counts_tokens_across_a_deeper_tree ................. 100+30+7 counted test_none_and_empty_values_still_count_zero ............. CAN-FAIL boundary: the fix must not crash on a missing value nor count None as a token test_a_locked_multi_element_node_is_still_counted_as_locked -> 4 passed + 2 subtests; 16 passed across the 715 + 695 + 694 mem_cache suites. ruff clean. CAN-FAIL PROVEN BY MUTATION: restoring the original expression fails 5 of 6; reverted and re-verified 4 passed. The suite would have caught the bug on the day it was written.

…nothing closed it Desk analysis of the 3x FLIP ABANDONED on 0b61699, plus the guard that makes the real state legible. 1. THE 1748-vs-1693 ARITHMETIC, reconstructed. phase_flip_runtime.py:4993-5019: usable = from_driver = max(0, driver_free - reserve) spendable = usable # allocator cache is NOT counted So "spendable" is driver-free minus the kept-free reserve, measured AFTER _reclaim_cached_blocks() has returned what it can; whatever the caching allocator still holds is deliberately excluded. The 1748 requirement comes from _staging_bytes, which is incoming + max(outgoing, local) over the WIDEST WAVE, using the same row_nbytes the move itself uses -- derived from the plan, not estimated, and waved so it scales with pool geometry rather than prompt length. I find no overcount on that side. 2. THE RUNG LINE IS ARITHMETICALLY IMPOSSIBLE, and that is the root. _floor_rows is max_live + 1 + margin_rows + admission_reserve_rows. margin_rows DEFAULTS TO 0 and is never passed at the construction site (kv_backing_relief.py:2158-2167), and admission_reserve_rows is chunked_prefill_size = 512. So floor = max_live + 513, and floor=398471 means max_live = 397,958 -- against a current cap of 137,216. A live row id 2.9x ABOVE the cap. Compare the healthy shape this module documents itself (line 875): "max_live=644 + admission reserve 512, slack=405894". There the high-water id is tiny. Here it outlived the pool it was measured in: ids from a larger id space surviving a reshard/shrink. Because slack is max(0, current - floor_rows), it pins to 0 for as long as that holds, so the rung can NEVER propose a shrink. The evict-rung funding path (sgl-project#688) is therefore permanently unavailable at this operating point, and every flip falls back on the raw seam fund alone. That is why the instance abandoned three times over 55 MiB instead of funding it from KV once -- the backstop was gone, not merely small. So: the floor FORMULA is right and its INPUT is impossible. The defect is upstream of this file, in whatever leaves live ids above the cap. 3. WHAT LANDS HERE. Only the guard: a floor above the cap now says FLOOR UNREACHABLE, names the gap in rows, and states the implication (this rung can never fund; max_live is above the cap). "slack=0" alone is indistinguishable from a rung that merely had no room this round, which is exactly the confusion this ticket started in. The healthy path is untouched and a floor exactly AT the cap is reported as tight, not impossible. I did NOT change the floor, the eviction handoff, or the live-set derivation. Re-basing live ids after a shrink/reshard is the actual fix and it sits in the area F4-r4 holds for sgl-project#715, so per the coordination rule it goes back through the operator rather than being edited here. Tests: 5, red first -- two new assertions failed, three controls (healthy shape, boundary at floor == cap, never-ran) passed before the change and still do. managers + mem_cache: 2892 passed, 0 failed. ruff clean.

…ts is unreachable Pre-approved review amendment: _post_evict_rows() was the one undefended term inside the diagnostic's logger call, and a raise there would kill the scheduler round the line exists to observe -- the sgl-project#715 failure mode exactly. It now uses the same try/except-RAISED pattern as the other two probes. BUT THE MUTATION SAYS THE CONCERN IS UNREACHABLE, AND I WILL NOT DRESS THAT UP. My first test for it broke the accessors underneath _post_evict_rows and passed 10/10 against a deliberately UNDEFENDED call site -- a vacuous test, caught only because I mutation-check every can-fail. The reason: _post_evict_rows swallows its own accessor exceptions internally, so it cannot raise. Forcing a raise means patching the bound method, which then explodes inside _layout_admits (also a bare call) before the diagnostic is ever reached -- testing neither. So: the hardening stays, as cheap insurance against a future edit that makes _post_evict_rows raise, and because a uniform pattern is worth more than a justified exception. It is NOT claimed to fix a reachable defect today, the unreachable arm is removed rather than left as a passing test that proves nothing, and the limit is written into the test class docstring so the next reader does not re-derive it. The mamba probe IS reachable and remains tested: a raising accessor must be NAMED as "RAISED <type>", never silently read as 0, because zero-by-exception and zero-by-measurement are different states and only one is a defect. Tests, hermetic: 10 passed + 1 subtest; and the 677/689/708 adjacency suites pass alongside. ruff clean.

…d their honest cost Slice 1 verified a checkpoint's references and refused to branch when any had been evicted. Correct failure, still a failure: the store is an LRU cache and a conversation checkpoint is not cache-shaped -- its whole value is staying restorable after its pages stop being hot. So a checkpoint now PINS what it references, and the cost of that promise is accounted rather than hidden. WHAT THE PIN DOES * Eviction SKIPS pinned entries, using the same skip-and-repin step in-flight writes already get. Deliberately the same: it reuses the bounded skip budget in _evict_while, so a store whose entries are ALL pinned exhausts its attempts and the caller learns the space is not there, instead of spinning. * The TTL sweeper skips a pinned page's .part706 as well, because a checkpoint may reference a page another stage is still assembling and age alone cannot tell that from abandonment -- a pin can. * Pins are REF-COUNTED by checkpoint. Branching exists to SHARE a prefix, so unpinning one holder must not strip its sibling, and a branch that shares its parent's whole prefix costs ZERO new pinned bytes (tested). * Pins are DURABLE ({store}/pins706/{checkpoint}.pin.json, atomic replace). A checkpoint outlives the process; a memory-only pin would silently stop protecting anything at the next restart. BUDGET HONESTY (sgl-project#715: never count as deliverable what the actuator cannot deliver). capacity_stats now reports pinned_entries, pinned_bytes and reclaimable_bytes = used - pinned, so a capacity decision reads the number eviction can actually free. Creating a checkpoint whose pins would cross SGLANG_HICACHE_PIN_BUDGET_BYTES is REFUSED with all four numbers (want, held, budget, overshoot) and changes nothing -- a cache quietly degraded into a pin museum is the outcome worth refusing. UNIT BUG FOUND BY THE TESTS, and it was the sgl-project#715 sin in miniature: the ledger charged APPARENT size (st_size) while the evictor accounts ALLOCATED size (st_blocks; the incident filesystem charged 8704 bytes for a 512-byte page, 17x over 5.8M files). reclaimable = used - pinned was therefore subtracting centimetres from inches, and could go NEGATIVE. The ledger now charges through LRUFileEvictor._allocated_size -- one unit, one authority. ORPHANS: a pin whose manifest is gone protects nothing and blocks eviction forever -- the .part706 leak shape one layer up, reaped the same way. Age-gated against the authority that knows whether the checkpoint exists, because a checkpoint being written has pins BEFORE it has a manifest and reaping those would delete the protection at the exact moment it is needed. Tests (hermetic, CUDA_VISIBLE_DEVICES=""), 13 new, 181 in the family. The three named red-first cases, all against the REAL evictor under REAL pressure: * test_the_pinned_sibling_survives_eviction_pressure -- two identical pages, one pinned, cap exceeded: the unpinned one dies, the pinned one is there. * test_pinned_bytes_are_not_reported_as_reclaimable -- the arithmetic. * test_an_orphan_pin_is_reaped (+ test_a_young_orphan_is_left_alone). Plus ref-counting, restart durability, the budget refusal with its numbers, zero-charge re-pinning, and the sweeper honouring pins. Two test-construction bugs of my own, both found by running rather than reading, both recorded because they are the same class: the eviction cap was computed from tensor length while the evictor charges allocated blocks, so the page was evicted before it could be pinned (the pin then correctly charged nothing, and the test failed for a reason unrelated to pinning). The cap is now MEASURED from a probe store -- which itself needs a cap, since byte accounting only runs when eviction is configured. Regression, same env, base c3e9487 vs this commit: unit/mem_cache 940 failed / 779 passed -> 940 failed / 938 passed (+159) unit/managers 4 failed / 2145 passed -> 4 failed / 2145 passed Still slice 3, named not smuggled: wiring pin/unpin to the checkpoint create and delete endpoints, which needs the live session layer. CAN-FAIL PROOF (mutation applied, suite re-run, reverted): T1 eviction ignores pins -> test_the_pinned_sibling_survives_eviction_pressure, test_a_store_of_only_pins_cannot_spin T2 reclaimable reported as used (the sgl-project#715 sin, planted) -> test_pinned_bytes_are_not_reported_as_reclaimable, +1 T3 unpin drops shared pages -> test_pins_are_ref_counted_across_checkpoints, and ONLY that one T4 the budget never refuses -> test_the_budget_refuses_with_the_numbers, and ONLY that one T5 orphan reaping ignores age -> test_a_young_orphan_is_left_alone, and ONLY that one T6 the sweeper ignores pins -> test_a_pinned_pages_partial_is_not_reaped, and ONLY that one T7 pins not persisted -> test_pins_survive_a_restart, and ONLY that one Restored tree re-verified green after every mutation (13 passed).

…e they raise ROOT, and it is not a new one. free_group_begin is called from the event loop (batch_result_processor.py:92 and :741). While that window is open, PagedTokenToKVPoolAllocator.free appends to free_group instead of extending free_pages (allocator/paged.py:293-308), so the pages sit in neither free_pages nor release_pages and available_size cannot see them -- while the tree has already counted them as evicted. That is sgl-project#681's third root, and flush_free_group (allocator/base.py:208) is its remedy. WHY IT CRASHED AGAIN ANYWAY. The remedy was wired into alloc_token_slots only. The paged twins reached their raise without ever asking whether the pages they needed were already freed and merely staged. The relief NET was carried across to the extend path under "sgl-project#681 RULE 3: every alloc path reachable from prefill admission gets the same net"; the third root was not carried with it, and the decode path had no net of any kind. So: one root, wired on one of the three paths that need it. That is the 02:18 crash -- 512 tokens refused with 147,456 counted evictable. The receipt-checking added in sgl-project#681 cannot catch it, because the eviction's receipt is HONEST: the tokens really were freed. THE LABELLED CANDIDATE IS REFUTED. The proposal was that _evict_leaf_node's allocator.free(x.value) might route rows to one sub-pool of a HybridLinearKVPool while available_size/alloc read another. It cannot produce this divergence: the accounting is entirely allocator-side over index bookkeeping (free_pages / release_pages / free_group), and available_size is computed from those two lists alone (base.py:187-188), so a free and the available_size after it read the same structure however the pool splits its tensors underneath. Pinned in TestAccountingLivesInTheAllocator, including a run with kvcache=None throughout. This says only that the hybrid pool cannot cause THIS divergence, not that it is defect-free. FIX: flush staged frees and retry, on both paged paths, before the relief ladder -- same ordering as alloc_token_slots and for the same reason, that it is not relief. It gives up nothing: it applies frees already performed and already counted. Cold path only, reached after an allocation has already failed. The raises are unchanged, so fail-loud keeps the last word. Tests (hermetic, CUDA_VISIBLE_DEVICES="", no GPU, no serving contact): test/registered/unit/mem_cache/test_paged_staged_frees_715.py 9 passed can-fail proof: the two fix-pins fail without the fix, with the exact production messages ("Prefill out of memory" / "Decode out of memory"); the other 7 hold either way test/registered/unit/mem_cache/ 797 passed, 1651 skipped, 124 subtests test/registered/unit/managers/ 2119 passed, 18 skipped, 130 subtests ruff clean on both changed files The pins drive the REAL PagedTokenToKVPoolAllocator on CPU, inheriting the whole group protocol unmodified; only alloc_extend/alloc_decode are overridden, because the production ones dispatch to CUDA kernels (sgl-project#624: the stub stays off the load-bearing path).

…omits TICKET STATE FIRST, because it changes what the work is: the corruption sgl-project#568 describes is ALREADY FIXED. Commit 4222976 ("ledger: park/restore replaced every non-persistent buffer with garbage") carries the park-side copy and the restore-side re-registration, it is present in the deploy tree (4 references to _cpu_nonpersistent) and absent from main, and three tests already cover it (test_a_non_persistent_buffer_comes_back_bit_identical and siblings). Verified load-bearing rather than assumed: reverting the park loop fails exactly those three tests and nothing else. So this commit is the REST of that omission -- the same "state_dict() is not the module's state" mistake, one layer up, in two places the original fix did not reach. 1. THE LEDGER DID NOT COUNT THEM. `tensors()` enumerated `state_dict()` only, so `size_bytes()` under-reported while `park()` went on to free the non-persistent buffers as well. Measured on a 64x64 parameter plus a 512-element inv_freq: registered 16,384 bytes, freed 18,432. The register prices victim choices on the registered number, and the residency report showed a module freeing more than it was ever registered as holding. FAILURE DIRECTION, stated: under-counting is the SAFE direction -- the actuator delivers more than promised, the mirror image of the sgl-project#715 sin rather than the sin itself -- and it costs accuracy, since the ladder can pass over a victim that would have sufficed. This cannot over-count: every tensor now listed is one the park really frees. 2. THE RESTORE GUARD DID NOT LOOK FOR THEM, and this one is sharp. A module whose ENTIRE state is non-persistent buffers -- a rotary cache is exactly that shape -- parks with an empty `_cpu_state` and a full `_cpu_nonpersistent`. The guard tested only the first and declared the module "parked but holds no host copy ... unrecoverable in place -- reload it from the checkpoint", while its bytes sat on the host one attribute away. A park that succeeded followed by a restore that refused. Before fix (1) it was worse still: such a module's `tensors()` was empty, so park took the "nothing to do" early exit and marked it parked having saved nothing. Both fixed at their single sources. The guard stays strict where it should: with BOTH stores empty the module really is unrecoverable, and a test pins that it still refuses. SIBLING SWEEP. The rule is module-GENERIC and always was: the ledger registers arbitrary modules by dotted path (inprocess_tts.py:274), so every asset class it will ever be handed goes through this path -- drafters, vocoder, speaker encoder alike. Nor is the affected state only `inv_freq`: the rotary variants in this tree also carry `short_cos_sin_cache`, `long_cos_sin_cache`, `cos_sin_q*_cache` and `axis_map` as non-persistent buffers. So the general rule is pinned rather than the single instance, over three module shapes including one made of nothing but such buffers. THE 19 COUNT IS NOT REPRODUCIBLE HERMETICALLY, and I will not pretend otherwise: it is an instance count from a loaded checkpoint, while the source carries 13 `register_buffer("inv_freq")` DEFINITION sites -- how many submodules a checkpoint instantiates is a property of the model. What the next translator test needs is therefore an instrument, and park now logs one: the persistent and non-persistent tensor counts per parked asset. Confirming "19" on metal is then reading a log line rather than guessing which submodules loaded. Tests: 4 new (26 in the file, 640 in the translator suite). The accounting rule is asserted over three module shapes; the only-buffers module survives the round trip byte-identically; it is no longer treated as empty; and a genuinely interrupted park is still refused. Also recorded: `git stash` is BANNED by a repo hook (shared stack across worktrees) -- it refused an attempt to A/B this fix, correctly, and the run that looked like a "before" was actually the fixed code. Use a patch file.

Step 2 of the reconciliation, and a correctness fix in its own right. A filesystem charges the blocks it allocated, not the length the file reports. On this rig's ZFS a 64-byte page occupies 512 bytes -- 8x -- and the incident that produced _allocated_size measured 512-byte pages occupying 8704 bytes each across 5.8M files. A store that believes it is using a fraction of its real disk does not evict when it must. It also unblocks sgl-project#410's pin ledger, which charges allocated bytes. While the evictor charged apparent ones, reclaimable = used - pinned subtracted two different units and could go negative. That is the sgl-project#715 shape: not a wrong number, but two right numbers that cannot be combined. _scan_existing_files, commit and touch now all charge _allocated_size. reserve deliberately still estimates -- a reservation is taken before the file exists, so the payload length is the only number available -- and commit reconciles it against the filesystem's own answer once the write has landed. RED FIRST, ON AN EVICTION DECISION rather than an accounting field: six tiny pages into a store sized for three ALLOCATED pages. Under apparent accounting the store believes it is at 320 of 1536 bytes and evicts nothing; under allocated accounting it is over cap and must. A test asserting only used_bytes would have passed against an evictor that computed the right number and still evicted on the wrong one. Both halves of max(st_blocks*512, st_size) are covered: ZFS over-allocates small files AND reports st_blocks==1 for large ones under delayed allocation. THE REGRESSION THIS CAUSED, FOUND AND FIXED. The first sweep after the change came back 599 failed against the port-only 590 -- nine new failures, all in test_hicache_file_lru_unit.py, with runtime going 12s -> 235s because the store was over cap from the first write and churning. The cause was the suite's fixtures, not the change: it sizes everything in apparent bytes (max_size="300", three 100-byte tensors, asserting _total_bytes == 300), a premise that collapses once the accounting is correct. B had already solved it, and the filename-grep rule is what found it: the same suite exists in both lineages and B's copy sizes fixtures in a measured _UNIT = 512 with the reasoning inline. Adopted verbatim -- 34 passed in 8.6s, no B-only imports, and identical ruff findings (14 in both A's original and B's copy), so no new lint debt. The other two files in that failure list were checked at both commits rather than assumed collateral: test_swa_eviction_boundary.py 9 failed on both, test_mamba_checkpoint_interval.py 19 failed / 15 passed on both. Standing no-accelerator family. Test results, same selection and env throughout: unmodified base 03adbf8: 590 failed, 180 passed, 485 skipped port only b61d708: 590 failed, 190 passed, 488 skipped this commit: 590 failed, 195 passed, 487 skipped Failures identical to baseline; +15 passed. (881 vs 878 deselected is the three tests in the new suite that the -k filter does not select.) Targeted: 18 passed / 2 skipped across the pin + allocated suites, with the previously skipped 0 != -384 test un-skipped and green and accounting_overshoot_bytes == 0 asserted. A's test_session_checkpoint.py: 48 passed. ruff clean on files I authored. No boots.

…utover THE CAUSAL CHAIN, measured 2026-08-17 across two boots: the carry re-homed a request into running_batch/running_mbs[0] and left waiting_queue untouched -> the request existed TWICE: resident (invisible to the policy as runnable) and queued (counted) -> _pending_prefill_tokens summed the queue and the resident set without excluding their intersection, billing the same prompt twice: 51,369 -> 102,307 tokens across one cutover, within rounding of 2x -> the inflated backlog drove the flip policy past its threshold -> six cutovers, FLIP-CARRY announcing a resident carry while the policy read `running bs 0`, the sgl-project#699 detector reporting "1 queued, 0 running", and the warmup generation never served. THE FLIP CHURN WAS A SYMPTOM. Six cutovers looked like a flip defect and were not one: the policy responded correctly to a number that was wrong. Nobody should "fix" the churn separately. THREE PARTS, because the defect has three surfaces: 1. STATE -- the carry now CONSUMES what it re-homes (`_consume_carried_from_waiting_queue`). After a cutover the resident set owns the request and the queue must not also claim it. Both edges pinned: a request that is ONLY queued stays untouched, and a carry over an already-consumed queue is a no-op rather than a corruption. Bookkeeping never raises into a cutover (sgl-project#715 lesson). 2. NUMBER -- `_pending_prefill_tokens` de-duplicates at the INTERSECTION. Not a blanket "count each rid once anywhere": a request genuinely holding budget in two places is a real state a future reader may need to see, and a global dedup would make that class silent the way this one was. Exactly one overlap is excluded, and only this one. The sgl-project#713(a) resident term did not create the duplicate state -- the carry did -- but a counter that sums two sets without excluding their intersection is the second half, and that half is mine. 3. GUARD -- `duplicate_resident_reqs` gains the waiting queue in its universe. It compared batches against each other and never consulted the queue, so it reported "no duplicates" meaning "none of the kind I look for". Queue-side hits carry a `queued:` prefix so the 2026-08-09 resident-vs-resident specimen and this one stay distinguishable. Plus the hardening: `_arriving_prefill_tokens` asserted "ARRIVED but not yet on the queue" and enforced nothing. Its only inflight-bearing call site is pre-queue so the invariant holds today, but an asserted-never-checked invariant is exactly how this double count stayed silent, so it is checked now. TESTS 18 passed. Mutation-proven THREE ways, each caught: removing the carry's call site, removing the counter's dedup, and removing the guard's queue universe. TWO OF THOSE TESTS WERE VACUOUS FIRST, and both are worth recording: * the carry test called the helper directly, so deleting the call from `install_resident_set` left everything green -- the helper worked and nothing used it. It now drives the real entry point. * the counter test's duplicate had `extend_range=None`, which the sgl-project#713(a) term skips by design, so the resident side contributed 0 and the intersection was never exercised. The duplicate is now made countable on purpose. Both were found by running the mutations rather than by reading the tests. REGRESSION managers + mem_cache: 19 failed / 3588 passed against a captured baseline of 19 -- ZERO new failures. Ruff: scheduler.py 94 errors before and after, all pre-existing; the other two files clean.

…refused on structure Desk-only classification. Verdict and fix SHAPE only -- no fix built; the boot wrapper is F4-r4's and the operator's ack is required. THE HYPOTHESIS DOES NOT HOLD, and it is refused structurally rather than for want of evidence. The store port cannot collide across boots: route_a_631_prod_boot.sh pins neither --nccl-port nor --dist-init-addr, so server_args.py:18787-18788 applies -- nccl_port = get_free_port() draws a port that is free at that moment, and a predecessor still holding its own store port is not a candidate. A lost race is caught anyway at :18873 by wait_port_available, which polls 30 s and names the holding process. There is also a shape argument that does not depend on this codebase: "Connection closed by peer" is a connection ESTABLISHED and then broken, whereas a stale predecessor holding a port yields EADDRINUSE at bind or a refused connect. The observed error is the wrong shape for the hypothesis. So the answer to "does wait_host_release check the store port" is no -- and it should not need to. Extending it there would encode a refuted hypothesis into the boot wrapper, where the next reader would trust it. AT LEAST TWO ROOTS, not one. The specimens split on evidence: - 18:23:43 is not NEAR an OOM event, it IS one. syslog:1789 records "A process of this unit has been killed by the OOM killer" at 18:23:43.842677. The same signature repeats at 18:39:05 against e66bde7's own measurement at 18:39:07 -- F4-r4 called that boot "killed by an external process exit", which is what an OOM kill of a peer looks like from inside a survivor. A third sits at 18:45:30. - 19:31:45 rank2 is NOT memory. The host ledger reads avail=103 headroom=97 at 19:32:24, 39 seconds later, and there is no OOM line anywhere in the 19:3x window. SENDBYTES IS A TOMBSTONE. In 3 of 3 recorded instances it is the SECOND event: HANDOFF_663:696-698 (peer dies, no traceback -- "exactly what a SIGKILL looks like"; that run "died of host RAM"), HANDOFF_658:256 (rank 0's TCPStore died, survivors then spun on sendBytes), and e66bde7's gloo frame naming a dead peer process. Any verdict that makes the socket primary is fighting the prior. THE REAL HOLE is elsewhere and is the recurring shape. wait_host_release.sh computes S -- the count of surviving schedulers -- on every iteration, PRINTS it in the clear-to-boot line, and never puts it in a condition. The gate is available-RAM and nothing else, so a predecessor whose schedulers are alive but whose allocations are already unmapped passes it. The single-instance guard does not cover the gap either: boot script :250 pgreps sglang.launch_server, the LAUNCHER, while the store and ranks live in the sglang::scheduler children -- an orphaned scheduler set whose launcher has exited passes cleanly. That is the counter-without-a-reachable-actuator family (sgl-project#679/sgl-project#681/sgl-project#684/sgl-project#715): the value that answers the question is discarded one line before the decision. FIX SHAPE (not built): gate on the counter that already exists -- require S -eq 0 alongside A -ge NEED -- which closes the predecessor window for RAM, GPU and the store socket at once, without a bespoke port probe. Two cautions handed to the owner: grep -c "sglang::schedul" reads ps comm, truncated at 15 chars, so the match is one rename from silently counting zero and a miscounting gate that reports "clear" is worse than none; and requiring zero turns a soft wait hard, which is the right direction but must fail with PIDs named rather than timing out anonymously. HONEST GAPS, recorded in section 6 rather than papered over: dmesg is permission-denied here and journalctl -k returns "No entries", so kernel OOM detail is UNAVAILABLE from this session, not absent -- which process the killer took at 18:23:43 is therefore not established, only that it fired. The serving tree runs under setsid outside systemd, so the absence of an sglang unit line is not evidence it survived. Specimen B's root and the third specimen's log remain open; a timestamp tied to a specimen is not the same as its log read, and I have not read the second. Adds "schedul" to .codespellrc: it is the literal 15-char truncation ps comm reports, not a typo.

…remise POSTEN 1 -- MEASURED FIRST, AND THE PREMISE DOES NOT SURVIVE THE MEASUREMENT. The instruction was: the guard is inert because there is no readable total, so cap `memory.current` against a constant 111.3 GiB from sgl-project#721. Measured on this box before touching anything: pinned_host_memory_bytes() -> total=118.05 GiB available=112.95 GiB so it does NOT land on the :245 "no honest number" branch. The guard is ACTIVE, a total IS readable, and `PINNED_HOST_RESERVE_BYTES` = 10 GiB is therefore NOT inert as a subtraction. The clamp binds too: MemAvailable 113.19 -> 112.95 GiB, reduced by the cgroup's own resident accounting exactly as rule 2 of `honest_host_memory_bytes` documents. `MemAvailable > MemTotal` is FALSE right now, so the pathological lxcfs reading is not present either. AND THE PROPOSED CAP WOULD NOT HAVE BOUND ANYTHING. Every level of the visible hierarchy is unlimited -- root, system.slice and system.slice/claude.service all read `memory.max = max`, and all three report `oom_kill 0` today (sgl-project#721's 17 kills predate this boot; `memory.peak` 101.98 GiB of 118.05). With no enforcing cgroup anywhere, the physical machine IS the ceiling, which is what MemTotal reports. A hand-carried 111.3 GiB constant would have been a third floor with no enforcer behind it -- the rule this ticket already established, applied to its own instruction. Not built. THE REAL DEFECT IS ONE BUCKET OVER, AND IT IS THE sgl-project#695 CLASS AGAIN. Prior art via the new index: commit c043235 measured that "CUDA pinned host memory is accounted in the cgroup's `file` bucket, not `anon` ... the offload ledger reported 20.78 + 14.44 + 14.44 = 49.66 GiB of pinned pool while `anon` sat steady at 14.6 GiB." Confirmed live on this box: anon 2.12 GiB, file 19.06 GiB, current 21.39 GiB -- the clamp charges ~2.4 GiB of a cgroup holding 21.4. `honest_host_memory_bytes` charges anon + kernel + unreclaimable shmem and NEVER charges `file`, correctly, because page cache is reclaimable. Pinned bytes there are not. sgl-project#695 already fixed one member of this class (shmem hiding in `file`); pinned host memory is the second and is still uncounted. IT ALSO PUTS A DOCUMENTED PREMISE IN DOUBT, and the doubt is precise: `pinned_host_budget.py:253` credits already-registered posts BACK to available because "their bytes are therefore already missing from it". For a post whose bytes land in `file`, they are NOT missing from it -- the clamp never subtracted them -- so the credit-back would count them as free twice. The 2026-08-17 measurement quoted there is sound for the weight images it was taken against; whether it holds for THIS pin depends on which bucket the pin's bytes land in. DELIBERATELY NOT FIXED HERE. That question is decidable only by watching `anon` and `file` across an actual pin allocation, which is a boot. Changing admission arithmetic on a shared path used by HiCache and kv-session-offload, on a guess about which bucket, would be the "capping on a fabricated figure" the module exists to prevent. So it is INSTRUMENTED instead -- the window script prints the before/after pair and names the consequence -- and the arithmetic is left alone until the boot answers it. POSTEN 2 -- `chunk_blocks_quiescence` (phase_flip_runtime.py:144): SAME CLOCK DIVERGENCE, and the determination is not a formality. It is the same class as the retention gap: a consumer (the cutover) and a writer (the chunk boundary that inserts into the tree) on different clocks. Under STRICT the clocks are SYNCHRONISED -- the cutover waits for the prefill, which is what "waiting for prefill to finish IS drain-and-flip" means in its docstring -- and the residual is exactly the NON-STRICT case where they are not. THE DIFFERENCE THAT MATTERS, and why this is one root and not two: the chunk case has a BOUNDED wait available (a prefill completes in finitely many chunks), while a mid-decode resident does not. That is why STRICT can fix one and nothing can fix the other by waiting. But the NON-STRICT chunk case cannot be fixed by waiting either -- an unconditional block re-creates sgl-project#631 defect O, the 32768-token prefill that ran in the slow layout and paid two cutovers for nothing. So BOTH halves need the same missing capability: persisting PARTIAL work at the seam. Neither is closable without it, and that capability is `seam_copy_state` / `restore_seam_state` (schedule_batch.py:2054/:2089) -- sgl-project#875's active territory. Reported, not touched, per Posten 4. Unlike the sgl-project#813/sgl-project#852/sgl-project#715 convergence I rejected last round, this one IS a shared root, and I am saying so because the evidence supports it, not because it tidies. POSTEN 3 -- `scripts/window_871a_verify.py`: one call, PASS/FAIL with numbers. Decides all three open claims from a boot log plus the cgroup. Exit 0 all passed, 1 a decided negative, 2 UNDECIDED -- and 2 is never a pass: "the evidence was not there" is a different fact from "the claim is false", and it sends the reader back to the boot. It boots nothing, claims no card, restores no serving. It also prints the sgl-project#721 HOST-LEDGER pair (posts + memory.current / peak / anon / file, before vs after) that Posten 1 needs. IT SHIPPED A FALSE PASS AND I CAUGHT IT BEFORE THE WINDOW, which is the whole argument for mock-smoking a window script. The first version summed `acked=` over the WHOLE log and returned PASS on the W40 boot -- the very boot in which all 21 fences reported `acked=0`. Seven lines there carry `acked=` from an unrelated subsystem, three of them `acked=24`, so it credited 72 acknowledgements no fence ever made. A window script that reports a false PASS is worse than none: it CLOSES an open claim. Now scoped to the fence's own lines, with that exact log shape as a permanent self-test case. The strongest desk check available is wired in: run against the real W40 log it must reproduce that boot's known split -- tier armed PASS, store delivery FAIL, exit 1 -- and it does. TESTS (hermetic; no boot; no card touched). CUDA_VISIBLE_DEVICES="" verified at the process and `nvidia-smi --query-compute-apps` empty throughout; GPUs 0/0/0. Runner reported its exclusions. * the script's own --self-test decides SIX cases in the intended direction, both polarities, including the false-PASS regression. * can-fail by mutation: un-scoping the acked regex reddens the real-boot reproduction and the self-test (2 red); restored green. * A SECOND SELF-INFLICTED DEFECT FOUND AND FIXED: the test located the script by counting `dirname` calls, landed on `test/` instead of the repo root, and failed all six cases -- a green arm red for a reason unrelated to the thing under test. It now walks up to the root. * partitioned tier-2 gate: 0 failing, 4421 passed (wide 3775 / narrow 292 / serial 354). Serial +6 over 348 is exactly the six tests added here. names=0 agrees with the summary on every lane. * test/registered/unit/mem_cache: 2 failed / 1802 passed -- the known pre-existing test_acceptance_emitters_758 RefillTiming pair, unchanged. POSTEN 4 -- boundary held. Nothing in schedule_batch.py, phase_flip_spill.py or seam_kv_recover was read into or written. Both Posten 2's root and A's remedy lead there; both are reported rather than resolved. NOT ESTABLISHED. Which cgroup bucket the phase-flip pin's bytes land in -- the question Posten 1 turns on -- needs the boot. Whether the admission ever refuses on this box, likewise. And sgl-project#721's 17 oom_kills are from a previous boot: today's counters are zero at every level, so nothing here re-measures that incident, it only records that the enforcement it implies is absent.

zhyncs added 2 commits July 25, 2024 03:43

misc: update doc

0d3e54c

update

9c899ee

zhyncs requested review from Ying1123, hnyls2002 and merrymercy July 24, 2024 17:46

Ying1123 approved these changes Jul 24, 2024

View reviewed changes

Ying1123 merged commit 321a963 into sgl-project:main Jul 24, 2024

zhyncs deleted the upd branch July 24, 2024 23:07

timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025

misc: update doc (sgl-project#715)

2739db4

Navigation Menu

Uh oh!

misc: update doc - #715

misc: update doc#715
Ying1123 merged 2 commits into
sgl-project:mainfrom
zhyncs:upd

zhyncs commented Jul 24, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

zhyncs commented Jul 24, 2024

Motivation

Modification

Checklist

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants