Update docs - #486
Merged
Merged
Update docs#486
Conversation
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
EdwardXuy
pushed a commit
to shun8686/sglang
that referenced
this pull request
May 9, 2026
Skip the execution of new model accuracy test cases in nightly.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 3, 2026
…rch guard stricter than upstream _varlen_arch_ok; rebase-gate list persisted; sgl-project#33348 root not shared by our replicated draft-KV path (loose end -> sgl-project#487); EAGLE 2x reserve carried unmodified -> sgl-project#486
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 3, 2026
…ne dead write Follow-up to the derived-reserve commit, from finishing the per-lane sweep. Coverage is now stated explicitly in the ledger: the tree has exactly two per-decode watermark sites and both derive the reserve -- eagle_prepare_for_decode (EAGLE / EAGLE3 / NEXTN-MTP / STANDALONE / frozen-KV-MTP / multi-layer, none of which override it) and dflash_info_v2.prepare_for_decode (DFLASH and DSpark; dspark_components/ has no allocation code of its own). The ngram lane holds no reserve at all: it allocates a flat draft_token_num from batch.seq_lens each step and never advances kv_allocated_len, so it meets the formula only in the admission estimator. EagleDraftInput.ALLOC_LEN_PER_DECODE, assigned in multi_layer_eagle_worker_v2 and standalone_worker_v2, is a DEAD WRITE -- the class has no such attribute and nothing in the tree reads it (verified: hasattr is False, zero readers). It reads like the W term and is not; recorded as a trap rather than left to mislead the next person. Harmless today only because adaptive spec requires topk=1, where its expression and get_alloc_len_per_decode agree. DSV4F offload recipe corrected in the GPU ticket: that boot line carries no speculative flags, so its reserve is the plain-decode 1+1 = 2 slots, and its MLA rows are replicated rather than TP-split (DeepSeekV4TokenToKVPool has no tp_size; _reject_uneven_tp_mla refuses uneven-TP MLA) at 43 x 584 B = 24.5 KiB/slot on every rank -- 49 KiB total at --max-running-requests 1. Nothing for sgl-project#486 to reclaim there either. Tests unchanged and re-run: 6 passed, 33 subtests. codespell clean.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 3, 2026
…stead of a blanket 2W W = get_alloc_len_per_decode() (this step's draft+verify write footprint) L = get_commit_lag_per_decode() (what one in-flight verify can still commit) Both eagle_utils.eagle_prepare_for_decode and dflash_info_v2.prepare_for_decode now derive from the same formula in mem_cache/common.py. On the NEXTN production recipe (steps 3, topk 1, 4 draft tokens, overlap on) W == L == 4, so the reserve is unchanged there; the reclaimable part (W - L) is nonzero on topk>1, page>1 topk>1, chains with num_steps > num_draft_tokens, and every --disable-overlap-schedule run. See docs/dev/TICKET_486_reserve_radix.md and DESIGN_330 §3b.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 9, 2026
…sified VERIFY BEFORE FIXING, applied. The hypothesis recorded last commit was that build_flip_live_slots_fn enumerates req_to_token[idx, :seqlen] while the allocator owns kv_allocated_len -- structurally different under sgl-project#486, whose spec reserve is W + L slots ahead of kv_committed_len (W = the draft/verify write footprint, several slots on this rig's NEXTN config, not one). It fits the symptom exactly. It is wrong. Measured with a census bracket around the flip, identical on all three ranks (2026-08-09 02:08:17Z): POOL CENSUS at-arm live_reqs=1 cached=80 unaccounted=1 [81] POOL CENSUS pre-cutover live_reqs=0 cached=80 unaccounted=1 [81] POOL CENSUS post-cutover live_reqs=0 cached=80 unaccounted=1 [81] Page 81 is already unaccounted before the flip moves a byte, and the set is unchanged across the move and the cutover. A no-flip control boot (POLICY=manual, one request served to completion, server idle) stayed clean: leaks=0, flips=0, health 200. WHAT THIS LOCALISES. At arm the row is legitimately held -- live_reqs=1, charged by the checker as uncached = kv_allocated_len - cache_protected_len, so the invariant balances. By pre-cutover the request has FINISHED, its 80 committed rows are in the tree, and its one uncached row was never freed: nothing owns page 81 and no live request remains to charge it to. The defect is in the COMPLETION PATH OF A REQUEST THAT FINISHES WHILE A FLIP IS ARMED, not in the flip's KV move. First suspects are what the armed state defers or suppresses around completion (the abort-deferral window), and the checker's own invariant assert req.kv_committed_freed == req.kv_overallocated_freed. Changes, all diagnostics; no behaviour change to what the flip moves: - _pool_census(when, direction), reproducing the invariant checker's expected - free - cached arithmetic, called at arm and on both sides of the cutover. Read-only and exception-swallowing: a census must never be able to affect the flip it watches. - _probe_allocated_extent, reporting seqlen vs kv_allocated_len vs cache_protected_len per live request, so the sgl-project#486 reserve is visible rather than inferred. - _live_reqs (running_batch + last_batch, the pair the checker walks), used by diagnostics ONLY. - build_flip_live_slots_fn is UNCHANGED, deliberately. I had widened it while adding the probe; the census falsified the reason, so it is reverted. Widening it would have moved rows nobody had shown were missing, in the one place where a wrong guess corrupts a request's context silently and the leak detector would never have said a word. Test results: - bash scripts/run_631_flip_family.sh -> 379 passed. - Metal: automatic flip reproduced twice more (cutover complete on all three ranks); census bracket above; no-flip control clean. - Production restored on POLICY=manual and verified: GET /health_generate 200.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 11, 2026
…acity is now a named posten Second metal kill of the shift, and it proves the cap WORKS: with a commit chunk in place the rung shrank the pool and withheld 80165 slots, and the scheduler's first idle check then killed all three ranks with ValueError: pool memory leak detected! [full] total=500000, available=419745, evictable=90, protected=0, session_held=0, uncached=0 419745 + 90 + 80165 = 500000. Nothing leaked. The invariant is available + evictable + protected + session_held + uncached == total, and capacity deliberately held out of circulation is in none of those buckets. Added a 'withheld' term, sourced from the allocator's residency_withheld_slots, which KvRowCap publishes on every mutation. It is a NAMED posten for the sgl-project#486 reason: anything that durably removes pool slots must be named here, or the next unexplained delta is attributed to the wrong holder. A test pins that the term does not become a licence -- the same shortfall with no cap engaged is still a leak. Published in TOKENS, not ids: available_size() multiplies the free list by page_size, so a raw id count would be wrong by that factor on every paged lane. Tests: 881 pass in the 631 flip family (24 in the relief file, 4 new), 1 inherited red. ruff clean. FEATURE_CATALOG updated with both new rungs and the commit-chunk requirement.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.