misc: add compute capability in check_env - #965
Merged
Merged
Conversation
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 28, 2026
…te the group, and let the guard speak sgl-project#965 -- THE CO-DERIVED MATCH TUPLE. `Req.init_next_round_input` reads the radix tree ONCE and unpacks that single `match_result` into EIGHT attributes in one tuple assignment: prefix_indices, last_node, last_host_node, best_match_node, host_hit_length, swa_host_hit_length, mamba_host_hit_length, mamba_branching_seqlen. One reading of one geometry wearing eight names. `truncate_prefix_to` moves the prefix they were derived from. Its own docstring says why that matters -- "the geometry was DERIVED from the prefix that just moved, so it is now a stale reading rather than a report" -- and that sentence is true, word for word, of every field in the tuple. It wrote prefix_indices, and over two separate windows, each discovered by a boot, cache_protected_len (sgl-project#930) and extend_range (sgl-project#958). Six were left standing. This is the complete sibling sweep those two never got. NOTHING RE-DERIVES IN BETWEEN. scheduler.py calls init_next_round_input (all eight derived), both arms of the `pp_size > 1` fork call truncate_prefix_to, then adder.add_one_req reads them. No match_prefix on that path. WHAT THE STALE READINGS DO, at PrefillAdder.add_one_req: * `real_input_tokens = cand_extend_input_len - req.host_hit_length` subtracts a host hit no longer part of this prefix, under-counting against the budget; * `needs_host_load_back()` is still true, so init_load_back runs on a stale best_match_node/host_hit_length and does `prefix_indices = torch.cat([prefix_indices, new_indices])` -- leaving `[0, told)` then `[L_dev, L_dev+H)` with a HOLE between, while prepare_for_extend sizes the cross-stage tensor off len(prefix_indices) as though it were contiguous. The silently-wrong-context class; * that branch then sets `cache_protected_len = prefix_len`, re-raising the exact value the truncation lowered nine lines earlier -- undoing sgl-project#930. FIXED AS ONE CONTRACT, not a seventh one-field patch, because two of the eight were already bought one at a time by two boots. The ratchet (`test_every_co_derived_field_is_accounted_for`) reads the producer tuples from the AST -- not by grep, since a grep needs a name and the whole point is a field nobody has thought of yet -- and FAILS when a ninth joins, telling the author to decide TRUNCATED / CLEARED / HANDLE. Planted-ninth-field proof: it names `newly_added_ninth_reading`. `last_node` IS NOT NULLED, and that asymmetry is deliberate: it is not a reading but a RESOURCE HANDLE holding an outstanding inc_lock_ref, and this attribute is the only surviving reference to it. Nulling it would leak the ref and make the node permanently unevictable -- a defect that already exists on the PP void path (`_release_dynamic_chunk_probe` frees rows without cache_finished_req / dec_lock_ref, unlike schedule_batch.py:2170-2175 and disaggregation/prefill.py :1159-1161, which do). Registered, NOT fixed here. A NO-OP TRUNCATION STILL INVALIDATES NOTHING. My first test asserted the opposite; the invalidator's own documented rule -- "a no-op truncation leaves a valid geometry valid, clearing it there would void healthy passes for nothing" -- is right, and clearing there would discard a host hit every rank still holds. The test was inverted to pin the documented behaviour instead of contradicting it. sgl-project#967 -- THE sgl-project#959 GUARD WAS UNMEASURABLE. It is closed by two bare `return AddReqResult.OTHER` statements. A refusal that leaves no trace is indistinguishable from a scheduler that simply built nothing, which is precisely the state the sgl-project#963 investigation spent a boot separating by hand out of per-rank coverage databases. Unconditional per-site counter plus a rate-limited `[sgl-project#967] SECOND CONTINUATION REFUSED` line, following the neighbour that already got this right (`_note_seam_chunk_refused`: first three, then every thousandth). Module-level, because PrefillAdder is rebuilt every pass and an instance counter would reset before anyone could read it. THE TWO GUARDS ARE IN DIFFERENT METHODS -- add_one_req_ignore_eos:1692 and add_one_req:1935 -- and my first wiring pin inspected only add_one_req, so it measured nothing and failed on the UNMUTATED tree. That is how the mistake was caught. Rewritten to walk the module AST, and it now names the offending line number when a guard's counter is stripped (proof: "guard(s) at line(s) [1935] return without counting"). The site labels are the method names. CAN-FAIL, read rather than asserted: sgl-project#965 clearing withdrawn -> 8 failed, one per field, each naming its own field sgl-project#965 ninth field planted -> ratchet names it sgl-project#967 one guard's note stripped -> wiring pin names line 1935 NUMBERS, scripts/gate_tier2_partitioned.py, CVD="" : BEFORE 2 genuine (test_collective_family_siblings_610.py x2), 685.00 s AFTER 2 genuine (the same two), 580.49 s, 358 modules count probe 2 named == 2 summary, SUBFAILED and ERRORS included truncation family + prefill adder + schedule_policy suites: 54 passed ruff: schedule_batch.py 45 before / 45 after, schedule_policy.py 21 / 21, both new test files clean TICKET_961_WINDOW.md gains section 8: the "race" of section 7 is NOT a race (ServerArgs byte-identical, the LOAD ARM differed -- 8422 chunked vs 2822 sub-chunk, and an over-one-chunk prompt forces a chunked_req onto every rank, so Arm A masks the defect), and the acceptance now counts flip EPOCHS and phase=tp BATCHES SEPARATELY: 3+3 "both directions" was 3 RANKS confirming ONE epoch, and no batch ever ran phase=tp, so every post-flip guard's green was vacuous. NOT CLAIMED: no boot, no metal, no effect measurement.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 28, 2026
…, not at the exits Boot 8 died 25s into load: prepare_for_extend's sgl-project#965 invariant (len(prefix_indices) == extend_range.start, asserted at schedule_batch.py:3262) failed on PP0. R11 rooted the writer pair, refuting both suspects (neither the sgl-project#944 told=0 terminator -- it took its report-only branch -- nor the sgl-project#987 adopt, which moves only the fill): the host load-back in PrefillAdder.add_one_req grows prefix_indices in place and is followed by budget/align/sgl-project#967 early returns that never reach set_extend_range; reachable because the waiting_queue loop carried no membership guard, so a request already resident in can_run_list (the void family's deliberately preserved co-residency) was visited a second time, its prefix moved, and the bail-out left the first visit's extend_range behind it. Fix, the sgl-project#965 lesson (one junction, never per-branch bail patches): re-derive Range(prefix, prefix) IMMEDIATELY at the mutation -- every current and future early return inherits a consistent parked shape automatically; the success paths overwrite it with the real range as before. Plus the identity membership guard at the loop head (skip census 'already_in_batch'; identity not rid -- two objects with one rid are a different defect and must not be masked). Plus the sgl-project#988 LOADBACK instrument line (module-level, rate-limited). Sibling class registered as ONE posten, not fifteen: 11 further prefix_indices reassignments without same-function re-derive across the cache stash paths (R11 sweep; they run where the next admission re-derives, so not reachable as this crash). Execution proof: import smoke green; the load-back ran on every cache-hit admission in boot 8 (FILL-ADOPT and cached_pos=8 prove the hit path hot); the guard sits in the loop that produced boot 8's crash batch. Boot 8's collapse numbers stand: unexec 513 -> 10, abort net 6x bounded, batches and cached at family bests -- this closes the last named layer of that boot.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 28, 2026
…boot 8, at the mutation sgl-project#988 shipped 56 lines with no tests. Unlike sgl-project#980 this one admits a genuine behavioural red-first: the arms name no symbol it introduces (set_extend_range predates it) and assert sgl-project#965's co-derived invariant itself -- extend_range.start == len(prefix_indices), which prepare_for_extend asserts at the batch boundary. The specimen is boot 8's: PrefillAdder's host load-back grows prefix_indices in place, and before sgl-project#988 went on to the budget checks without touching extend_range, so every early return below that line left the previous visit's range behind a moved prefix. The arms drive a bail-out that happens AFTER the load-back and assert the invariant on the way out, deliberately not pinned to one specific return -- the fix's argument is that re-deriving at the mutation makes every current AND FUTURE exit inherit a consistent shape, and per-branch patches are how sgl-project#965 was paid for twice. An arm pinned to one exit goes stale the moment another is added. PICKING THAT BAIL-OUT TOOK A WRONG TURN, recorded in the file. add_one_req carries TWO near-identical 'rem_chunk_tokens is None and input_tokens >= rem_input_tokens' returns, one before the _lock_node block and one after. Configuring for the input budget hits the FIRST, so the prefix never moves and the file measures nothing -- caught only by the precondition assert, which is exactly what it is for. Nor can budget separate them: the post-load-back input_tokens is SMALLER than the pre one, so any budget tripping the second has already tripped the first. The truncation path (chunk budget 0) is reachable only after the load-back, and that is the one used. MEASURED, same file: 4ea93b8 (pre-sgl-project#988): 2 failed, 2 passed 8380555 (post) : 4 passed The reds read 'the prefix advanced to 4096 and extend_range still starts at 0' and '(0, 8422) != (4096, 4096)'. The two pre-fix PASSES are load-bearing: the prefix-preservation arm and the can-fail hold on the old tree too, so the failures are the defect and not a broken harness. Chain at 8380555: 164 defs / 164 passed across eleven suites.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 29, 2026
`set_extend_range(prefix, prefix + new_len)` means
`extend_range.start == len(prefix_indices)`, and `prepare_for_extend` slices
`fill[len(prefix) : extend_range.end]`. So
len(input_ids) = end - len(prefix) = end - start = new_len
holds ONLY while `start == len(prefix_indices)`. That is one condition, and
`prepare_for_extend` is the one place that reads both halves.
WHY THIS REPLACES THE ENUMERATION. There are 24 writers of `extend_range`
and 27 of `prefix_indices` across five directories. Two were read -- the
sgl-project#791 truncation and the #797b park -- and both carry the group; the park
does so precisely because it derives its shape FROM the prefix in the same
call. But 2 of 24 is not a basis, and reading the other 22 still could not
show the case where every writer is individually correct and their ORDER
breaks the pair. A condition at the consumer covers all 51 writers and that
case as well. Third time today that checking the consumer beat enumerating
the producers.
IT IS ALSO THE MEASUREMENT. The guard reports 254 rows against 301 tokens;
this decomposes the 301 into its two terms, per rid, on every rank. If
`end` or `len(prefix)` stands differently on the sender, the root is read
rather than inferred -- and the rid then names which writer touched it last,
so the enumeration survives as a fallback reduced to one entry.
Discipline unchanged, and each clause was paid for in this window: counter
at the event, emission in `pp_ring_note` (independent of this probe, its
execution on this config established), no hit filter, `seen` and `breaks`
always printed so a zero is distinguishable from a never-run, and -1 as a
sentinel because 0 is a legitimate break value.
Predictions recorded before the boot, both directions. A: `break == 0`
everywhere, which puts the divergence in the READ or the TIMING rather than
the setting -- the first question of this window an enumeration cannot
reach. B: `break != 0` somewhere, which is the root. I expect A weakly, and
say why: the coupling is deliberately built at the truncation junction
("sgl-project#965 THE WHOLE CO-DERIVED GROUP, AS A GROUP"), but that is an argument
from 2 of 24, and I have been wrong twice in this window reasoning from a
subset.
The 47 will not be computed into anything either way. The question is
whether the invariant breaks.
Evidence: desk. py_compile on both files; the module bridge imports; the
invariant arithmetic checked on a holding and a broken case. Belegstufe:
DESK-BEWIESEN.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 31, 2026
…old decide HOW MUCH, not whether Three corrections, two of them to defects in my own slice that boot 2 exposed. PREFIX-KEY KEYING (the reason boot 2 returned 0 applied load-backs). The fact was keyed by rid, but a request reaches the load-back site EXACTLY ONCE and only learns the extent during that visit -- measured: 5 rids x 3 ranks, every rid deferring once (occurrence=1..5 per rank). So a rid-keyed fact is always one visit too late for the rid that named it. And the warm hit is by definition a DIFFERENT request: the cell's mid_cold/mid_warm/mid_warm2 carry identical content under three distinct rids. Keyed by offered_prefix_key the extent travels across rids, across cutovers and across the one-lap lag at once. Keyed on the FULL FILL, not the device match, because offered_prefix_key returns None for prefix_len<=0 and the device match is measured as 0 on exactly the warm path this serves. S1 -- TOLD DECIDED *WHETHER*, NOT *HOW MUCH* (silently wrong, would have reintroduced the death class one layer down). init_load_back consumes host_hit_length only as a boolean trigger (unified_radix_cache.py:4981-4988); the amount returned is this rank's LOCAL node run at :4990, and load_back() never sees the number. Without a clamp, "every rank applies the same number" was unbacked -- each rank grew by its own local amount, which is the 1815081 divergence. And the skew is the NORMAL case, not an edge: the extent is published one lap before use and keyed by content, so it is stale-low by construction (same rid measured 1215 -> 1216 -> 2114). Now clamped at the caller to exactly told, with a loud refusal when a rank cannot even reach it. Clamp rather than refuse because stale-low is the safe direction: fewer revived tokens is more recompute, never more memory. Gated on pp_load_back_told, so pp_size<=1 stays byte-for-byte upstream. F1 -- A DOCSTRING CLAIM THAT THE CODE DID NOT BACK, retracted rather than left to be read as a guarantee. It said "the first chunk computes without the prefix and every later chunk honours it". The only consumer is in add_one_req; add_chunked_req has no load-back site, so later chunks of an already-admitted request do NOT consume it. Deliberately not mirrored there: that would put a second prefix mutation on the chunked path, the seam sgl-project#965/sgl-project#988 record as having already cost a boot. What is actually paid is now stated: the request that discovers an extent never benefits from it; the next request over the same content does. Plus the success instrument the chain was missing (sgl-project#968 LOAD-BACK PROMOTED at the promote site). Until now there was only a failure marker and an application marker, so "0 applied" could not be attributed to a missing publish, a missing hop or a missing consume. Desk checks: imports; clamp present, gated, and ordered before the prefix growth; clamp arithmetic yields exactly told; prefix key content-stable, length-sensitive, None at prefix_len=0, and STABLE ACROSS PYTHONHASHSEED (ranks are separate processes, so a builtin hash would disagree only on multi-process runs); codec round-trip and width tolerance at row widths 12/11/10. Lifecycle table recorded for all seven new states incl. deleters and their separating events -- effective is deleted by the SPEND, not by a clock or an epoch. Ruff finding set unchanged (172/172).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
inspired by flashinfer-ai/flashinfer#426 (comment)
Modification
as titled
Checklist
pre-commit run --all-filesor other linting tools are used to fix potential lint issues.