misc: disable auto release - #862
Merged
Merged
Conversation
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 26, 2026
…ct#868's 15, and one keyword wide sgl-project#864. The question was whether the 16 standing failures are the same hermeticity form as the 15 that sgl-project#868 resolved. They are its MIRROR IMAGE, and the mirror is the direction that hurts: hermetic cards the desk gate shows sgl-project#868's 15 FAIL pass a red nobody can act on sgl-project#864's 16 SKIP FAIL a GREEN over a standing red Same class -- the invocation decides the verdict -- but sgl-project#868's announce themselves on every desk run, while these announce themselves nowhere anybody looks: the machine that can see them has cards, and the gate people read does not. CAUSE, proved without a card. hiradix_cache.py:266-278 calls all_reduce(tensor, op=op, group=group, async_op=True) unconditionally. The double at test_unified_radix_cache_unittest.py:2382 is fake(tensor, op=None, group=None). All 16 die with a TypeError inside mock's dispatch BEFORE their first assertion, so they are not evidence about SWA prefetch adoption in either direction. Checked the other half of the call too rather than assuming: the double returns None, and hicache_collective.py:190-196 documents None as a completed no-op -- so the drift is exactly one keyword wide and a one-line fix is the whole fix. TEST-ONLY. Zero live defects. sgl-project#783 gets nothing from them. Corrects the prior art it rests on: NOTE_861fg's "16 passed" hermetic is a numeric COINCIDENCE, not the same 16. Measured: -k tp_swa_prefetch under CVD="" gives 38 skipped / 0 passed / 0 failed (8+8 for want of a card, 11+11 non-SWA parametrisations; tally 38 == summary 38). They are inside the 1506 skips, not the 16 passes. sgl-project#862. mem_cache hermetic: 2 failed / 1774 passed / 1658 skipped, both tally axes OK (names 2 == summary 2; extracted 1658 == summary 1658). Both failures DEVICE-caused; REAL defects 0. The 1658 split by reason: HARDWARE 996 | DEPENDENCY 12 | CONFIG-not-darkness 645 | BY ACCIDENT 0 UNCLASSIFIED 5 So the darkness is BY DESIGN -- BY ACCIDENT is zero and the conftest mechanism explains itself in its own docstring. The ticket is still right for a finer reason: 1008 of those would run on a card and 645 would never run anywhere, and the summary line renders those identical facts. My own classifier was wrong first and the data said so: two buckets put 636 of 1658 in neither. Folding them into "dark" would have inflated the figure by 62 %. All 36 survivors were read individually and encoded; 5 remain unclassified and are reported as such rather than tuned away. Two more corrections to prior art: mem_cache DOES run directory-wide on CPU here (3396 collected, no abort), so the recorded collection-time abort does not reproduce on this tree -- reported, not bisected. And the escape behind it is real but at RUNTIME and two tests wide: rotation_executor.py:344 allocates with pin_memory=True, a device door that never calls get_device(), which is the one door the conftest guard patches. THE CHECK, in the CPU lane on purpose. New test_double_signature_drift_864.py (2 tests, 0.11 s, register_cpu_ci) parses the double and the production call sites and asserts the first can bind the second -- no device, no ranks, no import of a module that needs them. It also asserts production still passes async_op, so the guard reports a change instead of silently testing a call that no longer exists. Can-fail proven: re-introducing the drift turns it red with the exact diagnostic. Regression: mem_cache after the change is 2 failed / 1776 passed / 1658 skipped -- same failures, same skips, +2 from the new guard. NOT done and recorded: the 16 were never run WITH cards, so the fix is proved to remove the drift and not proved to make them pass. Every number here is a CVD="" number, verified on the process; nvidia-smi compute-apps showed only the other runner's serving ranks throughout. (cherry picked from commit a428907)
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 26, 2026
…ndarkened, the Y2 fix, and a mem_cache partition table) into the flip train
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 3, 2026
…follower speaks before it can be stuck
boot_855_weg1b5_cd5bb69607_0903_115008, rid 0c34259f, 11:55:13-11:55:30.
All three ranks issued the same re-admission prefetch (gen 4, keys=13224,
byte-identical FETCH CAP lines). PP0's completed in ~3 s, printed
completed_synced=12288, and ADMITted prefix_lens=12288. PP1's and PP2's had
not terminated. PP1 took the forwarded row into execute_scheduled_prefix and
raised the designed group STOP 14 s later. The number PP0 admitted on was
rank-local by construction: the MIN reduce behind "completed_synced" runs
only under tp_world_size > 1, and this boot is --tp-size 1 --pp-size 3 --
attn_reduce_world=1 on 307/307 HICACHE-ROUND lines of the whole log.
E1 managers/pp_prefetch_completion.py (new, pure, no I/O, no clock) plus a
per-rid completion carrier on the sgl-project#791 ring lap, modelled exactly on
sgl-project#968's parked-continuation table (stamp on the way up, absorb at PP0).
No new all_reduce, no new message, no new blocking point on the
admission path -- the PP0-authority order records a collective there as
fatal. PP0 admits only when the group floor covers what it wants to
schedule; a peer that is silent or still running contributes NO NUMBER
(INCOMPLETE, never a zero), otherwise the rid is DEFERRED by name while
other work proceeds. When the same length-priced bound the follower's
own wait uses expires, PP0 CLAMPS to the group floor through the
existing sgl-project#1059 note_observed_coverage channel instead of over-telling.
want <= 0 (no store span) admits unconditionally, so a boot with no
storage hit takes exactly the pre-sgl-project#1175 path. Kill switch
SGLANG_PP_GROUP_COMPLETION=0 restores the old admission for an A/B.
completed_synced now prints synced=yes|no and attn_reduce_world=N beside
itself, so the field can no longer claim a sync it did not do.
E2 pp_admission_congruence.execute_scheduled_prefix now DRIVES the writer it
used to suspend: after each poll it calls check_prefetch_progress when
the tree reports that path collective-free (world 1 = every all_reduce
skipped by construction), and the expiry text carries
prefetch_driven_in_loop=<bool> so an expiry can never be read as "the
bytes were not there" when nothing drove the sole writer. Bounds
unchanged. The sgl-project#1157 reaper's comment now states where it structurally
cannot fire (it reports terminations; a prefetch that never terminates
produces no line, so 0 REAPED is not evidence that nothing was reaped).
E3 Every follower prints one bounded "sgl-project#1175 PREFIX-EXEC UNDER-COVERAGE"
line at ENTRY, before any bound can expire, naming rid/local/scheduled/
deficit/bound. PP2 printed nothing at all about this rid and its silence
was indistinguishable from health. The line also states honestly that a
rank which never RECEIVES a decision row never calls this function and
therefore still emits nothing -- that case is what E1+E2 remove.
E4 (a) the healthy local == scheduled path no longer returns silently: one
bounded, counted "sgl-project#968 PREFIX-EXEC no-op" line, so "0 materialised lines
on the whole boot" reads as "never needed" rather than "never reached"
(INDIKATOR-GESETZ). (b) the sgl-project#939 census line now names its population
(population=retract_closure_only) -- SEAM_READMIT_ATTR is stamped only
by the retract closure, so the two 13225-token queue-occupant
re-admissions of this boot produced not one census line. The field is
placed BEFORE fence_proceeds so sgl-project#1068's tail pin still holds.
EVIDENCE (DESK-PROVEN; no boot, no GPU)
red-first, parent cd5bb69 + only the three new test files:
16 failed, 3 passed, 1 collection error (pp_prefetch_completion absent)
fixed tree: 36 passed (25 managers + 11 mem_cache)
mutants: 12/12 killed, >= 3 per decision (E1 x4, E2 x3, E3 x3, E4 x2)
matched checks re-run on the fixed tree: test_pp_admission_congruence_791,
test_968_starvation_umbau, test_968_deletion_falsifiers,
test_producer_phase_census_631, test_double_prefill_census_fence_1068,
test_producer_phase_census_wiring_1061 -> 114 passed, 1 failed
(test_the_module_imports_no_torch_at_all -- identical failure on the
parent, "import torch" count 1 on both trees: pre-existing)
bounded suite once, -n 8, hermetic, managers + touched mem_cache files:
parent 262 failed / 5435 passed / 12 errors
fixed 286 failed / 5472 passed / 12 errors
name-level comm: 35 "regressions" and 11 "cures", ALL of them inside
test_unified_radix_cache_unittest.py; 0 outside it, 0 involving the new
files. That module run SOLO and SERIAL is 642 passed / 860 skipped /
0 failed on BOTH trees -- the delta is the known -n 8 crowding family
(sgl-project#862/sgl-project#899), not this diff.
ruff --select=F401,F821,UP037: 2 errors on both trees, the same two
pre-existing ones; black/isort per file identical on both trees for all
five touched production files, all four new files clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Thank you for your contribution, we really appreciate it. The following instructions will help improve your pull request and make it easier to receive feedback. If there are any items you don't understand, don't worry. Just submit the pull request and ask the maintainers for help.
Motivation
Please explain the motivation behind this PR and the goal you aim to achieve with it.
Modification
Briefly describe the changes made in this PR.
Checklist
pre-commit run --all-filesor other linting tools are used to fix potential lint issues.