Compat with latest VLLM 0.4.2 main + fork.number rename + Flashinfer 0.0.4 - #380
Merged
Merged
Conversation
no need to int cast
merrymercy
requested changes
Apr 30, 2024
Contributor
|
@Qubitium Thanks for the contributions! We would like to merge this soon. |
Contributor
Author
|
@merrymercy The vllm main is moving quite fast. We have not tested main compatibility as of today and staff is on a 5 day vacation so response will be delayed. |
Contributor
Author
|
@merrymercy Remerged with vllm main as of today and confirmed working. |
# Conflicts: # python/sglang/srt/utils.py
merrymercy
approved these changes
May 11, 2024
Contributor
|
@Qubitium @ZhouXingg Thanks! |
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
…0.0.4 (sgl-project#380) Co-authored-by: ZX <zx@lbx.dev> Co-authored-by: ZhouXingg <165115237+ZhouXingg@users.noreply.github.com>
vschandramourya
pushed a commit
to vschandramourya/sglang
that referenced
this pull request
Feb 3, 2026
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 1, 2026
…t held to its own standard), reject arms assert their named guard (conjunction fixed), nine arm definitions repaired incl. drop_flags with loud no-op refusal, stale reject retired into a boot arm pinning the new sgl-project#108 contract, axis reader on ACHIEVED= lines with mixed reported loudly, fatal detector respects ignore-framed tracebacks (sgl-project#380)
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 3, 2026
Adds docs/dev/AUDIT_506_unswept_axes.md (1231 lines): a targeted sweep of four axes this fork's existing audits (sgl-project#500, sgl-project#492) never covered -- index/dtype overflow (native byte-stride, not Python index tensors), state-changing HTTP endpoints (auth/CORS), persistent-cache key completeness (the sgl-project#241 class generalised), and test honesty (the sgl-project#380 "cannot fail" class). Top PRIO findings: /hibernate takes an unvalidated hibernate_dir override from the request body and accepts GET (A2-F1); the api-key middleware is skipped entirely unless a key is configured AND a route carries ADMIN_FORCE, so the fork's newest state-changing routes are open by default, plus wildcard CORS with credentials defeats the loopback-only bind (A2-F2/F3); the only numeric gate on the native GGUF MXFP4 kernels is loose enough that an all-zeros or sign-flipped output passes (A4-F1); a GGUF MoE MMQ byte-stride computed in 32-bit crosses at 256 experts on DSV4-Flash Q4_K (A1-1). Pure docs, no code diff (git diff --stat d653405..HEAD -- '*.py' empty). Clean merge, single new file, no conflict. codespell clean. Scrubbed for personal email/PAT/private-IP leakage (grep for gmail/googlemail, github_pat/ghp_, 192.168., 10.10.10., the user's private address) -- no hits; the only IPs present are loopback (127.0.0.1), 0.0.0.0 bind examples, and the well-known 169.254.169.254 cloud-metadata address used as an attack-payload example, none of which are a leak.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 3, 2026
…pert stride; catch-all probe registration Three sgl-kernel csrc fixes, each with an off-GPU can-discriminate test (pure tree change -- wheel NOT built/installed here): - sgl-project#511 (audit sgl-project#506 axis 4, sgl-project#380 class): the GGUF MXFP4 numeric gate was loose enough that an all-zeros or sign-flipped output passed (atol=1.5, rtol=3e1 against RMS 5.1e3 outputs). Replaced with a tolerance DERIVED from the one physical error source (q8_1 activation rounding), plus spread/magnitude preconditions. Every gate in test_gguf_mxfp4_cuda.py now has an off-GPU test proving it rejects a zeroed and a sign-flipped output, including the refuted baseline executed as its own test. - sgl-project#512 (audit sgl-project#506 A1-1): the GGUF MoE MMQ kernel's expert-address byte stride (moe.cuh) was declared `int` (32-bit) while the call site already passed int64_t -- wraps negative once a rank's per-layer expert tensor passes 2 GiB (DSV4-Flash Q4_K, 256 experts, first bad local expert 227; TP=3 sharding is the only reason this rig never hit it). Widened to int64_t through all instantiations. - sgl-project#518: three GGUF capability probes (ggml_moe_get_block_size, ggml_mmvq_kq_tuned, ggml_mxfp4_native) were registered for a DEVICE dispatch key but take no tensor argument, so the dispatcher can never route them -- every call raised "no tensor arguments" before any kernel ran, on every arch. Serving never noticed because gguf.py had grown a python mirror around the raise. Fixed with the same keyless m.impl pattern apply_token_bitmask_inplace_cuda already uses. test_no_tensor_op_dispatch_518.py pins it twice: the dispatcher behaviour on throwaway in-process ops (executed, off-GPU, no wheel needed) and a ratchet over every TORCH_LIBRARY_FRAGMENT schema. FEATURE_CATALOG.md §12 conflict as expected (both this branch and the already-merged sgl-project#510/sgl-project#513 insert a new family paragraph immediately after the §12 header): kept all three additive paragraphs in sequence (Unauthenticated-state-change sgl-project#510, Incomplete-cache-key sgl-project#513, Unreachable-registration sgl-project#81/sgl-project#518, Byte-stride width sgl-project#109/sgl-project#112/sgl-project#512, Tolerance-that-cannot-fail sgl-project#380/sgl-project#511). All 17 sections and every other previously-preserved additive paragraph verified present. Tests (CUDA_VISIBLE_DEVICES=99, PYTHONPATH pinned to this worktree, no wheel rebuild): - The three sgl-project#511-touched test files directly: 29 passed, 14 skipped (the CUDA-gated MXFP4 kernel cases correctly skip without a rebuilt wheel; the off-GPU can-discriminate portions all run and pass). - test/registered/unit/quantization/ (full): 6 failed, 100 passed, 14 skipped, 27 subtests -- all 6 failures confined to test_gguf_dequant_scratch_budget.py, untouched by sgl-project#511's diff. - test/registered/unit/layers/ (full): 3 failed, 646 passed, 39 skipped, 995 subtests -- all 3 failures in test_deterministic_fp8_gemm.py / test_flashinfer_workspace_zeroing.py, both untouched by sgl-project#511's diff. - test/registered/unit/model_loader/ (full, since gguf.py is touched): 32 failed, 229 passed, 9 skipped, 41 subtests. Diffed the FULL failing-test-ID list (not just the tail) against the pre-sgl-project#511 tip (post-sgl-project#514): BYTE-IDENTICAL, 32 lines each side, empty diff. An earlier truncated `tail` comparison looked different due to test-order variance in what scrolled past -- re-verified against the complete list before trusting it. - ruff check --select=F401,F821,UP037: clean. - codespell: clean on all 9 touched/added files including the .cc/.cuh sources.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 3, 2026
… pool -- established unreachable, not fixed sgl-project#487 audits whether the stock even-DCP allocator branch (model_runner_kv_cache_mixin.py, the else of the allocator chain -- which inflates both index space and page granularity by dcp_size, assuming a token-sharded pool) can ever run against a draft worker at --draft-kv-layout replicated (the default), whose pool sizing has the opposite geometry (draft_pool_is_replicated) and which the allocator selection never mentions. sgl-project#108 never audited this crossing. Answer established WITHOUT a boot, by enumerating every producer of is_draft_worker=True rather than assuming there is one shape: UNREACHABLE on CUDA. Given dcp_size > 1 the stock branch is taken exactly when rank_tp_ratio is None and not weightless_kv_active(); (1) a speculative draft worker cannot exist in that shape because ServerArgs._handle_dcp_validation refuses dcp_size>1 + speculation on CUDA unless the boot is uneven-weighted DCP or the weightless fast lane; (2) a sgl-project#274 dual-group-lane runner also sets is_draft_worker=True and is NOT speculative, so leg 1 misses it -- closed instead by _lane_server_args_view forcing view.dcp_size = 1. Leg 2 exists only because "a draft pool implies a speculative algorithm" turned out false once the producer set was enumerated, not assumed -- the new falsifier pins that set so a third producer lands as a red test instead of a wrong address. Residual, named rather than fixed: on HIP/ROCm leg 1 does not run (is_hip() short-circuits before the CUDA branch), so the crossing IS admitted there -- left alone, since this fork does not serve ROCm and a desk-guessed change to address computation is exactly the sgl-project#345 right-token/wrong-slot class waiting to happen. Also folds in docs(sgl-project#482): the ARM3_COMPUTE work-matched counter rule and its script updates (scripts/dev/394_s2_proof/). FEATURE_CATALOG.md §12 conflict as expected (this branch and the already-merged sgl-project#510/sgl-project#513/sgl-project#511 all insert a new family paragraph at the same point, right after the §12 header): kept all six additive paragraphs in sequence (Unauthenticated-state-change sgl-project#510, Incomplete-cache-key sgl-project#513, Unreachable-registration sgl-project#81/sgl-project#518, Byte-stride width sgl-project#109/sgl-project#112/sgl-project#512, Tolerance-that-cannot-fail sgl-project#380/sgl-project#511, Reach-before-fix sgl-project#487). All 17 sections and every other previously-preserved additive paragraph (two-axis §1, matrix doctrine §2, sgl-project#514's five families, sgl-project#499's resolution-ordering, MERGE-DUTY owner-sites + SITREP §12, NVENC §13) verified present. Tests (CUDA_VISIBLE_DEVICES=99, PYTHONPATH pinned to this worktree): - test_stock_dcp_allocator_reach_487.py alone: 20 passed. - test/registered/unit/distributed/ (full, since model_runner_kv_cache_mixin.py is touched): 24 failed, 2167 passed, 8 skipped, 723 subtests -- same 24 failing test IDs as the established baseline, byte-identical; the +20/+2 deltas are exactly this merge's own new test file. - ruff check --select=F401,F821,UP037: clean. - codespell: clean on all 8 touched/added files.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 5, 2026
…tbeat) Boot 13 ran the census and its log contained NO census line at all. That is not reassuring, it is unfalsifiable: an instrument that is silent when healthy AND silent when unwired cannot be told apart from a dead one, and "armed, counts uniform" reads exactly like "import path never reached on this flagset" (the sgl-project#380 test-honesty trap). Two additions, both cheap: * ONE INFO line per scheduler rank, the first time the tick runs: "collective census armed (rank N): interval=50 iterations, heartbeat=..., families tracked=K (...)". Emitted from the TICK rather than at import on purpose -- an import-time line proves only that the module loaded, while this one proves the wired production path is actually reached. Guarded by a flag so it appears once, not once per iteration. * A coarse local heartbeat (default every 10000 iterations, INFO, carrying this rank's snapshot) so a long-running boot keeps evidence the detector is still ticking. It rides the round counter that is incremented anyway, so the hot path is one extra modulo. SGLANG_COLLECTIVE_CENSUS_HEARTBEAT tunes it; <=0 disables the heartbeat only. Tests: 4 added to test_collective_census_583.py (17 total). * arming line emitted EXACTLY once, naming rank, interval, family count * heartbeat cadence rides the replicated counter * default coarse, overridable, junk-tolerant * can-fail proof: removing the once-only guard turns the arming test red
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 5, 2026
…by line number REGRESSION I CAUSED, and the fragility that let one line do so much damage. _bar1_marker_source.py parses the real emitter sources with `ast` and pulls the format strings out of them -- a good design, built precisely so BAR1 log fixtures cannot drift into hand-typed copies (sgl-project#315, the sgl-project#380 family). But it located each call by ABSOLUTE LINE NUMBER (LINE_BAR1_SETUP = 2236, ...). That coupled the module to the LAYOUT of files it does not own. Adding the collective-census counter to parallel_state.py shifted line 787, the AST lookup found no call there, and the assertion fired at IMPORT -- taking the whole BAR1 marker suite down at COLLECTION (test_gpu_battery_checks_bar1.py + test_bar1_marker_coupling.py, 136 tests). Note the actual trigger was the census BUMPS in parallel_state.py, not the abort-handler edit in barlink_bar1.py: the failing assertion names parallel_state.py:787. A line number is the same fragility class as a hand-typed fixture string: a second, unmaintained copy of a fact the source already states. So the anchors are now the MARKER TEXT itself (MARK_* instead of LINE_*), resolved by `_call_with_marker`: the unique call whose first argument contains the fragment. For the exception class the anchor is its DOCSTRING, not its name -- anchoring on the name would make `bar1_unavailable_class_name` a tautology, when its whole job is to report a rename to the consumers. This is strictly better coupling, not weaker: it survives every edit that does not touch the marker, and still fails loudly on the edit that does -- which is the drift the module exists to catch. Ambiguity (a marker matching two calls) is a loud error naming both, never a silent first-match. Tests: * test_gpu_battery_checks_bar1.py + test_bar1_marker_coupling.py: 136 passed (were: collection error) * robustness proof: inserting 40 unrelated lines above the old pin in parallel_state.py -- the exact edit class that broke it -- still 136 passed * can-fail proof: renaming the emitted marker ("barlink enabled for group " -> "barlink ACTIVATED for group ") fails loudly with a message naming the file, the marker and both things to update * WHOLE test/registered/unit/distributed directory: 2370 collected, 0 collection errors, 2358 passed / 12 skipped / 726 subtests passed * the remaining 3 subtest failures (test_dcp_context_ceiling.py, '_PoolRowStub' object has no attribute 'server_args') are PRE-EXISTING, not from this branch: verified identical at 38e7c69 in a throwaway worktree (3 failed, 17 passed there too). The call site is flashinfer_backend.py:691, which this branch does not touch. Process note: this slipped through because I ran only my own test files. The directory-wide run is the check that would have caught it, and it is what I ran here.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 5, 2026
…o local sgl-project#603 and the sgl-project#583 retraction fix both decide from a once-per-iteration MIN-reduced pool headroom instead of the rank-local `available_size()`. Both kept a fallback for "reduced value not available": * `Scheduler.uniform_min_avail` -- getattr(self, "_uniform_min_avail", None), returning the local pool when the reduce had not run * `ScheduleBatch.decode_mem_avail` -- returning the local pool when `uniform_avail_floor` was unset Justified as "correct for a single rank and for tests", which is true and beside the point: on a MULTI-RANK boot they hand back exactly the rank-local predicate both fixes removed. Any path reaching a decode-mem decision without setting the floor silently restores the desync, with the protection still plainly visible in the source. That is the getattr-default trap (sgl-project#606) -- a guard that reads as armed and is absent at runtime, the same shape as a fixture string nobody ever printed (sgl-project#380). Fix: single rank keeps the local value (nothing to diverge from); a group raises with a message naming the ordering that was violated. The world-size probe (`_group_world_size`) never raises -- a probe that could fail would just move the silent-default problem one level down. Tests: test/registered/unit/distributed/test_uniform_floor_fallbacks_583.py * falsifier: pins the pre-fix silent-local behaviour first, then asserts both paths refuse on a group * single-rank behaviour preserved on both paths (else this is a regression, not a fix) * a present reduced value always wins; a ZERO floor is honoured and not treated as missing (`if not floor` would be the same bug in disguise) * 10 passed; can-fail proof: reverting either guard turns the two refusal tests red Also updates the sgl-project#603 and sgl-project#583 stubs to model `self.ps.tp_size` rather than inventing a flat `self.tp_size` -- a stub inventing a surface the real class lacks is precisely what let the census ship dead. Full run: test/registered/unit/distributed + unit/managers = 2875 passed, 30 skipped, 735 subtests passed. Remaining failures are pre-existing and not from this branch: 3 subtests in test_dcp_context_ceiling.py (verified identical at 38e7c69) and test_bar1_host_cleanup.py's pgrep self-match test, which passes in isolation (28/28) and is sensitive to concurrently running processes. Directory ruff count unchanged at 4.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
…tched off sgl-project#380 CLASS: A TEST THAT COULD NOT FAIL AGAINST THE PROPERTY ITS NAME PROMISED. `test_the_solved_pool_clears_the_operators_bar` asserted that "the solved pool" reaches 495000 -- "within 10% of the 550000 the hand-pin encodes". It computed that from a record built with `per_row_bytes=1.0` and `have_bytes=8 GiB` ("deliberately non-binding here"). This rig's own records carry 424.1 / 550.7 / 2360.3 B/row. So the assertion measured the FLOOR solve in isolation with the seam term switched off, while on metal the seam binds rank 0 at 530237 and the group lands at 435319. A green bar at 495000 coexisted with a 435319 boot for a whole task, and the gap between them was read as 115k of missing pool. The isolation test still has value -- it is what shows the floor arithmetic is not itself the conservative term -- so it is kept and RENAMED to say what it does: `test_the_floor_solve_alone_clears_the_bar_with_the_seam_NEUTRALISED`. THE SHIP BAR NOW USES THE RECORDS THE BOOT ACTUALLY WRITES. Each rank's fixture goes through `write_seam_reserve` -> `read_seam_reserve`, the same pair the boot uses, so a change to the record schema breaks this file instead of leaving it testing a shape nothing writes. The values are this rig's records of 2026-08-16T03:56Z. Reconstructed that way the six solves (three ranks x floor and seam) give a group minimum of ~435696 against the boot's own 435319 -- the fixture reproduces the metal. FOUR PROPERTIES, AND THE FIRST IS THE ONE THE OLD BAR INVERTED: * nothing ships at or above a pool measured UNSAFE. 537076 could not hold its floors on two of three ranks. The old bar pushed UP toward 550000, which is 13k ABOVE that. This one fails if the sizer ever reaches it. * no regression below what the recorded inputs support today. * the withdrawn 550000 is recorded AS withdrawn, with its reason, so a re-introduction is recognisable rather than arriving as a fresh "empirical bound". * THE INVALIDATION MECHANISM: the pin's recorded basis (seam slopes, arming floors) is compared against the records it was derived from. 550000 outlived its regime silently because nothing ever compared it to its inputs; when a change moves a slope or a floor this goes red and the pin is re-derived. Mutation-checked: perturbing one floor by 67 MiB fails the test. AND THE REMAINING GAP IS NAMED, NOT HIDDEN. A fifth case asserts the sizer is still BELOW the demonstrated-safe 482490, so the ~47k is visible in the suite rather than papered over by lowering the safe number to meet the solver. It inverts when the follow-up lands, which is the signal that it can close. SHIP_PIN ITSELF landed in bb1620e, which touched the same module: a frozen `SizingPin` carrying demonstrated_safe=482490, demonstrated_unsafe=537076, the regime commit 48ba9fe ("the pool must reserve for the floor the GATE arms at"), the per-rank slopes and floors it stands on, and the withdrawn pin with its reason. Provenance lives in the artifact precisely so the next regime change invalidates it visibly instead of silently surviving, which is the whole failure this task turned out to be about. Suites: 60 in this file, 2394 across unit/managers + unit/mem_ledger. The 4 failures in that sweep are pre-existing and unrelated, parity-proven earlier against the untouched tree.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
…faster A permanently-red suite hides every new regression (sgl-project#380/sgl-project#585 test-honesty class). Hermetic mem_cache stood at 944 failed / 777 passed / 707 skipped in 248 s. It now runs 777 passed / 1651 skipped / 0 failed in 32 s. CLASSIFICATION, and it is simpler than the count suggested: (a) GPU-required under CVD="" ...... 944 of 944 (100%) (b) stub-drift / sgl-project#624-class ........ 0 (c) genuinely broken code paths .... 0 (d) obsolete tests ................. 0 Every one of the 944 shared a single root: get_device() raising "No accelerator (CUDA, XPU, HPU, NPU, MUSA, MPS) or platform plugin is available". Red meant "no GPU here", never "broken". Verified by taking the non-GPU remainder of the FAILED lines, which is EMPTY. Two secondary findings, both of which made this worse than it had to be: * CustomTestCase._callTestMethod wraps every test in utils.common.retry, which re-raises a bare Exception("retry() exceed maximum number of retries.") WITHOUT chaining the cause (no `from e`). The real one-line reason was discarded, so 841 of 944 reported an opaque retry message. That is why a single environmental cause looked like a suite-wide catastrophe. * the same wrapper retried a DETERMINISTIC environment failure several times each, which was most of the runtime -- hence 248 s -> 32 s. FIX. retry() honours exactly one exception: SkipTest is re-raised immediately and never retried. So a directory conftest converts the environmental error into a skip AT ITS SOURCE, which fixes the masking and the retry storm together and needs no edit to any test. It patches get_device on both the defining module and the package namespace, because the tests bind the name at import time and conftest is imported first. Installed ONLY when there is genuinely no accelerator, so a GPU run is unchanged and any real failure stays red; it fires only for tests that actually call get_device, so the 777 that already pass keep running -- the passing count is identical before and after, which is the check that no green test was silenced into a skip. One test needed an individual marker: test_memory_allocated calls torch.cuda.memory_allocated() directly and never goes through get_device. Marked with skipUnless and the reason, rather than widening the patch to cover torch.cuda. PHANTOM MOCK removed (mine, from the sgl-project#701 harness): tc.mamba_recoverable_size was mocked and NO SUCH ACCESSOR EXISTS -- the real one is mamba_evictable_size (mamba_radix_cache.py:1177, unified_radix_cache.py:3189). Nothing read it, so it never produced a false green, but a mock of a non-existent API teaches the next reader an interface the tree does not have, which is the same sgl-project#380 shape. Both it and its unused `recoverable` parameter are gone, with a comment naming the real accessor. mem_cache + managers together: 2883 passed, 0 failed (baseline 945 failed / 2870 passed). ruff clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reason for PR:
fork(number=N)param tofork(size=N)for clarity.