Improve logging & fix litellm dependency. - #512
Merged
Merged
Conversation
merrymercy
added a commit
that referenced
this pull request
Jun 7, 2024
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
6 tasks
vschandramourya
pushed a commit
to vschandramourya/sglang
that referenced
this pull request
Feb 3, 2026
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
3 tasks
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 3, 2026
…pert stride; catch-all probe registration Three sgl-kernel csrc fixes, each with an off-GPU can-discriminate test (pure tree change -- wheel NOT built/installed here): - sgl-project#511 (audit sgl-project#506 axis 4, sgl-project#380 class): the GGUF MXFP4 numeric gate was loose enough that an all-zeros or sign-flipped output passed (atol=1.5, rtol=3e1 against RMS 5.1e3 outputs). Replaced with a tolerance DERIVED from the one physical error source (q8_1 activation rounding), plus spread/magnitude preconditions. Every gate in test_gguf_mxfp4_cuda.py now has an off-GPU test proving it rejects a zeroed and a sign-flipped output, including the refuted baseline executed as its own test. - sgl-project#512 (audit sgl-project#506 A1-1): the GGUF MoE MMQ kernel's expert-address byte stride (moe.cuh) was declared `int` (32-bit) while the call site already passed int64_t -- wraps negative once a rank's per-layer expert tensor passes 2 GiB (DSV4-Flash Q4_K, 256 experts, first bad local expert 227; TP=3 sharding is the only reason this rig never hit it). Widened to int64_t through all instantiations. - sgl-project#518: three GGUF capability probes (ggml_moe_get_block_size, ggml_mmvq_kq_tuned, ggml_mxfp4_native) were registered for a DEVICE dispatch key but take no tensor argument, so the dispatcher can never route them -- every call raised "no tensor arguments" before any kernel ran, on every arch. Serving never noticed because gguf.py had grown a python mirror around the raise. Fixed with the same keyless m.impl pattern apply_token_bitmask_inplace_cuda already uses. test_no_tensor_op_dispatch_518.py pins it twice: the dispatcher behaviour on throwaway in-process ops (executed, off-GPU, no wheel needed) and a ratchet over every TORCH_LIBRARY_FRAGMENT schema. FEATURE_CATALOG.md §12 conflict as expected (both this branch and the already-merged sgl-project#510/sgl-project#513 insert a new family paragraph immediately after the §12 header): kept all three additive paragraphs in sequence (Unauthenticated-state-change sgl-project#510, Incomplete-cache-key sgl-project#513, Unreachable-registration sgl-project#81/sgl-project#518, Byte-stride width sgl-project#109/sgl-project#112/sgl-project#512, Tolerance-that-cannot-fail sgl-project#380/sgl-project#511). All 17 sections and every other previously-preserved additive paragraph verified present. Tests (CUDA_VISIBLE_DEVICES=99, PYTHONPATH pinned to this worktree, no wheel rebuild): - The three sgl-project#511-touched test files directly: 29 passed, 14 skipped (the CUDA-gated MXFP4 kernel cases correctly skip without a rebuilt wheel; the off-GPU can-discriminate portions all run and pass). - test/registered/unit/quantization/ (full): 6 failed, 100 passed, 14 skipped, 27 subtests -- all 6 failures confined to test_gguf_dequant_scratch_budget.py, untouched by sgl-project#511's diff. - test/registered/unit/layers/ (full): 3 failed, 646 passed, 39 skipped, 995 subtests -- all 3 failures in test_deterministic_fp8_gemm.py / test_flashinfer_workspace_zeroing.py, both untouched by sgl-project#511's diff. - test/registered/unit/model_loader/ (full, since gguf.py is touched): 32 failed, 229 passed, 9 skipped, 41 subtests. Diffed the FULL failing-test-ID list (not just the tail) against the pre-sgl-project#511 tip (post-sgl-project#514): BYTE-IDENTICAL, 32 lines each side, empty diff. An earlier truncated `tail` comparison looked different due to test-order variance in what scrolled past -- re-verified against the complete list before trusting it. - ruff check --select=F401,F821,UP037: clean. - codespell: clean on all 9 touched/added files including the .cc/.cuh sources.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 3, 2026
… pool -- established unreachable, not fixed sgl-project#487 audits whether the stock even-DCP allocator branch (model_runner_kv_cache_mixin.py, the else of the allocator chain -- which inflates both index space and page granularity by dcp_size, assuming a token-sharded pool) can ever run against a draft worker at --draft-kv-layout replicated (the default), whose pool sizing has the opposite geometry (draft_pool_is_replicated) and which the allocator selection never mentions. sgl-project#108 never audited this crossing. Answer established WITHOUT a boot, by enumerating every producer of is_draft_worker=True rather than assuming there is one shape: UNREACHABLE on CUDA. Given dcp_size > 1 the stock branch is taken exactly when rank_tp_ratio is None and not weightless_kv_active(); (1) a speculative draft worker cannot exist in that shape because ServerArgs._handle_dcp_validation refuses dcp_size>1 + speculation on CUDA unless the boot is uneven-weighted DCP or the weightless fast lane; (2) a sgl-project#274 dual-group-lane runner also sets is_draft_worker=True and is NOT speculative, so leg 1 misses it -- closed instead by _lane_server_args_view forcing view.dcp_size = 1. Leg 2 exists only because "a draft pool implies a speculative algorithm" turned out false once the producer set was enumerated, not assumed -- the new falsifier pins that set so a third producer lands as a red test instead of a wrong address. Residual, named rather than fixed: on HIP/ROCm leg 1 does not run (is_hip() short-circuits before the CUDA branch), so the crossing IS admitted there -- left alone, since this fork does not serve ROCm and a desk-guessed change to address computation is exactly the sgl-project#345 right-token/wrong-slot class waiting to happen. Also folds in docs(sgl-project#482): the ARM3_COMPUTE work-matched counter rule and its script updates (scripts/dev/394_s2_proof/). FEATURE_CATALOG.md §12 conflict as expected (this branch and the already-merged sgl-project#510/sgl-project#513/sgl-project#511 all insert a new family paragraph at the same point, right after the §12 header): kept all six additive paragraphs in sequence (Unauthenticated-state-change sgl-project#510, Incomplete-cache-key sgl-project#513, Unreachable-registration sgl-project#81/sgl-project#518, Byte-stride width sgl-project#109/sgl-project#112/sgl-project#512, Tolerance-that-cannot-fail sgl-project#380/sgl-project#511, Reach-before-fix sgl-project#487). All 17 sections and every other previously-preserved additive paragraph (two-axis §1, matrix doctrine §2, sgl-project#514's five families, sgl-project#499's resolution-ordering, MERGE-DUTY owner-sites + SITREP §12, NVENC §13) verified present. Tests (CUDA_VISIBLE_DEVICES=99, PYTHONPATH pinned to this worktree): - test_stock_dcp_allocator_reach_487.py alone: 20 passed. - test/registered/unit/distributed/ (full, since model_runner_kv_cache_mixin.py is touched): 24 failed, 2167 passed, 8 skipped, 723 subtests -- same 24 failing test IDs as the established baseline, byte-identical; the +20/+2 deltas are exactly this merge's own new test file. - ruff check --select=F401,F821,UP037: clean. - codespell: clean on all 8 touched/added files.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 4, 2026
…test, correct install-state documentation Single commit, docs + test only, zero drift (branch base is exactly this line's prior tip). The sgl-project#398 native-MXFP4 wheel has been installed in the serving venv since 2026-08-03 12:37 (direct_url.json names the pinned wheel, sha 67f03cfa) -- so MXFP4_NATIVE flipped True and the load-time MXFP4->Q5_0 repack became a no-op there, silently invalidating the premise test_gguf_mxfp4_native.py's _FakeNativeOp helper depended on. The helper probed hasattr(torch.ops.sgl_kernel, "ggml_mxfp4_native") BEFORE importing sgl_kernel; torch only registers ops when the extension .so loads, not when the namespace is touched, so on the now- installed wheel the probe still answered "absent", the fake schema got defined, and the real .so loading moments later inside _reload_gguf registered the same schema a second time -- a C++-level duplicate- registration abort ("Fatal Python error: Aborted"), not something pytest can report as a failure, killing the file mid-run and taking 6+ remaining tests with it silently. Fix: import sgl_kernel (best-effort) before the hasattr probe, so the probe answers about the wheel actually on disk. The absent-wheel branch is preserved -- a genuinely missing wheel still takes the fake path, which the dispatch-flip's red arm needs. Falsifier executed both directions: unfixed, the file aborts after 6 tests (reproduced with and without pytest-randomly, and independently in four lines outside pytest -- define the fake, import sgl_kernel, abort); fixed, 16 passed. TICKET_398_mxfp4_validation.md gains a top-of-file correction (the ticket header and the original merge message both said "built, not installed" -- that stopped being true on 2026-08-03 12:37, proof cited rather than inferred: direct_url.json's sha plus a live hasattr probe returning True) and a SS7 continuation note for a successor without this session's memory: both kernel stages are already merged (08bde23, do not rebuild), the gate is a pure registration probe so sgl-project#518's dispatch-key defect never reaches it, Gate A on GPU is next (window sgl-project#537), and the still-unbuilt sgl-project#512/sgl-project#518 kernel-bundle wheel is confirmed NOT on MXFP4's critical path (checked against current preconditions, which are not met -- the DSV4F/translator boot is live on all three cards). FEATURE_CATALOG.md section 8 sharpened at merge time (the branch itself doesn't touch the catalog): the existing "GPU-pending" line didn't carry a false claim, but per instruction it now states the install proof explicitly (wheel sha, live hasattr probe) and names window sgl-project#537 by the same number the ticket's own continuation note uses, rather than a bare "GPU-pending" cross-reference. Verification, hermetic, CUDA_VISIBLE_DEVICES=99, PYTHONPATH pinned to this worktree. Serving (30030 production tenant, 30800 translator front door, user testing live) confirmed untouched -- 30030 on its unchanged PID throughout; nothing here boots a server or touches a GPU: - test_gguf_mxfp4_native.py: 16 passed (was: interpreter abort). - test_gguf_mxfp4_cuda.py + test_gguf_mxfp4_dsv4f_moe_479.py + test_gguf_mxfp4_bridge.py: 30 passed, 24 subtests, 14 skipped (the 14 are the GPU-only Gate-A set, consistent with SS7's "next" item). - test/registered/unit/model_loader/, BOTH wheel states (default and SGLANG_GGUF_MXFP4_NATIVE=0): IDENTICAL both times, 341 passed, 0 failed, 15 skipped, 67 subtests passed. - ruff --select=F401,F821,UP037 and codespell: clean on all 3 touched files. All 17 catalog sections and every previously tracked additive paragraph verified intact. - No rebuild performed or needed: change is test-only, the installed wheel already carries every sgl-project#398 kernel.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.