Repository navigation
chore: bump v0.1.25 - #725
Merged
Merged
Conversation
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 17, 2026
… as data, and the embed-quant checks sgl-project#725. THE BRIEF'S PREMISE DOES NOT HOLD, and saying so is the finding. The task was to locate where the cost model prices quantized-linear lanes as M-independent or with a wrong crossover, and to stop bs=1-decode points being priced through the quant stage. Neither happens: the planner has NO activation-quant axis at all. decode is weight_bytes / effective_decode_bw (key_solver.decode_weight_time), prefill is 2*params / gemm_tflops (key_solver.prefill_compute_time), and quantization enters those two formulas only as a smaller weight-byte count and as which lane rate is picked (sgl-project#324). A T=2 decode point never passes through a quant stage, so there is no misprice to make red. Per the brief's own escape clause this reduces to data-with-provenance plus pins. TWO CORRECTIONS TO THE FRAMING, both load-bearing: 1. The real dispatch decision is not in the planner. It is layers/quantization/fp8_dequant_gemv.py:324 (fused_gemv_applicable, aspect test N >= K) and the sm120 NVFP4 buckets in nvfp4_scaled_mm_sm120.cuh: 192-201 -- per-call runtime branches. 2. sgl-project#368 measured the small-M quant penalty as largely a LAUNCH CONSTANT that CUDA-graph replay removes: int8_quant 0.0266 -> 0.0012 ms (~21x), quant share of the fused op 61% -> 11%. NInfer's thresholds are dispatch branches in a C++ header, i.e. taken per call in an eager engine, and a captured graph cannot branch on T at replay. Wiring an eager-measured crossover into our graph-captured decode path would import an artefact. The table is therefore recorded with NO consumer, deliberately (sgl-project#421's audit class: an instrument that looks wired and is not is worse than one honestly inert). WHAT THE HARVEST IS WORTH TODAY: corroboration, and it is real. Two codebases, two mechanisms, two GPU classes agree on which family must not take the small-M quant/fused path -- the K>N shapes. Our own sm86 aspect gate separates exactly those ([5120,6144] 0.86x LOSE, [5120,17408] 0.90x LOSE). Five of six shapes agree in direction. (d) sm86 stays EMPTY. NInfer measured sm120a only, and sgl-project#368 established sm86 is GEMM-slow rather than quant-dominant, so its crossovers are not merely unmeasured but expected to differ in KIND. An estimate would be a number with no measurement wearing the same type as five that have one. sgl-project#724, checks only, no port. Checkpoint read by headers only (120 and 2624 bytes of header JSON; no shard mmapped, no tensor data touched). (a) YES on all three counts, from two independent places. config.json: tie_word_embeddings False, and quantization_config.ignore names 'lm_head' and 're:.*embed_tokens.*'. Safetensors headers confirm it without trusting the config: lm_head.weight BF16 [248320, 5120], embed_tokens.weight BF16 [248320, 5120]. This CLOSES the one open divergence from sgl-project#725. That shape is exactly NInfer's lm_head row -- the single shape where their "A16 always" points opposite to our N>=K gate (N/K=48.5, selects fused). On this checkpoint lm_head is BF16, so no activation-quant gate ever sees it: moot. (b) NO gap in that form; the kernel exists and IS wired, just scoped. main model qwen3_5.py:1386-1405 builds embed_tokens quantized-resident via GGUFEmbeddingMethod (gguf.py:1532), gated explicitly to GGUF; every other quantization keeps quant_config=None and takes the dense path. NEXTN draft qwen3_5_mtp.py:185-204 set_embed_and_head_modules hands the draft the target's MODULE OBJECTS rather than tensors -- precisely because a packed qweight has no .weight to hand over -- so the draft inherits whatever the main model built and has no independent path to carry a gap. The only real delta is a capability question for its own decision: extending quantized-resident vocab beyond GGUF would require re-quantizing embed and lm_head, which this checkpoint's producer deliberately excluded, so the ~1.1 GiB/rank prize is not available without changing the checkpoint. Tests, hermetic (CUDA_VISIBLE_DEVICES="", no GPU, no model load, no serving): test_activation_quant_crossover_725.py 13 passed + 31 subtests MUTATION PROOF: estimating an sm86 row and making ABSENT answer False turns 3 pins red test/registered/unit/planner 2524 passed, 123 skipped, 188 subtests, 2 failed -- PROVEN PRE-EXISTING by re-running the failing file with both new files moved aside: same 2 fail (test_rejected_evidence_pins.py). ruff + codespell clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
cc @merrymercy @Ying1123
Modification
as titled
Checklist
pre-commit run --all-filesor other linting tools are used to fix potential lint issues.