misc: update bug issue template by zhyncs · Pull Request #727 · sgl-project/sglang

zhyncs · 2024-07-25T10:52:24Z

Thank you for your contribution, we really appreciate it. The following instructions will help improve your pull request and make it easier to receive feedback. If there are any items you don't understand, don't worry. Just submit the pull request and ask the maintainers for help.

Motivation

Please explain the motivation behind this PR and the goal you aim to achieve with it.

Modification

Briefly describe the changes made in this PR.

Checklist

Ensure pre-commit pre-commit run --all-files or other linting tools are used to fix potential lint issues.
Confirm that modifications are covered by complete unit tests. If not, please add more unit tests for correctness.
Modify documentation as needed, such as docstrings or example tutorials.

* Update daily-build-test-npu-innersource.yml * Create mmmu-val.yaml * Update test_disaggregation_decode_tp.py * Update test_disaggregation_decode_dp.py * Update run_suite.py

…thesis, not a veto My desk risk-split withheld the lm_head half on the argument that a ~0.4% per-output-channel error lands on logit DIFFERENCES, where softmax and argmax can see it. The argument stands as a HYPOTHESIS; it is not evidence, and it is not mine to spend as a veto on a decision the measurement can make. Second artifact built on top of the first, so the two differ in exactly one tensor: Qwen3.8-27B-INT8-vocabint8-both carries embed_tokens AND lm_head as I8 [248320, 5120] with BF16 [248320, 1] scales and neither entry left in ignore. 2.7 GB on disk against 1.2 GB for embed-only, both by hardlinking the 17 unchanged shards. The switchover ticket is now THREE arms -- baseline BF16, embed-only, both -- because the middle arm is what makes lm_head attributable: a delta present in C and absent in B belongs to lm_head alone. Quality gate is the club-3090 suite plus determined-answer probes per arm, greedy and fixed-order, scored against the baseline's own A-vs-A floor first because this model is not deterministic across boots. The decision rule is written down BEFORE the run so it cannot be argued afterwards, and it includes the branch where the hypothesis is refuted and both halves ship. 11 tests still green, ruff clean, no boots.

…and the crossing count Independent re-derivation (NOTE_735_arithmetic_check.md, adopted here from an untracked working copy) confirmed all six load-bearing numbers at their claimed values. Three of its findings land on this document. 1. SLOT CEILING used the NOMINAL card total. The spec's 4738 MiB free on the 5090 was taken against 32768 MiB; the live NVML total is 32607, so the real figure is 4577 MiB and the ladder shifts by one slot at every graph pool size: 31/24/21 -> 30/23/20. The headline becomes "roughly 20-23 concurrent mamba slots", and the bf16 no-sgl-project#727 case 8 -> 7. The width-canon lesson applies to totals as well as widths: use what NVML reports, not the sticker. Both columns are shown so the correction is auditable. 2. CROSSING COUNT reconciled with DESIGN_family_fullplan.md, which places lm_head on the 5090. The two counts are 31 and 32 and both are right about different things: 31 is the INTER-LAYER crossing count, which is what the schedule emits and what a transport matches sends to; 32 is the full-forward MOVEMENT count, including the terminal layer's output to the head. Cost figures derived from 31 understate the transport term by ~3%, which is below the resolution of every verdict they support, so they are left as printed and the note says so rather than pretending the gap is not there. 3. The acceptance probe in section 9.1 now names both counts, since that is the one place someone will compare a wire trace against a number. The NOTE is adopted onto this branch rather than committed where it was found: that working copy sits on another lane's branch (fix/699-progress-clock-wiring), and adopting it there would contaminate unrelated work. Docs only; no code change, no test impact.

…ication, and the sgl-project#735 arm flag PRIOR-ART FIRST: the coordinator's 'second artifact' ask (embed+lm_head int8) was already built by 5745a54 as Qwen3.8-27B-INT8-vocabint8-both -- re-VERIFIED here independently from the shard headers and inodes rather than the build report: embed I8+BF16-scale in 00003, lm_head I8+BF16-scale in 00018 (corrected shard table), no vocab entries left in ignore, 17/18 shards inode-shared with -embed and exactly one rewritten (00018) -- the arms differ in exactly one tensor. Verification record in the ticket; the register name 'vocabint8-embed-lmhead' maps to '-both' on disk (doc-fix over rename). RAM discipline held: 99G free, and no requant was re-run against an already-verified artifact. NEW, the window-ready half: - tools/ab_vocab_int8_727.py -- the ticket's protocol as a turnkey runner: artifact verification BEFORE the first boot, then A1, A2 (the A-vs-A floor -- this model is not deterministic across boots), B, C, gates 0/A/B/C per arm, stop at first failure, decision rule applied mechanically with the results JSON written. Boot/suite/perf legs are operator-filled command templates; --mock replaces them with fixtures (desk-written-never-executed: the desk boots nothing). - GATE 0 observable: ct_embedding.py now logs 'INT8-VOCAB ENGAGED' per loaded vocab tensor, so a silent dense fallback (which would show a WORSE VRAM number while proving nothing) is caught by counting lines (0/1/2 for A/B/C). - sgl-project#735 flag, as required: DESIGN_pp_layer_set.md prices GDN slots 21-24 on the 5090 AFTER the sgl-project#727 saving and the full plan places lm_head ON the 5090 -- so ARM C is the arm that satisfies sgl-project#735; arm B funds nothing there. The decision rule's verdict texts carry the consequence both ways (C passes -> slots funded; C fails -> the 21-24 plan loses its funder, ~7 slots on BF16 lm_head). Tests (hermetic, CUDA_VISIBLE_DEVICES=99): test_ab_runner_727.py 23 tests -- every gate driven in BOTH directions, the floor-widens-with- baseline-noise property, the verdict table, abort-on-baseline-failure, two full mocked CLI runs (SHIP-BOTH and ABORT), and the artifact verifier against synthetic checkpoints broken in each way it must catch (BF16 lm_head in C, broken hardlink economy) plus the REAL rig artifacts. Full quantization suite green alongside: 157 passed / 14 skipped. ruff and codespell clean.

…gated, resumable machine scripts/merge_train_pass3.py executes LEDGER_merge_train_0818 (c) mechanically, with the map's own disciplines enforced in code: - GATE: refuses without the comp4-green marker (/spinning/evidence-665-f1/COMP4_ACCEPTED) -- pass 3 stays boot-proof-gated per the ledger's rule; the marker path is a CLI parameter so the refusal is testable. - STANDING RULE first: re-derives the (a) lineage facts against the ACTUAL tip (contains review tip + composite, every step ref present, already-contained refs auto-skip) and aborts if the world moved. - Mapped resolutions ONLY: 'ours' (the sgl-project#754 fixture hunk drop), 'theirs' + load-bearing marker verification (the sgl-project#735 doc numbers 32607/4577), 'manual' (resumable stop with instructions, the sgl-project#727 gate hunks). An UNPREDICTED conflict aborts loudly WITHOUT resolving -- a stale map stops the train, it is never improvised around. - Stop-on-first-failure with per-step targeted suites and a resumable JSON state file; an all-ours pick that becomes empty is skipped as the mapped outcome. Nothing in the script pushes (pinned by test). SMOKED per the desk-written-never-executed rule, and the smoke EARNED ITS KEEP: the --dry-run scratch-clone execution of the real plan caught a ledger error -- d11b29d (sgl-project#740 s5a) is only a diff and conflicts without the original note commit 0480f4b underneath; the executor aborted on exactly that unpredicted conflict. Plan and ledger corrected to the sgl-project#740 PAIR; the re-run completes all 10 steps clean in the scratch. Bonus finding recorded in the ledger: the sgl-project#727 'manual' gate-hunk predictions did NOT fire -- those picks apply cleanly on the comp4 tip. Unit suite (hermetic, synthetic mini-repos): 12 passed -- gate refusal (helper + CLI end-to-end exit 2), unpredicted-conflict abort leaving the conflict for the operator, ours/theirs resolutions, marker-loss abort, manual resumable stop, plan-mirrors-ledger pins, state roundtrip. ruff and codespell clean. The real pass was NOT run; COMP4_ACCEPTED does not exist and no real branch moved.

…ce in one harvest plan, with a proven runner Consolidated from the comp4 gate run (progress.662-F4-r5), WINDOW_TICKET_745/755, NOTE_747 par.8-9, NOTE_738/755, TICKET_727 and the operator ledger. Two hard incompatibilities shape the plan: WINDOW_TICKET_745 Arm 1 excludes the checkpoint interval that sgl-project#758's anchor-cadence observable requires (-> ARM I hicache harvest, ARM II = one flag more), and sgl-project#713's TTFT<3s needs a quiet router while every other loaded gate needs the soak backlog (-> sgl-project#713 is the idle sub-phase BEFORE the backlog, not a separate boot; the 06:44Z soak driver is the load source per the Lastprobe rule). ARM I phases: load-time (sgl-project#738 no-99G-plateau, file-backed-image reclaim), idle-quiet (sgl-project#713, health, corridor), loaded (Gates A/B/C, - sgl-project#757 race-holds, sgl-project#748 all three shapes, sgl-project#744/sgl-project#717 rung-funded flip, - sgl-project#690 refill census, sgl-project#758-2 mamba host resume, WT_745's three lines, corridor minima), teardown (image reclaim). ARM II adds --mamba-checkpoint-interval 8192 for sgl-project#758-1 anchor cadence + NOTE_747 par.8.1-8.3. SEPARATE windows named with reasons: sgl-project#727 four-boot A/B, WT_755 slots A/B (pool-geometry confound), sgl-project#755 metal retraction (mechanism not built -- nothing to measure), sgl-project#709, sgl-project#735 Step-2. sgl-project#602 and sgl-project#536/sgl-project#537 carried as HONEST unresolved slots (owner-held detail / not found with acceptance shape) rather than invented readouts. Runner run_window_ladder.sh: PASS/FAIL/UNOBS table from boot log + live server; never boots, never kills, never touches the soak driver; soak-tolerant by construction (loaded checks are log observations, the one latency check runs only in --phase idle, and choosing that phase IS the operator's quiet-router assertion). Missing-emitter cases (anchor cadence) report UNOBS, never FAIL -- absence of an instrument is not absence of the property. Mock-smoked per the desk rule, both directions: a fixture built from the comp4 specimen lines reproduces the real run's verdicts exactly (GATE-C crash, both sgl-project#748 shapes + vacuous relief, sgl-project#757 sentence = 5 FAIL; sgl-project#744/sgl-project#690 PASS; exit 1) and a clean fixture goes fully green incl. the ARM II cadence line (exit 0). bash -n clean, codespell clean. Nothing was booted.

… end to end, and the A/B reads accept length Second-pass verdict on the re-issued second-artifact task: the artifact half was already DONE (Qwen3.8-27B-INT8-vocabint8-both, independently shard-verified in the previous pass -- embed+lm_head I8+BF16-scale in the corrected shards, 17/18 inodes shared with -embed, exactly 00018 rewritten; the runner's real-artifact check re-ran green here). What the re-issue surfaced as genuinely open was the HEAD-SIDE WIRING question, and the answer is: the chain exists but nothing pinned it. Pinned now (test_ct_lmhead_chain_727.py, 7 tests, both directions, 2 mutants killed): - selection: qwen3_vl:1298 hands quant_config to ParallelLMHead unconditionally; get_quant_method matches the head via the VocabParallelEmbedding isinstance gated on the checkpoint's ignore list -- the arm-B/C discriminator pinned both ways; - GATE 0: the ENGAGED line carries the layer class name, so the two engagements of arm C are tellable apart in the boot log; - routing: should_apply_lm_head_quant_method's DEFAULT arm sends the head matmul through apply() -- listing the method in _UNQUANTIZED_LM_HEAD_METHODS reds 2 tests (the silent-garbage refactor is now loud), and a dense head still skips apply; - numerics: apply() is BIT-identical to the dense head on exact-int8 rows (the checkpoint's own contract). Runner: optional accept_len metric (meta_info-sourced per the acceptance rule, never spec_ema_accept_len) joins GATE B -- near-tie flips decay accept before the suite score moves, the sharpest logit-sensitivity readout the stack exposes. Skipped when either baseline boot lacks it, never invented; 3 new tests incl. the absence-tolerance direction. Hermetic (CUDA_VISIBLE_DEVICES=99): 44 passed across the three 727 suites. ruff and codespell clean.

…pectations, and the sgl-project#735-funder flag Per-arm model paths (A1/A2 baseline, B embed-only, C both), all other flags the composite recipe unchanged with spec ON so the new accept_len readout fires; GATE 0 engagement counts 0/0/1/2, GATE A per-stage 1212 MiB expectations, GATE B incl. the optional accept-length floor. Arm C flagged as the sgl-project#735 GDN-slot funder in the row itself.

The re-issued lm_head-arm task found the standing state intact and verified it LIVE rather than trusting the reports: the -both artifact re-passed the shard-header/ignore/inode checks through the runner's own verify_artifacts (embed I8+scale in 00003, lm_head I8+scale in 00018 -- the corrected table, re-derived from the indices both times it was checked; 17/18 inodes shared, exactly 00018 rewritten), the head-chain pins are green, and the runner mock-smoke stands. Artifact naming: the register's 'vocabint8-full/embed-lmhead' maps to '-both' on disk -- doc-fix over rename, a verified artifact does not churn. NEW: optional ppl metric in the A/B (suite JSON key 'ppl', perplexity on the fixed probe set). LOWER is better, so the delta direction is INVERTED relative to score -- pinned both ways, including that a ppl DROP never fails. Same optionality contract as accept_len: compared only when both baseline boots report it, skipped otherwise, never invented. GATE B now reads suite score, determined-answer accuracy, perplexity, and spec accept length against the A-vs-A floor. Hermetic (CUDA_VISIBLE_DEVICES=99): 46 passed across the three 727 suites. ruff and codespell clean.

…ntract behind it was wrong TWO DEFECTS, ONE HIDING THE OTHER. The sgl-project#727 requantized checkpoint (Qwen3.8-27B-INT8-vocabint8-embed) generated token soup under plain TP=3 uneven-DCP: greedy "The capital of France is" returned " a a a a a a a". Swapping ONLY the checkpoint to INT8-yarn1.5, same code path and TP vector, restored coherent output at identical throughput (473 vs 474 tok/s at bs=24), so the defect cost correctness and nothing else. ROOT: A GATE THAT COULD NEVER FIRE. qwen3_5.py selected the embedding method with get_name() == "compressed-tensors", but CompressedTensorsConfig.get_name returns the UNDERSCORE spelling "compressed_tensors" (compressed_tensors.py:262-263). The hyphen is what the checkpoint's quant_method field and the prose use; the config names itself with an underscore. So embedding_quant_config stayed None, the embedding was built DENSE, and the checkpoint's int8 rows were cast into a BF16 embedding with no scale applied while the scale had nowhere to land. Serving said so on every rank: "Parameter model.embed_tokens.weight_scale not found in params_dict" (bench_tp3.log:110-112). Nothing else in the sgl-project#727 lineage was wrong. The ignore-list scan answers True for every embed_tokens prefix on this checkpoint (measured), and get_quant_method dispatches VocabParallelEmbedding correctly. The feature was simply never reached, which is exactly why every test that drove the method directly stayed green. SECOND, MASKED BY THE FIRST. create_weights registered weight and weight_scale without output_dim. VocabParallelEmbedding.weight_loader reads that attribute (vocab_parallel_embedding.py:524, branch at :577) to decide whether to narrow the checkpoint tensor to this rank's rows; without it a parameter takes the "copy onto all gpus" path meant for shard-invariant tensors like gptq's g_idx and asserts at :584. Invisible at tp_size 1 -- the whole vocab IS the local shard, which is why the PP=3 layout never saw it -- and fatal the moment the vocab is row-sharded. Fixing only the gate would have turned token soup into a hard TP boot crash, so both halves land together. The per-row scale shards on dim 0 with the rows it belongs to; slicing one without the other pairs every row with a stranger's scale. THE GATE IS NOW TESTABLE. The predicate moved next to the method it selects as is_compressed_tensors_config, matching the normalized family name so a second spelling cannot re-open this, instead of a bare string duplicated in a model file. TESTS. TestItSurvivesTpVocabSharding loads TP=3 shards through the REAL VocabParallelEmbedding.weight_loader and requires the reduced result to equal the TP=1 reference -- confirmed RED before the fix (AssertionError at vocab_parallel_embedding.py:584). TestTheFamilyGateActuallyMatches binds the predicate to the real config object. The existing suite drove one unsharded partition with a dummy loader (weight_loader=lambda *a, **k: None), which is how both defects passed 17 green tests. Now 18 passed, hermetic on CPU. Selection verified end-to-end at the desk: the fixed gate yields CompressedTensorsEmbeddingMethod with ['weight', 'weight_scale'] both carrying output_dim 0, the old gate yields UnquantizedEmbeddingMethod with ['weight'] alone in bf16 -- the production symptom, reproduced. ruff clean on the files I touched (the unused triton import at qwen3_5.py:24 is pre-existing, verified against HEAD). NOT CLAIMED: that the requantized checkpoint now serves coherently under TP=3 end to end. That needs a boot and has not been run.

misc: update bug issue template

e5a3897

zhyncs requested review from Ying1123, hnyls2002 and merrymercy July 25, 2024 10:52

zhyncs merged commit 6010589 into sgl-project:main Jul 25, 2024

zhyncs deleted the temp branch July 25, 2024 10:52

timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025

misc: update bug issue template (sgl-project#727)

ec66ca6

Navigation Menu

Uh oh!

misc: update bug issue template - #727

misc: update bug issue template#727
zhyncs merged 1 commit into
sgl-project:mainfrom
zhyncs:temp

zhyncs commented Jul 25, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

zhyncs commented Jul 25, 2024

Motivation

Modification

Checklist

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant