Tmp fix illegal sample by hnyls2002 · Pull Request #676 · sgl-project/sglang

hnyls2002 · 2024-07-20T09:45:51Z

No description provided.

ispobock · 2024-07-20T13:15:52Z

@hnyls2002 I also faced this illegal sample result issue. In my cases (run llama2-13b-chat model on A100-80G) it can be reproduced by this request from ShareGPT dataset:

curl -X POST "http://127.0.0.1:30000/generate" \
     -H "Content-Type: application/json" \
     -d '{
           "text": "| Article Topic | Target Persona | Target Search Phrase |\n| --- | --- | --- |\n| Understanding Defects Liability Periods in Building Contracts | Developers | Defects Liability Period in Building Contracts |\n| What Homeowners Need to Know About Defects Liability Periods | Homeowners | Defects Liability Period for Homeowners |\n| How Builders Can Manage Defects Liability Periods | Builders | Defects Liability Period Management for Builders |\n| Contractors'\'' Obligations During the Defects Liability Period | Contractors | Contractor Obligations in Defects Liability Period |\n| Benefits of a Defects Liability Period for Developers | Developers | Benefits of Defects Liability Period for Developers |\n| Homeowners'\'' Rights During the Defects Liability Period | Homeowners | Homeowner Rights in Defects Liability Period |\n| Importance of Documenting Defects During the Liability Period | Builders | Documenting Defects in Defects Liability Period |\n| Defects Liability Period Extensions and Their Implications | Contractors | Extensions of Defects Liability Period |\n| How to Ensure Compliance with the Defects Liability Period | Developers | Ensuring Compliance in Defects Liability Period |\n| Common Misconceptions About Defects Liability Periods | Homeowners | Misconceptions About Defects Liability Periods |\n| Legal Implications of Defects Liability Periods for Builders | Builders | Legal Implications of Defects Liability Periods |\n| Remedies Available for Homeowners During Defects Liability Periods | Homeowners | Remedies for Homeowners in Defects Liability Period |\n| How to Prepare for the End of Defects Liability Period | Developers | Preparing for End of Defects Liability Period |\n| Understanding Defects Liability Periods in Subcontracting | Contractors | Defects Liability Period in Subcontracting |\n| Best Practices for Managing Defects Liability Periods | Builders | Best Practices for Defects Liability Period Management |",
           "sampling_params": {
               "temperature": 1,
               "ignore_eos": true,
               "max_new_tokens": 200
           }
         }'

If I don't set the top_k, the first decoded token_id will be illegal (1057103872)
If I set top_k=1, the 78th decoded token_id will be illegal (-1062346752)

It seems like an issue for top_k_top_p_sampling_from_probs kernel. (ref: flashinfer-ai/flashinfer#384)
Hope we can find the root cause soon.

…he wedge was mine LIVE INCIDENT 2026-08-16 06:47:48, and the defect is in hot fix 2. The policy armed tp_to_pp every ~3 s and CorridorGuard refused the seam staging on two ranks with STATIC numbers, 76 refusals in a row: PP1 staging needs 1651 MiB -> want 2163 (+512 entry margin) free 2456, arming floor 1536 -> 293 short PP2 want 2858, free 3560 -> 702 short, "every provider is exhausted" PP0 cleared THE FLOOR GUARANTEES LESS THAN THE SEAM NEEDS. The wants (2163-2858) exceed the arming floor (1536) because staging scales with the live KV cells a 4-carrier bundle holds, while the floor was solved against a smaller draw. Every degradation had already stood down -- abandon cap, backoff, entry margin -- and it still could not fund. THAT ALONE WOULD HAVE BEEN A SLOW BOOT. It became a TOTAL wedge because of hot fix 2. Before drain mode, a refused tp_to_pp still prefilled in the TP layout, so the backlog drained slowly instead of not at all. Suppressing prefill in TP removed that fallback: an unfundable seam became an idle server with 727004 tokens waiting and nothing running in either layout. The rule hot fix 1 was built on -- a failure must degrade to the fallback, never to a wedge -- applies to hot fix 2, and I did not apply it. This does. THE YIELD. Once tp_to_pp has been refused DRAIN_SUPPRESSION_YIELD_AFTER times in a row, drain mode stops suppressing prefill in TP and the layout goes back to draining the backlog slowly. The threshold is 2 and is NOT a new number: the seam entry margin already yields after two consecutive abandoned attempts, and an instance should not wait longer to stop IDLING than it waits to lower its own guard. NOT A LATCH. `arm_refusals` is reset by the first successful arm, so an instance that recovers returns to the user's semantics by itself. This chain has spent four tasks removing one-way ratchets and is not adding a fifth. THE CALL SITE CARRIES THE COUNT, because a yield the hook never learns about is a yield that never happens -- the third instance of that shape this file has caught (the `phase_policy_config` attribute name, the env knob, now this). Pinned by `test_the_purity_hook_passes_the_refusal_count`, which drives the real hook with a state carrying 76 refusals. WHAT THIS DOES NOT FIX, and is the real sizing defect underneath: the arming floor does not cover a full bundle's staging. Candidates weighed: (a) chunk the staging so per-wave want fits under the guard -- the seam already runs in 16 waves, so this is a wave-size question, not new machinery; (b) derive the floor from max_running x per-req staging cells, which feeds sgl-project#676 pool sizing and costs pool; (c) bounded degradation. This commit is (c), chosen because it is the only one that unwedges a LIVE instance without a sizing change, and because the fallback it restores is the behaviour the instance had two hours ago. (a) is the structural fix and is the follow-up. 5 new hermetic cases: suppression holds while the flip is viable, yields at the threshold and at the incident's 76, the threshold matches the seam margin's own, the yield is not a latch, and the hook passes the count. 33 in the two sgl-project#677 files, 266 across every phase-policy/purity suite.

…ut arming floor consumed Three deliverables from the solver conflict, plus a retraction of my own rev4 validation claim. (1) ADJUDICATED KV-SCALING RULE -- Slot-3 is right, verified against the allocator, not taken on report. Token-scaling KV lives ONLY on full-attention layers: HybridLinearKVPool is documented "KV cache with separate pools for full and linear attention layers" (memory_pool.py:3606), full_kv_pool takes layer_num=self.full_layer_nums (:3688), and full_layer_nums = len(full_attention_layer_ids) (:3637). Linear/GDN layers hold per-SEQUENCE MambaPool slots, which is a residency subtraction and never a divisor. My rev3 divided by the rank's TOTAL layer count and was wrong. RETRACTION of my own rev4 test. I called two live boots a "metal discriminator". They are not, on two counts. First, this checkpoint places attention uniformly every 4th layer, so any multiple-of-4 cut has total/attn == 4 on every rank and the two rules agree exactly -- and BOTH available points are multiple-of-4. Second, I had solved two free constants from those two points, so reproducing them was circular. The rule is adopted on the allocator source. A test now pins the non-discrimination so nobody repeats the claim. (2) PER-LAYOUT SOLVED ARMING FLOOR. F4-r4's prime suspect is confirmed and quantified rather than asserted: rank0's floor rises 1728 -> 2255 MiB between the two layouts, and at 8 attention layers and 2048 B/token that is 33,728 tokens against a 39,919-token over-prediction -- 84 percent of the +9.6 percent common mode, leaving ~1.5 percent residual. arming_floor_mib is now a REQUIRED per-rank field with no default, so a constant cannot be passed by accident, and the docstring points callers at arming_floor_target_bytes (sgl-project#676). Known gap recorded in the docstring rather than smoothed: that solver derives the floor from a MEASURED seam draw, so an unbooted layout has no solved floor. Predicting one needs a draw-versus-layout model that does not exist. Every row for an unbooted cut therefore carries its proxy's uncertainty, about +-500 MiB = +-32,000 tokens = ~7 percent. (3) BACKTEST of both metal points: incumbent 436,767 against live 436,766, and [32,16,16] 416,794 against live 416,796, with the binding rank switching r1 -> r0 across them. [42,11,11] is treated as an INEQUALITY, not a point, because it OOM'd and never solved a pool; the model puts it far below the pin and no constant is fitted to it. Calibration honesty guard added: two metal points give two equations for three ranks, so rank2's free constant is BOUNDED, not identified -- a test perturbs it and shows neither reproduced point moves. (4) RE-EMITTED CUT TABLE, and it changes the recommendation. [33,15,16], the cut queued for boot, is predicted at 387,949 -- 11.2 percent BELOW the incumbent, i.e. it would fail the same pool gate that [32,16,16] failed at -4.4 percent. Better candidates on both axes: [30,18,16] at 539,003 (+23.4 percent pool, 1.111x pipelined) and [31,17,16] at 509,301 (+16.6 percent pool, 1.176x pipelined). The rev3-vs-rev5 gap at [33,15,16] is +3.12 percent, so that cut does discriminate the rules -- but only by about twice the residual and half the floor uncertainty, which is worth knowing before spending a boot on it. Tests: 12, red first. Planner suite 2622 passed, 2 failed -- both the pre-existing test_rejected_evidence_pins failures verified earlier against clean integration/r2. ruff clean. Canonical-implementation convergence with Slot-3's parallel pp_cut is NOT done here and needs the review gate; this commit makes the rule and its calibration limits explicit so the merge has one adjudicated form to converge on.

…h reserve is the blocker Review-gate item 2. Every term from config or instruments, nothing fitted. KV term is byte-exact from config, not fitted: kv_cache_dtype fp8_e4m3 (1 byte) x num_key_value_heads 4 x head_dim 256 = 1024 B per token per ATTENTION layer for K, 2048 B for K+V. The boot log's per-rank K sizes at 436,766 tokens match on all three ranks (2.92 / 2.08 / 1.67 GiB predicted and logged). This falsifies the 4096 B bf16 constant in Slot-3's doc directly. Mamba residency derived from the allocation sites (memory_pool.py:583-608, :655-665, :693-705) with max_mamba_cache_size 12 (13 slots), max_running_requests 4 (5 spec slots), speculative_num_draft_tokens 4, temporal_state_shape (48,128,128), conv_dim 10240, win 3, bf16: temporal_state 13 x 48x128x128 x 2B = 19.50 MiB/GDN-layer conv_state 13 x 10240x3 x 2B = 0.762 intermediate_ssm_state_cache 5 x 4 x 48x128x128x2B = 30.00 intermediate_conv_window_cache 5 x 10240x6 x 2B = 0.586 total = 50.85 MiB/GDN-layer The gate's 19.5 MB/GDN-layer is the temporal component ALONE. The full residency is 50.85, dominated by the SPECULATIVE intermediate cache at 30.00, so a model charging only 19.5 under-charges GDN layers by 2.6x. Forward validation against the live [28,20,16] boot chain (avail-after-weights minus avail-after-pool): predicted 6.874 / 4.910 / 3.928 GiB against measured 6.91 / 4.95 / 3.97 -- under 1.1 percent on every rank. The residual is nearly CONSTANT at ~40 MiB rather than scaling with layers or attention, which is the evidence that the per-layer terms are complete: a missing per-layer term would have shown a slope. CORRECTION to my own blocker claim. I previously named the +-7 percent unbooted-floor uncertainty as what blocks bootable predictions. Wrong in emphasis. Retro-prediction runs the equation backwards and needs everything the sizer sets aside before the pool; every term is now config-derived except the GRAPH RESERVE, which the chain puts at 6,372 / 2,650 / 2,377 MiB per rank after subtracting the measured floors. Next to that the floor uncertainty is second-order. The fix is not to model the graph reserve independently but to have the pool solve consume the sizer's own reserve terms, as it must already consume the sgl-project#676 floor. Structural finding encoded (gate 3.5): no cut keeping rank2 = layers 48-63 can beat the incumbent pool. Rank2 is byte-identical across both boots (weights 10.40 GiB in each log) and binds at 436,766, a hard min-rule ceiling. Pool gains require shrinking rank2's ATTENTION count or Part B decoupling, not rank0/rank1 rebalancing. Also on record: the incumbent's binder is PP2 per the boot log (PP1 cap 463,406, PP2 436,766). My rev5 calibration assigned it to rank1 and solved a free constant from that assumption; this derivation needs no such fit and supersedes it. Docs only. Items 1 (canonical pp_cut convergence) and 3 (sgl-project#701 slice-3 rework) not started.

…take the four gate corrections The review gate adjudicated the attention-only divisor in this branch's favour -- white-box at the allocator, byte-exact against the boot log's K sizes -- and found four defects of mine. This lands the convergence and all four. ## Convergence: one canonical solver Merged c5afff7 (Slot-2 rev5), which adopts the attention-only rule and adds the per-layout arming floor and mamba terms. My duplicate FamilyPoolModel / stage_family_capacities / family_phase_pool are DELETED; rev5's PhasePoolModel is the only pool model. Slot-2 leads the solver; this branch owns layout_ladder, ladder_controller and the arena model, which now consume rev5 instead of reimplementing it. Two additions to pp_cut.py that do not duplicate rev5: - kv_mib_per_token_per_attn_layer_from_config() + kv_dtype_width_bytes() - decoupled_phase_pool(), the part B projection, rebased on PhasePoolModel ## E1: the KV cell was FITTED, and was wrong by 2x DESIGN_704 read the cell as 4096 B (bf16). The shipped config is fp8_e4m3 and the cell is 2048 B for K+V, 1024 B for K. Fitting it against an observed pool is what produced both the dtype error and the bogus "0.83 of observed" fudge -- they cancelled into something that looked calibrated. The cell is now consumed from config (2 x kv_heads x head_dim x dtype_width). Unknown dtypes raise rather than default, because a wrong default is a silent 2x on every pool number; 'auto' raises too, since it names no width. ## E2/E3: the binding rank, and the retraction I claimed the incumbent binds on rank1. The boot log says PP2 binds at 436,766. The functional form was right; my free-bytes vector was not, and that produced the retraction: [33,15,16] leaves rank2 holding layers 48-63 -- byte-identical to the incumbent's rank2, measured cap 436,766. Under the min-rule no cut keeping that rank2 can exceed it, so my claimed 457,604 over-predicted an UNCHANGED rank's measured capacity by 4.8%. Expected actual ~387k, about -11%. The arm is withdrawn. [32,16,16] had already failed its gate on metal at 416,796, the same error class. The "discriminating experiment" justification was void: the divisor was settled white-box, so the window would have bought nothing. What replaces it is stronger than the arm I lost: NO cut that keeps rank2 = layers 48-63 can beat the incumbent pool. Pool-positive rungs must shrink rank2's attention count, or wait for part B. That is structural, not a number needing re-measurement. ## E4: GDN residency was under-charged 2.6x, and it falsified a claim The full per-GDN-layer figure is 50.85 MiB, not 19.5: temporal_state 19.5 + speculative intermediate_ssm_state_cache 30.0 (5 spec slots x 4 draft tokens) + conv_state 0.762 + intermediate_conv_window 0.586. This falsified a structural claim of mine. §3.7 said two rungs sharing an attention profile price EXACTLY the same, so the deeper strictly dominates. That held only because the 30 MiB speculative term was missing. Corrected, such rungs differ by their GDN residency (~3,250 tokens at 8 attention layers): a weak real trade, not a domination. test_under_an_arena_equal_attention_profiles _have_equal_pool failed the moment the constant landed and is replaced by test_under_an_arena_same_attention_profile_differs_only_by_gdn_residency, which asserts the gap is explicable by GDN residency and nothing else. ## Part B defects D4/D5/D6 - D4: the byte-identity gate as written could never pass. "decoupled vs coupled" is A-vs-B -- an LSE merge sums in a different float order than monolithic attention -- so it would fail forever on correct code and be waived. Respecified: (1) decoupled-vs-decoupled determinism, byte-identical across runs and boots, CPU-sampled inputs; (2) agreement with the coupled reference within a tolerance fixed before the run. Gate 1 is never waived. - D5: "TP identity wins" was unconditional and cannot be. The measured TP vector puts ~43.8% of KV rows on rank0, the rank with the least free bytes at exactly the deep rungs part B exists to unlock, so prize 1 and prize 3 can contradict. TP identity now carries a per-rung feasibility bound, and the rung yields before the vector does. - D6: committing rung changes only at a fully-quiescent boundary can starve. 640 chunks per max-length prompt with overlapping admissions means the boundary may never arrive -- and ascent is needed precisely while chunked prefills are active. That is the ladder's own sgl-project#701-shaped wedge. Paired with an admission hold plus bounded drain and an urgency-derived deadline; asymmetric, since only a safety-seeking ascent may hold admission. ## Model discipline LadderInputs now REQUIRES an arming_floor_for(counts) provider and refuses construction without one -- a constant floor is the E3 error. Rev5's own docstring records the known gap: the sgl-project#676 solver derives the floor from a measured seam draw, so an unbooted cut has no solved floor, leaving ~±500 MiB (~±32,000 tokens, ~7%) of uncertainty on every unbooted rung. No rung's predicted pool is a boot gate on its own; the retro-prediction gate against all four measured points (434,878 / 435,822 / 436,766 / 416,796) comes first, and Slot-2 owns it. The withdrawn headline table is deleted rather than patched. Re-deriving those numbers before the retro gate passes would repeat the mistake. ## Test results 63 passed, hermetic (CUDA_VISIBLE_DEVICES=""), ruff clean, codespell clean. - test_kv_cell_from_config_704.py (7, replaces test_pp_cut_family_pool_704.py): reproduces the logged K sizes 2.92/2.08/1.67 GB from config alone with zero free parameters; pins that a bf16 reading misses the log by exactly 2x; refuses unknown dtypes and 'auto'; keeps the 0.47-bytes-per-element dimensional record that settled the dispute. - test_layout_ladder_704.py (25) and test_ladder_controller_704.py (12): ported to rev5's API. test_descend_only_happens_at_low_fill now states its thresholds RELATIVE to the solved ladder -- its absolute token constants silently encoded the KV dtype and became meaningless when the cell was corrected. - rev5's test_pp_cut_phase_pool_702.py + test_pp_cut_prefill_speed_702.py (19): green across the merge. The rig fixtures are labelled STRUCTURAL, NOT CALIBRATED PREDICTORS: they pin monotonicity, hysteresis and arena residency, which are invariant to the free vector. No pool value in them is bootable.

This is the instrument that unblocks the four-boot retro-prediction gate. The blocker was not the arming floor and not the graph reserve as a quantity. It was that the sizer builds budget_posts -- its own named decomposition of the KV budget -- on every boot and hands it to budget_exhausted_message ONLY when the budget runs out. The sizer names every term of its arithmetic exactly when it fails, and discards the naming when it succeeds. So the planner had no instrument for the reserve it must not double-count, and had to re-derive it from config in a different process, where it could not even be compared. Three independent re-derivations, three misses against the live [28,20,16] boot: subtracting derived_rank_auto_reserve_mib (4160 MiB, called directly) gave PP2 = 524,620 tokens against a measured 436,766, i.e. +20 percent; the non-budget slack path gave -3.8 percent; the configured-budget variant gave -12 percent. The boot actually takes the budget_mib branch because rank_gpu_memory_mib is set, which is a fourth path again. Re-deriving a number the sizer already computes is the shape this corpus has paid for repeatedly. So the fix is the one the gate's meta-lesson implies: create the instrument and consume it. One info line emits the posts and the final rest_memory on the success path, in the same GiB units the failure message already uses. The pool solve then consumes these exactly as it already consumes the sgl-project#676 arming floor, rather than recomputing them. Not included, and stated rather than implied: the consuming side. PhasePoolModel cannot require a sizer_reserve field until a boot has emitted one -- the posts do not exist in any log on disk today, which is precisely the defect. The next boot produces them, and the consuming field lands with real values instead of a placeholder. Verification: compiles, ruff clean, logger already in scope at module line 168. model_executor suite 594 passed / 15 failed, and the 15 are PRE-EXISTING -- verified by reverting this file, re-running, and getting the identical 594/15, then restoring. This commit adds one log statement and changes no control flow.

…laySSM gate built sgl-project#707 CLOSED. The holdback is not a reserve in the intuitive sense and never was, which is why four attempts to model it as one failed. It is a CAP: allowed_tokens = id_space + (free_at_measure - arming_floor - margin)/cell adjusted_bytes = allowed_tokens * cell holdback_frac = 1 - allowed_tokens / (profiled_bytes / cell) from phase_flip_seam_reserve.floor_allowed_tokens -- "largest id space whose RESTING FREE COLUMN still holds the arming floor". An equality, solved in closed form, reached through the one funnel every sizing path crosses. Verified on all three ranks of the instrumented boot: model 45.141 / 44.072 / 60.256 percent against reported 45.143 / 44.074 / 60.258. Agreement 0.002 pp, the residual being MiB rounding in the logged budget. id_space 435,334 on all three. The three questions answered. (1) The percentage is a CONSEQUENCE of capping the pool so the resting free column still holds the floor, not a parameter anything computes. (2) The PP pass pays because it is the layout a flip departs from and must rest above the floor; the TP-stack pass has no flip to arm from and spends its whole budget -- holdback 0.000 percent on all three ranks confirms. (3) The BINDER holds back most because both extremes meet on PP2: smallest cell (4 attn -> 8192 B) gives it the LARGEST raw token capacity of the three (1,097,728), while its resting free column sits 7.4 MiB above its arming floor -- essentially on it -- pinning allowed at barely more than id_space. It binds for the same reason it holds back most, which is why the two always co-occurred and looked like coincidence. Retired: all fitted candidates, and the [33,15,16] discriminator boot, which now has no calibration value. The rank2 blind spot is preserved explicitly because it still governs TIMING calibration, which this result does not touch. Handed to Slot-3: rung pools can go from extrapolated to EXACT without booting each rung. cell is config-derived, id_space/free_at_measure/margin come from the current seam record, arming_floor from sgl-project#676, and the layout shift is arithmetic over two established constants -- free_at_measure(cut) = free_at_measure(booted) - delta_weights (374.2 full-attention / 476.2 linear per layer) - delta_mamba (51.20 MiB/GDN-layer). What stays extrapolated is only that shift, which is a far smaller claim than carrying another layout's capture behaviour and should be labelled as such rather than as measured. sgl-project#700, both halves built. The wiring contradiction is RESOLVED in-code: the kernel header's "NOT yet wired into the memory pool / radix cache / scheduler / backend dispatch" was stale and false in all four places it named, and the header now carries the file:line evidence for each plus the note that reading it as "inert" cost sgl-project#325 revision 1 a wrong conclusion. The byte-identity half is planner/replayssm_identity.py plus scripts/replayssm/identity_probe.py: A-vs-A floor before A-vs-B, probe capped under the ~109-token GDN determinism ceiling, CPU-sampled inputs only, GDN scalar gate only, and same-tokens-with-nonzero-delta classified as NOT byte-identical rather than rounded up. An unrun measurement is a refusal, never a default-on. The probe refuses before touching a GPU -- verified by running it with 256 tokens. --enable-linear-replayssm stays OFF: wiring resolved, fidelity unmeasured. Tests: 11 for the identity gate, red first. Planner suite 2665 passed / 2 failed -- the pre-existing test_rejected_evidence_pins pair. ruff clean.

hnyls2002 and others added 4 commits July 20, 2024 09:45

tmp fix

4ce0ad7

fix io_struct

0355b48

fix

ed48c8d

update

723ef9c

merrymercy merged commit f424e76 into main Jul 20, 2024

merrymercy deleted the fix-sample branch July 20, 2024 10:11

merrymercy restored the fix-sample branch July 20, 2024 10:16

merrymercy deleted the fix-sample branch July 20, 2024 10:16

timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025

Fix illegal tokens during sampling (sgl-project#676)

5c19900

Navigation Menu

Uh oh!

Tmp fix illegal sample - #676

Tmp fix illegal sample#676
merrymercy merged 4 commits into
mainfrom
fix-sample

hnyls2002 commented Jul 20, 2024

Uh oh!

ispobock commented Jul 20, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

hnyls2002 commented Jul 20, 2024

Uh oh!

ispobock commented Jul 20, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants