Conversation
merrymercy
reviewed
Aug 10, 2024
| self.logit_bias[i][: len(int_token_logit_bias)] = int_token_logit_bias | ||
| self.logit_bias[i][ | ||
| : len(self.int_token_logit_bias) | ||
| ] = self.int_token_logit_bias |
Contributor
There was a problem hiding this comment.
We can remove int_token_logit_bias now. It was used before we had regex decoding. But please keep logit_bias because it is still useful and it is in OpenAI API (although we haven't supported it yet)
hnyls2002
marked this pull request as draft
August 10, 2024 19:00
hhhh1252023
pushed a commit
to hhhh1252023/sglang_public
that referenced
this pull request
Jul 28, 2026
Add npu rl testcases.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 30, 2026
…here the PP bytes go sgl-project#1011 -- CHECKED THE LIVE LOG FIRST, as the order required, and the honest answer is that most of it needs no rebuild. Across three boots (restore, final, wb): holds in tp with pending prefill > 0 ............ 0 holds citing tp-decode-floor-s or pp-window-s ... 0 holds citing min dwell ......................... 1345 / 1248 / 1 holds citing IDLE-LOCKED "< break-even N tok" ... 1342 / 1234 / 0 The exits are ALREADY work-gated: the policy never holds in TP while prefill work waits, and the two clocks never appear as a hold reason at all. The only reasons that fire are min-dwell (the sanctioned last thrash guard) and the PRICED economy ("0 tok < break-even 27410 tok"), which is exactly the sgl-project#677/sgl-project#819 form the order wants anti-thrash to have. pp_window_s is dead code on this config by construction: phase_policy.py:3840 gates it on `cap <= 0`, and cap > 0 whenever decode_stall_slo_s > 0 (ours is 180). CHANGED, the one place that did actuate against the drain law: the decode stall cap (phase_policy.py:3828). It was the only thing permitted to cut a drain short, and its own message printed "N tok prefill still pending" while doing it. It now DEGRADES TO A DETECTOR whenever pending_prefill_tokens > 0 -- a named WARNING with wait time and backlog -- and actuates only on an empty backlog. THE ONE HONEST CONSEQUENCE, stated and not softened: under a continuous prefill stream the backlog never reaches zero, so carried decodes wait UNBOUNDEDLY. That is the operator's decision; the warning is the whole mechanism that keeps it visible. BOOT-PROVEN: boot up, coherent (Paris / 143-67=76), 39 cutovers under agent-shaped load, 0 detector fires and 0 actuations -- i.e. no regression. The actuation path was already latent (0 fires in the two prior boots), so this boot proves the change is harmless, NOT that the detector fires. Naming that limit rather than claiming a proof I do not have. sgl-project#1014 -- WHERE THE BYTES GO. Answered from the ledger's own per-rank posts, same boot, both phases (GiB): post PP0 pp -> tp PP1 pp -> tp PP2 pp -> tp weights + runtime state 13.572 -> 4.252 7.408 -> 2.719 8.342 -> 3.662 gapped corridor holdback 1.000 -> 1.000 1.000 -> 1.000 1.000 -> 1.000 mamba state pool 0.731 -> 0.731 0.426 -> 0.365 0.304 -> 0.365 speculative intermediate 0.877 -> 0.877 0.511 -> 0.438 0.365 -> 0.438 prefill activation reserve 1.000 -> (none) 1.000 -> (none) 1.000 -> (none) rest (= the KV pool) 13.875 ->24.195 8.014 ->13.837 8.324 ->13.870 COUNT-CHECK PASSES in BOTH phases: 31.055 / 18.359 / 19.335 GiB = 31800 / 18800 / 19800 MiB = rank_gpu_memory_mib exactly. No unattributed MiB. So the entire PP->TP KV gain decomposes into exactly two terms: weights booked differently 18.69 GiB (29.32 -> 10.63 summed over ranks) prefill activation reserve 3.00 GiB (1.0 per rank, PP only) total 21.69 GiB = the measured KV delta (30.21 -> 51.90) AND THE FIRST TERM DOES NOT SURVIVE INSPECTION. The PP-phase weight posts (13.572 / 7.408 / 8.342, sum 29.32) are right: a layer-wise split holds the whole 27.5 GiB checkpoint plus runtime state. The TP-phase posts (4.252 / 2.719 / 3.662, sum 10.63) are NOT: with the flip vector 32,16,16 the expected shards are ~13.75 / 6.9 / 6.9. The TP weight post is low by ~3x. VERDICT per post, as asked: weights + runtime state ... MIS-BOOKED, not mobilisable. The TP phase's larger KV budget is substantially an accounting artifact, and the 1,274,048-token figure rests on it. Corroborating evidence from the same boot: "TP pool sized to the PP id space: 616670 tokens" and the earlier "max_total_tokens=616670 is larger than the profiled value 450402 -- use the profiled value" -- the inflated budget is largely NOT spent. prefill activation reserve NEEDED-IN-PP (prefill runs there; absent in TP by construction). 3.00 GiB, not free. gapped corridor holdback NEEDED, flip-functional, and per sgl-project#707 the floor is itself measured. Arming floor is 1037 MiB solver-derived (band floor 819 + seam entry 218 + arming margin 192). Shrink-by-measurement only. mamba / speculative NEEDED, and they are already near-identical across phases (<= 0.073 GiB delta). CONSEQUENCE FOR THE 1M CHALLENGE, computed rather than aspired: the world per-token cost is 32,768 B and is layout-invariant (ANALYSE_799 §5.2). 1M tokens therefore needs ~31,250 MiB of KV world-wide against the ~20,300 MiB the PP phase holds today -- about +10,950 MiB of REAL bytes. The ledger above shows no idle posten of that size: the only 18.69 GiB "difference" is a mis-booking in the other phase, not memory sitting unused in this one. So PP does not reach 1M by copying the TP budget. Whether it can reach it at all is a question for the post-#1009a solver with the FA split solved budget-proportionally instead of pinned 8/4/4 -- and the token axis alone is capped at 639,800 (+6.51%) at this cut, also from ANALYSE_799. I am not projecting a number I cannot source.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 30, 2026
… degrades to a detector Inherited as an UNCOMMITTED working-tree change from the predecessor strand (file mtime 2026-08-30 06:46). Named here so every boot from this tree has a patchstand, per "Patchstand vor Last". WHAT IT DOES: while `pending_prefill_tokens > 0`, the decode-stall cap no longer returns PP_TO_TP; it logs "sgl-project#1011 DECODE-STALL DETECTOR (not actuated)" and holds in pp. Only with an empty prefill backlog does it still actuate. The honest consequence is stated in the code comment: under a continuous prefill stream carried decodes wait unboundedly. That is the operator's order (2026-08-30), and the warning is the mechanism that keeps it visible. EVIDENCE TIER: BOOT-PROVEN PRESENT, ACTUATION PATH UNEXERCISED. The change was already live in the std boot (boot_855_std_0840f82601_0830_091223.log, world 591,726, /generate greedy coherent), so it is part of the current baseline, not a new variable. Trap-safe count of marker 'sgl-project#1011 ' in that log: bare 0 / genuine 0 -- the detector branch never fired, i.e. no sample of a stalled decode alongside a non-empty prefill backlog. Not claimed as tested.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 30, 2026
…n, die es nicht gibt
== DER BEFUND (gemessen, nicht vermutet) ==
Boot_855_704bgroup2: 57 Prefill-Batches, NULL Decode-Batches, eine 6-Token-Probe
-- und 42 Flips. Die Gruende standen im Log, es hatte sie nur nie jemand
gezaehlt (`PHASE-POLICY arming`, scheduler.py:13627 -- die Zeile existierte
bereits, ich musste kein Instrument bauen):
21x pp_to_tp: "idle Ns >= Ns, returning to the decode resting layout"
21x tp_to_pp: "pending prefill N tok > 0 (purity: prefill cannot run in tp,
nothing decoding)"
21 Rundtrips auf einer LEEREN Kiste. Die zwei Regeln zertifizierten einander:
diese hier flippte fuer nichts nach TP, die Prefill-Arbeit der Health-Probe
konnte dort nicht laufen, die tp-waertige Regel flippte sofort zurueck. Bei
gemessenen 8,07 s je Rundtrip (4,016 + 4,050 s, RECONCILED `flips`-Tabelle)
sind das ~169 s Naht fuer nichts.
== DIE WURZEL ==
`idle` ist an dieser Stelle der STARKE Leere-Term (`decode_work_bs() == 0 AND
not work_exists()`) -- es ist also BEWIESEN, dass weder ein residenter
Decode-Bundle noch ein Prefill irgendwo geschuldet ist. Die Regel flippte
trotzdem, um sich in der Ruhe-Schicht zu POSITIONIEREN.
Diese Praemisse ist fuer eine KALTE Ankunft verkehrt herum, und eine andere
bekommt eine leere Kiste nicht: ein neuer Request braucht ZUERST einen
Prefill-Pass, und Prefill laeuft unter strict purity nicht in TP. In TP zu
ruhen heisst, die naechste Ankunft zahlt tp_to_pp zum Prefillen UND pp_to_tp
zum Dekodieren. In PP zu ruhen ist fuer genau diese Ankunft strikt besser.
KEINE Dwell-Erhoehung: ein laengerer Dwell macht die Schleife langsamer, nicht
abwesend, und bei NULL Arbeit ist kein Timer lang genug, um den Handel positiv
zu machen (sgl-project#819-Preisfrage, die ein Timer per Konstruktion nicht beantwortet).
== FEHLERKLASSEN-PASSENDER CHECK (Direktaufruf von decide(), 3 Faelle) ==
Klasse a) Format-Spec auf einem Conditional im f-String: beide Zweige
ausgefuehrt -> "idle 0.0s" / "idle 11.5s". AST + Import gruen.
Klasse b) falscher Zweig / Verhungern:
CASE1 idle+leer in PP -> direction=None, Reason traegt sgl-project#1011 PASS
CASE2 Prefill geschuldet -> "prefilling in pp (50000 tok pending)" PASS
(nimmt den idle-Zweig NICHT)
CASE3 Decode-Arbeit da -> direction=pp_to_tp via DRAINED PASS
<- das ist der sgl-project#858/sgl-project#1006-Verhungerungsbeweis: mit echter Arbeit
feuert der Flip weiterhin. Belegt, nicht argumentiert.
== ZWEISEITIGE ABNAHME AUF METALL (boot_855_1011idle) ==
(i) LEERLAUF, 11 min 39 s, Deadman aktiv, 7 health_generate-Proben bedient:
Flips 0 | PHASE-POLICY armings 0 | sgl-project#1011-Refusals 77 | health 200
Vorher auf vergleichbarem Leerlauf: 42 Flips. JETZT NULL.
(ii) LAST, conc=4 x 200 tok: 4/4 fertig, 0 Fehler,
TTFT min 0,30 / med 0,31 / MAX 0,31 s -- keine Verhungerung
Completion 11,38-11,85 s
Flips: GENAU 1, armiert via DRAINED "N req decoding" -- der Flip feuert,
wenn echte Arbeit wartet.
== WAS DIESER SCHNITT NICHT TUT -- und meine eigene frueher zu grosse Behauptung ==
Ich hatte A als groessten Hebel gegen die 68-%-Flip-Steuer benannt. GEMESSEN
FALSCH, und die Korrektur gehoert hierher:
conc=4, 600 tok: 88,64 tok/s (vorher 90,47) -- unveraendert in der Streuung
bs1, 600 tok: 25,09 tok/s (vorher 25,37) -- unveraendert
Unter DAUERLAST sind die Flips nachfrage-getrieben (bs1 zahlt weiter ~2 Flips je
Request: 12 Flips auf 6 Requests), und genau das ist die 68-%-Steuer. Der
Leerlauf-Befund bleibt gueltig und der Schnitt beseitigt reine Verschwendung
ohne Preis -- aber er kauft die Last-Steuer NICHT zurueck.
BELEG-STUFE: BOOT-BEWIESEN fuer beide Abnahmehaelften (Zahlen und n oben).
Der Drain-and-Flip-Kontrakt (sgl-project#925) ist unberuehrt; nur der Leerlauf-Zweig mit
`rest_phase == PHASE_TP` aendert sein Verdikt.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 30, 2026
… die Policy vergleicht == EIN FEHLENDES ARGUMENT, GEMESSEN STATT VERMUTET == `SUBCHUNK-SERVED-IN-TP` feuerte auf boot_855_1011idle NULL Mal, waehrend das sgl-project#887-Gate 11x gewaehrte und die Policy 10x tp_to_pp auf "pending prefill 51 tok > 0" armte. Alle statischen Terme lasen erfuellbar, also war Weiterlesen nicht die Antwort: ein Direktaufruf von `decide()` reproduzierte die LIVE-Zeile byte-genau mit grant=0 und lieferte bei grant=51 die Unterdrueckung -- die Policy-Haelfte ist korrekt, der Grant kam als 0 an. WURZEL: `_pending_prefill_tokens(inflight=None)` -- der eigene Docstring sagt "``inflight`` ... Passing it is what the flip policy MUST do". Die sgl-project#942-Sonde liegt auf dem Flip-Policy-Pfad und uebergab es NICHT. `recv_requests` wertet die Policy AUS, BEVOR die Requests in `waiting_queue` stehen (genau deshalb existiert der Parameter), also mass die Sonde bei einer FRISCHEN ANKUNFT 0, waehrend die Policy den echten Prompt sah. `0 < 0 < chunk` = False -> Grant 0 -> Unterdrueckung unmoeglich, und zwar fuer genau den Fall, fuer den sgl-project#887 gebaut wurde. Ein bs1-Strom macht ausschliesslich frische Ankuenfte. == DER SCHNITT: EIN OPTIONALES DURCHREICHEN, EINE REGEL == `tp_compute_fits_in_one_chunk(scheduler, inflight=None)`. Default None haelt JEDEN anderen Aufrufer byte-identisch -- insbesondere das ECHTE GATE (phase_purity.py:1093) reicht bewusst nichts durch: ein Batch-Builder kann nur bauen, was wirklich in der Queue steht. Die Fit-Regel bleibt EINE Regel (nicht dupliziert), sie beantwortet nur je nach Aufrufer die passende Frage. LECK-FAMILIE UNBERUEHRT (sgl-project#890/sgl-project#906): beide Sonden sind PURE READS, `_spend_tp_compute_chunk` wird ausschliesslich aus `prefill_blocked_here` gerufen. Zu sehen ist nicht zu verbrauchen. == FEHLERKLASSEN-PASSENDE CHECKS == * Gefaehrlichste Klasse: `inflight_reqs` nicht im Scope -> NameError, vom umgebenden `except Exception` verschluckt -> stiller No-op. AST-geprueft: es ist PARAMETER von `maybe_arm_phase_policy` (13020-13664). In Scope. * Verhalten je Aufrufer, Stand-in-Scheduler: frische Ankunft (Queue 0, inflight 51): Gate=False (UNVERAENDERT), Policy-Sonde=True <- der Fix bereits gequeued (51): beide True (unveraendert) ueber einem Chunk (9000): beide False (Refusal erhalten) Legacy-Accessor ohne Argument: True (TypeError-Fallback haelt) * Diagnose-Zeile rate-limitiert (unlimitiert: 1457 Zeilen in EINER Zelle); Zaehler-Vorlauf per Direktaufruf belegt (3 nach 3 decide()). * sgl-project#1011-Nichtregression per Direktaufruf: CASE1 refuses, CASE3 flips. == ZELLEN (gleiche Harness, greedy, 600 tok, fuehrendes Unique-Token) == Zelle vorher (bc43f0d) nachher (dieser Commit) bs1 25,09 n=5 Spr. 4,1% 72,0 n=7 Spr. 69,0% conc=4 88,64 n=3 Spr. 3,4% 134,93 n=3 Spr. 100,8% Flips/bs1-Zelle 12 auf 6 Requests 3 SUBCHUNK-SERVED 0 10 KEIN DOPPEL-PREFILL: `cached_tokens` ist in JEDEM Lauf 0, in schnellen wie langsamen -- der Grant-Pfad fuehrt keinen Re-Prefill ein (sgl-project#939 unberuehrt). EHRLICHE SCHRANKE: die Verteilung ist BIMODAL (bs1-Laeufe 67-80 tok/s, mit einzelnen bei ~23), daher die grosse Spreizung. Der Median bewegt sich stark, die Streuung ist NICHT Rauschen sondern zwei Regime -- welcher Anteil der Requests noch ins alte Regime faellt, ist offen und nicht von diesem Commit beantwortet. BELEG-STUFE: BOOT-BEWIESEN (boot_855_942c, boot_855_942cfinal).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Thank you for your contribution, we really appreciate it. The following instructions will help improve your pull request and make it easier to receive feedback. If there are any items you don't understand, don't worry. Just submit the pull request and ask the maintainers for help.
Motivation
Move the sampler out of ScheduleBatch and try to move it into a part of the forward pass.
Checklist
pre-commit run --all-filesor other linting tools are used to fix potential lint issues.