Allow disabling streaming in bench - #687
Merged
Merged
Conversation
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
shiyu7
pushed a commit
to shiyu7/sglang
that referenced
this pull request
Aug 5, 2026
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
…ed inert under load TWO FINDINGS FROM THE LOADED CORRIDOR RUN, one a fix and one a negative result that stops a default from moving. 1. A RUNG THAT RETURNS NOTHING NOW SAYS SO. The pre-gate KV relief was logged only under `if kv_freed > 0`. On 2026-08-16 the seam was refused 76 times from 06:47:48 with no relief line anywhere in the log, so "the rung returned 0" and "the rung never ran" were indistinguishable -- and the guard's own "reclaimed 0 MiB from [nothing]" was then read (by me, in the sgl-project#687 diagnosis) as the rung being exhausted. IT NEVER SAID THAT. That string is the GUARD LADDER's provider list, and the ladder has exactly two providers, allocator-cache and draft-weights. No KV provider is registered with the guard anywhere in the tree, deliberately: the cap is a group decision and the ladder is rank-local, so the rung's bytes arrive as `kv_freed` BEFORE the guard probes and can never appear in that list however much it paid. The zero line now says that outright, and names the three non-interchangeable causes to check: the admission floor (healthy), an empty evictable set (the pool is genuinely live), a disqualified rung (a defect). 2. SGLANG_FLIP_SEAM_CHUNK_MIB=16 BUYS NOTHING UNDER LOAD. DEFAULT STAYS 0. Row-blocking's own comment required a loaded corridor run before its default could move, because its measured 488.7 -> 305.6 -> 276.5 MiB shrink at B = 1, 4, 16 "was taken at 90 live slots and prices the seam's constant, not its behaviour under a full pool". That caution was right. Two boots of 8974994, same argv, same 4x25625-token load, one variable: chunk OFF 185650 live slots staging 1456.14 / 1354.00 / 2016.64 MiB chunk 16 288904 live slots staging 1657.42 / 1555.57 / 2250.81 MiB Fitting staging = a + b*slots per rank across each boot's own points: slope 0.001953 (off) vs 0.002064 (armed) MiB/slot intercept 1093.6 / 991.4 / 1686.6 vs 1061.3 / 959.4 / 1654.6 MiB Identical within 3%. The knob was verified live in the worker environment, not merely exported. The shrink does not appear because the backing term is `max(ceil(max(com_w, B*com_w - (B-1)*rel_w)/B), min(com_w, chunk_floor))`: with a release leg small against the commit leg the numerator stays near B*com_w and the division cancels, and the arena chunk floor binds what is left. At 90 live slots those terms sit differently, which is precisely why the comment demanded this run. Recorded rather than deleted: the knob and its 16-block default remain, so a successor with a different pool geometry can re-price it without rebuilding the machinery. What must not happen is shipping it as a default on the strength of a 90-slot measurement. BOTH BOOTS SERVED CLEANLY on the sgl-project#687 law-advisory code: 12 and 15 flips, 0 refusals, 0 delays, 0 withholds, 0 drain-mode yields, 4/4 requests each. The law warning fired 2 and 3 times and the seam proceeded every time. TESTS: 2 new source-assertion cases pinning the zero-report branch and the clause that ends the guard-ladder confusion (the same convention this file already uses for the rung's wiring), 29 in that file, 974 across every corridor / phase / purity / seam / margin / kv_backing suite.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
…ed inert under load TWO FINDINGS FROM THE LOADED CORRIDOR RUN, one a fix and one a negative result that stops a default from moving. 1. A RUNG THAT RETURNS NOTHING NOW SAYS SO. The pre-gate KV relief was logged only under `if kv_freed > 0`. On 2026-08-16 the seam was refused 76 times from 06:47:48 with no relief line anywhere in the log, so "the rung returned 0" and "the rung never ran" were indistinguishable -- and the guard's own "reclaimed 0 MiB from [nothing]" was then read (by me, in the sgl-project#687 diagnosis) as the rung being exhausted. IT NEVER SAID THAT. That string is the GUARD LADDER's provider list, and the ladder has exactly two providers, allocator-cache and draft-weights. No KV provider is registered with the guard anywhere in the tree, deliberately: the cap is a group decision and the ladder is rank-local, so the rung's bytes arrive as `kv_freed` BEFORE the guard probes and can never appear in that list however much it paid. The zero line now says that outright, and names the three non-interchangeable causes to check: the admission floor (healthy), an empty evictable set (the pool is genuinely live), a disqualified rung (a defect). 2. SGLANG_FLIP_SEAM_CHUNK_MIB=16 BUYS NOTHING UNDER LOAD. DEFAULT STAYS 0. Row-blocking's own comment required a loaded corridor run before its default could move, because its measured 488.7 -> 305.6 -> 276.5 MiB shrink at B = 1, 4, 16 "was taken at 90 live slots and prices the seam's constant, not its behaviour under a full pool". That caution was right. Two boots of 8974994, same argv, same 4x25625-token load, one variable: chunk OFF 185650 live slots staging 1456.14 / 1354.00 / 2016.64 MiB chunk 16 288904 live slots staging 1657.42 / 1555.57 / 2250.81 MiB Fitting staging = a + b*slots per rank across each boot's own points: slope 0.001953 (off) vs 0.002064 (armed) MiB/slot intercept 1093.6 / 991.4 / 1686.6 vs 1061.3 / 959.4 / 1654.6 MiB Identical within 3%. The knob was verified live in the worker environment, not merely exported. The shrink does not appear because the backing term is `max(ceil(max(com_w, B*com_w - (B-1)*rel_w)/B), min(com_w, chunk_floor))`: with a release leg small against the commit leg the numerator stays near B*com_w and the division cancels, and the arena chunk floor binds what is left. At 90 live slots those terms sit differently, which is precisely why the comment demanded this run. Recorded rather than deleted: the knob and its 16-block default remain, so a successor with a different pool geometry can re-price it without rebuilding the machinery. What must not happen is shipping it as a default on the strength of a 90-slot measurement. BOTH BOOTS SERVED CLEANLY on the sgl-project#687 law-advisory code: 12 and 15 flips, 0 refusals, 0 delays, 0 withholds, 0 drain-mode yields, 4/4 requests each. The law warning fired 2 and 3 times and the seam proceeded every time. TESTS: 2 new source-assertion cases pinning the zero-report branch and the clause that ends the guard-ladder confusion (the same convention this file already uses for the rung's wiring), 29 in that file, 974 across every corridor / phase / purity / seam / margin / kv_backing suite.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
…requests hung LIVE, 2026-08-16 08:03, and my own sgl-project#687 fix caused it. An 18-token request hung for two minutes with all three GPUs at 0% while the policy logged, every ten seconds: holding in tp: pending prefill 18 tok <= N=7004, running it in tp (pending prefill 18 tok, running bs 0) THE POLICY WAS RIGHT AND THE SUPPRESSION OVERRODE IT. Below the break-even N a flip costs more than it saves, so the policy deliberately keeps the work in the TP layout and arms no flip at all. The purity valve cannot rescue that case: the flip is not FAILING, it was never asked for, so `flip_unavailable` is false and drain mode held a request that nothing was ever going to run. EVERY METAL PROOF SO FAR HID IT. The 4-carrier runs use 25625-token prompts, which sit far above N=7004 and therefore go to PP and prefill there. Short requests -- the common case on this box -- could not be served at all under drain mode. A 12-token smoke request is what found it, which is the argument for keeping one in the loop next to the heavy load. THE RULE. Drain mode exists to stop a TP window admitting the work it was entered to escape, and that window is defined by A DECODE BUNDLE IN FLIGHT. With `running_bs == 0` the bundle is finished, there is nothing left to drain, and the only thing suppression can still do is idle the instance. So suppression now requires a running bundle. -1 IS NOT ZERO. An unmeasured count must never be read as "no bundle", or the contract quietly stops applying wherever a call site forgets to pass it. The scheduler passes `len(running_batch.reqs)`; the default of -1 keeps the contract. THE CALL SITE, FOR THE FOURTH TIME IN THIS CHAIN. A condition the hook is never told about is a condition that never fires -- after the `phase_policy_config` attribute name, the env knob, and the refusal count. Pinned by source assertions on both the scheduler's call and the hook's forward, not just on the predicate. TESTS: 5 new cases (a running bundle is still protected, an empty one suppresses nothing, -1 is not read as empty, and both ends of the wiring), 983 across every corridor / phase / purity / seam / margin / kv_backing suite.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 16, 2026
…requests hung LIVE, 2026-08-16 08:03, and my own sgl-project#687 fix caused it. An 18-token request hung for two minutes with all three GPUs at 0% while the policy logged, every ten seconds: holding in tp: pending prefill 18 tok <= N=7004, running it in tp (pending prefill 18 tok, running bs 0) THE POLICY WAS RIGHT AND THE SUPPRESSION OVERRODE IT. Below the break-even N a flip costs more than it saves, so the policy deliberately keeps the work in the TP layout and arms no flip at all. The purity valve cannot rescue that case: the flip is not FAILING, it was never asked for, so `flip_unavailable` is false and drain mode held a request that nothing was ever going to run. EVERY METAL PROOF SO FAR HID IT. The 4-carrier runs use 25625-token prompts, which sit far above N=7004 and therefore go to PP and prefill there. Short requests -- the common case on this box -- could not be served at all under drain mode. A 12-token smoke request is what found it, which is the argument for keeping one in the loop next to the heavy load. THE RULE. Drain mode exists to stop a TP window admitting the work it was entered to escape, and that window is defined by A DECODE BUNDLE IN FLIGHT. With `running_bs == 0` the bundle is finished, there is nothing left to drain, and the only thing suppression can still do is idle the instance. So suppression now requires a running bundle. -1 IS NOT ZERO. An unmeasured count must never be read as "no bundle", or the contract quietly stops applying wherever a call site forgets to pass it. The scheduler passes `len(running_batch.reqs)`; the default of -1 keeps the contract. THE CALL SITE, FOR THE FOURTH TIME IN THIS CHAIN. A condition the hook is never told about is a condition that never fires -- after the `phase_policy_config` attribute name, the env knob, and the refusal count. Pinned by source assertions on both the scheduler's call and the hook's forward, not just on the predicate. TESTS: 5 new cases (a running bundle is still protected, an empty one suppresses nothing, -1 is not read as empty, and both ends of the wiring), 983 across every corridor / phase / purity / seam / margin / kv_backing suite.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.