Fix LiteLLM kwargs - #817
Merged
Merged
Conversation
Ying1123
approved these changes
Jul 30, 2024
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
litmei
pushed a commit
to litmei/sglang_litmei
that referenced
this pull request
Jun 23, 2026
modify 27b ang glm5.1 param
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 23, 2026
…rom memory The first draft of `_blocked_arm()` typed the reason string out by hand and got its tail backwards: "-- exit condition: admission blocked" for the real "-- exit condition: blocked admission". The denylist replay in the same class would never have caught it, because that check keys on three OTHER substrings (IDLE_LOCKED, DRAINED, decode starvation cap) and passes either way. That is exactly how a test comes to assert something about a string no code produces -- the failure mode this ticket exists to fix, reproduced one level up. The specimen is now built by driving the real rules, so it is always the string being shipped, and the class asserts up front that the arm it got really is the blocked-admission exit. Verified independently after the fact: the arm's text is built at phase_policy.py:2186-2192 and ends "-- exit condition: blocked admission". TESTS: 10 passed. Red-first improves from 8/10 to 9/10 against the unmodified tree, because the derived specimen now also fails there. All three mutants stay KILLED (base denylist wrapper restored; blocked exit put back in the allowlist; allowlist inverted back into a denylist). black and ruff clean.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 23, 2026
… the allowlist is empty Operator decision on the fork the previous commit left open, and it turns on a fact from the code rather than on the wrapper's prose. The sgl-project#677 hold was licensed to veto "the plain timer/economics exit". That licence was never really about the arm being a timer; it rested on an UNSTATED ASSUMPTION -- that vetoing the timer leaves some other backstop armed. The legacy pp_window stopwatch destroys the assumption: it sits behind a `cap <= 0` guard, so it fires ONLY when the decode-starvation cap is absent, which makes it the LAST anti-pinning bound in every state where it fires. Vetoing the last bound is an unbounded hold. That is verbatim the condition test_sustained_backlog_still_leaves_pp_via_the_window exists to prevent ("PP returned 'holding in pp' on every call, without end") and the shape of the live wedge family this ticket started from. The assumption outranks the prose, so the stopwatch is an exit as well. THE RULE THAT FOLLOWS, written into the dataclass and pinned by a test rather than left in this message: An arm may carry hold_eligible=True only if, in EVERY state where that arm fires, a SECOND INDEPENDENT anti-starvation bound is armed. THE ALLOWLIST IS THEREFORE EMPTY, and empty is the honest state, not a loss. The seam stays: it is the socket for a future arm that really is backstopped, so such an arm is added by stating the claim instead of re-deriving this whole argument -- and with no member, no arm can be held at all. sgl-project#677's economics is not dead with it; it lives in the window-length machinery and in the threshold repricing (sgl-project#819), on the flip-DECISION side where a flip can be weighed before one is chosen, rather than as a veto on an exit the rules already decided. That wiring pointer is recorded in the code and in the COORD so nobody re-attaches it here. The three anti-starvation tests are left untouched. They are the guarantee carriers, and they now pass because the code agrees with them again. TESTS (hermetic, CUDA_VISIBLE_DEVICES="", PYTHONPATH at this worktree; no boot -- a flip-decision change is acceptable only under load, which is a window post): test/registered/unit/managers/test_hold_allowlist_817.py: 12 passed. New pins: the allowlist is empty and the admission condition is written where it binds (the dataclass a future author actually reads, not the history), the stopwatch is shown from the source to be the arm that fails the condition, and -- as behaviour rather than as a count -- no reachable PP_TO_TP arm can be held while the list has no member. ALL EIGHT wrapper-caused failures are now green. Suite diff over the 25 suites importing phase_policy, branch vs base 587e4c2: 14 named failures -> 6, and the diff contains ONLY fixed entries, no new failure. The remaining 6 are test_vacuous_decode_exit_730.py, pre-existing and unrelated to the wrapper (proven earlier by disabling the wrapper outright: they do not move). Mutants, all KILLED: base denylist wrapper restored -> 15 failed stopwatch back to hold_eligible=True -> 5 failed blocked exit put back in the allowlist -> red allowlist inverted back into a denylist -> 11 failed black, isort, ruff, codespell clean.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 23, 2026
Carries [sgl-project#790] carry-instrument gating and [sgl-project#777] threshold honesty. feat/797 is an ancestor of this head, so the previous stage is subsumed; kept as its own merge for per-ticket attribution. Clean merge. Touches phase_policy.py, which sgl-project#817/sgl-project#820 also touch -- those branch from 587e4c2 and are merged after, see their stage notes.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 23, 2026
Closes the Cluster 4 defect that sgl-project#815 escalated rather than fixed: the sgl-project#677 HOLD wrapper swallowed the blocked-admission exit. sgl-project#817 inverts it into an allowlist exactly as the wrapper's own comment prescribed. Clean merge against the sgl-project#790 phase_policy.py edits.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 23, 2026
[sgl-project#820] removes the hold's TP mirror, which had no reachable caller. sgl-project#817 is an ancestor of this head, so the previous stage is subsumed; kept as its own merge for per-ticket attribution. Clean merge.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 23, 2026
…t#816, sgl-project#810, sgl-project#806, sgl-project#797, sgl-project#790/sgl-project#777, sgl-project#817/sgl-project#820, sgl-project#818 Merge-checklist duty for the nine stages on this branch. Entries follow each section's own house style, and the mechanism text is taken from the commits' measured evidence rather than restated from the ticket titles. §3 KV backing relief + the allocator cap -- UPDATED IN PLACE rather than given a second bullet, because sgl-project#814 and sgl-project#816 are follow-on defects OF the KvRowCap mechanism that bullet already describes: the census reading the withheld block as a leak (340262 of 465190 ids), the lift being reachable only from a cutover (one boot at 26.8% of its id space for the life of the process), and exposure exceeding the backing (417850 rows over 105413 committed, the device-side assert in masked_set_kv_buffer_kernel). §3 HiCache staging write-through ring (sgl-project#810), new bullet, plus its two companion refusals -- the unbounded-file-tier refusal and the boot preflight ledger entry, the latter being why 22.01 GB of MHATokenToKVPoolHost across three PP ranks previously reached the preflight as nothing. §7 BAR1 deadline + loud abort -- appended the sgl-project#818 peer-liveness half to the existing narrative: the gate could wait forever on a peer that no longer exists, and neither Bar1CollectiveStalled (reset by every resolved read) nor defer_stall_for_building_peer (900 s off a build marker) caught it. §12 Robustness canon -- three new families: contradictory-flag (sgl-project#806), read-back-after-construction (sgl-project#797), denylist-of-reasons (sgl-project#817, sgl-project#820). §18.3 hicache staging sizing (sgl-project#810) -- §18's own rule is that a merge adding a reusable module adds its entry in the SAME merge, and this module had none. Records the removal of fits_pinned_host_budget so it is not reintroduced. §18.6 mamba carry instrument (sgl-project#767, gated by sgl-project#790) and flip break-even N (sgl-project#777). No existing entry was contradicted. Checked before writing: none of sgl-project#814, sgl-project#810, sgl-project#806, sgl-project#772, sgl-project#797, sgl-project#790, sgl-project#777, sgl-project#817, sgl-project#820, sgl-project#818 had a catalog entry, and the one sgl-project#677 line (§19.2, RESTORE-NEVER-REBUILD) describes a different mechanism than the sgl-project#677 layout hold, so it is not stale and was left alone. Gates for the tree this documents (hermetic, CUDA_VISIBLE_DEVICES=""): battery test/registered/unit/{managers,planner,server_args,mem_cache} baseline integ @ 78d27da 44 failed c56d238 (through sgl-project#818) 30 failed, 8452 passed 0 new ids, 14 fixed 1c4eadb (through sgl-project#816) 30 failed, 8461 passed 0 new ids, same set test_barlink_abort_gate_liveness_818.py (outside the battery dirs) 10 passed ruff --select=F401,F821,UP037 and codespell: 0 new findings vs the same file set on 78d27da (16 ruff / 6 codespell exist identically on base) docs-only change; codespell clean on the catalog itself
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Fixes the LiteLLM kwarg conversion.
Modification
Change from Anthropic to LiteLLM