[Feat] Expose logprob options to sgl.gen API - #503
Merged
Merged
Conversation
Contributor
Author
|
Hi @merrymercy, if there are any problems, please feel free to provide feedback, and I'll be happy to address them! |
merrymercy
requested changes
Jun 8, 2024
merrymercy
requested changes
Jun 8, 2024
merrymercy
changed the title
sgl.gen API
merrymercy
changed the title
sgl.gen APIsgl.gen API
Contributor
Author
Here's how it looks with Llama3-8b-Instruct |
Contributor
Author
|
I’m not sure if there’s a simpler way to replace the final end of turn token with 'So the answer is' and then continue generating. |
Contributor
Author
|
Any feedbacks? |
merrymercy
force-pushed
the
main
branch
2 times, most recently
from
July 4, 2024 07:57
2463404 to
d737da5
Compare
Co-authored-by: huyiwen <1020030101@qq.com>
merrymercy
approved these changes
Jul 9, 2024
Contributor
|
@huyiwen It is merged. Thanks for the contribution. |
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
Co-authored-by: Lianmin Zheng <lianminzheng@gmail.com>
shiyu7
added a commit
to shiyu7/sglang
that referenced
this pull request
May 22, 2026
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 3, 2026
…tall predicate; refute sgl-project#500-B1 sgl-project#500-B1 claimed the phase-prefill enumerator's kv-head grid was an artefact and that the runtime always ran on a replicated-KV / continuous-token-vector geometry. sgl-project#503 executed the check and REFUTES that claim: two independent predicates govern this family, and sgl-project#500-B1 conflated them. `attn_kv_replicated(tp_size, total_num_kv_heads)` is strictly `kv < tp` (`distributed/utils.py:1081`) and is what `linear.py:1423` / `model_config.py:1332` read to build the q/k/v shards -- so at `kv_heads >= tp` (4 kv heads over 3 ranks on the reference rig) the k/v PROJECTIONS head-shard and the enumerator's kv-head grid is correct. `uneven_dcp_kv_replicated` (`dcp_size > 1 and get_tp_partition_ratios() is not None`, `distributed/utils.py:346`, now consolidated in plan_uneven_dcp_kv_replicated and called from placement.py) replicates the KV POOL, not the projections. Pricing attention weights on the pool predicate models a layout the sgl-project#105 ragged-kernel guard refuses at the first forward. Fix: the planner now reports the token axis only where its own install predicate (`uneven_dcp_kv_replicated`) actually holds -- a bare `--rank-kv-ratio coupled` boot without a non-uniform base plan never reaches `dcp_size > 1`, head-shards the KV cache, and installs no token vector, so the axis is refused by name there instead of being priced as a lever the boot cannot actuate. `planner/placement.py:813` is flagged as an open (narrower) defect: it still uses one flag for both mechanisms and reports k/v projection heads as replicated under uneven DCP where the runtime shards them. Corrects FEATURE_CATALOG.md §1/§2, NOTE_485_joint_phase_vectors.md (three correction sites) and TICKET_485_int8_joint_arm.md to carry this refutation alongside the sgl-project#492/sgl-project#500 correction history already there (append-only, strikethrough + REFUTED markers, nothing deleted). Clean auto-merge -- no conflict: the touched lines in FEATURE_CATALOG.md §1/§2 were untouched by every merge since d653405 (the sgl-project#500-audit merge these lines came from), so the 3-way merge applied sgl-project#503's diff directly. All 17 catalog sections and the four previously-preserved additive paragraphs (two-axis §1, matrix doctrine §2, MERGE-DUTY owner-sites + SITREP §12, NVENC §13) verified present. Tests (CUDA_VISIBLE_DEVICES=99, PYTHONPATH pinned to this worktree): - test/registered/unit/planner/ (HTSGLANG_TEST_MODEL_DIR set): 2205 passed, 1 skipped, 334 subtests -- above the >=2202 target. - test_attn_replication_axis_492.py alone: 21 passed (up from 18 at the sgl-project#492 merge -- sgl-project#503 added 3 new methods to TestWhatTheRuntimeCanActuallyDoToday). - test/registered/unit/distributed/ (distributed/utils.py is touched): 24 failed, 2131 passed, 8 skipped, 703 subtests. Verified NOT a regression: every one of the 24 failing tests lives in a file sgl-project#503's diff never touches (test_bar1_host_cleanup.py, test_dcp_token_vector_collective.py, test_s12_log_analyse.py, test_uneven_tp_nccl_env.py, test_vmm_utils.py, test_dcp_context_ceiling.py), and the failing code paths themselves (model_runner_kv_cache_mixin.py's uneven_kv_derived_mode() call, the _PoolRowStub fixture, the MPS-warning assertion, LOCAL_RANK env lookups) already existed byte-identical in the pre-merge base d653405 -- confirmed by diff --stat, not just a count match. Root causes are a mix of missing test-fixture attributes, missing LOCAL_RANK env var under hermetic CPU-only invocation, and historical-log numeric drift, none related to the attention/KV geometry this merge touches. - ruff check --select=F401,F821,UP037: 2 findings (placement.py:166 `"PlacementFlags"`, uneven_perf.py:5028 `"Dict[str, List[float]]"`), both confirmed pre-existing at d653405 and outside sgl-project#503's diff hunks. - codespell clean on all 8 touched/added files.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 3, 2026
…each with a falsifier B3 (draft-KV-DCP now keys on the installer's own predicate uneven_weighted_dcp_enabled(), admitting the --rank-kv-ratio route the sibling speculation x DCP gate already accepted -- the -67% draft-KV win is no longer unreachable for flag-route boots); B8 (validate_breakable_boot's None arm split into a NO_SERVER_ARGS sentinel that skips vs. an unresolvable backend that refuses by name); B10 (verdict: the kvso x speculation refusal is DELIBERATE, not stale -- reason and the one unobserved round now in the message, KVSO_ALLOW_SPEC surfaced in --enable-kv-session-offload's CLI help for the first time); B2 (--rank-kv-ratio without a placement refuses by name instead of being accepted-and-inert); B18 (the marlin uneven-TP coarsening fold reads a per-backend marlin_packable_linear declaration instead of a class-name list; of the three configs the audit named, only MarlinConfig was a genuine latent gap, W8A8Fp8Config/QuarkConfig reach no marlin repack entry point); planner/flags.py's two inverted registry edges (I-2/I-3) plus a new contract test that drives every declared uneven-TP edge against the live ServerArgs validation instead of trusting the registry by inspection. Adds a status header to AUDIT_500_mechanism_reach.md §5 recording the six fixes and two corrections-from-execution. Manually extended that header with the sgl-project#503 B1-refutation (the sgl-project#503 branch predates B1's discovery entirely, so it couldn't record it): combined board is now B1 REFUTED (sgl-project#503), B2/B3/B8/B10/B18 + the flags.py registry FIXED (sgl-project#504), B4-B7, B9, B11-B17, B19, B20 still OPEN -- one place to read the whole status rather than two disjoint headers. Clean auto-merge, no conflict: FEATURE_CATALOG.md's touched regions (near the preamble, and in §1/§3/§12-adjacent prose) were untouched by every merge since the ac84f5d base; server_args.py was untouched by sgl-project#503/sgl-project#505/sgl-project#506/ sgl-project#509 (its only other change on this line, sgl-project#493, is already inside the base). All 17 catalog sections and the four previously-preserved additive paragraphs (two-axis §1, matrix doctrine §2, MERGE-DUTY owner-sites + SITREP §12, NVENC §13) verified present; the sgl-project#505 §16 corrections (gpu-arb convention-not-enforcement, forward_peak off-by-default, CollectiveClock prefill-only) verified present and unaffected. Tests (CUDA_VISIBLE_DEVICES=99, PYTHONPATH pinned to this worktree): - test_flag_registry_contract_500.py alone: 15 passed (exact target). - test/registered/unit/server_args/: 532 passed, 86 subtests (up from 526 pre-merge -- the new/extended test files account for the delta). - test/registered/unit/planner/ (HTSGLANG_TEST_MODEL_DIR set): 2223 passed, 1 skipped, 352 subtests -- crossing-check against sgl-project#503 (which also worked the planner edge): both merges coexist without collision, count is above the sgl-project#503-alone baseline of 2205 as expected from sgl-project#504's added tests. - test/registered/unit/layers/moe/: 153 passed, 395 subtests (includes the new test_breakable_boot_gate_500.py). - test/registered/unit/distributed/ (touched via test_marlin_unit_coarsening.py): 24 failed, 2134 passed, 8 skipped, 721 subtests. Diffed the failing-test list against the sgl-project#503-merge state line by line: BYTE-IDENTICAL set of 24 failures, only the passed/subtests counts grew (from marlin coarsening's new cases) -- confirmed non-regression, not just a count match. - ruff check --select=F401,F821,UP037: 2 findings (planner/flags.py:1717 `"Profile"`, test_rank_kv_ratio_args.py's unused `math` import), both confirmed pre-existing at the ac84f5d base and outside sgl-project#504's diff hunks. No ruff format run (CLAUDE.md: do not reformat planner/* and other listed subtrees -- unrelated churn buries the semantic diff). - codespell clean on all 18 touched/added files.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 17, 2026
…le, and what the phase matrix owes Docs only. No code changed, nothing built or wired. ## sgl-project#702 decision table (docs/DECISION_702_cut_table.md) Re-derived on the sgl-project#707 closed form with the calibrated fixture. Incumbent pool reads 436,275 here against the 436,766 in older notes; all percentages use 436,275 so they are internally consistent. cut attn pool speed noise(14.1%) gate [28,20,16] (7,5,4) 436,275 1.0000 -- is the gate [30,18,16] (7,5,4) 436,275 (+0.0%) 1.1111 BELOW IT PASSES [33,15,16] (8,4,4) 415,859 (-4.7%) 1.3299 above FAILS Switch cost 1575 ms from the incumbent either way (distance-independent, per #704a); payback 15.75 s and 6.35 s respectively. THE TRADE: the two candidates fail in OPPOSITE places, which is the decision. [30,18,16] passes the stage-1 "pool >= incumbent" gate and cannot be validated by measurement -- its 11.1% gain sits BELOW this rig's measured 14.1% A-vs-A noise floor, so no boot here can tell it from the incumbent. [33,15,16] clears the noise floor comfortably but needs a 4.7% capacity waiver. No ranking is offered; the pick is the user's. [30,18,16]'s pool is identical STRUCTURALLY, not coincidentally: it keeps the (7,5,4) attention split and leaves PP2 (layers 48-63) untouched, and PP2 is the binder. Per-rank capacity is [594,615/462,920/436,275] at the incumbent and [517,464/570,931/436,275] here -- PP2's column is unchanged, so moving two layers between PP0 and PP1 cannot move a pool PP2 sets. CORRECTION: I could not reproduce a "+20% pool" figure for [30,18,16]; its coupled pool is +0.0%. The nearest +17.7% on the list is the DECOUPLED pool, which is cut-INDEPENDENT (the #704b R6 sum-rule result: 513,875 for every cut including the incumbent) and therefore not attributable to any cut. SOLVER DEFECT FOUND, flagged not fixed: prefill_frontier.py:144-156 enumerates, per lead depth, ONLY the tail split that minimises pipelined time. At n0=30 that is [30,16,18] (130.95 ms), so [30,18,16] (139.32 ms) is never generated -- yet [30,16,18] holds 343,951 tokens (-21.2%) against [30,18,16]'s 436,275 (+0.0%). The frontier is Pareto-optimal in SPEED but not complete in POOL: a roomier, slightly slower cut at the same lead depth cannot appear on it. [30,18,16] strictly dominates the incumbent (equal pool, more speed) and was invisible for exactly that reason, which is why it never reached the sgl-project#702 block. The fix would be to keep the Pareto set over (pool, speed) per lead depth instead of the speed-argmin; other pool-preserving cuts may also be hidden. Validation-boot evidence is specified in falsification order, and includes the one that matters most: for [30,18,16] the prefill A/B is expected to be INCONCLUSIVE by construction -- a boot reporting a clean 1.11x has measured its own noise, and that result must be rejected rather than celebrated. Appended to the morning plan file as a dated block per its amendment convention; head-298 sha256 verified byte-identical before and after (81560e94b7628e6c8...). That file is not version-controlled, hence this copy. ## sgl-project#485 completion verdict (docs/VERDICT_485_phase_matrix.md) Directive quoted verbatim from /spinning/htsglang/CLAUDE.md:87-96 (user law 2026-08-03), which is NOT tracked in this worktree. The two facets under sgl-project#485 are the same ticket, not a number collision: the matrix doctrine plus its prefill-column solver, and the seam/memory certification gating it. VERDICT: PARTIALLY DELIVERED, do not close. DELIVERED prefill column for the attention-vs-GDN/linear boundary (c08f613, b675630, b3c9802, e645aa7), with a second axis on the attention row (sgl-project#492 d82a778, corrected by sgl-project#503 447249a). Its own feasibility facet (407b382) is RED: the threshold is not reachable at the certification pool. SUPERSEDED d937d5f refuses the decode-side family split on sound MEASURED reasoning -- against the honest proportional (uneven-TP) baseline the split is worth +0.090 ms/round, ~0.3% of a 30 ms bs=1 round. But this closes ONE CANDIDATE CELL (concentration), not the column. Named tension: the directive says single-family arms are DIAGNOSTIC only and must never be phrased as a phase-level verdict, and that commit is worded as one. The narrow reading is adopted. OPEN O1 no (family x phase) structure exists anywhere -- pp_cut.py solves prefill only, family_split.py returns by_family with NO phase argument or dimension; the "matrix" is prose over two single-phase solvers. O2 the decode solver is DEAD CODE: solve_family_placement has exactly two references, its definition (family_split.py:141) and its test, zero call sites. O3 vocab, experts, nonlinear kernels and per-quant-lane linear layers were never cut at all. O4 the law covers diffusion, SR/video, TTS/ASR and training tenants -- entirely untouched, and the largest open surface. O5 the prefill column is not arm-able while Facet B is red. NOT sgl-project#485 sgl-project#704/#704a/#704b are a PP-cut layout LADDER and its arming machinery. They consume the prefill-column solve as an input and must not be counted toward the matrix; counting them would inflate its delivery with work on a different axis. Status in one line: one of two columns, one of many families, one of several workloads -- and the table is still prose. Every load-bearing claim spot-verified against the tree rather than taken from the survey: directive text read at source, solve_family_placement call sites grepped, d937d5f's baselines read from the commit body, [30,18,16] priced directly rather than inferred from frontier membership. Sanity: 35 passed (test_prefill_frontier_702.py, test_fill_ladder_704a.py). Desk only, no GPU, deploy tree untouched.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Key changes
Fixes
return_logprobandtop_logprobs_numwithLogitsProcessorJSONDecodeErrorwhen runtime returns a 500 error