Add disk cache for loading ShareGPT dataset. - #542
Merged
Merged
Conversation
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
Garrybest
pushed a commit
to Garrybest/sglang
that referenced
this pull request
Jan 9, 2026
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 4, 2026
…ats both arms Nine runs of a planned sixteen, stopped on instruction to restart serving with disk hicache and preserve_thinking. Two arms differing in one wire field: thinking disabled (today's default) against adaptive. The benchmark does not answer its question, and both reasons are measured. The A-vs-A noise floor spans everything: the same task in the same arm, back to back, went 179 s with a complete answer and then 600 s with no answer at all, 0.54 M against 2.58 M prompt tokens. And the thinking arm ran without preserve_thinking, so it re-prefills more per turn (sgl-project#542) -- 48.5 % prefix reuse against 40.3 % on the one pair where the counter was already wired up. Arm-B wall times are a serving artefact, not a model signal. Two results survive both problems. Adaptive thinking self-doses by task shape: 72 % and 56 % of generated tokens on the judgement audit, 7 % and 9 % on the mechanical inventory sweeps. And neither arm ever invented a defect -- every audit error in every run was a stale citation waved through, never a fabricated finding. Task material is real open work, not retro-eval: two sgl-project#538 building-block sweeps over disjoint directory slices, the standing FEATURE_CATALOG audit continued into the unchecked section 16, and the open sgl-project#533 warmup question. Each output is kept as work product rather than discarded, listed per task in the writeup. Yield: four stale section-16 citations confirmed by hand (uneven_perf.py:2617 -> 2642, metrics_reporter.py:1018 -> 1020, :1020 -> 1022, :962 -> 964), two usable module inventories, and a sgl-project#533 analysis whose twelve file:line claims were each opened and confirmed exact. One grading trap recorded because it nearly produced a false result: the first grader flagged sixteen importer claims as wrong and was wrong in all sixteen -- it excluded slice-internal importers, never searched scripts/, and its AST pass missed annotated module constants. ANALYSE_532's capability ladder gains three rows, all Tier 1, including an upgrade of bounded analysis-with-recommendation out of Tier 2. Thinking on vs off moves no tier. codespell clean.
xiangyuT
added a commit
to analytics-zoo/sglang
that referenced
this pull request
Aug 20, 2026
… packaged op esimd_qkv_split_norm_rope gained a 13th parameter, normalize_v, in llm-scaler PR sgl-project#542. It defaults to False for Qwen3 compatibility, but gemma4_causal.py called the op with only 12 arguments while its comment stated "V norm is now fused into the ESIMD kernel ... No separate call". Gemma4's weight-free V RMSNorm was therefore never applied on the ESIMD fast path. Symptom: locally coherent but degenerate, repeating generations ("la la la ...", looping Chinese clauses). Isolated by an in-container single-variable A/B starting from every XPU fast path disabled: the prefill paths (SGL_XPU_PREFILL_DPAS, SGLANG_XPU_FP8_W8A16_PREFILL) and the ESIMD decode attention were all clean; enabling only SGLANG_DISABLE_ESIMD_QKV=0 reproduced the degeneration. - Pass normalize_v=True explicitly. - Gate the fast path on `not self.v_norm.with_scale`, matching the precondition the non-ESIMD gemma_qkv_rmsnorm path already checks. - Add SGLANG_GEMMA4_DISABLE_ESIMD_NORM as a diagnostic kill switch for the three fused-norm fast paths, which had no individual env gate. - fp8_utils: prefer onednn_fp8_gemm_w8a16 from the packaged custom_esimd_kernels_sglang, keeping the standalone mini_fp8_C.so as fallback so the W8A16 prefill path works from a plain wheel install. Verified on gemma-4-31B-it, TP=2 eager, --dtype float16, online fp8 (gsm8k_chat_eval.py --n 100 --parallel 1, same server build, back to back): all fast paths OFF : 0.990 (99/100), 1187.6 s all fast paths ON : 0.990 (99/100), 1092.8 s Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.