Skip to content

Disk FSM cache and adjust code. - #63

Merged
merrymercy merged 8 commits into
mainfrom
disk_fsm_cache
Jan 21, 2024
Merged

Disk FSM cache and adjust code.#63
merrymercy merged 8 commits into
mainfrom
disk_fsm_cache

Conversation

@hnyls2002

@hnyls2002 hnyls2002 commented Jan 21, 2024 •
edited
Loading

Copy link
Copy Markdown
Collaborator
  • A disk cache mechanism was applied, caching the most time-consuming part in the local storage.
  • I removed the unused code in tokenizer.py and built TransformersTokenizer at the beginning of ModelRpcServer to avoid some checking errors in the outlines code.
  • Aligned the fsm.py with outlines==0.0.24

@merrymercy
merrymercy merged commit ca13f3b into main Jan 21, 2024
@merrymercy
merrymercy deleted the disk_fsm_cache branch January 22, 2024 00:54
@CSEEduanyu CSEEduanyu mentioned this pull request Jan 26, 2025
Closed
5 tasks done
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
@hnyls2002
Disk FSM cache and adjust code. (sgl-project#63)
bbb000e
blzheng pushed a commit to blzheng/sglang that referenced this pull request Apr 22, 2025
@ZailiWang
branch name changable (sgl-project#63)
2a88456
NorthmanPKU pushed a commit to NorthmanPKU/sglang that referenced this pull request May 16, 2025
@xinhaoc
Fix some existing issues (sgl-project#63)
b227d42 
* fix decompose output tensor guid issue

* .

* .
pi314ever pushed a commit to pi314ever/sglang that referenced this pull request Jun 21, 2025
Chandrakant Khandelwal
[SW-228160] Update HPUForwardBatch and hpu forward batch creation for…
3435406 
… logprob calculation (sgl-project#63)

* Update HPUForwardBatch and hpu forward batch creation for prefill for returning logprob

* fix typo

* Add TODO with the ticket details for circular dependency
blzheng pushed a commit to blzheng/sglang that referenced this pull request Aug 13, 2025
@jianan-gu
add moe a16w4 gptoss kernel (sgl-project#63)
2da2d20
zhuyijie88 pushed a commit to zhuyijie88/sglang that referenced this pull request Sep 4, 2025
@cherryblo @iforgetmyname
Update test_ascend_mla_w8a8int8.py (sgl-project#63)
789cf23
key4ng pushed a commit to key4ng/sglang that referenced this pull request Nov 9, 2025
@CatherineSue
feat(data): normalize HF dataset types and enable lazy image sampling (…
4a42ed4 
…sgl-project#63)
apinge referenced this pull request in apinge/sglang Dec 8, 2025
@ZLkanyo009
Merge pull request #63 from ZLkanyo009/dev/perf
3f5e835 
[Accelerate] acc by delete D2D in torch.where and index_put
efschu added a commit to efschu/htsglang that referenced this pull request Jul 17, 2026
@efschu @claude
Qwen3.6-27B shvllm
[GGUF] Perf overhaul: flat layout, batched MMVQ, quantized vocab, Q8 …
e251804 
…support

Closes the measured GGUF performance gap to llama.cpp (M20 analysis:
61.6 vs ~41.5 ms/MTP-iteration at Q6 TP=2) and turns it around at
TP>=3. Four measures plus the Q8 loader fix, all GGUF-gated — AWQ/FP8/
non-GGUF paths byte-identical (proven, see validation).

1. Flat single-buffer shard layout (gguf.py): merged shards live
   back-to-back in ONE 16-byte-aligned uint8 buffer with static 2-D
   views; apply() uses the views — removes the ~1.9 GB/rank/forward
   slice+contiguous() copy traffic of mixed-dtype GDN qkvz layers.
   Graph-safe by construction.
2. Persistent dequant workspace (sgl-project#63): grow-only per-(device,dtype)
   workspace, pre-reserved BEFORE KV profiling; ggml_dequantize gains
   an optional out= (schema-detected; old wheel = byte-identical
   fallback). Kills the 86-MiB-per-forward alloc OOM class.
3. Quantized-resident lm_head/embed (default; SGLANG_GGUF_DENSE_VOCAB=1
   escape hatch): token_embd/output stay Q8_0-class instead of bf16
   materialization (1.26 vs 2.37 GiB; 4 lm_head passes per MTP
   iteration read half the bytes). NEXTN shares embed/lm_head at
   MODULE level (draft 2.62 -> 0.24 GB). Row-granular qweight vocab
   sharding replaces the M24 fail-fast — works even AND with
   --rank-vocab-ratio.
4. Batched MMVQ ncols<=8 (mmvq.cuh): one weight-block load serves up
   to 8 columns (llama.cpp pattern), bit-identical accumulation order
   per column, dispatcher behind all 19 type launchers; default-on iff
   the new-wheel schema is detected (SGLANG_GGUF_BATCHED_MMVQ env
   override; old wheel keeps the device-adaptive crossover).
5. (sgl-project#64) Mixed-dtype fused GDN qkvz (UD-Q8_K_XL: F16 z next to Q8_0
   qkv etc.): flat layout carries mixed dtypes, dense shards cast once
   at load; dense-module forcing now only for F32. Q8 at TP>1 was
   INFEASIBLE in both forks — now boots and serves.
Build: sgl-kernel gains gated SGL_KERNEL_LIMIT_CUDA_ARCHS /
SGL_KERNEL_SKIP_SM90_VARIANT options (default-off = stock builds).
Requires a rebuilt sgl_kernel wheel for measures 2/4 (graceful
fallback without).

Validation (T66_VALIDATION.md + HANDOFF M29; all boots thermal-gated,
GPUs 0 MiB after):
- Token identity: kill-switch boot (dense vocab, batched off) is
  BYTE-IDENTICAL to HEAD (3x1024 greedy long-forms, logprob maxdelta
  0.000000, accepts equal) -> measures 1+2 exact. Batched-MMVQ boot is
  not byte-identical BY CONSTRUCTION (dispatch shift MMQ->MMVQ at
  M=3..8): both kernels 4.93e-3 relmax to fp32 reference, 1.18e-4 to
  each other, logprob sampled IDs identical over 32 steps, maxdelta
  0.041, accepts within 0.1. Quantized vocab: IDs identical, sampled
  maxdelta 0.104, accepts equal or better (3.91/3.62/2.61).
- Kernel bit-parity batched-vs-per-column MMVQ M=1..8 + out= parity
  PASS on sm86 and sm120.
- Q6 TP=2: 54.5 -> 66.0 tok/s code (+21%), weights 14.03 -> 12.74
  GB/rank; Q6 TP=3-uneven+DCP: 65.9/54.6 -> 91.1/79.1 (+37%/+47%) —
  above every llama.cpp cell on this rig (79.6/74.5); accept lengths
  unchanged (3.35 vs 3.36).
- Q8_K_XL TP=3: boots coherent at 91.1/82.3 tok/s (previously
  INFEASIBLE, gguf.py:475 mixed-dtype assert).
- Q4_K_M TP=2: +5% decode, maxKV 153k -> 208k (+36%), 14.7k prefill
  without the sgl-project#63 OOM. Q5_K_M + heretic-Q6 smokes pass.
- --rank-vocab-ratio auto on quantized GGUF vocab: boots, coherent.

Operational notes: freed VRAM shifts auto-sizing — boot recipes in
HANDOFF M29 (TP=2 memfrac 0.80; TP=3 reserve 3500 +
--cuda-graph-max-bs-decode 8); sizing follow-up tracked. MoE-GGUF
(qwen35moe) keeps its own unmodified path.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Qwen3.6-27B <qwen3.6-27b@localhost>
efschu added a commit to efschu/htsglang that referenced this pull request Jul 20, 2026
@efschu
[sgl-project#89] hibernate: reserve sgl-project#63 dequant scratch on…
2cd3079 
… restore (KV-profiling parity) -> uneven-TP=3 green

Restore skips process_weights_after_loading, which is where the cold path
reserves the sgl-project#63 persistent dequant workspace BEFORE KV-cache profiling so the
pool leaves room for it. Without that reservation the restored model shows more
apparent free VRAM, the KV pool over-sizes, and decode CUDA-graph capture OOMs
on a tight rank (observed on the 3080 at TP=3, 802 MiB free). Re-run just that
side effect on the restored GGUF layers (module-global buffer; no effect on the
byte-hash gate).

Validation -- uneven-TP=3 + DCP=3 + GGUF (5090 + 2x3080, --rank-tp-ratio 2,1,1,
graphs ON):
- Cold boot parks 3 UNEVEN per-rank shards (rank0/5090 8.1GB, ranks1-2/3080
  4.36GB each), distinct NVML UUIDs + distinct per-rank byte-hashes, 96
  unquantized prefixes, manifest with all 3 ranks; park ~15s (parallel D2H).
- Restart auto-selects HIBERNATE; each rank resolves its own NVML UUID, matches
  its shard, restores uneven shapes: rank0 14.4s, ranks1-2 8.4s vs ~50s cold
  load. All 3 ranks byte-hash == manifest (bit-identical). Reserved dequant
  scratch on 210 GGUF layers/rank; prefill+decode CUDA graphs re-captured clean;
  server fires.
- Weight correctness: per-rank byte-hash machine-zero vs cold (the sgl-project#89 gate).
  Greedy A/B at TP=3 is within the config's own run-to-run envelope: the SAME
  restored server yields different greedy continuations on repeated calls to the
  same open-ended prompt (uneven-TP cross-rank fp reassociation on near-ties,
  == sgl-project#124 DECODE_CLASS, not machine-zero) -- so cold-vs-restore divergence there
  is inherent to the config, not a hibernate defect. TP=1 (no cross-rank merge)
  restore is character-identical to cold (machine-zero), proving the mechanism
  is exact. KV pool differs ~0.1% (131805 vs 131954 tokens) from imperfect
  memory parity -- harmless, notable.
efschu added a commit to efschu/htsglang that referenced this pull request Jul 22, 2026
@efschu
[GGUF] Perf overhaul: flat layout, batched MMVQ, quantized vocab, Q8 …
5a5baf8 
…support

Closes the measured GGUF performance gap to llama.cpp (M20 analysis:
61.6 vs ~41.5 ms/MTP-iteration at Q6 TP=2) and turns it around at
TP>=3. Four measures plus the Q8 loader fix, all GGUF-gated — AWQ/FP8/
non-GGUF paths byte-identical (proven, see validation).

1. Flat single-buffer shard layout (gguf.py): merged shards live
   back-to-back in ONE 16-byte-aligned uint8 buffer with static 2-D
   views; apply() uses the views — removes the ~1.9 GB/rank/forward
   slice+contiguous() copy traffic of mixed-dtype GDN qkvz layers.
   Graph-safe by construction.
2. Persistent dequant workspace (sgl-project#63): grow-only per-(device,dtype)
   workspace, pre-reserved BEFORE KV profiling; ggml_dequantize gains
   an optional out= (schema-detected; old wheel = byte-identical
   fallback). Kills the 86-MiB-per-forward alloc OOM class.
3. Quantized-resident lm_head/embed (default; SGLANG_GGUF_DENSE_VOCAB=1
   escape hatch): token_embd/output stay Q8_0-class instead of bf16
   materialization (1.26 vs 2.37 GiB; 4 lm_head passes per MTP
   iteration read half the bytes). NEXTN shares embed/lm_head at
   MODULE level (draft 2.62 -> 0.24 GB). Row-granular qweight vocab
   sharding replaces the M24 fail-fast — works even AND with
   --rank-vocab-ratio.
4. Batched MMVQ ncols<=8 (mmvq.cuh): one weight-block load serves up
   to 8 columns (llama.cpp pattern), bit-identical accumulation order
   per column, dispatcher behind all 19 type launchers; default-on iff
   the new-wheel schema is detected (SGLANG_GGUF_BATCHED_MMVQ env
   override; old wheel keeps the device-adaptive crossover).
5. (sgl-project#64) Mixed-dtype fused GDN qkvz (UD-Q8_K_XL: F16 z next to Q8_0
   qkv etc.): flat layout carries mixed dtypes, dense shards cast once
   at load; dense-module forcing now only for F32. Q8 at TP>1 was
   INFEASIBLE in both forks — now boots and serves.
Build: sgl-kernel gains gated SGL_KERNEL_LIMIT_CUDA_ARCHS /
SGL_KERNEL_SKIP_SM90_VARIANT options (default-off = stock builds).
Requires a rebuilt sgl_kernel wheel for measures 2/4 (graceful
fallback without).

Validation (T66_VALIDATION.md + HANDOFF M29; all boots thermal-gated,
GPUs 0 MiB after):
- Token identity: kill-switch boot (dense vocab, batched off) is
  BYTE-IDENTICAL to HEAD (3x1024 greedy long-forms, logprob maxdelta
  0.000000, accepts equal) -> measures 1+2 exact. Batched-MMVQ boot is
  not byte-identical BY CONSTRUCTION (dispatch shift MMQ->MMVQ at
  M=3..8): both kernels 4.93e-3 relmax to fp32 reference, 1.18e-4 to
  each other, logprob sampled IDs identical over 32 steps, maxdelta
  0.041, accepts within 0.1. Quantized vocab: IDs identical, sampled
  maxdelta 0.104, accepts equal or better (3.91/3.62/2.61).
- Kernel bit-parity batched-vs-per-column MMVQ M=1..8 + out= parity
  PASS on sm86 and sm120.
- Q6 TP=2: 54.5 -> 66.0 tok/s code (+21%), weights 14.03 -> 12.74
  GB/rank; Q6 TP=3-uneven+DCP: 65.9/54.6 -> 91.1/79.1 (+37%/+47%) —
  above every llama.cpp cell on this rig (79.6/74.5); accept lengths
  unchanged (3.35 vs 3.36).
- Q8_K_XL TP=3: boots coherent at 91.1/82.3 tok/s (previously
  INFEASIBLE, gguf.py:475 mixed-dtype assert).
- Q4_K_M TP=2: +5% decode, maxKV 153k -> 208k (+36%), 14.7k prefill
  without the sgl-project#63 OOM. Q5_K_M + heretic-Q6 smokes pass.
- --rank-vocab-ratio auto on quantized GGUF vocab: boots, coherent.

Operational notes: freed VRAM shifts auto-sizing — boot recipes in
HANDOFF M29 (TP=2 memfrac 0.80; TP=3 reserve 3500 +
--cuda-graph-max-bs-decode 8); sizing follow-up tracked. MoE-GGUF
(qwen35moe) keeps its own unmodified path.
efschu added a commit to efschu/htsglang that referenced this pull request Jul 22, 2026
@efschu
[GGUF] Shape-aware dispatch, capture-safe dequant chunking, auto reserve
7c031df 
Measures from the sgl-project#69 investigation (T72; two predictions honestly
corrected during implementation):

1. Shape-aware MMVQ/MMQ dispatch (sm<9, K-quants only): re-measurement
   at the REAL TP=2 down-proj shard (K=8704) corrected the analysis --
   MMQ wins from M>=5 there (not M>=3; that held only for the K=17408
   full shape which does not exist under TP). Rules: M>=8 -> MMQ (wins
   on all measured shapes, up to 2.27x), M>=5 & K>=8192 -> MMQ.
   Single-stream decode gain is ZERO (verify M=4 untouched) -- the
   predicted +2-3 tok/s does not materialize at TP=2/num_spec=3; the
   real gain sits in bs>=2 verify and M=5-8 prefill chunks. Quality:
   32-step logprob-ID identity vs base (IDs identical, maxdelta
   0.000000 on normal prompts; crafted M=8 probe shows one near-tie
   flip at kernel relmax 3.1e-4 -- the sgl-project#66-documented class).
   Kill switch SGLANG_GGUF_SHAPE_DISPATCH=0; SGLANG_GGUF_MMVQ_SAFE
   still wins.
2. ssm_out GEMM tile measure: FALSIFIED, nothing built -- the
   "290 GB/s bad tile" premise was a bytes-accounting artifact (96
   calls/iter = 2 weight-stream passes; per call the GEMM runs at
   630 GB/s = 87% peak, standalone bench reproduces the trace time).
   All layout variants <=3% delta. Predicted +4 tok/s does not exist.
3. (sgl-project#70) Capture-safe dequant: over-cap dequants during CUDA-graph
   capture now row-chunk through the persistent sgl-project#63 workspace
   (_mul_mat_dequant_chunked); eager path byte-identical. The exact
   previously-OOMing boot (Q4-XL-MTP without mrr cap) now captures:
   verify 0.24 GB (was 1.42), draft-extend bs=4 at 0.09 GB (was
   1.19 GiB OOM); maxKV 213,546 @mrr=4, bs=4 burst coherent.
4. (sgl-project#68) --rank-auto-reserve-mib defaults to 'auto': reserve derived
   as stock_runtime_reserve + decode_max_bs x (2D+1) x 2 MiB x
   colocated_ranks (three spec graph families counted, no magic
   constants). TP=3-uneven default boot: formula yields 4400 MiB/GPU,
   one-boot convergence, full bs-list captured, decode 90.08/80.40
   (91.1-class). Honest cost: maxKV 450,752 vs 546,560 with the
   hand-tuned M29 recipe (-17.5%, formula over-reserves ~0.9 GB/rank)
   -- documented; hand-tuning remains possible via explicit values.
   The analogous even-path memfrac multiplier was built and REVERTED
   after an A/B showed it regressed the flagless multimodal TP=2 boot
   (rationale in code).
5. The historical 32k KV cap lived only in the t66 boot script (not in
   repo code); removed there. Uncapped recipe boot: maxKV 95,402 at
   baseline-exact decode (67.78/54.24, accept 3.346).

Regressions: 27B FP8-MTP recipe max_total 487,616 exact, coherent;
flagless Q6 TP=2 sizing byte-identical to stock.
efschu added a commit to efschu/htsglang that referenced this pull request Jul 22, 2026
@efschu
[sgl-project#89] hibernate: reserve sgl-project#63 dequant scratch on…
fb39cb8 
… restore (KV-profiling parity) -> uneven-TP=3 green

Restore skips process_weights_after_loading, which is where the cold path
reserves the sgl-project#63 persistent dequant workspace BEFORE KV-cache profiling so the
pool leaves room for it. Without that reservation the restored model shows more
apparent free VRAM, the KV pool over-sizes, and decode CUDA-graph capture OOMs
on a tight rank (observed on the 3080 at TP=3, 802 MiB free). Re-run just that
side effect on the restored GGUF layers (module-global buffer; no effect on the
byte-hash gate).

Validation -- uneven-TP=3 + DCP=3 + GGUF (5090 + 2x3080, --rank-tp-ratio 2,1,1,
graphs ON):
- Cold boot parks 3 UNEVEN per-rank shards (rank0/5090 8.1GB, ranks1-2/3080
  4.36GB each), distinct NVML UUIDs + distinct per-rank byte-hashes, 96
  unquantized prefixes, manifest with all 3 ranks; park ~15s (parallel D2H).
- Restart auto-selects HIBERNATE; each rank resolves its own NVML UUID, matches
  its shard, restores uneven shapes: rank0 14.4s, ranks1-2 8.4s vs ~50s cold
  load. All 3 ranks byte-hash == manifest (bit-identical). Reserved dequant
  scratch on 210 GGUF layers/rank; prefill+decode CUDA graphs re-captured clean;
  server fires.
- Weight correctness: per-rank byte-hash machine-zero vs cold (the sgl-project#89 gate).
  Greedy A/B at TP=3 is within the config's own run-to-run envelope: the SAME
  restored server yields different greedy continuations on repeated calls to the
  same open-ended prompt (uneven-TP cross-rank fp reassociation on near-ties,
  == sgl-project#124 DECODE_CLASS, not machine-zero) -- so cold-vs-restore divergence there
  is inherent to the config, not a hibernate defect. TP=1 (no cross-rank merge)
  restore is character-identical to cold (machine-zero), proving the mechanism
  is exact. KV pool differs ~0.1% (131805 vs 131954 tokens) from imperfect
  memory parity -- harmless, notable.
@Andrew88Lin Andrew88Lin mentioned this pull request Aug 25, 2026
Closed
TianHao65 added a commit to TianHao65/sglang that referenced this pull request Oct 22, 2026
@TianHao65
Show native Perfetto area selection and kernel lists for PR sgl-proje…
3e9c6b2 
…ct#63
TianHao65 added a commit to TianHao65/sglang that referenced this pull request Oct 26, 2026
@TianHao65
Replace PR sgl-project#63 Perfetto figures with full original trace s…
a52caa3 
…creenshots
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@hnyls2002 @merrymercy