Skip to content

Fix select - #64

Merged
merrymercy merged 2 commits into
mainfrom
fix-select
Jan 21, 2024
Merged

Fix select#64
merrymercy merged 2 commits into
mainfrom
fix-select

Conversation

@merrymercy

@merrymercy merrymercy commented Jan 21, 2024

Copy link
Copy Markdown
Collaborator

No description provided.

@merrymercy
merrymercy merged commit 11f3cca into main Jan 21, 2024
@merrymercy
merrymercy deleted the fix-select branch January 21, 2024 07:20
@merrymercy
merrymercy restored the fix-select branch January 21, 2024 07:21
@CSEEduanyu CSEEduanyu mentioned this pull request Jan 26, 2025
Closed
5 tasks done
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
@merrymercy
Fix select (sgl-project#64)
3855678
pengchengneo pushed a commit to pengchengneo/sglang that referenced this pull request Jun 5, 2025
@Prayer3th @mmxiaolin
fix eplb updater (sgl-project#64)
395f754 
Co-authored-by: xiaolin <xiaolin@minimaxi.com>
blzheng added a commit to blzheng/sglang that referenced this pull request Aug 13, 2025
@blzheng
add dequant and fix issue in tp=1 (sgl-project#64)
34c9b3f
zhuyijie88 pushed a commit to zhuyijie88/sglang that referenced this pull request Sep 4, 2025
@cherryblo @iforgetmyname
Block the original PR workflow of the sglang community and allow manu…
4a2e1de 
…al triggering only. (sgl-project#64)

* Update test_ascend_mla_w8a8int8.py

* Update nightly-test-amd.yml

* Update nightly-test.yml

* Update pr-benchmark-rust.yml

* Update pr-test-amd.yml

* Update pr-test-h20.yml

* Update pr-test-pd-router.yml

* Update pr-test-rust.yml

* Update pr-test-sgl-kernel.yml

* Update pr-test-xeon.yml

* Update pr-test.yml

* Update pr-test-npu.yml

* Update vllm-dependency-test.yml
key4ng pushed a commit to key4ng/sglang that referenced this pull request Nov 9, 2025
@CatherineSue
Revert "[Plot] Rename plot axis label and xlimt start from 0 and upda…
f8d66e4 
…te docs" (sgl-project#64)
vschandramourya pushed a commit to vschandramourya/sglang that referenced this pull request Feb 3, 2026
@zhyncs @github-actions
Copy OSS code from 373080e on 20250916 (sgl-project#64)
57031a5 
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
xq25478 pushed a commit to xq25478/sglang that referenced this pull request Apr 3, 2026
zhangfeiyu.17 @xq25478
fix: resolve memory leak for prefix caching requests in Draft mode (s…
755c939 
…gl-project#64)

* refactor: optimize paused_reqs lock acquisition in can_schedule

* fix: resolve memory leak for prefix caching requests in Draft mode
xq25478 pushed a commit to xq25478/sglang that referenced this pull request Apr 3, 2026
xiaoqi.31 @xq25478
feat & perf: wait & pre-verify (sgl-project#68)
47ed152 
* chore: cleanup

* Merge branches 'RemoteSpec-v057','RemoteSpec-v057-dev-yw' into 'RemoteSpec-v057-dev-yw'

* feat & perf: wait & pre-verify

* fix(cpp zmq)

* fix: resolve memory leak for prefix caching requests in Draft mode (sgl-project#64)

* chore: debug some log

* chore: correct log in draft

* feat: add retry_fail_ratio arg to control the trigger of retry

* perf: zmq pack send/recv @zhangyu

* fix(zmq):添加 ZMQ 链路耗时分析

* perf: skip req.origin_input_ids send when spec_cnt > 0

* perf: only send req.sampling_params when spec_cnt = 0  to reduce transfer time

* perf: pre-verify wait logic add.

* perf: adjust draft sleep time when recv draft tokens

* perf: let draft recv full batch

* chore: adjust time cnt

* chore: add some time count log

* perf: Target optimize

* perf: wait for draft tokens & add DraftCircuitBreaker

* fix: adjust first token in first pad token position

* perf: optimize 2 step verify logic

* perf: optimize tensor init
This was referenced May 8, 2026
Open
Open
efschu added a commit to efschu/htsglang that referenced this pull request Jul 17, 2026
@efschu @claude
Qwen3.6-27B shvllm
[GGUF] Perf overhaul: flat layout, batched MMVQ, quantized vocab, Q8 …
e251804 
…support

Closes the measured GGUF performance gap to llama.cpp (M20 analysis:
61.6 vs ~41.5 ms/MTP-iteration at Q6 TP=2) and turns it around at
TP>=3. Four measures plus the Q8 loader fix, all GGUF-gated — AWQ/FP8/
non-GGUF paths byte-identical (proven, see validation).

1. Flat single-buffer shard layout (gguf.py): merged shards live
   back-to-back in ONE 16-byte-aligned uint8 buffer with static 2-D
   views; apply() uses the views — removes the ~1.9 GB/rank/forward
   slice+contiguous() copy traffic of mixed-dtype GDN qkvz layers.
   Graph-safe by construction.
2. Persistent dequant workspace (sgl-project#63): grow-only per-(device,dtype)
   workspace, pre-reserved BEFORE KV profiling; ggml_dequantize gains
   an optional out= (schema-detected; old wheel = byte-identical
   fallback). Kills the 86-MiB-per-forward alloc OOM class.
3. Quantized-resident lm_head/embed (default; SGLANG_GGUF_DENSE_VOCAB=1
   escape hatch): token_embd/output stay Q8_0-class instead of bf16
   materialization (1.26 vs 2.37 GiB; 4 lm_head passes per MTP
   iteration read half the bytes). NEXTN shares embed/lm_head at
   MODULE level (draft 2.62 -> 0.24 GB). Row-granular qweight vocab
   sharding replaces the M24 fail-fast — works even AND with
   --rank-vocab-ratio.
4. Batched MMVQ ncols<=8 (mmvq.cuh): one weight-block load serves up
   to 8 columns (llama.cpp pattern), bit-identical accumulation order
   per column, dispatcher behind all 19 type launchers; default-on iff
   the new-wheel schema is detected (SGLANG_GGUF_BATCHED_MMVQ env
   override; old wheel keeps the device-adaptive crossover).
5. (sgl-project#64) Mixed-dtype fused GDN qkvz (UD-Q8_K_XL: F16 z next to Q8_0
   qkv etc.): flat layout carries mixed dtypes, dense shards cast once
   at load; dense-module forcing now only for F32. Q8 at TP>1 was
   INFEASIBLE in both forks — now boots and serves.
Build: sgl-kernel gains gated SGL_KERNEL_LIMIT_CUDA_ARCHS /
SGL_KERNEL_SKIP_SM90_VARIANT options (default-off = stock builds).
Requires a rebuilt sgl_kernel wheel for measures 2/4 (graceful
fallback without).

Validation (T66_VALIDATION.md + HANDOFF M29; all boots thermal-gated,
GPUs 0 MiB after):
- Token identity: kill-switch boot (dense vocab, batched off) is
  BYTE-IDENTICAL to HEAD (3x1024 greedy long-forms, logprob maxdelta
  0.000000, accepts equal) -> measures 1+2 exact. Batched-MMVQ boot is
  not byte-identical BY CONSTRUCTION (dispatch shift MMQ->MMVQ at
  M=3..8): both kernels 4.93e-3 relmax to fp32 reference, 1.18e-4 to
  each other, logprob sampled IDs identical over 32 steps, maxdelta
  0.041, accepts within 0.1. Quantized vocab: IDs identical, sampled
  maxdelta 0.104, accepts equal or better (3.91/3.62/2.61).
- Kernel bit-parity batched-vs-per-column MMVQ M=1..8 + out= parity
  PASS on sm86 and sm120.
- Q6 TP=2: 54.5 -> 66.0 tok/s code (+21%), weights 14.03 -> 12.74
  GB/rank; Q6 TP=3-uneven+DCP: 65.9/54.6 -> 91.1/79.1 (+37%/+47%) —
  above every llama.cpp cell on this rig (79.6/74.5); accept lengths
  unchanged (3.35 vs 3.36).
- Q8_K_XL TP=3: boots coherent at 91.1/82.3 tok/s (previously
  INFEASIBLE, gguf.py:475 mixed-dtype assert).
- Q4_K_M TP=2: +5% decode, maxKV 153k -> 208k (+36%), 14.7k prefill
  without the sgl-project#63 OOM. Q5_K_M + heretic-Q6 smokes pass.
- --rank-vocab-ratio auto on quantized GGUF vocab: boots, coherent.

Operational notes: freed VRAM shifts auto-sizing — boot recipes in
HANDOFF M29 (TP=2 memfrac 0.80; TP=3 reserve 3500 +
--cuda-graph-max-bs-decode 8); sizing follow-up tracked. MoE-GGUF
(qwen35moe) keeps its own unmodified path.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Qwen3.6-27B <qwen3.6-27b@localhost>
efschu added a commit to efschu/htsglang that referenced this pull request Jul 17, 2026
@efschu @claude
Qwen3.6-27B shvllm
[Feature] TP > num_kv_heads: replicated-KV + token-sharding (generic)
dcf1d0f 
Models whose attention layers carry fewer KV heads than TP ranks could
not run under (uneven) TP at all -- the whole-head split requires
kv >= tp (and even TP requires tp % kv == 0). This lands the analyzed
approach generically: when the uneven plan is active and kv_total < tp,
the affected full-attention layers switch to REPLICATED-KV mode -- all
kv heads on every rank (deterministic recompute from byte-identical
replicated k/v projection weights, no broadcast), q heads split in
GQA-group-sized units (q/kv, keeping num_qo % num_kv == 0 per rank),
tokens sharded via the existing weighted-DCP owner rule and merged with
the graph-validated LSE all-gather. NO KV-cache duplication (disjoint
token shards -> capacity stays the sum) and comm-neutral (the per-layer
kv-head all-gather in _dcp_masked_write is skipped for these layers).
GDN/linear-attention layers keep normal head sharding; the NEXTN draft
(also kv-deficient) replicates the same way incl. its full-context
pool. Parametric geometry only -- no model special cases: covers kv=2
models on TP=3/4 (Qwen3.6-35B-A3B, Gemma-4-26B-A4B) AND e.g. the 27B
(kv=4) on TP=5+ ([8,4,4,4,4] q-split). Missing DCP -> fail fast naming
SGLANG_UNEVEN_DCP=1; tp > GQA group size -> clear ValueError.

Also wires the previously caller-less fused-MoE FP8 Marlin fallback
(pre-existing platform gap; A3B's first forward crashed with fp8e4nv
unsupported on sm86): Fp8MoEMethod auto-enables Marlin W8A16 per rank
on 80<=sm<89 (native fp8 stays on sm120 -> mixed-arch TP with disjoint
expert shards, combined by all-reduce), zero-padding non-64-aligned
uneven expert shards locally (numerically exact). Micro-test vs fp32
dequant reference on sm86 and sm120: cos >= 0.99998. By-catch fixes:
group-quant expert units in fused_moe_triton (was FP8-block-only) and
a fused-aware AWQ skip list.

Validation (A3B FP8 TP=3 uneven, weighted DCP; full battery GREEN):
- REPLICATED-KV active on all 10 global layers (q [8,4,4]); coherence,
  1024-tok greedy long-form clean; needle at ~20k tokens HIT in eager
  AND graphs -- the core correctness proof of token-sharded replicated
  attention.
- Calibrated max_total_num_tokens = 2,187,648 WITH CUDA graphs
  (2.19M-token KV pool on 72 GB of consumer GPUs).
- MTP/NEXTN works under replicated-KV: 134.3 tok/s single-stream,
  conc-8 8/8 (bs>1 verify graphs); no-MTP decode 121.0 tok/s, prefill
  4170 tok/s; max_total 1,631,168 with draft.
- HiCache: storage restore of 12,992 tokens with identical greedy IDs
  cold<->restore, needle from restored cache correct, conc-8 8/8 (the
  M26 owner-rule translation carries replicated-KV unchanged).
- Boot-to-boot determinism: first-request logprob maxdelta 0.000000,
  top-5 identical over 32 steps.
- 27B regression: byte-identical (git-stash A/B: max_total 487616
  exact, greedy long-form byte-identical, logprob maxdelta 0.000000);
  MoE method never instantiated for dense models.

Open (documented): GGUF-A3B MoE tensor name-map + AWQ moe_wna16 uneven
loader (sgl-project#64 family), gemma4 uneven bring-up, 27B TP=5 co-located
Docker recipe prepared, OQ-1 falsifier not run.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Qwen3.6-27B <qwen3.6-27b@localhost>
efschu added a commit to efschu/htsglang that referenced this pull request Jul 22, 2026
@efschu
[GGUF] Perf overhaul: flat layout, batched MMVQ, quantized vocab, Q8 …
5a5baf8 
…support

Closes the measured GGUF performance gap to llama.cpp (M20 analysis:
61.6 vs ~41.5 ms/MTP-iteration at Q6 TP=2) and turns it around at
TP>=3. Four measures plus the Q8 loader fix, all GGUF-gated — AWQ/FP8/
non-GGUF paths byte-identical (proven, see validation).

1. Flat single-buffer shard layout (gguf.py): merged shards live
   back-to-back in ONE 16-byte-aligned uint8 buffer with static 2-D
   views; apply() uses the views — removes the ~1.9 GB/rank/forward
   slice+contiguous() copy traffic of mixed-dtype GDN qkvz layers.
   Graph-safe by construction.
2. Persistent dequant workspace (sgl-project#63): grow-only per-(device,dtype)
   workspace, pre-reserved BEFORE KV profiling; ggml_dequantize gains
   an optional out= (schema-detected; old wheel = byte-identical
   fallback). Kills the 86-MiB-per-forward alloc OOM class.
3. Quantized-resident lm_head/embed (default; SGLANG_GGUF_DENSE_VOCAB=1
   escape hatch): token_embd/output stay Q8_0-class instead of bf16
   materialization (1.26 vs 2.37 GiB; 4 lm_head passes per MTP
   iteration read half the bytes). NEXTN shares embed/lm_head at
   MODULE level (draft 2.62 -> 0.24 GB). Row-granular qweight vocab
   sharding replaces the M24 fail-fast — works even AND with
   --rank-vocab-ratio.
4. Batched MMVQ ncols<=8 (mmvq.cuh): one weight-block load serves up
   to 8 columns (llama.cpp pattern), bit-identical accumulation order
   per column, dispatcher behind all 19 type launchers; default-on iff
   the new-wheel schema is detected (SGLANG_GGUF_BATCHED_MMVQ env
   override; old wheel keeps the device-adaptive crossover).
5. (sgl-project#64) Mixed-dtype fused GDN qkvz (UD-Q8_K_XL: F16 z next to Q8_0
   qkv etc.): flat layout carries mixed dtypes, dense shards cast once
   at load; dense-module forcing now only for F32. Q8 at TP>1 was
   INFEASIBLE in both forks — now boots and serves.
Build: sgl-kernel gains gated SGL_KERNEL_LIMIT_CUDA_ARCHS /
SGL_KERNEL_SKIP_SM90_VARIANT options (default-off = stock builds).
Requires a rebuilt sgl_kernel wheel for measures 2/4 (graceful
fallback without).

Validation (T66_VALIDATION.md + HANDOFF M29; all boots thermal-gated,
GPUs 0 MiB after):
- Token identity: kill-switch boot (dense vocab, batched off) is
  BYTE-IDENTICAL to HEAD (3x1024 greedy long-forms, logprob maxdelta
  0.000000, accepts equal) -> measures 1+2 exact. Batched-MMVQ boot is
  not byte-identical BY CONSTRUCTION (dispatch shift MMQ->MMVQ at
  M=3..8): both kernels 4.93e-3 relmax to fp32 reference, 1.18e-4 to
  each other, logprob sampled IDs identical over 32 steps, maxdelta
  0.041, accepts within 0.1. Quantized vocab: IDs identical, sampled
  maxdelta 0.104, accepts equal or better (3.91/3.62/2.61).
- Kernel bit-parity batched-vs-per-column MMVQ M=1..8 + out= parity
  PASS on sm86 and sm120.
- Q6 TP=2: 54.5 -> 66.0 tok/s code (+21%), weights 14.03 -> 12.74
  GB/rank; Q6 TP=3-uneven+DCP: 65.9/54.6 -> 91.1/79.1 (+37%/+47%) —
  above every llama.cpp cell on this rig (79.6/74.5); accept lengths
  unchanged (3.35 vs 3.36).
- Q8_K_XL TP=3: boots coherent at 91.1/82.3 tok/s (previously
  INFEASIBLE, gguf.py:475 mixed-dtype assert).
- Q4_K_M TP=2: +5% decode, maxKV 153k -> 208k (+36%), 14.7k prefill
  without the sgl-project#63 OOM. Q5_K_M + heretic-Q6 smokes pass.
- --rank-vocab-ratio auto on quantized GGUF vocab: boots, coherent.

Operational notes: freed VRAM shifts auto-sizing — boot recipes in
HANDOFF M29 (TP=2 memfrac 0.80; TP=3 reserve 3500 +
--cuda-graph-max-bs-decode 8); sizing follow-up tracked. MoE-GGUF
(qwen35moe) keeps its own unmodified path.
efschu added a commit to efschu/htsglang that referenced this pull request Jul 22, 2026
@efschu
[Feature] TP > num_kv_heads: replicated-KV + token-sharding (generic)
76e6142 
Models whose attention layers carry fewer KV heads than TP ranks could
not run under (uneven) TP at all -- the whole-head split requires
kv >= tp (and even TP requires tp % kv == 0). This lands the analyzed
approach generically: when the uneven plan is active and kv_total < tp,
the affected full-attention layers switch to REPLICATED-KV mode -- all
kv heads on every rank (deterministic recompute from byte-identical
replicated k/v projection weights, no broadcast), q heads split in
GQA-group-sized units (q/kv, keeping num_qo % num_kv == 0 per rank),
tokens sharded via the existing weighted-DCP owner rule and merged with
the graph-validated LSE all-gather. NO KV-cache duplication (disjoint
token shards -> capacity stays the sum) and comm-neutral (the per-layer
kv-head all-gather in _dcp_masked_write is skipped for these layers).
GDN/linear-attention layers keep normal head sharding; the NEXTN draft
(also kv-deficient) replicates the same way incl. its full-context
pool. Parametric geometry only -- no model special cases: covers kv=2
models on TP=3/4 (Qwen3.6-35B-A3B, Gemma-4-26B-A4B) AND e.g. the 27B
(kv=4) on TP=5+ ([8,4,4,4,4] q-split). Missing DCP -> fail fast naming
SGLANG_UNEVEN_DCP=1; tp > GQA group size -> clear ValueError.

Also wires the previously caller-less fused-MoE FP8 Marlin fallback
(pre-existing platform gap; A3B's first forward crashed with fp8e4nv
unsupported on sm86): Fp8MoEMethod auto-enables Marlin W8A16 per rank
on 80<=sm<89 (native fp8 stays on sm120 -> mixed-arch TP with disjoint
expert shards, combined by all-reduce), zero-padding non-64-aligned
uneven expert shards locally (numerically exact). Micro-test vs fp32
dequant reference on sm86 and sm120: cos >= 0.99998. By-catch fixes:
group-quant expert units in fused_moe_triton (was FP8-block-only) and
a fused-aware AWQ skip list.

Validation (A3B FP8 TP=3 uneven, weighted DCP; full battery GREEN):
- REPLICATED-KV active on all 10 global layers (q [8,4,4]); coherence,
  1024-tok greedy long-form clean; needle at ~20k tokens HIT in eager
  AND graphs -- the core correctness proof of token-sharded replicated
  attention.
- Calibrated max_total_num_tokens = 2,187,648 WITH CUDA graphs
  (2.19M-token KV pool on 72 GB of consumer GPUs).
- MTP/NEXTN works under replicated-KV: 134.3 tok/s single-stream,
  conc-8 8/8 (bs>1 verify graphs); no-MTP decode 121.0 tok/s, prefill
  4170 tok/s; max_total 1,631,168 with draft.
- HiCache: storage restore of 12,992 tokens with identical greedy IDs
  cold<->restore, needle from restored cache correct, conc-8 8/8 (the
  M26 owner-rule translation carries replicated-KV unchanged).
- Boot-to-boot determinism: first-request logprob maxdelta 0.000000,
  top-5 identical over 32 steps.
- 27B regression: byte-identical (git-stash A/B: max_total 487616
  exact, greedy long-form byte-identical, logprob maxdelta 0.000000);
  MoE method never instantiated for dense models.

Open (documented): GGUF-A3B MoE tensor name-map + AWQ moe_wna16 uneven
loader (sgl-project#64 family), gemma4 uneven bring-up, 27B TP=5 co-located
Docker recipe prepared, OQ-1 falsifier not run.
@fooling fooling mentioned this pull request Oct 22, 2026
Open
@kpham-sgl kpham-sgl mentioned this pull request Nov 5, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@merrymercy