Repository navigation
Additional k3 sync fixes - #67
Merged
Merged
Conversation
_grouped_conv shifts rows across the flattened [bs * block_size] token dimension and masked the taps that cross a block boundary by multiplying with a 0/1 mask. NaN * 0 and Inf * 0 are NaN, so a non-finite value in request i's last block rows reached request i+1's first rows, and the DFlash2 draft stack (attention_conv and mlp_conv, prepare and finish, in every layer) carried it one request further per layer. Select the cross-block taps to exact zeros with torch.where before the multiply. Finite outputs are unchanged apart from the sign of exact zeros. The boundary check and the DFlashGroupedConv prepare/finish check run both eagerly and through the compiled function (inductor on the CPU runner), since the engine calls _grouped_conv through torch.compile. Upstream PR: sgl-project#41736 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…ucer The bf16 -> e4m3 KV producers used overflow-to-NaN (NOSAT) casts, so a finite value beyond the e4m3 range was stored as a NaN byte. Saturate to +-448 instead (SATFINITE) in every producer. to_fp8_satfinite is a no-op for inputs already in the target dtype, matching Tensor.to semantics. Port note: v0.5.20 names the fused MLA KV/Q-concat JIT kernel set_mla_kv_concat_q.cuh; the NOSAT -> SATFINITE change is applied there. MLATokenToKVPool keeps the base None/empty-rope guard. Upstream PR: sgl-project#41732 Co-Authored-By: Rahul Chalamala <22563365+rchalamala@users.noreply.github.com> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…o_fp8_satfinite Port note: v0.5.20 _quantize_fp8_qkv inlines the checkpoint-scale lookup; only the three raw casts are replaced. Upstream PR: sgl-project#41732 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…n all-NaN score row stays in range The regression test runs the Triton kernel on CUDA (base-b-kernel-unit, 1-gpu-large) and skips without a GPU. Upstream PR: sgl-project#41737 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…he CuTe DSL MTP kernel Graph-padded requests (ssm_state_indices == -1) all share the last real cu_seqlens endpoint: graph metadata collapses every padded request's query_start_loc to the real-token endpoint (hybrid_linear_attn_backend.py, verify branch with num_padding > 0), so the kernel's pad branch cleared only the first padded request's T_LOOP rows and left the remaining padded rows as torch.empty bytes. The wrapper enforces a fixed dense 1 + num_spec width per request, so the pad branch now clears row block i_n * T_LOOP. An instrumented engine run observed the first padded request's dense output interval zero in 100% of 5.3M observations across 8 ranks while padded requests 2 and 3 were nonzero in 6.1%-59.0% of theirs. Adds test_cutedsl_multiple_padding_outputs_and_states: sentinel-filled padded output intervals and state slots exercised through the real wrapper, in eager and CUDA-graph replay, with snapshot and ReplaySSM-ring state arms and raw and fused-onorm outputs, for one, two and three padded requests. Real-request outputs must match a real-requests-only control run bitwise and all untouched padding state must stay unchanged. Port note: the v0.5.20 test helper has no split_v parameter; only the new kwargs_only parameter is added. Upstream PR: sgl-project#41733 Co-Authored-By: Rahul Chalamala <22563365+rchalamala@users.noreply.github.com> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Under DFLASH the scheduler never advanced the penalizer orchestrator, so repetition, frequency and presence penalties had no effect and min_new_tokens never counted up. spec_prepare_for_decode now feeds every token committed since the previous step (all accepted speculative tokens, not only the last) through a new cumulate_output_tokens_multi path. Port note: the test's fixtures follow v0.5.20 (req.eos_token_ids, get_exec() instead of get_server_args()). Upstream PR: sgl-project#41738 Co-Authored-By: Rahul Chalamala <22563365+rchalamala@users.noreply.github.com> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…and adopts the other's delta When an earlier session turn had no M-RoPE delta, merge() dropped the new turn's delta, and it never cleared mrope_position_delta_repeated_cache, so decode positions after a multimodal session continuation came from a stale or missing delta. Upstream PR: sgl-project#41739 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
prepare_load_back raw-allocs the request's device mamba slot, bypassing HybridReqToTokenPool.alloc's fresh-slot hygiene, and the H2D load-back writes only temporal+conv (the host tier carries no ring state). The GDN ReplaySSM decode kernel replays ring contents whenever write_pos>0, so a load-back slot inherited the ring of the slot's previous request on top of the loaded checkpoint: cross-conversation recurrent-state bleed, observed as sticky deterministic wrong retrievals (~10% of revisits) served through HiCache load-back only. Reset write_pos on the freshly allocated slot before the load-back lands, mirroring the fresh-slot reset in HybridReqToTokenPool.alloc. Adds a CPU unit test for prepare_load_back. Upstream PR: sgl-project#41740 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…d-back HiCache load-back gates device state reads with Python-level per-layer waits (layer_transfer_counter.wait_until in the KV / mamba pool accessors). A captured graph body never executes those waits on replay. The breakable backend runs attention and linear attention as eager graph breaks, so the waits still fire there, and tc_piecewise is already disabled with hierarchical cache. The full prefill backend replays the whole transformer body as one graph, so a prefill batch carrying a live consumer index could read KV / mamba state before the H2D load landed (the standing "disable cuda graph execution if hicache loading triggered" TODO in Scheduler.get_new_batch_prefill). Under the full backend, enqueue one wait on the pending load op's FINAL event on the forward stream before replay. This keeps the graph and trades the layerwise H2D/compute pipelining for that batch only. Upstream PR: sgl-project#41741 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…ow the sampler SamplingParams.verify() checked logit_bias keys but not values. A value beyond the float32 range raises inside the scheduler loop while the batch is built, which takes the server down; NaN, infinite and non-numeric values fail the same way or leave the request with no finite probability. A bias that fits in float32 can still become inf after division by the temperature, and a tiny repetition_penalty does the same, so torch.multinomial raises inside the scheduler. verify() now accepts |logit_bias| <= 1e30 and repetition_penalty in [1e-6, 2], so these requests get a 400 before they reach the scheduler. Both bounds sit far past any value that changes sampling: a bias of -1e9 still bans a token, and repetition_penalty 1e-3 is still accepted. Upstream PR: sgl-project#41734 Co-Authored-By: Rahul Chalamala <22563365+rchalamala@users.noreply.github.com> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Sibling of the zero-scale check; raises ValueError instead of asserting. Upstream PR: sgl-project#41742 Co-Authored-By: Rahul Chalamala <22563365+rchalamala@users.noreply.github.com> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Check numel before the finite check so multi-element scales raise a clear ValueError instead of an ambiguous RuntimeError, and evaluate isfinite on CPU copies. Upstream PR: sgl-project#41742 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The final multimodal placement check silently keeps a suffix when an embedding has too many rows, which can shift image rows onto different placeholder tokens, and counted only the first dimension of higher-rank encoder outputs. Mismatches are now counted over flattened token rows and reported with both counts and a chunked-prefill hint. Cropping overlong embeddings with a warning stays the default. SGLANG_ENABLE_STRICT_MM_EMBEDDING_LENGTH=1 raises on any mismatch instead. Model-owned padding/trimming and EVS placeholder redistribution keep their existing behavior. Upstream PR: sgl-project#41744 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Validate normalized prompts against the text input vocabulary and the active processor markers for their supplied modalities, and bound synthetic warmup inputs. Output-token bounds, embedding overrides, session continuation and batch metadata setup keep their existing contracts. Upstream PR: sgl-project#41735 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
SGLANG_MM_STRIP_PROCESSOR_INPUT_IDS is an independent payload optimization tracked in sgl-project#41776. The caller token-ID validation and warmup bound are unchanged. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
rodamani
requested review from
Fridge003,
Qiaolin-Yu and
hnyls2002
as code owners
September 30, 2026 21:55
github-actions
Bot
added
quant
jit-kernel
npu
memory-pool
unified-radix-cache
blackwell
hicache
labels
When SGLANG_MM_STRIP_PROCESSOR_INPUT_IDS is set, a processor output whose input_ids list was copied into the canonical request array is replaced with a copy that drops that duplicate list. Outputs with attached state or weak references are left unchanged. Off by default. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Adds correctness fixes on top of
upstream/k3-sync(v0.5.20 + K3 performance commits). Each fix is also open againstsgl-project/sglangmain; the commits here are the v0.5.20 ports. Defaults are unchanged except where a fix replaces a NaN/out-of-range result with a finite or rejected one.Modifications
One commit per fix:
to_fp8_satfinite, no-op for inputs already FP8)_quantize_fp8_qkvsaturateMultimodalInputs.mergeinvalidates the M-RoPE delta cachelogit_bias/repetition_penaltyvalues that overflow the samplerSGLANG_ENABLE_STRICT_MM_EMBEDDING_LENGTH=1); cropping stays the defaultinput_idslist once copied into the request (SGLANG_MM_STRIP_PROCESSOR_INPUT_IDS=1); off by defaultThe "Use neutral wording" commit only rewords a few new comments.
Accuracy Tests
CPU: the 14 new or changed CPU test files pass locally (the 2 CUDA-only DFlash2 selector cases skip). Not run here, GPU only:
kernel/jit/test_set_mla_kv_concat_q_fp8.py,kernel/speculative/test_dflash_selector_walk_nan_row.pyandkernels/test_kda_mtp_cutedsl_replayssm_ring.py. The DFlash2 selector test passed on H100 and the KDA test on B200 in the upstream PRs' CI.Speed Tests and Profiling
Not benchmarked on this branch.
Checklist
Link to Devin session: https://modal.devinenterprise.com/sessions/e00892013a0d4707b3c872eb7e9b8ba6
Open in Devin Desktop: https://modal.devinenterprise.com/desktop/session/e00892013a0d4707b3c872eb7e9b8ba6?variant=devin
Requested by: @rodamani
CI States
Latest PR Test (Base): ❌ Run #36783100041
Latest PR Test (Extra): ❌ Run #36783099428
Latest PR Test (AMD ROCm 10): ❌ Run #36783099616