Skip to content

[NPU] use causal_conv1d_update_v2 for performance#24595

Merged
sglang-npu-bot merged 3 commits into
sgl-project:mainfrom
iridiumine:causal_conv1d_update_v2
May 12, 2026
Merged

[NPU] use causal_conv1d_update_v2 for performance#24595
sglang-npu-bot merged 3 commits into
sgl-project:mainfrom
iridiumine:causal_conv1d_update_v2

Conversation

@iridiumine
Copy link
Copy Markdown
Contributor

@iridiumine iridiumine commented May 7, 2026

Motivation

Use causal_conv1d_update_v2 instead of torch.ops.npu.causal_conv1d_update for performance.

Modifications

  • Use causal_conv1d_update_v2 instead of torch.ops.npu.causal_conv1d_update.
  • Fix sgl_kernel can not import bug by adding is_npu().

Accuracy Tests

+-----------------+-----------+----------+----------+-------+---------+---------+
| Model           | Dataset   | Metric   | Subset   |   Num |   Score | Cat.0   |
+=================+===========+==========+==========+=======+=========+=========+
| Qwen3.5-35B-A3B | gsm8k     | mean_acc | main     |   200 |   0.975 | default |
+-----------------+-----------+----------+----------+-------+---------+---------+

Speed Tests and Profiling

  • Qwen3.5-397B: improve from ~300us to ~140us.
  • Qwen3-next: improve from ~550us to ~200us.

Checklist

Review and Merge Process

  1. Ping Merge Oncalls to start the process. See the PR Merge Process.
  2. Get approvals from CODEOWNERS and other reviewers.
  3. Trigger CI tests with comments or contact authorized users to do so.
    • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
  4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

@github-actions github-actions Bot added the npu label May 7, 2026
gemini-code-assist[bot]
gemini-code-assist Bot reviewed May 7, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the NPU backend to use causal_conv1d_update_v2 and refactors the Triton backend to conditionally import utilities based on the hardware environment. Feedback suggests replacing a hardcoded padding value with the class attribute self.pad_slot_id to maintain consistency across the NPU backend.

Comment thread python/sglang/srt/hardware_backend/npu/attention/ascend_gdn_backend.py
gemini-code-assist[bot]
gemini-code-assist Bot reviewed May 7, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the NPU backend to utilize the causal_conv1d_update_v2 kernel and introduces conditional imports in the Triton backend to support NPU environments. Feedback indicates a potential bug where calling .contiguous() on conv_states could lead to lost state updates if a copy is created. It is also suggested to replace the hardcoded pad_slot_id with self.pad_slot_id for better consistency.

Comment thread python/sglang/srt/hardware_backend/npu/attention/ascend_gdn_backend.py
Comment thread python/sglang/srt/hardware_backend/npu/attention/ascend_gdn_backend.py
@sglang-npu-bot
Copy link
Copy Markdown
Collaborator

sglang-npu-bot commented May 7, 2026

/tag-and-rerun-ci

@github-actions github-actions Bot added the run-ci label May 7, 2026
@iridiumine
Copy link
Copy Markdown
Contributor Author

iridiumine commented May 11, 2026

/rerun-failed-ci

@iridiumine
Copy link
Copy Markdown
Contributor Author

iridiumine commented May 12, 2026

/rerun-failed-ci

@sglang-npu-bot sglang-npu-bot merged commit 693f497 into sgl-project:main May 12, 2026
100 of 126 checks passed
SpencerGarnets added a commit to ai-blaise/optimization-playground that referenced this pull request May 12, 2026
@SpencerGarnets
Merge sgl-project/sglang upstream main into fork (preserves custom st…
76ccea8 
…ack)

Brings in upstream sgl-project/sglang main commits since
096ad02 (merge base, Laguna-XS.2 model support).
Total: 28 upstream commits composed.

Custom-stack files preserved intact (entirely-ours, byte-identical to
origin/main):
  - Blackwell CuTe kernel suite (warp_decode_cute, g1_attention_cute,
    gated_norm_cute, layersplit_cute, fused_store_index_cache)
  - TurboQuant 2.5-bit dense KV cache path
  - HIGGS 2-bit dense KV cache path (with split-K decode)
  - NVFP4 IndexCache dispatcher (active gate)
  - quantization_config_dispatch (HF-config-driven runtime routing)
  - All custom server-args flags and runtime methods preserved

Verification:
  - 200+ merged Python files compile cleanly
  - Dispatcher symbol presence verified
  - HIGGS pool / TurboQuant pool classes present at expected lines
  - compressed_tensors_w4a4_nvfp4_moe imports clean
  - All custom server-args flags present (enable_higgs_dense_2bit_kv_cache,
    enable_turboquant_dense_kv_cache, turboquant_dense_kv_preset,
    indexer_quantization_declared, higgs_mla_decode_num_splits, etc.)

Manual-merged shared files (auto-merge gave broken/mixed output; cleaned
up post-merge):
  - python/sglang/srt/disaggregation/mooncake/conn.py: upstream's PR#24932
    refactored maybe_send_extra into a state-types-loop. Replayed our
    LayerSplit NSA state-index-length-mismatch check inside the SWA/NSA
    branch of the new loop body.
  - sgl-kernel/python/sgl_kernel/__init__.py: upstream's PR#23449 (Apple
    Silicon Metal kernel) wrapped the entire module body in
    `if darwin/arm64: from sgl_kernel.metal import * else: ...`. The
    auto-merge duplicated the file body; rewrote cleanly with upstream's
    structure and re-injected our `g1_gate_forward`,
    `warp_decode_cute_moe_forward`, and
    `warp_decode_cute_moe_packed_forward` imports plus `g1_gate_forward`
    in _DEBUG_EXPORT_NAMES.
  - python/sglang/srt/managers/scheduler_output_processor_mixin.py: line
    628 still referenced `result.num_accepted_drafts` (renamed by PR
    sgl-project#25038 to `num_correct_drafts`). Renamed in place.
  - python/sglang/srt/observability/scheduler_metrics_mixin.py: a block
    around the spec-decode logging path had mixed old/new names from
    auto-merge (lines 553/557/560). Renamed `spec_num_accepted_tokens`
    -> `spec_num_accept_tokens` and local `num_accepted_drafts` ->
    `num_correct_drafts` to match the rest of the file.
  - test/test_smc_info.py: stub Req mock used the old field names
    `spec_accepted_drafts` and `update_spec_acceptance_histogram`.
    Renamed to `spec_num_correct_drafts` and
    `update_spec_correct_drafts_histogram` per PR sgl-project#24081.

Auto-merge cleanly integrated upstream changes to:
  - server_args.py (new fields: prefill_only_disable_kv_cache,
    weight_loader_drop_cache_after_load, prefill_delayer_queue_min_ratio,
    prefill_delayer_max_delay_ms, speculative_draft_window_size, etc.)
  - mem_cache/memory_pool.py (new NoOpMHATokenToKVPool)
  - model_executor/model_runner_kv_cache_mixin.py (NoOpMHATokenToKVPool
    pool factory + _validate_prefill_only_disable_kv_cache_pool_family)
  - layers/attention/nsa_backend.py (spec rename
    num_accepted_drafts -> num_correct_drafts;
    num_accepted_tokens -> num_accept_tokens)
  - layers/attention/nsa/nsa_indexer.py (new _apply_q_scale_and_softmax_scale
    compile method; torch.mm replaces deep_gemm wrapper)
  - 28+ disaggregation/spec/runner files with mostly clean
    upstream-side-only integration.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

----- upstream commit subjects (28) -----
fd3eb77 [Cookbook]: add Laguna-XS.2 (Poolside) (sgl-project#24730)
6be1a45 Fix swa component host hit (sgl-project#25085)
693f497 [NPU] use causal_conv1d_update_v2 for performance (sgl-project#24595)
1efe9e2 [Bug Fix] Reject incompatible combination of --disable-cuda-graph-padding and --enable-torch-compile (sgl-project#23903)
8d27ce7 Optimize uvicorn startup command (sgl-project#25041)
b35fd5f [fix] skip legacy minicpmv conv template for MiniCPM-V 4.6 (sgl-project#24998)
7582237 [Tiny Fix] Disable BCG when inner layer_model unresolved (sgl-project#25021)
ca3bc05 Deepseek-v4-Pro share expert tp1 (sgl-project#24949)
a72d3ae [Spec] Multi-layer mamba scatter cleanup; fix positional call bug (sgl-project#25030)
7128533 Revert "Migrate Intel CPU cases to the test/registered." (sgl-project#25044)
1f985c5 [Spec] Rename `accepted_indices` -> `accept_indices`; drop `_token_id` suffix per Rule 5 (sgl-project#25038)
ecf5d84 Migrate Intel CPU cases to the test/registered. (sgl-project#22670)
d7f4761 [PD] Refactor hybrid state transfer (sgl-project#24932)
91907b7 [UnifiedTree]: Fix Unified HiCache tombstone lock release replay (sgl-project#24972)
4ad63ad [Spec] Rename `accepted_drafts` -> `correct_drafts` for unambiguous naming (sgl-project#24081)
6bfb365 [PD] Rate limit prefill inflight polling warnings (sgl-project#24967)
6bb79c1 [Linear Attn] Add CUSTOM enum and plugin extensibility for kernel backends (sgl-project#24937)
cfc41d5 Fix kimi k2.5 mla eagle + dp attention (sgl-project#25033)
0f3932c [Fix] Qwen3-ASR config: set thinker_config before super().__init__ (sgl-project#24187)
f526e3f [Spec] Mamba scatter cleanup; fix multi-layer positional bug; dflash naming (sgl-project#25029)
10375a1 [NIXL][XPU] Fix uint64 overflow for mismatched P/D TP sizes (e.g. prefill_tp=1, decode_tp=2) (sgl-project#24648)
0a37d24 [diffusion] hardware: support sage attention backend on MUSA (attn backend, 21/N) (sgl-project#24752)
5495026 [HiCache] feat: default storage prefetch timeout (sgl-project#23309)
186eb42 Feat: Support SWA (Sliding Window Attention) for EAGLE-3 drafter (sgl-project#24664)
a75b79e Feat: Support newer EAGLE-3 drafters (sgl-project#24663)
f3a8189 [Spec] Internal rename per N2 v2 naming rule (sgl-project#25014)
bfc2eda [MUSA] Use MUSA-optimized operators in piecewise CUDA graph (sgl-project#23633)
74d70af [Apple Silicon] Add Metal kernel support in sgl-kernel (sgl-project#23449)
xjpang pushed a commit to xjpang/sglang that referenced this pull request May 13, 2026
@iridiumine
[NPU] use causal_conv1d_update_v2 for performance (sgl-project#24595)
fc0c1ce 
Co-authored-by: iridiumine <iridiumine@users.noreply.github.com>
Shunkangz pushed a commit to Shunkangz/sglang that referenced this pull request May 27, 2026
@iridiumine
[NPU] use causal_conv1d_update_v2 for performance (sgl-project#24595)
21b0449 
Co-authored-by: iridiumine <iridiumine@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sglang-npu-bot sglang-npu-bot sglang-npu-bot approved these changes

@merrymercy merrymercy Awaiting requested review from merrymercy

@Fridge003 Fridge003 Awaiting requested review from Fridge003

@ispobock ispobock Awaiting requested review from ispobock

@Qiaolin-Yu Qiaolin-Yu Awaiting requested review from Qiaolin-Yu

@hebiao064 hebiao064 Awaiting requested review from hebiao064

@HaiShaw HaiShaw Awaiting requested review from HaiShaw

@ping1jing2 ping1jing2 Awaiting requested review from ping1jing2 ping1jing2 is a code owner

@iforgetmyname iforgetmyname Awaiting requested review from iforgetmyname iforgetmyname is a code owner

+1 more reviewer

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

npu run-ci

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@iridiumine @sglang-npu-bot