Support double sparsity by andy-yang-1 · Pull Request #1459 · sgl-project/sglang

andy-yang-1 · 2024-09-18T22:57:50Z

Motivation

Support double sparsity (post-training sparse attention) for long context inference in SGLang
See paper

Modifications

Add triton implementation in sglang/python/sglang/srt/layers/sparse_decode_attention.py
Add serving-related parts

Speedup Evaluation

Run double sparsity with:

python -m sglang.bench_latency --model-path lmsys/longchat-7b-v1.5-32k \
    --attention-backend triton --disable-cuda-graph \
    --ds-channel-config-path /path/to/lmsys/longchat-7b-v1.5-32k.json \
    --input-len 20000 --output-len 200 \
    --batch-size 3 \
    --enable-double-sparsity \
    --ds-heavy-channel-num 16 \
    --ds-heavy-token-num 1024 \
    --ds-sparse-decode-threshold 0 \
    --max-total-tokens 70000

Benchmark ...
Prefill. latency: 7.83636 s, throughput:   7656.62 token/s
Decode.  latency: 0.02351 s, throughput:    127.58 token/s
Decode.  latency: 0.02124 s, throughput:    141.22 token/s
Decode.  latency: 0.02037 s, throughput:    147.26 token/s
Decode.  latency: 0.01950 s, throughput:    153.81 token/s
Decode.  latency: 0.01935 s, throughput:    155.04 token/s
Decode.  median latency: 0.01923 s, median throughput:    156.04 token/s
Total. latency: 11.821 s, throughput:   5126.36 token/s

Original triton implementation:

python -m sglang.bench_latency --model-path lmsys/longchat-7b-v1.5-32k \
    --attention-backend triton \
    --input-len 20000 --output-len 200 \
    --batch-size 3

Benchmark ...
Prefill. latency: 7.79627 s, throughput:   7695.98 token/s
Decode.  latency: 0.07196 s, throughput:     41.69 token/s
Decode.  latency: 0.06514 s, throughput:     46.05 token/s
Decode.  latency: 0.06475 s, throughput:     46.33 token/s
Decode.  latency: 0.06463 s, throughput:     46.41 token/s
Decode.  latency: 0.06457 s, throughput:     46.46 token/s
Decode.  median latency: 0.06487 s, median throughput:     46.25 token/s
Total. latency: 20.720 s, throughput:   2924.74 token/s

Original flashinfer implementation:

python -m sglang.bench_latency --model-path lmsys/longchat-7b-v1.5-32k \
    --attention-backend flashinfer \
    --input-len 20000 --output-len 200 \
    --batch-size 3

Benchmark ...
Prefill. latency: 5.68892 s, throughput:  10546.83 token/s
Decode.  latency: 0.03240 s, throughput:     92.60 token/s
Decode.  latency: 0.02993 s, throughput:    100.23 token/s
Decode.  latency: 0.02970 s, throughput:    101.01 token/s
Decode.  latency: 0.02959 s, throughput:    101.39 token/s
Decode.  latency: 0.02959 s, throughput:    101.38 token/s
Decode.  median latency: 0.02961 s, median throughput:    101.32 token/s
Total. latency: 11.585 s, throughput:   5231.00 token/s

With Llama-3.1-8B:

# Double Sparsity
python -m sglang.bench_latency --model-path meta-llama/Llama-3.1-8B-Instruct \
    --attention-backend triton \
    --ds-channel-config-path /path/to/meta-llama/Llama-3.1-8B-Instruct.json \
    --input-len 60000 --output-len 200 \
    --batch-size 3 \
    --enable-double-sparsity \
    --ds-heavy-channel-num 32 \
    --ds-heavy-channel-type k \
    --ds-heavy-token-num 3000 \
    --ds-sparse-decode-threshold 0 \
    --max-total-tokens 200000

Benchmark ...
Prefill. latency: 42.96801 s, throughput:   4189.16 token/s
Decode.  latency: 0.02843 s, throughput:    105.50 token/s
Decode.  latency: 0.02518 s, throughput:    119.16 token/s
Decode.  latency: 0.02465 s, throughput:    121.72 token/s
Decode.  latency: 0.02442 s, throughput:    122.84 token/s
Decode.  latency: 0.02434 s, throughput:    123.24 token/s
Decode.  median latency: 0.02421 s, median throughput:    123.90 token/s
Total. latency: 47.793 s, throughput:   3778.77 token/s

# Triton
python -m sglang.bench_latency --model-path meta-llama/Llama-3.1-8B-Instruct \
    --attention-backend triton \
    --input-len 60000 --output-len 200 \
    --batch-size 3 \
    --max-total-tokens 200000

Benchmark ...
Prefill. latency: 43.17160 s, throughput:   4169.41 token/s
Decode.  latency: 0.06359 s, throughput:     47.18 token/s
Decode.  latency: 0.05965 s, throughput:     50.30 token/s
Decode.  latency: 0.05927 s, throughput:     50.62 token/s
Decode.  latency: 0.05906 s, throughput:     50.80 token/s
Decode.  latency: 0.05906 s, throughput:     50.80 token/s
Decode.  median latency: 0.05913 s, median throughput:     50.73 token/s
Total. latency: 54.950 s, throughput:   3286.63 token/s

# Flashinfer
python -m sglang.bench_latency --model-path meta-llama/Llama-3.1-8B-Instruct \
    --attention-backend flashinfer \
    --input-len 60000 --output-len 200 \
    --batch-size 3 \
    --max-total-tokens 200000

Benchmark ...
Prefill. latency: 27.50800 s, throughput:   6543.55 token/s
Decode.  latency: 0.03014 s, throughput:     99.54 token/s
Decode.  latency: 0.02834 s, throughput:    105.86 token/s
Decode.  latency: 0.02821 s, throughput:    106.36 token/s
Decode.  latency: 0.02819 s, throughput:    106.41 token/s
Decode.  latency: 0.02823 s, throughput:    106.28 token/s
Decode.  median latency: 0.02821 s, median throughput:    106.34 token/s
Total. latency: 33.125 s, throughput:   5452.12 token/s

Checklist

Format your code according to the Contributor Guide.
Add unit tests as outlined in the Contributor Guide.
Update documentation as needed, including docstrings or example tutorials.

merrymercy · 2024-09-19T09:32:15Z

Great work. Some tips for rebasing:

Recently, we did some refactoring to introduce AttentionBackend. This will make the code of supporting multiple attention backend cleaner Refactor attention backend #1381
Following the above refactor, you can use cuda graph and torch.compile to make it run up to 4x faster (Support cuda graph in the triton attention backend #1401, Enable torch.compile for triton backend #1422)

ghost · 2024-09-24T20:57:26Z

Quick question @andy-yang-1 - Does this PR support just Double Sparsity or DS-Offload as well?

andy-yang-1 · 2024-09-24T21:10:01Z

@vnkc1 Hi, this PR doesn't support DS-Offload for now. DS-Offload may be integrated in other PR if needed.

fengyang95 · 2024-10-09T05:07:50Z

Is there a plan to merge this PR?

merrymercy · 2024-10-11T09:02:14Z

Yes. It should be merged within one week.
@andy-yang-1 please

Resolve the conflicts.
Add an end-to-end accuracy unit test

merrymercy · 2024-10-14T02:55:10Z

Please fix the lint error and add an end-to-end accuracy test

merrymercy · 2024-10-14T07:48:59Z

Give two example commands and past their results in the description of this PR. This is for tracking the progress. It should be something like this

# baseline
python3 -m sglang.bench_latency --model meta-llama/Meta-Llama-3.1-8B-Instruct --batch-size 1 --input 1024 --output 8

# double sparsity
python3 -m sglang.bench_latency --model meta-llama/Meta-Llama-3.1-8B-Instruct --batch-size 1 --input 1024 --output 8 --enable-double-sparsity ...

merrymercy · 2024-10-14T08:16:23Z

@andy-yang-1 Can you also paste the latency results?

andy-yang-1 · 2024-10-16T17:51:19Z

@max99x You can use this link to generate channel config file.

@fengyang95 We may support it in the next PR

fengyang95 · 2024-10-18T17:26:07Z

hi @andy-yang-1 Does this support the deepseek-v2 architecture? How can I obtain the config for this structure? I see that the example here https://github.com/andy-yang-1/DoubleSparse/blob/main/evaluation/group_channel_config.py only support llama/mixtral arch.

fengyang95 · 2024-10-19T18:16:50Z

@andy-yang-1 I tried running the deepseek-v2 model, but encountered the following issue:

File "/opt/tiger/custome_sglang/python/sglang/srt/layers/attention/double_sparsity_backend.py", line 162, in forward_extend
    k_label = torch.gather(
              ^^^^^^^^^^^^^
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:3 and cuda:0! (when checking argument for argument index in method wrapper_CUDA_gather)

  File "/opt/tiger/custome_sglang/python/sglang/srt/layers/attention/__init__.py", line 49, in forward
    return self.forward_extend(q, k, v, layer, forward_batch)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/tiger/custome_sglang/python/sglang/srt/layers/attention/double_sparsity_backend.py", line 162, in forward_extend
    k_label = torch.gather(
              ^^^^^^^^^^^^^
RuntimeError: Size does not match at dimension 1 expected index [7, 128, 16] to be smaller than self [7, 1, 576] apart from dimension 2

andy-yang-1 · 2024-10-19T21:41:03Z

@fengyang95 I haven't added support for deepseek-v2 model. I may add support for this later

fengyang95 · 2024-10-20T04:03:11Z

@fengyang95 I haven't added support for deepseek-v2 model. I may add support for this later

@andy-yang-1 Thank you very much! Looking forward to support for deepseek-v2 and cuda graph.

shreyansh26 · 2024-11-06T14:21:23Z

@andy-yang-1 - Loved the paper! I was trying this out and I am facing a few issues generating the config file using the mentioned script.

The line cos, sin = m.rotary_emb(v, seq_len=kv_seq_len) in stat_qk_max_hook of get_calib_qk_feat gives an error

TypeError: LlamaRotaryEmbedding got an unexpected keyword argument 'seq_len'

I replaced it with cos, sin = m.rotary_emb(v, position_ids=position_ids) which works. I'm not sure if that is correct but LlamaRotaryEmbedding indeed doesn't have the seq_len param

In the config file that gets generated, I only get keys of the form model.layers.{layer_num}.self_attn but the config file present in the test folder has keys in the form of model.layers.{layer_num}.self_attn.q_proj, model.layers.{layer_num}.self_attn.k_proj and model.layers.{layer_num}.self_attn.qk_proj. How were these generated?
On using my generated config with sglang, I am getting error of the type - Key model.layers.0.self_attn.k_proj was not found.

Any help on how to run this would be appreciated.

andy-yang-1 · 2024-11-06T18:08:07Z

@shreyansh26 The first problem is caused by older version of transformers, and I will update the base repo to fix it this week.
The q_outlier_config/k_outlier_config is generated with get_calib_feat function, and the qk_outlier_config is generated with get_qk_calib_feat function. You can merge this two config together to get all configs. I will also update it this week.

shreyansh26 · 2024-11-07T07:31:49Z

Thank you.
There may be another discrepancy, in get_calib_feat, with the following condition, k_proj gets filtered out because of GQA.

if y.shape[-1] != model.config.hidden_size:
    return

But in the Llama-3.1-8B-Instruct config file, k_proj keys are also present.

andy-yang-1 · 2024-11-10T23:13:48Z

@shreyansh26 Hi, I have updated the main repo. Can you try with this code?

shreyansh26 · 2024-11-11T10:02:03Z

Thank you @andy-yang-1!! This is working perfectly now.

yuguo-Jack · 2024-11-23T14:36:13Z

@vnkc1 Hi, this PR doesn't support DS-Offload for now. DS-Offload may be integrated in other PR if needed.
Is there a plan to support DS-Offload in Sglang?

hcyz33 · 2025-01-13T06:16:53Z

Motivation

Support double sparsity (post-training sparse attention) for long context inference in SGLang
See paper

Modifications

Add triton implementation in sglang/python/sglang/srt/layers/sparse_decode_attention.py
Add serving-related parts

Speedup Evaluation

Run double sparsity with:

python -m sglang.bench_latency --model-path lmsys/longchat-7b-v1.5-32k \
    --attention-backend triton --disable-cuda-graph \
    --ds-channel-config-path /path/to/lmsys/longchat-7b-v1.5-32k.json \
    --input-len 20000 --output-len 200 \
    --batch-size 3 \
    --enable-double-sparsity \
    --ds-heavy-channel-num 16 \
    --ds-heavy-token-num 1024 \
    --ds-sparse-decode-threshold 0 \
    --max-total-tokens 70000

Benchmark ...
Prefill. latency: 7.83636 s, throughput:   7656.62 token/s
Decode.  latency: 0.02351 s, throughput:    127.58 token/s
Decode.  latency: 0.02124 s, throughput:    141.22 token/s
Decode.  latency: 0.02037 s, throughput:    147.26 token/s
Decode.  latency: 0.01950 s, throughput:    153.81 token/s
Decode.  latency: 0.01935 s, throughput:    155.04 token/s
Decode.  median latency: 0.01923 s, median throughput:    156.04 token/s
Total. latency: 11.821 s, throughput:   5126.36 token/s

Original triton implementation:

python -m sglang.bench_latency --model-path lmsys/longchat-7b-v1.5-32k \
    --attention-backend triton \
    --input-len 20000 --output-len 200 \
    --batch-size 3

Benchmark ...
Prefill. latency: 7.79627 s, throughput:   7695.98 token/s
Decode.  latency: 0.07196 s, throughput:     41.69 token/s
Decode.  latency: 0.06514 s, throughput:     46.05 token/s
Decode.  latency: 0.06475 s, throughput:     46.33 token/s
Decode.  latency: 0.06463 s, throughput:     46.41 token/s
Decode.  latency: 0.06457 s, throughput:     46.46 token/s
Decode.  median latency: 0.06487 s, median throughput:     46.25 token/s
Total. latency: 20.720 s, throughput:   2924.74 token/s

Original flashinfer implementation:

python -m sglang.bench_latency --model-path lmsys/longchat-7b-v1.5-32k \
    --attention-backend flashinfer \
    --input-len 20000 --output-len 200 \
    --batch-size 3

Benchmark ...
Prefill. latency: 5.68892 s, throughput:  10546.83 token/s
Decode.  latency: 0.03240 s, throughput:     92.60 token/s
Decode.  latency: 0.02993 s, throughput:    100.23 token/s
Decode.  latency: 0.02970 s, throughput:    101.01 token/s
Decode.  latency: 0.02959 s, throughput:    101.39 token/s
Decode.  latency: 0.02959 s, throughput:    101.38 token/s
Decode.  median latency: 0.02961 s, median throughput:    101.32 token/s
Total. latency: 11.585 s, throughput:   5231.00 token/s

With Llama-3.1-8B:

# Double Sparsity
python -m sglang.bench_latency --model-path meta-llama/Llama-3.1-8B-Instruct \
    --attention-backend triton \
    --ds-channel-config-path /path/to/meta-llama/Llama-3.1-8B-Instruct.json \
    --input-len 60000 --output-len 200 \
    --batch-size 3 \
    --enable-double-sparsity \
    --ds-heavy-channel-num 32 \
    --ds-heavy-channel-type k \
    --ds-heavy-token-num 3000 \
    --ds-sparse-decode-threshold 0 \
    --max-total-tokens 200000

Benchmark ...
Prefill. latency: 42.96801 s, throughput:   4189.16 token/s
Decode.  latency: 0.02843 s, throughput:    105.50 token/s
Decode.  latency: 0.02518 s, throughput:    119.16 token/s
Decode.  latency: 0.02465 s, throughput:    121.72 token/s
Decode.  latency: 0.02442 s, throughput:    122.84 token/s
Decode.  latency: 0.02434 s, throughput:    123.24 token/s
Decode.  median latency: 0.02421 s, median throughput:    123.90 token/s
Total. latency: 47.793 s, throughput:   3778.77 token/s

# Triton
python -m sglang.bench_latency --model-path meta-llama/Llama-3.1-8B-Instruct \
    --attention-backend triton \
    --input-len 60000 --output-len 200 \
    --batch-size 3 \
    --max-total-tokens 200000

Benchmark ...
Prefill. latency: 43.17160 s, throughput:   4169.41 token/s
Decode.  latency: 0.06359 s, throughput:     47.18 token/s
Decode.  latency: 0.05965 s, throughput:     50.30 token/s
Decode.  latency: 0.05927 s, throughput:     50.62 token/s
Decode.  latency: 0.05906 s, throughput:     50.80 token/s
Decode.  latency: 0.05906 s, throughput:     50.80 token/s
Decode.  median latency: 0.05913 s, median throughput:     50.73 token/s
Total. latency: 54.950 s, throughput:   3286.63 token/s

# Flashinfer
python -m sglang.bench_latency --model-path meta-llama/Llama-3.1-8B-Instruct \
    --attention-backend flashinfer \
    --input-len 60000 --output-len 200 \
    --batch-size 3 \
    --max-total-tokens 200000

Benchmark ...
Prefill. latency: 27.50800 s, throughput:   6543.55 token/s
Decode.  latency: 0.03014 s, throughput:     99.54 token/s
Decode.  latency: 0.02834 s, throughput:    105.86 token/s
Decode.  latency: 0.02821 s, throughput:    106.36 token/s
Decode.  latency: 0.02819 s, throughput:    106.41 token/s
Decode.  latency: 0.02823 s, throughput:    106.28 token/s
Decode.  median latency: 0.02821 s, median throughput:    106.34 token/s
Total. latency: 33.125 s, throughput:   5452.12 token/s

Checklist

Format your code according to the Contributor Guide.
Add unit tests as outlined in the Contributor Guide.
Update documentation as needed, including docstrings or example tutorials.

I found that the throughput of prefill is lower when enable DS attention(from 6543.55 to 4189.16 ). The possible reason is that you use triton as attention-backend. Is it possible to use flashinfer attention in prefill to increase the throughput of prefill.

alex1720-web · 2025-10-10T01:11:21Z

Hi, @andy-yang-1 , that is a great work!
I have encountered a problem when using double sparsity in the latest version of SGLang. I followed the same command as you, but it turned out to fail. The error log is in the following. Could you help me to fix it ?

CUDA_VISIBLE_DEVICES=0 python -m sglang.bench_one_batch --model-path /home/lyy/model
s/Mistral-7B-v0.1 --tensor-parallel-size 1 --attention-backend triton --disable-cuda-graph --ds-channel-config-path /home/zongyi/DoubleSparse/config/mistralai/Mistral-7B-v0.1.json --input-len 20000 --output-len 200 --batch-size 1 --enable-double-sparsity --ds-heavy-channel-num 16 --ds-heavy-token-num 1024 --ds-sparse-decode-threshold 0 --max-total-tokens 70000

/root/miniconda3/envs/zy-sgl/lib/python3.11/site-packages/torch/cuda/init.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
torch_dtype is deprecated! Use dtype instead!
[2025-10-09 14:21:16 TP0] Double sparsity optimization is turned on. Use triton backend without CUDA graph.
[2025-10-09 14:21:16 TP0] Init torch distributed begin.
[rank0]:[W1009 14:21:17.340194889 ProcessGroupGloo.cpp:514] Warning: Unable to resolve hostname to a (local) address. Using the loopback address as fallback. Manually set the network interface to bind to with GLOO_SOCKET_IFNAME. (function operator())
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[2025-10-09 14:21:17 TP0] Init torch distributed ends. mem usage=0.00 GB
[2025-10-09 14:21:18 TP0] Ignore import error when loading sglang.srt.models.kimi_vl: cannot import name 'GELUTanh' from 'transformers.activations' (/home/zongyi/transformers/activations.py)
[2025-10-09 14:21:18 TP0] Ignore import error when loading sglang.srt.models.kimi_vl_moonvit: cannot import name 'GELUTanh' from 'transformers.activations' (/home/zongyi/transformers/activations.py)
[2025-10-09 14:21:18 TP0] Load weight begin. avail mem=46.94 GB
Loading safetensors checkpoint shards: 0% Completed | 0/3 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 33% Completed | 1/3 [00:00<00:01, 1.30it/s]
Loading safetensors checkpoint shards: 67% Completed | 2/3 [00:01<00:00, 1.25it/s]
Loading safetensors checkpoint shards: 100% Completed | 3/3 [00:02<00:00, 1.19it/s]
Loading safetensors checkpoint shards: 100% Completed | 3/3 [00:02<00:00, 1.21it/s]

[2025-10-09 14:21:21 TP0] Load weight end. type=MistralForCausalLM, dtype=torch.bfloat16, avail mem=33.37 GB, mem usage=13.57 GB.
[2025-10-09 14:21:21 TP0] Using KV cache dtype: torch.bfloat16
[2025-10-09 14:21:21 TP0] Memory pool end. avail mem=23.76 GB
max_total_num_tokens=70000
Warmup ...
[rank0]: Traceback (most recent call last):
[rank0]: File "", line 198, in _run_module_as_main
[rank0]: File "", line 88, in _run_code
[rank0]: File "/root/miniconda3/envs/zy-sgl/lib/python3.11/site-packages/sglang/bench_one_batch.py", line 660, in
[rank0]: main(server_args, bench_args)
[rank0]: File "/root/miniconda3/envs/zy-sgl/lib/python3.11/site-packages/sglang/bench_one_batch.py", line 624, in main
[rank0]: work_func(server_args, port_args, bench_args, 0)
[rank0]: File "/root/miniconda3/envs/zy-sgl/lib/python3.11/site-packages/sglang/bench_one_batch.py", line 531, in latency_test
[rank0]: latency_test_run_once(
[rank0]: File "/root/miniconda3/envs/zy-sgl/lib/python3.11/site-packages/sglang/bench_one_batch.py", line 433, in latency_test_run_once
[rank0]: next_token_ids, _, batch = extend(reqs, model_runner)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/miniconda3/envs/zy-sgl/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
[rank0]: return func(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/miniconda3/envs/zy-sgl/lib/python3.11/site-packages/sglang/bench_one_batch.py", line 275, in extend
[rank0]: logits_output, _ = model_runner.forward(forward_batch)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/miniconda3/envs/zy-sgl/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 1982, in forward
[rank0]: output = self._forward_raw(
[rank0]: ^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/miniconda3/envs/zy-sgl/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2033, in _forward_raw
[rank0]: ret = self.forward_extend(
[rank0]: ^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/miniconda3/envs/zy-sgl/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 1927, in forward_extend
[rank0]: return self.model.forward(
[rank0]: ^^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/miniconda3/envs/zy-sgl/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
[rank0]: return func(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/miniconda3/envs/zy-sgl/lib/python3.11/site-packages/sglang/srt/models/llama.py", line 469, in forward
[rank0]: hidden_states = self.model(
[rank0]: ^^^^^^^^^^^
[rank0]: File "/root/miniconda3/envs/zy-sgl/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/miniconda3/envs/zy-sgl/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
[rank0]: return forward_call(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/miniconda3/envs/zy-sgl/lib/python3.11/site-packages/sglang/srt/models/llama.py", line 342, in forward
[rank0]: hidden_states, residual = layer(
[rank0]: ^^^^^^
[rank0]: File "/root/miniconda3/envs/zy-sgl/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/miniconda3/envs/zy-sgl/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
[rank0]: return forward_call(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/miniconda3/envs/zy-sgl/lib/python3.11/site-packages/sglang/srt/models/llama.py", line 266, in forward
[rank0]: hidden_states = self.self_attn(
[rank0]: ^^^^^^^^^^^^^^^
[rank0]: File "/root/miniconda3/envs/zy-sgl/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/miniconda3/envs/zy-sgl/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
[rank0]: return forward_call(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/miniconda3/envs/zy-sgl/lib/python3.11/site-packages/sglang/srt/models/llama.py", line 197, in forward
[rank0]: attn_output = self.attn(q, k, v, forward_batch)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/miniconda3/envs/zy-sgl/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/miniconda3/envs/zy-sgl/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
[rank0]: return forward_call(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/miniconda3/envs/zy-sgl/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 108, in forward
[rank0]: return forward_batch.attn_backend.forward(
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/miniconda3/envs/zy-sgl/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 82, in forward
[rank0]: return self.forward_extend(
[rank0]: ^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/miniconda3/envs/zy-sgl/lib/python3.11/site-packages/sglang/srt/layers/attention/double_sparsity_backend.py", line 128, in forward_extend
[rank0]: k_label = torch.gather(
[rank0]: ^^^^^^^^^^^^^
[rank0]: RuntimeError: Size does not match at dimension 1 expected index [20000, 32, 16] to be no larger than self [20000, 8, 128] apart from dimension 2
[rank0]:[W1009 14:21:23.273092827 ProcessGroupNCCL.cpp:1538] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())

IgniteGo · 2026-01-29T02:53:31Z

Thank you for your excellent work! I have 3 questions I'd like to ask:

(1) Performance Comparison
Based on the test results you provided in the PR and my own local tests, Double Sparsity shows a slight improvement over SGLang + Triton, but compared to SGLang + FlashInfer (the default backend), there is no significant degradation (and sometimes it's roughly on par).
Is this mainly due to inherent efficiency limitations of the Triton sparse operator itself? Or are factors such as kernel optimization, lack of CUDA Graph support, etc., having a bigger impact?

(2) Model Differences
In the PR benchmarks, the relative improvement of Double Sparsity on LongChat-7B is clearly larger than on Llama-3.1-8B. What is the main reason for this difference?

(3) Main Advantage Scenarios of Double Sparsity
What are the primary scenarios where Double Sparsity shows the strongest advantages?
How does its speedup correlate with hardware (compute vs. memory bandwidth on H100/A100), input/output lengths, batch size, and model architecture (e.g., GQA, context length)?

I would greatly appreciate it if you could spare some time to answer these questions. Thank you in advance!

…*) replaces the stat-per-page probe; the front queues one request ahead on P Boot weg2xsn214: between two 100k prefills on P the GPU idled ~3 s -- the new request's intake store probe (sgl-project#1400 STORE-TOLD, span 94k after the 4k shared prefix) stat'ed ~94k page files to learn that none exists, and the front only sent the next request once the previous one was served, so the probe never overlapped compute. Six prompts: ~18 s of ~180 s. * arena.c: l3idx_* -- an open-addressing table of BLAKE2b-128 stem keys in shm (<arena dir>/l3idx.bin, 2^23 entries x 16 B = 134 MB), writers under a spinlock, readers lock-free; add/del/has by stem, clear, count. * l3_index.py: mapping + creator/joiner protocol; open_index never raises; SGLANG_HICACHE_L3_INDEX=0 turns it off. * lru_file_evictor: the one bookkeeper of the disk store keeps the table exact -- commit adds, the evict unlink removes. * hicache_storage: _l3_index() opens it beside the arena and hands it to the evictor; _stat_stems asks it first and stats only the stems it names (a stem it does not know is not on disk); clear() clears it. * front: P takes p_concurrency + SGLANG_WEG2_P_QUEUE_AHEAD (default 1) requests at once; the extra one waits in P's queue (max-running-requests bounds compute) while its probe runs in the shadow of the prefill. Tests: test_weg2_l3_index_1459 (4: two mappings, full table, env off, wiring) + arena claim/direct-write/L3-fill + dormant_hold_1445: passed.

…dir; hold-time presence claims adopt the group MIN instead of DRAFT-DISAGREE STOP Boot weg2xsn216 (sgl-project#1459): D died at the second park-test request -- WEG2 DRAFT-DISAGREE STOP rid=weg2-6-3 per_rank_claim=[94207, 97870]. The held request's probe ran 1 s after P's leg answered, while P's write- through was still landing the tail; TP2 probed ms before TP0/TP1 and read fewer COMPLETE pages. That inequality is the write-through's clock, not a divergent store: for rids in the dormant hold every rank adopts the group MIN (the same prefix everywhere, sgl-project#1461 DRAFT-CLAIM MIN-ADOPTED) and the hold's top-up (sgl-project#1456) fetches the rest; outside the hold the law stays STOP (assert_draft_claims_agree unchanged). The scheduler keeps cache_controller.weg2_hold_rids in step with the hold (add / release / abort). #1459b: the index file moves out of the arena dir into a sibling dir (<arena dir>-l3idx/l3idx.bin) -- nothing that scans the arena dir sees it. Tests: hold_refetch_1456 (+1) + dormant_hold_1445 + l3_index_1459: passed.

xsn217 measured the sgl-project#1459 pair form: -7 finished 26.2 s after -6 (the 3 s store probe overlapped), but -8 took 29.5 s again because the batch of p_concurrency+ahead was gathered whole before the next dispatch, so only every second request had a successor queued on P. _p_drain_pool keeps p_concurrency+ahead legs in flight and refills the moment ONE finishes; on_done runs in completion order; a state change stops new dispatches but never abandons a leg in flight (the gather had the same property). Tests: test_weg2_l3_index_1459 7 passed (3 new: pool refill timing, stop-dispatch drains in-flight, empty queue).

…faces (798/951 retired with #1015c, slice-A, fix3, 1290, 1233, 1153) Pre-existing reds on desk/27b-unified-0926 (b3180fb), all but 1153 also red on the 27B line (bfc6bd8) and the NF line (d1c7094). Test files only. - pp_void_slot_advance_798 x7 -> successor pins. sgl-project#1015 (01a391f) deleted the admission-decision arc the whole file drove (_pp_send/_recv_admission_ decision, _pp_drain_voided_proxy, ...) and made the proxy channel unconditional; #1015c (206661a) retired the downstream refusal (pp_upstream_void_pending False; boot_pp3solo_01a391fa03: every downstream pass refused, 0 served). The file now pins both halves of that retirement (sgl-project#631's restored _pp_reconcile_incoming_admission excluded by name). - pp_upstream_void_before_formation_951 x3 (found in the same family): the three refusal cases rewritten to the #1015c form (pass reaches the formation, predicate False on every shape, no void -> streak 0, slot kept). - scheduling_slice_a_0907: t1 sgl-project#1459 34b6d14 (P takes p_concurrency + SGLANG_WEG2_P_QUEUE_AHEAD, default 1); t4/g3 sgl-project#1356 bfc252c (argv_p keyword-only); f4c sgl-project#1444 dc15687 (front_argv_for reads ns.weg2_weight_source -> the launcher's own parser namespace); f1a: the case set awake='P' mid-flip, which since sgl-project#1443/xsn347 (4336934) is the P->D flip D legitimately admits into; it now models the D->P flip as flip() runs it (awake stays 'D' until the end). Mutant (post-acquire re-check removed) -> red. t3 is the front defect, next commit. - sched_fix3_0907 b1d-b1g: H42b 7370b10 hoisted the pass-level burst verdict into a local tested in the loop (b1d: locals assigned from a call/attribute count as state; b1e-g: the slice gets its stock values _burst_hold=None, _carried_n=0) and xsn326 a3489cc reads module `time` (real clock). Round-5 mutant (`if req is not None: continue` above the gate) still red on all four. - no_route_1290 x2: Front.__new__ harness + vision (sgl-project#58 0aeb52e), the sgl-project#49 epoch keyword of price_remainder (196f6a8), H84 x_start_tokens (b895928). - s3s4_refusals_1233 w25: #1474 c7cd8ae wrapped handle_generate_request around _handle_generate_request_impl; the seam is the impl's first statement, the wrapper is pinned to clock-read + delegation. - pp_forwarded_refusal_stop_1153 x2 (unified only): H93 RU 6cda2aa plants the usable-match floor on tree_cache; harness tree_cache=None.

…-09-27: "der l3 hicache wird jedesmal verworfen, oder? der soll natürlich persistent sein" / "aber natürlich zwischen 27b und nf verschiedene L3 caches. sonst knallts") Defect: plan_store named the store directory after the BOOT TAG and sweep_store_residue rmtree'd every other directory under the root at the next boot, so every boot started with an empty L3 (the NF docker mount held only dkrnfh91dprbar1dauer09271632). Hidden second half: the shared L3 stem index (sgl-project#1459) answers "not on disk" for any stem it does not know, and a freshly created index knows none -- a kept directory would still read as all misses. Fix (switch SGLANG_WEG2_L3_PERSIST, default on; 0 = the per-boot store as before): - launcher: the store is <root>/l3-<profile>-<model>-<identity10>; identity = model realpath, sha1 of the model config, profile, form_kv, kv_cache_dtype, on-disk generation (706). The directory carries L3_IDENTITY.json, checked BEFORE the boot touches a file: a mismatch is a named W57 refusal, never a shared store. The sweep spares every l3-* directory (this model's and the other's) and removes only per-boot tag directories. l3_persist_attach walks once before any rank starts: counts the pages ("L3-PERSIST reuse|fresh dir=... files=... bytes=... staging_removed=...") and unlinks crash-left .tmp.<uuid> staging files (no writer is alive yet); canonical partials stay with HiCacheFile's marker sweep. The disk check credits the reused directory's own bytes (walked only when the plain check would refuse), so a full store does not refuse its own reboot. - hicache_storage: the process that CREATES the L3 index seeds it from _iter_existing_files ("L3-PERSIST index_seeded=N"); joiners querying before the seed completes see misses only. Facts checked: page keys already carry compute_model_identity_hash; Mamba/GDN state is in L3 as the sgl-project#706 canonical mamba blob; clear_storage_backend runs only on the /clear_hicache_storage_backend API, not on reset or flip; the evictor's attach census adopts the previous boot's pages (own_known=None), so the cap holds across boots; the docker arm mounts $ACC/store per line (27b / nf), so the volume already persists. Cap: per model = --store-max-gb of that model's boot; the disk check counts the other model's store as used space, so the sum cannot exceed the disk minus min_free. W8b on the warm store (closed, not left for metal): the evictor's census only indexes its own group's suffixes, so the sibling group's private-suffix pages a previous boot left would count as seen-but-unindexed and could drop coverage under the 50 % floor. In a persistent store the launcher proved (L3_IDENTITY.json present, switch on) such unindexed files OLDER than the evictor are counted as inherited_bytes/entries, logged ("L3-PERSIST W8b graded without N inherited file(s)") and left out of the gate's denominator; files written after the evictor was born still count, so a blind scan filter on this boot's own writes is still refused at the wake re-scan. Disk sum (user "L3, jeder 150gb"): each store records L3_CAP.json {max_size_bytes, bytes_at_attach}; at attach "L3-PERSIST DISK-SUM ok|WARN" checks sum over all l3-* stores of (cap - bytes at their last attach) + min_free against free space, a named warning with the numbers when short, never a silent cap. Measured: /mnt/nvfast (XFS, bind-mounted as $ACC/store for both lines) 1.8 TB, 1.4 TB free -> 2 x 150 GB + 32 GiB min_free fits. Tests: test_weg2_l3_persist_0927.py 23 passed (W8b: 60 % sibling-private pages inherited, no refusal; without identity file / switch off / files newer than the owner the refusal stands; disk sum fits, short = named WARN, no-record stores named) (switch; 27B/NF two named directories, stable name, kv format / generation / re-quantised config -> new directory; identity written, matched, foreign refused; dry run writes nothing; sweep keeps both models' l3-* dirs and removes the tag dir; attach counts pages and drops staging, dry run removes nothing; disk check credits own bytes, no walk when roomy, tag dir without a name; index seed in 4096 batches, switch off, creator only; wiring order sweep < identity < attach < prepare). Comparison fix vs base e1eb275, 31 neighbouring files (incl. every LRUFileEvictor suite) one by one via test27b.sh: identical pass/fail on every file (pre-existing reds in test_fence_store_decider_1063, test_hicache_era_purge, test_weg2_s5_store_carrier, test_weg2_dormant_vram_1y, test_hicache_file_lru_unit on both sides).

…miss -- the P followers reach one store depth (z30y14 root) z30y14 (27B image 02adfaa, 01.10. 03:15:38Z, sgl-project#1400 STORE-TOLD MISMATCH rid=weg2-64-273 told=97631 own_prefix=93858). The store VIEW was the same on every rank: PP1 and PP2 both logged ARENA-EVICT n=2 want=256 need=88 at 03:15:36 -- need = len(full) of arena_fill_from_disk's first claim, so all 88 L3 candidates had passed the rank's stat (the shared sgl-project#1459 L3 stem index). What diverged was the CONCURRENT FILL: both followers filled the same 88 canonical pages at the same second, and arena_fill_from_disk treated a claim answered status 1 (JOIN: arena.c "write your extents, then complete") as a miss that ends the prefix. Each rank stopped at the other's first claim -- "sgl-project#1433 L3->L2 fill: 5 of 5" / "12 of 12", "ARENA-FREE reason= l3fill_past_prefix slots=59" / "55" -- and reached 95684 / 95716 tokens, while PP0, reading alone one pass earlier, reached 97631. The join's open writer was also never resolved, and claims given back past the prefix were freed whole (free_slots) even when the other rank had joined them (#1427r). The frozen follower census (PP1/PP2 and TP1/TP2: the same file count and 0 B sibling at every wake; NF y6b 66664 files) is NOT a stale read view: LRUFileEvictor.rescan returns the attach-time census for a non-owner by design (no walk; rescan_eviction_index docstring). The wake line now says owner=yes / owner=no (attach-time census reprinted, reads ask sgl-project#1459). Fix (shared modules, NF-portable): - python/sglang/srt/mem_cache/hicache_storage.py arena_fill_from_disk: a status-1 join with a slot is read and completed like a fresh claim (the whole page from the same file; the second completer gets status 2 = usable); unread claims (past the prefix, failed read) go back through arena.release_claims (frees only a sole claimant), new helper _release_fill_claims; a join refused at completion is not freed by the joiner; the sgl-project#1433 line counts joined=. - python/sglang/srt/managers/scheduler_components/weight_updater.py: the WEG2-STORE-RESCAN line names owner vs non-owner (appended field only). PF (d0fd844) stays on as the safety net. Tests (CUDA_VISIBLE_DEVICES="", real shm arena + real files): - new test/registered/unit/mem_cache/test_l3fill_join_z30y14.py: red on d0fd844 (3 failed: the prefix ends at the other rank's first claim [5..], the two followers' depths differ, the joined page is not read), green here (4 passed). - changed area: L3 fill / fast read / probe hold / anchor drop / park L3 / substages 59 passed + 1 pre-existing failure (rc12z double lacks prefix=, same on d0fd844); test_arena_* family: the same 4 pre-existing failures before and after, all else passed; rescan/wake suites 35 passed.

…L2/L3-Store-Praefix der exakten IDs (src=l3_index), 27B-Seite 1 Token Port of NF 2c1150f onto desk/27b-nfport-route-1002 (f405217). At arrival, after the exact count, the front asks the shared sgl-project#1459 L3 stem index (<arena dir>-l3idx/l3idx.bin, joined with L3Index(create=False) -- the ranks create and seed it) for the leading run of page keys as the tree writes them (bigram, complete pages, the store's group-wide L3_SUFFIXES suffix), cut by every all-pages component, ended at the deepest page carrying the mamba blob; recorded as a store fact (TokenSpans.record_store_depth, label l3_index). Launcher hands the front SGLANG_HICACHE_BIGRAM_KEYS and SGLANG_HICACHE_ARENA_DIR (the groups' own values). Markers: 'WEG2 L3-INDEX-PRICE armed ...' at the open, 'WEG2 L3-INDEX-PRESENCE rid= depth= pages= kv_pages= tokens= probe_ms=', 'X-EXACT-PRICE ... src=l3_index'; /weg2/state presence_price.l3_index(_tokens). 27B check (boot ..._1002_083747): both groups page_size=1, DFLASH, SGLANG_HICACHE_BIGRAM_KEYS=1; the ranks joined '/dev/shm/weg2-arena-<tag>-l3idx/l3idx.bin (cap 8388608, entries 1806831)' -- the same path front_store derives from the arena dir; the 27B store (l3-qwen27b-...) writes L3_SUFFIXES.<group> (L3P N1) and keys shared KV and shared mamba pages with the same kv_config_suffix (one suffix fits). 27B adaptations: (1) page = ONE token: a 111k prompt is 111k page keys; hashing 16 ms, the forward anchor scan asked every page (~200 ms per probe measured on a real L3Index). The trailing-component scan now walks from the end, one CHUNK at a time (same answer: the deepest anchor sits in the last chunk on a hit). (2) the first page hash imports the radix key code (20 s cold measured, beside the 3 s price timeout): warmed up in the worker thread at the probe's open, not on the first arrival. Not ported: NF's d_inflight presence_price counters (no D-INFLIGHT on 27B). Coverage of the 27B classes: N3y weg2-8-15 (P hit 109703 = 60551 + 48 x 1024, an inner chunk-end anchor of weg2-0-4's prefill) is credited when the store holds that anchor -> pending 1553 <= X=4896, SHORT. N3y weg2-0-2 is NOT store credit (its hit was its twin 0-1 computed in the same P phase, 0-1 cached 0): the probe credits 0 -- correct, no over-credit. Tests: test_weg2_l3_index_price_1002.py (NF, unchanged, green on 27B) + test_weg2_l3_index_price_27b_1002.py (5, page 1: chained tree keys, 8-15 -> 109703, 0-2 -> 0, KV without anchor -> 0, backward scan == forward answer).

…cord AND sgl-project#1459 index and queue; 0 file I/O on the scheduler thread, transitively (SOLL R7 R11; user 20.10.) User order 20.10. (verbatim): "UND WEHE DAS SCHREIBEN IN DEN CACHE BLOCKIERT DURCH IRGENDEINE SYNCHRON ODER WARTESCHEISSE". (1) ArenaMHAHostPool._evict_for_claim (sgl-project#257 d, scheduler thread): l3_secured.unsecured_keys_lo (one C census of the COMPLETE slots + memory reads) puts every page without an L3 copy -- by the write-behind's published record or the shared sgl-project#1459 L3 stem index (HiCacheFile.l3_listed: the evictor's exact shm bookkeeping, no stat, no open) -- on the keep list of every stage and queues it for the write-behind; the taken candidates are verified, never written (l3_secured.verify_candidates; one that re-completed between census and clock is the old stage-iii loss, named). (2) HiCacheFile._arena_evict_to_disk (clock evict): the same keep list (extra_lo); none copied: S109d ARENA-EVICT PENDING, asked again; a race: S109d ARENA-EVICT RACE dropped_without_l3. Note: its callers are the storage / fill threads (batch_get, _batch_set_each, _arena_fill_from_disk_impl, _await_fill_room_loop), not the scheduler thread; converted per the order (writing into the cache is never held by a synchronous write). Return value unchanged (pages written: 0 now). (3) ArenaMambaPoolHost.anchor_held: arena COMPLETE -> True; else the sgl-project#1459 index (l3_listed), never a stat; no opened index -> None (cannot tell). A reader attaches the shared record at most once a second until the owner made it (ATTACH_RETRY_S). Review 20.10. ("NEIN in dieser Form", deskq/done/nf-s108-s109-review-1020.md), fixed before this commit: A1 (data loss) l3_secured.secured / unsecured_keys_lo = published record (this generation) AND the sgl-project#1459 L3 index (HiCacheFile.l3_listed, registered per arena at _open_arena); and the L3 LRU evictor's on_evict (HiCacheFile._l3sec_on_evict, its own thread) voids the record of every slot holding the evicted stem. A file the L3 cap unlinked is never read as a copy again. Review tests R4/R5 green (R1-R3 skip without S108). A2 ArenaMambaPoolHost._secure_before_drop is read-only (returns the keep set; drop_unreferenced drops only copied anchors, queues the rest, check writer "anchor_drop"); old body kept as _secure_before_drop_now (writer form). H19 displacement (park spill, flush spill) therefore no longer stats or writes 56 MiB + fsync on the scheduler thread. The check is TRANSITIVE now: a runtime guard counts every os.stat/lstat/open/fsync, builtins/io.open, HiCacheFile.arena_secure_to_disk/_stat_stems call on the calling thread through claim, clock, secure_rows_to_l3, H19 displacement (3-rank harness) and the S86 sleep release + sgl-project#1470 flush round: 0. It found one more: hicache_arena._evict_gens_for did os.path.realpath (an lstat per path component) on every claim/clock call -- resolved once at ShmArena attach now. A3 verify_candidates returns a mask; claim and clock free only candidates with a copy. A race candidate without one (re-completed between census and clock) is NOT freed and NOT counted lost: l3_secured.hand_over queues it for the hand-over writer (thread pdflip-l3-evict-handover: arena_secure_to_disk writer=evict_handover, then free_evicted_named); a failed copy keeps it EVICTING, named (S109-A3 EVICT-HANDOVER). A4 request() sets l3_secured.WAKE; the write-behind waits with wait_or_stop(stop, wait_s), so a queued page is written at once instead of at the next 2-s tick. S86 before the sleep (rc19): with the pinned pages copied by the write-behind the park-end chain gets in, lost_tokens 0, 0 file I/O; with half of them still uncopied S86 keeps those (not_on_l3) and the released half is room enough, lost 0, the request woke the write-behind. Boundary, named not hidden: with NONE of the pinned pages copied at the sleep, S86 releases nothing (not_on_l3:N) and the chain is lost as on the S85 base (S85 PARK-END-UNBACKED lost_tokens>0 in the log) -- the old S86 wrote the copies synchronously there, which the order of 20.10. forbids. Source check over all 17 scheduler forms (test_s109d_zero_file_io_on_every_scheduler_form) + the transitive runtime guard above: 0 file I/O. Mutants (test/registered/unit/mem_cache/mutants_s109_1020.py, serial under /spinning/gpu-arb/pytest.lock): 16 of 16 killed (M1-M8: secured counts all; claim / clock without the keep list; anchor_held stat; no publish; aux stage off; S109c worker decides; pending counted as copied. A1a secured record alone; A1c census record alone; A1b LRU evict leaves the record; A2a drop_unreferenced drops uncopied anchors; A2b realpath per call; A3a claim frees uncopied race candidates; A3b hand-over frees without a copy; A4 request does not wake). Checks of the order: (a) the write-behind never pauses for flip or sleep: quiet_reason() is None since S36 (18.10.); rc26/rc25 D+P 0x "L3-REUSE WRITE-BEHIND gate="; D TP0 26 passes paused=- inside its 25 sleep windows (383 s), P PP0 21 while asleep. (b) "== RANKS -> == 1" is right under Form A: one shared /dev/shm arena attached by TP0/TP1/TP2 (rc26 D 5539/5542/ 5559), token cut [0, 9, 7] with each owner writing its own rows of every page (D 8/10, arena_pool.py:1462 sgl-project#239 S4b F14), COMPLETE only when the page is full (hicache_arena.py:1060), the write-behind writes whole COMPLETE slots (hicache_storage.py:3842/3967); the mamba anchor arena is TP0's alone (D 5564). Tests seam: the conftests of mem_cache/, pdflip/ and (new) managers/ swap in the writer's forms (verify_candidates_now = the pre-S109d sgl-project#257 write, byte for byte; unsecured_keys_lo_none) for the claim/clock LOGIC tests; adapted pin: test_pdflip_anchor_only_backup_0930::test_y6b_anchor_held_reads_the_arena_then_the_l3_index (the index, not a stat). TESTS (serial, flock /spinning/gpu-arb/pytest.lock, systemd-run --scope -p MemoryMax=2G, CUDA_VISIBLE_DEVICES=""): 91 affected suites (incl. test_review_s108_s109_1020 from review/nf-s108-s109-1020 66db115b2f): 85 green, 6 with only the pre-existing reds below (10 tests, identical to the base). The review snapshot's 8 reds (test_pdflip_d_norecompute_1002 1, test_pdflip_mamba_last_resort_1008 3, test_pdflip_park_end_anchor_first_1303 4): green (9/7/19 passed). test_s109_backup_wall_aux_leaves_1020 35 passed, test_s86_sleep_l3_release_1020 11, test_pdflip_mamba_arena_displace_h19 9, test_review_s108_s109_1020 2 passed 3 skipped. Pre-existing red on rc27 56f3a0c (checked in a base worktree): test_pdflip_read_room_1015 (1), test_pdflip_arenafull_1015 (2), test_arena_l3_fill_group_rc12z (1), test_flush_arena_full_stop_flushfast_1012 (2), test_l3_fill_orphan_claim_nf (2), test_pdflip_prefix_trace_in (2).

…index (A5-N1), hand-over stamps its candidate against the reaper (A8-N1), freed only when freed (A8-N2), A5-M / A6-M counters (SOLL R7 R11; user 20.10.) Re-review of 8e8e427 ("FREIGABE MIT AUFLAGEN", deskq/done/nf-s109-rereview-a5a8-1020.md; counter-tests review/nf-s109-a5a8-1020 @ 089b0f2a53, Q1 and Q3 red there -- the file is taken over here unchanged). User order 20.10. (verbatim): "UND WEHE DAS SCHREIBEN IN DEN CACHE BLOCKIERT DURCH IRGENDEINE SYNCHRON ODER WARTESCHEISSE" -- nothing below waits, locks the evictor or does I/O on the scheduler thread. A5-N1 (before boot, R11 through a race): every publish of the write-behind pass (on disk, written, written by the sibling group) goes through HiCacheFile._l3wb_publish: publish, then ONE re-read of the stems in the sgl-project#1459 index (memory); a stem the index no longer lists is voided in the baseline AND the shared record. The LRU removes the index entry before on_evict, so an unlink between the pass's stat/commit and its publish can no longer leave the page "copied" forever. Line S109-A5-N1 L3WB-PUBLISH-RACE voided=. Source pin updated (3 _l3wb_publish, no bare publish, publish before the re-read before the void). Q1 green. A8-N1 (before boot, wrong KV): new C function arena_touch_evicting(slot, gen) -- stamps touched_ms only while the slot is EVICTING under gen, then re-reads state and generation; arena_reap_stale_aged re-reads touched_ms after its CAS and backs off (arena.c), so a candidate the hand-over stamped stays its candidate for the stale age. _hand_drain stamps BEFORE its check (ShmArena.touch_evicting via _hand_still_ours) and checks again AFTER the copy: a slot that moved while the copy ran (copy longer than the stale age) gets its written file voided (unlink, LRU forget, index remove, record void -- on the hand-over thread) and counts dropped_without_l3. No wait, no lock. Q3 green; real-reaper test: the reaper firing mid-copy frees nothing. A8-N2: free_evicted_named returns the number freed; HAND_STATS["freed"] counts only a free that happened, a refused one is free_refused (S109-A8-N2 line). A5-M: _l3sec_on_evict counts the L3 LRU taking the file of a page still COMPLETE in L2 (l3_secured.LRU_L2_STATS n/bytes; S109-A5-M L3-LRU-TOOK-L2-PAGE, throttled) -- the rewrite trigger, so ping-pong is visible on metal. A6-M: PROMOTE_STATS gains `missed` (stems no longer in the LRU at the drain); one throttled line every 60 s when a counter moved: S109 L3-STATS promote queued/refreshed/missed/overflow | handover queued/freed/free_refused/ copy_failed/reaped/voided/dropped_without_l3 | quiesce n/queued | lru_took_l2 n/bytes. Also: _hand_drain drains pending promotions between hand-over items (re-review, low). Merge: the S109b seam block in test/registered/unit/pdflip/conftest.py moved off the file end (before the published- form fixture), so S107's decider_off append merges cleanly; git merge-tree S109 + S108 (4218aba) + S107 (56f91d8): only tools/rig_dashboard/rigdash/profil_data/catalog.json conflicts (generator file). Tests: two own tests made deterministic (the hand-over writer waits until the test looked at the scheduler-side state; the reuse-boot passes run with slice_s=0 so a busy host does not split them at the slice deadline). Mutants (test/registered/unit/mem_cache/mutants_s109_1020.py, against the S109 file, the review file and the re-review file, serial under /spinning/gpu-arb/pytest.lock; every mutant string checked to match exactly once, a mutant that does not import is INVALID, not SURVIVED): 27 of 27 killed -- the 22 of 8e8e427 plus A5N1 publish without the index re-read, A8N1a hand-over does not stamp, A8N1b C touch does not stamp touched_ms, A8N1c no re-check after the copy, A8N2 freed counted on a refused free. TESTS (serial, flock /spinning/gpu-arb/pytest.lock, systemd-run --scope -p MemoryMax=2G, CUDA_VISIBLE_DEVICES=""): 96 suites (the 95 of 8e8e427 + test_rereview_s109_a5a8_1020): 90 green, 6 with only the pre-existing reds of rc27 56f3a0c (10 tests, identical): read_room_1015 (1), arenafull_1015 (2), arena_l3_fill_group_rc12z (1), flush_arena_full_stop_flushfast_1012 (2), l3_fill_orphan_claim_nf (2), prefix_trace_in (2). test_s109_backup_wall_aux_leaves_1020 46 passed, test_rereview_s109_a5a8_1020 4 passed (Q1, Q2a, Q2b, Q3), test_review_s108_s109_1020 5 passed 5 skipped, l3_reuse_after_restart_0928 9, s86_sleep_l3_release_1020 11, bubble_publish_1406 5, quiesce_publish_f5_1016 6.

merrymercy reviewed Sep 19, 2024

View reviewed changes

Comment thread python/sglang/srt/layers/radix_attention.py Resolved

Comment thread python/sglang/srt/layers/test_ds_kernel.py Resolved

Comment thread python/sglang/srt/mem_cache/memory_pool.py Resolved

Comment thread python/sglang/srt/model_executor/model_runner.py Resolved

Ying1123 mentioned this pull request Sep 22, 2024

Development Roadmap (2024 Q4) #1487

Closed

21 of 37 tasks

merrymercy mentioned this pull request Sep 22, 2024

[Feature] KV Cache Compression #1347

Closed

2 tasks done

andy-yang-1 force-pushed the double-sparsity branch from 9798dc2 to 57c998b Compare September 25, 2024 00:45

merrymercy mentioned this pull request Oct 6, 2024

[Feature] 4-bit quantized prefix cache #1374

Closed

merrymercy added the high priority label Oct 12, 2024

resolve all conflicts

5f71afa

andy-yang-1 force-pushed the double-sparsity branch from 6b07a3d to 5f71afa Compare October 13, 2024 22:41

andy-yang-1 added 2 commits October 14, 2024 05:26

add unittest & lint

bb38db3

fix lint

831250a

merrymercy requested changes Oct 14, 2024

View reviewed changes

Comment thread python/sglang/srt/model_executor/forward_batch_info.py Resolved

Comment thread python/sglang/test/Llama-3.1-8B-Instruct.jsonconfig Resolved

Comment thread test/srt/test_double_sparsity.py Resolved

merrymercy changed the title ~~[WIP] Support double sparsity~~ Support double sparsity Oct 14, 2024

fix lint & add unit test

daf0400

merrymercy reviewed Oct 14, 2024

View reviewed changes

Comment thread test/srt/test_double_sparsity.py Resolved

merrymercy reviewed Oct 14, 2024

View reviewed changes

Comment thread test/srt/test_double_sparsity.py Resolved

add automatic backend & comment for channel config

fb592cb

merrymercy requested changes Oct 14, 2024

View reviewed changes

Comment thread test/srt/test_double_sparsity.py Resolved

Comment thread test/srt/run_suite.py Resolved

fix style & unset cuda graph

3ea16f0

merrymercy approved these changes Oct 14, 2024

View reviewed changes

merrymercy enabled auto-merge (squash) October 14, 2024 08:32

merrymercy disabled auto-merge October 14, 2024 09:00

merrymercy merged commit 061e546 into sgl-project:main Oct 14, 2024

ChenlongDeng mentioned this pull request Dec 18, 2024

[Feature] Support for Evicting Specific KV Cache to Save GPU Memory #2510

Closed

2 tasks done

shadowpa0327 mentioned this pull request Jan 17, 2025

[Feature] Enhancement on Sparse Attention and KV-Cache Compression #2946

Closed

2 tasks done

zhaochenyang20 mentioned this pull request Mar 3, 2025

Development Roadmap (2025 H1) #4035

Closed

1 of 22 tasks

timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025

Support double sparsity (sgl-project#1459)

fee496f

Jiminator mentioned this pull request Apr 16, 2026

[Feature] Restore Double Sparsity attention backend on latest main #22992

Closed

4 of 5 tasks

Conversation

andy-yang-1 commented Sep 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Speedup Evaluation

Checklist

Uh oh!

merrymercy commented Sep 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ghost commented Sep 24, 2024

Uh oh!

andy-yang-1 commented Sep 24, 2024

Uh oh!

fengyang95 commented Oct 9, 2024

Uh oh!

merrymercy commented Oct 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

merrymercy commented Oct 14, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

merrymercy commented Oct 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

merrymercy commented Oct 14, 2024

Uh oh!

Uh oh!

Uh oh!

andy-yang-1 commented Oct 16, 2024

Uh oh!

fengyang95 commented Oct 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fengyang95 commented Oct 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andy-yang-1 commented Oct 19, 2024

Uh oh!

fengyang95 commented Oct 20, 2024

Uh oh!

shreyansh26 commented Nov 6, 2024

Uh oh!

andy-yang-1 commented Nov 6, 2024

Uh oh!

shreyansh26 commented Nov 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andy-yang-1 commented Nov 10, 2024

Uh oh!

shreyansh26 commented Nov 11, 2024

Uh oh!

yuguo-Jack commented Nov 23, 2024

Uh oh!

hcyz33 commented Jan 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Speedup Evaluation

Checklist

Uh oh!

alex1720-web commented Oct 10, 2025

Uh oh!

IgniteGo commented Jan 29, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

andy-yang-1 commented Sep 18, 2024 •

edited

Loading

merrymercy commented Sep 19, 2024 •

edited

Loading

merrymercy commented Oct 11, 2024 •

edited

Loading

merrymercy commented Oct 14, 2024 •

edited

Loading

fengyang95 commented Oct 18, 2024 •

edited

Loading

fengyang95 commented Oct 19, 2024 •

edited

Loading

shreyansh26 commented Nov 7, 2024 •

edited

Loading

hcyz33 commented Jan 13, 2025 •

edited

Loading