[HiCache] Add synchronization for context parallelism

[vladnosiv]

HiCache previously synchronized state only within tp_group, which is no longer sufficient after the CP split.
This could cause different CP ranks to make different decisions about prefetch completion/revoke, write-through ack handling, and host-cache updates.

This change passes attn_cp / attn_tp groups into HiCache and switches the relevant sync points to CP-aware reductions/barriers, including storage-prefetch synchronization.

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[libratiger]

@vladnosiv does this feature improve the performance?

[vladnosiv]

@vladnosiv does this feature improve the performance?

Hi ! No, according to my observations, without this change, CP + HiCache (L2) may fail after some time (up to 30-40 minutes according to my observations)
So it's for reliability.

[libratiger]

@vladnosiv does this feature improve the performance?

Hi ! No, according to my observations, without this change, CP + HiCache (L2) may fail after some time (up to 30-40 minutes according to my observations) So it's for reliability.

Thanks, is there any script or command to reproduce this situation.

[vladnosiv]

Thanks, is there any script or command to reproduce this situation.

I think that such a launch + some cache-heavy traffic like the mooncake dataset in bench_serving should reproduce the crash:

python3 -m sglang.launch_server \
      --model-path deepseek-ai/DeepSeek-V3.2 \
      --trust-remote-code \
      --tp-size 8 \
      --attn-cp-size 8 \
      --enable-nsa-prefill-context-parallel \
      --chat-template examples/chat_template/tool_chat_template_deepseekv32.jinja \
      --mem-fraction-static 0.8 \
      --enable-hierarchical-cache \
      --hicache-ratio 2.0 &> sglang.out

In my case, the repro setup was with P/D + CP on Prefills + HiCache and real prod traffic. After 30-40 minutes, prefills stopped responding to generation requests in any way without an obvious fail.
I can confirm that with this commit, I don't see any problems for a weeks.

I also saw that @whybeyoung worked on CP + PP + P/D + HiCache, maybe he has more information.

[vladnosiv]

@hzh0425 done

[vladnosiv]

fixed the flag --moe-dp-size 2 that was lost during the moving of the test

[hzh0425]

/rerun-test test_qwen35_hicache.py test_hicache_storage_mooncake_backend.py test_hicache_storage_file_backend.py

[github-actions]

✅ 4-gpu-h100 (1 test): View workflow run

cd test/ && python3 registered/4-gpu-models/test_qwen35_hicache.py

✅ 2-gpu-h100 (2 tests): View workflow run

cd test/ && python3 registered/hicache/test_hicache_storage_mooncake_backend.py
cd test/ && python3 registered/hicache/test_hicache_storage_file_backend.py

[vladnosiv]

/rerun-failed-ci

[vladnosiv]

stage-b test crashed a second time due to hf error 429

[ShangmingCai]

/rerun-failed-ci

[ShangmingCai]

/rerun-test test_hicache_storage_mooncake_backend.py test_hicache_storage_file_backend.py

[hzh0425]

HiCache's CI has been temporarily removed due to some incompatibilities with the CUDA 13 environment.
We might need to wait until the CI is restored before proceeding further.

[ShangmingCai]

Please address conflict

[vladnosiv]

[vladnosiv]

CI is done