[HiCache] Merge HiCache event checks to reduce decode overhead

[huangtingwei9988]

Motivation

Although HiCache is primarily used during the prefill phase, in single-node deployments the scheduler still has to complete HiCache event checks on every decode step. When storage is enabled, each decode step performs four all-reduce operations to finish the HiCache event polling path:

• writing_check() inside check_hicache_events()
• loading_check() inside check_hicache_events()
• the storage control queue drain inside check_hicache_events()
• an extra repeated writing_check() from flush_write_through_acks()

Each all-reduce costs about 150+ us, so four of them add roughly 600+ us of CPU-side synchronization overhead per decode step. On small models or small-batch decode workloads, this overhead can dominate TPOT and cause up to ~30% TPOT regression, with the GPU becoming idle while the scheduler waits on these CPU-side all-reduce calls.

Modifications

Accuracy Tests

Speed Tests and Profiling

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review and Merge Process

1. Ping Merge Oncalls to start the process. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
   • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

---

CI States

Latest PR Test (Base): ✅ Run #30440233721
Latest PR Test (Extra): ❌ Run #30440236718

[gemini-code-assist]

Warning

Gemini encountered an error creating the review. You can try again by commenting /gemini review.

[huangtingwei9988]

/tag-and-rerun-ci

[ishandhanani]

@huangtingwei9988 - just for my curiosity can you share the steps to get that same profile? I think it would be helpful as a snippet

[huangtingwei9988]

Qwen3-8B Benchmark Summary
Configuration:
Model: /data/nas/yongke.zyk/model_hub/Qwen3-8B
TP: 4
HiCache: enabled
HiCache backend/layout/policy: direct / page_first_direct / write_through
Workload: 16k input, 2k output

BS	Base TPOT (ms)	Opt TPOT (ms)	TPOT Improvement	Base Output tok/s	Opt Output tok/s	Throughput Improvement
1	13.13	12.91	+1.68%	71.08	72.64	+2.19%
2	13.48	13.39	+0.67%	133.73	134.75	+0.76%
4	14.39	14.33	+0.42%	236.68	240.45	+1.59%
8	17.47	16.70	+4.41%	367.97	388.45	+5.57%
16	21.11	20.22	+4.22%	561.36	591.17	+5.31%

[huangtingwei9988]

/rerun-failed-ci