Skip to content

test(hicache): cover buffer pipeline idle flush gate - #37177

Open
vuuihc wants to merge 1 commit into
sgl-project:mainfrom
vuuihc:codex/hicache-buffer-idle
Open

vuuihc wants to merge 1 commit into
sgl-project:mainfrom
vuuihc:codex/hicache-buffer-idle

Conversation

@vuuihc

@vuuihc vuuihc commented Aug 31, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

Buffer-only HiCache must not report the scheduler as fully idle while D2H or H2D pipeline work is still in flight. Otherwise, flush_cache() can reset cache state before transfer acknowledgements and staging ownership have drained.

The production predicate was completed on main by #37567 and #37883 while this PR was in draft. This PR now focuses on preserving that behavior with registered regression coverage and documenting the full idle contract.

Modifications

  • Add registered scheduler tests covering all seven buffer-pipeline activity states:
    • pending backup batches
    • ongoing backup
    • ongoing write-through
    • pending load-back batches
    • ongoing load-back
    • ongoing buffer load-back
    • completed-ack queue
  • Verify that each active state rejects flush without resetting the radix cache or KV allocator.
  • Verify that flush succeeds once every pipeline state has drained.
  • Preserve the scheduler health-check semantics while the pipeline is busy.
  • Clarify the buffer-only idle contract in the scheduler comment.

No transfer, allocator, or cache implementation is changed.

Tests

Unit tests

PYTHONPATH=python .venv/bin/python -m pytest \
  test/registered/unit/managers/test_scheduler_buffer_mode_idle.py \
  test/registered/unit/managers/test_scheduler_flush_cache.py -q

Result:

10 passed, 7 subtests passed

The old predicate reproduces the regression for both missing states:

D2H: current_idle=False old_idle=True old_flush=True
H2D: current_idle=False old_idle=True old_flush=True

Formatting, Ruff, isort, codespell, registered-test validation, and git diff --check pass.

Ascend A3 lifecycle validation

Configuration: Qwen3-0.6B, Ascend backend, buffer-only HiCache, file storage.

Observed real transfer activity:

launched d2h node=10 tokens=4096
buffer busy ... d2h=1 storage=0
launched h2d ack_id=-3 tokens=4032
buffer busy ... h2d=1 ... d2h=0
prefetch fill committed ... filled=4032

Results:

first_cached_tokens:       0
flush_after_d2h:           HTTP 200, 3.258455s
replay_cached_tokens:      4032
flush_after_h2d:           HTTP 200, 13.0699s
post_flush_cached_tokens:  4032

Both flush calls waited for the corresponding pipeline state to drain before succeeding. Temporary tracing was removed after validation.

Checklist

  • Rebased onto current main
  • Registered focused regression tests
  • Supported local test execution
  • Real A3 D2H/H2D lifecycle validation
  • No public API or documentation change

@github-actions github-actions Bot added the hicache Hierarchical Caching for SGLang label Aug 31, 2026
@vuuihc
vuuihc force-pushed the codex/hicache-buffer-idle branch from 4116278 to 1d3ed5e Compare September 17, 2026 03:08
@vuuihc vuuihc changed the title [WIP] fix(hicache): include in-flight buffer transfers in idle detection test(hicache): cover buffer pipeline idle flush gate Sep 17, 2026
@vuuihc
vuuihc marked this pull request as ready for review September 17, 2026 05:05

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

hicache Hierarchical Caching for SGLang

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant