Skip to content

perf: add opt-in DRAM descriptor coalescing and UCX batching - #113

Closed
ziqifan617 wants to merge 1 commit into
mainfrom
ziqifan617/hicache-transfer-batching
Closed

ziqifan617 wants to merge 1 commit into
mainfrom
ziqifan617/hicache-transfer-batching

Conversation

@ziqifan617

@ziqifan617 ziqifan617 commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Add optional transfer controls needed by the SGLang HiCache performance integration, separately from basic enablement and operator documentation. Existing branch/head is preserved at bb48f3843a292f514becd3efedf29429e272c0af.

  • Opt-in remote framework DRAM descriptor coalescing preserves paired source/destination byte ranges.
  • Add validated optional nixl_ucx_split_batch_size; reject invalid integer/range/backend combinations before native startup.
  • Forward the option through configuration/runtime initialization.
  • Record native descriptor telemetry before releasing transfer handles.
  • Keep defaults, request isolation, error handling and terminal native-lifetime rules unchanged. No dependency/model changes.

Companion: sgl-project/sglang#45649. The initial HiCache integration sgl-project/sglang#45647 does not require enabling these tuning options.

Scope and tests

Four runtime files and two unit-test files: 285 additions / three deletions overall. Added cases cover split-option validation and backend propagation, coalescing address-pair integrity, overlap/gap behavior and descriptor telemetry.

Focused test targets for a compatible Linux/NIXL environment:

python -m pytest tests/unit/test_progress.py tests/unit/test_kvcr_remote_source.py

This publication pass checks exact branch identity, diff whitespace and clean composition with current main 0521fb1981ab34fc5c80a8115849c8987e99f76c. It does not claim a fresh Linux/native test execution or an isolated end-to-end speedup for this SDK-only change. The SGLang companion reports a combined-stack matched-workload regression screen; that is not a causal benchmark of this individual PR.

Draft / merge gates

  • Rerun focused unit and native Linux cases for the final rebased source tuple.
  • Preserve per-pool byte checks, failure/uncertain-DMA lifetime and unsupported-option rejection.
  • Attach an isolated matched-workload comparison before making a SDK-only performance claim.
  • Complete review and upstream CI.

The separate SGLang quick-start preview does not depend on these optional tuning settings.

Operator documentation preview (independent): #114

Preserve source/destination byte pairs when grouping remote framework DRAM descriptors. Add a validated optional UCX split-batch setting and native descriptor telemetry, with unit coverage.

Defaults remain unchanged. No dependency or model changes.
@copy-pr-bot

copy-pr-bot Bot commented Oct 21, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant