Skip to content

[Perf] unified-memory: Reduce allocation and SWA retirement overhead - #44970

Merged
ch-wan merged 2 commits into
mainfrom
cheng/dev/unified-allocator-overhead
Oct 18, 2026
Merged

ch-wan merged 2 commits into
mainfrom
cheng/dev/unified-allocator-overhead

Conversation

@ch-wan

@ch-wan ch-wan commented Oct 18, 2026 •

Copy link
Copy Markdown
Collaborator

This PR is part of a stack (oldest at bottom):

Current head d541e806cd, based on main a5b21145e1. Stack predecessor: #44969. The Files changed tab isolates this PR from its predecessor.

Motivation

Unified allocation and sliding-window retirement repeatedly queried stable geometry and launched avoidable copies, mapping lookups, and tombstone writes in the scheduler path.

Modifications

The bind kernels reuse compiled code when consuming an odd number of int64 free-list entries changes pointer alignment. A GPU test verifies both mappings and JIT-cache reuse; the prefix-view comment states its lifetime contract.

  • Snapshot allocator geometry once for each capacity query.
  • Retain consumed free-list views instead of copying lists that are rebound rather than modified in place.
  • Fuse FLOAT-sub-pool unbinding and skip signal-only invariant tensor work when invariant detection is disabled.
  • Keep enabled invariant checks and add coverage for capacity, ownership, deferred frees, and absence of host synchronization.

CI registration updates

Updated est_time in 1 affected test files from observed file-level CI timings, including process/import cost and margin. Existing estimates with sufficient margin are retained. The expanded weekly configurator test also passed all 11 cases in a 30.1-second CPU measurement.

The CI registry check validated 2,960 registrations before landing. An AST comparison confirmed that this follow-up changes registration metadata only. Runtime test measurements elsewhere in this description refer to their stated earlier revisions.

Accuracy Tests

Earlier combined stack b550cacde6 adds a test-fixture correction in #43326: the Gemma4 read-stream fixture declares its FULL address space, and the HiSparse registry test uses real CacheInitParams for the retained-window calculation. Both complete test files pass locally with GPUs hidden: 4 Gemma4 tests and 26 registry tests. Pre-commit passed. The five descendant patches are unchanged (git range-diff); this follow-up changes no production code. The earlier 455-test run did not cover these two files; GitHub CI has not yet been revalidated for this head.

Validation before the test-fixture follow-up, on combined stack 4cf27a551c: 455 tests and 921 subtests passed on B300, plus the two widened-history regressions against Rust TreeCore. The new FULL-fallback metadata cases cover eager execution and CUDA graph replay. New allocation-reclaim, explicit-prefill and JIT-reuse regressions fail on the previous code and pass with the fixes. These are combined-stack checks, not independent model reruns of each PR.

157 tests and 732 subtests passed on B300 at the previous head 88e2d59d29, covering multi-ended allocation, capacity memoization, no-sync frees, and allocation eviction.

Speed Tests and Profiling

Performance validation is reported for the final combined stack below; this run does not attribute an isolated throughput gain to this PR.

Prior combined-stack E2E output/acceptance and throughput results: #44973. These measurements include all five changes before the review updates.

Checklist

  • Commit pre-commit hooks passed.
  • Regression tests passed on the combined stack; earlier individual-head results are labeled above.
  • Followed SGLang code style.
  • Upstream CI pending.

CI States

Latest PR Test (Base): 🚫 Run #39128235423
Latest PR Test (Extra): ❌ Run #39128235132
Latest PR Test (AMD ROCm 10): ❌ Run #39128235129

@ch-wan
ch-wan force-pushed the cheng/dev/unified-allocator-overhead branch from 88e2d59 to c59c07f Compare October 18, 2026 08:13
@ch-wan
ch-wan force-pushed the cheng/refactor/decode-metadata-mappings branch from 95903cb to d9b03fe Compare October 18, 2026 08:13
@ch-wan
ch-wan force-pushed the cheng/refactor/decode-metadata-mappings branch from d9b03fe to 9273bf9 Compare October 18, 2026 09:02
@ch-wan
ch-wan force-pushed the cheng/dev/unified-allocator-overhead branch from c59c07f to 68553ac Compare October 18, 2026 09:02
@ch-wan
ch-wan force-pushed the cheng/refactor/decode-metadata-mappings branch from 9273bf9 to d68283d Compare October 18, 2026 12:39
Base automatically changed from cheng/refactor/decode-metadata-mappings to main October 18, 2026 12:39
@ch-wan
ch-wan force-pushed the cheng/dev/unified-allocator-overhead branch from 68553ac to d541e80 Compare October 18, 2026 12:39
@ch-wan
ch-wan merged commit d318a54 into main Oct 18, 2026
84 of 98 checks passed
@ch-wan
ch-wan deleted the cheng/dev/unified-allocator-overhead branch October 18, 2026 12:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant