Skip to content

fix: enforce tensor packing limits and remove redundant clones - #129

Open
yulangz wants to merge 1 commit into
inclusionAI:mainfrom
yulangz:fix/tensor-packing-memory
Open

yulangz wants to merge 1 commit into
inclusionAI:mainfrom
yulangz:fix/tensor-packing-memory

Conversation

@yulangz

@yulangz yulangz commented Oct 13, 2026

Copy link
Copy Markdown

Tensor packing currently checks the byte limit after inserting a tensor and uses torch.cat(...).clone(). This can produce oversized groups from individually small tensors and unnecessarily allocate/copy another whole packed group.

Split a nonempty group before an insertion would exceed max_tensor_size, keeping individually oversized tensors isolated. Remove the post-cat clone: concatenation already owns storage independently of the inputs, including single-input groups. Reconstruction metadata and dtype grouping remain unchanged.

Fixes #128.

Validation:

  • Tensor utilities and CUDA IPC tests: 31 passed, including exact bounds, oversized-tensor isolation, noncontiguous/single-input storage ownership after input release, and output-only CUDA allocation peak.
  • bash ci/test_cpu.sh: 279 passed, 1 skipped in the local environment. Added the packing and CUDA IPC test files to this suite; the device-memory test skips without CUDA.
  • Ruff 0.15.20 formatting and lint checks across the repository passed.

The issue's 256 MiB CUDA reproduction measured a 512 MiB packing peak above live inputs with the old cat().clone(), versus 256 MiB with cat() alone. This change targets allocation/copy overhead and the group-size contract; it does not change the CUDA IPC acknowledgement/lifetime protocol.

@yulangz
yulangz requested a review from chaokunyang as a code owner October 13, 2026 09:10
Copilot AI balanced review requested due to automatic review settings October 13, 2026 09:10

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Tensor packing exceeds max_tensor_size and duplicates group-sized CUDA allocations

2 participants