Skip to content

fix: Reserve fixed-size-list child capacity - #28980

Merged
ritchie46 merged 3 commits into
pola-rs:mainfrom
0guban0v:fix/fixed-size-list-child-reservation
Aug 27, 2026
Merged

ritchie46 merged 3 commits into
pola-rs:mainfrom
0guban0v:fix/fixed-size-list-child-reservation

Conversation

@0guban0v

@0guban0v 0guban0v commented Aug 25, 2026 •

Copy link
Copy Markdown
Contributor

Closes #28872

More measurements are underway, but PR is ready for review. I realized it has a bigger blast radius than just arr.dot.
Added comment with broader impact test.

Benchmark used arr.dot streaming broadcast, width 768, 16,776,960 products, and validated native/composed equality.

Revisions: main a6782fa065, PR 56ddea3e98

100 × (composed heap allocation − native heap allocation) / (rows × width × sizeof(inner_dtype))

+100% → native avoids heap allocation equal to one product buffer
   0% → equal heap allocation
 −33% → native allocates 33% of one product buffer more

Early numbers:

Inner dtype   1t before   1t after   8t before   8t after  16t before   16t after
Float32       -33.3280%   +0.0049%   -33.2688%   +0.0647%   -33.2005%   +0.1342%
Float64       -33.3282%   +0.0050%   -33.2827%   +0.0506%   -33.2300%   +0.1023%
Int8          -33.3262%   +0.0091%   -33.1841%   +0.1491%   -33.0234%   +0.3106%
UInt8         -33.3276%   +0.0078%   -33.1845%   +0.1493%   -33.0188%   +0.3119%
Int16         -33.3266%   +0.0062%   -33.2402%   +0.0930%   -33.1385%   +0.1920%
UInt16        -33.3280%   +0.0062%   -33.2401%   +0.0930%   -33.1395%   +0.1914%
Int32         -33.3281%   +0.0052%   -33.2686%   +0.0646%   -33.2013%   +0.1324%
UInt32        -33.3284%   +0.0057%   -33.2685%   +0.0647%   -33.2022%   +0.1322%
Int64         -33.3284%   +0.0050%   -33.2828%   +0.0507%   -33.2302%   +0.1030%
UInt64        -33.3280%   +0.0051%   -33.2828%   +0.0506%   -33.2305%   +0.1031%
Int128        -33.3286%   +0.0047%   -33.2897%   +0.0437%   -33.2453%   +0.0883%
UInt128       -33.3286%   +0.0048%   -33.2897%   +0.0436%   -33.2452%   +0.0881%

Post-fix, native arr.dot allocates less heap than composed, and that saving grows with thread count.

@github-actions github-actions Bot added fix Bug fix python Related to Python Polars rust Related to Rust Polars title needs formatting labels Aug 25, 2026
@0guban0v 0guban0v changed the title fix: reserve fixed-size-list child capacity fix: Reserve fixed-size-list child capacity Aug 25, 2026
@codecov

codecov Bot commented Aug 26, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 50.00000% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 81.33%. Comparing base (a6782fa) to head (c25a47c).
⚠️ Report is 99 commits behind head on main.

Files with missing lines Patch % Lines
.../polars-arrow/src/array/fixed_size_list/mutable.rs 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #28980      +/-   ##
==========================================
- Coverage   81.58%   81.33%   -0.26%     
==========================================
  Files        1890     1886       -4     
  Lines      268762   267093    -1669     
  Branches     3264     3061     -203     
==========================================
- Hits       219280   217246    -2034     
- Misses      48637    49069     +432     
+ Partials      845      778      -67     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Comment thread crates/polars/tests/it/arrow/array/fixed_size_list/reserve.rs Outdated
@0guban0v
0guban0v force-pushed the fix/fixed-size-list-child-reservation branch from ffab9b6 to c4efe4f Compare August 26, 2026 15:16
@0guban0v

0guban0v commented Aug 27, 2026 •

Copy link
Copy Markdown
Contributor Author

To test broader impact, I isolated generic construction path DataFrameBuilder::reserve → SeriesBuilder::reserve → FixedSizeListArrayBuilder::reserve

I directly compare reserve(additional) with reserve(additional * size). Each case processes 4,194,048 Int64 child values at widths 1, 8, 64, 768. Both modes reserve all outer rows first: single extend copies input once, while repeated subslice_extend copies approx 65,536 child values per call.

Revisions: main 73dd5bea65, PR c25a47ca73.

Arrows in table mean main → PR.
100% means final child capacity equals required payload.

Mode Width Growth
reallocations
Final child capacity
/ required payload
single extend 1 0 → 0 100.000% → 100.000%
single extend 8 1 → 0 100.000% → 100.000%
single extend 64 1 → 0 100.000% → 100.000%
single extend 768 1 → 0 100.000% → 100.000%
repeated subslice_extend 1 0 → 0 100.000% → 100.000%
repeated subslice_extend 8 3 → 0 100.000% → 100.000%
repeated subslice_extend 64 6 → 0 100.000% → 100.000%
repeated subslice_extend 768 8 → 0 199.231% → 100.000%

Corrected reservation eliminates every observed child-buffer growth reallocation for widths above 1. Most cases reach same capacity without intermediate growth, but width 768 also reduces final capacity from 199.231% to 100% of required payload.

@ritchie46
ritchie46 merged commit 442c479 into pola-rs:main Aug 27, 2026
31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fix Bug fix python Related to Python Polars rust Related to Rust Polars

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fixed-size-list builders reserve outer rows instead of child values

2 participants