Repository navigation
Reuse cached small files during wheel extraction - #21375
Closed
charliermarsh wants to merge 2 commits into
Closed
charliermarsh wants to merge 2 commits into
charliermarsh wants to merge 2 commits into
Conversation
charliermarsh
had a problem deploying
to
automations
August 31, 2026 13:22 — with
GitHub Actions
Error
charliermarsh
force-pushed
the
charlie/avoid-small-file-cache-writes
branch
from
August 31, 2026 13:33
0f66630 to
e7eafd3
Compare
charliermarsh
temporarily deployed
to
automations
August 31, 2026 13:34 — with
GitHub Actions
Inactive
uv test inventory changesThis PR changes the tests when compared with the
|
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When a small wheel file already exists in the content-addressed cache, we currently write out another copy during extraction, delete it, and replace it with a hardlink. This prototype buffers files up to 64 KiB and links existing objects directly into the extracted archive, avoiding the redundant write.
Streaming extraction retains at most 1 MiB of file contents and 1,024 files until the central directory supplies executable permissions. Larger files and exhausted budgets keep streaming; cache misses and failed hardlinks write a private copy.
RECORDstays private, and ZIP checksums and sizes are still validated.We compared against #21327 on Linux/ext4 using optimized builds, preselected wheels, and localhost HTTP. Each cell is the median of eight runs per binary in alternating ABBA/BAAB blocks, with baseline → prototype wall time. Installed paths, contents, and permissions matched across all 192 runs.
The existing-object case retains only
files-v0and forces wheel extraction again, isolating the reuse opportunity. Ordinary warm installs can reuse the unpacked archive directly. These are exploratory timings with substantial variation in several cases, especially the synthetic HTTP workload.The clearest gain was local Jupyter extraction with existing objects: 18% lower median time, improvement in all four paired blocks, and 42% fewer reported filesystem output blocks. HTTP Jupyter was slower both with an empty cache and with existing objects, so this remains a draft while we evaluate the streaming tradeoff.