Skip to content

Prefetch wheel central directories for streamed file publication - #21338

Draft
charliermarsh wants to merge 22 commits into
mainfrom
charlie/prefetch-wheel-central-directory
Draft

charliermarsh wants to merge 22 commits into
mainfrom
charlie/prefetch-wheel-central-directory

Conversation

@charliermarsh

Copy link
Copy Markdown
Member

Summary

Alternative to #21334 and #21333, stacked on #21327.

We fetch the ZIP central directory with range requests before streaming extraction, so we know each
file's executable status as soon as its payload is unpacked. We set its permissions while the file
is still private, then publish it to files-v0 through a bounded queue while the remaining payloads
download and extract. File hashes still include executable status, and installation keeps its
existing link behavior; we don't need permission manifests or copies to restore executable bits.

We start with a 16 KiB suffix request and fetch more of the directory when needed. Strong ETags from
the full response are sent as If-Match on the range requests. If prefetching fails, we fall back
to learning modes from the streamed directory and publishing files then.

The full streamed ZIP is still validated before we expose the archive. If its executable modes
differ from the prefetched directory, we reject the download without changing any shared inode's
permissions. RECORD stays private. Rejected or interrupted downloads can leave orphan file
objects, which cache pruning removes.

This adds range requests to cold downloads. I haven't measured whether the overlap outweighs that
cost yet.

This branch had an error being deployed

1 failed deployment
automations — 58478fcd Deployed Aug 28, 2026 by charliermarsh via review / security review #45412
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants