feat: Add PyPI support by e8035669 · Pull Request #116 · Quantco/pixi-pack

Jeff Zhang (e8035669) · 2025-03-12T15:11:58Z

Motivation

closes #38

pixi-pack is a great work to deploy packages to anywhere, but I hope to support pypi packages as well. So I am trying to make support for pypi packages.

But I am not familar with how pixi or uv works. So it roughly imitates the way pixi works.

Changes

Use pixi-pack pack --experimental-pypi-support to package a project which has pypi dependencies.

It will do these things internally:

Collect all pypi record from pixi.lock, It accept wheel files currently. source distribution would reject
Create simple repository
Download all wheel files into simple directory
Create some index.html make it becomes PyPI-compatible package repository
Writing pip section when creating environment.yml, make it compatible to conda/micromamba

Use pixi-pack unpack to unpack. When simple directory is exist, it will continue install wheel files after conda prefix is installed.

It will do these things internally:

Find installed python from this prefix
Load python interpreter and python environment
Collect all .whl files
Unpack these wheel files
Install them into this python environment

If updating documentation:

Updated documentation in https://github.com/prefix-dev/pixi/blob/main/docs/deployment/pixi_pack.md as well docs: Mention pypi support in pixi-pack prefix-dev/pixi#3508

….2 to 2.1.2

Jeff Zhang (e8035669) · 2025-03-12T15:48:34Z

All right.... uv needs rustc 1.83, but current rustc in pixi is 1.81. But I am not sure about downgrading uv because pixi was using this version last week.

Can I upgrade rustc version?

Pavel Zwerschke (pavelzw) · 2025-03-12T18:12:15Z

Can I upgrade rustc version?

yes, sure!

Pavel Zwerschke (pavelzw)

haven't reviewed the logic yet as i'm also not really familiar with pypi/pip/uv's inner workings, just two comments

src/main.rs

tests/integration_test.rs

Jeff Zhang (e8035669) · 2025-03-15T14:33:40Z

examples/pypi-bdist-packages/pixi.toml

+# torch has local version, which is percent encoded on url, it should be decode.
+# torch-2.6.0%2Bcpu + torch-2.6.0+cpu
+torch = { version = ">=2.2", index = "https://download.pytorch.org/whl/cpu" }


I only know that this package has a local version, but it is big (170MB). If there is a smaller package. We can change it.

for the general unit tests, i would prefer a very small pypi package like ordered-enum which is a wheel with zero dependencies that is only 3KB large.

how about you verify the behavior with URL encoding in a unit test in pack.rs test_url_encoding where you just call download_pypi_package with the specific pytorch url?
this way pytorch will only get downloaded a single time in that test and all other integration tests just use a very small pixi environment which should make the tests quicker

Jeff Zhang (e8035669) · 2025-03-15T14:39:00Z

.github/workflows/ci.yml

          key: tests
      - name: Run test
-        run: pixi run test --color always${{ startsWith(matrix.os, 'ubuntu') && ' --no-default-features --features rustls-tls' || '' }}
+        run: pixi run test --color always${{ startsWith(matrix.os, 'ubuntu') && ' --no-default-features --features rustls-tls' || '' }} -- --test-threads=4


I edited CI script to reduce parallel test, or it would make too many requests to download packages at the same time.

Tim de Jager (tdejager) · 2025-03-17T10:34:45Z

Pavel Zwerschke (@pavelzw) asked me for a quick eyes on this. I think it is already looking pretty good , two questions.

Should the index.html generation maybe be optional. And AFAICS this is creating a flat index right?
I think it might be nicer to have a nicer error when a wheel is not found, like maybe saying only wheel files are supported or something? There can be quite some lock files with source distributions in there.

…io-tar by astral-tokio-tar to prevent long filename truncate bug

Jeff Zhang (e8035669) · 2025-03-18T14:20:24Z

Should the index.html generation maybe be optional. And AFAICS this is creating a flat index right?

flat index is better, I didn't think of it before, I update to use flat index.

I think it might be nicer to have a nicer error when a wheel is not found, like maybe saying only wheel files are supported or something? There can be quite some lock files with source distributions in there.

Currently it only show the first source distribution it found, and exit program immediately, should I list all the source distribution in lock file?

…heel file

Jeff Zhang (e8035669) · 2025-03-22T01:13:49Z

Did I break something? Why do Windows and MacOS builds sometimes fail....

Pavel Zwerschke (pavelzw) · 2025-03-25T13:26:13Z

stderr: "ERROR conda.core.link:_execute(938): An error occurred while installing package 'file:///private/var/folders/1z/67kzkrxn55x733bk922y4y440000gn/T/.tmpbhkkO6/channel::tk-8.6.13-h1abcd95_1'.\n\nCondaError: Cannot link a source that does not exist. /Users/runner/work/pixi-pack/pixi-pack/.pixi/envs/default/pkgs/tk-8.6.13-h1abcd95_1/include/X11/DECkeysym.h\nRunning `conda clean --packages` may resolve your problem.\n()\n\n" }

maybe this is because test_compatibility_with_pypi and test_compatibility run simultaneously and conda doesn't like that? we could try limiting the parallelism in a way that these two tests never run at the same time 🤔

Jeff Zhang (e8035669) · 2025-03-27T11:16:06Z

stderr: "ERROR conda.core.link:_execute(938): An error occurred while installing package 'file:///private/var/folders/1z/67kzkrxn55x733bk922y4y440000gn/T/.tmpbhkkO6/channel::tk-8.6.13-h1abcd95_1'.\n\nCondaError: Cannot link a source that does not exist. /Users/runner/work/pixi-pack/pixi-pack/.pixi/envs/default/pkgs/tk-8.6.13-h1abcd95_1/include/X11/DECkeysym.h\nRunning `conda clean --packages` may resolve your problem.\n()\n\n" }
maybe this is because test_compatibility_with_pypi and test_compatibility run simultaneously and conda doesn't like that? we could try limiting the parallelism in a way that these two tests never run at the same time 🤔

Wow, it works!

Daniel Elsner (delsner) · 2025-03-28T14:43:50Z

Thanks for your contribution Jeff Zhang (@e8035669), we'll take a look at this early next week!

Pavel Zwerschke (pavelzw)

looks very good already, thanks Jeff Zhang (@e8035669)! i have a couple of comments and after those are addressed, we can merge :)

Currently it only show the first source distribution it found, and exit program immediately, should I list all the source distribution in lock file?

i think it is fine as you implemented it now

.github/workflows/ci.yml

Cargo.toml

src/main.rs

src/pack.rs

tests/integration_test.rs

src/pack.rs

Pavel Zwerschke (pavelzw) · 2025-04-01T12:27:37Z

examples/pypi-bdist-packages/pixi.toml

+# torch has local version, which is percent encoded on url, it should be decode.
+# torch-2.6.0%2Bcpu + torch-2.6.0+cpu
+torch = { version = ">=2.2", index = "https://download.pytorch.org/whl/cpu" }


for the general unit tests, i would prefer a very small pypi package like ordered-enum which is a wheel with zero dependencies that is only 3KB large.

how about you verify the behavior with URL encoding in a unit test in pack.rs test_url_encoding where you just call download_pypi_package with the specific pytorch url?
this way pytorch will only get downloaded a single time in that test and all other integration tests just use a very small pixi environment which should make the tests quicker

examples/pypi-bdist-packages/pixi.toml

src/unpack.rs

tests/integration_test.rs

…d serial_test and mark conda tests are serial

Pavel Zwerschke (pavelzw)

thanks! some minor last comments

tests/integration_test.rs

Jeff Zhang (e8035669) · 2025-04-04T10:36:28Z

Thanks for the code review, I learned a lot

Jeff Zhang (e8035669) added 2 commits March 12, 2025 22:00

add experimental pypi support

abd0fa1

Merge branch 'main' into pypi_support

ba65eba

Jeff Zhang (e8035669) requested review from Daniel Elsner (delsner) and Pavel Zwerschke (pavelzw) as code owners March 12, 2025 15:11

[fix] downgrade indexmap from 2.8.0 to 2.7.1, priority-queue from 2.2…

4ff8d8e

….2 to 2.1.2

Jeff Zhang (e8035669) changed the title ~~Pypi support~~ feat: Add PyPI support Mar 12, 2025

Jeff Zhang (e8035669) added 3 commits March 13, 2025 19:51

update to rust 1.83.0

ef692fa

output message minor fix

f2b8c83

apply suggestions from cargo-fmt and cargo-clippy

9446bf7

Pavel Zwerschke (pavelzw) reviewed Mar 13, 2025

View reviewed changes

src/main.rs Outdated Show resolved Hide resolved

tests/integration_test.rs Show resolved Hide resolved

Jeff Zhang (e8035669) added 5 commits March 13, 2025 22:15

make --experimental-pypi-support conflict with --ignore-pypi-errors

01fd1e0

[WIP] add test to check pypi-support is work, add sdists test

a31df5c

[WIP] fix tests in windows

cc98d5a

[feat] add reproducible test

d073dbd

try reduce test parallel

545e1ea

Jeff Zhang (e8035669) force-pushed the pypi_support branch from 104b4b6 to 545e1ea Compare March 15, 2025 13:57

Jeff Zhang (e8035669) commented Mar 15, 2025

View reviewed changes

Jeff Zhang (e8035669) added 3 commits March 18, 2025 20:14

Merge branch 'main' into pypi_support

fcadc4c

[feat] use flat index instead of simple index, replace dependency tok…

c18f239

…io-tar by astral-tokio-tar to prevent long filename truncate bug

[feat] revert to use tokio-tar

b755a51

Jeff Zhang (e8035669) added 2 commits March 18, 2025 22:42

[fix] improve error message when project contains a package without w…

29120e4

…heel file

Merge branch 'main' into pypi_support

7974bc2

[fix] disable test parallel when ci testing

c868e2e

Pavel Zwerschke (pavelzw) requested changes Apr 1, 2025

View reviewed changes

Jeff Zhang (e8035669) added 2 commits April 2, 2025 22:38

Merge branch 'main' into pypi_support

b3897b6

[WIP] apply some minor suggestions

2d27fe8

Pavel Zwerschke (pavelzw) reviewed Apr 2, 2025

View reviewed changes

examples/pypi-bdist-packages/pixi.toml Outdated Show resolved Hide resolved

src/unpack.rs Outdated Show resolved Hide resolved

tests/integration_test.rs Outdated Show resolved Hide resolved

feat: remove experimental flag, merge pypi tests with exist tests, ad…

4733c20

…d serial_test and mark conda tests are serial

Pavel Zwerschke (pavelzw) approved these changes Apr 4, 2025

View reviewed changes

tests/integration_test.rs Outdated Show resolved Hide resolved

tests/integration_test.rs Outdated Show resolved Hide resolved

tests/integration_test.rs Show resolved Hide resolved

tests/integration_test.rs Outdated Show resolved Hide resolved

review

005fa99

bump package version, add to readme

903f33b

Pavel Zwerschke (pavelzw) mentioned this pull request Apr 4, 2025

docs: Mention pypi support in pixi-pack prefix-dev/pixi#3508

Merged

Pavel Zwerschke (pavelzw) added the enhancement New feature or request label Apr 4, 2025

Pavel Zwerschke (pavelzw) merged commit 21f8630 into Quantco:main Apr 4, 2025
13 of 17 checks passed

BrewTestBot mentioned this pull request Apr 4, 2025

pixi-pack 0.4.0 Homebrew/homebrew-core#218247

Merged

Jeff Zhang (e8035669) deleted the pypi_support branch May 21, 2025 13:04

Conversation

Jeff Zhang (e8035669) commented Mar 12, 2025 • edited by pavelzw Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Uh oh!

Jeff Zhang (e8035669) commented Mar 12, 2025

Uh oh!

Pavel Zwerschke (pavelzw) commented Mar 12, 2025

Uh oh!

Pavel Zwerschke (pavelzw) left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Jeff Zhang (e8035669) Mar 15, 2025

Choose a reason for hiding this comment

Uh oh!

Pavel Zwerschke (pavelzw) Apr 1, 2025

Choose a reason for hiding this comment

Uh oh!

Jeff Zhang (e8035669) Mar 15, 2025

Choose a reason for hiding this comment

Uh oh!

Tim de Jager (tdejager) commented Mar 17, 2025

Uh oh!

Jeff Zhang (e8035669) commented Mar 18, 2025

Uh oh!

Jeff Zhang (e8035669) commented Mar 22, 2025

Uh oh!

Pavel Zwerschke (pavelzw) commented Mar 25, 2025

Uh oh!

Jeff Zhang (e8035669) commented Mar 27, 2025

Uh oh!

Daniel Elsner (delsner) commented Mar 28, 2025

Uh oh!

Pavel Zwerschke (pavelzw) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Pavel Zwerschke (pavelzw) Apr 1, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Pavel Zwerschke (pavelzw) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Jeff Zhang (e8035669) commented Apr 4, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Jeff Zhang (e8035669) commented Mar 12, 2025 •

edited by pavelzw

Loading

Pavel Zwerschke (pavelzw) left a comment •

edited

Loading