More sized-efficient pixi.lock (with care to be optimized for git packfile).

[jleibs]

Problem description

The pixi.lock file in our repository is now over 1 megabyte (https://github.com/rerun-io/rerun/blob/main/pixi.lock).

It still compresses reasonably in git object storage within the packfile (taking up about 3MB of storage across history), but it is fast-becoming a meaningful contributor to repository growth.

This is a tricky one to do something about, as we ultimately care more about contribution to the delta-compressed packfile than we care about the actual size of the file-on-disk. Compression strategies that make the file smaller in a single checkout but harder to compress would still be a net-negative.

[baszalmstra]

Do you have an idea on how we could change the format to achieve this?

I think one of the biggest issues is the presence of sha hashes because those compress terribly. We tried to minimize the places where these occur for that reason.

[jleibs]

Not off the top of my head -- I definitely acknowledge it's a hard problem, so I was at least somewhat relieved to find that it still compressed reasonably well in the packfile.

Looking at the file itself it seems like there is still maybe a lot of meta information that is redundant with package management meta-data from conda/pypi as well.

From an information theory perspective does the lockfile need to have more than a table of: (feature, platform, package-name, version-number)

All the information about the kind of package, where to find it, it's own transitive dependencies, etc. seem like they could be re-computed from the pixi.toml file again.

Maybe there need to be two files here? One is a strict minimal .lock that can be included in the repository, while the other is a materialized file that can be cached somewhere.

[tdejager]

There has been some talk in the python packaging space along these lines, where the lock-file would still need to be rendered into a suitable format. Could be food for thought :)

[glemaitre]

We are facing a bit the same issue when we intend to use pixi to manage our CI/CD.

From an information theory perspective does the lockfile need to have more than a table of: (feature, platform, package-name, version-number)

In scikit-learn, when doing the flow for which we would like to use pixi, we are currently storing file from conda-lock files (eg https://github.com/scikit-learn/scikit-learn/blob/main/build_tools/azure/pylatest_conda_forge_mkl_osx-64_conda.lock) that have less meta-information.

In skrub, we went with pixi and automatic weekly update of some dependencies bring a significant amount of changes.

So the "environment" part of the pixi.lock + the checksum I think would be almost enough in this particular setting of CIs.

I clearly can understand that in some other cases, you might need more information to have a fully and secure reproducible environment.

Maybe there is a way to set the cursor for the underlying use-case.

seem like they could be re-computed from the pixi.toml file again

If this is something possible, this could be one of the trade-off where less information are stored at a small cost of recomputing some potential information when the CI is triggered.

[tdejager]

Like discussed in the discord. I'm also curious what we can do if we deduplicate some of the common prefixes.

[tdejager]

Not off the top of my head -- I definitely acknowledge it's a hard problem, so I was at least somewhat relieved to find that it still compressed reasonably well in the packfile.

BTW @jleibs what (commands) are you using to compare in the packfile?