Migrate development workflow to Pixi

[VeckoTheGecko]

(posting now for visibility and to request feedback/pixi debugging help)

Overview

Fixes #10732

This PR migrates the dev workflow and CI for Xarray across to Pixi, providing the following benefits:

• Composable environments via dependency groups (in pixi called "features")
• Support for multiple environments
• Task running
• lock file support

See the original issue for more info.

Changes so far in this PR:

• Add pixi badge to readme
• Migrated most environment files to Pixi config in pixi.toml split apart into features that I thought were sensible . I left out environment-benchmarks.yml, binder/environment as that has interactions with asv, and Binder - this PR is already big enough, and I think those should be explored another time.
  • I made the environments in Pixi have similar names as the original conda environments to ease migration
• Introduced a cache-pixi-lock.yml workflow (see below section "Considerations")
• Updated ci.yaml
  • Fixed now! 98% there - for some reason the CI of Pixi is finding which pytest to be .pixi/envs/default/Scripts/pytest while local pixi run -e test-all-deps-py313 which pytest is finding .pixi/envs/test-all-deps-py313/bin/pytest (see test-pixi-dust branch, example action run) . Any ideas why @lucascolley ?
• Update CI additional
• Update RTD build
• Migrate minimal environment to Pixi as well
• Update contributing guidelines (see "Feedback wanted" section below)
• Migrate hypothesis tests
• Migrate nightly dev testing

I've tried to make the commits tidy to help with reviewing commit by commit, which might be easier. I also was quite diligent when migrating from the old env files to make sure versions were the same.

Testing instructions

Resources: Pixi Scipy 2025 talk | Docs: Manifest Reference

• pixi info -> show info about the pixi environments
• Build documentation: pixi run doc
• Run tests: pixi run test then choose the environment you want to run the tests in (or pixi run -e environment_name test)
  • Most often you'll want the test-all-deps-py313 environment (corresponding to the old environment ci/requirements/environment.yml)
• Run pre-commit: pixi run pre-commit
• Run mypy typing: pixi run typing

Enter an environment (equivalent to conda activate): pixi shell -e env_name
Exit an environment (equivalent to conda activate): exit or Ctrl+D

See all tasks: pixi run

Considerations

Lock files o' lock files

There was some interesting conversation in #10732 (comment) about lock files. To summarise:

We have two choices to handle the lock files, either (a) generate them in CI, or (b) commit them to the repo and periodically update them.

(a) generating in CI (done in this PR):

• add pixi.lock to .gitignore
• have a workflow which generates the lock file. Cache this under a key that is date + hash(pixi.toml)
• have all workflows restore this pixi.lock file for environment creation

Pros:

• lock file is only generated once a day and shared across workflows - saving 40s per run
• close to what was previously done (with daily caches)
• minimal changes to workflows (only need to add a few lines - cache-pixi-lock.yml is re-usable across different projects).

Cons:

• Mismatch of devs pixi.lock and what's in CI. Local developers need to periodically delete pixi.lock and regenerate it.
• Missed benefit of perfectly reproducible dev environments cross developers and with CI

(b) commit the lock files

(I think this is the gist of it)

• commit the lock file (now local devs and CI can use this lockfile) - around 40k lines
• add GitHub PR automation to automatically update the lockfile every 3 weeks
• Most of CI works from committed lockfile, but there can be a job bleeding-edge which runs every few days by taking the current lockfile, running an update, and then running tests. Any failures can be automatically reported in an issue
  • Then, to "resolve" that issue you can add a pin in the pixi.toml manifest and talk with upstream to see whats up

Pros:

• no need to generate lock files in CI
• perfectly reproducible dev environments cross developers and CI

Cons:

• there is a bunch of added complexity/maintainer burden to setting this up (automated workflows etc)

@lucascolley knows the full extent as he's been exploring this setup at Scipy

Conclusion

Approach (a) has minimal setup/maintenance with little downside. I think that it's a good solution for smaller projects in particular (we've adopted it at Parcels - cc @maxrjones might be interesting based on your comment )

Approach (b) is more robust if having the same environment between all devs is highly valued (@shoyer mentioned during a dev meeting that this would be good for xarray), but requires more setup.

I recommend we go for (a) as is done in this PR, and consider (b) separately .

@lucascolley would it be beneficial to do a write-up of all this on prefix.dev sometime to help guide others dealing with this? I'm happy to write or collab on a blog post.

Feedback wanted: To what extent do we promote Conda dev workflows

Yeah - I don't know. In the projects I'm working on I've gone full Pixi, but those are smaller projects.

I've deleted the old environment files to avoid duplication, but can re-add them to the extent which you want to support conda dev workflows.

I've held off on updating the contributing instructions for this reason.

EDIT: Joined the dev meeting - @keewis doesn't think its a bad idea to fully migrate dev instructions from conda to Pixi. Later (if people really want conda instructions) we can show how to use pixi to export a conda compatible env file - no need for us to maintain two separate env files.

---

I think that's about it! I don't think I've forgotten anything, but it is late on a Friday so maybe - will update if that's the case :)

Let me know if you want me to drop by the dev meeting on 5 Nov - but I'm happy to keep this async otherwise.

---

(🎉 for my first significant contribution to Xarray!!!)

[dcherian]

cc @kmuehlbauer

[kmuehlbauer]

I'm looking for the first time in a pixi-based setup, please bear with me.

To my understanding in the current nightly we are using h5py, hdf5, netcdf4-python and libnetcdf from conda forge (where libnetcdf/netcdf4-python as well as h5py are build from the same hdf5.

In this pixi based setup it looks like we get h5py-wheel ( https://pypi.anaconda.org/scientific-python-nightly-wheels/simple/h5py/3.15.1/h5py-3.15.1-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl with bundled hdf5) and netcdf4-wheel (netcdf4-1.7.3-cp311-abi3-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl with bundled libnetcdf/hdf5).

Not sure if compatible versions of hdf5 are in the above two wheels. Do we have any means of control, which versions of the underlying c libraries are bundled into those wheels?

[keewis]

maybe we need to make sure to only install the packages we want from the index (numpy, scipy, pandas, matplotlib) instead of all of the related ones?

So for example h5py should be installed from conda-forge, but h5netcdf should be installed from its main branch.

[kmuehlbauer]

@keewis That looks reasonable. Nevertheless, should we aim for nightly testing netcdf4-python, too?

I remember vaguely, that we removed that from nightly testing because of some numpy 2 issues.

[keewis]

did we ever do that? As far as I remember I only removed the conda-forge versions from the environment because they were clashing with numpy=2, but those should have been added back (unless I forgot).

As for using the conda-forge version of h5py, that's not what we did for the nightly CI before. We don't have a way to control the version of hdf5 with PyPI, though, whatever is in the wheel is what you get (this was one of the main motivation for the creation of conda) – or at least that was the case a couple years ago, and I don't think much has changed since then. So I'll retract my suggestion, if the versions of hdf5 installed by h5py is incompatible with the dependency of netcdf4 we have an issue.

[kmuehlbauer]

It seems my memory failed me @keewis. #9136

Not sure how to tackle this, though.

[keewis]

if you're sure the issue is the mismatch in hdf5 versions I'm sure we can try to come up with something (but for now I think it would be easiest to exclude the nightly h5py from the deps and use the conda-forge version instead)

[VeckoTheGecko]

but for now I think it would be easiest to exclude the nightly h5py from the deps and use the conda-forge version instead

I assume this would mean moving h5netcdf from a Pypi git dep to a conda dep (there is no direct dep on h5py in the nightly feature currently).

The following diff, however, doesn't work locally for me either:

diff --git a/pixi.toml b/pixi.toml
index cdb9553f..dbc28e51 100644
--- a/pixi.toml
+++ b/pixi.toml
@@ -259,6 +259,7 @@ extra-index-urls = [
 
 [feature.nightly.dependencies]
 python = "*"
+h5netcdf = "*"
 
 [feature.nightly.pypi-options.dependency-overrides]
 dask = { git = "https://github.com/dask/dask" }
@@ -283,7 +284,7 @@ bottleneck = { git = "https://github.com/pydata/bottleneck" }
 fsspec = { git = "https://github.com/intake/filesystem_spec" }
 nc-time-axis = { git = "https://github.com/SciTools/nc-time-axis" }
 flox = { git = "https://github.com/xarray-contrib/flox" }
-h5netcdf = { git = "https://github.com/h5netcdf/h5netcdf" }
+# h5netcdf = { git = "https://github.com/h5netcdf/h5netcdf" } # https://github.com/pydata/xarray/pull/10888#issuecomment-3544194835
 opt_einsum = { git = "https://github.com/dgasmith/opt_einsum" }
 # sparse = { git = "https://github.com/pydata/sparse"}
 

FAILED xarray/tests/test_backends.py::TestNetCDF4ViaDaskData::test_encoding_kwarg_vlen_string[input_strings1-False] - RuntimeError: NetCDF: HDF error
FAILED xarray/tests/test_backends.py::TestNetCDF4ViaDaskData::test_encoding_kwarg_vlen_string[input_strings2-False] - RuntimeError: NetCDF: HDF error
FAILED xarray/tests/test_backends.py::TestNetCDF4ViaDaskData::test_roundtrip_string_with_fill_value_vlen[XXX] - RuntimeError: NetCDF: Problem with HDF5 dimscales.
FAILED xarray/tests/test_backends.py::TestNetCDF4ViaDaskData::test_roundtrip_string_with_fill_value_vlen[] - RuntimeError: NetCDF: Problem with HDF5 dimscales.
FAILED xarray/tests/test_backends.py::TestNetCDF4ViaDaskData::test_roundtrip_string_with_fill_value_vlen[b\xe1r] - RuntimeError: NetCDF: Problem with HDF5 dimscales.
FAILED xarray/tests/test_backends.py::TestNetCDF4ViaDaskData::test_roundtrip_character_array - RuntimeError: NetCDF: HDF error
...

Feeling a bit out of my depth here - is having h5netcdf as a nightly dep a blocker for this PR, or is it something that can be tracked for later? (and if a blocker, can someone please submit a patch)

[dcherian]

Feeling a bit out of my depth here - is having h5netcdf as a nightly dep a blocker for this PR,

I am +1 on getting this in and making h5netcdf a git dependency in a later PR. This PR is already a great upgrade

[keewis]

it should be sufficient to add h5py to the conda dependencies of the feature (and keep h5netcdf as a git dependency), but if that turns out to be complicated I'd be fine with skipping h5netcdf for now.

[keewis]

I've got another two questions on the CI setup

[keewis]

is there a reason why we pin this? I'm asking because this may be a pain to keep in sync or upgrade. If not, would it be possible to use the most recent version, somehow?

If that's not possible, is there a reason why we don't use the environment variable?

Suggested change

	pixi-version: "v0.58.0" # keep in sync with env var above
	pixi-version: "${{ env.PIXI_VERSION }}"

[VeckoTheGecko]

Its because of this thread in the Prefix Discord https://discord.com/channels/1082332781146800168/1433751284317687868

TLDR(ish)

I have a CI setup that does the following:

• Generate a pixi.lock file based on the pixi.toml (saved under a cache ID of the date and a hash of the contents of the pixi.toml)
• Use the cached pixi.lock to set up the environment

This workflow has been working quite well, and has been reducing CI times significantly without us having to commit lock files.

Though after some weeks i've run into flakiness (https://github.com/Parcels-code/Parcels/actions/runs/18939417196) for the first time, with the message

pixi install --locked
Error:   × lock-file not up-to-date with the workspace

There was lockfile incompatability between pixi versions which surfaced since we were using latest pixi all the time, and the lockfile (for that day) was generated with an older version of Pixi hence breaking runs. Nasty to debug since it requires manually clearing caches from GitHub and re-requesting runs.

@lucascolley mentioned in the thread they try to maintain lockfile compatability across versions, but I thought it would be good to pin it in case just to be safe

---

If that's not possible, is there a reason why we don't use the environment variable?

Will update

[VeckoTheGecko]

If that's not possible, is there a reason why we don't use the environment variable?

Oh yes, the env context isn't available at this level

"Available expression contexts: github, inputs, vars, needs, strategy, matrix"

[keewis]

we'll have to find a way to automatically upgrade this version through some bot like dependabot, then; there's no way we can keep this up-to-date manually

[lucascolley]

in my experience I've never had a particularly pressing reason to bump the Pixi version.

EDIT: of course this would be valuable in case of big security fixes

[VeckoTheGecko]

We can set the Pixi version on a repo level using the vars context https://docs.github.com/en/actions/how-tos/write-workflows/choose-what-workflows-do/use-variables#defining-variables-for-multiple-workflows

[keewis]

Oh yes, the env context isn't available at this level

Is that because you're reusing a workflow? Could we maybe convert the caching of the lock file to an action?

[VeckoTheGecko]

it should be sufficient to add h5py to the conda dependencies of the feature (and keep h5netcdf as a git dependency)

Didn't work for me :/ (testing locally)

[keewis]

I did some experiments myself and realized two things:

• pixi first solves for conda and only after that for PyPI
• there's no way to override the conda solver with the PyPI solver

This has the ugly side-effect that e.g. numpy is installed from conda-forge because it is (indirectly?) depended on by the typing feature, and thus we don't get the nightly version (and the same for packaging, which is also depended on by the test feature). Removing the typing feature from nightly we get the proper environment if we don't count packaging.

I've taken the liberty of pushing three commits to pin python to 3.13, remove the typing feature from nightly, and switch to using the dependency-overrides (which I believe makes this a bit easier to understand?).

[keewis]

but that doesn't resolve our issue with mismatching hdf5 versions, for that the pinning of conda versions before the PyPI solve is a blocker

[lucascolley]

but that doesn't resolve our issue with mismatching hdf5 versions, for that the pinning of conda versions before the PyPI solve is a blocker

is it possible to install all packages which depend on hdf5 from PyPI? Or is it then not possible to ensure compatible versions without trial and error?

[keewis]

I don't think so (but I'm not an expert on this), as there is nothing the solver could use to detect which version of hdf5 is contained within the wheel. As soon as you have a fundamental mismatch between the two, you may get segfaults and other weird errors.

[lucascolley]

perhaps the long-term solution is nightly conda packages, although that will require quite broad ecosystem coordination I imagine

[keewis]

I guess for pure-conda solves something similar to the scientific-python index would be nice

[lucascolley]

I think this would be somewhat feasible once we can build all the projects we are interested in with pixi-build. I guess we are wandering a bit off-topic from this PR, though.