create a development environment

[keewis]

• closes Add pytest-accept to a pixi developer environment #10990

We have quite a few environments now, but for development there are a few other tools needed. So far, we have:

• ipython, black, and ipdb
• pytest-accept

Feel free to accept other tools that need access to the environment. (This environment explicitly excludes typing, but it shouldn't be too hard to add another one should we need it).

Note: environments are only materialized once they are accessed, so if you never use the dev env, pixi should never download and install pytest-accept.

cc @Illviljan

[dcherian]

I've started using pdbpp now, and really like it.

Here's a list from my personal dev tool yaml. Can we add

• pdbpp
• line_profiler
• memory_profiler
• memray
• snakeviz
• icecream
• ipykernel
• snoop (haven't tried this one yet, had forgotten about it but it looks great)

Is asv in the dev env?

[keewis]

hmm... looks like it might be better to have a profiling feature / env, in that case. I'd probably include pyinstrument in that list, as well. memory_profiler does not appear to be maintained anymore (not sure if that's a problem).

I don't think asv is anywhere yet, I believe. Is it possible to use that with pixi, or does that have its own dependency installation system? I think I remember it using conda envs before.

[dcherian]

looks like it might be better to have a profiling feature / env

Why not just one big fat one? :P

memory_profiler does not appear to be maintained anymore (not sure if that's a problem).

Interesting. I still find %mprun -f in a notebook is very effective (recent example).

I don't think asv is anywhere yet, I believe. Is it possible to use that with pixi, or does that have its own dependency installation system

I think it still manages its own; but you can tell it to use rattler as a solver. I think it's useful to jsut add asv so we can run benchmarks.

[ruben-arts]

If you see this env as the "default" env for users, you can just rename it to default instead of dev. This would then allow users to run pixi run xxx or pixi shell to start this developer env. And in scripts you can still use -e to specify the more precise env.
If you still want to use the current default env for something else you can rename it with:

new-default = []

Also, you can use solve-group to specify that all these environments should have the same versions for the overlapping dependencies. These guarantees lower file system usage and reproducible environments between the different sets. That would look like this:

typing = { features = ["typing"], solve-group = "main" }
doc = { features = [
  "doc",
  "backends",
  "test",
  "accel",
  "viz",
  "extras",
], solve-group = "main"}
dev = { features = [ "py313",
  "test",
  "backends",
  "accel",
  "numba",
  "dask",
  "viz",
  "extras",
  "dev",
], solve-group = "main"}

If this works, pixi will guarantee that they all use the exact same python and other base dependencies. When they conflict in dependencies, you can still leave out the solve-group.

[VeckoTheGecko]

Thanks for taking the time to look at this @ruben-arts ! (and for responding to my Discord comment about this :)) ) . Also useful for another project I'm working on

[benbovy]

I agree with @Illviljan #10990 (comment) it's nice to have such a "default" development environment with everything + all bells and whistles, especially for onboarding new contributors.

I'm wondering whether it is possible with pixi to define a local-only environment, though. I.e., an environment that is built on top of the Xarray's pixi workspace manifest but that is not included in that manifest (not VCS tracked nor shared with other Xarray contributors). I haven't seen anything in pixi's documentation and repository issues / discussions but I might have missed it?

The motivation is that everyone has its own preferences regarding development tools such as profilers and debuggers. Some of those tools might be already installed globally and could automatically detect virtual environments. Also in a large project like Xarray, contributors (even the regular ones) are rarely working on every feature of the library nor need all the optional dependencies.

Before #10888 I had a local pixi.toml manifest file for Xarray in which I defined a lightweight development environment for my own needs. This was pretty handy, for example I could quickly run all the unit tests locally and skip all tests that are not relevant for the contributions and reviews I usually do (e.g., specific backends, visualization, etc.). Is there a way to keep doing that now that Xarray has an "official" pixi configuration? I guess a workaround would be to continue maintaining my pixi manifest file in a non-default location and use --manifest-path, but that's sub-optimal.