Bringing our environment to modern standards (also with removal of NCL)

[valeriupredoi]

This is an umbrella issue that is motivated by our dependencies environment being fairly patchy and out of date; there are some outstaning issues at play:

• the need for NCL to be moved to an "optional dependency" Remove ncl and its dependencies from main dependencies and add documentation how to install it #3853 : ncl is lagging behind with every Python and co upgrade, specifically for us, it makes it impossible to:
  • move to support Python 3.13 via Support Python 3.13 #3805
  • move to support recent versions of esmpy>=8.6
  • start supporting numpy>=2
  • altogether get rid of importlib_metadata <8 (predicated by older esmpy, predicated, in turn, by NCL)
  • makes the environment unsolvable for Python <3.13, when curl is unpinned and let to latest (see below)
• unpin curl, hopefully with a correct curl=8.11, caused by the issues it has with Julia installing packages, see Unpin curl #3758 -> curl 8.12 still affecting Julia pkgs installation since Julia hasn't moved past 1.10.4 in conda-forge; unpinning curl to latest makes the env unsolvable for Python <3.13 (if we keep NCL as mandatory dep)
• Julia conda-forge package has deplorable support (ie currently lagging behind by four minor versions), so suggest removing it from our conda env and use its "classic" install as @bouweandela suggests, see Remove Julia from conda dependency environment (but keep tests for esmvaltool install Julia) and add docs how to install from source #3921
• unpin cartopy, see Unpin cartopy #3769 -> this is done in Update environment: pin iris>=3.11, unpin cartopy and allow for numpy >=2 #3811 - it is safe to do that since we pin iris there
• unpin pandas - that will be done when we release ESMValCore v2.12 since the fix for pandas is there now Unpin pandas  #3924
• unpin numpy<2 ie upper bound - iris 3.11 is safe for numpy>=2 - we should free it too, and consider issues with diagnostics not being numpy2 compatible as they come, probably during the release process for v2.12; unpinning upper bound for `numpy done in Update environment: pin iris>=3.11, unpin cartopy and allow for numpy >=2 #3811
• unpin r-base <4.3 - Unpin pandas  #3924

[valeriupredoi]

@ESMValGroup/technical-lead-development-team this is a big red flag and something looming above us:

• currently we are pinning curl so that we avoid an issue with Julia installing packages; this has now been resolved at Julia end, but we are awaiting a new Julia release to have the fix in...
• however, that pinned curl is keeping us from a fairly catastrophic environment solving (or not solving), see the tests where curl pin is removed (curl removed altogether) - solving for Python 3.11 takes about 10 minutes, and for Python 3.12 it actually fails after about 15 minutes - this is unacceptable! NB: I am still to investigate the reason why the env solving fails for 3.12, but...
• look at the tests where I remove NCL and its dependencies - all good and in sub 3 minutes times
• we really-really need to remove NCL from our main dependencies

🍺

[valeriupredoi]

the Python 3.12 solver issue from above appears to be the solver really getting stuck in a corner case, see eg mamba-org/mamba#3290 I tried to reproduce the issue locally, but my ye olde machine promptly ran out of memory ie the solver was using more than 4GB of my free memory, again, not acceptable

[bouweandela]

We could consider switching from mamba to pixi. That would have the advantage that we can define "extra"s as available for PyPI packages, but then for conda-forge packages, and multiple environments, such as the Linux (environment.yml) and OSX (environment_osx.yml) one, all in the pyproject.toml file. I had a go at that for ESMValCore a few months ago (see here) and then I ran into prefix-dev/pixi#2458, but that seems to be kind of solved now.

[valeriupredoi]

thanks bud! I'll look into that, though I am bit weary of going for something that (I bet) not many people have heard of - let's not try that at least for the upcoming release, there are already quite a few major bits that we bring in ie Python 3.12 (hopefully 3.13 too) etc, but will defo look at it after the next release 🍺

[valeriupredoi]

love the title of that issue BTW 🤣

[rbeucher]

I'll take a look at Pixi, but it seems to be primarily aimed at developers.

[bouweandela]

This discussion seems to be mostly about our environment.yml files, which are aimed at developers. People who only use the tool can already install it without NCL using e.g. conda install esmvaltool-python esmvaltool-r

[valeriupredoi]

@bouweandela I've updated the requirements table in the Issue description, and also did have a look at Pixi - unfortunately, Pixi looks a great little tool, but well distanced from our current infrastructure - I think it may be a good idea to test it in the future, but not just yet ie in the immediate aftermath of 2.12 release; until then NCL is a BIG roadblock