Use conda-lock for reproducible conda env

[tobiasdiez]

The conda workflow is broken since a few months due to various issues that arise from some packages being updated in conda-forge and sage not being quite compatible with the new version. For example, currently it fails as Cython 3.0.3 is used.

To prevent such issues happening in the feature, we use conda-lock to create a lock file for each os + python combination, from which one can then create a reproducible conda env. These lock files are created using commands of the form:

conda-lock --channel conda-forge --kind env --platform linux-64 --platform osx-64 --file ./src/environment-dev-3.10.yml --filename-template "src/environment-dev-3.10-{platform}"

Follow-up: Add a github workflow that updates these lock files automatically, similar to:

• https://github.com/ibis-project/ibis/blob/master/.github/workflows/conda-lock.yml
• https://github.com/svopper/munaiah-analyser/blob/main/out/SciTools/iris/.github_workflows_refresh-lockfiles.yml
• https://github.com/chipsalliance/fpga-interchange-tests/blob/main/.github/workflows/update_conda_lock.yml
• https://github.com/ESMValGroup/ESMValTool/blob/main/.github/workflows/create-condalock-file.yml

📝 Checklist

• The title is concise, informative, and self-explanatory.
• The description explains in detail what this PR is about.
• I have linked a relevant issue or discussion.
• I have created tests covering the changes.
• I have updated the documentation accordingly.

⌛ Dependencies

• bootstrap-conda: Refactor, generate versioned environment files #36405: to get env files for each python version
• Reduce execution time of conda workflow #36767

[mkoeppe]

How can conda-lock (and in particular an automatic workflow that runs conda-lock and commits the lock files) be able to resolve to a working set of versions, when the problem is that the resolution does not work for users when they install from our environment?

Is it not just that some version constraints were missing (which you added here)?

[tobiasdiez]

How can conda-lock (and in particular an automatic workflow that runs conda-lock and commits the lock files) be able to resolve to a working set of versions, when the problem is that the resolution does not work for users when they install from our environment?

Is it not just that some version constraints were missing (which you added here)?

The point is that you always have a working environment based off the lock files. If the automatic update of the lock file fails, manual work is needed (e.g. adding a version constraint). But only then.

[mkoeppe]

Could you explain the changes to bootstrap-conda?

Do I read it correctly that unversioned conda environment files are no longer generated?

[tobiasdiez]

Could you explain the changes to bootstrap-conda?

Do I read it correctly that unversioned conda environment files are no longer generated?

Conda-lock needs a python=3.x line in the environment file to know what kind of python version to use. Thus, the bootstrap conda file now generates a bunch of enviroment files that are the same as the existing ones, except that they contain a python version specifier. All of them are git-ignored.

[maresb]

Hey, as a fan of sage and maintainer of conda-lock, I'm very happy you're finding it useful.

Could you please test again with removing --no-mamba? We hope that the issues mentioned above were resolved with the release of Conda 23.7 a few days ago.

[tobiasdiez]

@maresb For me removing the --no-mamba switch worked even with the slightly older versions of mamba 1.4.9 and conda 23.5.2. And I've used it to create the lock files in the last commit. Based on this experience conda/conda-lock#292 could be closed.

@mkoeppe @isuruf Anything else to do here?

[mkoeppe]

Work items:

• Rebase on Add needed pip packages to src/environment.yml, minimize conda environments #35593 (which has cherry-picked from here)
• Maintenance documentation

[tobiasdiez]

All tests are passing now, so this should be ready to go.

[tobiasdiez]

@dimpase are you okay with these changes here?

[dimpase]

would you like to rebase it?

[tobiasdiez]

Okay, done.

[dimpase]

OK

[vbraun]

merge conflict

[tobiasdiez]

Thanks Dima for the review. Merge was trivial, so I'll keep the "positive review" label.

[github-actions]

Documentation preview for this PR (built with commit bc91e96; changes) is ready! 🎉

[tobiasdiez]

Could someone of the @sagemath/core members please change the required workflows in the branch protection rules to include all of the current 6 conda workflows and remove the ones with the old naming "ubuntu-latest". Thanks!