-
-
Notifications
You must be signed in to change notification settings - Fork 438
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use conda-lock for reproducible conda env #35986
Conversation
How can conda-lock (and in particular an automatic workflow that runs conda-lock and commits the lock files) be able to resolve to a working set of versions, when the problem is that the resolution does not work for users when they install from our environment? Is it not just that some version constraints were missing (which you added here)? |
The point is that you always have a working environment based off the lock files. If the automatic update of the lock file fails, manual work is needed (e.g. adding a version constraint). But only then. |
Could you explain the changes to Do I read it correctly that unversioned conda environment files are no longer generated? |
Conda-lock needs a |
Co-authored-by: Isuru Fernando <[email protected]>
Hey, as a fan of sage and maintainer of conda-lock, I'm very happy you're finding it useful. Could you please test again with removing |
@maresb For me removing the |
Work items:
|
All tests are passing now, so this should be ready to go. |
@dimpase are you okay with these changes here? |
would you like to rebase it? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Okay, done.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK
merge conflict |
Thanks Dima for the review. Merge was trivial, so I'll keep the "positive review" label. |
Documentation preview for this PR (built with commit bc91e96; changes) is ready! 🎉 |
<!-- ^^^^^ Please provide a concise, informative and self-explanatory title. Don't put issue numbers in there, do this in the PR body below. For example, instead of "Fixes sagemath#1234" use "Introduce new method to calculate 1+1" --> <!-- Describe your changes here in detail --> <!-- Why is this change required? What problem does it solve? --> <!-- If this PR resolves an open issue, please link to it here. For example "Fixes sagemath#12345". --> <!-- If your change requires a documentation PR, please link it appropriately. --> The conda workflow is broken since a few months due to various issues that arise from some packages being updated in conda-forge and sage not being quite compatible with the new version. For example, currently it fails as Cython 3.0.3 is used. To prevent such issues happening in the feature, we use [conda- lock](https://conda.github.io/conda-lock/) to create a lock file for each os + python combination, from which one can then create a reproducible conda env. These lock files are created using commands of the form: ```bash conda-lock --channel conda-forge --kind env --platform linux-64 --platform osx-64 --file ./src/environment-dev-3.10.yml --filename- template "src/environment-dev-3.10-{platform}" ``` Follow-up: Add a github workflow that updates these lock files automatically, similar to: - https://github.com/ibis- project/ibis/blob/master/.github/workflows/conda-lock.yml - https://github.com/svopper/munaiah- analyser/blob/main/out/SciTools/iris/.github_workflows_refresh- lockfiles.yml - https://github.com/chipsalliance/fpga-interchange- tests/blob/main/.github/workflows/update_conda_lock.yml - https://github.com/ESMValGroup/ESMValTool/blob/main/.github/workflows/ create-condalock-file.yml ### 📝 Checklist <!-- Put an `x` in all the boxes that apply. --> <!-- If your change requires a documentation PR, please link it appropriately --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> <!-- Feel free to remove irrelevant items. --> - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [ ] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies - sagemath#36405: to get env files for each python version - sagemath#36767 <!-- List all open PRs that this PR logically depends on - sagemath#12345: short description why this is a dependency - sagemath#34567: ... --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> URL: sagemath#35986 Reported by: Tobias Diez Reviewer(s): Dima Pasechnik, Isuru Fernando, Matthias Köppe, Tobias Diez
<!-- ^^^^^ Please provide a concise, informative and self-explanatory title. Don't put issue numbers in there, do this in the PR body below. For example, instead of "Fixes sagemath#1234" use "Introduce new method to calculate 1+1" --> <!-- Describe your changes here in detail --> <!-- Why is this change required? What problem does it solve? --> <!-- If this PR resolves an open issue, please link to it here. For example "Fixes sagemath#12345". --> <!-- If your change requires a documentation PR, please link it appropriately. --> The conda workflow is broken since a few months due to various issues that arise from some packages being updated in conda-forge and sage not being quite compatible with the new version. For example, currently it fails as Cython 3.0.3 is used. To prevent such issues happening in the feature, we use [conda- lock](https://conda.github.io/conda-lock/) to create a lock file for each os + python combination, from which one can then create a reproducible conda env. These lock files are created using commands of the form: ```bash conda-lock --channel conda-forge --kind env --platform linux-64 --platform osx-64 --file ./src/environment-dev-3.10.yml --filename- template "src/environment-dev-3.10-{platform}" ``` Follow-up: Add a github workflow that updates these lock files automatically, similar to: - https://github.com/ibis- project/ibis/blob/master/.github/workflows/conda-lock.yml - https://github.com/svopper/munaiah- analyser/blob/main/out/SciTools/iris/.github_workflows_refresh- lockfiles.yml - https://github.com/chipsalliance/fpga-interchange- tests/blob/main/.github/workflows/update_conda_lock.yml - https://github.com/ESMValGroup/ESMValTool/blob/main/.github/workflows/ create-condalock-file.yml ### 📝 Checklist <!-- Put an `x` in all the boxes that apply. --> <!-- If your change requires a documentation PR, please link it appropriately --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> <!-- Feel free to remove irrelevant items. --> - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [ ] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies - sagemath#36405: to get env files for each python version - sagemath#36767 <!-- List all open PRs that this PR logically depends on - sagemath#12345: short description why this is a dependency - sagemath#34567: ... --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> URL: sagemath#35986 Reported by: Tobias Diez Reviewer(s): Dima Pasechnik, Isuru Fernando, Matthias Köppe, Tobias Diez
<!-- ^^^^^ Please provide a concise, informative and self-explanatory title. Don't put issue numbers in there, do this in the PR body below. For example, instead of "Fixes sagemath#1234" use "Introduce new method to calculate 1+1" --> <!-- Describe your changes here in detail --> <!-- Why is this change required? What problem does it solve? --> <!-- If this PR resolves an open issue, please link to it here. For example "Fixes sagemath#12345". --> <!-- If your change requires a documentation PR, please link it appropriately. --> The conda workflow is broken since a few months due to various issues that arise from some packages being updated in conda-forge and sage not being quite compatible with the new version. For example, currently it fails as Cython 3.0.3 is used. To prevent such issues happening in the feature, we use [conda- lock](https://conda.github.io/conda-lock/) to create a lock file for each os + python combination, from which one can then create a reproducible conda env. These lock files are created using commands of the form: ```bash conda-lock --channel conda-forge --kind env --platform linux-64 --platform osx-64 --file ./src/environment-dev-3.10.yml --filename- template "src/environment-dev-3.10-{platform}" ``` Follow-up: Add a github workflow that updates these lock files automatically, similar to: - https://github.com/ibis- project/ibis/blob/master/.github/workflows/conda-lock.yml - https://github.com/svopper/munaiah- analyser/blob/main/out/SciTools/iris/.github_workflows_refresh- lockfiles.yml - https://github.com/chipsalliance/fpga-interchange- tests/blob/main/.github/workflows/update_conda_lock.yml - https://github.com/ESMValGroup/ESMValTool/blob/main/.github/workflows/ create-condalock-file.yml ### 📝 Checklist <!-- Put an `x` in all the boxes that apply. --> <!-- If your change requires a documentation PR, please link it appropriately --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> <!-- Feel free to remove irrelevant items. --> - [x] The title is concise, informative, and self-explanatory. - [x] The description explains in detail what this PR is about. - [ ] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation accordingly. ### ⌛ Dependencies - sagemath#36405: to get env files for each python version - sagemath#36767 <!-- List all open PRs that this PR logically depends on - sagemath#12345: short description why this is a dependency - sagemath#34567: ... --> <!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> URL: sagemath#35986 Reported by: Tobias Diez Reviewer(s): Dima Pasechnik, Isuru Fernando, Matthias Köppe, Tobias Diez
Could someone of the @sagemath/core members please change the required workflows in the branch protection rules to include all of the current 6 conda workflows and remove the ones with the old naming "ubuntu-latest". Thanks! |
The conda workflow is broken since a few months due to various issues that arise from some packages being updated in conda-forge and sage not being quite compatible with the new version. For example, currently it fails as Cython 3.0.3 is used.
To prevent such issues happening in the feature, we use conda-lock to create a lock file for each os + python combination, from which one can then create a reproducible conda env. These lock files are created using commands of the form:
conda-lock --channel conda-forge --kind env --platform linux-64 --platform osx-64 --file ./src/environment-dev-3.10.yml --filename-template "src/environment-dev-3.10-{platform}"
Follow-up: Add a github workflow that updates these lock files automatically, similar to:
📝 Checklist
⌛ Dependencies
bootstrap-conda
: Refactor, generate versioned environment files #36405: to get env files for each python version