ModuleNotFoundError: No module named 'build' with no command line options

[wholtz]

Checklist

• I added a descriptive title
• I searched open reports and couldn't find a duplicate

What happened?

$ conda-lock
/Users/example/.local/pipx/venvs/conda-lock/lib/python3.9/site-packages/urllib3/__init__.py:35: NotOpenSSLWarning: urllib3 v2 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020
  warnings.warn(
Traceback (most recent call last):
  File "/Users/example/.local/bin/conda-lock", line 5, in <module>
    from conda_lock import main
  File "/Users/example/.local/pipx/venvs/conda-lock/lib/python3.9/site-packages/conda_lock/__init__.py", line 3, in <module>
    from conda_lock.conda_lock import main
  File "/Users/example/.local/pipx/venvs/conda-lock/lib/python3.9/site-packages/conda_lock/conda_lock.py", line 76, in <module>
    from conda_lock.pypi_solver import solve_pypi
  File "/Users/example/.local/pipx/venvs/conda-lock/lib/python3.9/site-packages/conda_lock/pypi_solver.py", line 28, in <module>
    from conda_lock.interfaces.vendored_poetry import (
  File "/Users/example/.local/pipx/venvs/conda-lock/lib/python3.9/site-packages/conda_lock/interfaces/vendored_poetry.py", line 20, in <module>
    from conda_lock._vendor.poetry.installation.chooser import Chooser
  File "/Users/example/.local/pipx/venvs/conda-lock/lib/python3.9/site-packages/conda_lock/_vendor/poetry/installation/__init__.py", line 3, in <module>
    from conda_lock._vendor.poetry.installation.installer import Installer
  File "/Users/example/.local/pipx/venvs/conda-lock/lib/python3.9/site-packages/conda_lock/_vendor/poetry/installation/installer.py", line 9, in <module>
    from conda_lock._vendor.poetry.installation.executor import Executor
  File "/Users/example/.local/pipx/venvs/conda-lock/lib/python3.9/site-packages/conda_lock/_vendor/poetry/installation/executor.py", line 20, in <module>
    from conda_lock._vendor.poetry.installation.chef import Chef
  File "/Users/example/.local/pipx/venvs/conda-lock/lib/python3.9/site-packages/conda_lock/_vendor/poetry/installation/chef.py", line 11, in <module>
    from build import BuildBackendException
ModuleNotFoundError: No module named 'build'

I expected to get the the brief help output, not a stack trace.

Additional Context

This was on a Apple M2 CPU (osx 14.6.1). I installed conda-lock via:

pipx install --python /usr/bin/python3 git+https://github.com/conda/conda-lock.git@9611d11

(Note I am installing the main branch, not a published release. I am having a solving issue with the current release that I hoped was fixed.)

The above stack trace is from using the system python (3.9.6)

I also tried using a macport installed version of python (3.10.11) and got the same error (minus the OpenSSL warning).

[maresb]

Thanks so much @wholtz for your vigilance on this! I confirm that main is currently quite broken. I'm looking into fixes, but it will be a significant amount of work (e.g. we also need to revendor poetry-core and cleo).

CC @romain-intel

On an unrelated note, @wholtz, is there any particular reason you haven't migrated everything to pixi? I'd really like to understand any remaining use cases of conda-lock as per #615.

[wholtz]

I haven't looked at pixi in a while. The last time I looked it still seemed alpha level. But it sounds like I should take another look.

[romain-intel]

Is it just the build dependency that's missing from the pyproject.toml? I was able to repro this in a brand new environment (conda-lock worked in my dev env -- the one I was developing conda-lock in but not in a pristine one). It seems to work though if I just install the build package. I think when I moved from 1.8.2 to 1.8.3 I stupidely assumed that nothing had changed but it seems it did... I'll push a PR to fix that. Sorry about that and good catch.

(in the meantime, you can just manually install build in your enviornment and you should be good to go)

[maresb]

@romain-intel I think it goes a lot deeper than this unfortunately. As noted above, note that we forgot to revendor cleo and poetry-core. I need to review my original vendoring scripts and make sure we've covered all the necessary steps.

[romain-intel]

@maresb -- looking at the commit when I ran vendoring, cleo did change (bunch of adds) as well as poetry/core (which is where poetry-core gets installed -- it won't create a separate top-level package). When I ran it I do remember it downloading those things so I believe it did do the proper thing. Why do you say that it goes deeper? In a clean install, at least for me, I was able to:

• create a clean environment (just with python)
• install conda-lock using the command above (pip install git+https:/....)
• install build (pip install build)
• run conda-lock
• resolve a very simple environment with conda-lock

In short, it seems to work.
Are the above steps not working for you?

[maresb]

Maybe I'm mistaken, apologies if that's the case @romain-intel. I will take a closer look.

[romain-intel]

No worries -- I feel stupid enough for not having checked the new dependencies. Am working on a PR now for it.

[maresb]

I was simply confused @romain-intel, sorry about that!!! 🙈 (I must have been working in an older branch when I last checked.)

Indeed, as far as we know this was just a simple dependency mistake. I would still like to review my original vendor stuff before releasing to ensure there's nothing else we overlooked. Looking forward to your PR!

[romain-intel]

Added a PR. Sorry about that miss. I went through the dependencies again. There is one we don't have which is https://github.com/python-poetry/poetry-plugin-export but I don't think we use it. Can always add if needed though.

[maresb]

@romain-intel, @wholtz, I'm getting very close to a major release of conda-lock. I would be very interested to know the results if you could try main in some real-world scenarios to ensure that we didn't break anything.

[romain-intel]

Awesome. Yes, I saw a lot of activity :). I can run through some of the test cases we have here over the week-end. If you are interested, I may also have one small PR coming your way to add support for "path" dependencies and "subdirectory" (for the already supported "git" dependency). I haven't finished but I think it may just be a few lines (since poetry already supports it).

[wholtz]

On python 3.12 (installed via macports), I get an error that distutils is missing:

Traceback (most recent call last):
  File "/Users/willholtz/.local/bin/conda-lock", line 5, in <module>
    from conda_lock import main
  File "/Users/willholtz/.local/pipx/venvs/conda-lock/lib/python3.12/site-packages/conda_lock/__init__.py", line 3, in <module>
    from conda_lock.conda_lock import main
  File "/Users/willholtz/.local/pipx/venvs/conda-lock/lib/python3.12/site-packages/conda_lock/conda_lock.py", line 50, in <module>
    from conda_lock.conda_solver import solve_conda
  File "/Users/willholtz/.local/pipx/venvs/conda-lock/lib/python3.12/site-packages/conda_lock/conda_solver.py", line 19, in <module>
    from conda_lock.interfaces.vendored_conda import MatchSpec
  File "/Users/willholtz/.local/pipx/venvs/conda-lock/lib/python3.12/site-packages/conda_lock/interfaces/vendored_conda.py", line 2, in <module>
    from conda_lock._vendor.conda.models.match_spec import MatchSpec
  File "/Users/willholtz/.local/pipx/venvs/conda-lock/lib/python3.12/site-packages/conda_lock/_vendor/conda/models/match_spec.py", line 20, in <module>
    from .channel import Channel
  File "/Users/willholtz/.local/pipx/venvs/conda-lock/lib/python3.12/site-packages/conda_lock/_vendor/conda/models/channel.py", line 17, in <module>
    from ..base.context import context, Context
  File "/Users/willholtz/.local/pipx/venvs/conda-lock/lib/python3.12/site-packages/conda_lock/_vendor/conda/base/context.py", line 55, in <module>
    from ..common.configuration import (Configuration, ConfigurationLoadError, MapParameter,
  File "/Users/willholtz/.local/pipx/venvs/conda-lock/lib/python3.12/site-packages/conda_lock/_vendor/conda/common/configuration.py", line 41, in <module>
    from .path import expand
  File "/Users/willholtz/.local/pipx/venvs/conda-lock/lib/python3.12/site-packages/conda_lock/_vendor/conda/common/path.py", line 21, in <module>
    from distutils.spawn import find_executable
ModuleNotFoundError: No module named 'distutils'

[wholtz]

Using python 3.10 to run conda-lock, I have succesfully completed a pretty nasty test case:

• 45 direct conda dependencies from conda-forge and bioconda
• 3 direct pip dependencies (lots of transative dependencies)
• pip dependencies come from private AWS CodeArtifcat pypi repo
• locked for linux-64 and osx-64
• run on Apple M2 CPU

[maresb]

Amazing, thanks so much @wholtz for your always invaluable help.

I'm relieved that your nasty test case works.

I'm disappointed that my new integration test didn't catch the distutils problem; I suppose the runner's environment is not as pristine as I hoped.

[maresb]

Ok, I reproduced the distutils problem in #698 and fixed it in #699. Thanks so much again for pointing that out!!!

[romain-intel]

I've been playing around with it a little bit and things look good. I ran it against a few scenarios that I have including private GIT repo, GIT urls, extra repositories, etc.

One nice thing I noticed:

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

[tool.conda-lock]
channels = ['conda-forge']
pip-repositories = ['https://<internal pypi>', 'https://download.pytorch.org/whl/cu118']
allow-pypi-requests = false

[tool.conda-lock.dependencies]
"pyarrow" = ">=0.17.1"
"requests" = ">=2.21.0"
"platformdirs" = "*"
"fastavro" = ">=1.6.0"
"boto3" = ">=1.14.0"
"pandas" = ">=0.24.0"
"cffi" = ">=1.13.0,!=1.15.0"
"python" = "3.10.*"
"pip" = "*"

[tool.poetry.dependencies]
"polars" = {version = "<=0.19",  source="pypi"}
"clip" = {git = "git+https://github.com/openai/CLIP.git@d50d76daa670286dd6cacf3bcd80b5e4823fc8e1"}
"outlier-detector" = {version = "0.0.3", source="pypi"}

would initially not resolve at all because it would try finding wrong packages on that second repo and get 403s. This now works but with warnings like so:

WARNING:conda_lock._vendor.poetry.repositories.repository.LegacyRepository:<c1>Source (702e2f9cb7f0a7087ac298fc8571ee0ce24dfc3f5c9966286fa73a129b8c114a):</c1> Authorization error accessing https://download.pytorch.org/whl/cu118/google-re2/

(so it works so that's a big plus :) -- mentioning since we changed stuff around the auth)

Ooh, and I did get it to work with local directories too :). I'll submit a PR for that shortly (need to clean up a few things and see if I can improve it a bit).

[maresb]

@romain-intel, very glad to see you trying this out!!! Please see #708 for the first draft of my guide.

There is a lot of low-hanging fruit regarding easy PRs for Git and path deps. I haven't thought about private pip repositories; that may or may not be easy.

I'm very eager to get your feedback and contributions!

Also @wholtz I partially had you in mind as my audience while drafting #708. My hope is that can efficiently get you up-to-speed with exactly what you need to know to start using pixi.