Poetry doesn't install build-system.requires from dependencies not using Poetry.

[adamgreig]

• Poetry version: 1.4.1 and current master 71b3d6e
• Python version: 3.10.6
• OS version and name: Ubuntu 22.04
• pyproject.toml: https://github.com/adamgreig/poetry_build_requires_bug/blob/master/poetry_bug_2/pyproject.toml

• I am on the latest stable Poetry version, installed using a recommended method.
• I have searched the issues of this repo and believe that this is not a duplicate.
• I have consulted the FAQ and blog for any relevant entries or release notes.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

Poetry doesn't install entries in [build-system.requires] from dependencies not using Poetry. In my case a dependency has a build system requirement on setuptools_scm, which must be installed for setup.py to run successfully. setuptools_scm is listed in the dependency's [build-system.requires] but isn't installed by Poetry, so the overall installation fails.

Adding setuptools_scm to the dependency's actual dependencies list causes it to be installed first, and then the build succeeds.

I ran into this issue trying to install amaranth but have reproduced the error in a minimal test case here. Run poetry install in poetry_bug_2/ to see the error.

It seems like the issue (thanks to @whitequark for investigating) is here:

poetry/src/poetry/inspection/info.py

Line 599 in 71b3d6e

*PEP517_META_BUILD_DEPS,

which should possibly include the extra requirements:

-                *PEP517_META_BUILD_DEPS,
+                *(PEP517_META_BUILD_DEPS + PyProjectTOML(path.joinpath("pyproject.toml")).build_system.requires),

I'm happy to prepare a PR with that change if you would like.

Full error output is below but the pertinent error is:

Unable to determine package info for path: /home/adam/Projects/tmp/poetry_bug/poetry_bug_1

Fallback egg_info generation failed.

Command ['/tmp/tmpq237jcb0/.venv/bin/python', 'setup.py', 'egg_info'] errored with the following return code 1

Output:
Traceback (most recent call last):
  File "/home/adam/Projects/tmp/poetry_bug/poetry_bug_1/setup.py", line 1, in <module>
    import setuptools_scm
ModuleNotFoundError: No module named 'setuptools_scm'

Output of `poetry install -vvv`

Using virtualenv: /home/adam/.cache/pypoetry/virtualenvs/poetry-bug-2-MKW0P_o3-py3.10
Updating dependencies
Resolving dependencies...
   1: fact: poetry-bug-2 is 0.1.0
   1: derived: poetry-bug-2
   1: Version solving took 7.074 seconds.
   1: Tried 1 solutions.
Stack trace:
1  ~/.local/lib/python3.10/site-packages/poetry/utils/env.py:1530 in run

1528│         try:

1529│             if input:

→ 1530│                 output = subprocess.run(

1531│                     cmd,

1532│                     stdout=subprocess.PIPE,
CalledProcessError
Command '['/tmp/tmpfrovmlq9/.venv/bin/python', '-']' returned non-zero exit status 1.
at /usr/lib/python3.10/subprocess.py:524 in run

520│             # We don't call process.wait() as .exit does that for us.

521│             raise

522│         retcode = process.poll()

523│         if check and retcode:

→  524│             raise CalledProcessError(retcode, process.args,

525│                                      output=stdout, stderr=stderr)

526│     return CompletedProcess(process.args, retcode, stdout, stderr)

527│

528│
The following error occurred when trying to handle this error:
Stack trace:
3  ~/.local/lib/python3.10/site-packages/poetry/inspection/info.py:601 in get_pep517_metadata

599│                 *PEP517_META_BUILD_DEPS,

600│             )

→ 601│             venv.run(

602│                 "python",

603│                 "-",
2  ~/.local/lib/python3.10/site-packages/poetry/utils/env.py:1500 in run

1498│     def run(self, bin: str, *args: str, **kwargs: Any) -> str | int:

1499│         cmd = self.get_command_from_bin(bin) + list(args)

→ 1500│         return self._run(cmd, **kwargs)

1501│

1502│     def run_pip(self, *args: str, **kwargs: Any) -> int | str:
1  ~/.local/lib/python3.10/site-packages/poetry/utils/env.py:1782 in _run

1780│     def _run(self, cmd: list[str], **kwargs: Any) -> int | str:

1781│         kwargs["env"] = self.get_temp_environ(environ=kwargs.get("env"))

→ 1782│         return super()._run(cmd, **kwargs)

1783│

1784│     def get_temp_environ(
EnvCommandError
Command ['/tmp/tmpfrovmlq9/.venv/bin/python', '-'] errored with the following return code 1
Output:

Traceback (most recent call last):

File "/tmp/tmpfrovmlq9/.venv/lib/python3.10/site-packages/build/init.py", line 494, in _handle_backend

yield

File "/tmp/tmpfrovmlq9/.venv/lib/python3.10/site-packages/build/init.py", line 486, in _call_backend

basename: str = callback(outdir, config_settings, **kwargs)

File "/tmp/tmpfrovmlq9/.venv/lib/python3.10/site-packages/pep517/wrappers.py", line 189, in prepare_metadata_for_build_wheel

return self._call_hook('prepare_metadata_for_build_wheel', {

File "/tmp/tmpfrovmlq9/.venv/lib/python3.10/site-packages/pep517/wrappers.py", line 309, in _call_hook

self._subprocess_runner(

File "/tmp/tmpfrovmlq9/.venv/lib/python3.10/site-packages/build/init.py", line 327, in _runner

self._hook_runner(cmd, cwd, extra_environ)

File "/tmp/tmpfrovmlq9/.venv/lib/python3.10/site-packages/pyproject_hooks/_impl.py", line 71, in quiet_subprocess_runner

check_output(cmd, cwd=cwd, env=env, stderr=STDOUT)

File "/usr/lib/python3.10/subprocess.py", line 420, in check_output

return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,

File "/usr/lib/python3.10/subprocess.py", line 524, in run

raise CalledProcessError(retcode, process.args,

subprocess.CalledProcessError: Command '['/tmp/build-env-hbzwyj_3/bin/python', '/tmp/tmpfrovmlq9/.venv/lib/python3.10/site-packages/pep517/in_process/_in_process.py', 'prepare_metadata_for_build_wheel', '/tmp/tmpslsegi50']' returned non-zero exit status 1.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):

File "", line 17, in 

File "/tmp/tmpfrovmlq9/.venv/lib/python3.10/site-packages/build/init.py", line 454, in metadata_path

metadata = self.prepare('wheel', output_directory)

File "/tmp/tmpfrovmlq9/.venv/lib/python3.10/site-packages/build/init.py", line 411, in prepare

return self._call_backend(

File "/tmp/tmpfrovmlq9/.venv/lib/python3.10/site-packages/build/init.py", line 485, in _call_backend

with self._handle_backend(hook_name):

File "/usr/lib/python3.10/contextlib.py", line 153, in exit

self.gen.throw(typ, value, traceback)

File "/tmp/tmpfrovmlq9/.venv/lib/python3.10/site-packages/build/init.py", line 502, in _handle_backend

raise BuildBackendException(  # noqa: B904 # use raise from

build.BuildBackendException: Backend subprocess exited when trying to invoke prepare_metadata_for_build_wheel
Input:

import build

import build.env

import pyproject_hooks
source = '/home/adam/Projects/tmp/poetry_bug/poetry_bug_1'

dest = '/tmp/tmpfrovmlq9/dist'
with build.env.IsolatedEnvBuilder() as env:

builder = build.ProjectBuilder(

srcdir=source,

scripts_dir=env.scripts_dir,

python_executable=env.executable,

runner=pyproject_hooks.quiet_subprocess_runner,

)

env.install(builder.build_system_requires)

env.install(builder.get_requires_for_build('wheel'))

builder.metadata_path(dest)
at ~/.local/lib/python3.10/site-packages/poetry/utils/env.py:1546 in run

1542│                 )

1543│             else:

1544│                 output = subprocess.check_output(cmd, stderr=stderr, env=env, **kwargs)

1545│         except CalledProcessError as e:

→ 1546│             raise EnvCommandError(e, input=input)

1547│

1548│         return decode(output)

1549│

1550│     def execute(self, bin: str, *args: str, **kwargs: Any) -> int:
The following error occurred when trying to handle this error:
Stack trace:
2  ~/.local/lib/python3.10/site-packages/poetry/utils/env.py:1544 in run

1542│                 )

1543│             else:

→ 1544│                 output = subprocess.check_output(cmd, stderr=stderr, env=env, **kwargs)

1545│         except CalledProcessError as e:

1546│             raise EnvCommandError(e, input=input)
1  /usr/lib/python3.10/subprocess.py:420 in check_output

418│         kwargs['input'] = empty

419│

→  420│     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,

421│                **kwargs).stdout

422│
CalledProcessError
Command '['/tmp/tmpfrovmlq9/.venv/bin/python', 'setup.py', 'egg_info']' returned non-zero exit status 1.
at /usr/lib/python3.10/subprocess.py:524 in run

520│             # We don't call process.wait() as .exit does that for us.

521│             raise

522│         retcode = process.poll()

523│         if check and retcode:

→  524│             raise CalledProcessError(retcode, process.args,

525│                                      output=stdout, stderr=stderr)

526│     return CompletedProcess(process.args, retcode, stdout, stderr)

527│

528│
The following error occurred when trying to handle this error:
Stack trace:
3  ~/.local/lib/python3.10/site-packages/poetry/inspection/info.py:622 in get_pep517_metadata

620│             os.chdir(path.as_posix())

621│             try:

→ 622│                 venv.run("python", "setup.py", "egg_info")

623│                 info = PackageInfo.from_metadata(path)

624│             except EnvCommandError as fbe:
2  ~/.local/lib/python3.10/site-packages/poetry/utils/env.py:1500 in run

1498│     def run(self, bin: str, *args: str, **kwargs: Any) -> str | int:

1499│         cmd = self.get_command_from_bin(bin) + list(args)

→ 1500│         return self._run(cmd, **kwargs)

1501│

1502│     def run_pip(self, *args: str, **kwargs: Any) -> int | str:
1  ~/.local/lib/python3.10/site-packages/poetry/utils/env.py:1782 in _run

1780│     def _run(self, cmd: list[str], **kwargs: Any) -> int | str:

1781│         kwargs["env"] = self.get_temp_environ(environ=kwargs.get("env"))

→ 1782│         return super()._run(cmd, **kwargs)

1783│

1784│     def get_temp_environ(
EnvCommandError
Command ['/tmp/tmpfrovmlq9/.venv/bin/python', 'setup.py', 'egg_info'] errored with the following return code 1
Output:

Traceback (most recent call last):

File "/home/adam/Projects/tmp/poetry_bug/poetry_bug_1/setup.py", line 1, in 

import setuptools_scm

ModuleNotFoundError: No module named 'setuptools_scm'
at ~/.local/lib/python3.10/site-packages/poetry/utils/env.py:1546 in run

1542│                 )

1543│             else:

1544│                 output = subprocess.check_output(cmd, stderr=stderr, env=env, **kwargs)

1545│         except CalledProcessError as e:

→ 1546│             raise EnvCommandError(e, input=input)

1547│

1548│         return decode(output)

1549│

1550│     def execute(self, bin: str, *args: str, **kwargs: Any) -> int:
The following error occurred when trying to handle this error:
Stack trace:
21  ~/.local/lib/python3.10/site-packages/cleo/application.py:327 in run

325│

326│             try:

→ 327│                 exit_code = self._run(io)

328│             except BrokenPipeError:

329│                 # If we are piped to another process, it may close early and send a
20  ~/.local/lib/python3.10/site-packages/poetry/console/application.py:190 in _run

188│         self._load_plugins(io)

189│

→ 190│         exit_code: int = super()._run(io)

191│         return exit_code

192│
19  ~/.local/lib/python3.10/site-packages/poethepoet/plugin.py:249 in _run

247│                 tokens.insert(task_name_index, "--")

248│

→ 249│             continue_run(self, io)

250│

251│         # Apply the patch
18  ~/.local/lib/python3.10/site-packages/cleo/application.py:431 in _run

429│             io.input.interactive(interactive)

430│

→ 431│         exit_code = self._run_command(command, io)

432│         self._running_command = None

433│
17  ~/.local/lib/python3.10/site-packages/cleo/application.py:473 in _run_command

471│

472│         if error is not None:

→ 473│             raise error

474│

475│         return terminate_event.exit_code
16  ~/.local/lib/python3.10/site-packages/cleo/application.py:457 in _run_command

455│

456│             if command_event.command_should_run():

→ 457│                 exit_code = command.run(io)

458│             else:

459│                 exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED
15  ~/.local/lib/python3.10/site-packages/cleo/commands/base_command.py:119 in run

117│         io.input.validate()

118│

→ 119│         status_code = self.execute(io)

120│

121│         if status_code is None:
14  ~/.local/lib/python3.10/site-packages/cleo/commands/command.py:62 in execute

60│

61│         try:

→  62│             return self.handle()

63│         except KeyboardInterrupt:

64│             return 1
13  ~/.local/lib/python3.10/site-packages/poetry/console/commands/install.py:156 in handle

154│         self.installer.verbose(self.io.is_verbose())

155│

→ 156│         return_code = self.installer.run()

157│

158│         if return_code != 0:
12  ~/.local/lib/python3.10/site-packages/poetry/installation/installer.py:116 in run

114│             self._execute_operations = False

115│

→ 116│         return self._do_install()

117│

118│     def dry_run(self, dry_run: bool = True) -> Installer:
11  ~/.local/lib/python3.10/site-packages/poetry/installation/installer.py:263 in _do_install

261│                 source_root=self._env.path.joinpath("src")

262│             ):

→ 263│                 ops = solver.solve(use_latest=self._whitelist).calculate_operations()

264│         else:

265│             self._io.write_line("Installing dependencies from lock file")
10  ~/.local/lib/python3.10/site-packages/poetry/puzzle/solver.py:74 in solve

72│         with self._progress(), self._provider.use_latest_for(use_latest or []):

73│             start = time.time()

→  74│             packages, depths = self._solve()

75│             end = time.time()

76│
9  ~/.local/lib/python3.10/site-packages/poetry/puzzle/solver.py:157 in _solve

155│

156│         try:

→ 157│             result = resolve_version(self._package, self._provider)

158│

159│             packages = result.packages
8  ~/.local/lib/python3.10/site-packages/poetry/mixology/init.py:18 in resolve_version

16│     solver = VersionSolver(root, provider)

17│

→  18│     return solver.solve()

19│
7  ~/.local/lib/python3.10/site-packages/poetry/mixology/version_solver.py:112 in solve

110│             while next is not None:

111│                 self._propagate(next)

→ 112│                 next = self._choose_package_version()

113│

114│             return self._result()
6  ~/.local/lib/python3.10/site-packages/poetry/mixology/version_solver.py:428 in _choose_package_version

426│             package = locked

427│

→ 428│         package = self._provider.complete_package(package)

429│

430│         conflict = False
5  ~/.local/lib/python3.10/site-packages/poetry/puzzle/provider.py:657 in complete_package

655│                     if locked is not None and locked.package.is_same_package_as(dep):

656│                         continue

→  657│                     self.search_for_direct_origin_dependency(dep)

658│         else:

659│             for dep in _dependencies:
4  ~/.local/lib/python3.10/site-packages/poetry/puzzle/provider.py:280 in search_for_direct_origin_dependency

278│         elif dependency.is_directory():

279│             dependency = cast("DirectoryDependency", dependency)

→  280│             package = self._search_for_directory(dependency)

281│

282│         elif dependency.is_url():
3  ~/.local/lib/python3.10/site-packages/poetry/puzzle/provider.py:426 in _search_for_directory

424│     def _search_for_directory(self, dependency: DirectoryDependency) -> Package:

425│         dependency.validate(raise_error=True)

→  426│         package = self.get_package_from_directory(dependency.full_path)

427│

428│         self.validate_package_for_dependency(dependency=dependency, package=package)
2  ~/.local/lib/python3.10/site-packages/poetry/puzzle/provider.py:439 in get_package_from_directory

437│     @classmethod

438│     def get_package_from_directory(cls, directory: Path) -> Package:

→  439│         return PackageInfo.from_directory(path=directory).to_package(root_dir=directory)

440│

441│     def _search_for_url(self, dependency: URLDependency) -> Package:
1  ~/.local/lib/python3.10/site-packages/poetry/inspection/info.py:499 in from_directory

497│                         info = cls.from_setup_files(path)

498│                     else:

→ 499│                         info = get_pep517_metadata(path)

500│                 except PackageInfoError:

501│                     if not info:
PackageInfoError
Unable to determine package info for path: /home/adam/Projects/tmp/poetry_bug/poetry_bug_1
Fallback egg_info generation failed.
Command ['/tmp/tmpfrovmlq9/.venv/bin/python', 'setup.py', 'egg_info'] errored with the following return code 1
Output:

Traceback (most recent call last):

File "/home/adam/Projects/tmp/poetry_bug/poetry_bug_1/setup.py", line 1, in 

import setuptools_scm

ModuleNotFoundError: No module named 'setuptools_scm'
at ~/.local/lib/python3.10/site-packages/poetry/inspection/info.py:625 in get_pep517_metadata

621│             try:

622│                 venv.run("python", "setup.py", "egg_info")

623│                 info = PackageInfo.from_metadata(path)

624│             except EnvCommandError as fbe:

→ 625│                 raise PackageInfoError(

626│                     path, "Fallback egg_info generation failed.", fbe

627│                 )

628│             finally:

629│                 os.chdir(cwd.as_posix())

[finswimmer]

Hello @adamgreig,

I'm able to reproduce it with the example repo you have prepared.

But this minimal example have two problems.

1. The pyproject.toml of poetry_bug_1 is invalid. The ^ in requires-python = "^3.8" is not valid here. You have to write it like this: requires-python = ">=3.8, <4"
2. poetry_bug_1 is not a valid package at all. It does not contain any data and the setup.py does not contain any instruction for the build.

Once I have fixed these two problems I cannot reproduce the issue any more.

I believe that there is nothing to fix on poetrys side here. Can you confirm this?

fin swimmer

[SolidTux]

Not OP, but I managed to still reproduce the problem, as long as there is a build.py file (see repo).

Running git clean -dfx; pushd poetry_bug_2; poetry lock; popd works fine, but git clean -dfx; pushd poetry_bug_3; poetry lock; popd results in the following error:

Unable to determine package info for path: ###/poetry_build_requires_bug/poetry_bug_1

Command ['/tmp/tmpfwl5v_j4/.venv/bin/python', '-'] errored with the following return code 1

Output:
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
ModuleNotFoundError: No module named 'build.env'; 'build' is not a package


Input:
import build
import build.env
import pyproject_hooks

source = '###/poetry_build_requires_bug/poetry_bug_1'
dest = '/tmp/tmpfwl5v_j4/dist'

with build.env.IsolatedEnvBuilder() as env:
    builder = build.ProjectBuilder(
        srcdir=source,
        scripts_dir=env.scripts_dir,
        python_executable=env.executable,
        runner=pyproject_hooks.quiet_subprocess_runner,
    )
    env.install(builder.build_system_requires)
    env.install(builder.get_requires_for_build('wheel'))
    builder.metadata_path(dest)

Fallback egg_info generation failed.

Command ['/tmp/tmpfwl5v_j4/.venv/bin/python', 'setup.py', 'egg_info'] errored with the following return code 1

Output:
Traceback (most recent call last):
  File "###/poetry_build_requires_bug/poetry_bug_1/setup.py", line 1, in <module>
    import setuptools_scm
ModuleNotFoundError: No module named 'setuptools_scm'

It does not matter whether I specify

[tool.poetry.build]
generate-setup-file = false
script = "build.py"

or

[tool.poetry.build]
generate-setup-file = true
script = "build.py"

or nothing in pyproject.toml.

[rashidnhm]

I have hit the exact same error @SolidTux outlines in his comment

Unable to determine package info for path: /home/user/.cache/pypoetry/virtualenvs/project-qxo8sVZj-py3.10/src/ldpc                                                                                             
                                                                                                                                                                                                                   
Command ['/tmp/tmp50jhlm2z/.venv/bin/python', '-'] errored with the following return code 1                                                                                                                        
                                                                                                                                                                                                                   
Output:                                                                                                                                                                                                            
Traceback (most recent call last):                                                                                                                                                                                 
  File "<stdin>", line 2, in <module>                                                                                                                                                                              
ModuleNotFoundError: No module named 'build.env'; 'build' is not a package                                                                                                                                         
                                                                                                                                                                                                                   
                                                                                                                                                                                                                   
Input:                                                                                                                                                                                                             
import build                                                                                                                                                                                                       
import build.env                                                                                                                                                                                                   
import pyproject_hooks                                                                                                                                                                                             
                                                                                                                                                                                                                   
source = '/home/user/.cache/pypoetry/virtualenvs/project-qxo8sVZj-py3.10/src/ldpc'                                                                                                                             
dest = '/tmp/tmp50jhlm2z/dist'                                                                                                                                                                                     
                                                                                                                                                                                                                   
with build.env.IsolatedEnvBuilder() as env:                                                                                                                                                                        
    builder = build.ProjectBuilder(                                                                                                                                                                                
        srcdir=source,                                                                                                                                                                                             
        scripts_dir=env.scripts_dir,                                                                                                                                                                               
        python_executable=env.executable,                                                                                                                                                                          
        runner=pyproject_hooks.quiet_subprocess_runner,                                                                                                                                                            
    )                                                                                                                                                                                                              
    env.install(builder.build_system_requires)                                                                                                                                                                     
    env.install(builder.get_requires_for_build('wheel'))                                                                                                                                                           
    builder.metadata_path(dest)                                                                                                                                                                                    
                                                                                                                                                                                                                   
Fallback egg_info generation failed.                                                                                                                                                                               
                                                                                                                                                                                                                   
Command ['/tmp/tmp50jhlm2z/.venv/bin/python', 'setup.py', 'egg_info'] errored with the following return code 1                                                                                                     
                                                                                                                                                                                                                   
Output:                                                                                                                                                                                                            
Traceback (most recent call last):                                                                                                                                                                                 
  File "/home/user/.cache/pypoetry/virtualenvs/project-qxo8sVZj-py3.10/src/ldpc/setup.py", line 2, in <module>                                                                                                 
    from Cython.Build import cythonize                                                                                                                                                                             
ModuleNotFoundError: No module named 'Cython' 

The ldpc package in this case, does list cython as a build requirement See Here

[dimbleby]

original reproduction was bogus, reproduction at #7702 (comment) has been removed, no reproduction in #7702 (comment)

it's entirely unclear what this issue is even about: since the original bug report was wrong and there are no repros since then it's hard to say whether any of these are the "same" or whether folk are just hijacking this issue.

suggest closing this out and people should raise new issues - with reproduction - for new problems.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.