poetry fails building flat directory hierarchy

[saver-r]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• Window 10:
• Poetry 1.0.9:

Issue

Im trying to make poetry work on a flat hierarchy, i.e. on a directory tree that does not contain a package subdirectory. I looked for duplicates and found #92 (Poetry build can't find source files) but #92 refers to an alternative src/-subdirectory while we strive for a flat directory structure (no subdirectory).

test_flat/
├── __init__.py
│   └── foo/
│       └── bar.py
└── pyproject.toml

It is intended to be imported like this:

import test_flat

test_flat.foo.bar.print_result()

Setup

mkdir -p test_flat/foo
cd test_flat
cat <<"EOF" > pyproject.toml
[tool.poetry]
name = "test_flat"
version = "0.1.0"
description = "bla"
license = "Proprietary"
authors = ["nobody <nobody@invalid>"]

packages = [
  { include = "test_flat", from = ".." }, 
]

[tool.poetry.dependencies]
python = "^3.6"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry>=1.0.9"]
build-backend = "poetry.masonry.api"
EOF
cat <<"EOF" > __init__.py
EOF
cat <<"EOF" > foo/bar.py
def print_result():
  print("foo-bar")
EOF

Build it

poetry build -vvv

Expected result

Should create a dist/test_flat-0.1.0.tar.gz and a wheel file and should report no errors.

Actual result

Using virtualenv: %USERPROFILE%\.conda\envs\test_flat
Building test_flat (0.1.0)
 - Building sdist
 - Adding: ..\test_flat\__init__.py
 - Adding: ..\test_flat\dist\test_flat-0.1.0.tar.gz
 - Adding: ..\test_flat\foo\bar.py
 - Adding: ..\test_flat\pyproject.toml
 - Adding: pyproject.toml
 - Built test_flat-0.1.0.tar.gz


[AssertionError]
['test_flat', 'test_flat-0.1.0']

Traceback (most recent call last):
  File "%USERPROFILE%\.conda\envs\test_flat\lib\site-packages\clikit\console_application.py", line 131, in run
    status_code = command.handle(parsed_args, io)
  File "%USERPROFILE%\.conda\envs\test_flat\lib\site-packages\clikit\api\command\command.py", line 120, in handle
    status_code = self._do_handle(args, io)
  File "%USERPROFILE%\.conda\envs\test_flat\lib\site-packages\clikit\api\command\command.py", line 171, in _do_handle
    return getattr(handler, handler_method)(args, io, self)
  File "%USERPROFILE%\.conda\envs\test_flat\lib\site-packages\cleo\commands\command.py", line 92, in wrap_handle
    return self.handle()
  File "%USERPROFILE%\.conda\envs\test_flat\lib\site-packages\poetry\console\commands\build.py", line 30, in handle
    builder.build(fmt)
  File "%USERPROFILE%\.conda\envs\test_flat\lib\site-packages\poetry\masonry\builder.py", line 21, in build
    return builder.build()
  File "%USERPROFILE%\.conda\envs\test_flat\lib\site-packages\poetry\masonry\builders\complete.py", line 53, in build
    with self.unpacked_tarball(sdist_file) as tmpdir:
  File "%USERPROFILE%\.conda\envs\test_flat\lib\contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "%USERPROFILE%\.conda\envs\test_flat\lib\site-packages\poetry\masonry\builders\complete.py", line 71, in unpacked_tarball
    assert len(files) == 1, files

Creates garbled tar.gz:

tar -tzf dist/test_flat-0.1.0.tar.gz
tar: Removing leading `test_flat-0.1.0/../' from member names
test_flat-0.1.0/../test_flat/__init__.py
test_flat-0.1.0/../test_flat/foo/bar.py
test_flat-0.1.0/../test_flat/pyproject.toml
test_flat-0.1.0/pyproject.toml
test_flat-0.1.0/setup.py
test_flat-0.1.0/PKG-INFO

Fails at creating wheel-file.

[saver-r]

Rebuilt with poetry 1.1.0a1

poetry -V
Poetry version 1.1.0a1

Building with poetry 1.1.0a1 yields a different error. It looks like poetry does not correctly handle ..-indirection.

poetry build -vvv
Using virtualenv: %USERPROFILE%\.conda\envs\test_flat
Building test_flat (0.1.0)
 - Building sdist
 - Adding: pyproject.toml
 - Built test_flat-0.1.0.tar.gz

  ValueError

  %USERPROFILE%\AppData\Local\Temp\tmpm29lt396\test_flat-0.1.0\..\test_flat does not contain any element

  at %USERPROFILE%\.conda\envs\test_flat\lib\site-packages\poetry\core\masonry\utils\package_include.py:54 in check_elements
      50|
      51|     def check_elements(self):  # type: () -> PackageInclude
      52|         if not self._elements:
      53|             raise ValueError(
    > 54|                 "{} does not contain any element".format(self._base / self._include)
      55|             )
      56|
      57|         root = self._elements[0]
      58|         if len(self._elements) > 1:

  Stack trace:

   1  %USERPROFILE%\.conda\envs\test_flat\lib\site-packages\poetry\core\masonry\utils\package_include.py:15 in __init__
       13|
       14|         super(PackageInclude, self).__init__(base, include, formats=formats)
     > 15|         self.check_elements()
       16|
       17|     @property

   2  %USERPROFILE%\.conda\envs\test_flat\lib\site-packages\poetry\core\masonry\utils\module.py:72 in __init__
        70|                     package["include"],
        71|                     formats=formats,
     >  72|                     source=package.get("from"),
        73|                 )
        74|             )

   3  %USERPROFILE%\.conda\envs\test_flat\lib\site-packages\poetry\core\masonry\builders\builder.py:82 in __init__
        80|             self._path.as_posix(),
        81|             packages=packages,
     >  82|             includes=includes,
        83|         )
        84|

   4  %USERPROFILE%\.conda\envs\test_flat\lib\site-packages\poetry\core\masonry\builders\wheel.py:44 in __init__
        42|
        43|     def __init__(self, poetry, target_dir=None, original=None):
     >  44|         super(WheelBuilder, self).__init__(poetry)
        45|
        46|         self._records = []

   5  %USERPROFILE%\.conda\envs\test_flat\lib\site-packages\poetry\core\masonry\builders\wheel.py:54 in make_in
        52|     @classmethod
        53|     def make_in(cls, poetry, directory=None, original=None):
     >  54|         wb = WheelBuilder(poetry, target_dir=directory, original=original)
        55|         wb.build()
        56|

   6  %USERPROFILE%\.conda\envs\test_flat\lib\site-packages\poetry\core\masonry\builders\complete.py:46 in build
       44|             with self.unpacked_tarball(sdist_file) as tmpdir:
       45|                 WheelBuilder.make_in(
     > 46|                     Factory().create_poetry(tmpdir), dist_dir, original=self._poetry
       47|                 )
       48|

   7  %USERPROFILE%\.conda\envs\test_flat\lib\site-packages\poetry\core\masonry\builder.py:19 in build
       17|         builder = self._FORMATS[fmt](self._poetry)
       18|
     > 19|         return builder.build()
       20|

   8  %USERPROFILE%\.conda\envs\test_flat\lib\site-packages\poetry\console\commands\build.py:35 in handle
       33|
       34|         builder = Builder(self.poetry)
     > 35|         builder.build(fmt)
       36|

   9  %USERPROFILE%\.conda\envs\test_flat\lib\site-packages\cleo\commands\command.py:92 in wrap_handle
        90|         self._command = command
        91|
     >  92|         return self.handle()
        93|
        94|     def handle(self):  # type: () -> Optional[int]

  10  %USERPROFILE%\.conda\envs\test_flat\lib\site-packages\clikit\api\command\command.py:171 in _do_handle
       169|         handler_method = self._config.handler_method
       170|
     > 171|         return getattr(handler, handler_method)(args, io, self)
       172|
       173|     def __repr__(self):  # type: () -> str

  11  %USERPROFILE%\.conda\envs\test_flat\lib\site-packages\clikit\api\command\command.py:120 in handle
       118|     def handle(self, args, io):  # type: (Args, IO) -> int
       119|         try:
     > 120|             status_code = self._do_handle(args, io)
       121|         except KeyboardInterrupt:
       122|             if io.is_debug():

  12  %USERPROFILE%\.conda\envs\test_flat\lib\site-packages\clikit\console_application.py:131 in run
       129|             parsed_args = resolved_command.args
       130|
     > 131|             status_code = command.handle(parsed_args, io)
       132|         except KeyboardInterrupt:
       133|             status_code = 1

[finswimmer]

Hello @saver-r,

may I ask you why you want to have this folder structure? It is intended that the pyproject.toml lives outside the package itself.

With a folder structure like this:

test-flat
├── pyproject.toml
└── test_flat
    ├── __init__.py
    └── foo
        └── bar.py

with this in the __init__.py:

import test_flat.foo.bar

and a default pyproject.toml:

[tool.poetry]
name = "test-flat"
version = "0.1.0"
description = ""
authors = ["finswimmer <[email protected]>"]

[tool.poetry.dependencies]
python = "^3.8"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry>=1.0"]
build-backend = "poetry.masonry.api"

you can run the code snippet you've shown.

fin swimmer

[saver-r]

Thanks for you question. Let me elaborate on the rationale.

Overview

This module is part of a large multi-language project whose directory hierarchy is preset.

large_project/
├── src/
│   └── main/
│       ├── python/
│       │   ├── module_1/
│       │   │   ├── __init__.py
│       │   │   └── ...
│       │   ├── module_2/
│       │   │   ├── __init__.py
│       │   │   └── ...
│       │   ...
│       │   └── module_n/
│       │       ├── __init__.py
│       │       └── ...
│       ├── java/
│       └── .../
└── ...

large_project consists of many small python modules denoted here as module_1, module_2, etc. In order to access the python modules from within large_project, the PYTHON_PATH will be set to large_project/src/main/python.

Then we attempted to replace setup.py hell with declarative pyproject.toml files and poetry as the build system.

Goal

Each module_n is to be packaged as a separate module and to be uploaded individually to a private pypi-server such that other projects can perform hassle-free imports of modules without resorting to hackery like git-submodule or extending PYTHON_PATH to checkouts of large_project.

Impediments

I know perfectly well the recommended module layout you described. Using this layout, poetry works as intended. However, having to alter the directory layout in large_project will break existing workflows that emerged a long time before a python build system beyond setup.py was considered.

For intra-project access the modules module_1 to module_n should still be imported from the working directory and not from package versions from the pypi-server.

Current workflow

0. Hack locally
1. Build locally via IDE
2. Test locally

Only one PYTHON_PATH pointing to large_project/src/main/python needed. This has to be configured once, is well supported by IDEs and does not need to be touched any more during the whole existence of large_project.

Workflow changes on directory structure change

Pretended we changed the directory structure to the poetry-suggested default:

large_project/
├── src/
│   └── main/
│       ├── python/
│       │   ├── module_1/
│       │   │   ├── pyproject.toml
│       │   │   └── module_1/
│       │   │       ├── __init__.py
│       │   │       └── ...
│       │   ├── module_2/
│       │   │   ├── pyproject.toml
│       │   │   └── module_2/
│       │   │       ├── __init__.py
│       │   │       └── ...
│       │   ...
│       │   └── module_n/
│       │       ├── pyproject.toml
│       │       └── module_n/
│       │           ├── __init__.py
│       │           └── ...
│       ├── java/
│       └── .../
└── ...

This will inflict one of two changes to the workflow of which both are non favourable.

Workflow change 1: Use pypi-modules

0. Hack locally.
1. Invent some temporary version (e.g. append commit id to version-tag)
2. Build locally via poetry. poetry will upload all modules to private pypi-server.
3. For each module_n update dependencies
4. Test locally

This is an undue complication of our local hack-build-test cycle not well supported by IDEs and therefore not acceptable. Additionally it litters the private pypi-repository with transient development versions of packages that are of no use except for an individual developer's hack-build-test cycle.

Workflow change 2: Keep amending PYTHON_PATH

PYTHON_PATH has to be preset like this: large_project/src/main/python/module_1:large_project/src/main/python/module_1:...:large_project/src/main/python/module_n

0. Introduce new module locally
1. Amend PYTHON_PATH by adding path to new module
2. Build locally
3. Test locally

If no module is added, workflow remains the same as initially depicted.

One might naively think, there's no problem in a change of workflow if the change is only rarely to be regarded.

However, this is actually the biggest impediment. If the developer is used to the simple workflow, and only on rare occations, he has to use another workflow, guess which workflow the developer will forget to apply.

Now you will think: "Bah! Amending the PYTHON_PATH only takes one minute!" Experience tells that if you think some action takes one minute, it actually takes five minutes to deal with dependent side-actions not duely considered before.

To make matters worse, amending PYTHON_PATH not only strikes the developer adding the module. No, as soon as the new module is pushed, every other developer working on large_project now has to amend his PYTHON_PATH. So multiply five minutes with the count of affected developers, and probably even more as half of the devs will ask back the original developer what the heck he did.

We make software for a living and strive to keep development workflows lean and efficient. Therefore, we cannot introduce exceptions to the current workflow.

Wrap up

poetry is a very promising tool we'd really like to replace our setup.py hell with. From my point of view, it looks like poetry already provides the basis to support the current workflow of using a flat hierarchy.

poetry supports

• package include specifying a different base directory to package from,
• exclude to exclude certain build related files in the flat hierarchy, e.g. pyproject.toml.

What's missing?

• Proper resolution of relative paths containing .. indirections.

E.g. poetry could be able support a layout like this:

├── src/extra_package/
└── test_flat/
    ├── __init__.py
    │   └── foo/
    │       └── bar.py
    └── pyproject.toml

With pyproject.toml containing:

[tool.poetry]
name = "test_flat"
version = "0.1.0"
description = "bla"
license = "Proprietary"
authors = ["nobody <nobody@invalid>"]

packages = [
  { include = "test_flat", from = ".." }, 
  { include = "extra_package", from = "../src" }, 
]

exclude = [ "test_flat/pyproject.toml" ]

[tool.poetry.dependencies]
python = "^3.6"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry>=1.0.9"]
build-backend = "poetry.masonry.api"

Up to here, poetry already supports specifying sources and excludes like denoted above.

The resulting tar.gz should have a directory layout like this:

test_flat-0.1.0/test_flat/__init__.py
test_flat-0.1.0/test_flat/foo/bar.py
test_flat-0.1.0/extra_package/__init__.py
test_flat-0.1.0/pyproject.toml
test_flat-0.1.0/setup.py
test_flat-0.1.0/PKG-INFO

This is where poetry currently fails, but it looks like it could be easily solved by properly resolving indirect paths.

[dmvianna]

Same problem here. I want to use a dependency that is in

<root>
├── <other languages>...
└── python/
    ├── my_module
    │   └── my_submodule.py
    └── setup.cfg
    └── setup.py

of a remote git repository. And I don't control that project. As it stands, poetry won't let me

• specify a path inside the repository where the python module is; or
• find it algorithmically.

[neersighted]

@dmvianna your issue is unrelated and will be solved by #5811. As to the original ask here, it's actually relative path includes. I couldn't find a more general issue, but #4583 is a better blanket for this feature anyway as the semantics of what you want to achieve here is a bit blurry on interaction with other features/aspects of Poetry.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.