default pyproject.toml from poetry init -n improperly handles src layout; reports installation success even though it cannot actually be imported

[ckp95]

• Poetry version: 1.4.0
• Python version:

Poetry
Version: 1.4.0
Python:  3.10.9

Virtualenv
Python:         3.9.11

• OS version and name: 5.13.19-2-MANJARO
• pyproject.toml (see below)

• I am on the latest stable Poetry version, installed using a recommended method.
• I have searched the issues of this repo and believe that this is not a duplicate.
• I have consulted the FAQ and blog for any relevant entries or release notes.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

Trying to make simple hello-word package in src layout. The default pyproject.toml created with poetry init -n does not let me install it properly.

To begin with, my file structure is like this:

$ tree
.
├── README.md
└── src
    └── fancy_project
        ├── __init__.py
        └── something.py

3 directories, 3 files

The __init__.py file is blank. The something.py file contains this:

def hello():
    print("this is a fancy project!")

When I run poetry init -n, it generates the following (sanitized) pyproject.toml.

[tool.poetry]
name = "fancy-project"
version = "0.1.0"
description = ""
authors = ["aaaa <[email protected]>"]
readme = "README.md"
packages = [{include = "fancy_project"}]

[tool.poetry.dependencies]
python = "^3.9"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

When I run poetry install, I get this error:

$ poetry install
Updating dependencies
Resolving dependencies... (0.1s)

Writing lock file

/tmp/fancy-project/fancy_project does not contain any element

When I amend pyproject.toml to change the include = "fancy_project" to include = "src", the poetry install command seems to succeed:

$ poetry install
Installing dependencies from lock file

Installing the current project: fancy-project (0.1.0)

However, the package is not actually importable:

$ poetry run python -c 'from fancy_project.something import hello; hello()'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'fancy_project'

I tried changing the include line to say include = "src/fancy_project". It similarly reports a successful install, but the import fails just the same.

I tried removing the packages line from pyproject.toml entirely. This time it works:

$ poetry install
Installing dependencies from lock file

Installing the current project: fancy-project (0.1.0)
$ poetry run python -c 'from fancy_project.something import hello; hello()'
this is a fancy project!

But I don't understand why. I don't remember ever having to muck around with the packages line; everything "just worked" the last time I tried this with a src layout.

I believe there is at least one bug here:

• the default poetry init -n should make a pyproject.toml that can work with src layout
• poetry install should not say that an installation succeeded when it actually didn't
• the docs for packages don't clear up any of this confusion
• the "does not contain any element" error message does not make any sense

(PS: I ran rm -rf .venv before every poetry install command, to start from a clean slate each time)

EDIT:

bash script to precisely reproduce the problem:

#! /usr/bin/bash
set -euo pipefail

cd /tmp
rm -rf fancy-project
mkdir fancy-project
cd fancy-project

touch README.md
mkdir -p src/fancy_project
touch src/fancy_project/__init__.py
echo 'def hello():' >> src/fancy_project/something.py
echo '    print("this is a fancy project!")' >> src/fancy_project/something.py

poetry init -n

poetry install || true

sed -i 's/include = "fancy_project"/include = "src"/g' pyproject.toml
rm -rf .venv
poetry install
poetry run python -c 'from fancy_project.something import hello; hello()' || true

sed -i 's/include = "src"/include = "src\/fancy_project"/g' pyproject.toml
rm -rf .venv
poetry install
poetry run python -c 'from fancy_project.something import hello; hello()' || true

sed -i 's/^packages =.*//g' pyproject.toml
rm -rf .venv
poetry install
poetry run python -c 'from fancy_project.something import hello; hello()'

[dimbleby]

packages = [
    { include = "poetry", from = "src" }
]

per poetry's own usage

It's impossible for poetry init -n to produce a pyproject.toml that works for both src and not-src layouts: so it has to choose one. I don't see that as a bug.

I expect docs improvements explaining this would be welcome, please contribute!

[ckp95]

Okay, was that changed recently? I must have done this sequence of commands dozens of times and I never ran into this problem before.

Also I think the "does not contain any element" error message would be clearer as "directory does not exist"? since it's looking for a nonexistent /tmp/fancy_project/fancy_project.

[dimbleby]

not so far as I know

[ckp95]

Okay I'm not crazy, I tested it and the following works on Poetry version 1.1.15, like how I remember:

#! /usr/bin/bash
set -euo pipefail

cd /tmp
rm -rf fancy-project
mkdir fancy-project
cd fancy-project

touch README.md
mkdir -p src/fancy_project
touch src/fancy_project/__init__.py
echo 'def hello():' >> src/fancy_project/something.py
echo '    print("this is a fancy project!")' >> src/fancy_project/something.py

poetry init -n

poetry install
poetry run python -c 'from fancy_project.something import hello; hello()'

The culprit seems to be that 1.1.15 doesn't populate the packages line in pyproject.toml when you run poetry init -n. It just makes this:

[tool.poetry]
name = "fancy-project"
version = "0.1.0"
description = ""
authors = ["aaaa <[email protected]>"]

[tool.poetry.dependencies]
python = "^3.10"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

I looked in the release notes but I can't figure out why this was changed. The default config used to be able to figure out src layouts but now it can't.

[dimbleby]

it's because you're using a name with a dash in it. You can probably make it skip the explicit package by making this comparison be against canonicalize_name(include)

edit: or possibly it would be better the other way round, and the comparison should be against module_name(self._project).

again, contributions welcome

[ckp95]

Yes you're right, it works when I change everything to fancyproject.

I'll see if I can figure out a PR for it.

[dimaqq]

Thank you for this issue.

I've stumbled across the same error, and sure it is confusing!

In my case, my some-project was initialised from a "misnamed" directory hacking-some-project with the latter recorded in pyproject.toml and of course I've missed that.

[dimbleby]

you - and all readers - are invited to submit an MR with a fix as outlined at #7609 (comment)

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.