pdm for applications doesn't make their modules importable

[qris]

Describe the bug

I am very new to PDM, I like it but I'm really confused about the support for "application" projects. Perhaps I'm misunderstanding something, but if so then I think the docs could be improved to help others.

PDM supports developing applications as well as libraries, and some consider it better suited (and most useful) to applications. However for applications it does nothing to help Python to find an application's Python modules, which is especially problematic if the src layout is used, which seems to be recommended generally, and supported (and automatically detected) by PDM.

For example, if you create a structure like this:

pyproject.toml
pdm.lock
src
  bin
    run-myapp.py
  myapp
    __init__.py
    mymodule.py

Then pdm run python src/bin/run-myapp.py (or the equivalent user script) cannot import myapp.mymodule, which is where all the functionality is supposed to be (assuming that run-myapp.py is just a trivial stub).

This is because my application is not "installed" in the virtualenv, and nor is src added to PYTHONPATH, so Python just cannot find the modules anywhere.

Of course I could use relative imports everywhere instead, but I think that looks ugly, pollutes sys.modules and stack traces, makes it harder to refactor/move code, and breaks if the application is installed as a package (because the stub script moves to the bin or scripts directory, depending on the OS).

I looked at various options to work around this, including using sitecustomize.py to add src to sys.path, a PEP-582 __pypackages__ directory, and adding pth files, but the only one that I could make work was to install my app, either as a package:

pdm add .

Or in editable mode (which PDM only allows as a dev dependency):

pdm add --dev .

This seems to be required for an application to be usable (if it contains any modules at all, and doesn't hack about with sys.path), but I don't think it's actually documented anywhere. I don't know how it works, because applications have no buildsystem. Also, both methods have problems:

• I don't know how installing as a package works at all, because applications have no build-system (by default).
• Installing it as a package adds the project to its own dependencies in pyproject.toml which seems wrong (circular dependency).
• Installing it as a package means that it would need to be reinstalled after making any local changes (because a copy of the source is installed in the virtualenv, not an editable link).
• Installing it editable only works as a development dependency. According to the docs, all users should have to run pdm install with the --dev option to install it (even in CI and production deployments, or not use PDM at all), but in practice it always installs even without --dev for reasons unknown.

Perhaps I'm missing the point of application (non-library) projects, but it seems like they should always install the application itself (either editable if --dev is specified, or as a package if not), or that should be a configurable option. Otherwise they just seem to be crippled libraries.

To reproduce

Create a new project with this structure:

pyproject.toml
pdm.lock
src
  bin
    run-myapp.py
  myapp
    __init__.py
    mymodule.py

Have run-myapp.py import myapp.mymodule.

Run pdm run python src/bin/run-myapp.py

Expected Behavior

run-myapp.py successfully imports myapp.mymodule.

Environment Information

PDM version:
2.20.1
Python Interpreter:
C:\Users...\Documents....venv\Scripts\python.exe (3.12)
Project Root:
C:/Users/.../Documents/...
Local Packages:

{
"implementation_name": "cpython",
"implementation_version": "3.12.3",
"os_name": "nt",
"platform_machine": "AMD64",
"platform_release": "10",
"platform_system": "Windows",
"platform_version": "10.0.19045",
"python_full_version": "3.12.3",
"platform_python_implementation": "CPython",
"python_version": "3.12",
"sys_platform": "win32"
}

pdm -v output

Usage: pdm [-h] [-V] [-c CONFIG] [-v | -q] [--no-cache] [-I] [--pep582 [SHELL]] [-n] ...

____  ____  __  ___

/ __ / __ / |/ /
/ // / / / / /|/ /
/ / // / / / /
// /_____// //

Options:
-h, --help Show this help message and exit.
-V, --version Show the version and exit
-c CONFIG, --config CONFIG
Specify another config file path [env var: PDM_CONFIG_FILE]
-v, --verbose Use -v for detailed output and -vv for more detailed
-q, --quiet Suppress output
--no-cache Disable the cache for the current command. [env var: PDM_NO_CACHE]
-I, --ignore-python Ignore the Python path saved in .pdm-python. [env var: PDM_IGNORE_SAVED_PYTHON]
--pep582 [SHELL] Print the command line to be eval'd by the shell for PEP 582
-n, --non-interactive
Don't show interactive prompts but use defaults. [env var: PDM_NON_INTERACTIVE]

Commands:
add Add package(s) to pyproject.toml and install them
build Build artifacts for distribution
cache Control the caches of PDM
completion Generate completion scripts for the given shell
config Display the current configuration
export Export the locked packages set to other formats
fix Fix the project problems according to the latest version of PDM
import Import project metadata from other formats
info Show the project information
init Initialize a pyproject.toml for PDM. Built-in templates: - default: pdm init, A simple template with
a basic structure. - minimal: pdm init minimal, A minimal template with only pyproject.toml.
install Install dependencies from lock file
list List packages installed in the current working set
lock Resolve and lock dependencies
outdated Check for outdated packages and list the latest versions on indexes.
publish Build and publish the project to PyPI
python (py) Manage installed Python interpreters
remove Remove packages from pyproject.toml
run Run commands or scripts with local packages loaded
search Search for PyPI packages
self (plugin) Manage the PDM program itself (previously known as plugin)
show Show the package information
sync Synchronize the current working set with lock file
update Update package(s) in pyproject.toml
use Use the given python version or path as base interpreter. If not found, PDM will try to install one.
venv Virtualenv management

Additional Context

No response

Are you willing to submit a PR to fix this bug?

• Yes, I would like to submit a PR.

[o-moe]

Here's my take on that matter, maybe this helps you find a proper work-flow with pdm for your project:

For me, the difference between a library and an app is not about whether it is distributed/published but rather if it has a __main__. An application is meant to work stand-alone. A library needs another library or application to be used (I know there are hybrids but this is just for laying out my basic mental-model).

With this being said, in your example, you technically have an application within the bin folder and a library within the myapp folder - so you could also make use of a monorepo approach.

In general: if you do not want to publish your application, why not making use of pdm's distribution feature anyway? When you enable distribution, you get all that magic coming from pdm's buildsystem (if you choose to use it). That includes:

• automatically installing your application into the venv as an editable dev dependency (without configuring it)
• automatically setting PYTHONPATH
• building a wheel (that can be used for sharing/installing by pip's install --find-links)
• optionally using dynamic versioning
• optionally publish to pypi or any private index locally
• optionally using project scripts (that may substitute your application stub)
• ...

Even though you do not plan on publishing your application on pypi or any private index, enabling distribution makes the development process much easier imho.

Again, these are just thought that may help you to figure out your development process with pdm.

[qris]

@o-moe thanks for your suggestions. I was trying to follow the guidance in PDM's docs and that pointed me towards my code being an application (there was nothing about not being able to import from itself). If PDM agrees with your view of what an application is, then I think the docs need to be updated to reflect that. I wish I had chosen library in the first place, since applications don't seem to offer any advantages (that I can see), only disadvantages.

I could turn my project into a library by adding the [build-system] table in pdm.lock. Is that what you meant by "making use of pdm's distribution feature"? I will probably do that if there's no cleaner solution. I don't really want to have two pyproject.toml files, so I don't think I want a monorepo.

[o-moe]

I am not saying that the PDM's author's point of view regarding applications and libraries is incorrect. I don't even think there's a single correct way. Everybody has his/hers own view of what a library and what an application is. PDM chose to interpret it the way it did and chose some default configuration for it.

The beauty of standards is that it gives you the liberty and flexibility to configure your project as your projects needs it. If pdm calls it "library configuration“ but you use it for your "application" - why not.

Regarding "enabling distribution" I mean adding this to your pyproject.toml (I think that's the default - but not 100% sure) and with it, you can use a build-backend, pdm installs your library/application into the venv etc - basically everything that is defined in this guide (note that this guide talks about the "project" - not about a library or application since it's not relevant).

[tool.pdm]
distribution = true

And if you do not want to use it, adding your sources as an editable dev dependency is another way of installing your sources into the venv alongside all the other dependencies which your project may have.

[qris]

@o-moe thanks again. It's fine for PDM authors to have their own definition of "application", but the one given in the docs doesn't seem to match that definition, or at least it led me to make the wrong choice for my project.

It seems like there's no advantage to making an application project, so library should just be the default, there should be no need to ask for it in pdm init, and application mode could be relegated to a footnote in the docs that if you want to use --no-self every time you run pdm install, you can just set distribution = false in pyproject.toml. At which point the term "application mode" is redundant and can be forgotten.