Editable mode is gone since 1.1.0a2

[snejus]

• I am on the latest Poetry version.

  • Though this covers more than one release

• I have searched the issues of this repo and believe that this is not a duplicate.

• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Ubuntu 18.04.4 LTS x86_64

• Poetry version: 1.1.0a2-b2

• Link of a Gist with the contents of your pyproject.toml file:

Issue

I've been using 1.1.0a1 version since the beginning and having seen the recent pre-release updates I thought I'd try them out. Unfortunately, it doesn't seem that Poetry has been able to install the root project in editable mode since 1.1.0a2 pre-release. I've checked also checked a3 and b2 and found the same behaviour.

pip list output using 1.1.0a1, as expected:

requests-toolbelt     0.9.1
safety                1.9.0
my-package 4.4.0     /projects/my-package/src

and what I saw for the more recent pre-releases mentioned above:

requests-toolbelt     0.9.1
safety                1.9.0
my-package 4.4.0     /projects/my-package/.venv/lib/python3.6/site-packages

It's telling me that the editable installation is in the site-packages - by this logic every each of my dependencies is an editable one too 😅

Would be good to have sorted since it's something that's blocking me from updating. I'd be happy to help with this if you need extra hands.

See below for the pyproject.toml. The package metadata is altered a bit

[tool.poetry]
name = "my-package"
version = "4.4.0"
authors = ["Sarunas Nejus <[email protected]>"]
description = "My cool cool package"

readme = "README.rst"
repository = "https://github.com/my/package"

packages = [{ include = "mypackage", from = "src" }]
include = ["packagedata/**/*"]

[tool.poetry.urls]
Changelog = "https://github.com/my/package/CHANGELOG.md"

[[tool.poetry.source]]
name = "pripy"
url = "https://hidden-pypi.com"

[tool.poetry.dependencies]
python = "~3.6"

pydantic = "^1.5"
dataclasses = "^0.7"
dj-database-url = "^0.4"
psycopg2-binary = "^2.8"
python-dateutil = "*"
typing-extensions = "*"

[tool.poetry.dev-dependencies]
mypy = "^0.782"
flake8 = "^3.8"
safety = "^1.9"
pytest = "^5.4"
pylint = "^2.5"
pytest-cov = "^2.8"
pytest-randomly = "^3"


[tool.coverage.run]
branch = true
dynamic_context = "test_function"

[tool.coverage.report]
precision = 2
show_missing = true
exclude_lines = [
    "pragma: no cover",
    "if TYPE_CHECKING",
    "raise AssertionError",
    "raise NotImplementedError"
]

[tool.coverage.html]
show_contexts = true

[tool.pylint.MASTER]
class-rgx = "[A-Z_]+[a-zA-Z0-9]+$"
deprecated-modules = ["tkinter.tix", "formatter", "pycrypto", "optparse", "httpoxy", "xmlrpc",
                      "parser", "email", "pipes", "crypt", "spwd", "imp", "xml", "lxml"]
disable = [
    "no-member",  # buggy
    "bad-continuation",  # buggy
    "ungrouped-imports",  # incompatible with mypy
    "no-name-in-module",  # buggy
    "undefined-variable",  # covered by mypy
    "not-callable",  # covered by mypy
    "unused-import", # covered by flake8 which does it correctly
]
docstring-min-length = 10
good-names = "_,pk,pytest_plugins"
ignore-long-lines = 'noqa|^\s*(# )?<?https?://\S+>?$'
max-line-length = 90
max-nested-blocks = 3
msg-template="{path}:{module}:{line}: [{msg_id}({symbol}), {obj}] {msg}"
min-similarity-lines = 5
no-docstring-rgx = "^test_"
persistent = "no"

[tool.black]
line_length = 90

[tool.isort]
line_length = 90
multi_line_output = 3
include_trailing_comma = true

[build-system]
requires = ["poetry-core>=1.0.0a5"]
build-backend = "poetry.core.masonry.api"

[abn]

The short answer here is that this is expected and highlights the fragility of how editable installs are handled today. This is an expected side-effect of the transition to a more standard approach to handle editable installs in PEP 517 build backends as recommended in PEP 610. This does not have any functional change in behaviour, and it is important to note that curently there is no formal implemented standard around this.

Prior to #2400 (1.1.0.a2), poetry install would lead to the creation of *.egg-link files of the project being developed. The use of *.egg-link files has always been more of a hacky solution [1, 2] that setuptools and by extension pip has relied on, , together with the absense of *.dist-info directory, to "detect" editable packages. This problem of communicating metadata relating to an editable install will go away once the PEP 610 specification is fully implemented, in particular the following.

When url refers to a local directory, the dir_info key MUST be present as a dictionary with the following key:

editable (type: boolean): true if the distribution was installed in editable mode, false otherwise. If absent, default to false.
When url refers to a local directory, it MUST have the file sheme and be compliant with RFC 8089 [9]. In particular, the path component must be absolute. Symbolic links SHOULD be preserved when making relative paths absolute.

Additional References:

• https://discuss.python.org/t/pip-freeze-vcs-urls-and-pep-517-feat-editable-installs/1473/3

[abn]

Would be good to have sorted since it's something that's blocking me from updating. I'd be happy to help with this if you need extra hands.

@snejus would be great to get some more details on how this is blocking your upgrade.

[snejus]

Okay, this makes sense. Though considering that many people do rely on editable installs, I wouldn't say that releasing these changes is the best option (even if we're talking about pre-releases). I take my words back if (a) majority of things going on now depend on it being present or (b) it's about to be completed. For example, maybe it would be possible to isolate PEP 610 to a specific branch/letter of pre-releases (is that even a thing, really?), say x.x.xbx. At the same time, x.x.xax could include every update except PEP 610 work. Then people who don't care about editable installs can go ahead with b, while people that rely on them can stay with a and benefit from the rest of updates.

I'm assuming here though that there is less coupling between things than there may actually be. Also, I've never seen a setup like this - probably because now I realise that it does sound like it'd be a pain in the ass to manage this kind of thing.

@snejus would be great to get some more details on how this is blocking your upgrade.

For example, I often have djangos python manage.py runserver running locally to make use of its auto-reload capabilities.
Then, for some packages I run mypy on the package instead of paths. pylint also.

In general, I find having the most recent code available in all contexts (i.e. editable mode) more preferable than not.

[abn]

To be clear, in this context, functionality has not changed.

The issue reported here has to do with how pip detects the "editable" package's location. When you run poetry install the current project's included directories are written to <purelib>/python-<version>/site-packages/<package>.pth file as described here. This means that your package source is loaded from the your source tree. This means commands like poetry run python manage.py runserver will run your current version of the code.

In general, I find having the most recent code available in all contexts (i.e. editable mode) more preferable than not.

In development environments this may be true for most cases excluding isolated testing. This is not desireable nor recommended in managed environments.

[adamf]

This is also breaking our upgrade to 1.1.0 - we install a package in editable mode so we can, later in a docker build step, inject built protocol buffers into that package. Without editable mode this process breaks. Any ideas how to work around it in 1.1.0 series?

[finswimmer]

Hello @adamf,

the editable mode still works. The default behavior has changed. So if you define a path dependency, you have to set the develop key to true explicit:

my-package = {path = "../path/to/dependency", develop = true}

[dpompeu-xgeeks]

I've struggled with this: apparently, you cannot simply add the key develop = true: this will not update changes.

My solution was to do this:

1. remove the package
2. lock + install the changes
3. add the package directly to pyproject.toml with develop = true
4. lock + install the changes

I wish I could simply run $ poetry add ../my-lib --develop=true instead.

[nishantvarma]

@dpompeu-xgeeks, Removing a package will "forget" its dependencies, isn't that a problem? When you add again, you get the latest. Switching with minimal impacts will the best behaviour. Refer the linked discussion; --editable seems to be available in Poetry 1.2.

[dpompeu-xgeeks]

@dpompeu-xgeeks, Removing a package will "forget" its dependencies, isn't that a problem? When you add again, you get the latest. Switching with minimal impacts will the best behaviour.

That's a good point.

We got some changes in our poetry.lock file when we didn't want to: not sure if this was the reason , but it's possible. In the end, we pinned all library versions in pyproject.toml to guarantee there were no unexpected updates.

This works for us, but doesn't seem feasible for other use cases, eg a public library which should be more flexible with its dependencies versions.

[dpompeu-xgeeks]

Another workaround we used which worked more reliably, but with great effort:

1. develop changes in library
2. change the library's version to a temporary version, eg 0.5.1-experimental-d7ba
3. git tag 0.5.1-experimental-d7ba
4. git push --tag 0.5.1-experimental-d7ba
5. in the main application, use poetry add git+ssh://[email protected]/user/lib#0.5.1-experimental-d7ba
6. test the changes
7. repeat all previous steps if there was a bug

(We use git to host our internal libraries).

Being able to change existing dependency to editable mode would be much quicker.

[snejus]

@dpompeu-xgeeks have you tried specifying those dependencies as local paths?

I have been successfully using the following configuration:

[tool.poetry.dependencies]
python = ">=3.6.2,<4"

beets = { path = "../beets", develop = true, extras = ["chroma", "discogs", "fetchart", "lastgenre", "mpdstats", "web"] }
beetcamp = { path = "../beetcamp", develop = true }
rich_tables = { path = "../rich-tables", develop = true }
beets-importreplace = { git = "https://github.com/edgars-supe/beets-importreplace.git" }
discogs-client = "*"

As expected, poetry uses a .pth file for each of the editable dependencies:

$ print -l $VIRTUAL_ENV/lib/python3.10/site-packages/*.pth(:t)
beetcamp.pth
beets.pth
distutils-precedence.pth
rich_tables.pth
_virtualenv.pth

Edit:
A quick test to see whether a change in the dependency's source code is reflected in the main project:

$ $VIRTUAL_ENV/bin/beetcamp
usage: beetcamp [-h] [-a] [-l] [-t] [release_url | query]

Get bandcamp release metadata from the given <release-url> or perform bandcamp search with <query>. Anything that does not start
with https:// will be assumed to be a query. Search type flags: -a for albums, -l for labels and artists, -t for tracks. By
default, all types are searched.

positional arguments:
  release_url  Release URL, starting with https:// OR
  query        Search query

options:
  -h, --help   show this help message and exit
  -a, --album  Search albums
  -l, --label  Search labels and artists
  -t, --track  Search tracks
$ sed 's/Get bandcamp release metadata//' ../beetcamp/beetsplug/bandcamp/__init__.py -i
$ $VIRTUAL_ENV/bin/beetcamp
usage: beetcamp [-h] [-a] [-l] [-t] [release_url | query]

from the given <release-url> or perform bandcamp search with <query>. Anything that does not start with https:// will be assumed
to be a query. Search type flags: -a for albums, -l for labels and artists, -t for tracks. By default, all types are searched.

positional arguments:
  release_url  Release URL, starting with https:// OR
  query        Search query

options:
  -h, --help   show this help message and exit
  -a, --album  Search albums
  -l, --label  Search labels and artists
  -t, --track  Search tracks

As you can see, the sed command edited a source file of the dependency, and I didn't need to poetry install nor update to synchronise.

If this for some reason is not feasible for your application, I think pip version 22 (and higher) now supports editable installations using only pyproject.toml.

[nishantvarma]

@snejus , Are you talking about a scenario where there is a tag in your project and switch to git develop = true?

pendulum = "^2.1.2"
to
pendulum = { git = "https://github.com/sdispater/pendulum.git", branch = "2.1", develop = true }

@dpompeu-xgeeks,

Another possibility:

Start with pendulum = "^2.1.2", lock and install.
Update to the git with develop = true.
Do a poetry lock --no-update to update the changes.
Do an rm -rf .venv and poetry install. # Direct poetry install won't work for the same reason as above. This kind of a removal but preserves the lock.

The main problem is the nuking part; as long as it is there in ~/.cache/pypoetry/, it should be OK. If not, there could be some ways to extend this hack. Need to try Poetry 1.2 to see if the --editable option helps.

[snejus]

@nishantvarma, in the case above (specifying a git-based dependency) inclusion of develop = true indeed makes the source code editable, however, poetry downloads the repository during the install and keeps it in your virtual environment ($VIRTUAL_ENV/src/pendulum) which is not ideal.

Instead of that I provide a path to a local repository (e.g. beetcamp = { path = "../beetcamp", develop = true }). I do not need to tag or push the code to a remote repository in order to use it as a dependency in some other project which I am working on locally.

[dpompeu-xgeeks]

@dpompeu-xgeeks have you tried specifying those dependencies as local paths?

@snejus: My problem is when I change from git to local paths: poetry does not apply the changes.

I can reproduce this by creating a lib and app (using poetry 1.1.13):

# mylib
$ echo 'print("Hello World from git tag")' > mylib.py
$ git commit -m "some message"
$ git rev-parse HEAD
cad773cb633c3d8be5ddd16ef6767d5d0bc6e7cc
$ git push

# myapp
$ poetry add git+ssh://[email protected]:dpompeu-xgeeks/example-lib.git#cad773cb633c3d8be5ddd16ef6767d5d0bc6e7cc
$ cat myapp.py
import mylib.mylib
$ poetry run python myapp.py 
Hello World from git tag

Now let's say I want to make some changes and debug them locally before pushing:

#mylib
$ echo 'print("Hello World from path")' > mylib.py
$ vim pyproject.toml # bump version, not sure if needed

#myapp
$ vim pyproject.toml # mylib = {path = "../mylib/", develop = true }
$ poetry lock
$ poetry install
$ poetry run python myapp.py
Hello World from git tag #  <--- no changes!!!

My expectation is that poetry would use the local files.

Instead, I'm either removing the package before, or giving up and just pushing experimental versions to git, which is a slow process.

[dpompeu-xgeeks]

@dpompeu-xgeeks,

Another possibility:

Start with pendulum = "^2.1.2", lock and install. Update to the git with develop = true. Do a poetry lock --no-update to update the changes. Do an rm -rf .venv and poetry install. # Direct poetry install won't work for the same reason as above. This kind of a removal but preserves the lock.

@nishantvarma: Thanks for the tip, I can confirm this works! It does seem better than removing the packages.

However, I'd still hope this would work more simply: this way is prone to mistakes, eg forgetting to remove the package and then wondering why there are no changes. It has happened with different people in my project.

Reproducing with the example above:

$ rm -rf .venv/
$ poetry install
$ poetry run python myapp.py
Hello world from file

[snejus]

@dpompeu-xgeeks Could you confirm I understood it correctly:

1. Initially added the dependency through the git reference:
   $ poetry add git+ssh://[email protected]:dpompeu-xgeeks/example-lib.git#cad773cb633c3d8be5ddd16ef6767d5d0bc6e7cc

2. Later removed the reference to the repository and replaced it with
   mylib = {path = "../mylib/", develop = true }

Could you also check what output do you see after running the following?

poetry --version
poetry shell
grep . $VIRTUAL_ENV/lib/*/*/*.pth | grep -oP '\K/lib.*'
# one of the lines should say '/lib/python*/site-packages/mylib.pth:<absolute-path-to-../mylib>'

[dpompeu-xgeeks]

Thanks for the followup @snejus .

@dpompeu-xgeeks Could you confirm I understood it correctly:

1. Initially added the dependency through the git reference:
   `$ poetry add git+ssh://[email protected]:dpompeu-xgeeks/example-lib.git#cad773cb633c3d8be5ddd16ef6767d5d0bc6e7cc`

2. Later removed the reference to the repository and replaced it with
   `mylib = {path = "../mylib/", develop = true }`

Yes, exactly. However, I don't get the latest changes after step 2.

Could you also check what output do you see after running the following?

poetry --version
poetry shell
grep . $VIRTUAL_ENV/lib/*/*/*.pth | grep -oP '\K/lib.*'
# one of the lines should say '/lib/python*/site-packages/mylib.pth:<absolute-path-to-../mylib>'

Sure, here's the output:

$ poetry run python myapp.py
Hello World from git tag # <--- no changes
$ poetry --version
Poetry version 1.1.13
$ poetry shell
Virtual environment already activated: /home/dpompeu/dev/experiments/poetry-experiment/myapp/.venv
$ grep . $VIRTUAL_ENV/lib/*/*/*.pth | grep -oP '\K/lib.*'
/lib/python3.8/site-packages/_virtualenv.pth:import _virtualenv
/lib/python3.8/site-packages/distutils-precedence.pth:import os; var = 'SETUPTOOLS_USE_DISTUTILS'; enabled = os.environ.get(var, 'local') == 'local'; enabled and __import__('_distutils_hack').add_shim();
/lib/python3.8/site-packages/myapp.pth:/home/dpompeu/dev/experiments/poetry-experiment/myapp
/lib/python3.8/site-packages/mylib.pth:/home/dpompeu/dev/experiments/poetry-experiment/mylib

If I remove .venv and install again , I get the result I expected, same output from grep:

$ rm -rf .venv
$ poetry install
Creating virtualenv myapp in /home/dpompeu/dev/experiments/poetry-experiment/myapp/.venv
Installing dependencies from lock file
Warning: The lock file is not up to date with the latest changes in pyproject.toml. You may be getting outdated dependencies. Run update to update them.

Package operations: 1 install, 0 updates, 0 removals

  • Installing mylib (0.2.0 /home/dpompeu/dev/experiments/poetry-experiment/mylib)
$ poetry run python myapp.py
Hello world from file # <--- correct changes
$ grep . $VIRTUAL_ENV/lib/*/*/*.pth | grep -oP '\K/lib.*'
/lib/python3.8/site-packages/_virtualenv.pth:import _virtualenv
/lib/python3.8/site-packages/distutils-precedence.pth:import os; var = 'SETUPTOOLS_USE_DISTUTILS'; enabled = os.environ.get(var, 'local') == 'local'; enabled and __import__('_distutils_hack').add_shim();
/lib/python3.8/site-packages/myapp.pth:/home/dpompeu/dev/experiments/poetry-experiment/myapp
/lib/python3.8/site-packages/mylib.pth:/home/dpompeu/dev/experiments/poetry-experiment/mylib

[snejus]

I've tried doing the same in my machine and encountered your issue. After some investigation, it seems that poetry does not manage to uninstall the outdated (git) version of your dependency which then conflicts with your editable install later.

To fix this, before adding the path dependency, run both poetry remove mylib and pip uninstall mylib to cleanly get rid of it from your environment. You can double-check with poetry run python myapp.py - it should fail with ModuleNotFoundError: No module named 'mylib'.

Then, add the local path dependency and lock/install. It worked okay for me.

[dpompeu-xgeeks]

Thanks for the input. I prefer the workaround provided by @nishantvarma , which does preserves poetry.lock:

Start with pendulum = "^2.1.2", lock and install. Update to the git with develop = true. Do a poetry lock --no-update to update the changes. Do an rm -rf .venv and poetry install. # Direct poetry install won't work for the same reason as above. This kind of a removal but preserves the lock.

I'm glad there's a workaround but I don't think it's a good solution for the long term. In my team, there has been a lot of confusion about this, especially because poetry fails to update the dependency but writes successful messages anyway. It will be inevitable that someone will forget one of the steps and gets some unnecessary pain and loss of time.

I think this deserves its own issue. I can create it and link to this thread.

[dpompeu-xgeeks]

FYI this seems to be fixed by the recent release 1.2.0.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.