editable mode is not supported

[jaraco]

Recently, tests that were previously passing have started failing, possibly due to pypa/pip#6331. The error message is:

Obtaining file:///home/travis/build/jaraco/pip-run
ERROR: Error installing 'file:///home/travis/build/jaraco/pip-run': editable mode is not supported for pyproject.toml-style projects. This project is being processed as pyproject.toml-style because it has a pyproject.toml file with a "build-backend" key in the "build_system" value. See PEP 517 for the relevant specification.
Traceback (most recent call last):
  File "pip-10-install.py", line 19, in <module>
    __name__ == '__main__' and main()
  File "pip-10-install.py", line 16, in main
    'python -m pip install') + sys.argv[1:])
  File "/opt/python/2.7.15/lib/python2.7/subprocess.py", line 190, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['python', '-m', 'pip', 'install', '--exists-action', 'w', '-e', '/home/travis/build/jaraco/pip-run[testing]']' returned non-zero exit status 1

I've been (internally) bemoaning the fact that the 'editable' use-case is poorly-supported (and relies on deprecated functionality in setuptools such as easy_install and setup_requires to function). But I'd also been assuming that the tool chains would retain fallback compatibility, such as editable installs, for projects that support them (such as this pip-run project).

I'm on the hook to do more investigation here, but I wanted to report the issue thusfar.

[jaraco]

I do want to ask - are there plans to support editable installs in a PEP 517 world? Or are editable installs deprecated and the build tool must be run on every edit to convey changes?

[gaborbernat]

Broken due to pypa/pip#6434

There's some discussion to have some support, but no agreement yet.

[pfmoore]

Editable installs are waiting on another round of standards work to define a PEP 517-style interface for them. There's nothing blocking that, it was dropped from PEP 517 simply because of a lack of time. If someone were to champion a proposal for an editable install standard, it should be possible to get this under way again fairly easily. (Not that I'm guaranteeing that achieving consensus would be easy ;-))

[jaraco]

I did confirm that the pip-10-install script is not implicated - the tests still fail after removing it.

[pganssle]

@jaraco We're seeing similar problems in the pyo3 project, and we're definitely not intending to use any sort of editable install. @gaborbernat Does tox use editable installs under the hood for some reason?

[brainwane]

People discussed the future of editable installs during the packaging minisummit at PyCon North America in May 2019 and I'm moving the notes from that discussion here:

https://discuss.python.org/t/pip-19-1-and-installing-in-editable-mode-with-pyproject-toml/1553 & https://discuss.python.org/t/specification-of-editable-installation/1564/34

• PEP 517, standardized.

• We have a mechanism to specify in pyproject.toml what the backend is that is to actually build the package

  • This grows out of the prior standard mechanism, of setup.py/setuptools

• Originally, editable installs were too complicated and out of scope

  • This is becoming a major blocker for using pep 517

Open Questions:

• What is the responsibility of the backend
• What is the responsibility of the front end

Proposal/Suggestion:

• Frontend says 'give editable' Declares its needs
• Backend gives a dist, like a wheel, and provide where the files/list to install
• The front end will decide how to expose this
  • Tbd. pth, something else?
• Argument made (P Ganssle?) @pganssle that the B.E. shouldn't have to know about the surrounding system

Options from Nick Coghlan @ncoghlan

• .pth files are one implementation; symlinks/hardlinks are another
  • Windows symlinking continues to improve; as older Win software ages out, the cross-platform viability of symlinking becomes more attractive
• Give a full list of files and it is up to the front end to decide what to expose. Even if that means exposing more than a full import lib
• Backend says 'these are the directories I need)
  • Paul doesn't like this one.

What responsibility should the front end take on?

• • Conda may want to do things different than pip.
  • Same applies for instance, for AWS lambda
• How do we get the editable install to play nice?

Metadata is also imported. Setup.py uses egg info and this lacks many of the structures we want

• • What are the extras, files, etc
  • Whatever is designed needs to be able to have this metadata.
  • Metadata format. (generate_metadata_for_build_wheel)?

We cannot require front ends to respect excludes.

• • Today it doesn't for pip + setup.py (pip install -e .)
  • We can provide the info to respect this but we can't enforce this behaviour
  • If the F.E. was to use symlinks it would respect it. Today pip uses .pth files.

We cannot require front ends to respect updates to say, an `.so` file for a cffi module.

• • Frontend's implementation will determine this.
  • Having the backend provide a list of files provides the best chance of this working as it would allow front ends to use symlinks which would allow all of the updates to be reflected. (nick coghlan)
  • Query to the backend? API?
  • External service/daemon to watch key files and trigger builds when they change?
  • Can be designed later.

Actions:

• Would need a P.o.C. from pip + wheel + setuptools + pep517 (.pth-file approach ok for this P.o.C.)
  • Paul Ganssle for SetupTools
  • Dan Ryan can likely help @techalchemy
• Draft PEP

end of meeting notes

And more recently people have been discussing the next steps on Discourse.

[brainwane]

One more note: @ofek mentioned in a comment to those notes, regarding Windows symlinking:

I'm unsure of the current semantics, but the last time I tested this you needed to be a privileged user

[ofek]

Oh, it seems the docs now say so too https://docs.python.org/3/library/os.html#os.symlink

[ncoghlan]

Some useful discussion regarding Windows symlinks at https://security.stackexchange.com/questions/10194/why-do-you-have-to-be-an-admin-to-create-a-symlink-in-windows

I must have seen the announcement of the more fine-grained permissions structure a few years ago, but not looked into the details (i.e. an admin can configure their particular system to let unprivileged users create symlinks, but that still isn't, and likely will never be, the default behaviour)

[jaraco]

I had heard that more recent versions of Windows support symlinks without the special privileges (confirmed here), but it seems this more lenient behavior is still hamstrung by requiring both the caller to pass a special value and the user to have configured the system for development mode, meaning it's basically non-viable until those changes trickle out.

[pradyunsg]

requiring both the caller to pass a special value and the user to have configured the system for development mode

Yep. The documentation for the caller's special flag for this functionality states:

Developer Mode must first be enabled on the machine before this option will function.

---

@ncoghlan It seems that "Developement Mode" is a per-user flag (ref) so we can in theory ask users to flip this flag for us to enable symlinks for their system.

While it's not the default behavior, we can detect this situation, ask the user to flip a flag and things work from then on -- which is a decent experience. :)

[st-pasha]

From https://docs.microsoft.com/en-us/windows/uwp/get-started/enable-your-device-for-development#accessing-settings-for-developers:

Note
Enabling Developer mode requires administrator access. If your device is owned by an organization, this option might be disabled.

So there will be at least some users who will not be able to turn this option on.

[takluyver]

I'm taking another look at this, as I was reminded recently how much I dislike setuptools' editable install mechanism.

PEP 517 already includes a hook to prepare the .dist-info folder, which for Flit I take to include entry_points.txt.

I propose two additional optional hooks

• get_requires_for_build_editable - like the existing get_requires_* hooks.
• get_source_module_paths, which should return one or more paths to modules (.py files, directories containing __init__.py, etc.), to be made available on sys.path by some method (symlinking, pth files, PYTHONPATH). This would also be allowed to raise UnsupportedOperation (already part of the spec) if the backend refuses to do an editable install of that module, e.g. if the backend only supports editable installs when there's no compiled code.

When asked for an editable install, e.g. pip install -e ., the frontend would:

1. Install the static buildsystem dependencies from pyproject.toml in a build environment.
2. Call get_requires_for_build_editable. If the backend doesn't support it, bail out.
3. Install any requirements from step 2 in the build environment.
4. Call the new get_source_module_paths hook. If the backend doesn't support it, bail out.
5. Create a temporary folder and call prepare_metadata_for_build_wheel to put the metadata there.
6. Install any necessary dependencies from step 5 to the target environment.
7. Clean up any other version of the same distribution in the install target.
8. By whatever means is preferred, put the .dist-info folder from step 5 and the modules from step 4 on the import path, and create scripts on PATH from entry points.

I think this is flexible enough to allow both persistent editable installs (like symlinking), and session editable installs (e.g. launch a shell with a modified environment referring to tmp folders).

It's probably not perfect. But I think we've concluded that editable installs of this nature can never be perfect, because of the way they mix copying information and referencing information. But they're still handy, especially for expert users who understand what changes still need a reinstall.

The other possible approach is a 'watcher' mechanism, where something spots when the files are changed and quickly rebuilds/reinstalls the package. That is in many ways more powerful, because in principle any change can take effect automatically, and I think we should be open to adding support for that in the future. But it also seems much more complicated to do in general, so unless someone has a great idea, I think we should focus on the simpler mechanism for now.

[pganssle]

@takluyver That sounds a lot like the proposal we decided upon at PyCon and mentioned in the thread on discourse.

Honestly I am fairly sure that the only reasonable choice is for the backend to prepare a "virtual wheel" that contains references to the locations of the files that would be installed if you were to install them. Everything else is details.

I don't think there is much if any controversy here, it's just that no one has created a proof of concept with setuptools yet, and there's not much point trying to figure out the details before we have a working understanding of the kinds of problems we're going to run into.

[takluyver]

Yup, sorry. I hadn't read enough of that thread to see it.

[sbidoul]

As an intermediate step would it make sense to think about standardizing metadata for editable installs?

The goal would be that tools could discover that a distribution has been installed in editable (or should we say development?) mode and make some sense of it. I'm thinking about use cases such as pip list or pip freeze that need only one piece of additional metadata to make sense of something installed with e.g. flit install --symlink: namely the project location.

The spec could start small, by saying an "editable" installation must have regular .dist-info metadata, plus a new metadata file (develop.json?) that specifies the local directory where the project resides (i.e. the place where pyproject.toml can be found).

[pganssle]

@sbidoul I really don't see that any intermediate step is necessary or wise. If setuptools doesn't support a mechanism for PEP 517 editable installs, then for all practical purposes the mechanism is not useful. Standardizing metadata (or any of this) before we have a working implementation is putting the horse before the cart, in my opinion.

I think we have a pretty good "big picture" idea of what this should look like: backend produces a a pseudo-wheel consisting of wheel metadata plus a mapping between the locations of the files as they should exist in the package and the locations that they exist on disk, front-end uses whatever mechanism it deems proper to expose those files to Python in such a way that they will update as the files change on disk.

We need a proof of concept of that for setuptools and pip. A rough proof of concept for pip is likely relatively easy. I have taken a rough crack at a proof of concept for setuptools and it was a bigger job than I've had time for since then because of the way distutils marshals the files to be packaged (it does so incrementally, rather than first creating a manifest of files to be copied and then copying them). I'm happy to see anyone take a crack at that, but I can pretty much guarantee you that completing that task is the major bottleneck.

One might suggest building such a facility into flit or something similar for the proof of concept, but there are two problems with this: 1. pretty much everything except setuptools delegates the "hard cases" to setuptools, meaning that we won't get a good sense of the edge cases that need to be included in the spec (e.g. generated files, compiled artifacts) and 2. even if we succeed, there's no point in accepting a PEP that can't or won't be implemented in setuptools, so we're again hitting the setuptools bottleneck. The only plausible benefit I could see for implementing a PoC using flit or similar would be to allow someone to build a pip proof of concept in parallel to the setuptools one, but the outputs are simple enough that pip can mock them up easily enough for development purposes if that's a problem.

[sbidoul]

To clarify I'm not talking about progressing the PEP 517 interface for editable installs.
I'm talking about specifying what minimal metadata are necessary to be recorded in the database of installed distribution for editable installs.

This is independent of what the PEP 517 interface will look like in the end, and is useful today when we manually invoke the backend a tool such as flit to perform the installation.

[pganssle]

To clarify I'm not talking about progressing the PEP 517 interface for editable installs.
I'm talking about specifying what minimal metadata are necessary to be recorded in the database of installed distribution for editable installs.

I'm not sure it is independent of this, but if it is then it's not really an "intermediate step", and probably should have its own thread. That said, in your original post you write:

The spec could start small, by saying an "editable" installation must have regular .dist-info metadata, plus a new metadata file (develop.json?) that specifies the local directory where the project resides (i.e. the place where pyproject.toml can be found).

I'm not clear on what purpose this metadata is serving, but it doesn't seem like it's terribly useful information in my concept of what PEP 517 editable installs will look like, which is why I'm skeptical that this can be done in parallel to the development of a spec for editable installs.

If you do think it's orthogonal to the question of editable installs for pyproject.toml installations (e.g. PEP 517), I recommend a separate thread to discuss the merits, either here or on discuss.python.org.

[sbidoul]

what purpose this metadata is serving

When one has done setup.py develop, pip list can show an additional column with the project location, because it has special knowledge of .egg-link and such.

That project location information is important for users inspecting what is installed.

When you do flit install --symlink pip can't show the project location because the information is not available. This metadata would solve that.

When we will do pip install --editable in PEP 517 mode, that metadata will be necessary too, for the same reason.

[ncoghlan]

@sbidoul Editable installs have to be able to achieve two things:

• the Python import system has to be able to find the modules to import
• Python packaging tools have to be able to find the installed distribution metadata

The available mechanisms for the first part are either OS level (i.e. symlinks) or Python level (i.e. *.pth files - even *.egg-link files ultimately rely on this via some arcane trickery in setuptools). There's a hard constraint on this aspect, which is "Must not require any changes to the Python import system".

The second is less complicated, since the distribution metadata can just go in the normal location. There's no real need for any "this is an editable install" metadata (it might be nice to have some for informational purposes, but it's not a blocker to "editable install support")

[sbidoul]

There's a hard constraint on this aspect, which is "Must not require any changes to the Python import system".

Totally agree. I'm not questioning that. I'm using the symlink option of flit install as nothing more than an example of another tool that does some sort of editable install today.

The second is less complicated, since the distribution metadata can just go in the normal location.

Agreed too.

There's no real need for any "this is an editable install" metadata (it might be nice to have some for informational purposes, but it's not a blocker to "editable install support")

I think there is a need, beyond informational purposes. For example to enable pip freeze to do something sensible with editable installs: if pip freeze ignores the editable characteristic, it produces an incorrect output (package==version instead of, say, a VCS URL, or some sort of warning that an editable install is not a reproducible requirement).

[ncoghlan]

OK, I think I follow now: the key missing installation metadata is "How to pin this specific version" when "package_name == full_version_number" won't do the right thing.

That wouldn't just be beneficial for editable installs, it would help with VCS installs and other cases where using just the version number won't give the right result.

[merwok]

There's no real need for any "this is an editable install" metadata

One need I’ve had recently was related to hot-reload of web app: the reload tool watched files in my project directory, ignored files in site-packages (as an optimization), and so would not monitor the modules made available via flit install --symlink (more details in Pylons/hupper#61).

[sbidoul]

OK, I think I follow now: the key missing installation metadata is "How to pin this specific version" when "package_name == full_version_number" won't do the right thing.

@ncoghlan it is one use case, yes. Note that for VCS URLs and direct URL references in general, there is the draft PEP 610, which is currently stuck for lack of a core dev sponsor. My first impression is that metadata for editable instalis is different enough to warrant a separate spec (if only because the "location" of an editable install is always a local directory).

[ncoghlan]

@sbidoul I replied over on the PEP 610 thread, as I believe the local directory handling needs some adjustment anyway (does it really make sense to hash the whole directory tree?), and that creates an opportunity to store an "editable or not?" flag.

[pradyunsg]

Closing this out, since PEP 660 has happened.

[sbidoul]

🎆