Clarify how development dependencies are installed

[cjolowicz]

There's sometimes confusion why session.install does not get the development dependencies installed. The documentation could be clearer here.

[cjolowicz]

See #389

[matteosantama]

Is it possible to install the dev dependencies as well? Or is it a design choice to disallow it?

[matteosantama]

The reason I ask is I have a session that runs mypy. With mypy>=0.900, you must explicitly install type-stubs as dependencies. The easiest way to manage this (I think) would be to add each of the type-stubs as a dev dependency of my library, and then install all the dev dependencies in the nox session.

[cjolowicz]

You can do this:

session.install("mypy", "types-foobar")

Or this:

session.run_always("poetry", "install", external=True)

session.install has no concept of development dependencies, because it is a front-end to pip install.

nox-poetry does not change that. It just makes it respect the lockfile. So if you add mypy and types-foobar as development dependencies, the first example will install them at the versions recorded in the lockfile.

This is by design. Nox sessions are for isolated, repeatable, dedicated checks. That's what makes them different from a development environment like the one Poetry sets up for you, where you do want everything installed for convenience (and an editable install of your own package, so you see changes immediately).

[cjolowicz]

To expand on this, session.install(".") installs your package and its dependencies, but not development dependencies. That's because the development dependencies are not a part of your package metadata. They're a Poetry-specific extension.

So you need to install these dependencies explicitly: session.install("mypy", ...).

nox-poetry is very much about development dependencies. It makes sure that you can manage these tools in Poetry, and the Nox sessions will use the expected version.

If you really want to install every development dependency in a Nox session, use Poetry as an external command, as shown above in the second example.

[matteosantama]

Got it, thanks!

[kentmartin73]

A little hacky, but I think I have a better way. nox-poetry creates requirements.txt at a predictable location which contains all the dev dependencies, so rather than calling poetry externally, we can pip -r install that

session.install("-r", f"{session.virtualenv.location}/tmp/requirements.txt")

I think there is support in Poetry for having other dependency groups beyond just dev and "normal" so this would probably include those too - haven't tried

[cjolowicz]

Help me understand, why don't you want to invoke Poetry directly to install development dependencies?

The location of the generated requirements file is an implementation detail. For example, this location would change when #350 is implemented.

[kentmartin73]

That's good to know - so not an ideal solution. Perhaps nox-poetry could provide session.requirements_file to allow it to keep working?

There's no hugely compelling reasoning. This makes pip install everything, instead of a mix of pip when you use session.install and poetry when you use session.run_always("poetry", blah)

Why does nox provide session.install at all when you could just use session.run_always("pip", "install", "foo", external=True)? I guess by not calling pip externally, nox can make changes to the install logic if they want to without noxfiles being changed - although calling session.install with --some-pip-flag rather limits this possibility.

What might be nice as an alternative/addition is if nox-poetry provided a session.poetry_install in a similar vein to session.conda_install which nox provides, then, we get to choose which install tool to use to install (install for pip, conda_install for Conda and poetry_install for Poetry)

[cjolowicz]

That's good to know - so not an ideal solution. Perhaps nox-poetry could provide session.requirements_file to allow it to keep working?

You can use session.poetry.export_requirements.

While this function is (historically) part of the public API, let me explain why you may not want to use it (and why we should really deprecate this function). The requirements file returned by this function is no longer one-to-one what you get from poetry export. These days we need to postprocess the file to get valid constraints for pip's new dependency resolver.

There's no hugely compelling reasoning. This makes pip install everything, instead of a mix of pip when you use session.install and poetry when you use session.run_always("poetry", blah)

I understand how this seems preferable, but the transformation from Poetry's dependency information to something that pip understands is (alas) not lossless. For example, we had to drop hashes from the export, and that's not the only thing. So if you can use Poetry directly, you should.

The main use case where you can't (yet) use Poetry directly is when you want specific development dependencies installed. With Nox you generally create a lot of environments for different Python versions. Being able to install only the dependencies you actually need is nice both for efficiency and isolation. And that's essentially why nox-poetry exists.

Maybe this will change when Poetry's dependency groups become a thing (#481).

Why does nox provide session.install at all when you could just use session.run_always("pip", "install", "foo", external=True)? I guess by not calling pip externally, nox can make changes to the install logic if they want to without noxfiles being changed - although calling session.install with --some-pip-flag rather limits this possibility.

I'm not sure I understand what you're getting at here. Yes, the session.install method is a thin layer over pip install. But it uses the environment's pip, not an external command. Apart from being a convenient shortcut, marking the command as an installation allows Nox to handle it differently from other commands, e.g. with respect to logging, or for options like --no-install and --install-only.

What might be nice as an alternative/addition is if nox-poetry provided a session.poetry_install in a similar vein to session.conda_install which nox provides, then, we get to choose which install tool to use to install (install for pip, conda_install for Conda and poetry_install for Poetry)

But nox-poetry does not use Poetry as an installer. When you invoke session.install in a nox-poetry session, you still install with pip. The main difference to a plain nox session is that pip's installation is constrained by dependency information from Poetry's lockfile.

The session.poetry_install you propose is very different from session.install and session.conda_install, because it does not let you install individual packages.

If you want to use both installation methods (plain nox vs nox-poetry), that is already possible:

import nox
import nox_poetry

@nox.session
def test(session):
    session.install("foo")  # use plain pip
    nox_poetry.Session(session).install("foo")  # use pip with constraints

[kentmartin73]

Thanks for the clear answer, if nothing else, I've learned something today. I guess the fundamental difference is we want to use an "external" Poetry, but "per-venv" pip

It still makes some sense to me to have an abstraction wrapped around Poetry with the units Poetry understands (dependencies, dev-dependencies, groups(?) etc), in the same way that a wrapper around pip understands package names, but I guess it boils down to little more than a trivial convenience wrapper if I follow your comments above correctly. It would enable handling, to take an absurd case, things like, if the command poetry install changed to poetry install2 in nox-poetry without any changes to noxfiles. It is a struggle to see the point though, and, doesn't do anything to further the core purpose of this project - I withdraw the suggestion