Allow users to install a project's dependencies, without the project itself

[charliermarsh]

Not sure if we want this -- it's like Poetry's --no-root. Let's wait and see if we can find motivating use-cases for it. Poetry also has this concept of "operating modes" which toggle this automatically.

[charliermarsh]

(Not decided, filed to revisit later.)

[zanieb]

Lots of prior chatter over in python-poetry/poetry#800

[kdebrab]

Maybe see also pypa/pip#11440 for potential use cases.

[charliermarsh]

(Interestingly, we actually kind of already support what's described in that issue because we allow pip install -r pyproject.toml. But that's separate from this issue.)

[kdebrab]

(Interestingly, we actually kind of already support what's described in that issue because we allow pip install -r pyproject.toml. But that's separate from this issue.)

That doesn't support the [extras] syntax (e.g. pip install -r pyproject.toml[doc])?

Personally, I'd love to see support for the syntax suggested here:

pip install --only-deps  .[doc]  # project.dependencies and project.optional-dependencies.doc

[EDIT]
Just discovered that pip install -r pyproject.toml --extra doc gives the functionality I was looking for. That's really great!

[firtrfl]

This works great pip install -r pyproject.toml :) It would be still great to add a flag and also add it to the pyproject.toml

[tool.uv]
package-mode = false

So it would also work with uv add <package>. Many things that I work on are not packages, but it would still be great to use uv to manage dependencies.

PS: uv is such an amazing tool - thanks for building it ❤️

[kdeldycke]

I have a use case for that: my blog, which is actually Markdown content only, and is built by Pelican, a pure Python static website generator.

In that case the package-mode = false from Poetry was handy.

In the mean time, I hacked the process and added a dummy package at the root of my repository:

$ ls -lah ./blog
Permissions Size User Date Modified    Name
drwxr-xr-x     - kde  2024-07-13 14:12  .
drwxr-xr-x@    - kde  2024-07-13 14:17  ..
.rw-r--r--     0 kde  2024-07-13 14:11  __init__.py

And added the following directive in pyproject.toml:

[tool.setuptools.packages.find]
include = ["blog"]

[Vigilans]

My use case: ansible project with bundled ansible and other related packages.

For ansible project, the majority of code are written in yaml, by providing pyproject.toml we can build a self-contained ansible environment in project's .venv folder, with guaranteed ansible modules versions.

Besides, non-python-package programs like sshpass can be distributed into .venv/bin, so by source .venv/bin/activate ansible will be able to use functionally like remote host with password.

[msw-kialo]

We use --no-root to better caching of container image layers.

The first layer just installs the project dependencies; the second adds the projects itself.
So if dependencies don't change, the layer can be reused.

We hope to write something like:

WORKDIR /app
ADD pyproject.toml uv.lock /app
RUN uv sync --no-dev --no-root
ADD src /app
RUN uv sync --no-dev

[sbidoul]

In the above scenario of a layered Dockerfile to leverage caching of infrequently changing dependencies, it is important that there is a way to install from uv.lock without involving pyproject.toml. That is because when pyproject.toml gets involved, it may in turn require the presence of the whole project (because of an in-tree build backend, dynamic dependencies, etc).

At first sight, uv.lock contains everything that is needed to do this, except for an indication about the top level package(s).

IMHO this is a critical feature for adoption of uv.lock, as without it the space and time efficiency of dockerfile builds will be severely impacted, compared to the traditional approach of installing requirements.txt in a layer, then the app in another.

[charliermarsh]

I agree that something like this is necessary.

[adriangb]

I'll chime in that --no-root is not enough. You probably also want a --no-directory or --no-local, maybe just generalize to a source selector or something.

Motivation: https://python-poetry.org/docs/faq#poetry-busts-my-docker-cache-because-it-requires-me-to-copy-my-source-files-in-before-installing-3rd-party-dependencies

In particular the idea is that if you have a monorepo you can still copy in your top-level lockfile and not have to copy in anything (not even the pyproject.toml) from your sub-projects/packages.

[charliermarsh]

Yeah 100% agree that you should be able to sync from just the lockfile. The lockfile itself is actually designed to support that (it doesn’t rely on pyproject.toml), but we need to expose the right options to users…

[charliermarsh]

I can see a few approaches here (all fairly easy to implement, mostly about getting the API right)...

• --no-project (or --no-workspace) along with a separate --no-local flag (to ignore path dependencies that aren't part of the workspace)
• --only-remote (the inverse)
• Something like --seed or another semantic term that indicates why this is useful

[hynek]

In case you’re looking for opinions, I like “seed”. The others are rather generic and I’d have to guess what they exactly mean.

[charliermarsh]

(Gonna see if we can come up with an API and ship this ~today.)

[charliermarsh]

Opinions very welcome.

[adriangb]

I think --no-project is pretty clear but don't care too much as well as it's documented.

[baggiponte]

Would --no-self and --only-self be more intuitive?

[sbidoul]

--seed {package-name}[extra,...], because you need to tell it what top level package's dependencies you want to install?

[zanieb]

Some thoughts....

I think extras would be covered by the normal --extra and --all-extras options.

I worry about --no-project and --no-workspace because those have a different meaning elsewhere; which is, don't discover a project or workspace entirely. We still should perform normal project or workspace discovery, e.g., to find the uv.lock file if you're in a subdirectory for some reason.

I think we might want to use --without- and --only- prefixes, to match the our --with options.

The difficulty is in choosing how to describe what we're selecting. There are a few things we want to be able exclude:

• The current project
• A workspace member
• A path dependency

Unfortunately --without-editables doesn't work because a path dependency can be either editable or not editable. We also want to be able to toggle whether or not the current project and workspace members are installed as editable in the future.

All of these are "local" dependencies, as in they're on the local file system, so --without-local might make sense. Does this include the current project though? Do we need an inverse flag, like --only-remote? Is that easier to reason about?

It feels like we need a separate flag for excluding just the current project, like --without-self or --without-root (I think --without-project isn't an option because it overlaps too much with --no-project). Presumably we'd pair this with an inverse flag like --only-self or --only-root.

Do we need the ability to exclude all local dependencies except the current project? Or can it be done in two steps:

uv sync --without-local
uv sync --inexact --only-self

What's the use-case for that?

[charliermarsh]

I think the only use-case for the Docker caching situation is "exclude all locals".

I think excluding the current project (but not other workspace members) is independently useful, like for "virtual projects." We could choose not to solve that here, but we should probably have a design that won't cause us problems in the future if we pursue it.

[hynek]

Your example doesn't include cache busting by copying the uv.lock file into the build container which is kind the whole point? You want the layer to be rebuilt if the lock file changes. but keep it around otherwise.

I suspect your example is incomplete on purpose to just sketch out the idea, but I think it would be good to complete it, because those COPYs and cds are kinda elemental to the whole thing.

I for one could live with explicitly naming the package I want to sync later (i.e. uv sync --skip-package="my-app") but I'm not 100% sure we're on the same page.

[paveldikov]

My use case here is very similar to the ones listed here, but with a twist. We do not copy the workspace into the build context, rather, we mount a wheel that was already built in a previous CI stage.

This reduces duplication, both in terms of stages (build-backend only invoked once), as well as storage (if I COPY src and then proceed to uv sync that, I'd have an src tree left over in my resulting image, taking space and causing confusion).

Most importantly, by using a wheel and not a raw workspace, this also lets me have consistent builds between different targets (docker, pypi, rpm/deb/nix, etc etc)

Being able to uv sync without the root package will help to an extent, but I am not sure if it is a globally-optimal solution. I'd end up with:

RUN uv sync --frozen --no-locals
RUN uv pip install /mnt/my-actual-application.whl       # This part is lockfile agnostic; no guarantee that things won't drift!

For context, right now we do:

RUN uv pip install /mnt/my-actual-application.whl -c /mnt/constraints.txt

Which works exactly as intended -- but it's not a proper lockfile (and being able to slice it into layers is a very nice thing to be able to do)

[mitsuhiko]

Your example doesn't include cache busting by copying the uv.lock file into the build container which is kind the whole point? You want the layer to be rebuilt if the lock file changes. but keep it around otherwise.

Yes, the lock file needs to be copied first, but only prior to the first partial sync and if it's not there, uv sync --partial would fail.

My point is mostly, without going too much into the implementation, that a partial sync would be an explicit, multi-stage approach precisely to safely make a multi-layer caching solution work. The benefit for the user is that it can be guided without the user having too many ways to unintentionally fuck it up or changes in uv breaking layer caching accidentally.

I for one could live with explicitly naming the package I want to sync later (i.e. uv sync --skip-package="my-app") but I'm not 100% sure we're on the same page.

My reasoning with explicit skipping here is not so much that this is how it should be (eg: this is how you name what has not been synced) but that it's a very intentional process and you don't accidentally start interacting with a partial venv and then end up surprised about random failures later down the line.

[charliermarsh]

I think it would be pretty reasonable to use --skip-package rather than --no-locals. It would enable strictly more use-cases. The downside is that you need to include the package name (of course) and that, if you have a large workspace, it's a little tedious to enumerate them all. But it might be a better, more flexible solution than --no-locals.

[sbidoul]

Does the lock file include dev dependencies? In that case it would be very tedious if we don't want these in the docker image.

How about --dependencies-of package ?

[charliermarsh]

Yeah it does include dev dependencies. But you can omit those with --no-dev.

[sbidoul]

Ah, right I now see the lock file knows the distinction between regular and dev dependencies 👍

[zanieb]

Might be worth considering #4422 too when designing a --skip-package interface.

[bluss]

What stands out to me is that uv sync is not a command designed for application install or deployment. It sets up the venv and workspace (for development/test?) and all the defaults point towards that use case (?)

[charliermarsh]

Today we discussed doing something like:

• --no-install-root (skip root package)
• --no-install-workspace (skip any workspace dependencies, including the root)
• --no-install-paths (skip any path dependencies)
• --no-install-package [package] (skip a specific package)

So most usages would just be --no-install-root or --no-install-workspace.

We may even want to have a dedicated separate command for this with different defaults (per the above).

[zanieb]

This will be available when we release (soon, I'm sure). Let me know if you encounter any problems!

[zanieb]

This is available now in 0.3.3

[justenstall]

Is --no-install-paths still planned?

[zanieb]

@justenstall we're waiting to see how necessary it is since we have --no-install-package and the amount of options could get excessive. We can track in #6695

[ddorian]

Can --no-install-project be set in config?

[zanieb]

No, it's not intended for long-term usage. You're probably looking for #6585 instead (which will release today).