uv pip compile drops mention of extras, totally losing dependency annotations via constraints

[funkybob]

Using : uv-0.2.10-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Environment: python:3.11-slim docker container

I recently switched from pip-compile to uv and am generally pleased, however today I noticed (after discovering it doesn't write the output by default) that when a package requirement has extras, these aren't written out.

e.g. django-storages[s3] is written out as simply django-storages

django-storages==1.14.2
    # via -r requirements.in

At the first level, this has no direct impact; All the requirements are still commented as being required by this package:

boto3==1.34.15
    # via
    #   -r requirements.in
    #   django-storages

Howerver, my -dev.in file which has a -c constraint on the main requirements.txt no longer carries over these comments, as it can no longer see the information -- the requirements.txt doesn't have django-storages[s3], only django-storages

django-storages==1.14.2
    # via -r requirements.txt

[charliermarsh]

You can pass --no-strip-extras to retain these.

[charliermarsh]

This shouldn't matter though -- extras just add additional dependencies, and they're all written out in the output requirements anyway. I believe pip-compile plans to make this the default too in the next release.

[funkybob]

Firstly, thanks for the very quick response!

You can pass --no-strip-extras to retain these.

OK, so another unexpected difference from pip-tools behavior. Got it.

This shouldn't matter though -- extras just add additional dependencies, and they're all written out in the output requirements anyway. I believe pip-compile plans to make this the default too in the next release.

Whilst I recognise it makes no difference to the packages that eventually get installed, it matters because I, the developer, lose the information that helps me track dependencies.

Otherwise, why emit those comments in the first place?

[charliermarsh]

Ah, I see, you want django-storages[s3] in the comment. I don't think we include the extra in the comment regardless.

[charliermarsh]

(I thought you were referring to the emitted dependency.)

[charliermarsh]

Or am I again misinterpreting? Because pip-compile doesn't include the extras in the comments either.

[funkybob]

No, to clarify I want my requirements-dev.txt to include in the comments that one of the reasons boto3 was included was django-storages.

Adding --no-strip-extras has achieved this.

[charliermarsh]

Ahh, ok, I see. Glad it's working! And sorry for the confusion.

[funkybob]

If one doesn't exist already, I might contribute a docs update to include some tips on differences when moving from pip-tools.

[dperetti]

Amazing, I was experimenting the same kind of confusion at the exact same time as @funkybob, as I was installing django-anymail[mailgun]. I came here for clarification and boom, this post!

That said, I noticed something that should probably be addressed:

If I install an extra that does not exist ("blah", here), pip warns about it:

pip install "django-anymail[blah]"

>WARNING: django-anymail 8.2 does not provide the extra 'blah'
...

But uv says nothing about it:

uv pip -v install "django-anymail[blah]"
Searching for Python interpreter in virtual environments
DEBUG Found CPython 3.11.8 at `/home/project/.venv/bin/python3` (virtual environment)
DEBUG Using Python 3.11.8 environment at .venv/bin/python3
DEBUG Acquired lock for `.venv`
DEBUG Requirement satisfied: asgiref<4, >=3.7.0
DEBUG Requirement satisfied: certifi>=2017.4.17
DEBUG Requirement satisfied: charset-normalizer<4, >=2
DEBUG Requirement satisfied: django-anymail[mailgunqqsdqd]
DEBUG Requirement satisfied: django>=2.0
DEBUG Requirement satisfied: idna<4, >=2.5
DEBUG Requirement satisfied: requests>=2.4.3
DEBUG Requirement satisfied: sqlparse>=0.3.1
DEBUG Requirement satisfied: urllib3<3, >=1.21.1
DEBUG Requirement satisfied: urllib3>=1.25.0
Audited 1 package in 5ms

[funkybob]

Ah, should I open a separate ticket about how uv is not emitting the --extra-index-url line from the requirements.in?

I should have spotted this when I had to add it to the requirements-dev.in.

[charliermarsh]

You can add --emit-index-url. Check out uv pip compile --help for the full list!

[charliermarsh]

@dperetti -- We do emit it on install:

Downloaded 7 packages in 528ms
Installed 9 packages in 44ms
 + asgiref==3.8.1
 + certifi==2024.6.2
 + charset-normalizer==3.3.2
 + django==5.0.6
 + django-anymail==10.3
 + idna==3.7
 + requests==2.32.3
 + sqlparse==0.5.0
 + urllib3==2.2.1
warning: The package `django-anymail==10.3` does not have an extra named `blah`.

But not when the package is already installed.

[funkybob]

You can add --emit-index-url. Check out uv pip compile --help for the full list!

So many changes in behavior :/

Perhaps a "--pip-tools-compat" flag is warranted?

[notatallshaw]

I might contribute a docs update to include some tips on differences when moving from pip-tools

It seems like this won't be a difference for very long, if I don't specify --strip-extras or --no-strip-extras on the latest version of pip-tools I get this warning:

$ pip-compile requirements.in
WARNING: --strip-extras is becoming the default in version 8.0.0. To silence this warning, either use --strip-extras to opt into the new default or use --no-strip-extras to retain the existing behavior.

[funkybob]

Indeed, however it's a loud difference with a clear deprecation path and timeline.

I'm slowly discovering all the quiet differences between uv and pip-tools.

[funkybob]

You can add --emit-index-url. Check out uv pip compile --help for the full list!

For those playing along at home, this does work, but it also emits --index-url -- another departure frompip-tools behavior, though granted "explicit is better than implicit".

[charliermarsh]

I appreciate where you're coming from but I consider these to be fairly minimal differences. We have different defaults for the --strip and --emit flags, and one of them is about to pip-tools default anyway. A contribution to the PIP_COMPATIBILITY.md to cover these would be welcome, if you're up for it, but I think all the choices we made here are very reasonable.

[funkybob]

Not arguing the validity of the choices.

It's more that since this is pitched as a drop in replacement ... it should default to behaving the same as the tools it's replacing.

Or have a flag to do that.

I will make time this afternoon to formulate a patch for the docs.

[zanieb]

Sorry to hear you've been having a hard time transitioning. We've tried to avoid implementing deprecated behavior in both pip and pip tools so there are definitely some differences — which is why we try to clearly outline the expectations in the compatibility guide.

[charliermarsh]

Closed by #4302. Thanks!