add support for specifying conflicting extras

[BurntSushi]

This PR adds support for conflicting extras. For example, consider
some optional dependencies like this:

[project.optional-dependencies]
project1 = ["numpy==1.26.3"]
project2 = ["numpy==1.26.4"]

These dependency specifications are not compatible with one another.
And if you ask uv to lock these, you'll get an unresolvable error.

With this PR, you can now add this to your pyproject.toml to get
around this:

[tool.uv]
conflicting-groups = [
    [
      { package = "project", extra = "project1" },
      { package = "project", extra = "project2" },
    ],
]

This will make the universal resolver create additional forks
internally that keep the dependencies from the project1 and
project2 extras separate. And we make all of this work by reporting
an error at install time if one tries to install with two or more
extras that have been declared as conflicting. (If we didn't do this,
it would be possible to try and install two different versions of the
same package into the same environment.)

This PR does not add support for conflicting groups, but it is
intended to add support in a follow-up PR.

Closes #6981

Fixes #8024

Ref #6729, Ref #6830

This should also hopefully unblock
dagster-io/dagster#23814, but in my testing, I
did run into other problems (specifically, with pywin). But it does
resolve the problem with incompatible dependencies in two different
extras once you declare test-airflow-1 and test-airflow-2 as
conflicting for dagster-airflow.

NOTE: This PR doesn't make conflicting-groups public yet. And in a follow-up PR, I plan to switch the name to conflicts instead of conflicting-groups, since it will be able to accept conflicting extras and conflicting groups.

[charliermarsh]

Epic!

[BurntSushi]

Reviewers should read this commit-by-commit. It should help a lot.

Also, I haven't made any attempt at hiding the schema yet, since I'm guessing we'll want to merge this without exposing it yet. cc @zanieb I know we talked about this a couple days ago.

[charliermarsh]

This generally looks good to me. A few questions.

[charliermarsh]

To clarify: does this mean conflicting groups must be defined in the workspace root?

Could we instead have each project define its own conflicts, and then remove package = from the schema? I don't think it's necessary to allow arbitrary packages to be included in these groups, since we only allow enabling extras for members in the workspace.

[BurntSushi]

I think this makes sense to me. From the resolver's perspective, it definitely needs the package name, but from the user's perspective, I can see how it makes sense to keep the conflicting declaration local to where the corresponding extras/groups are themselves defined. And in that context, the package name is implicit.

I do wonder if there exists a conflict that does require the user to specify the package name, but I can't quite think of how that might manifest.

I'll plan to remove package from the schema.

[BurntSushi]

(But note that we still need package in the lock file.)

[charliermarsh]

I am wondering if the terminology here should just around something more general, like "Conflicts", since it has to cover both extras and dependency groups. The use of "groups" here makes it somewhat overloaded.

[BurntSushi]

Yeah @zanieb suggested just using conflicts, so I think I'll switch to that. I don't like that it doesn't say what is conflicting, but "extra" and "group" are already such general names that it's hard to think of something more general than that but still captures their essence. And in theory, we could add more types of conflicts in the future, although the use cases aren't quite clear.

[T-256]

Can you add a test case for this? where packages are different.
I noticed all test cases use package = "project". could you make it as default value to avoid explicitly writing?

[BurntSushi]

Ah yeah, I'll add one of these in a follow-up PR. Originally I thought we were going to get rid of package, but now it's optional. I'll add the test from the comments below, which should cover "conflicts with distinct package values" case.

[konstin]

Can't comment that high, but now diverging packages can be empty and the debug message has a hole:

DEBUG Splitting resolution on dummy==0.1.0 over  into 3 resolution with separate markers

[BurntSushi]

Ah I missed this. I'll have this fixed in a follow-up PR.

[konstin]

Sharing this from our conversation:Currently, the following example leads to an install without proxy-fork-1 but with anyio==4.2.0, while it should conflict, the conflict is unconditional.

[project]
name = "dummy"
version = "0.1.0"
requires-python = "==3.12.*"
dependencies = [
  "proxy-fork-1[project1,project2]"
]

[tool.uv.sources]
proxy-fork-1 = { path = "../proxy-fork-1" }

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "proxy-fork-1"
version = "0.1.0"
requires-python = "==3.12.*"
dependencies = []

[project.optional-dependencies]
project1 = ["anyio==4.1.0"]
project2 = ["anyio==4.2.0"]

[BurntSushi]

@konstin Aye, I think that should be resolved now. Whenever a requirement is seen with an unconditional dependency on a conflicting extra, we error. This is perhaps overly conservative, but if we figure out a way to relax it (and a use case that wants it), we can do so in the future.

[konstin]

The comment is out of date, and i think the example, too? (dev conflicting with dev sounds like it should warn/error)

[BurntSushi]

Aye, fixed. I'll do some more validation in a follow-up PR.

[BurntSushi]

@konstin found another interesting case. In this case, resolution fails, even though it should be possible to declare conflicting extras in a way that makes it work. To reproduce, create a pyproject.toml with this:

[project]
name = "dummy"
version = "0.1.0"
requires-python = "==3.12.*"

[project.optional-dependencies]
project1 = [
  "proxy-fork-1[project1]",
  # "dummysub[project1]" # Fails with or without this
]
project2 = [
  "proxy-fork-1[project2]"
  # "dummysub[project2]" # Fails with or without this
]

[tool.uv.sources]
proxy-fork-1 = { path = "./proxy-fork-1" }
dummysub =  { workspace = true }

[tool.uv.workspace]
members = ["dummysub"]

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[tool.uv]
conflicting-groups = [
  [
    { extra = "project1" },
    { extra = "project2" },
  ],
]

And another at dummysub/pyproject.toml:

[project]
name = "dummysub"
version = "0.1.0"
requires-python = "==3.12.*"

[project.optional-dependencies]
project1 = [
  "proxy-fork-1[project1]",
]
project2 = [
  "proxy-fork-1[project2]"
]

[tool.uv.sources]
proxy-fork-1 = { path = "../proxy-fork-1" }

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[tool.uv]
conflicting-groups = [
  [
    { extra = "project1" },
    { extra = "project2" },
  ],
]

And finally, another one that isn't part of the workspace but is just an "external" package:

[project]
name = "proxy-fork-1"
version = "0.1.0"
requires-python = "==3.12.*"
dependencies = []

[project.optional-dependencies]
project1 = ["anyio==4.1.0"]
project2 = ["anyio==4.2.0"]

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

In this case, we have project1 and project2 declared as conflicting extras in both dummy and dummysub. But they manifest as two different sets of conflicting extras. In order to make this example work, I believe they need to all be in one set, because they are all mutually conflicting. But since these extras span two different packages, the easiest way to do this is to bring back package (but we can make it optional). That is, in the root pyproject.toml, we have this:

[tool.uv]
conflicting-groups = [
  [
    { extra = "project1" },
    { extra = "project2" },
    { package = "dummysub", extra = "project1" },
    { package = "dummysub", extra = "project2" },
  ],
]

And we can remove conflicting-groups from dummysub/pyproject.toml. Once done, this makes resolution succeed.

The only alternative I can see here, AFAIK, is to automatically combine conflicting groups among workspace members, so that every set of conflicts is combined with every other set of conflicts. But I think this leads to a combinatorial explosion, which would be bad juju. So I think we are left with allowing package but making it optional. (And now I think we've come full circle to @zanieb's suggestion. :P)

[BurntSushi]

The above solution means you can't install both dummy[project1] and dummysub[project1] at the same time. So I think you actually have to list out all pairs of possible conflicts:

[tool.uv]
conflicting-groups = [
  [
    { extra = "project1" },
    { extra = "project2" },
  ],
  [
    { package = "dummysub", extra = "project1" },
    { package = "dummysub", extra = "project2" },
  ],
  [
    { extra = "project1" },
    { package = "dummysub", extra = "project2" },
  ],
  [
    { package = "dummysub", extra = "project1" },
    { extra = "project2" },
  ],
]

And I do believe you need all four entries here, since you need to declare, e.g., that dummy[project1] and dummysub[project2] are conflicting.

[charliermarsh]

@BurntSushi is it then any different than taking all pairs of groups? I don't fully see the advantage.

[BurntSushi]

@BurntSushi is it then any different than taking all pairs of groups? I don't fully see the advantage.

I think the advantage is that not all conflicting extras are mutually exclusionary. So the question becomes then, in what circumstances do you take all pairs? I don't think you want to do it in all cases.

[BurntSushi]

And also, the above isn't all pairs. For example, dummy[project1] and dummysub[project1] aren't marked as conflicting. You very specifically want to be able to install both of those at the same time.

[BurntSushi]

NOTE: I accidentally squash-merged this PR. For anyone looking for better history in the future, my apologies.

[martinitus]

Hey folks, I have a followup question to the "conflicting extras / groups" features that were merged the last weeks.

I would like to declare conflicting extras - however these dependencies have no inherent conflict in the package or version. Instead they happen to use the same module namespace - databricks-connect and pyspark both use pyspark as module names.

What I tried so far was

[project.optional-dependencies]
databums = ["databricks-connect>=16.0.0",]
vanilla = ["pyspark>=3.5.4",]

[tool.uv]
conflicts = [
    [
        {extra = "databums" },
        {extra = "vanilla" },
    ]
]

However, when I run uv lock or uv pip install -v -e .[databums,vanilla] or even just uv pip -e . it happily installs both packages (and screws up the site-package directory afaik).

I presume the current implementation only prevents scenarios where the dependency resolution would have failed due to conflicting optional dependencies. Whereas what I want is explicitly declaring dependencies as conflicting?

Let me know if I missed something and this should actually be possible!

BTW: I also tried it with optional groups, but groups are not exposed to pip (yet, afaik). As I want to be able to install my package with pip on downstream systems, dependency groups seem to not be an option yet.

-- Edit --
I opened #10238

[zanieb]

@martinitus I don't think we support that, but it does seem interesting.

Could you open a new issue please?