Warn when two packages write to the same module by konstin · Pull Request #13437 · astral-sh/uv

konstin · 2025-05-13T20:55:00Z

We regularly get confusing bug reports where a package sometimes works and sometimes doesn't and it's not clear to the user why. Ultimately, it turns out that two packages contain the same module and there is a race condition when installing the two packages. Usually, it's one of the opencv-python distributions, but recently it's been z3, too. These error are completely inscrutable to users.

We now warn for top-level modules (pattern: <identifier>/__init__.py) that collide in a single installation, naming the offending wheels. Checking for __init__.py excludes namespace packages.

Test script:

uv venv -q && cargo run -q --profile fast-build pip install --no-progress --link-mode clone opencv-python opencv-contrib-python --no-build --no-deps
uv venv -q && cargo run -q --profile fast-build pip install --no-progress --link-mode copy opencv-python opencv-contrib-python --no-build --no-deps
uv venv -q && cargo run -q --profile fast-build pip install --no-progress --link-mode hardlink opencv-python opencv-contrib-python --no-build --no-deps
uv venv -q && cargo run -q --profile fast-build pip install --no-progress --link-mode symlink opencv-python opencv-contrib-python --no-build --no-deps

We currently only catch conflicts in a single installation. Should we prime the lock database with the site-packages contents, and would that carry overhead?

konstin · 2025-05-13T20:55:46Z

crates/uv-install-wheel/src/linker.rs

+    modules: Mutex<FxHashMap<OsString, WheelFilename>>,
+}
+
+impl Locks {
+    /// Warn when a module exists in multiple packages.
+    fn warn_module_conflict(&self, module: &OsStr, filename: &WheelFilename) {
+        if let Some(existing) = self
+            .modules
+            .lock()
+            .unwrap()
+            .insert(module.to_os_string(), filename.clone())


This should get <1000 hits so I've implemented the naive approach.

cthoyt · 2025-05-14T06:14:11Z

An example that's vexed me in the past:

https://pypi.org/project/Flask-Bootstrap/ (original, unmaintained project)
https://pypi.org/project/Bootstrap-Flask/ (maintained fork, uses same package name but
different on PyPI)

both install to flask_bootstrap

charliermarsh · 2025-05-20T14:16:33Z

I don't know that we should make this user-facing, to be honest. Isn't it going to trigger any time anyone installs Jupyter?

konstin · 2025-05-20T14:58:00Z

I strongly think this should be user-facing, I linked some of the issues this would have prevented above (we had another one just today: #13550), especially since this is otherwise non-deterministic, silent and almost impossible to figure out if you don't know what you're looking for.

It doesn't warn for uv pip install jupyter, is there a reason we would expect it to? Note that we're checking for an __init__.py, so this does not affect namespace packages that are allowed to interleave.

atinary-bvollmer · 2025-05-21T06:34:42Z

Hey @charliermarsh,

I'm the author of #13550. Thanks to @konstin I found the issue but I would strongly side with here that especially for less experienced users this can lead to very frustrating problems. A warning during the install would have most likely prevented it for me.

Thanks!

konstin · 2025-05-27T21:09:15Z

Added a test case

konstin · 2025-05-30T11:31:05Z

One place where this could fail is the Python 2 path = import("pkgutil").extend_path(__path__, name) hack, but I haven't seen that one in a while, and we can special case it if necessary.

zanieb · 2025-05-30T12:02:37Z

(I'll review this)

crates/uv/tests/it/pip_install.rs

zanieb · 2025-06-06T19:01:56Z

crates/uv/tests/it/pip_install.rs

+    ----- stderr -----
+    Resolved 2 packages in [TIME]
+    Prepared 2 packages in [TIME]
+    warning: The module built_by_uv exists in two packages! This leads to a race condition and likely to a broken installation. Consider removing either built-by-uv (built_by_uv-0.1.0-py3-none-any.whl) or also-built-by-uv (also_built_by_uv-0.1.0-py3-none-any.whl).


Why do we show the wheel filenames here? Is that an important detail?

For debugging, I usually need to go open the wheel and look inside of it, so I consider this a valuable detail. We should show at least the version though, otherwise we can't find where this is from.

See also https://github.com/astral-sh/uv/pull/13437/files#r2132758628

I worry about this as a user-facing detail in the warning though, isn't the file name going to be in the verbose logs?

Also, aren't the versions included in the install summary?

The error should by itself contain enough information to track down the problem, so I would like it to at least contain the versions.

I'm okay with the versions, styled as we do elsewhere, e.g., foo (v1.0.0). I think the filenames sound like a developer-facing rather than user-facing detail.

The exclamation mark is a bit much, I don't think we do that anywhere else.

That's dropped in #13889

I updated the error message

zanieb · 2025-06-06T19:07:53Z

crates/uv-install-wheel/src/linker.rs

+                (wheel_a, &wheel_b)
+            };
+            warn_user!(
+                "The module {} exists in two packages! \


I reworded this in #13889, curious for your thoughts

Will this trigger if two packages include the same file?

Yes, do we have a case where this is expected and valid to happen?

Isn't it a feature that two packages could contain the same shared module, and could just vendor it into their own wheel?

Is it? That seems pretty problematic from a versioning perspective.

I haven't seen any vendoring cases where the vendored modules would overlap.

zanieb · 2025-06-06T19:14:56Z

crates/uv-install-wheel/src/linker.rs

+        && relative.components().next_back().unwrap().as_os_str() == "__init__.py"
+    {
+        // Modules must be UTF-8, but we can skip the conversion using OsStr.
+        locks.warn_module_conflict(relative.components().next().unwrap().as_os_str(), filename);


What does this look like for modules like foo.bar? Are we only going to show bar? Or are you only checking for the top-level modules because of relative.components.count() == 2?

That's the TODO above, we're only looking for foo/__init__.py rn and don't track foo/bar/__init__.py

crates/uv/tests/it/pip_install.rs

codspeed-hq · 2025-06-11T20:55:40Z

CodSpeed WallTime Performance Report

Merging #13437 will not alter performance

_{Comparing konsti/warn-on-module-conflicts (f18ca49) with main (8968d78)}

Summary

✅ 3 untouched benchmarks

MalteEbner · 2025-06-13T16:02:14Z

Suggested solution: I think it should be possible for the user to define different packages under the same module name similar to the sources logic. Something similar to this:

[tool.uv.sources]
pillow = [
  { package = "pillow-simd>=9,<10", marker = "platform_machine == 'x86_64'"},
  { package = "pillow>=11,<12", marker = "platform_machine == 'aarch64'"},
]

geofft · 2025-08-08T19:30:21Z

I sort of thought I've seen pre-PEP-420 namespace packages recently enough. Debian Code Search for path:__init__.py extend_path shows a good amount of hits, though several are in vendored and possibly old packages. Protobuf has a google/__init__.py in its sources, but it doesn't seem to actually ship it in the wheel. The latest PyQt6 wheel does have a PyQt6/__init__.py, but I'm not sure if anything else installs into that namespace.

[edit: meant to write "pre-PEP-420", not "PEP-420"]

konstin · 2025-08-08T19:33:17Z

Valid PEP 420 packages that don't clash are not affected by this change.

In the previous iteration, conflict detection was based on top level modules. This would work if all namespace packages correctly omitted the `__init__.py`, but e.g. the nvidia packages include an empty `nvida/__init__.py`. Instead, we track overlapping top level modules during installation and perform conflict analysis after installation, recursing only as far as necessary. See #13437 for a list of reported conflicts. Before: ``` $ uv venv -c -q && uv pip install --preview nvidia-nvjitlink-cu12==12.8.93 nvidia-nvtx-cu12==12.8.90 Resolved 2 packages in 0.99ms ░░░░░░░░░░░░░░░░░░░░ [0/2] Installing wheels... warning: The module `nvidia` is provided by more than one package, which causes an install race condition and can result in a broken module. Consider removing your dependency on either `nvidia-nvtx-cu12` (v12.8.90) or `nvidia-nvjitlink-cu12` (v12.8.93). Installed 2 packages in 3ms + nvidia-nvjitlink-cu12==12.8.93 + nvidia-nvtx-cu12==12.8.90 ``` After: ``` $ uv venv -c -q && cargo run -q pip install --preview nvidia-nvjitlink-cu12==12.8.93 nvidia-nvtx-cu12==12.8.90 Resolved 2 packages in 3ms Installed 2 packages in 4ms + nvidia-nvjitlink-cu12==12.8.93 + nvidia-nvtx-cu12==12.8.90 ``` Still detected true positive: ``` $ uv venv -c -q && cargo run -q pip install --no-progress opencv-python opencv-contrib-python --no-build --no-deps --preview Resolved 2 packages in 5ms warning: The file `cv2/__init__.pyi` is provided by more than one package, which causes an install race condition and can result in a broken module. Packages containing the file: * opencv-contrib-python (opencv_contrib_python-4.13.0.90-cp37-abi3-manylinux_2_28_x86_64.whl) * opencv-python (opencv_python-4.13.0.90-cp37-abi3-manylinux_2_28_x86_64.whl) Installed 2 packages in 6ms + opencv-contrib-python==4.13.0.90 + opencv-python==4.13.0.90 ```

konstin requested a review from charliermarsh May 13, 2025 20:55

konstin added the bug Something isn't working label May 13, 2025

konstin commented May 13, 2025

View reviewed changes

konstin force-pushed the konsti/warn-on-module-conflicts branch from 1dce177 to b052c16 Compare May 19, 2025 08:29

konstin mentioned this pull request May 20, 2025

Github action installation differs from local #13550

Closed

konstin force-pushed the konsti/warn-on-module-conflicts branch from b052c16 to 8bcf72a Compare May 27, 2025 20:08

zanieb self-assigned this May 28, 2025

konstin requested a review from zanieb June 5, 2025 10:08

zanieb reviewed Jun 6, 2025

View reviewed changes

crates/uv/tests/it/pip_install.rs Outdated Show resolved Hide resolved

zanieb reviewed Jun 6, 2025

View reviewed changes

zanieb mentioned this pull request Jun 6, 2025

Update error message for module conflict hint #13889

Closed

zanieb reviewed Jun 6, 2025

View reviewed changes

crates/uv/tests/it/pip_install.rs Show resolved Hide resolved

konstin temporarily deployed to uv-test-publish June 11, 2025 19:35 — with GitHub Actions Inactive

konstin temporarily deployed to uv-test-publish June 11, 2025 20:43 — with GitHub Actions Inactive

konstin mentioned this pull request Jun 13, 2025

Handling of different packages with same name (Pillow and Pillow-SIMD) is confusing. #14030

Open

This comment was marked as outdated.

Sign in to view

konstin force-pushed the konsti/warn-on-module-conflicts branch from dcefc54 to 6f9e320 Compare July 24, 2025 10:05

konstin temporarily deployed to uv-test-registries July 24, 2025 10:07 — with GitHub Actions Inactive

konstin temporarily deployed to uv-test-registries July 24, 2025 10:13 — with GitHub Actions Inactive

konstin requested a review from zanieb August 7, 2025 14:30

zanieb approved these changes Aug 7, 2025

View reviewed changes

konstin added 2 commits August 8, 2025 10:48

Warn when two packages write to the same module

0754418

Use source trees directly

f18ca49

konstin force-pushed the konsti/warn-on-module-conflicts branch from d31f63c to f18ca49 Compare August 8, 2025 08:48

konstin enabled auto-merge (squash) August 8, 2025 08:48

konstin temporarily deployed to uv-test-registries August 8, 2025 08:50 — with GitHub Actions Inactive

konstin merged commit 1843c90 into main Aug 8, 2025
95 checks passed

konstin deleted the konsti/warn-on-module-conflicts branch August 8, 2025 09:01

dagardner-nv mentioned this pull request Aug 8, 2025

[BUG]: Creating an env using uv v0.8.7 issues a warning NVIDIA/NeMo-Agent-Toolkit#611

Closed

2 tasks

BrewTestBot mentioned this pull request Aug 8, 2025

uv 0.8.7 Homebrew/homebrew-core#232884

Merged

konstin mentioned this pull request Aug 18, 2025

Stabilize conflicting modules warning #15357

Open

4 tasks

zanieb mentioned this pull request Sep 17, 2025

Better support for conflicting transitive dependencies #15813

Closed

samypr100 mentioned this pull request Nov 3, 2025

report an error when installing a package that overrides another #16546

Open

konstin mentioned this pull request Jan 20, 2026

Better detection for conflicting packages #17623

Merged

Conversation

konstin commented May 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cthoyt commented May 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

charliermarsh commented May 20, 2025

Uh oh!

konstin commented May 20, 2025

Uh oh!

atinary-bvollmer commented May 21, 2025

Uh oh!

konstin commented May 27, 2025

Uh oh!

konstin commented May 30, 2025

Uh oh!

zanieb commented May 30, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zanieb Jun 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

codspeed-hq bot commented Jun 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging #13437 will not alter performance

Summary

Uh oh!

MalteEbner commented Jun 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

geofft commented Aug 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

konstin commented Aug 8, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

konstin commented May 13, 2025 •

edited

Loading

cthoyt commented May 14, 2025 •

edited

Loading

zanieb Jun 11, 2025 •

edited

Loading

codspeed-hq bot commented Jun 11, 2025 •

edited

Loading

MalteEbner commented Jun 13, 2025 •

edited

Loading

geofft commented Aug 8, 2025 •

edited

Loading