Skip to content

Comments

WIP extra-build-dependencies#14092

Closed
Gankra wants to merge 4 commits intomainfrom
gankra/add-build-deps
Closed

WIP extra-build-dependencies#14092
Gankra wants to merge 4 commits intomainfrom
gankra/add-build-deps

Conversation

@Gankra
Copy link
Contributor

@Gankra Gankra commented Jun 16, 2025

This introduces a global setting

[tool.uv.extra-build-dependencies]
pytest = ["setuptools"]

Which forces extra requirements into a build-isolated sdist build for any package that has the given name. Included is a test the demonstrates the feature is at all useful/functional using an example from

no-build-isolation-package was used as the template for all the places this setting should be threaded, as they both mess with the isolated build of a package by-name. There's a very high chance this implementation is insufficient, as I didn't go out of my way to do anything with caching (but maybe I ended up shoving it somewhere that does).

@Gankra Gankra marked this pull request as draft June 16, 2025 21:32
Comment on lines +487 to +535
let name = pyproject_toml
.project
.as_ref()
.map(|project| &project.name)
.or(package_name);
let extra_build_dependencies = name
.as_ref()
.and_then(|name| extra_build_dependencies.get(name).cloned())
.unwrap_or_default();

let backend = if let Some(mut build_system) = pyproject_toml.build_system {
// Apply extra-build-dependencies if there are any
build_system.requires.extend(extra_build_dependencies);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Happy path where I'm confident Sources/Indexes are appropriately handled.

Comment on lines 582 to 626
}
default_backend.clone()
let mut backend = default_backend.clone();
// Apply extra_build_dependencies
// TODO(Gankra): should Sources/Indexes be applied on this path?
backend
.requirements
.extend(extra_build_dependencies.into_iter().map(Requirement::from));
backend
};
Copy link
Contributor Author

@Gankra Gankra Jun 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sad path 1 where I probably need to handle sources where previously they were always irrelevant (pyproject.toml but missing [build-system])

.requirements
.extend(extra_build_dependencies.into_iter().map(Requirement::from));
Ok((backend, None))
}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sad path 2 where I am also probably mishandling sources/indexes (no pyproject.toml).

Comment on lines 1375 to 1605
[tool.uv.extra-build-dependencies]
fasttext = ["setuptools", "wheel", "pybind11"]
"#,
)?;

uv_snapshot!(&filters, context.sync(), @r"
success: true
exit_code: 0
----- stdout -----

----- stderr -----
Resolved [N] packages in [TIME]
Prepared [N] packages in [TIME]
Installed [N] packages in [TIME]
+ fasttext==0.9.2
+ myproject==0.1.0 (from file://[TEMP_DIR]/)
+ numpy==1.26.4
+ pybind11==2.11.1
+ setuptools==69.2.0
");
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Glorious success!

@notatallshaw
Copy link
Collaborator

I think this effectively fixes #12447

@konstin
Copy link
Member

konstin commented Jun 17, 2025

The test fails to pass for me locally, with the same error as when I try to install asttext==0.9.2 on its own. Does it need a specific, older compiler version?

$ uv pip install fasttext==0.9.2 --no-build-isolation-package fasttext
Resolved 4 packages in 3ms
  × Failed to build `fasttext==0.9.2`
  ├─▶ The build backend returned an error
  ╰─▶ Call to `setuptools.build_meta:__legacy__.build_wheel` failed (exit status: 1)

      [stdout]
      running bdist_wheel
      running build
      running build_py
      copying python/fasttext_module/fasttext/FastText.py -> build/lib.linux-x86_64-cpython-313/fasttext
      copying python/fasttext_module/fasttext/__init__.py -> build/lib.linux-x86_64-cpython-313/fasttext
      copying python/fasttext_module/fasttext/util/util.py -> build/lib.linux-x86_64-cpython-313/fasttext/util
      copying python/fasttext_module/fasttext/util/__init__.py -> build/lib.linux-x86_64-cpython-313/fasttext/util
      copying python/fasttext_module/fasttext/tests/test_script.py -> build/lib.linux-x86_64-cpython-313/fasttext/tests
      copying python/fasttext_module/fasttext/tests/test_configurations.py -> build/lib.linux-x86_64-cpython-313/fasttext/tests
      copying python/fasttext_module/fasttext/tests/__init__.py -> build/lib.linux-x86_64-cpython-313/fasttext/tests
      running build_ext
      c++ -pthread -fno-strict-overflow -Wsign-compare -Wunreachable-code -DNDEBUG -g -O3 -Wall -fPIC -fPIC -I/home/konsti/projects/uv/.venv/include
      -I/home/konsti/.local/share/uv/python/cpython-3.13.2-linux-x86_64-gnu/include/python3.13 -c /tmp/tmpl6kbi6ua.cpp -o tmp/tmpl6kbi6ua.o -std=c++14
      c++ -pthread -fno-strict-overflow -Wsign-compare -Wunreachable-code -DNDEBUG -g -O3 -Wall -fPIC -fPIC -I/home/konsti/projects/uv/.venv/include
      -I/home/konsti/.local/share/uv/python/cpython-3.13.2-linux-x86_64-gnu/include/python3.13 -c /tmp/tmp91txd3ad.cpp -o tmp/tmp91txd3ad.o
      -fvisibility=hidden
      building 'fasttext_pybind' extension
      c++ -pthread -fno-strict-overflow -Wsign-compare -Wunreachable-code -DNDEBUG -g -O3 -Wall
      -fPIC -fPIC -I/home/konsti/projects/uv/.venv/lib/python3.13/site-packages/pybind11/include
      -I/home/konsti/projects/uv/.venv/lib/python3.13/site-packages/pybind11/include -Isrc -I/home/konsti/projects/uv/.venv/include
      -I/home/konsti/.local/share/uv/python/cpython-3.13.2-linux-x86_64-gnu/include/python3.13 -c python/fasttext_module/fasttext/pybind/fasttext_pybind.cc
      -o build/temp.linux-x86_64-cpython-313/python/fasttext_module/fasttext/pybind/fasttext_pybind.o -DVERSION_INFO=\"0.9.2\" -std=c++14
      -fvisibility=hidden
      c++ -pthread -fno-strict-overflow -Wsign-compare -Wunreachable-code -DNDEBUG -g -O3 -Wall
      -fPIC -fPIC -I/home/konsti/projects/uv/.venv/lib/python3.13/site-packages/pybind11/include
      -I/home/konsti/projects/uv/.venv/lib/python3.13/site-packages/pybind11/include -Isrc -I/home/konsti/projects/uv/.venv/include
      -I/home/konsti/.local/share/uv/python/cpython-3.13.2-linux-x86_64-gnu/include/python3.13 -c src/args.cc -o
      build/temp.linux-x86_64-cpython-313/src/args.o -DVERSION_INFO=\"0.9.2\" -std=c++14 -fvisibility=hidden

      [stderr]
      /home/konsti/projects/uv/.venv/lib/python3.13/site-packages/setuptools/dist.py:599: SetuptoolsDeprecationWarning: Invalid dash-separated key
      'description-file' in 'metadata' (setup.cfg), please use the underscore name 'description_file' instead.
      !!

              ********************************************************************************
              Usage of dash-separated 'description-file' will not be supported in future
              versions. Please use the underscore name 'description_file' instead.
              (Affected: fasttext).

              By 2026-Mar-03, you need to update your project and remove deprecated calls
              or your builds will no longer be supported.

              See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.
              ********************************************************************************

      !!
        opt = self._enforce_underscore(opt, section)
      /home/konsti/projects/uv/.venv/lib/python3.13/site-packages/setuptools/dist.py:759: SetuptoolsDeprecationWarning: License classifiers are
      deprecated.
      !!

              ********************************************************************************
              Please consider removing the following classifiers in favor of a SPDX license expression:

              License :: OSI Approved :: MIT License

              See https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#license for details.
              ********************************************************************************

      !!
        self._finalize_license_expression()
      python/fasttext_module/fasttext/pybind/fasttext_pybind.cc: In lambda function:
      python/fasttext_module/fasttext/pybind/fasttext_pybind.cc:345:35: warning: comparison of integer expressions of different signedness: ‘int32_t’
      {aka ‘int’} and ‘std::vector<long int>::size_type’ {aka ‘long unsigned int’} [-Wsign-compare]
        345 |             for (int32_t i = 0; i < vocab_freq.size(); i++) {
            |                                 ~~^~~~~~~~~~~~~~~~~~~
      python/fasttext_module/fasttext/pybind/fasttext_pybind.cc: In lambda function:
      python/fasttext_module/fasttext/pybind/fasttext_pybind.cc:359:35: warning: comparison of integer expressions of different signedness: ‘int32_t’
      {aka ‘int’} and ‘std::vector<long int>::size_type’ {aka ‘long unsigned int’} [-Wsign-compare]
        359 |             for (int32_t i = 0; i < labels_freq.size(); i++) {
            |                                 ~~^~~~~~~~~~~~~~~~~~~~
      src/args.cc: In member function ‘void fasttext::Args::parseArgs(const std::vector<std::__cxx11::basic_string<char> >&)’:
      src/args.cc:120:23: warning: comparison of integer expressions of different signedness: ‘int’ and ‘std::vector<std::__cxx11::basic_string<char>
      >::size_type’ {aka ‘long unsigned int’} [-Wsign-compare]
        120 |   for (int ai = 2; ai < args.size(); ai += 2) {
            |                    ~~~^~~~~~~~~~~~~
      src/args.cc:221:19: warning: catching polymorphic type ‘class std::out_of_range’ by value [-Wcatch-value=]
        221 |     } catch (std::out_of_range) {
            |                   ^~~~~~~~~~~~
      src/args.cc: In member function ‘int64_t fasttext::Args::getAutotuneModelSize() const’:
      src/args.cc:468:3: error: ‘uint64_t’ was not declared in this scope
        468 |   uint64_t multiplier = 1;
            |   ^~~~~~~~
      src/args.cc:17:1: note: ‘uint64_t’ is defined in header ‘<cstdint>’; did you forget to ‘#include <cstdint>’?
         16 | #include <unordered_map>
        +++ |+#include <cstdint>
         17 |
      src/args.cc:471:5: error: ‘multiplier’ was not declared in this scope
        471 |     multiplier = units[lastCharacter];
            |     ^~~~~~~~~~
      src/args.cc:474:11: error: expected ‘;’ before ‘size’
        474 |   uint64_t size = 0;
            |           ^~~~~
            |           ;
      src/args.cc:478:5: error: ‘size’ was not declared in this scope
        478 |     size = std::stol(modelSize, &nonNumericCharacter);
            |     ^~~~
      src/args.cc:490:10: error: ‘size’ was not declared in this scope
        490 |   return size * multiplier;
            |          ^~~~
      src/args.cc:490:17: error: ‘multiplier’ was not declared in this scope
        490 |   return size * multiplier;
            |                 ^~~~~~~~~~
      error: command '/usr/bin/c++' failed with exit code 1

      hint: This usually indicates a problem with the package or the build environment.
$ gcc --version
gcc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0
Copyright (C) 2023 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
$ clang --version
Ubuntu clang version 18.1.3 (1ubuntu1)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

@Gankra
Copy link
Contributor Author

Gankra commented Jun 17, 2025

Oh hrm that's annoying, there's other packages I could use for this problem though.

@konstin
Copy link
Member

konstin commented Jun 17, 2025

We should pick something where the successful build is still fast, we had problems with slow builds in test the past. The fastest should be handrolling something with build-system.backend-path where our custom build system imports a marker package and then forwards us to hatchling the regular way.

@Florent-H
Copy link

Any updates on this pull request? I am having build isolation issues (#2252). Thanks for your time and effort!

@konstin konstin added the enhancement New feature or improvement to existing functionality label Jul 16, 2025
@konstin konstin force-pushed the gankra/add-build-deps branch from b2caf47 to c4245ca Compare July 16, 2025 07:22
@konstin konstin temporarily deployed to uv-test-registries July 16, 2025 07:24 — with GitHub Actions Inactive
@konstin konstin temporarily deployed to uv-test-registries July 16, 2025 10:09 — with GitHub Actions Inactive
@zanieb zanieb force-pushed the gankra/add-build-deps branch from 59fe8f9 to 4ca5770 Compare July 18, 2025 13:17
@zanieb zanieb self-assigned this Jul 18, 2025
@zanieb zanieb temporarily deployed to uv-test-registries July 18, 2025 13:20 — with GitHub Actions Inactive
@zanieb zanieb closed this Jul 18, 2025
zanieb added a commit that referenced this pull request Jul 30, 2025
Replaces #14092

Adds `tool.uv.extra-build-dependencies = {package = [dependency, ...]}`
which extends `build-system.requires` during package builds.

These are lowered via workspace sources, are applied to transitive
dependencies, and are included in the wheel cache shard hash.

There are some features we need to follow-up on, but are out of scope
here:

- Preferring locked versions for build dependencies
- Settings for requiring locked versions for build depencies

There are some quality of life follow-ups we should also do:

- Warn on `extra-build-dependencies` that do not apply to any packages
- Add test cases and improve error messaging when the
`extra-build-dependencies` resolve fails


-------

There ~are~ were a few open decisions to be made here

1. Should we resolve these dependencies alongside the
`build-system.requires` dependencies? Or should we resolve separately?
(I think the latter is more powerful? because you can override things?
but it opens the door to breaking your build)
2. Should we install these dependencies into the same environment? Or
should we layer it on top as we do elsewhere? (I think it's fine to
install into the same environment)
3. Should we respect sources defined in the parent project? (I think
yes, but then we need to lower the dependencies earlier — I don't think
that's a big deal, but it's not implemented)
4. Should we respect sources defined in the child project? (I think no,
this gets really complicated and seems weird to allow)
5. Should we apply this to transitive dependencies? (I think so)

---------

Co-authored-by: Aria Desires <aria.desires@gmail.com>
Co-authored-by: konstin <konstin@mailbox.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or improvement to existing functionality

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants