Port to scikit-build-core #397

maxbachmann · 2024-08-01T21:26:44Z

@henryiii this is a first attempt at porting to scikit-build-core. It did work on my machine, but still has to be tested on things like android where no cmake wheels are available.

maxbachmann · 2024-08-01T21:36:01Z

I should have ported over all build system features I was previously using and didn't run into any problems so far.

Can you have a quick look over the build system?

maxbachmann · 2024-08-01T21:37:44Z

Amazing experience so far. This allows me to get rid of quite a few things in the build system. From a development perspective I am really pleased with the fact that I don't have to delete the _skbuild directory between each build anymore 👍

codecov · 2024-08-01T21:54:39Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 82.87%. Comparing base (d8238f5) to head (344c3cd).

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #397   +/-   ##
=======================================
  Coverage   82.87%   82.87%           
=======================================
  Files          46       46           
  Lines        4496     4496           
=======================================
  Hits         3726     3726           
  Misses        770      770

Flag	Coverage Δ
unittests	`82.87% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

pyproject.toml

tools/sdist.patch

maxbachmann · 2024-08-02T20:10:32Z

In the released version I generate the cython files and ship them as part of the sdist. These generated files get the cxx file ending and are gitignored. The patch is applied before packing the sdist to remove cython from the dependency list since it's not required if the generated files are packed.

pyproject.toml

henryiii · 2024-08-06T20:12:20Z

FYI, 0.10 is out. It takes a bit for it to filter down everywhere (conda takes ~24 hours to pick it up), but just letting you know it's becoming available. :)

pyproject.toml

tools/sdist.patch

henryiii · 2024-08-06T21:10:03Z

pyproject.toml

+if.any.env.PIWHEELS_BUILD = true
+if.any.env.RAPIDFUZZ_BUILD_EXTENSION = true
+wheel.cmake = true
+error-message = "failed to build C++ Extension in a packaged build"


Suggested change

error-message = "failed to build C++ Extension in a packaged build"

messages.after-success = "{yellow}Failed to build C++ Extension in a packaged build, using Python fallback."

FYI, an override that never matches won't be validated, so this likely never matched in your tests.

The goal was that it would:

print a warning when falling back to the pure Python fallback in

[[tool.scikit-build.overrides]] if.failed = true wheel.cmake = false

so I guess this would have to be added there

in case the environment variables are set I actually want the build to fail completely + print an error message

I don't have any tests for this in github actions yet since this requires a platform where cmake wheels are not available. Do you know an easy way to test this in github actions?

I misunderstood your correction back then. It should certainly use messages.*. In which context would this color actually have an effect?

This is a success / failure message. If the override matches and the pure Python wheel is built, it "succeeds". And the color is the same color system that scikit-build-core uses, it shows up with pypa/build directly and with buffered builders like pip if you set FORCE_COLOR.

Co-authored-by: Henry Schreiner <[email protected]>

maxbachmann · 2024-09-19T23:13:33Z

@henryiii I took this up again and started to test the fallback handling.

As a first step I did add invalid code in C++ to test the Python fallback. This falls back, but then crashes:

  *** CMake build failed - retrying due to override...
  Traceback (most recent call last):
    File "/tmp/pip-build-env-ca_6j1zw/overlay/lib/python3.12/site-packages/scikit_build_core/build/wheel.py", line 175, in _build_wheel_impl
      return _build_wheel_impl_impl(
             ^^^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-ca_6j1zw/overlay/lib/python3.12/site-packages/scikit_build_core/build/wheel.py", line 423, in _build_wheel_impl_impl
      builder.build(build_args=build_args)
    File "/tmp/pip-build-env-ca_6j1zw/overlay/lib/python3.12/site-packages/scikit_build_core/builder/builder.py", line 265, in build
      self.config.build(
    File "/tmp/pip-build-env-ca_6j1zw/overlay/lib/python3.12/site-packages/scikit_build_core/cmake.py", line 264, in build
      self._build(*local_args, *build_args)
    File "/tmp/pip-build-env-ca_6j1zw/overlay/lib/python3.12/site-packages/scikit_build_core/cmake.py", line 275, in _build
      raise FailedLiveProcessError(msg) from None
  scikit_build_core.errors.FailedLiveProcessError: CMake build failed

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/usr/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
      main()
    File "/usr/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
      return _build_backend().build_wheel(wheel_directory, config_settings,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-ca_6j1zw/overlay/lib/python3.12/site-packages/scikit_build_core/build/__init__.py", line 31, in build_wheel
      return _build_wheel_impl(
             ^^^^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-ca_6j1zw/overlay/lib/python3.12/site-packages/scikit_build_core/build/wheel.py", line 204, in _build_wheel_impl
      return _build_wheel_impl_impl(
             ^^^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-ca_6j1zw/overlay/lib/python3.12/site-packages/scikit_build_core/build/wheel.py", line 232, in _build_wheel_impl_impl
      metadata = get_standard_metadata(pyproject, settings)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-ca_6j1zw/overlay/lib/python3.12/site-packages/scikit_build_core/build/metadata.py", line 35, in get_standard_metadata
      raise KeyError(msg)
  KeyError: 'version is not in project.dynamic'

I did attempt a couple of other things:

hard code the version using version = "3.9.7" + removing tool.scikit-build.metadata.version and dynamic = ["version"]. This still fails with:

  *** CMake build failed - retrying due to override...
  *** scikit-build-core 0.10.6 (wheel)
  *** Making wheel...
  2024-09-20 01:10:50,905 - scikit_build_core - ERROR - Metadata mismatch in WHEEL: b'Wheel-Version: 1.0\nGenerator: scikit-build-core 0.10.6\nRoot-Is-Purelib: false\nTag: cp312-cp312-linux_x86_64\n\n' != b'Wheel-Version: 1.0\nGenerator: scikit-build-core 0.10.6\nRoot-Is-Purelib: true\nTag: py3-none-any\n\n'
  Traceback (most recent call last):
    File "/tmp/pip-build-env-zd1ax4eg/overlay/lib/python3.12/site-packages/scikit_build_core/build/wheel.py", line 175, in _build_wheel_impl
      return _build_wheel_impl_impl(
             ^^^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-zd1ax4eg/overlay/lib/python3.12/site-packages/scikit_build_core/build/wheel.py", line 423, in _build_wheel_impl_impl
      builder.build(build_args=build_args)
    File "/tmp/pip-build-env-zd1ax4eg/overlay/lib/python3.12/site-packages/scikit_build_core/builder/builder.py", line 265, in build
      self.config.build(
    File "/tmp/pip-build-env-zd1ax4eg/overlay/lib/python3.12/site-packages/scikit_build_core/cmake.py", line 264, in build
      self._build(*local_args, *build_args)
    File "/tmp/pip-build-env-zd1ax4eg/overlay/lib/python3.12/site-packages/scikit_build_core/cmake.py", line 275, in _build
      raise FailedLiveProcessError(msg) from None
  scikit_build_core.errors.FailedLiveProcessError: CMake build failed

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/usr/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
      main()
    File "/usr/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
      return _build_backend().build_wheel(wheel_directory, config_settings,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-zd1ax4eg/overlay/lib/python3.12/site-packages/scikit_build_core/build/__init__.py", line 31, in build_wheel
      return _build_wheel_impl(
             ^^^^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-zd1ax4eg/overlay/lib/python3.12/site-packages/scikit_build_core/build/wheel.py", line 204, in _build_wheel_impl
      return _build_wheel_impl_impl(
             ^^^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/pip-build-env-zd1ax4eg/overlay/lib/python3.12/site-packages/scikit_build_core/build/wheel.py", line 508, in _build_wheel_impl_impl
      raise AssertionError(msg)
  AssertionError: Metadata mismatch in WHEEL
  error: subprocess-exited-with-error

I then tried to change the order to:

[[tool.scikit-build.overrides]]
if.failed = true
wheel.cmake = false

[[tool.scikit-build.overrides]]
if.any.system-cmake = ">=3.15"
if.any.cmake-wheel = true
wheel.cmake = true

my expectation here was that the second is implicitly if.failed = false and so this would still fallback to Python. However it attempts to build using cmake twice. It's unclear to me why this happens.

henryiii · 2024-09-20T03:49:45Z

Is the current state of the PR the one that produces your first traceback? Might be an issue with pyproject-metadata - I've just rewritten the way this works in pyproject-metadata, so we might get this fixed for free by upgrading.

For the second one, that might be tricky. prepare_metadata_for_build_wheel is required to return the same metadata that the wheel build itself produces. But you can't know that CMake is going to fail. We could either check to see if there's an override that uses if.failed, and not provide this method if that's the case (it's great that it's only an override and not a config-setting!), or we could ease up on our check and not verify some fields like Root-Is-Purelib: false and Tag are included in the verification. Or we could reduce the verification to a warning (this technically must have also happened with setuptools + skbuild before the way you were doing it, we just didn't verify back then).

Also we produce the correct error message at the top, I should see if we can get rid of the traceback.

my expectation here was that the second is implicitly if.failed = false and so this would still fallback to Python. However it attempts to build using cmake twice. It's unclear to me why this happens.

Isn't that still what this is asking for? The order only matters if both match, then the second replaces any fields defined (unless you set inherit).

henryiii · 2024-09-20T04:01:20Z

uv venv --python python3.13
uv pip install -e ../../scikit-build/scikit-build-core
FORCE_COLOR=1 uv build --no-build-isolation --wheel

This reproduces, and you even get color in Python 3.13's traceback too! :)

And something is modifying the pyproject dictionary, dynamic.version is empty at this point.

henryiii · 2024-09-20T04:06:20Z

Ah, bug, and before it gets to pyproject-metadata. It's making a shallow copy but then modifying project.dynamic which is the same array. A deep copy fixes that bug.

I didn't get the collision in metadata, though, what tool are you using? Ahh, uv must not call the (optional) metadata hook. Pretty sure build doesn't either. But pip does.

henryiii reviewed Aug 2, 2024

View reviewed changes

pyproject.toml Outdated Show resolved Hide resolved

henryiii reviewed Aug 2, 2024

View reviewed changes

pyproject.toml Show resolved Hide resolved

henryiii reviewed Aug 2, 2024

View reviewed changes

tools/sdist.patch Show resolved Hide resolved

henryiii reviewed Aug 6, 2024

View reviewed changes

pyproject.toml Outdated Show resolved Hide resolved

henryiii reviewed Aug 6, 2024

View reviewed changes

pyproject.toml Outdated Show resolved Hide resolved

pyproject.toml Show resolved Hide resolved

pyproject.toml Show resolved Hide resolved

tools/sdist.patch Outdated Show resolved Hide resolved

henryiii reviewed Aug 6, 2024

View reviewed changes

maxbachmann force-pushed the scikit-build-core branch 2 times, most recently from a8e0bc6 to 0b0e045 Compare August 7, 2024 09:09

maxbachmann force-pushed the scikit-build-core branch from 0b0e045 to 9fc8891 Compare September 19, 2024 23:01

maxbachmann and others added 11 commits September 20, 2024 01:04

port to scikit-build-core

13561cc

fix build

2d3c291

update patch

70edc21

fix project properties

aa7b4d3

Update pyproject.toml

fb9ff36

Update pyproject.toml

69aa805

Co-authored-by: Henry Schreiner <[email protected]>

Update pyproject.toml

1cf3643

Co-authored-by: Henry Schreiner <[email protected]>

Update pyproject.toml

615a9ac

Co-authored-by: Henry Schreiner <[email protected]>

Update tools/sdist.patch

0bc454b

Co-authored-by: Henry Schreiner <[email protected]>

fix patch

1d1abe8

further cleanup

35b379a

maxbachmann force-pushed the scikit-build-core branch from 9fc8891 to 35b379a Compare September 19, 2024 23:05

maxbachmann added 2 commits September 20, 2024 01:19

fix failure message

03e5167

add warning message

344c3cd

This was referenced Sep 20, 2024

fix: avoid modifing the input dict scikit-build/scikit-build-core#903

Open

fix: avoid providing prepare-metadata methods if rebuild is True scikit-build/scikit-build-core#904

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Port to scikit-build-core #397

Port to scikit-build-core #397

maxbachmann commented Aug 1, 2024

maxbachmann commented Aug 1, 2024

maxbachmann commented Aug 1, 2024

codecov bot commented Aug 1, 2024 •

edited

Loading

maxbachmann commented Aug 2, 2024

henryiii commented Aug 6, 2024

henryiii Aug 6, 2024 •

edited

Loading

henryiii Aug 6, 2024

maxbachmann Aug 7, 2024

maxbachmann Sep 19, 2024

henryiii Sep 20, 2024

maxbachmann commented Sep 19, 2024 •

edited

Loading

henryiii commented Sep 20, 2024

henryiii commented Sep 20, 2024

henryiii commented Sep 20, 2024 •

edited

Loading

	error-message = "failed to build C++ Extension in a packaged build"
	messages.after-success = "{yellow}Failed to build C++ Extension in a packaged build, using Python fallback."

Port to scikit-build-core #397

Are you sure you want to change the base?

Port to scikit-build-core #397

Conversation

maxbachmann commented Aug 1, 2024

maxbachmann commented Aug 1, 2024

maxbachmann commented Aug 1, 2024

codecov bot commented Aug 1, 2024 • edited Loading

Codecov Report

maxbachmann commented Aug 2, 2024

henryiii commented Aug 6, 2024

henryiii Aug 6, 2024 • edited Loading

Choose a reason for hiding this comment

henryiii Aug 6, 2024

Choose a reason for hiding this comment

maxbachmann Aug 7, 2024

Choose a reason for hiding this comment

maxbachmann Sep 19, 2024

Choose a reason for hiding this comment

henryiii Sep 20, 2024

Choose a reason for hiding this comment

maxbachmann commented Sep 19, 2024 • edited Loading

henryiii commented Sep 20, 2024

henryiii commented Sep 20, 2024

henryiii commented Sep 20, 2024 • edited Loading

codecov bot commented Aug 1, 2024 •

edited

Loading

henryiii Aug 6, 2024 •

edited

Loading

maxbachmann commented Sep 19, 2024 •

edited

Loading

henryiii commented Sep 20, 2024 •

edited

Loading