Better defaults for native build backend cache keys

[rimathia]

Summary

This change extends the default initialized projects with scikit and maturin build backends to ensure that the rust or C++ build gets invoked when a source file changes.

Test Plan

cargo run init cppextension-test --build-backend=scikit followed by manual testing of the behaviour of the resulting project under uv run --with jupyter jupyter lab and changes to the source file of the extension.
Analogous for cargo run init maturin-test --build-backend=maturin .

Relevant Issues

The question of why the python extension is not rebuilt on source changes has been discussed in #15701 (comment).

[rimathia]

It is of course a judgement call whether the rebuild on rust or C++ source file changes is more likely to meet developer expectations than the rebuild on --reinstall. Just decline the PR if you disagree with my premise.

[zanieb]

cc @charliermarsh / @konstin

I don't have strong feelings here.

[konstin]

I think this is a good idea, but we should change the cache keys to something that captures all Rust/C/C++ source files, such as:

cache-keys = [{ file = "pyproject.toml" }, { file = "src/**/*.rs" }, { file = "Cargo.toml" }, { file = "Cargo.lock" }]

[rimathia]

Thanks for the suggestion, I was deterred by the documentation mentioning that glob patterns can be costly to check, but in a source directory it definitely makes sense. Unfortunately, C++ doesn't have a strong filename convention, so I just went with {cpp,h}, hoping that people will become aware of that line in the pyproject.toml before wondering why a change to their .hpp file doesn't trigger recompilation.

[konstin]

We can do {h,c,hpp,cpp}. The costly part of the glob is traversing many files, the regex part of it is really fast. For each directory we look, we need to make a syscall, and for each syscall, there needs to be a random IO read. These are still fast for most source tree, but it's easy to misconfigure the glob and e.g. traverse an entire venv or data or cache directory with tenthousands of small files, where it becomes slow, hence the warning.

[Ravencentric]

Should pyproject.toml and requirements.txt be a part of the cache-key by default in this case? I don't think adding/removing/updating a python dependency or changing the configuration of a completely unrelated tool should rebuild the expensive native code.

[konstin]

Speaking from a maturin perspective, pyproject.toml should be included, settings can be defined in this file, but requirements.txt isn't used.

CC @henryiii Do you have thoughts about which files should cause a scikit-build-core project to be rebuilt?

[konstin]

Don't we need the double {{ and }} here also?

[rimathia]

It turns out that at present, the Scikit project string does not use format! like the Maturin project string does. As a result, {wheel_tag} is still present in the resulting pyproject.toml. I'll have to investigate why this even works.

[henryiii]

That's correct, scikit-build-core has to inject the wheel tag there when it builds, since you don't know the wheel tag before you start building a wheel. You want {wheel_tag} in the pyproject.toml.

[henryiii]

pyproject.toml (at least some tables, like project, build-system, and tool.scikit-build) makes sense for scikit-build-core as well. **/CMakeLists.txt, **.cmake would make sense too. We don't have dedicated support for presets yet, but CMakePreset.json and CMakeUserPreset.json also can affect the builds (and we'll have better support at some point). I've also seen the .C convention for C++ (pretty rare, but was somewhat common in the early nineties, so still pops up occasionally, especially with ROOT in HEP). I've also seen .inc for includes that aren't headers.

[henryiii]

Oh, also the fortran extensions (.f, .f90, .F90, .F), and cuda (.cu, .cuh). I think those are the most common languages, though CMake also supports others, like Swift.

[konstin]

Thanks @henryiii!

For the PR, further file extensions are cheap to support, we can cover as many of them as we want. I would avoid **/CMakeLists.txt and **.cmake as the require traversing everything, we need to limit those more if we want to include them.

[henryiii]

How is that different from source file extensions? At least .cmake literally is an extension.

[henryiii]

Oh, I think I see. Hadn’t looked at the code. src/**/{*.h,*.c,*.hpp,*.cpp,*.cmake,CMakeLists.txt} might work?

[zanieb]

I'm sort of wary to go far beyond the files types that we're generating in the initial template.

[henryiii]

I think a nice minimal set then might be ..., "{ file = "src/**/*.{h,c,hpp,cpp}" }, { file = "CMakeLists.txt" }]", then? Since it's part of the template, that seems fine to have it minimal.

[rimathia]

Thanks for giving me specifics, I hope I've understood the emerging consensus.

[konstin]

Let's start with these cache keys and see how it goes.

[konstin]

Thank you!

[zanieb]

I think this is missing a docs update in https://docs.astral.sh/uv/concepts/projects/init/#projects-with-extension-modules where we say you'll need to use --reinstall when main.cpp changes