Skip to content

Conversation

@arrdem
Copy link
Collaborator

@arrdem arrdem commented Jan 8, 2026

The Python ecosystem presently lacks (PEP-720) a comprehensive crossbuilding story.

As a workaround for this and issues which could arise by "faking" crossbuilds eg by lying about the current interpreter platform, we can instead lean on RBE to ignore the issue by executing sdist to bdist build actions on workers matching the target configuration.

With thanks to @fmeum for the guidance, this is achieved by using Bazel's execution groups feature. We define a dummy toolchain of no files which can be resolved only in a configuration matching the target configuration, and consequently inherits the exec_properties of the target platform. Then we create an execution group which contains that dummy toolchain. ctx.actions.run calls referencing an exec group inherit the exec properties of the chosen platform for which the toolchains in the group were resolved. Thus we have a way to enforce that the action will be placed on a host matching the target's configuration.

The new e2e case exercises this behavior by forcibly marking a dependency which has no bdists (it was installed via sdist url) as being native and thus requiring platform-specific builds which must occur as platform-specific actions. This behavior can be observed in the following query

bazel aquery \
   --platforms=//cases/uv-deps-650/source_crossbuild:arm64_linux \
   --host_platform=//cases/uv-deps-650/source_crossbuild:amd64_linux \
   --extra_execution_platforms=//cases/uv-deps-650/source_crossbuild/... \
   'deps(cases/uv-deps-650/source_crossbuild:image_index)'

The image index depends on both configurations of the binary, each of which has a dependency on the "native" source build of cowsay. Consequently we expect to see two PySdistBuild actions,

In order to make performing these builds more efficient in RBE, this patch also reworks the sdist build helper so that we use Python to unzip the sdist archive in the same action step as we perform the build. In RBE this eliminates the time and transit overhead of using one action to run tar.bzl to unzip the sdist only to re-pack the resulting tree artifact and shovel it to a different worker for the subsequent sdist build action.

Changes are visible to end-users: yes

  • Searched for relevant documentation and updated as needed: yes
  • Breaking change (forces users to change their own code or config): no
  • Suggested release notes appear below: yes

The UV extension will now attempt to place sdist builds of native extensions on the target platform via RBE.

Test plan

See #771 for the test case code.

Inspect the build plan for the image index, which should show one sdist build on arm64 and one on amd64.

❯ bazel aquery \
    --platforms=//cases/uv-deps-650/source_crossbuild:arm64_linux \
    --extra_execution_platforms=//cases/uv-deps-650/source_crossbuild/... \
    'deps(cases/uv-deps-650/source_crossbuild:image_index)' \
    | grep -A14 'PySdistBuild' | sed 's/Inputs: \(\[.*\]\)/Inputs: .../g' 

Mnemonic: PySdistBuild
  Target: @@aspect_rules_py++uv+sbuild__pypi__source-build__cowsay//:whl
  Configuration: k8-fastbuild-ST-93c492559deb
  Execution platform: //cases/uv-deps-650/source_crossbuild:arm64_linux
  ActionKey: 4d5bf8cafcebd3c553c2e38548995188c20986fd614ce920ba4e90f351ac255f
  Inputs: ...
  Outputs: [bazel-out/k8-fastbuild-ST-93c492559deb/bin/external/aspect_rules_py++uv+sbuild__pypi__source-build__cowsay/build (TreeArtifact)]
  Command Line: (exec bazel-out/k8-fastbuild-ST-93c492559deb/bin/external/aspect_rules_py++uv+sbuild__pypi__source-build__cowsay/.build_venv/bin/python3 \
    external/aspect_rules_py+/uv/private/sdist_build/build_helper.py \
    external/aspect_rules_py++uv+sdist__cowsay__47445cb2/file/cowsay-6.0.tar.gz \
    bazel-out/k8-fastbuild-ST-93c492559deb/bin/external/aspect_rules_py++uv+sbuild__pypi__source-build__cowsay/build)
# Configuration: 7329af2666e1bf46b2f1b89d88d6e90d5311d6fd69df686f2a70377db1254561
# Execution platform: //cases/uv-deps-650/source_crossbuild:arm64_linux
  ExecutionInfo: {Arch: arm64, OSFamily: Linux}

--
  Mnemonic: PySdistBuild
  Target: @@aspect_rules_py++uv+sbuild__pypi__source-build__cowsay//:whl
  Configuration: k8-fastbuild-ST-01f0fdcf49e8
  Execution platform: //cases/uv-deps-650/source_crossbuild:amd64_linux
  ActionKey: 0b683ffca3367a28a07bee2fdc24a0913268d83a46b3444cc9f6a3e724571807
  Inputs: ...
  Outputs: [bazel-out/k8-fastbuild-ST-01f0fdcf49e8/bin/external/aspect_rules_py++uv+sbuild__pypi__source-build__cowsay/build (TreeArtifact)]
build__cowsay/.build_venv/bin/python3 \
    external/aspect_rules_py+/uv/private/sdist_build/build_helper.py \
    external/aspect_rules_py++uv+sdist__cowsay__47445cb2/file/cowsay-6.0.tar.gz \
    bazel-out/k8-fastbuild-ST-01f0fdcf49e8/bin/external/aspect_rules_py++uv+sbuild__pypi__source-build__cowsay/build)
# Configuration: 9475f7f1bf5753f90f43a325cac5dd2ab8bef1567797b087a6cb1a77b871b825
# Execution platform: //cases/uv-deps-650/source_crossbuild:amd64_linux
  ExecutionInfo: {Arch: amd64, OSFamily: Linux}

@aspect-workflows
Copy link

aspect-workflows bot commented Jan 8, 2026

Bazel 8 (Test)

All tests were cache hits

38 tests (100.0%) were fully cached saving 1m.


Bazel 9 (Test)

All tests were cache hits

35 tests (100.0%) were fully cached saving 1m 32s.


Bazel 8 (Test)

e2e

1 test target passed

Targets
//cases/uv-deps-650/say:say [k8-fastbuild]634ms

Total test execution time was 634ms. 13 tests (92.9%) were fully cached saving 11s.


Bazel 9 (Test)

e2e

1 test target passed

Targets
//cases/uv-deps-650/say:say [k8-fastbuild]481ms

Total test execution time was 481ms. 11 tests (91.7%) were fully cached saving 6s.


Bazel 8 (Test)

examples/uv_pip_compile

All tests were cache hits

1 test (100.0%) was fully cached saving 335ms.

@arrdem arrdem force-pushed the arrdem/uv-sdist-builds-with-exec-groups branch from 3e9ed4f to 2dced8d Compare January 8, 2026 22:29
@arrdem arrdem force-pushed the arrdem/uv-sdist-builds-with-exec-groups branch from ee0bafb to 7a3dbaa Compare January 15, 2026 20:07
@arrdem arrdem merged commit e85ed83 into main Jan 15, 2026
4 checks passed
@arrdem arrdem deleted the arrdem/uv-sdist-builds-with-exec-groups branch January 15, 2026 20:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants