diff --git a/docs/howto/multi-platform-pypi-deps.md b/docs/howto/multi-platform-pypi-deps.md new file mode 100644 index 0000000000..6cc7f842ea --- /dev/null +++ b/docs/howto/multi-platform-pypi-deps.md @@ -0,0 +1,194 @@ +:::{default-domain} bzl +::: + +# How-to: Multi-Platform PyPI Dependencies + +When developing applications that need to run on a wide variety of platforms, +managing PyPI dependencies can become complex. You might need different sets of +dependencies for different combinations of Python version, threading model, +operating system, CPU architecture, libc, and even hardware accelerators like +GPUs. + +This guide demonstrates how to manage this complexity using `rules_python` with +bzlmod. If you prefer to learn by example, complete example code is provided at +the end. + +In this how to guide, we configure for using 4 requirements files, each +for a different variation using Python 3.14 on Linux: + +* Regular (non-freethreaded) Python +* Freethreaded Python +* Regular Python for CUDA 12.9 +* Freethreaded Python for ARM and Musl + +## Mapping requirements files to Bazel configuration settings + +Unfortunately, a requirements file doesn't tell what it's compatible with, +so we have to manually specify the Bazel configuration settings for it. To do +that using rules_python, there are two steps: defining a platform, then +associating a requirements file with the platform. + +### Defining a platform + +First, we define a "platform" using {obj}`pip.default`. This associates an +arbitrary name with a list of Bazel {obj}`config_setting` targets. While any +name can be used for a platform (its name has no inherent semantic meaning), it +should encode all the relevant dimensions that distinguish a requirements file. +For example, if a requirements file is specifically for the combination of CUDA +12.9 and NumPy 2.0, then the platform name should represent that. + +The convention is to follow the format of `{os}_{cpu}{threading}`, where: + +* `{os}` is the operating system (`linux`, `osx`, `windows`). +* `{cpu}` is the architecture (`x86_64`, `aarch64`). +* `{threading}` is `_freethreaded` for a freethreaded Python runtime, or an + empty string for the regular runtime. + +Additional dimensions should be appended and separated with an underscore (e.g. +`linux_x86_64_musl_cuda12.9_numpy2`). + +The platform name should not include the Python version. That is handled by +`pip.parse.python_version` separately. + +:::{note} +The term _platform_ here has nothing to do with Bazel's `platform()` rule. +::: + +#### Defining custom settings + +Because {obj}`pip.parse.config_settings` is a list of arbitrary `config_setting` +targets, you can define your own flags or implement custom config matching +logic. This allows you to model settings that aren't inherently part of +rules_python. + +This is typically done using [bazel_skylib flags](https://bazel.build/extending/config), but any [Starlark +defined build setting](https://bazel.build/extending/config) can be used. Just +remember to use `config_setting()` to match a particular value of the flag. + +In our example below, we define a custom flag for CUDA version. + +#### Predefined and common build settings + +rules_python has some predefined build settings you can use. Commonly used ones +are: + +* {obj}`@rules_python//python/config_settings:py_linux_libc` +* {obj}`@rules_python//python/config_settings:py_freethreaded` + +Additionally, [Bazel @platforms](https://github.com/bazelbuild/platforms) +contains commonly used settings for OS and CPU: + +* `@platforms//os:windows` +* `@platforms//os:linux` +* `@platforms//os:osx` +* `@platforms//cpu:x86_64` +* `@platforms//cpu:aarch64` + +Note that these are the raw flag names. In order to use them with `pip.default`, +you must use {obj}`config_setting()` to match a particular value for them. + +### Associating Requirements to Platforms + +Next, we associate a requirements file with a platform using +{obj}`pip.parse.requirements_by_platform`. This is a dictionary attribute where +the keys are requirements files and the value is a platform name. The platform +value can use a trailing or leading `*` to match multiple platforms. It can also +specify multiple platform names using commas to separate them. + +Note that the Python version is _not_ part of the platform name. + +Under the hood, `pip.parse` merges all the requirements (for a `hub_name`) and +constructs `select()` expressions to route to the appropriate dependencies. + +### Using it in practice + +Finally, to make use of what we've configured, perform a build and set +command line flags to the appropriate values. + +```shell +# Build for CUDA +bazel build --//:cuda_version=12.9 //:binary + +# Build for ARM with musl +bazel build --@rules_python//python/config_settings:py_linux_libc=musl \ + --cpu=aarch64 //:binary + +# Build for freethreaded +bazel build --@rules_python//python/config_settings:py_freethreaded=yes //:binary +``` + +Note that certain combinations of flags may result in an error or undefined +behavior. For example, trying to set both freethreaded and CUDA at the same +time would result in an error because no requirements file was registered +to match that combination. + +## Multiple Python Versions + +Having multiple Python versions is fully supported. Simply add a `pip.parse()` +call and set `python_version` appropriately. + +## Multiple hubs + +Having multiple `pip.parse` calls with different `hub_name` values is fully +supported. Each hub only contains the requirements registered to it. + +## Complete Example + +Here is a complete example that puts all the pieces together. + +```starlark +# File: BUILD.bazel +load("@bazel_skylib//rules:common_settings.bzl", "string_flag") + +# A custom flag for controlling the CUDA version +string_flag( + name = "cuda_version", + build_setting_default = "none", +) + +config_setting( + name = "is_cuda_12_9", + flag_values = {":cuda_version": "12.9"}, +) + +# A config_setting that uses the built-in libc flag from rules_python +config_setting( + name = "is_musl", + flag_values = {"@rules_python//python/config_settings:py_linux_libc": "muslc"}, +) + +# File: MODULE.bazel +pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip") + +# A custom platform for CUDA on glibc linux +pip.default( + platform = "linux_x86_64_cuda12.9", + os = "linux", + cpu = "x86_64", + config_settings = ["@//:is_cuda_12_9"], +) + +# A custom platform for musl on linux +pip.default( + platform = "linux_aarch64_musl", + os = "linux", + cpu = "aarch64", + config_settings = ["@//:is_musl"], +) + +pip.parse( + hub_name = "my_deps", + python_version = "3.14", + requirements_by_platform = { + # Map to default platform names + "//:py3.14-regular-linux-x86-glibc-cpu.txt": "linux_x86_64", + "//:py3.14-freethreaded-linux-x86-glibc-cpu.txt": "linux_x86_64_freethreaded", + + # Map to our custom platform names + "//:py3.14-regular-linux-x86-glibc-cuda12.9.txt": "linux_x86_64_cuda12.9", + "//:py3.14-freethreaded-linux-arm-musl-cpu.txt": "linux_aarch64_musl", + }, +) + +use_repo(pip, "my_deps") +``` diff --git a/docs/pypi/index.md b/docs/pypi/index.md index c32bafc609..17928898c5 100644 --- a/docs/pypi/index.md +++ b/docs/pypi/index.md @@ -11,6 +11,7 @@ Using PyPI packages (aka "pip install") involves the following main steps: With the advanced topics covered separately: * Dealing with [circular dependencies](./circular-dependencies). +* Handling [multi-platform dependencies](../howto/multi-platform-pypi-deps). ```{toctree} lock @@ -22,6 +23,9 @@ use ## Advanced topics ```{toctree} +:maxdepth: 1 + circular-dependencies patch +../howto/multi-platform-pypi-deps ``` diff --git a/python/private/pypi/pip_repository.bzl b/python/private/pypi/pip_repository.bzl index 6d539a5f24..2cf20cd5a7 100644 --- a/python/private/pypi/pip_repository.bzl +++ b/python/private/pypi/pip_repository.bzl @@ -266,6 +266,9 @@ code will be re-evaluated when any of files in the default changes. Those dependencies become available in a generated `requirements.bzl` file. You can instead check this `requirements.bzl` file into your repo, see the "vendoring" section below. +For advanced use-cases, such as handling multi-platform dependencies, see the +[How-to: Multi-Platform PyPI Dependencies guide](/howto/multi-platform-pypi-deps). + In your WORKSPACE file: ```starlark