Skip to content

Create core module extensions #363

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mergify merged 13 commits into from
Apr 5, 2023
Merged

Create core module extensions #363

Changes from 1 commit
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
316ee14
Refine the module extension design document
aherrmann Mar 31, 2023
04f43d7
Define a helper module for hub repositories
aherrmann Mar 31, 2023
d3fa107
Add unit tests for the module registry
aherrmann Apr 3, 2023
35c240f
Module extension for Nix repositories
aherrmann Apr 3, 2023
3a3371a
Use the nix_repo module extension in testing
aherrmann Apr 3, 2023
41968db
WORKSPACE compatibility stub
aherrmann Apr 3, 2023
206a5ff
Module extension for Nix packages
aherrmann Apr 3, 2023
b144105
Use the nix_pkg module extension in testing
aherrmann Apr 3, 2023
0481bec
WORKSPACE compatibility stub
aherrmann Apr 3, 2023
bdf13ba
module extension for nixpkgs-git-repository-test
aherrmann Apr 3, 2023
2e3cf44
fix typos
aherrmann Apr 5, 2023
06e8c91
err --> fail_on_err
aherrmann Apr 5, 2023
33c59fd
Merge branch 'master' into bzlmod
mergify[bot] Apr 5, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
201 changes: 84 additions & 117 deletions design/bzlmod/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,8 +88,7 @@ Nix expressions, most commonly [nixpkgs].
E.g. `name = "nixpkgs-stable"`.\
* [`NIX_PATH` entry][nix-path] to Nix expressions.\
For angle-bracket reference in Nix expression, e.g. `import <nixpkgs>`.
E.g. `import_name = "nixpkgs"`.
Defaults to `import_name = name`.
E.g. `repositories = {"nixpkgs": ...}`.
* (to-consider) Alias tag to map repository to another `NIX_PATH` entry.
* (to-consider) Set default repository.\
Allowed in root or `rules_nixpkgs_core`.
Expand Down Expand Up @@ -253,6 +252,33 @@ Support the use of Nix built packages as Bazel toolchains.
[hub-and-spokes module][hub-and-spokes] or [hub repository][hub-repo] for
further details.

### Module Scope Repositories May be Added to Bazel

* Constraint
* A future follow-up of the [Automatic `use_repo` fixups for modul
Comment thread
aherrmann marked this conversation as resolved.
Outdated
extensions][auto-use-repo] proposal was discussed that could introduce
external workspaces generated by module extensions that are scoped to
specific Bazel modules. The constraint is that repository mappings need to
be calculable without loading the module extension, meaning they must be
fully defined in the `MODULE.bazel` files.
* Impact
* Repositories generated by the rules\_nixpkgs module extensions that are
globally unified, thereby potentially used by multiple Bazel modules,
should have a name that is not prefixed by any Bazel module name, i.e. not
scoped to any particular Bazel module. In the simplest case this can just
be the tag name. In light of the above proposal, these could be directly
imported via `use_repo`.
* Repositories generated by the rules\_nixpkgs module extensions that are
specific to the request Bazel module, i.e. only used by that module, should
be scoped to that particular module, e.g. by using a name that is prefixed
by that Bazel module's name and version. In future, this could be replaced
by the `use_local_repo` mechanism.
* Nix repositories could be passed to Nix package tags as labels imported via
`use_repo` or `use_local_repo` as described above. Contents of Nix packages
could be referenced by label directly in the same manner.

[auto-use-repo]: https://docs.google.com/document/d/1dj8SN5L6nwhNOufNqjBhYkk5f-BJI_FPYWKxlB3GAmA/edit?disco=AAAArdGBwhc

### Nixpkgs Repositories or Packages Have No Convenient Canonical Name

* Constraint
Expand Down Expand Up @@ -313,6 +339,10 @@ Support the use of Nix built packages as Bazel toolchains.
defaults.
* rules\_nixkgs module extensions should support targeted overrides of Nix
repositories and packages in transitive module dependencies.
Note, Bazel itself supports this via module overrides from the root module
Comment thread
aherrmann marked this conversation as resolved.
Outdated
Implementation of this feature is deferred until the need arises. If
Bazel's builtin mechanism is sufficient, then this feature will not be
implemented.

### Module Extensions Themselves Don't Generate Output

Expand Down Expand Up @@ -364,23 +394,18 @@ Support the use of Nix built packages as Bazel toolchains.

### Global Default Repository

rules\_nixpkgs\_core itself will define a global default `nixpkgs` repository.
rules\_nixpkgs\_core itself will define a global default `nixpkgs` repository,
any module can reference this global default repository like so.

```python
use_extension("//extensions:repository.bzl", "nix_repo")

nix_repo.global(name = "nixpkgs")
nix_repo.github(
name = "nixpkgs",
tag = "22.11",
sha256 = "ddc3428d9e1a381b7476750ac4dbea7a42885cbbe6e1af44b21d6447c9609a6f",
)
nix_repo.default(name = "nixpkgs")
Comment thread
aherrmann marked this conversation as resolved.
```

### Local Repository

Any Bazel module can define custom Nix repositories for local use.
The name may not collide with any global repositories.

```python
nix_repo.github(
Expand All @@ -392,11 +417,10 @@ nix_repo.github(

### Repository Override

The root module can define a global `nixpkgs` repository to override the
default set by rules\_nixpkgs\_core.
The root module can override the default set by rules\_nixpkgs\_core.

```python
nix_repo.global(name = "nixpkgs")
nix_repo.override(name = "nixpkgs")
nix_repo.http(
name = "nixpkgs",
url = "https://github.com/NixOS/nixpkgs/archive/1eeea1f1922fb79a36008ba744310ccbf96130e2.tar.gz",
Expand All @@ -405,19 +429,14 @@ nix_repo.http(
)
```

The root module can also override a local repository in a specific transitive
dependency module.
rules\_nixpkgs\_core uses the same mechanism to define the global default.

```python
nix_repo.override(
module_name = "some_module",
repository = "nixpkgs-unstable",
replacement = "custom-nixpkgs-unstable",
)
nix_repo.file(
name = "custom-nixpkgs-unstable",
import_name = "nixpkgs",
file = "//nix:custom-nixpkgs-unstable.nix",
```
nix_repo.override(name = "nixpkgs")
nix_repo.github(
name = "nixpkgs",
tag = "22.11",
sha256 = "ddc3428d9e1a381b7476750ac4dbea7a42885cbbe6e1af44b21d6447c9609a6f",
)
```

Expand All @@ -432,8 +451,8 @@ avoid diamond dependency issues, see above.
```python
use_extension("@rules_nixpkgs_core//extensions:package.bzl", "nix_pkg")

nix_pkg.attr(name = "jq")
nix_pkg.attr(name = "awk", attr = "gawk")
nix_pkg.attr(attr = "jq")
nix_pkg.attr(attr = "gawk")
```

### Local Nix Package
Expand All @@ -446,7 +465,7 @@ rules\_nixkgs will still generate two separate external repositories to import
the package for each module.

```python
nix_pkgs.local_attr(name = "jq", repository = "nixpkgs-unstable")
nix_pkgs.local_attr(name = "jq", repo = "nixpkgs-unstable")

nix_pkg.local_expr(
name = "awk",
Expand All @@ -456,39 +475,7 @@ gawk-with-extensions.override {
extensions = with gawkextlib; [ csv json ];
}
""",
repository = "nixpkgs-unstable",
)
```

### Package Override

The root module can override a unified Nix package to globally replace it by a
local Nix package.

```python
nix_pkgs.override(
attr = "jq",
replacement = "jq-latest",
)
nix_pkgs.local_attr(
name = "jq-latest",
attr = "jq",
repository = "nixpkgs-unstable",
)
```

The root module can also override a local package in a specific transitive
dependency module.

```python
nix_pkgs.override(
module_name = "some_module",
attr = "gawk",
replacement = "awk",
)
nix_pkgs.local_file(
name = "awk",
file = "//nix:gawk-with-extensions.nix",
repo = "nixpkgs-unstable",
)
```

Expand All @@ -499,10 +486,10 @@ nix_pkgs.local_file(
The `rules_nixpkgs_core` module exposes the module extension `nix_repo` which
offers tags to define Nix repositories:

* `github(name, import_name, org, repo, tag, commit)`\
* `default(name)`\
* `name`: `String`; Use this global default repository.
* `github(name, org, repo, tag, commit)`\
* `name`: `String`; unique name.
* `import_name`: optional, `String`; `NIX_PATH` entry name.\
Default: `name`.
* `org`: optional, `String`; The GitHub organization hosting the repository.\
Default: `NixOS`.
* `repo`: optional, `String`; The name of the GitHub repository.\
Expand All @@ -514,10 +501,8 @@ offers tags to define Nix repositories:
* `sha256`: optional, `String`; The SHA-256 hash of the downloaded archive.
* `integrity`: optional, `String`; Expected checksum of the archive, in
Subresource Integrity format.
* `http(name, import_name, url, urls, sha256, integrity, strip_prefix)`\
* `http(name, url, urls, sha256, integrity, strip_prefix)`\
* `name`: `String`; unique name.
* `import_name`: optional, `String`; `NIX_PATH` entry name.\
Default: `name`.
* `url`: optional, `String`; URL to download from.\
Specify one of `url` or `urls`.
* `urls`: optional, `String`; List of URLs to download from.\
Expand All @@ -526,29 +511,15 @@ offers tags to define Nix repositories:
* `integrity`: optional, `String`; Expected checksum of the archive, in
Subresource Integrity format.
* `strip_prefix`: optional, `String`; A directory prefix to strip from the extracted files.
* `file(name, import_name, file, file_deps)`\
* `file(name, file, file_deps)`\
* `name`: `String`; unique name.
* `import_name`: optional, `String`, `NIX_PATH` entry name.\
Default: `name`.
* `file`: `Label`; the file containing the Nix expression.
* `file_deps`: optional, List of `Label`, files required by `file`.
* `expr(name, import_name, expression)`\
* `expr(name, expression)`\
* `name`: `String`; unique name.
* `import_name`: optional, `String`, `NIX_PATH` entry name.\
Default: `name`.
* `expression`: `String`; the Nix expression.
* `alias(name, import_name, actual)`\
* `name`: `String`; unique name.
* `import_name`: optional, `String`; `NIX_PATH` entry name.\
Default: `name`.
* `actual`: `String`; Another repository.
* `default(repository)` (only allowed in root or `rules_nixpkgs_core`)\
* `repository`: `String`; The repository to expose as a global default.
* `override(module_name, name, actual)` (only allowed in root)\
* `module_name`: `String`; The module in which to apply the override.
* `name`: `String`; The name of the repository to override.
* `actual`: `String`; The name of the repository to replace it with.\
The root module must define a repository by that name.
* `override(repo)` (only allowed in rules\_nixpkgs\_core and root)\
* `repo`: `String`; The name of the repository to override.

All `name` attributes define a unique name for the given Nix repository within
the scope of the requesting module.
Expand All @@ -560,20 +531,18 @@ The extension generates a hub repository called `nixpkgs_repositories` that
exposes a macro from `//:defs.bzl` to access the imported repositories from the
scope of the calling module:

* `nix_repo(module_name, name)`\
* `nix_repo(module, name)`\
Attrs:
* `module_name`: `String`; name of the calling Bazel module.\
* `module`: `String`; name of the calling Bazel module.\
Needed until Bazel offers an API to infer the calling module.
See, [#17652][bazel-17652].\
Note, this is ambiguous for multi-version overrides.\
TODO: Handle multi-version overrides.
* `name`: `String`; name of the repository.\
This is the name used on the `nixpkgs_repository` tag.
This is the name used on the `nix_repo` tag.

Returns:\
`struct(import_name, label)`
* `import_name`: `String`; The name for the `NIX_PATH` entry.
* `label`: The resolved `Label` object to the repository.
The resolved `Label` object to the repository.

Users are not expected to invoke `nix_repo` directly. Instead, it will be
invoked by the package and toolchain module extensions to access the relevant
Expand All @@ -584,57 +553,55 @@ repositories.
The `rules_nixpkgs_core` module exposes the module extension `nix_pkg` which
offers tags to define Nix packages:

* `attr(name, attr, repository)` (globally unified)\
* `name`: `String`; unique name.
* `attr`: optional, `String`; the attribute path.\
Default: `name`.\
* `repository`: optional, `String`; repository to import from.\
Default: `nixpkgs`.\
Must be a global repository.
* `local_attr(name, attr, repository, build_file, build_file_content)`\
* `attr(attr)` (globally unified)\
* `attr`: `String`; the attribute path.\
* `local_attr(name, attr, repo, build_file, build_file_content)`\
* `name`: `String`; unique name.
* `attr`: optional, `String`; the attribute path.\
Default: `name`.
* `repository`: optional, `String`; repository to import from.\
* `repo`: optional, `String`; the `nixpkgs` repository to import from.\
Default: `nixpkgs`.
* `build_file`: optional, `Label`; `BUILD` file to write into the external
workspace.\
Specify at most one of `build_file` or `build_file_content`.
* `build_file_content`: optional, `Label`; `BUILD` file content to write into
the external workspace.\
Specify at most one of `build_file` or `build_file_content`.
* `local_file(name, attr, file, file_deps, repository, repositories)`\
* `local_file(name, attr, file, file_deps, repo, repos)`\
* `name`: `String`; unique name.
* `attr`: optional, `String`; the attribute path.\
Default: `name`.
* `file`: `Label`; the file containing the Nix expression.
* `file_deps`: optional, List of `Label`, files required by `file`.
* `repository`: optional, `String`; use this Nix repository.
* `repositories`: optional, List of `String`; use these Nix repositories.
* `repo`: optional, `String`; use this `nixpkgs` repository.
Equivalent to `repos = {"nixpkgs": repo}`.
Specify only one of `repo` or `repos`.
Default: `nixpkgs`.
* `repos`: optional, Dict of `String`; use these Nix repositories.
The dictionary key represents the name of the `NIX_PATH` entry.
Specify only one of `repo` or `repos`.
* `build_file`: optional, `Label`; `BUILD` file to write into the external
workspace.\
Specify at most one of `build_file` or `build_file_content`.
* `build_file_content`: optional, `Label`; `BUILD` file content to write into
the external workspace.\
* `local_expr(name, attribute, expression, repository, repositories)`\
* `local_expr(name, attr, expr, repo, repos)`\
* `name`: `String`; unique name.
* `attribute`: optional, `String`; the attribute path.\
* `attr`: optional, `String`; the attribute path.\
Default: `name`.
* `expression`: `String`; the Nix expression.
* `repository`: optional, `String`; use this Nix repository.
* `repositories`: optional, List of `String`; use these Nix repositories.
* `expr`: `String`; the Nix expression.
* `repo`: optional, `String`; use this `nixpkgs` repository.
Equivalent to `repos = {"nixpkgs": repo}`.
Specify only one of `repo` or `repos`.
Default: `nixpkgs`.
* `repos`: optional, Dict of `String`; use these Nix repositories.
The dictionary key represents the name of the `NIX_PATH` entry.
Specify only one of `repo` or `repos`.
* `build_file`: optional, `Label`; `BUILD` file to write into the external
workspace.\
Specify at most one of `build_file` or `build_file_content`.
* `build_file_content`: optional, `Label`; `BUILD` file content to write into
the external workspace.\
* `override(attr, repository, replacement, module_name)` (only allowed in root)\
* `attr`: `String`; the attribute path of the package to replace.
* `repository`: optional, `String`; the repository of the package to replace.\
Default: `nixpkgs`.
* `replacement`: `String`; the name of the replacement package.\
* `module_name`; optional, `String`; only override a local package in this
module, if specified; otherwise, replace the package globally,

All `name` attributes define a unique name for the given Nix repository within
the scope of the requesting module.
Expand All @@ -646,9 +613,9 @@ The extension generates a hub repository called `nixpkgs_packages` that exposes
a macro from `//:defs.bzl` to access the imported packages from the scope of
the calling module:

* `nix_pkg(module_name, name, label)`\
* `nix_pkg(module, name, label)`\
Attrs:
* `module_name`: `String`; name of the calling Bazel module.\
* `module`: `String`; name of the calling Bazel module.\
Needed until Bazel offers an API to infer the calling module.
See, [#17652][bazel-17652].\
Note, this is ambiguous for multi-version overrides.\
Expand Down