Integrate the kustomize tool into your Bazel projects to build Kubernetes-style manifests.
Contents
These Bazel rules allow you to define kustomizations within your project workspace and capture the output of building those kustomizations with the kustomize tool. Using kustomize within a Bazel project orchestrates both the preparation of the input files—for when static files won't do—and consumption of the resulting YAML document stream, situating kustomize as a transformer in the middle of that data flow.
kustomize operates on the Kubernetes Resource Model (KRM), and most often comes into play just ahead of Kubernetes API clients like kubectl, sending the YAML document stream to the Kubernetes API to apply the resources to the cluster. These rules do no include any such interaction with the Kubernetes API. Instead, they focus solely on emitting the resources defined by the kustomization.
Here we'll walk through a simple example to give a feel for how to use these Bazel rules.
Assume you have a kustomization that defines a single Kubernetes ConfigMap, taking its data entries from both a file containing "environment variable"-style line-by-line definitions and some inline definitions in the kustomization.yaml file. First, the kustomization.yaml file:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
configMapGenerator:
- name: translations
envs:
- config-bindings
literals:
- HELLO=hola
- GOODBYE=adiosNote that the "envs" field references a sibling file named config-bindings, with content as follows:
THANK_YOU=gracias
If we run the kustomize build command against the directory containing these two files, it emits the following YAML document:
apiVersion: v1
data:
GOODBYE: adios
HELLO: hola
THANK_YOU: gracias
kind: ConfigMap
metadata:
name: translations-74424tmdd8Now, let's teach Bazel to produce the same document.
By convention, we'll add a couple of Bazel targets to the BUILD.bazel file in the same directory. First, we define the kustomization itself, indicating where the kustomization file sits and which other files it depends on. For this, we use the kustomization rule.
load(
"@co_bisontrails_rules_kustomize//kustomize:kustomize.bzl",
"kustomization",
)
kustomization(
name = "base",
srcs = [
"config-bindings",
],
)By default, the kustomization rule assumes the kustomization file is named kustomization.yaml, but you can also point it at other file names, such as the kustomization.yml or kustomization alternatives that the kustomize tool accepts.
This "base" target we've defined doesn't produce any artifacts. It prepares the recipe for building artifacts, in the same way that a kustomization's files are inert input for the kustomize tool. When we'd like to build the kustomization using particular options—as we would by invoking kustomize build—we define another target in a BUILD.bazel file. Here we'll add to the same Bazel package, this time using the kustomized_resources rule.
When we tell Bazel to build this new "simple" target, it will invoke kustomize build and write the output to a file named simple.yaml. That's the default mapping from target name to output file name, but you can change it with the kustomized_resources rule's result attribute. Other Bazel targets can then demand this file as input, forcing Bazel to rebuild the file whenever—and only when—any of the input files change.
In order to use these rules in your Bazel project, you must instruct Bazel to download the source and run the functions that make the rules available. Add the following to your project's MODULE.bazel file.
bazel_dep(name = "rules_kustomize", version = "0.3.4")This declaration registers a particular version of the helm and kustomize tools, respectively. By default, it registers the latest version known to the rules. You can specify a preferred version for each tool by supplying the known version slug (e.g. "v4.5.7") as an argument to the respective module extension's download tag.
bazel_dep(name = "rules_kustomize", version = "0.3.4")
kustomize = use_extension("@rules_kustomize//kustomize:extensions.bzl", "kustomize")
kustomize.download(version = "v5.0.3")
helm = use_extension("@rules_kustomize//kustomize:extensions.bzl", "helm")
helm.download(version = "v3.11.3")If any number of modules wind up specifying different version values for these tags, the latest version—per Semantic Versioning sorting—among the proposed candidate versions wins. If any of the tags also include the tolerate_newer attribute with a value of False, then no competing version newer than that tag's proposed version can win.
With those calls in place, you're now ready to use the rules in your Bazel packages.
At present, these rules can load the following versions of these tools:
This defines a kustomization from a set of source files and other kustomizations, intended for referencing from one or more dependent kustomized_resources targets.
| Name | Type | Default value |
| name | string | mandatory value |
| A unique name for the kustomization. As there is usually only one such target defined per Bazel package (assuming that the target is in the same package as the kustomization file), a simple name like "base" is fitting. | ||
| deps | label_list | [] |
| The set of kustomizations referenced as resources by this kustomization. | ||
| file | label | kustomization.yaml |
| kustomization.yaml, kustomization.yml, or kustomization file for this kustomization. | ||
| requires_exec_functions | bool | False |
Whether this kustomization requires use of exec functions (raw executables) (per the
Even if this kustomization's top-level resources don't require such use but any of its base kustomizations do, this value is effectively |
||
| requires_helm | bool | False |
Whether this kustomization requires use of the Helm chart inflator generator (per the
Even if this kustomization's top-level resources don't require such use but any of its base kustomizations do, this value is effectively |
||
| requires_plugins | bool | False |
Whether this kustomization requires use of kustomize plugins (per the
Even if this kustomization's top-level resources don't require such use but any of its base kustomizations do, this value is effectively |
||
| requires_starlark_functions | bool | False |
Whether this kustomization requires use of Starlark functions (per the Even if this kustomization's top-level resources don't require such use but any of its base kustomizations do, this value is effectively |
||
| srcs | label_list | [] |
| Files referenced as resources for this kustomization. | ||
kustomization(
name = "overlay",
deps = [
# This target "base:base" is another kustomization.
"//apps/base"
],
# We can omit the "file" attribute because our kustomization
# file is named "kustomization.yaml," matching the default.
srcs = [
# This target "charts:charts" is a filegroup.
"//apps/base/charts",
"extras.yaml",
],
requires_helm = True,
)This defines an invocation of the kustomize build command against a kustomization target, creating a resulting set of resources (collectively, a variant).
See the Difficulties section below for considerations both with kustomizations that involve use of the helmCharts Kustomization (or Component) field and when executing Bazel actions on some computers.
| Name | Type | Default value |
| name | string | mandatory value |
| A unique name for the variant. | ||
| env_bindings | string_dict | {} |
Names and values of environment variables to be used by functions (per the These bindings specify a value for each environment variable. To forward an exported environment variable's through instead, use the env_exports attribute. |
||
| env_exports | string_list | [] |
Names of exported environment variables to be used by functions (per the These bindings forward each exported environment variable's value. To specify a value for each environment variable instead, use the env_bindings attribute. |
||
| kustomization | label | mandatory value |
The kustomization to build. This may refer to a target using the kustomization rule or another rule that yields a KustomizationInfo provider. |
||
| load_restrictor | string | RootOnly |
Control whether kustomizations may load files from outsider their root directory (per the
See the Difficulties section for cases where you may need to set this value to |
||
| result | output | <name>.yaml |
The built result, as a YAML stream of KRM resources in separate documents (per the --output
kustomize flag). |
||
kustomized_resources(
name = "production",
env_bindings = {
"CLUSTER_NAME": "prod1234",
"ENVIRONMENT": "production",
},
kustomization = ":overlay",
)KustomizationInfo summarizes a kustomization root, as provided by the kustomization rule.
| Name | Type |
requires_exec_functions |
bool |
Whether this kustomization requires use of exec functions (raw executables) (per the
--enable-exec kustomize flag). |
|
requires_helm |
bool |
Whether this kustomization requires use of the Helm chart inflator generator (per the
--enable-helm kustomize flag). |
|
requires_plugins |
bool |
Whether this kustomization requires use of kustomize plugins (per the
--enable-alpha-plugins kustomize flag). |
|
requires_starlark_functions |
bool |
Whether this kustomization requires use of Starlark functions (per the
--enable-star kustomize flag). |
|
target_file |
string |
| The top-level kustomization file defining this kustomization. | |
transitive_resources |
depset of File |
| The set of files (including other kustomizations) referenced by this kustomization. | |
These rules attempt to make using kustomize with Bazel easier, but there are a few features of the tools that interact poorly, or at least surprisingly, even when they're individually doing their job as intended. We can work around each problem once we know better what to expect.
kustomize prefers to load files only from the kustomization root directory—the one containing the kustomization.yaml file—or any of its subdirectories. The kustomize build subcommand runs with a load restrictor to enforce this restrictive policy. By default, the --load-restrictor flag uses the value LoadRestrictionsRootOnly. With that value in effect, kustomize build will refuse to read any files referenced by a kustomization that lie outside of the kustomization root directory tree, per this FAQ entry.
Bazel can execute the actions for its build and test commands in a restricted environment called a sandbox, using a technique called sandboxing. On some operating systems, Bazel uses symbolic links to make only some files available to programs it runs in its actions. These symbolic links point upward and outward to files that lie outside of the kustomization root in the sandbox. Even though the links are within the kustomization root, their target files are not. kustomize considers this to transgress its LoadRestrictionsRootOnly load restriction and blocks the attempt to load the referenced file.
There are three ways around this problem:
- Relax kustomize's load restrictor by passing
LoadRestrictionsNoneto its--load-restrictorflag, by way of specifying the valueNonefor the kustomized_resources rule's load_restrictor attribute. - Use a Bazel sandboxing implementation that doesn't rely on symbolic links, such as its sandboxfs FUSE file system. With the sandboxfs tool installed, pass the
--experimental_use_sandboxfsflag to bazel build, bazel test, or bazel run.
- Disable Bazel sandboxing entirely by omitting
sandboxedfrom the values supplied via its--spawn_strategyflag. With sandboxing disabled, Bazel will present the input files to kustomize as regular files. So long as those files lie within the kustomization root, theLoadRestrictionsRootOnlyload restrictor will not intervene.
The kustomize tool can expand Helm charts using the Kustomization manifest's helmCharts field. If the Helm chart's source files are not available already locally, kustomize can fetch the chart archive and unpack within the directory specified in the helmGlobals.chartHome field. By default, this chartHome field's value is charts, meaning that kustomize will download and expand chart archives in the charts/<chart name> directory within the kustomization root.
Now, first, let's acknowledge that the kustomize maintainers do not recommend downloading Helm charts automatically, nor even relying on Helm for repeated expansion at all. The capability is there in kustomize today, though, so let's clarify how Bazel might interfere.
What could go wrong? Consider:
If Bazel sandboxing is enabled, you can't have kustomize download and write files within the sandbox directory tree.
Instead, you can set the Kustomization
helmGlobals.chartHomefield to a directory to which Bazel is allowed to write, such as /tmp. Alternately, you can disable sandboxing entirely.If your kustomization directs kustomize to store Helm chart files outside of the kustomization root, or even just refers to such distant files, the default load restrictor will block kustomize from reading them.
You must relax the default load restrictor by specifying the value
Nonefor the kustomized_resources rule's load_restrictor attribute.
Given that your chosen use of Bazel likely implies a preference for hermetic and repeatable builds, it's best to at least acquire and unpack the Helm chart archives beforehand, committing the resulting files for future use. Expanding the Helm chart as manifests outside of kustomize is even better, though it's then harder to include artifacts generated by other Bazel rules. Finding the right balance will take some experimentation.