Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(bake): new --build-arg cli parameter overrides conf.py properties #38

Merged
merged 7 commits into from
Jul 8, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,13 @@ All notable changes to this project will be documented in this file.

## [Unreleased]

### Added

- New `bake` command line argument `--build-arg` to override conf.py build arguments ([#38])

[#38]: https://github.com/stackabletech/image-tools/pull/38


## [0.0.11]

### Added
Expand Down
71 changes: 64 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,26 +15,70 @@ The `bake` command provides the following features:
* Use one or more distributed docker cache servers
* Publish images

## Docker Build Cache

Docker's `buildx` plugin supports different types of build cache back ends. Since Stackable product images are built by distributed GitHub actions, the distributed back ends are relevant.

To use the build cache, you have to configure one or more back ends and enable them by calling `bake` with the `--cache` flag.

To configure one or more cache back ends, add the relevant properties to the `cache` property of the configuration module.

Here an example with the `registry` backend:

```python
cache = [
{
"type": "registry",
"ref_prefix": "build-repo.stackable.tech:8083/sandbox/cache",
"mode": "max",
"compression": "zstd",
"ignore-error": "true",
},
]
```

Here `ref_prefix` is used to build the unique `ref` property for each target.

NOTE: it's your responsibility to ensure that `bake` can read/write to the cache registry by performing a `docker login` before running `bake`.


For more information about the cache back ends, see the [Docker documentation](https://docs.docker.com/build/cache/backends/).

## Usage examples

Run either `bake` or `check-container` with `--help` to get an overview of the accepted flags and their functionality.
Below are some common usage examples:

```shell
# Build images of the hello-world containers
bake -p hello-world -i 0.0.0-dev
bake --product hello-world

# Build only one version [0.37.2] of OPA
bake -p opa=0.37.2 -i 0.0.0-dev
bake --product opa=0.37.2

# Dry run. Do not build anything. Print the the generated Bakefile.
bake --product hello-world --dry

# Build all OPA images and set the organisation to "sandbox"
lfrancke marked this conversation as resolved.
Show resolved Hide resolved
bake --product opa --organization sandbox

# Build all OPA images and set the image version to a release 24.7.0
bake --product opa --image-version 24.7.0

# Enable distributed docker cache (requires credentials to access the cache registry)
bake -p opa --cache
bake --product opa --cache
lfrancke marked this conversation as resolved.
Show resolved Hide resolved

# Build the HBase images but use Java 21 instead of the values in conf.py
# for the java-base and java-devel images.
# It doesn't matter if you use lower or upper case for the build argument names,
# bake will normalize all of them to upper case.
bake --product hbase --build-arg 'java-base=21' --build-arg 'java-devel=21'
lfrancke marked this conversation as resolved.
Show resolved Hide resolved

# Build half of all versions defined for OPA
bake -p opa -i 0.0.0-dev --shard-count 2 --shard-index 0
bake --product opa --shard-count 2 --shard-index 0

# Build the other half of all versions defined for OPA
bake -p opa -i 0.0.0-dev --shard-count 2 --shard-index 1
bake --product opa --shard-count 2 --shard-index 1
```

## Installation
Expand Down Expand Up @@ -63,8 +107,8 @@ let
image-tools = pkgs.callPackage (pkgs.fetchFromGitHub {
owner = "stackabletech";
repo = "image-tools";
rev = "caa4d993bcbb8b884097c89a54ee246f975e2ec6";
hash = "sha256-gjTCroHw4iJhXPW+s3mHBzIH8seIKH1tPb82lUb8+a0="; # comment out to find new hashes when upgrading
rev = "caa4d993bcbb8b884097c89a54ee246f975e2ec6"; # pragma: allowlist secret
hash = "sha256-gjTCroHw4iJhXPW+s3mHBzIH8seIKH1tPb82lUb8+a0="; # pragma: allowlist secret ; comment out to find new hashes when upgrading
} + "/image-tools.nix") {};
in
{
Expand Down Expand Up @@ -97,6 +141,19 @@ pipx install --editable .

With the activated virtual environment, you can now run the tools from the `docker-images` repository and any local changes are immediately in effect.

We also recommend installing the `pre-commit` hooks in the activated virtual environment.

```shell
pip install pre-commit
pre-commit install
```

To run the hooks, stage the changes you want to commit and run:

```shell
pre-commit run
```

## Release a new version

1. Create a release PR where you:
Expand Down
71 changes: 50 additions & 21 deletions src/image_tools/args.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@
import importlib.util
import sys
import os
from types import ModuleType
from typing import List, Tuple

from .version import version

Expand All @@ -20,14 +22,13 @@ def bake_args() -> Namespace:
)
parser.add_argument("-v", "--version", help="Display version", action="store_true")

(
parser.add_argument(
"-c",
"--configuration",
help="Configuration file. Default: './conf.py'.",
default="./conf.py",
),
parser.add_argument(
"-c",
"--configuration",
help="Configuration file. Default: './conf.py'.",
default="./conf.py",
)

parser.add_argument(
"-i",
"--image-version",
Expand Down Expand Up @@ -75,13 +76,19 @@ def bake_args() -> Namespace:
help="Image registry to publish to. Default: docker.stackable.tech.",
default="docker.stackable.tech",
)
(
parser.add_argument(
"--export-tags-file",
help="Write target image tags to a text file. Useful for signing or other follow-up CI steps.",
),
parser.add_argument(
"--export-tags-file",
help="Write target image tags to a text file. Useful for signing or other follow-up CI steps.",
)

parser.add_argument("--cache", help="Enable distributed build cache", action="store_true")

parser.add_argument(
"--build-arg",
help="Override build arguments. Expecting an KEY=VALUE format. The key is case insensitive.",
nargs="*",
type=check_build_arg,
)
(parser.add_argument("--cache", help="Enable distributed build cache", action="store_true"),)

result = parser.parse_args()

Expand Down Expand Up @@ -133,6 +140,13 @@ def check_image_version_format(image_version) -> str:
raise ValueError(f"Invalid image version: {image_version}")


def check_build_arg(build_args: str) -> Tuple[str, str]:
kv = build_args.split("=")
if len(kv) != 2:
raise ValueError
return kv[0], kv[1]


def preflight_args() -> Namespace:
parser = ArgumentParser(
description="Run OpenShift certification checks and submit results to RedHat Partner Connect portal"
Expand Down Expand Up @@ -180,13 +194,11 @@ def preflight_args() -> Namespace:
help="Name of the preflight program. Default: preflight",
default="preflight",
)
(
parser.add_argument(
"-c",
"--configuration",
help="Configuration file.",
default="./conf.py",
),
parser.add_argument(
"-c",
"--configuration",
help="Configuration file.",
default="./conf.py",
)

result = parser.parse_args()
Expand All @@ -206,7 +218,11 @@ def check_architecture_input(architecture: str) -> str:
return architecture


def load_configuration(conf_file_name: str):
def load_configuration(conf_file_name: str, user_build_args: List[Tuple[str, str]] = []) -> ModuleType:
"""Load the configuration module conf.py and potentially override build arguments
with values provided by the user with the --build-arg flag.
The build arguments are key, value pairs from the "conf.products.<product name>.versions.<version>" dictionary.
"""
module_name = "conf"
sys.path.append(str(os.getcwd()))
spec = importlib.util.spec_from_file_location(module_name, conf_file_name)
Expand All @@ -215,5 +231,18 @@ def load_configuration(conf_file_name: str):
sys.modules[module_name] = module
if spec.loader:
spec.loader.exec_module(module)
override_build_args(module, user_build_args)
return module
raise ImportError(name=module_name, path=conf_file_name)


def override_build_args(conf: ModuleType, user_build_args: List[Tuple[str, str]] = []) -> None:
if not user_build_args:
return
# convert user_build_args to a dictionary for easier lookup
user_build_args_dict = {kv[0]: kv[1] for kv in user_build_args}
for product in conf.products:
for conf_build_args in product["versions"]:
for arg in conf_build_args:
if arg in user_build_args_dict:
conf_build_args[arg] = user_build_args_dict[arg]
6 changes: 3 additions & 3 deletions src/image_tools/bake.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
from .version import version


def build_image_args(version: Dict[str, str], release_version: str):
def build_image_args(conf_build_args: Dict[str, str], release_version: str):
"""
Returns a list of --build-arg command line arguments that are used by the
docker build command.
Expand All @@ -33,7 +33,7 @@ def build_image_args(version: Dict[str, str], release_version: str):
"""
result = {}

for k, v in version.items():
for k, v in conf_build_args.items():
result[k.upper()] = v
result["RELEASE"] = release_version

Expand Down Expand Up @@ -213,7 +213,7 @@ def main() -> int:
print(version())
return 0

conf = load_configuration(args.configuration)
conf = load_configuration(args.configuration, args.build_arg)

bakefile = generate_bakefile(args, conf)

Expand Down