mzbuild is an build and orchestration system for Docker containers.
As a user or developer, you'll interact with mzbuild through two commands:
-
mzcomposeis a layer on top of Docker Compose that permits orchestrating and interacting with services via Python scripts.It automatically downloads cached images from Docker Hub if available, or otherwise builds them locally if you've made changes to the inputs of the image. This approach keeps things snappy when running a test on the latest tip of main, while ensuring that you don't need to modify the composition as you make changes to the source code.
-
mzimageis a lower-level command that allows inspection and finer-grained control over the build process for the images in the repository.
From the root of the repository, invoke the commands as bin/mzcompose and
bin/mzimage, respectively. Any directory with an mzcompose.py file will
also have a convenience script alongside it, which you can invoke as
./mzcompose from within that directory.
Table of contents:
Warning: mzbuild does not yet return particularly friendly error messages. This is under active improvement.
The core object of mzbuild is the Docker image. To ask mzbuild to manage a
Docker image for you, simply create a new directory in the repository with
a Dockerfile and a configuration file named mzbuild.yml.
Here's a simple example for a fictional Python load generator called
fancy-load:
# test/fancy/loadgen/Dockerfile
MZFROM ubuntu-base
RUN apt-get update && apt-get install -qy python3
COPY fancy-loadgen.py .
ENTRYPOINT ["python3", "fancy-loadgen.py"]# test/fancy/loadgen/mzbuild.yml
name: fancy-loadgen# test/fancy/loadgen/fancy-loadgen.py
print("fancy load")That's it! mzbuild will now automatically detect this image. You can see for
yourself with the mzimage list command:
$ bin/mzimage list
...
fancy-loadgen
...You can then also ask mzimage to run the image:
$ bin/mzimage run fancy-loadgen
==> Acquiring materialize/fancy-loadgen:7J7C34QTCEKQTI5G3NDIH7FFMMWZ3IWB
$ docker pull materialize/fancy-loadgen:7J7C34QTCEKQTI5G3NDIH7FFMMWZ3IWB
...
Successfully tagged materialize/fancy-loadgen:7J7C34QTCEKQTI5G3NDIH7FFMMWZ3IWB
$ docker run -it --rm --init materialize/fancy-loadgen:7J7C34QTCEKQTI5G3NDIH7FFMMWZ3IWB
fancy loadNotice that random string of characters, 7J7C34QTCEKQTI5G3NDIH7FFMMWZ3IWB, in
the output? That's the image's fingerprint, which is a Base32-encoded SHA-1 hash
of the inputs to your image. By default, mzbuild assumes all files in the
image's directory are inputs, which is usually a sane default.
Don't worry if your fingerprint isn't identical. A single stray newline will result in a completely different fingerprint.
If you run the image again, since we've already got an image for that
fingerprint available, mzimage won't need to rebuild the image, and the
command should be much speedier:
$ bin/mzimage run fancy-loadgen
$ docker run -it --rm --init materialize/fancy-loadgen:7J7C34QTCEKQTI5G3NDIH7FFMMWZ3IWB
fancy loadNow try changing the Python script to print some fancier load:
import sys
sys.stdout.buffer.write("🎩 load\n".encode("utf-8"))If we ask mzimage about the fingerprint, we'll see it's changed to reflect
the new Python code:
$ bin/mzimage fingerprint fancy-loadgen
TPOXKNHJOZBEYRN635UXLDHML6INVVMVWe can also add dependencies on other images to fancy-loadgen. Let's say we also want to include the billing-demo image. Edit the Dockerfile to look like this:
# test/fancy/loadgen/Dockerfile
MZFROM billing-demo AS billing-demo
MZFROM ubuntu-base
RUN apt-get update && apt-get install -qy python3
COPY fancy-loadgen.py .
COPY --from=billing-demo /usr/local/bin/billing-demo /usr/local/bin/billing-demo
CMD ["python3", "fancy-loadgen.py"]And let's verify that the new image contains billing-demo where we expect it:
$ bin/mzimage run fancy-loadgen ls -lh /usr/local/bin/billing-demo
==> Acquiring materialize/billing-demo:FM4STU42G7W44OLAPKZNEZWGEPTMIVE6
$ docker pull materialize/billing-demo:FM4STU42G7W44OLAPKZNEZWGEPTMIVE6
FM4STU42G7W44OLAPKZNEZWGEPTMIVE6: Pulling from materialize/billing-demo
...
docker.io/materialize/billing-demo:FM4STU42G7W44OLAPKZNEZWGEPTMIVE6
==> Acquiring materialize/fancy-loadgen:LLJS6MMDVOHZJQFBP42DFBW6Z5N3CW4F
$ docker pull materialize/fancy-loadgen:LLJS6MMDVOHZJQFBP42DFBW6Z5N3CW4F
Error response from daemon: manifest for materialize/fancy-loadgen:LLJS6MMDVOHZJQFBP42DFBW6Z5N3CW4F not found: manifest unknown: manifest unknown
$ docker build --pull -f - -t materialize/fancy-loadgen:LLJS6MMDVOHZJQFBP42DFBW6Z5N3CW4F test/fancy/loadgen
...
Successfully tagged materialize/fancy-loadgen:LLJS6MMDVOHZJQFBP42DFBW6Z5N3CW4F
-rwxr-xr-x 1 root root 8.3M Apr 15 01:34 /usr/local/bin/billing-demoNotice how mzbuild automatically downloaded a copy of the billing-demo from Docker Hub! Seamless.
To create an mzcompose configuration that uses the fancy-loadgen image we
built in the previous tutorial, just drop an mzcompose.py file into a
directory:
from materialize.mzcompose import Service
SERVICES = [
Service(
"fancy-loadgen",
{
"mzbuild": "fancy-loadgen",
},
),
]Now bring the configuration up with bin/mzcompose:
$ bin/mzcompose --find fancy up
==> Collecting mzbuild dependencies
materialize/billing-demo:FM4STU42G7W44OLAPKZNEZWGEPTMIVE6
materialize/fancy-loadgen:Z2GPU4TQMCV2PGFTNUPYLQO2PQAYD6OY
==> Delegating to Docker Compose
Starting fancy_fancy_1 ... done
Attaching to fancy_fancy_1
fancy_1 | 🎩 load
fancy_fancy_1 exited with code 0The argument you pass to --find is the name of the directory containing the
mzcompose.py. Don't worry: if this directory name is not unique across the
entire repository, mzcompose will complain.
Notice how mzcompose automatically acquired images for not just
fancy-loadgen but all of its dependencies before delegating to Docker Compose
to actually start the containers.
Typing that entire command out is painful, though. Let's ask mzcompose to generate a convenience script for us:
$ bin/mzcompose gen-shortcutsNow we can run ./mzcompose from within the test/fancy directory:
cd test/fancy
$ ./mzcompose ps
==> Collecting mzbuild dependencies
materialize/billing-demo:FM4STU42G7W44OLAPKZNEZWGEPTMIVE6
materialize/fancy-loadgen:Z2GPU4TQMCV2PGFTNUPYLQO2PQAYD6OY
==> Delegating to Docker Compose
Name Command State Ports
---------------------------------------------------------
fancy_fancy_1 python3 fancy-loadgen.py Exit 0Let's add another mzbuild dependency, this time on materialized:
from materialize.mzcompose import Service
from materialize.mzcompose.services.materialized import Materialized
SERVICES = [
Materialized(),
Service(
"fancy-loadgen",
{
"mzbuild": "fancy-loadgen",
},
),
]Notice how we reused the service definition in the services module. If
fancy-loadgen were used in multiple compositions, we'd similarly want to
create a reusable service definition called FancyLoadgen.
mzcompose will automatically acquire the new dependency on the next
invocation. Note that if you have local changes to any Rust code, you'll likely
want to stash them away now, or mzcompose will be spending a lot of time
recompiling a fresh version of the materialized image.
$ ./mzcompose up
==> Collecting mzbuild dependencies
materialize/billing-demo:FM4STU42G7W44OLAPKZNEZWGEPTMIVE6
materialize/fancy-loadgen:Z2GPU4TQMCV2PGFTNUPYLQO2PQAYD6OY
materialize/environmentd:EYBAS3HTGQS2SAVO3RBR5JS6AVGVRPJM
==> Acquiring materialize/environmentd:EYBAS3HTGQS2SAVO3RBR5JS6AVGVRPJM
$ docker pull materialize/environmentd:EYBAS3HTGQS2SAVO3RBR5JS6AVGVRPJM
EYBAS3HTGQS2SAVO3RBR5JS6AVGVRPJM: Pulling from materialize/environmentd
...
Status: Downloaded newer image for materialize/environmentd:EYBAS3HTGQS2SAVO3RBR5JS6AVGVRPJM
docker.io/materialize/environmentd:EYBAS3HTGQS2SAVO3RBR5JS6AVGVRPJM
==> Delegating to Docker Compose
Starting fancy_fancy_1 ... done
Creating fancy_materialized_1 ... done
Attaching to fancy_fancy_1, fancy_materialized_1
fancy_1 | 🎩 load
materialized_1 | booting...
fancy_fancy_1 exited with code 0
fancy_materialized_1 exited with code 1And that's it. Pretty simple. Note that you can add normal image services to
your mzcompose.py, too. That works just as it would in vanilla Docker
Compose.
from materialize.mzcompose import Service
from materialize.mzcompose.services.materialized import Materialized
SERVICES = [
Materialized(),
Service(
"fancy-loadgen",
{
"mzbuild": "fancy-loadgen",
},
),
Service(
"zookeeper",
{
"image": "zookeeper:3.4.13",
},
),
](Although, in a real composition, you'd use the built in Zookeeper service
rather than inlining the service definition.)
Via mzbuild, mzcompose supports building binaries in either release or
development mode. By default, binaries are built using release mode. You can
choose dev mode instead by passing the --dev flag to mzcompose:
$ bin/mzcompose --dev --find fancy upmzbuild is an input-addressable build system.
The key insight is that most Docker images are designed to be a pure function from a set of input files to a packaged image. Images generally only need to be built once for a given set of inputs.
In practice, Docker build processes tend to depend on non-reproducible state, like APT repositories or the current time, but these typically don't have a meaningful impact on the build, and we are happy to ignore these annoyances for now.
mzbuild uses SHA-1 hashes for its fingerprints, like Git. To prevent confusing a fingerprint for a Git commit SHA, mzbuild fingerprints are encoded in uppercase Base32. (Base32 is a bit easier to handle than Base64, as it doesn't include any non-alphanumeric characters.)
mzbuild and associated tools are written in Python 3 and live in misc/python/materialize.
Their only dependency is Python 3.5+, which is easy to find or pre-installed on most Linux distributions, and pre-installed on recent versions of macOS, too. Python dependencies are automatically installed into a virtualenv by the pyactivate wrapper script.
Using Python 3.6 would be a good bit more convenient, but our CI image runs on Ubuntu 16.04, which is still shipping Python 3.5. Supporting the oldest Ubuntu LTS release seems like a decent baseline, anyway.
Integration tests for mzcompose are in test/mzcompose.
An mzbuild.yml file describes how to build a Docker image from a Dockerfile
and a pre-docker build plugin.
The directory containing a mzbuild.yml file is called the "mzbuild context."
name: environmentd
description: "A Docker image containing just environmentd."
pre-image:
- type: cargo-build
bin: environmentd
strip: false
publish: true-
name(string, required) is an identifier for the image. It must be unique within the repository. If the image is publishable, it will be published to Docker Hub asmaterialize/<name>. -
pre-image(list of maps) specifies plugins to run before invokingdocker build. The plugins are run in order. This is where the magic happens for Rust code.At the moment
pre-imageonly supports three plugins:-
type: copyrecursively copies the contents of a directory into the mzbuild context.The
sourcefield specifies the directory from which files should be copied. It is relative to the root of the repository. Thedestinationfield specifies the directory into which files should be copied. It is relative to the mzbuild context. Both fields are required.The name of the file in the
destinationdirectory will be the name of the file in thesourcedirectory with thesourceprefix removed. So a file named/path/to/source/a/b/c.extwill be copied into/path/to/destination/a/b/c.ext.The optional
matchingfield specifies a glob that determines which files in thesourcedirectory to copy. -
type: cargo-buildbuilds a Rust binary with Cargo. Thebinfield is a string or a list of strings that indicates the name of one or more binary targets in the Cargo workspace to build. The resulting artifact will be placed into the mzbuild context. Theexamplefield works identically but names an example to build rather than a binary.All files within the crate directory, and all files within the directories of any transitive path dependencies of the crate (i.e., dependencies in this workspace, rather than on crates.io), will be considered as additional inputs to the build, plus the top-level
Cargo.toml,Cargo.lock, and.cargo/configfiles.Cargo is invoked with the
--releaseflag unless the--devflag is specified.In rare cases, it may be necessary to extract files from the build directory of a dependency. The
extractkey specifies a mapping from a dependent package to a mapping from source files and directories to destination directories. Source paths are interpreted relative to that crate's build directory while destination paths are interpreted relative to the build context. Note thatextractis only relevant if the dependency has a custom Cargo build script, as Rust crates without a build script do not have a build directory.
-
-
publish(bool) specifies whether the image should be automatically published to Docker Hub by CI. Non-publishable images can still be used by users and CI, but they must always be built from source. Use sparingly. The default istrue. -
description(str) is a short description for the image. Ifpublishis true, CI will automatically sync the description to Docker Hub. -
mainline(bool) indicates whether the image participates in the main Materialize versioning scheme. CI will automatically push version tags for mainline images for unprefixed tag builds (e.g.,v0.27.0). Non-mainline images must handle their own release scheme. The default istrue. -
build-args(map[str, str]) a list of parameters to pass as--build-argto Docker. For example:name: example build-args: VERSION: '1.0'
When using a pre-image plugin, arbitrary build artifacts will be copied into
the mzbuild context. Be sure to add a .gitignore to the mzbuild context and
ignore these files! Ignored files will be excluded from the mzbuild fingerprint,
and will be automatically deleted at the beginning of the pre-image phase to
ensure idempotent builds.
See mzcompose.md for details.
An mzbuild Dockerfile is like a normal Dockerfile, but it can depend on other mzbuild images.
MZFROM environmentd
MZFROM ubuntu-base
COPY --from=0 ...MZFROM <string> [AS <name>]sets the base image for the build stage to the specified mzbuild image. It is like the vanilla Dockerfile FROM command, except that the image named must be a valid mzbuild image in the repository, not a vanilla Docker image.
If an mzbuild.yml file contains a file named README.md and publish is true, CI
will automatically sync that file to Docker Hub as the repository's "full
description."
End-to-end tests of Materialize can involve orchestrating a dozen different services: ZooKeeper, Kafka, PostgreSQL, Prometheus, Grafana, load simulators, and so on. So far, the most workable solution for managing such a dizzying array of services has involved Docker and Docker Compose.
Docker has its shortcomings, but its popularity means it is more widely used and understood than any other tool we're aware of. Many developers have a passing familiarity with Docker. Empirically, many users of Materialize are willing to download Docker—or already have it installed—in order to take Materialize for a spin, and some prospective customers have explicitly requested the Docker distribution channel. The situation is similar with Docker Compose: it's not perfect, but it gives us a lot of power for free.
If you are not sold on Docker, see the Why docker? section below.
Our single biggest pain point with Compose has been the inability to seamlessly share Compose configurations between our developers, CI, and downstream users running demos. Developers want the Compose files to build all images from source, so that changes made locally are reflected in the containers. Users want Compose files to download images from Docker Hub, since building the images from source can take the better part of an hour, if they even have a build toolchain available. And CI wants both, as it needs to build from source on one machine, then distribute those pre-built images to downstream workers.
Compose, however, requires you to commit to an option for every service. Either a) a service is built from source or, b) it is downloaded from Docker Hub, and there is no in between.
To make matters worse, even if Compose had the desired behavior, building Rust inside of a Dockerfile-managed build process would take upwards of 10m. Since Docker builds start from a clean sandbox on every run, the Cargo cache is freezing cold, and all dependencies must be compiled from scratch. Instead, we need to build the Rust code outside of the Docker build process on a host machine with a warm Cargo cache, and then copy the built binary into the Docker build context.
To work around these limitations, our codebase has grown an increasingly complex collection of shell scripts and Buildkite configurations. Aside from the maintainability concerns, there is also a serious usability concern: every demo has a slightly different interface to run it, configure it, test changes to it, and deploy it.
So since we don't want to give up on Docker and Compose, the best option seemed to be a thin wrapper script on top of Compose that acquired all the necessary Docker images, using whatever means necessary, and then delegated to Compose once all the images were in place. That wrapper script is mzbuild.
Our requirements for a demo and testing tool are as follows:
- Support for macOS and Linux.
- Ideally, the platform differences would be entirely handled by the tool, and invisible to those using it.
- Automatic downloading and installation of myriad services.
- Version pinning, so that everyone runs the same version of the services.
- Automatic configuration of those services to talk to one another.
- High probability that if a demo or test works on one machine, it will work on another.
- Easy cleanup, so that data generated by the demo or test does not persist permanently on the user's machine.
Docker and Docker Compose satisfy all of the above criteria. The (fair) criticisms are that the tools are complicated, slow, and hard to understand and debug, but so far it seems the tradeoff seems worth it.
The closest competitors to Docker and Compose are Kubernetes and Helm, but these are more complicated than Docker and Compose, as they're more focused on the enterprise-grade production deployment scenario.