Add pubsys: make Bottlerocket repos using cargo make repo
#964
Conversation
|
Rebased on |
|
@zmrow and I addressed Ben's comments locally, but while testing them we realized something about the current structure that gave us pause - pubsys can't use one of its output repos as an input for a following run, because of the directory structure. We need to either decide that it's expected and move on as-is, or come up with a different directory structure that allows for it. As to why it happens - pubsys writes out to a flat directory structure of One option for a change would be to use the nested structure for pubsys output, too. The main difference from the current revision is that different pubsys runs would copy/link targets into the same directory, I think the tool can be a little opinionated about structure, rather than needing to have more required variables or a super flexible path system at the moment, so that seems OK to me... |
|
I think I'd rather not have a unified I'm wondering how often a user would want to use an outdir as the pubsys indir rather than just creating a new repository. Until we have a good answer to that question or an example of a solid use case, my vote is to leave pubsys as we currently have it. I'd rather get some more experience using it and an additional set of use cases before we make changes. |
Having the property where input, output, and remote structures match seems better to me. I'd like to be able to run We can split it out into two syncs for production use cases where it's important that targets go up before metadata, but otherwise it just seems more convenient. We could also make it more convenient to use the repositories by modeling the shared prefix ( |
|
We discussed the output repo structure offline, and decided for now to go with the plan to make the output structure match the input structure. It's clearer to have a single structure, and it's what Bottlerocket is using in production and what we'd recommend to others as well. |
Note: before merging, this requires a new |
|
Rebased and fixed trivial conflicts |
|
^ This force-push was just a rebase with no conflicts; fixes for comments are coming soon. |
|
This push addresses most of @bcressey's comments:
|
|
^ This push updates variable names in |
Simplify repo creation and tie it into the existing `cargo make` build system using a new `repo` target. This is intended to replace the common pattern of calls to tuftool and updata - either creating a repo or extending an existing one, adding the latest built artifacts, then updating the manifest to match. **Usage:** `cargo make repo` depends on the `build` target, so if your goal is building an update for a repo you don't have to separately build it first. It uses the same `BUILDSYS_VARIANT` and `BUILDSYS_ARCH` variables to determine the metadata for the update added to the repo. **Requirements:** * An Infra.toml file, based on Infra.toml.example, listing paths to keys, existing repos, etc. * Release.toml updated with a new version **Optional further configuration:** * Repo expiration policy - by default, uses a policy file with 2 week target and snapshot expiration and 1 week timestamp expiration. * Wave policy - same policy files you give to updata today; defaults to "default" wave policy. * Release start time - when waves start and when expiration starts counting down; defaults to now. * Can select which named repo and signing key to use from Infra.toml. **Design decisions:** * Built repo metadata is written to a directory like /build/repos/bottlerocket-0.4.1-5880e5d/aws-k8s-1.15/x86_64 so that you can prepare repos for multiple releases in parallel. Targets are written to a shared directory like /build/repos/bottlerocket-0.4.1-5880e5d/targets - they're unique across variants and arches so there's no conflict. The directory structure as a whole can be synced to your final repo location; it's the structure expected by Bottlerocket and updog. * buildsys uses environment variables set by cargo-make; we opted instead for more standard arg parsing. It seems more likely that someone would use pubsys separately from cargo-make, and pubsys has more input information, so arg parsing was clearer. * cargo-make environment variable expansion is done in phases, and you can't refer to a variable defined in the same section if you intend to let the user override the earlier variable on the command line. If you do, the variable won't expand, as seen in bottlerocket-os#963. Because of this, until we figure out a better strategy, a couple of variables can't be overridden - the path to Release.toml (which we made a variable in this change) and the repo output directory. Co-authored-by: Zac Mrowicki <[email protected]> Co-authored-by: Tom Kirchner <[email protected]>
|
This force-push updates the Cargo.lock to reflect the new |
Simplify repo creation and tie it into the existing
cargo makebuild systemusing a new
repotarget. This is intended to replace the common pattern ofcalls to tuftool and updata - either creating a repo or extending an existing
one, adding the latest built artifacts, then updating the manifest to match.
Usage:
cargo make repodepends on thebuildtarget, so if your goal is building anupdate for a repo you don't have to separately build it first. It uses the
same
BUILDSYS_VARIANTandBUILDSYS_ARCHvariables to determine the metadatafor the update added to the repo.
Requirements:
existing repos, etc.
Optional further configuration:
and snapshot expiration and 1 week timestamp expiration.
"default" wave policy.
down; defaults to now.
Design decisions:
/repo/bottlerocket-aws-k8s-1.15-x86_64-0.4.1-5880e5d that includes variant,
arch, and full version in the name, so that you can prepare repos for
multiple releases in parallel if desired. Inside are target/ and metadata/
directories that can be synced to your final repo location.
more standard arg parsing. It seems more likely that someone would use
pubsys separately from cargo-make, and pubsys has more input information, so
arg parsing was clearer.
refer to a variable defined in the same section if you intend to let the user
override the earlier variable on the command line. If you do, the variable
won't expand, as seen in
make BUILDSYS_OUTPUT_DIR work with overrides #963. Because of this,
until we figure out a better strategy, a couple of variables can't be
overridden - the path to Release.toml (which we made a variable in this
change) and the repo output directory.
Testing done:
the --link-target argument not used by Makefile
live Bottlerocket repo
v0.4.1 update, and using it with an instance to successfully update
Terms of contribution:
By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.