--- nav_order: 4 --- # Working with CoreOS Assembler {: .no_toc } 1. TOC {:toc} ## Understanding "config git" Conceptually, coreos-assembler ties together generating OSTree commits with disk images into a single "build schema". The build target is defined by the "config git" which should be in `src/config` (relative to the build directory). We can hack on some local input configs by exporting them in the `COREOS_ASSEMBLER_CONFIG_GIT` env variable. For example: ``` $ export COREOS_ASSEMBLER_CONFIG_GIT=/path/to/github.com/coreos/fedora-coreos-config/ $ cosa init --force /dev/null $ cosa fetch && cosa build ``` ## Components of "config git" ### `manifest.yaml` For generating OSTree commits, cosa uses `manifest.yaml`: An rpm-ostree "manifest" or "treefile", which mostly boils down to a list of RPMs and a set of rpm-md repositories they come from. It also supports `postprocess` to make arbitrary changes. See the rpm-ostree documentation for the [Treefile format reference](https://coreos.github.io/rpm-ostree/treefile/). ### `overlay.d/` coreos-assembler also supports a generic way to embed architecture-independent configuration and scripts by creating subdirectories in `overlay.d/`. Each subdirectory of the `overlay.d.` directory is added to the OSTree commit, in lexicographic order. It's recommended to name directories with a numeric prefix - e.g. `05core`, `10extras`. Non-directories are ignored. For example, a good practice is to add a `README.md` file into your overlay directories describing their structure. ### `image.yaml` This YAML file configures the output disk images. Supported keys are: - `size`: Size in GB for cloud images (OpenStack, AWS, etc.) Required. - `extra-kargs`: List of kernel arguments. - `include`: path to another YAML file to include. List values are appended. For other type, the values in the sourcing config win. It's likely in the future we will extend this to support e.g. a separate `/var` partition or configuring the filesystem types. If you want to do anything like that today it requires forking the assembler and rebuilding it. See the fedora-coreos-config for an example. ## Hacking on "config git" First you can expand the size of the image; edit `src/config/image.yaml` and e.g. change `8` to `9`. Rerun `cosa build`, and notice that the OSTree commit didn't change, but a new image is generated in `builds`. When you `cosa run`, you'll get it. Another thing to try is editing `src/config/manifest.yaml` - add or remove entries from `packages`. You can also add local rpm-md `file:///` repositories. ## Using overrides Development speed is closely tied to the "edit-compile-debug" cycle. coreos-assembler supports an `overrides/` sub-directory of the coreos-assembler working directory, which allows easily overlaying locally-generated content on top of the base OS content. There are two subdirectories of `overrides/`: - `overrides/rootfs` - `overrides/rpm` Let's say you want to hack on both ostree and ignition-dracut. See for example [this PR](https://github.com/coreos/ignition-dracut/pull/106) which added support for `make install DESTDIR=` to the latter. In general most upstream build systems support something like this; if they don't it's a good idea to add. Concretely, if `/path/to/cosa-workdir` is where you ran `cosa init`, then after doing edits in a project, run a command like this from the source repository for the component: ``` $ make install DESTDIR=/path/to/cosa-workdir/overrides/rootfs ``` This would then install files like `/path/to/cosa-buildroot/overrides/rootfs/usr/bin/ostree` etc. If you then run `cosa build` from the cosa workdir, those overrides will be automatically incorporated. You can also choose to use `overrides/rpm` - this accepts pre-built binary RPMs. This can be convenient when you want to quickly test a binary RPM built elsewhere, or if you want to go through a more "official" build process. If any RPMs are present here, then coreos-assembler will automatically run `createrepo_c` and ensure that they are used in the build. In the future, it's likely coreos-assembler will also support something like `overrides/src` which could be a directory of symlinks to local git repositories. ## Using cosa build-fast If you're working on e.g. the kernel or Ignition (things that go into the initramfs), then you probably need a `cosa build` workflow (or `cosa buildinitramfs-fast`, see below). However, let's say you want to test a change to something that runs purely in the real root, such as e.g. `rpm-ostree`, `podman`, or `console-login-helper-messages`. Rather than doing a full image build each time, a fast way to test out changes is to use `cosa build-fast`. This command assumes you have a previous local coreos-assembler build. From the git checkout of the project you want to add: ``` $ export COSA_DIR=/srv/builds/fcos $ cosa build-fast $ cosa run ``` The `cosa build-fast` command will run `make` and inject the resulting binaries on a qcow2 overlay file, which will appear in your project working directory. The `cosa run` command similarly knows to look for these `qcow2` files. This will not affect the "real" cosa build in `/srv/builds/fcos`, but will use it as a data source. ## Using cosa run --bind-ro for even faster iteration This workflow can be used alongside `cosa build-fast`, or separate from it. If you invoke e.g. ``` $ cosa run --bind-ro ~/src/github/containers/podman,/run/workdir ``` The target VM will have the source directory bound in `/run/workdir`; then you can directly copy binaries from your development environment into the VM. If you are running cosa in a container, you will have to change your current working directory to a parent directory common to both project directories and use relative paths: ``` $ cd ~ $ cosa run \ --qemu-image src/fcos/build/latest/x86_64/fedora-coreos-*.x86_64.qcow2 \ --bind-ro src/github/containers/podman,/run/workdir ``` Then in the booted VM, `/run/workdir` will point to the `libpod` directory on your host, allowing you to directly execute binaries from there. You can also use e.g. `rpm-ostree usroverlay` and then copy binaries from your host `/run/workdir` into the VM's rootfs. (This currently only works on Fedora CoreOS which ships `9p`, not RHCOS. A future version will use https://virtio-fs.gitlab.io/ ) ## Using host binaries Another related trick is: ``` $ cosa run --bind-ro /usr/bin,/run/hostbin ``` Then in the VM you have e.g. `/run/hostbin/strace`. (This may fail in some scenarios where your dev container is different than the target). If you are running cosa in a container, you will only have access to the binary installed in this container. You can install binaries before launching the VM with: ``` $ cosa shell $ sudo dnf install ... $ cosa run --bind-ro /usr/bin,/run/hostbin ``` ## Using cosa buildinitramfs-fast If you're iterating on changes *just* to the initramfs, you can also use `cosa buildinitramfs-fast`. For example, suppose you are working on `ignition`. Follow these steps: ``` $ make $ install -D -m 0755 bin/amd64/ignition /path/to/cosadir/overrides/initramfs/usr/bin/ignition $ cd /path/to/cosadir $ cosa buildinitramfs-fast $ cosa run --qemu-image tmp/fastbuild/fastbuildinitrd-fedora-coreos-qemu.qcow2 -i config.ign ``` (Or instead of `cosa run` use e.g. `cosa kola` to run tests, etc.) If you're adding new fields to the Ignition experimental spec, `-i` will silently remove those fields, since the copy of Ignition that's vendored into mantle won't know about them yet. Instead, you can specify that kola should pass the config to the machine unmodified: ``` $ kola qemuexec --qemu-image tmp/fastbuild/fastbuildinitrd-fedora-coreos-qemu.qcow2 -i config.ign --ignition-direct ``` You'll need to [manually configure autologin](https://docs.fedoraproject.org/en-US/fedora-coreos/tutorial-autologin/) in the Ignition config, since kola won't be able to do it for you. ## Performing an in-place OS update manually The output of coreos-assembler is conceptually two things: - an ostree container image - disk images (ISO, AWS AMI, qemu .qcow2, etc) In many cases, rather than booting from a new disk image with the new OS, you will want to explicitly test in-place upgrades. This uses an [ostree native container](https://fedoraproject.org/wiki/Changes/OstreeNativeContainer), which is in the form of an `.ociarchive` file generated by `cosa build ostree` (as well as the default `cosa build`, which *also* generates a `qemu` disk image). You will need to make the container image available to your targeted system (VM or physical). One way to do this is to push the container to a public registry such as quay.io: `cosa push-container quay.io/exampleuser/fcos` Performing an in-place update from the `cosa build` output boils down to invoking a command of the form: ``` $ rpm-ostree rebase --experimental ostree-unverified-registry:quay.io/exampleuser/fcos $ systemctl reboot ``` ## Using different CA certificates If you need access to CA certificates on your host (for example, when you need to access a git repo that is not on the public Internet), you can mount in the host certificates using the `COREOS_ASSEMBLER_CONTAINER_RUNTIME_ARGS` variable. **NOTE** Sharing the `/etc/pki/ca-trust` directory may be blocked by SELinux so you may have to use a directory with the `system_u:object_r:container_file_t:s0` file context. ``` $ export COREOS_ASSEMBLER_CONTAINER_RUNTIME_ARGS='-v /etc/pki/ca-trust:/etc/pki/ca-trust:ro' $ cosa init https://github.com/coreos/fedora-coreos-config.git $ cosa fetch && cosa build ``` See this [Stack Overflow question](https://stackoverflow.com/questions/26028971/docker-container-ssl-certificates) for additional discussion. ## Running CoreOS Assembler in OpenShift on Google Compute Platform This is a guide to run a COSA pod in an OpenShift 4.6+ cluster in Google Compute Platform (GCP). [Install the KVM device plugin](https://github.com/cgwalters/kvm-device-plugin). Up to this point, you needed to be `kubeadmin`. From this point on though, best practice is to switch to an "unprivileged" user. (In fact the steps until this point could be run by a separate team that manages the cluster; other developers could just use it as unprivileged users) Personally, I added a [httpasswd identity provider](https://docs.openshift.com/container-platform/4.1/authentication/identity_providers/configuring-htpasswd-identity-provider.html) and logged in with a password. I also did `oc new-project coreos-virt` etc. Schedule a cosa pod: ``` apiVersion: v1 kind: Pod metadata: labels: run: cosa name: cosa spec: containers: - args: - shell - sleep - infinity image: quay.io/coreos-assembler/coreos-assembler:latest name: cosa resources: requests: # Today COSA hardcodes 2048 for launching VMs. We could # probably shrink that in the future. memory: "3Gi" devices.kubevirt.io/kvm: "1" limits: memory: "3Gi" devices.kubevirt.io/kvm: "1" volumeMounts: - mountPath: /srv name: workdir volumes: - name: workdir emptyDir: {} restartPolicy: Never ``` Then `oc rsh pods/cosa` and you should be able to `ls -al /dev/kvm` - and `cosa build` etc!