Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Split-off dependencies.md from dev.md #290

Merged
merged 14 commits into from
Jul 1, 2021
59 changes: 59 additions & 0 deletions docs/dependencies.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
### Installing required dependencies for local build

To get requirements installed locally, run:

```bash
$> make deps
```

or you need:

- [`luet`](https://github.com/mudler/luet)
- [`luet-makeiso`](https://github.com/mudler/luet-makeiso)
- [`squashfs-tools`](https://github.com/plougher/squashfs-tools)
- `zypper in squashfs` on SLES or openSUSE
- [`xorriso`](https://dev.lovelyhq.com/libburnia/web/wiki/Xorriso)
- `zypper in xorriso` on SLES or openSUSE
- `yq` ([version `3.x`](https://github.com/mikefarah/yq/releases/tag/3.4.1)), installed via [packages/toolchain/yq](https://github.com/rancher-sandbox/cOS-toolkit/tree/master/packages/toolchain/yq) (optional)
- [`jq`](https://stedolan.github.io/jq), installed via [packages/utils/jq](https://github.com/rancher-sandbox/cOS-toolkit/tree/master/packages/utils/jq) (optional)

_Note_: Running `make` deps will install only `luet`, `luet-makeiso`, `yq` and `jq`. `squashfs-tools` and `xorriso` needs to be provided by the OS.

### Manually install dependencies

To install luet locally, you can also run as root:
```bash
$> curl https://get.mocaccino.org/luet/get_luet_root.sh | sh
kkaempf marked this conversation as resolved.
Show resolved Hide resolved
```
or either build from source (see [luet](https://github.com/mudler/luet)).

The Luet official repository that are being installed by the script above are:
kkaempf marked this conversation as resolved.
Show resolved Hide resolved
kkaempf marked this conversation as resolved.
Show resolved Hide resolved
- [official Luet repository](https://github.com/Luet-lab/luet-repo)
kkaempf marked this conversation as resolved.
Show resolved Hide resolved
- [mocaccino-extra repository](https://github.com/mocaccinoOS/mocaccino-extra) (installable afterwards also with `luet install -y repository/mocaccino-extra-stable`) that contains the `yq` and `jq` versions that are used by the CI.
kkaempf marked this conversation as resolved.
Show resolved Hide resolved


#### luet-makeiso

Available in the [official Luet
kkaempf marked this conversation as resolved.
Show resolved Hide resolved
repository](https://github.com/Luet-lab/luet-repo). After installing
`luet` with the curl command above, it's sufficient to:

```bash
$> luet install -y extension/makeiso
kkaempf marked this conversation as resolved.
Show resolved Hide resolved
```

to install it locally; otherwise grab the binary from [luet-makeiso](https://github.com/mudler/luet-makeiso) releases.

#### yq and jq
`yq` (version `3.x`) and `jq` are used to retrieve the list of
packages to build in order to produce the final ISOs. Those are not
strictly required, see the Note below.


They are installable with:

```bash
$> luet install -y utils/jq utils/yq
kkaempf marked this conversation as resolved.
Show resolved Hide resolved
```

_Note_: `yq` and `jq` are just used to generate the list of packages to build, and you don't need to have them installed if you manually specify the packages to be compiled.
99 changes: 25 additions & 74 deletions docs/dev.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
# cOS-Toolkit Developer Documentation

Welcome!

The cOS Distribution is entirely built over GitHub. You can check the pipelines in the `.github` folder to see how the process looks like.
The cOS (containerized OS) distribution is entirely built over GitHub. You can check the pipelines in the `.github` folder to see how the process looks like.

## Repository layout

Expand All @@ -13,7 +15,8 @@ The cOS Distribution is entirely built over GitHub. You can check the pipelines

## Forking and test on your own

By forking the `cOS` repository, you already have the Github Action workflow configured to start building and pushing your own `cOS` fork.
By forking the `cOS-toolkit` repository, you already have the Github Action workflow configured to start building and pushing your own `cOS` fork.

The only changes required to keep in mind for pushing images:
- set `DOCKER_PASSWORD` and `DOCKER_USERNAME` as Github secrets, which are needed to push the resulting docker images from the pipeline.
- Tweak or set the `Makefile`'s `REPO_CACHE` and `FINAL_REPO` accordingly. Those are used respectively for an image used for cache, and for the final image reference.
Expand All @@ -22,7 +25,7 @@ Those are not required for building - you can disable image push (`--push`) from

## Building locally

cOS has a docker image which can be used to build cOS locally in order to generate the cOS packages and the cOS iso from your checkout.
cOS has a Docker image which can be used to build cOS locally in order to generate the cOS packages and the cOS iso from your checkout.
kkaempf marked this conversation as resolved.
Show resolved Hide resolved

From your git folder:

Expand All @@ -38,99 +41,47 @@ $> source .envrc
$> cos-build
```

### Requirements for local build

To get requirements installed locally, run:

```bash
$> make deps
```

or you need:

- [luet](https://github.com/mudler/luet)
- [luet-makeiso](https://github.com/mudler/luet-makeiso)
- `squashfs-tools`
- `xorriso`
- `yq` (version `3.x`) (optional)
- `jq` (optional)

_Note_: Running `make` deps will install only `luet`, `luet-makeiso`, `yq` and `jq`. `squashfs-tools` and `xorriso` needs to be provided by the OS.

### Manually install dependencies

To install luet locally, you can also run as root:
```bash
$> curl https://get.mocaccino.org/luet/get_luet_root.sh | sh
```
or either build from source (see [luet](https://github.com/mudler/luet)).

The Luet official repository that are being installed by the script above are:
- [official Luet repository](https://github.com/Luet-lab/luet-repo)
- [mocaccino-extra repository](https://github.com/mocaccinoOS/mocaccino-extra) (installable afterwards also with `luet install -y repository/mocaccino-extra-stable`) that contains the `yq` and `jq` versions that are used by the CI.


#### luet-makeiso
### Build all packages locally

Available in the [official Luet repository](https://github.com/Luet-lab/luet-repo). After installing `luet` with the curl command above, is sufficient to:
Building locally has a [set of dependencies](dependencies.md) that
should be satisfied.

```bash
$> luet install -y extension/makeiso
Then you can run
```

to install it locally, otherwise grab the binary from [luet-makeiso](https://github.com/mudler/luet-makeiso) releases.

#### yq and jq
`yq` (version `3.x`) and `jq` are used to retrieve the list of packages to build in order to produce the final ISOs. Those are not strictly required, see the Note above.

Install the `mocaccino-extra` repository:

```bash
$> luet install -y repository/mocaccino-extra-stable
# make build
```
as root

They are installable with:

```bash
$> luet install -y utils/yq utils/yq
```

_Note_: `yq` and `jq` are just used to generate the list of packages to build, and you don't need to have them installed if you manually specify the packages to be compiled.

### Build all packages locally

```
$> make build
```

To clean from previous runs, run `make clean`.

_Note_: The makefile uses `yq` and `jq` to retrieve the packages to build from the iso specfile. If you don't have `jq` and `yq` installed, you must pass by the packages manually with `PACKAGES` (e.g. `PACKAGES="system/cos live/systemd-boot live/boot live/syslinux`).
_Note_: The makefile uses [`yq` and `jq`](dev.md#yq-and-jq) to
retrieve the packages to build from the iso specfile.

If you don't have `jq` and `yq` installed, you must pass by the packages manually with `PACKAGES` (e.g. `PACKAGES="system/cos live/systemd-boot live/boot live/syslinux"`).

You might want to build packages running as `root` or `sudo -E` if you intend to preserve file permissions in the resulting packages (mainly for `xattrs`, and so on).

### Build ISO

If using opensuse, first install the required deps:
If using SLES or openSUSE, first install the required deps:

```
$> zypper in -y squashfs xorriso dosfstools
# zypper in -y squashfs xorriso dosfstools
```

and then, simply run

```
$> make local-iso
# make local-iso
```

### Testing ISO changes

To test changes against a specific set of packages, you can for example:

```bash

$> make PACKAGES="live/init" build local-iso

```
# make PACKAGES="toolchain/yq" build local-iso
```

root is required because we want to keep permissions on the output packages (not really required for experimenting).
Expand All @@ -139,7 +90,7 @@ root is required because we want to keep permissions on the output packages (not

After you have the iso locally, run

```bash
```

$> QEMU=qemu-system-x86_64 make run-qemu

Expand All @@ -159,7 +110,7 @@ This will create a disk image at `.qemu/drive.img` and boot from the ISO.
With a fresh `drive.img`, `make run-qemu` will boot from ISO. You can then log in as `root` with password `cos` and install cOS on
the disk image with:

```bash
```
# cos-installer /dev/sda
```

Expand All @@ -176,15 +127,15 @@ We have a test suite which runs over SSH.

To create the vagrant image:

```bash
```

$> PACKER_ARGS="-var='feature=vagrant' -only virtualbox-iso" make packer

```

To run the tests:

```bash
```

$> make test

Expand Down
8 changes: 4 additions & 4 deletions docs/high_level_architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ This page tries to encompass the [`cos-toolkit`](https://github.com/rancher-sand

## Design goals

- Blueprints to build immutable linux derivatives from container images
- A Manifest that manipulates base images with a common abstraction layer. Currently Opensuse 15.3, Fedora and Ubuntu are supported
- Blueprints to build immutable Linux derivatives from container images
kkaempf marked this conversation as resolved.
Show resolved Hide resolved
- A manifest that manipulates base images with a common abstraction layer. Currently openSUSE 15.3, Fedora and Ubuntu are supported
- A workflow to maintain, support and deliver custom-OS and upgrades to end systems
- Derivatives have the same “foundation” manifest - easy to customize on top, add packages: systemd, dracut and grub as a foundation stack.
- Upgrades delivered with container registry images ( also workflow with `docker run` && `docker commit` supported! )
Expand All @@ -15,9 +15,9 @@ This page tries to encompass the [`cos-toolkit`](https://github.com/rancher-sand

## High level overview

Cos-toolkit encompasses several components required for building and distributing OS images. [This issue](https://github.com/rancher-sandbox/cOS-toolkit/issues/108) summarize the current state, and how we plan to integrate them in a single CLI to improve the user experience.
cOS-Toolkit encompasses several components required for building and distributing OS images. [This issue](https://github.com/rancher-sandbox/cOS-toolkit/issues/108) summarize the current state, and how we plan to integrate them in a single CLI to improve the user experience.

cos-toolkit is also a manifest, which includes package definitions of how the underlying OS is composed. It forms an abstraction layer, which is then translated to Dockerfiles and built by our CI (optionally) for re-usal. A derivative can be built by parts of the manifest, or reusing it entirely, container images included.
cOS-Toolkit is also a manifest, which includes package definitions of how the underlying OS is composed. It forms an abstraction layer, which is then translated to Dockerfiles and built by our CI (optionally) for re-usal. A derivative can be built by parts of the manifest, or reusing it entirely, container images included.

![High level overview](https://docs.google.com/drawings/d/e/2PACX-1vQQJOaISPbMxMYU44UT-M3ou9uGYOrzbXCRXMLPU8m7_ie3ke_08xCsyRLkFZJRB4VnzIeobPciEoQv/pub?w=942&h=532)

Expand Down