WIP

tests/README.md

@@ -3,8 +3,11 @@
 To run these tests, you need at least these dependencies on the host:
 
 1.    [bwrap](https://github.com/containers/bubblewrap/)
-1.    [gdb](https://www.gnu.org/software/gdb/)
-1.    [gnupg](https://www.gnupg.org/) >= 2.0
+
+If your host is *not* running Fedora Linux, you will also need either of:
+
+1.    [podman](https://github.com/containers/podman/)
+1.    [docker](https://github.com/docker/)
 
 Then run the command
 
@@ -14,6 +17,8 @@ The number of tests performed depends on features enabled at configure time,
 at least `--with-`/`--without-lua` and `--enable-`/`--disable-python`.
 See also the [INSTALL](../INSTALL) file for more information.
 
+## Selecting tests
+
 To run *single tests*, you can run the commands:
 
     make check TESTOPTS="$NNN $MMM"
@@ -34,66 +39,132 @@ For all available options, see the output of the command:
 By default, tests are executed in parallel using all available cores, pass
 a specific -jN value to limit.
 
-To drop into a test-like shell, run:
+## Developing RPM
 
-    make env
-
-See the printed help for details on how to use it.
-
-You can also run a containerized shell with your RPM checkout built, installed
-and ready to use:
+For manual testing of RPM, run:
 
     make shell
 
-This is equivalent to running:
+This will drop you into a containerized shell with your RPM checkout built and
+installed into the configured prefix.  The container is a minimal version of
+your host OS with only the basic tools installed, and has the `tests` directory
+mounted at `/srv` for your convenience.
 
-    make env
-    snapshot shell
+For any RPM source changes to take effect, simply rerun this target, any other
+changes to the filesystem will be retained.
 
-To factory-reset the container, run:
+To factory-reset the container (drop any of your changes), run:
 
     make reset
 
 ## How it works
 
+### Architecture
+
 The test suite is written using GNU Autotest and, once built, is a standalone
-Bourne shell script named `rpmtests`.  The script is supposed to be run as root
-and exercises the RPM installation in the root filesystem, preferably mounted
-as read-only.
+Bourne shell script named `rpmtests`.  The script is intended to be run as root
+and will exercise the RPM installation in the root filesystem.
 
 Each test in the suite that needs write access to the root filesystem (such as
 to install packages) runs RPM in a mutable, disposable container (a *snapshot*)
-on top of the root filesystem, using Bubblewrap and OverlayFS.
-
-Since running the test suite natively as root and installing build artifacts
-into the system is usually not desired on a workstation, `make check` runs the
-`rpmtests` script itself in a container on top of an OS filesystem tree that
-mirrors the running host and contains a fresh `make install` from the build
-directory.  This all works without root privileges thanks to the use of Linux
-`namespaces(7)`.
-
-### Building the tree
-
-The `mktree` executable is responsible for setting up the filesystem tree and
-running containers against it.  It is invoked by `make tree` which in turn is
-invoked by `make check`, `make env` and `make shell`, and may also be invoked
-directly if one wishes to avoid the CMake overhead.
-
-The executable is made during CMake configuration by choosing a *backend* (one
-of the files starting with `mktree.`) that's native to the host, configuring it
-and copying it to the build directory with the suffix stripped.
-
-Native backends use a package manager such as DNF or Zypper to bootstrap the
-tree with RPM's runtime dependencies matching the development headers used, RPM
-itself by running `make install` from the build directory with the `DESTDIR`
-environment variable set accordingly, and all the required test binaries.
-
-If no native backend is available, `mktree.podman` will be chosen as a fallback
-which performs the whole RPM build process in a Fedora container and reuses the
-same image for the tests.  This backend is primarily intended for use in our CI
-where portability is key (currently, Ubuntu VMs through GitHub Actions) and is
-therefore not optimized for iterative `make check` use.
-
-Developers wishing to contribute a native backend for their platform of choice
-are encouraged to consult the `mktree.README` file for the details on how to
-write one.
+on top of the root filesystem.  This is to prevent the individual tests from
+affecting each other.
+
+When hacking on RPM, one typically does not wish to install the build artifacts
+into the host system and run the risk of having their files purged by accident
+in case of a misbehaving test.  To avoid that, `make check` creates a minimal
+OS filesystem tree that mirrors the host, `make install`s RPM into it and runs
+the `rpmtests` script in a container on top.  This completely isolates the test
+suite from the host.
+
+The container technology used is a combination of
+[Bubblewrap](https://github.com/containers/bubblewrap/),
+[OverlayFS](https://docs.kernel.org/filesystems/overlayfs.html) and Linux
+`namespaces(7)`.  Thanks to the latter, this all works under a regular,
+non-root user.  Since most of these technologies are only available on Linux,
+this currently limits the test suite to Linux hosts only.
+
+### Making the tree
+
+The `mktree` executable is responsible for creating the filesystem tree and is
+invoked on `make tree` (which is a dependency of `make check` or `make shell`).
+The executable is made during CMake configuration by choosing an implementation
+(a *backend*) native to the running host, configuring it and copying it to the
+build directory under the `mktree` name.
+
+Most backends use the native package manager such as DNF or Zypper to install
+RPM's runtime dependencies matching the development headers used in the build,
+the test dependencies, and finally RPM itself using `make install` with the
+appropriate `$DESTDIR` value.  The tree is then cached in the form of layers
+(OS and RPM) and reused on subsequent `make check` runs, with only the RPM
+layer being redone.
+
+Currently, only Fedora Linux is supported natively with `mktree.fedora`, other
+distros automatically fall back to `mktree.podman` which is an implementation
+using Podman/Docker and the official Fedora OCI image to achieve the same, with
+the difference being that RPM is configured and built in a container.  This
+makes it more portable and thus ideal for our CI purposes where we currently
+run Ubuntu VMs, but it is not optimized for iterative `make check` use.
+
+## Advanced features
+
+### Common OS layer
+
+If you use multiple CMake build directories during development, you may want
+them to reuse the same OS layer to save time and disk space.  To enable that,
+simply create an empty `mktree.output` directory in the root of the source
+directory.  It will then be used by `mktree` to store the OS layer, instead of
+the build directory.
+
+Note that if you're using `git-worktree(1)`, the `mktree.output` directory will
+only be looked for in the original (full) checkout.
+
+### Interactive test-like environment
+
+This is like `make shell` on steroids, invoke it with:
+
+    make env
+
+This shell runs natively on your host, sources the `atlocal` file and mounts a
+test tree at `$RPMTEST`, much like what a typical test would do.  The advantage
+over `make shell` is that you can use your native tools to view and manipulate
+the contents of the filesystem.
+
+Note that the filesystem mounted at `$RPMTEST` is the same as the one used by
+`make shell`.  In fact, `make shell` is just a shorthand for
+
+    make env
+    runroot_other $SHELL
+
+with a few extras such as the `tests` directory mounted at `/srv`.
+
+A `motd(5)`-like piece of text is printed upon entering `make env` which
+contains more information on how to use it.
+
+### Specifying a mktree backend
+
+To override the autodetection, use the CMake option `MKTREE_BACKEND` with the
+desired backend name (`mktree` suffix).  This feature can be useful if you wish
+to use an alternative backend.
+
+For example, if you already have a throwaway development container or VM with
+RPM's build dependencies, you may prefer to just reuse the same filesystem to
+run the test suite against.  In such a case, configure your build with:
+
+    cmake -DMKTREE_BACKEND=rootfs ...
+
+In the future, this option may also be useful if we add an alternative, OS
+agnostic backend that e.g. reflinks the required binaries and libraries from
+the host instead of installing them from packages.
+
+### Running CI locally
+
+If you wish to verify that your changes pass in our CI before opening a PR, the
+following will run the same CI setup locally (needs Podman or Docker):
+
+    make ci
+
+This is equivalent to doing:
+
+    cmake -DMKTREE_BACKEND=podman ...
+    make check