diff --git a/layer.md b/layer.md index b2eac7ad7..abee44855 100644 --- a/layer.md +++ b/layer.md @@ -1,18 +1,74 @@ -# Creating an Image Filesystem Changeset +# Image Layer Filesystem Changeset -An example of creating an Image Filesystem Changeset follows. +This document describes how to serialize a filesystem and filesystem changes like removed files into a blob called a layer. +One or more layers are ordered on top of each other to create a complete filesystem. +This document will use a concrete example to illustrate how to create and consume these filesystem layers. -An image root filesystem is first created as an empty directory. -Here is the initial empty directory structure for a changeset using the randomly-generated directory name `c3167915dc9d` ([actual layer DiffIDs are generated based on the content](#id_desc)). +## Distributable Format + +Layer Changesets for the [mediatype](./media-types.md) `application/vnd.oci.image.layer.tar+gzip` MUST be packaged in [tar archive][tar-archive]. +Layer Changesets for the [mediatype](./media-types.md) `application/vnd.oci.image.layer.tar+gzip` MUST NOT include duplicate entries for file paths in the resulting [tar archive][tar-archive]. + +## Change Types + +Types of changes that can occur in a changeset are: + +* Additions +* Modifications +* Removals + +Additions and Modifications are represented the same in the changeset tar archive. + +Removals are represented using "[whiteout](#whiteouts)" file entries (See [Representing Changes](#representing-changes)). + +### File Types + +Throughout this document section, the use of word "files" includes: + +* regular files +* directories +* sockets +* symbolic links +* block devices +* character devices +* FIFOs + +### File Attributes + +Where supported, MUST include file attributes for Additions and Modifications include: + +* Modification Time (`mtime`) +* User ID (`uid`) + * User Name (`uname`) *secondary to `uid`* +* Group ID (`gid `) + * Group Name (`gname`) *secondary to `gid`* +* Mode (`mode`) +* Extended Attributes (`xattrs`) +* Symlink reference (`linkname`) + +[Sparse files](https://en.wikipedia.org/wiki/Sparse_file) SHOULD NOT be used because they lack consistent support across tar implementations. + +## Creating + +### Initial Root Filesystem + +The initial root filesystem is the base or parent layer. + +For this example, an image root filesystem has an initial state as an empty directory. +The name of the directory is not relevant to the layer itself, only for the purpose of producing comparisons. + +Here is an initial empty directory structure for a changeset, with a unique directory name `rootfs-c9d-v1`. ``` -c3167915dc9d/ +rootfs-c9d-v1/ ``` +### Populate Initial Filesystem + Files and directories are then created: ``` -c3167915dc9d/ +rootfs-c9d-v1/ etc/ my-app-config bin/ @@ -20,19 +76,37 @@ c3167915dc9d/ my-app-tools ``` -The `c3167915dc9d` directory is then committed as a plain Tar archive with entries for the following files: +The `rootfs-c9d-v1` directory is then created as a plain [tar archive][tar-archive] with relative path to `rootfs-c9d-v1`. +Entries for the following files: ``` -etc/my-app-config -bin/my-app-binary -bin/my-app-tools +./ +./etc/ +./etc/my-app-config +./bin/ +./bin/my-app-binary +./bin/my-app-tools ``` -To make changes to the filesystem of this container image, create a new directory, such as `f60c56784b83`, and initialize it with a snapshot of the parent image's root filesystem, so that the directory is identical to that of `c3167915dc9d`. -NOTE: a copy-on-write or union filesystem can make this very efficient: +### Populate a Comparison Filesystem + +Create a new directory and initialize it with an copy or snapshot of the prior root filesystem. +Example commands that can preserve [file attributes](#file-attributes) to make this copy are: +* [cp(1)](http://linux.die.net/man/1/cp): `cp -a rootfs-c9d-v1/ rootfs-c9d-v1.s1/` +* [rsync(1)](http://linux.die.net/man/1/rsync): `rsync -aHAX rootfs-c9d-v1/ rootfs-c9d-v1.s1/` +* [tar(1)](http://linux.die.net/man/1/tar): `mkdir rootfs-c9d-v1.s1 && tar --acls --xattrs -C rootfs-c9d-v1/ -c . | tar -C rootfs-c9d-v1.s1/ --acls --xattrs -x` (including `--selinux` where supported) + +Any [changes](#change-types) to the snapshot MUST NOT change or affect the directory it was copied from. + +For example `rootfs-c9d-v1.s1` is an identical snapshot of `rootfs-c9d-v1`. +In this way `rootfs-c9d-v1.s1` is prepared for updates and alterations. + +**Implementor's Note**: *a copy-on-write or union filesystem can efficiently make directory snapshots* + +Initial layout of the snapshot: ``` -f60c56784b83/ +rootfs-c9d-v1.s1/ etc/ my-app-config bin/ @@ -40,14 +114,16 @@ f60c56784b83/ my-app-tools ``` -This example change is going to add a configuration directory at `/etc/my-app.d` which contains a default config file. -There's also a change to the `my-app-tools` binary to handle the config layout change. -The `f60c56784b83` directory then looks like this: +See [Change Types](#change-types) for more details on changes. + +For example, add a directory at `/etc/my-app.d` containing a default config file, removing the existing config file. +Also a change (in attribute or file content) to `./bin/my-app-tools` binary to handle the config layout change. + +Following these changes, the representation of the `rootfs-c9d-v1.s1` directory: ``` -f60c56784b83/ +rootfs-c9d-v1.s1/ etc/ - .wh.my-app-config my-app.d/ default.cfg bin/ @@ -55,35 +131,70 @@ f60c56784b83/ my-app-tools ``` -This reflects the removal of `/etc/my-app-config` and creation of a file and directory at `/etc/my-app.d/default.cfg`. -`/bin/my-app-tools` has also been replaced with an updated version. -Before committing this directory to a changeset, because it has a parent image, it is first compared with the directory tree of the parent snapshot, `f60c56784b83`, looking for files and directories that have been added, modified, or removed. +### Determining Changes + +When two directories are compared, the relative root is the top-level directory. +The directories are compared, looking for files that have been [added, modified, or removed](#change-types). + +For this example, `rootfs-c9d-v1/` and `rootfs-c9d-v1.s1/` are recursively compared, each as relative root path. + The following changeset is found: ``` +Added: /etc/my-app.d/ Added: /etc/my-app.d/default.cfg Modified: /bin/my-app-tools Deleted: /etc/my-app-config ``` -A Tar Archive is then created which contains *only* this changeset: +This reflects the removal of `/etc/my-app-config` and creation of a file and directory at `/etc/my-app.d/default.cfg`. +`/bin/my-app-tools` has also been replaced with an updated version. + +### Representing Changes -- Added and modified files and directories in their entirety -- Deleted files or directory marked with a whiteout file +A [tar archive][tar-archive] is then created which contains *only* this changeset: -A whiteout file is an empty file that prefixes the deleted paths basename `.wh.`. -When a whiteout is found in the upper changeset of a filesystem, any matching name in the lower changeset is ignored, and the whiteout itself is also hidden. -As files prefixed with `.wh.` are special whiteout tombstones it is not possible to create a filesystem which has a file or directory with a name beginning with `.wh.`. +- Added and modified files and directories in their entirety +- Deleted files or directories marked with a [whiteout file](#whiteouts) -The resulting Tar archive for `f60c56784b83` has the following entries: +The resulting tar archive for `rootfs-c9d-v1.s1` has the following entries: ``` -/etc/my-app.d/default.cfg -/bin/my-app-tools -/etc/.wh.my-app-config +./etc/my-app.d/ +./etc/my-app.d/default.cfg +./bin/my-app-tools +./etc/.wh.my-app-config ``` -Whiteout files MUST only apply to resources in lower layers. +Where the basename name of `./etc/my-app-config` is now prefixed with `.wh.`, and will therefore be removed when the changeset is applied. + +## Applying + +Layer Changesets of [mediatype](./media-types.md) `application/vnd.oci.image.layer.tar+gzip` are applied rather than strictly extracted in normal fashion for tar archives. + +Applying a layer changeset requires consideration for the [whiteout](#whiteouts) files. +In the absence of any [whiteout](#whiteouts) files in a layer changeset, the archive is extracted like a regular tar archive. + + +### Changeset over existing files + +This section covers applying an entry in a layer changeset, if the file path already exists. + +If the file path is a directory, then the existing path just has it's attribute set from the layer changeset for that filepath. +If the file path is any other file type (regular file, FIFO, etc), then the: +* file path is unlinked (See [`unlink(2)`](http://linux.die.net/man/2/unlink)) +* create the file + * If a regular file then content written. +* set attributes on the filepath + +## Whiteouts + +A whiteout file is an empty file with a special filename that signifies a path should be deleted. +A whiteout filename consists of the prefix .wh. plus the basename of the path to be deleted. +As files prefixed with `.wh.` are special whiteout markers, it is not possible to create a filesystem which has a file or directory with a name beginning with `.wh.`. + +Once a whiteout is applied, the whiteout itself MUST also be hidden. +Whiteout files MUST only apply to resources in lower/parent layers. Files that are present in the same layer as a whiteout file can only be hidden by whiteout files in subsequent layers. The following is a base layer with several resources: @@ -117,6 +228,8 @@ a/.wh..wh..opq Implementations SHOULD generate layers such that the whiteout files appear before sibling directory entries. +### Opaque Whiteout + In addition to expressing that a single entry should be removed from a lower layer, layers may remove all of the children using an opaque whiteout entry. An opaque whiteout entry is a file with the name `.wh..wh..opq` indicating that all siblings are hidden in the lower layer. Let's take the following base layer as an example: @@ -139,7 +252,7 @@ bin/ ``` This is called _opaque whiteout_ format. -An _opaque whiteout_ file hides _all_ children of the `bin/` including sub-directories and all descendents. +An _opaque whiteout_ file hides _all_ children of the `bin/` including sub-directories and all descendants. Using _explicit whiteout_ files, this would be equivalent to the following: ``` @@ -151,8 +264,10 @@ bin/ In this case, a unique whiteout file is generated for each entry. If there were more children of `bin/` in the base layer, there would be an entry for each. -Note that this opaque file will apply to _all_ children, including sub-directories, other resources and all descendents. +Note that this opaque file will apply to _all_ children, including sub-directories, other resources and all descendants. Implementations SHOULD generate layers using _explicit whiteout_ files, but MUST accept both. Any given image is likely to be composed of several of these Image Filesystem Changeset tar archives. + +[tar-archive]: https://en.wikipedia.org/wiki/Tar_(computing)