Merge pull request #381 from amazonlinux/updater-docs

Add README files for updater/ and updog

workspaces/updater/README.md

@@ -0,0 +1,53 @@
+# Thar update infrastructure
+This document describes the Thar update system and its components, namely;
+
+- tough: implementation of "The Update Framework" (TUF)
+- updog: update client that interfaces with a TUF repository to find and apply updates
+- signpost: helper tool to update partition priority flags
+- dogswatch: an optional component that coordinates node updates with the rest of the cluster
+
+![Update overview](update-system.png)
+## TUF and tough
+A TUF repository is a collection of metadata files and 'target' files that clients can download.
+In Thar there are four metadata files used to establish trusted updates: root, timestamp, snapshot, and targets.
+Each metadata file is individually signed and serves a distinct purpose.
+
+The root.json file begins the chain of trust for working with a TUF repository.
+It lists the keys that a Thar instance trusts in order to verify the rest of the metadata.
+The root.json file is part of the Thar image, but can be updated from the TUF repo assuming the new root.json is signed by a certain number of keys from the old.
+In the Thar model multiple keys are used to sign root.json, and the loss of some amount of keys under the threshold does not prevent updating to a new root.json that contains new trusted keys.
+
+The timestamp.json file contains the hash of the current snapshot.json file and is frequently re-signed to prevent the use of out-of-date metadata.
+
+The snapshot.json file lists the current versions of all other metadata files in the TUF repository, aside from timestamp.json.
+Once verified by timestamp.json the snapshot file ensures the client only sees the most up-to-date versions of root.json and targets.json.
+
+The targets.json file lists all the available 'target' files in the TUF repository and their hashes.
+For Thar this includes a 'manifest.json' file and any update images or migration files that have been made available.
+(For more information on migrations see [migration](../api/migration))
+
+Update metadata and files can be found by requesting and verifying these metadata files in order, and then requesting the manifest.json target which describes all available updates.
+Any file listed in the manifest is also a TUF 'target' listed in targets.json and can only be downloaded via the TUF repository, preventing the client from downloading untrusted data.
+
+## Updog
+Updog is the client tool that interacts with a 'The Update Framework' (TUF) repository to download and write updates to a Thar partition.
+Updog will parse the manifest.json file from the TUF repository and will update to a new image if the following criteria are satisfied:
+### Version & Flavor
+By default Updog only considers updates resulting in a version increase; downgrades are possible by using the `--image` option to force a specific version.
+Updog will respect the `max_version` field in the update manifest and refuse to update beyond it.
+Updog also considers the Thar "flavor" of its current image and will not download updates for a different flavor.
+### Datastore version
+Each update image has an associated datastore version.
+If an update would cause a change in datastore version, Updog will ensure the appropriate migration files are available to safely transition between datastore versions.
+### Update wave
+Updates may include "wave" information which provides a way for updates to be scheduled over time for groups of Thar hosts.
+Updog will find the update wave the host belongs to and jitter its update time within that range.
+If the calculated time has not passed yet, Updog returns the update timestamp to the caller so it can be called again at the correct time.
+
+Assuming all the requirements are met, Updog requests the update images from the TUF repository and writes them to the "inactive" partition.
+
+For more information on what's Updog see [Updog](updog/)
+## Signpost
+Once an update has been successfully written to the inactive partition, Updog calls the Signpost utility.
+This updates the priority bits in the GUID partition table of each partition and swaps the "active" and "inactive" partitions.
+For more information see [Signpost](signpost/)

workspaces/updater/update-system.drawio

@@ -0,0 +1 @@
+<mxfile modified="2019-10-16T17:26:00.146Z" host="drawio.corp.amazon.com" agent="Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:69.0) Gecko/20100101 Firefox/69.0" etag="Rv5cDU7fOmM64uIt0t11" pages="1" version="10.9.7" type="device"><diagram id="3ayFpOOYjWGc_ZlwSSnA" name="Page-1">7V3bdps6EP2aPNoLcfdjbm3SnrQ5uawkfelSjGzrFIMLOLb79UcEYYMkm4sFJmn6UiOEjPfsGW2NBnKknU6XnwM4m1z5DnKPVMVZHmlnR6oKtIFO/otbVknLAGhJwzjADu20abjFfxBtVGjrHDsozHWMfN+N8CzfOPQ9Dw2jXBsMAn+R7zby3fy3zuAYcQ23Q+jyrQ/YiSa0FZiDzYkLhMcT+tW2aiUnpjDtTH9JOIGOv8g0aedH2mng+1Hyabo8RW4MXopLct2nLWfXNxYgLypzwflZ9Hin3Hr/Lv8MgrOTr9/uby56Or25F+jO6S+mdxutUggCf+45KB4FHGkniwmO0O0MDuOzC2J00jaJpi49vf6V8QF/i/SuX1AQoWWmid7yZ+RPURSsSBd61rDoHVL+qAo9XmysYQxo2yRjCF2jjZAyYLweewMS+UBxEmP2K4C9xT/3w2+X+OFirF9+f3h67AG9CmZKMWYSYLJVBqYBDxNQBTABvSmYNMkoOTCcrFkoATLLyENm6hoHmWoLIbP2h0zojVoHmQUGeZjWLMoySwSTJoNZQphUo4Mw1UTJBE2hxHPJ8cfhAkbDSY/MllHguy4KJPqk7NCvM5iapsFjagkwHTQFKTC7RzzTKBH5dQFKDcJkdQ+mQUqeDsFkczDdz4iP7oeVbC80zTy/dEXghSJlYTaG22BHZPPICqTDMY1F0xDI2XbRTL8sg+YVHgcwwr5Hmm8jP+ABJbDM4o9DGDjtKzdbz4OoAR5EoXBTGgMRcCAeDyP8QpBTrmEQ4QRNBkXycyPGecms/Aud+q4fkBbP90jPkxF2XaYJunjsxfgTEMkcrp3E4GGyaj2mJ6bYceKvEZom7w4564yIMLildyhYYVbnO6MeVZ03lTDqNmYpVTA3mW5sBwe/kI/j+OOlBwXmS7qRr830/BuNChRgdMyqvNidQg+PUBj1/wsFrlcppyEBMiZiAfPgYZ+XsnEqqjfbGq0ODJkhWlG1Cxmv1567DZl9cMh4qTYhkam7kJmCSNYuZBqvxyI8JYEMTmfdiGUqk2sEB6eZxsuv0IOzcOJ3JPyzkKkHj/8ar4MiGIxRFHYTMe3wjskntOMZsxtwsetK0ZKoZbhESdqM0FZSuUtaf8/jTbATKnzXxxuNnVwX69gczGnH+EQvfN07PCYdgDVbbh/lOW24u/9EOt+gmR9isspdZQT+M3sRaUu+nRkrEHTs/upAxrzJrAB4uq0p2A7deD17hSLowAh21AiyV2UqEwIECc5W12QanwW+S2aYv8QgdsfswWvxqoFVVUsFVjnD3k1gUBB+S0XqTnJNAr3Y/WugCaZ80faYJYFhP8zF19/655enH/M/Fy9fruY9YtESO/7Ic47jqpwYRheGIR7mbUHgCFaPMUh9Iz18yp47W1IEk6NVerTE0WPaj3x+2gxBjjYXxQfpNYnd06oerXSiOvTnwRDt8DSajEr09I5+6UYGcnI1SLzFs7UuAoOmbQFy4WvWMnu3IivTb7j28auDphqSCVgmu/OV/HB61YYr3ECWxgzEVgIkyHADEWbAVabbLO4Qbr9hi1kjWCBXAUU+JCNuGL3GtD7JS5RoFJE8v8PEsTDl8oa/T1n6Crlc12/qkzwlbzHL7U6xnCkqsNkysbIkZ9SWZZfjuCwaliiBydCQzmDcpJfSTpUQeqvQNSToRLyPvDZ/wm56k9lJ+EjVHIjs0ZCbsckZc2ij59HrrTn0+j3ZbZdkd5oV+GC3THYDPqEnjd4gS+4M17fQuwYHU4fIaRG1SIyU8oiS/C6mrSKkbWnPOBS/dUajcOWrDRO8qkRZ13ek1TNskXZBfxXs7s/WyKltSCBB4VI5DZQjebU5aT+nNZDt6CKntdVnzTQZR9vPtbqmdximg7qqviWPAayHD3Z7ANtfp4nnRj1AUDQlqCKZpnVUP0Ef/Bz5/rYkMXdlv9+v07dyOrrtbQpmYaiKdg5FSTEZaWOxIdVDhzKrKJaJ9XWa6BCrCSb+jewhGgpFy7Nt6IZSJf5lBcJO33hzgfKw0rlynGSVhdHCzK9KSPGVdo0dfmGuneYaBZj8qjgfuyMRaO0W34y7IEAEgyVyl4FpadBsZ2WZ+tGHe9RxDxvsdI/C/prVhoyom0yUNvsMgFU0/Yi8jJ0tqs83Wx3orU0brKbhHujqmGPoW+536wqT6a+bbcwzDaY398v/lKC2rBxNWqL7QfRaRGfv12hFIJV4QvD9E/cjQjNMk8OtEo9Vvn9udWvP/NDc2jcomlYbQbFuvng3d9OVXl9R7Nxqj4ra7Zvl5IBbN9ZNz3Ry7Zg+gd0RL3lra0d9p+Qt7G8otlSnEr5eRcrSsaSrVd8D3crxjjDSYl4HYNpG36hHShsUDiVPGAipUG2xJJcKxTtrb40KFtBlUUEwVMNUkLIzW48K+yR9ukoFXZVGBX6ohqnAl3uDpNz6nBj72cXhBHvjWEwE8zDiWLJ/mbSLRvEIsh5i38qdKi9JYyK1RVfr2e1H0bvkZNTki18lx+8jq4mRbsj4GL0kJprPHBjFLyOYNvdIS/fNZQnePKOmbe2Yiy9L0xJznfkLz/Whk9gLT+EYNfGYyxswksa/Ra9ln1K7OAcqr/9KG6CjE6JpWNJkMj9UwxMi4DevlcR7Lz0c4STCpqH2L3RdU7cP7boFzwwnVUzXKAhxGKHXh7zK1UjFb7GKX/T7th/bbfSlPjaTXGnznT4XX/6d3Ixt8NM7+/4QXDrXD9c/evuWgNd5JKx6ooOfEoQBvF6+T3sXWTuuppmtQJUX94VEqqYHSm2btEItmURS3yORNPZJ1oaJVLf4TYKwLL2tUEyXrmhJyxxIS67wQzVMBT77ridS8ng2c1f5VfvbkJKcUihFnW2rQNH7jSRJSXK4+aMJiT03f3pCO/8f</diagram></mxfile>

workspaces/updater/updog/README.md

@@ -1,3 +1,49 @@
 # what is updog
 
 not much what's up with you
+
+## no really
+
+The Updog client provides an interface to a TUF repository and prepares for, downloads, and applies updates to the Thar instance. Updog can be called manually, but will more commonly be called automatically by some cluster orchestrator. For usage run `updog --help`.
+
+## Quick reference
+
+### Check for the most recent update
+```
+# updog check-update
+aws-k8s-0.1.4 (v0.0)
+```
+
+### List all available updates, including older versions
+```
+# updog check-update --all
+aws-k8s-0.1.4 (v0.0)
+aws-k8s-0.1.2 (v0.0)
+aws-k8s-0.1.1 (v0.0)
+```
+
+### Specify JSON output
+```
+# updog check-update --json
+[{"flavor":"aws-k8s","arch":"x86_64","version":"0.1.4","max_version":"0.1.4","waves":{"512":"2019-10-03T20:45:52Z","1024":"2019-10-03T21:00:52Z","1536":"2019-10-03T22:00:52Z","2048":"2019-10-03T23:00:52Z"},"images":{"boot":"thar-x86_64-0.1.4-boot.ext4.lz4","root":"thar-x86_64-0.1.4-root.ext4.lz4","hash":"thar-x86_64-0.1.4-root.verity.lz4"}}]
+```
+
+### Try to update with wave information
+```
+# updog update
+Update available at 2019-10-03 21:24:00 UTC
+```
+Once timestamp has passed:
+```
+# updog update --timestamp 2019-10-03T21:24:00+00:00
+Starting update to 0.1.4
+Update applied: aws-k8s-0.1.4
+```
+
+### Force an immediate update, ignoring wave limits
+```
+# updog update --now
+Starting update to 0.1.4
+** Updating immediately **
+Update applied: aws-k8s-0.1.4
+```

workspaces/updater/updog/src/main.rs

@@ -134,16 +134,26 @@ USAGE:
 
 SUBCOMMANDS:
     check-update            Show if an update is available
+        [ -a | --all ]                Output all applicable updates
+
     prepare                 Download update files and migration targets
+
     update                  Perform an update if available
+        [ -i | --image version ]      Update to a specfic image version
+        [ -n | --now ]                Update immediately, ignoring wave limits
+        [ -r | --reboot ]             Reboot into new update on success
+        [ -t | --timestamp time ]     The timestamp from which to execute an update
+
     update-image            Download & write an update but do not update flags
-    update-apply            Update boot flags and reboot
-OPTIONS:
+        [ -i | --image version ]      Update to a specfic image version
+        [ -n | --now ]                Update immediately, ignoring wave limits
+        [ -t | --timestamp time ]     The timestamp to execute an update from
+
+    update-apply            Update boot flags (after having called update-image)
+        [ -r | --reboot ]             Reboot after updating boot flags
+
+GLOBAL OPTIONS:
     [ -j | --json ]               JSON-formatted output
-    [ -a | --all ]                Output all applicable updates
-    [ -n | --now ]                Update immediately, ignoring wave limits
-    [ -i | --image ]              Update to a specfic image version
-    [ -r | --reboot ]             Reboot upon updating boot flags
     [ --verbose --verbose ... ]   Increase log verbosity");
     std::process::exit(1)
 }