diff --git a/master/hooks.md b/master/hooks.md index 5097f4090e..3d042b8110 100644 --- a/master/hooks.md +++ b/master/hooks.md @@ -14,11 +14,6 @@ As of version v0.7.0, Ark also supports "post" hooks - these execute after all c completed, as well as after all the additional items specified by custom actions have been backed up. -An example of when you might use both pre and post hooks is freezing a file system. If you want to -ensure that all pending disk I/O operations have completed prior to taking a snapshot, you could use -a pre hook to run `fsfreeze --freeze`. Next, Ark would take a snapshot of the disk. Finally, you -could use a post hook to run `fsfreeze --unfreeze`. - There are two ways to specify hooks: annotations on the pod itself, and in the Backup spec. ### Specifying Hooks As Pod Annotations @@ -51,4 +46,38 @@ Ark v0.7.0+ continues to support the original (deprecated) way to specify pre ho Please see the documentation on the [Backup API Type][1] for how to specify hooks in the Backup spec. +## Hook Example with fsfreeze + +We are going to walk through using both pre and post hooks for freezing a file system. Freezing the +file system is useful to ensure that all pending disk I/O operations have completed prior to taking a snapshot. + +We will be using [example/nginx-app/with-pv.yaml][2] for this example. Follow the [steps for your provider][3] to +setup this example. + +### Annotations + +The Ark [example/nginx-app/with-pv.yaml][2] serves as an example of adding the pre and post hook annotations directly +to your declarative deployment. Below is an example of what updating an object in place might look like. + +```shell +kubectl annotate pod -n nginx-example -l app=nginx \ + pre.hook.backup.ark.heptio.com/command='["/sbin/fsfreeze", "--freeze", "/var/log/nginx"]' \ + pre.hook.backup.ark.heptio.com/container=fsfreeze \ + post.hook.backup.ark.heptio.com/command='["/sbin/fsfreeze", "--unfreeze", "/var/log/nginx"]' \ + post.hook.backup.ark.heptio.com/container=fsfreeze +``` + +Now test the pre and post hooks by creating a backup. You can use the Ark logs to verify that the pre and post +hooks are running and exiting without error. + +```shell +ark backup create nginx-hook-test + +ark backup get nginx-hook-test +ark backup logs nginx-hook-test | grep hookCommand +``` + + [1]: api-types/backup.md +[2]: examples/nginx-app/with-pv.yaml +[3]: cloud-common.md diff --git a/master/troubleshooting.md b/master/troubleshooting.md index 04854af913..8a15089666 100644 --- a/master/troubleshooting.md +++ b/master/troubleshooting.md @@ -1,12 +1,63 @@ # Troubleshooting -These tips can help you troubleshoot known issues. If they don't help, you can [file an issue][4], or talk to us on the [#ark-dr channel][25] on the Kubernetes Slack server. +These tips can help you troubleshoot known issues. If they don't help, you can [file an issue][4], or talk to us on the [#ark-dr channel][25] on the Kubernetes Slack server. -* [Debug installation/setup issues][2] +In `ark` version >= `0.1.0`, you can use the `ark bug` command to open a [Github issue][4] by launching a browser window with some prepopulated values. Values included are OS, CPU architecture, `kubectl` client and server versions (if available) and the `ark` client version. This information isn't submitted to Github until you click the `Submit new issue` button in the Github UI, so feel free to add, remove or update whatever information you like. + +Some general commands for troubleshooting that may be helpful: + +* `ark backup describe ` - describe the details of a backup +* `ark backup logs ` - fetch the logs for this specific backup. Useful for viewing failures and warnings, including resources that could not be backed up. +* `ark restore describe ` - describe the details of a restore +* `ark restore logs ` - fetch the logs for this specific restore. Useful for viewing failures and warnings, including resources that could not be restored. +* `kubectl logs deployment/ark -n heptio-ark` - fetch the logs of the Ark server pod. This provides the output of the Ark server processes. + +## Getting ark debug logs + +You can increase the verbosity of the Ark server by editing your Ark deployment to look like this: + + +``` +kubectl edit deployment/ark -n heptio-ark +... + containers: + - name: ark + image: gcr.io/heptio-images/ark:latest + command: + - /ark + args: + - server + - --log-level # Add this line + - debug # Add this line +... +``` + + +## [Debug installation/setup issues][2] + +## [Debug restores][1] + +## Miscellaneous issues + +### Ark reports `custom resource not found` errors when starting up. + +Ark's server will not start if the required Custom Resource Definitions are not found in Kubernetes. Apply +the `examples/common/00-prereqs.yaml` file to create these defintions, then restart Ark. + +### `ark backup logs` returns a `SignatureDoesNotMatch` error + +Downloading artifacts from object storage utilizes temporary, signed URLs. In the case of S3-compatible +providers, such as Ceph, there may be differences between their implementation and the official S3 +API that cause errors. + +Here are some things to verify if you receive `SignatureDoesNotMatch` errors: + + * Make sure your S3-compatible layer is using [signature version 4][5] (such as Ceph RADOS v12.2.7) + * For Ceph, try using a native Ceph account for credentials instead of external providers such as OpenStack Keystone -* [Debug restores][1] [1]: debugging-restores.md [2]: debugging-install.md [4]: https://github.com/heptio/ark/issues +[5]: https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html [25]: https://kubernetes.slack.com/messages/ark-dr \ No newline at end of file diff --git a/v0.9.0/troubleshooting.md b/v0.9.0/troubleshooting.md index 718d57e164..8a15089666 100644 --- a/v0.9.0/troubleshooting.md +++ b/v0.9.0/troubleshooting.md @@ -1,6 +1,8 @@ # Troubleshooting -These tips can help you troubleshoot known issues. If they don't help, you can [file an issue][4], or talk to us on the [#ark-dr channel][25] on the Kubernetes Slack server. +These tips can help you troubleshoot known issues. If they don't help, you can [file an issue][4], or talk to us on the [#ark-dr channel][25] on the Kubernetes Slack server. + +In `ark` version >= `0.1.0`, you can use the `ark bug` command to open a [Github issue][4] by launching a browser window with some prepopulated values. Values included are OS, CPU architecture, `kubectl` client and server versions (if available) and the `ark` client version. This information isn't submitted to Github until you click the `Submit new issue` button in the Github UI, so feel free to add, remove or update whatever information you like. Some general commands for troubleshooting that may be helpful: @@ -31,11 +33,31 @@ kubectl edit deployment/ark -n heptio-ark ``` -* [Debug installation/setup issues][2] +## [Debug installation/setup issues][2] + +## [Debug restores][1] + +## Miscellaneous issues + +### Ark reports `custom resource not found` errors when starting up. + +Ark's server will not start if the required Custom Resource Definitions are not found in Kubernetes. Apply +the `examples/common/00-prereqs.yaml` file to create these defintions, then restart Ark. + +### `ark backup logs` returns a `SignatureDoesNotMatch` error + +Downloading artifacts from object storage utilizes temporary, signed URLs. In the case of S3-compatible +providers, such as Ceph, there may be differences between their implementation and the official S3 +API that cause errors. + +Here are some things to verify if you receive `SignatureDoesNotMatch` errors: + + * Make sure your S3-compatible layer is using [signature version 4][5] (such as Ceph RADOS v12.2.7) + * For Ceph, try using a native Ceph account for credentials instead of external providers such as OpenStack Keystone -* [Debug restores][1] [1]: debugging-restores.md [2]: debugging-install.md [4]: https://github.com/heptio/ark/issues +[5]: https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html [25]: https://kubernetes.slack.com/messages/ark-dr \ No newline at end of file