-
Notifications
You must be signed in to change notification settings - Fork 255
Fix applies_to in 2 troubleshooting documents: snapshot failures and restore from snapshot #4487
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 6 commits
0020c30
281b0df
8cd3676
c584e2a
c50616a
259db0a
abd060c
20631dd
f95adf1
1afcab8
40c3e54
f32a9d5
a97d970
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -4,11 +4,6 @@ mapped_pages: | |
| - https://www.elastic.co/guide/en/elasticsearch/reference/current/repeated-snapshot-failures.html | ||
| applies_to: | ||
| stack: | ||
| deployment: | ||
| eck: | ||
| ess: | ||
| ece: | ||
| self: | ||
| products: | ||
| - id: elasticsearch | ||
| --- | ||
|
|
@@ -17,38 +12,43 @@ products: | |
|
|
||
| Repeated snapshot failures are usually an indicator of a problem with your deployment. Continuous failures of automated snapshots can leave a deployment without recovery options in cases of data loss or outages. | ||
|
|
||
| Elasticsearch keeps track of the number of repeated failures when executing automated snapshots. If an automated snapshot fails too many times without a successful execution, the health API will report a warning. The number of repeated failures before reporting a warning is controlled by the [`slm.health.failed_snapshot_warn_threshold`](elasticsearch://reference/elasticsearch/configuration-reference/snapshot-restore-settings.md#slm-health-failed-snapshot-warn-threshold) setting. | ||
| :::{include} /deploy-manage/_snippets/autoops-callout-with-ech.md | ||
| ::: | ||
|
|
||
| {{es}} keeps track of the number of repeated failures when executing automated snapshots with [{{slm}} ({{slm-init}})](/deploy-manage/tools/snapshot-and-restore/create-snapshots.md#automate-snapshots-slm) policies. If an automated snapshot fails too many times without a successful execution, the health API reports a warning. The number of repeated failures before reporting a warning is controlled by the [`slm.health.failed_snapshot_warn_threshold`](elasticsearch://reference/elasticsearch/configuration-reference/snapshot-restore-settings.md#slm-health-failed-snapshot-warn-threshold) setting. | ||
|
|
||
| In the event that an automated {{slm}} policy execution is experiencing repeated failures, follow these steps to get more information about the problem: | ||
| ## Review snapshot policy failures | ||
|
|
||
| If an automated {{slm-init}} policy execution is experiencing repeated failures, follow these steps to get more information about the problem: | ||
|
|
||
| :::::::{tab-set} | ||
|
|
||
| ::::::{tab-item} {{ech}} | ||
| In order to check the status of failing {{slm}} policies we need to go to Kibana and retrieve the [Snapshot Lifecycle Policy information](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-slm-get-lifecycle). | ||
| ::::::{tab-item} Using {{kib}} | ||
| In {{kib}}, you can view all configured {{slm-init}} policies and review their status and execution history. If the UI does not provide sufficient details about the failure, use the Console to retrieve the [snapshot policy information](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-slm-get-lifecycle) with the {{es}} API. | ||
|
|
||
| **Use {{kib}}** | ||
| 1. Go to **Snapshot and Restore > Policies** to see the list of configured policies. You can find the **Snapshot and Restore** management page using the navigation menu or the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). | ||
|
|
||
| 1. Log in to the [{{ecloud}} console](https://cloud.elastic.co?page=docs&placement=docs-body). | ||
| 2. On the **Hosted deployments** panel, click the name of your deployment. | ||
| :::{image} /troubleshoot/images/elasticsearch-reference-slm-policies.png | ||
| :alt: {{kib}} Console | ||
| :screenshot: | ||
| ::: | ||
|
|
||
| ::::{note} | ||
| If the name of your deployment is disabled your {{kib}} instances might be unhealthy, in which case contact [Elastic Support](https://support.elastic.co). If your deployment doesn’t include {{kib}}, all you need to do is [enable it first](../../deploy-manage/deploy/elastic-cloud/access-kibana.md). | ||
| :::: | ||
| 2. The policies table should be visible. Click on any of the policies to review the details and execution history. | ||
|
|
||
| 3. Open your deployment’s side navigation menu (placed under the Elastic logo in the upper left corner) and go to **Dev Tools > Console**. | ||
| 3. To get more detailed information about the failure, open {{kib}} **Dev Tools > Console**. You can find the **Console** using the navigation menu or the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. In other Troubleshooting pages where a similar approach was used (telling users to use Console), we separated out the common steps to only list them once. Step 4. in the Kibana tab is the exact same step as the one in the API tab. Maybe we could remove it from the Kibana tab and in the API tab, you can add a statement to say something like:
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. yes, definitely. I'll update the tabs to remove content duplication. |
||
|
|
||
| :::{image} /troubleshoot/images/elasticsearch-reference-kibana-console.png | ||
| :alt: {{kib}} Console | ||
| :screenshot: | ||
| ::: | ||
|
|
||
| 4. [Retrieve](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-slm-get-lifecycle) the {{slm}} policy: | ||
| 4. [Retrieve the affected {{slm-init}} policy](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-slm-get-lifecycle): | ||
|
|
||
| ```console | ||
| GET _slm/policy/<affected-policy-name> | ||
| ``` | ||
|
|
||
| The response will look like this: | ||
| The response looks like this: | ||
|
|
||
| ```console-result | ||
| { | ||
|
|
@@ -93,29 +93,20 @@ In order to check the status of failing {{slm}} policies we need to go to Kibana | |
| } | ||
| } | ||
| ``` | ||
|
|
||
| 1. The affected snapshot lifecycle policy. | ||
| 2. The information about the last failure for the policy. | ||
| 3. The time when the failure occurred in millis. Use the `human=true` request parameter to see a formatted timestamp. | ||
| 4. Error details containing the reason for the snapshot failure. | ||
|
|
||
|
|
||
| Snapshots can fail for a variety reasons. If the failures are due to configuration errors, consult the documentation for the repository that the automated snapshots are using. Refer to the [guide on managing repositories in ECE](/deploy-manage/tools/snapshot-and-restore/cloud-enterprise.md) if you are using such a deployment. | ||
|
|
||
|
|
||
| One common failure scenario is repository corruption. This occurs most often when multiple instances of {{es}} write to the same repository location. There is a [separate troubleshooting guide](diagnosing-corrupted-repositories.md) to fix this problem. | ||
|
|
||
| In the event that snapshots are failing for other reasons check the logs on the elected master node during the snapshot execution period for more information. | ||
| :::::: | ||
|
|
||
| ::::::{tab-item} Self-managed | ||
| [Retrieve](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-slm-get-lifecycle) the {{slm}} policy: | ||
| ::::::{tab-item} Using the {{es}} API | ||
| [Retrieve the affected {{slm-init}} policy](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-slm-get-lifecycle): | ||
|
|
||
| ```console | ||
| GET _slm/policy/<affected-policy-name> | ||
| ``` | ||
|
|
||
| The response will look like this: | ||
| The response looks like this: | ||
|
|
||
| ```console-result | ||
| { | ||
|
|
@@ -166,15 +157,14 @@ The response will look like this: | |
| 3. The time when the failure occurred in millis. Use the `human=true` request parameter to see a formatted timestamp. | ||
| 4. Error details containing the reason for the snapshot failure. | ||
|
|
||
| :::::: | ||
|
|
||
| Snapshots can fail for a variety reasons. If the failures are due to configuration errors, consult the documentation for the repository that the automated snapshots are using. | ||
| ::::::: | ||
|
|
||
| One common failure scenario is repository corruption. This occurs most often when multiple instances of {{es}} write to the same repository location. There is a [separate troubleshooting guide](diagnosing-corrupted-repositories.md) to fix this problem. | ||
| ## Possible causes | ||
|
|
||
| In the event that snapshots are failing for other reasons check the logs on the elected master node during the snapshot execution period for more information. | ||
| :::::: | ||
| Snapshots can fail for a variety of reasons. If the failures are due to configuration errors, consult the documentation for the repository type that the snapshot policy is using. Refer to the [guide on managing repositories in ECE](/deploy-manage/tools/snapshot-and-restore/cloud-enterprise.md) if you are using an Elastic Cloud Enterprise deployment. | ||
|
|
||
| ::::::: | ||
| One common failure scenario is repository corruption. This occurs most often when multiple instances of {{es}} write to the same repository location. There is a [separate troubleshooting guide](diagnosing-corrupted-repositories.md) to fix this problem. | ||
|
|
||
| :::{include} /deploy-manage/_snippets/autoops-callout-with-ech.md | ||
| ::: | ||
| If snapshots are failing for other reasons check the logs on the elected master node during the snapshot execution period for more information. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -3,18 +3,13 @@ mapped_pages: | |
| - https://www.elastic.co/guide/en/elasticsearch/reference/current/restore-from-snapshot.html | ||
| applies_to: | ||
| stack: | ||
| deployment: | ||
| eck: | ||
| ess: | ||
| ece: | ||
| self: | ||
| products: | ||
| - id: elasticsearch | ||
| --- | ||
|
|
||
| # Restore from snapshot [restore-from-snapshot] | ||
|
|
||
| Elasticsearch is using snapshots to store a copy of your data outside a cluster. You can restore a snapshot to recover indices and data streams for which there are no copies of the shards in the cluster. This can happen if the data (indices or data streams) was deleted or if the cluster membership changed and the current nodes in the system do not contain a copy of the data anymore. | ||
| {{es}} is using snapshots to store a copy of your data outside a cluster. You can restore a snapshot to recover indices and data streams for which there are no copies of the shards in the cluster. This can happen if the data (indices or data streams) was deleted or if the cluster membership changed and the current nodes in the system do not contain a copy of the data anymore. | ||
|
|
||
| ::::{important} | ||
| Restoring the missing data requires you to have a backup of the affected indices and data streams that is up-to-date enough for your use case. Don't proceed without confirming this. | ||
|
|
@@ -23,26 +18,17 @@ Restoring the missing data requires you to have a backup of the affected indices | |
|
|
||
| :::::::{tab-set} | ||
|
|
||
| ::::::{tab-item} {{ech}} | ||
| In order to restore the indices and data streams that are missing data: | ||
| ::::::{tab-item} Using {{kib}} | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think we can remove this tab as it just repeats the same steps as in the API tab. It's sufficient to mention that you can run these steps in the console or as direct API calls, but the steps themselves are identical. (Actually there's tiny language differences which makes it look even worse, as it they're drifting over time).
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I completely agree. I just didn't want to lose time on this, as the scope of this activity was to just fix the |
||
| To restore the indices and data streams with missing data: | ||
|
|
||
| **Use {{kib}}** | ||
|
|
||
| 1. Log in to the [{{ecloud}} console](https://cloud.elastic.co?page=docs&placement=docs-body). | ||
| 2. On the **Hosted deployments** panel, click the name of your deployment. | ||
|
|
||
| ::::{note} | ||
| If the name of your deployment is disabled your {{kib}} instances might be unhealthy, in which case contact [Elastic Support](https://support.elastic.co). If your deployment doesn’t include {{kib}}, all you need to do is [enable it first](../../deploy-manage/deploy/elastic-cloud/access-kibana.md). | ||
| :::: | ||
|
|
||
| 3. Open your deployment’s side navigation menu (placed under the Elastic logo in the upper left corner) and go to **Dev Tools > Console**. | ||
| 1. Go to **Dev Tools > Console**. You can find the **Console** using the navigation menu or the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). | ||
|
|
||
| :::{image} /troubleshoot/images/elasticsearch-reference-kibana-console.png | ||
| :alt: {{kib}} Console | ||
| :screenshot: | ||
| ::: | ||
|
|
||
| 4. To view the affected indices using the [cat indices API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cat-indices). | ||
| 2. Review the affected indices using the [cat indices API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cat-indices). | ||
|
|
||
| ```console | ||
| GET _cat/indices?v&health=red&h=index,status,health | ||
|
|
@@ -58,7 +44,7 @@ In order to restore the indices and data streams that are missing data: | |
|
|
||
| The `red` health of the indices above indicates that these indices are missing primary shards, meaning they are missing data. | ||
|
|
||
| 5. In order to restore the data we need to find a snapshot that contains these two indices. To find such a snapshot use the [get snapshot API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-snapshot-get). | ||
| 5. To restore the data we need to find a snapshot that contains these two indices. To find such a snapshot use the [get snapshot API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-snapshot-get). | ||
|
|
||
| ```console | ||
| GET _snapshot/my_repository/*?verbose=false | ||
|
|
@@ -227,10 +213,10 @@ In order to restore the indices and data streams that are missing data: | |
| For more guidance on creating and restoring snapshots see [this guide](../../deploy-manage/tools/snapshot-and-restore.md). | ||
| :::::: | ||
|
|
||
| ::::::{tab-item} Self-managed | ||
| In order to restore the indices that are missing shards: | ||
| ::::::{tab-item} Using the {{es}} API | ||
| To restore the indices and data streams with missing data: | ||
|
|
||
| 1. View the affected indices using the [cat indices API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cat-indices). | ||
| 1. Review the affected indices using the [cat indices API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cat-indices). | ||
|
|
||
| ```console | ||
| GET _cat/indices?v&health=red&h=index,status,health | ||
|
|
||
Uh oh!
There was an error while loading. Please reload this page.