Skip to content
Merged
64 changes: 27 additions & 37 deletions troubleshoot/elasticsearch/repeated-snapshot-failures.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,6 @@ mapped_pages:
- https://www.elastic.co/guide/en/elasticsearch/reference/current/repeated-snapshot-failures.html
applies_to:
stack:
deployment:
eck:
ess:
ece:
self:
products:
- id: elasticsearch
---
Expand All @@ -17,38 +12,43 @@ products:

Repeated snapshot failures are usually an indicator of a problem with your deployment. Continuous failures of automated snapshots can leave a deployment without recovery options in cases of data loss or outages.

Elasticsearch keeps track of the number of repeated failures when executing automated snapshots. If an automated snapshot fails too many times without a successful execution, the health API will report a warning. The number of repeated failures before reporting a warning is controlled by the [`slm.health.failed_snapshot_warn_threshold`](elasticsearch://reference/elasticsearch/configuration-reference/snapshot-restore-settings.md#slm-health-failed-snapshot-warn-threshold) setting.
:::{include} /deploy-manage/_snippets/autoops-callout-with-ech.md
:::

{{es}} keeps track of the number of repeated failures when executing automated snapshots with [{{slm}} ({{slm-init}})](/deploy-manage/tools/snapshot-and-restore/create-snapshots.md#automate-snapshots-slm) policies. If an automated snapshot fails too many times without a successful execution, the health API reports a warning. The number of repeated failures before reporting a warning is controlled by the [`slm.health.failed_snapshot_warn_threshold`](elasticsearch://reference/elasticsearch/configuration-reference/snapshot-restore-settings.md#slm-health-failed-snapshot-warn-threshold) setting.

In the event that an automated {{slm}} policy execution is experiencing repeated failures, follow these steps to get more information about the problem:
## Review snapshot policy failures

If an automated {{slm-init}} policy execution is experiencing repeated failures, follow these steps to get more information about the problem:

:::::::{tab-set}

::::::{tab-item} {{ech}}
In order to check the status of failing {{slm}} policies we need to go to Kibana and retrieve the [Snapshot Lifecycle Policy information](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-slm-get-lifecycle).
::::::{tab-item} Using {{kib}}
In {{kib}}, you can view all configured {{slm-init}} policies and review their status and execution history. If the UI does not provide sufficient details about the failure, use the Console to retrieve the [snapshot policy information](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-slm-get-lifecycle) with the {{es}} API.

**Use {{kib}}**
1. Go to **Snapshot and Restore > Policies** to see the list of configured policies. You can find the **Snapshot and Restore** management page using the navigation menu or the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md).

1. Log in to the [{{ecloud}} console](https://cloud.elastic.co?page=docs&placement=docs-body).
2. On the **Hosted deployments** panel, click the name of your deployment.
:::{image} /troubleshoot/images/elasticsearch-reference-slm-policies.png
:alt: {{kib}} Console
:screenshot:
:::

::::{note}
If the name of your deployment is disabled your {{kib}} instances might be unhealthy, in which case contact [Elastic Support](https://support.elastic.co). If your deployment doesn’t include {{kib}}, all you need to do is [enable it first](../../deploy-manage/deploy/elastic-cloud/access-kibana.md).
::::
2. The policies table should be visible. Click on any of the policies to review the details and execution history.
Comment thread
eedugon marked this conversation as resolved.
Outdated

3. Open your deployment’s side navigation menu (placed under the Elastic logo in the upper left corner) and go to **Dev Tools > Console**.
3. To get more detailed information about the failure, open {{kib}} **Dev Tools > Console**. You can find the **Console** using the navigation menu or the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In other Troubleshooting pages where a similar approach was used (telling users to use Console), we separated out the common steps to only list them once. Step 4. in the Kibana tab is the exact same step as the one in the API tab.

Maybe we could remove it from the Kibana tab and in the API tab, you can add a statement to say something like:

You can run the following steps using either [API console](/explore-analyze/query-filter/tools/console.md) or direct [Elasticsearch API](elasticsearch://reference/elasticsearch/rest-apis/index.md) calls.

@eedugon eedugon Jan 13, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, definitely. I'll update the tabs to remove content duplication.


:::{image} /troubleshoot/images/elasticsearch-reference-kibana-console.png
:alt: {{kib}} Console
:screenshot:
:::

4. [Retrieve](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-slm-get-lifecycle) the {{slm}} policy:
4. [Retrieve the affected {{slm-init}} policy](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-slm-get-lifecycle):

```console
GET _slm/policy/<affected-policy-name>
```

The response will look like this:
The response looks like this:

```console-result
{
Expand Down Expand Up @@ -93,29 +93,20 @@ In order to check the status of failing {{slm}} policies we need to go to Kibana
}
}
```

1. The affected snapshot lifecycle policy.
2. The information about the last failure for the policy.
3. The time when the failure occurred in millis. Use the `human=true` request parameter to see a formatted timestamp.
4. Error details containing the reason for the snapshot failure.


Snapshots can fail for a variety reasons. If the failures are due to configuration errors, consult the documentation for the repository that the automated snapshots are using. Refer to the [guide on managing repositories in ECE](/deploy-manage/tools/snapshot-and-restore/cloud-enterprise.md) if you are using such a deployment.


One common failure scenario is repository corruption. This occurs most often when multiple instances of {{es}} write to the same repository location. There is a [separate troubleshooting guide](diagnosing-corrupted-repositories.md) to fix this problem.

In the event that snapshots are failing for other reasons check the logs on the elected master node during the snapshot execution period for more information.
::::::

::::::{tab-item} Self-managed
[Retrieve](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-slm-get-lifecycle) the {{slm}} policy:
::::::{tab-item} Using the {{es}} API
[Retrieve the affected {{slm-init}} policy](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-slm-get-lifecycle):

```console
GET _slm/policy/<affected-policy-name>
```

The response will look like this:
The response looks like this:

```console-result
{
Expand Down Expand Up @@ -166,15 +157,14 @@ The response will look like this:
3. The time when the failure occurred in millis. Use the `human=true` request parameter to see a formatted timestamp.
4. Error details containing the reason for the snapshot failure.

::::::

Snapshots can fail for a variety reasons. If the failures are due to configuration errors, consult the documentation for the repository that the automated snapshots are using.
:::::::

One common failure scenario is repository corruption. This occurs most often when multiple instances of {{es}} write to the same repository location. There is a [separate troubleshooting guide](diagnosing-corrupted-repositories.md) to fix this problem.
## Possible causes

In the event that snapshots are failing for other reasons check the logs on the elected master node during the snapshot execution period for more information.
::::::
Snapshots can fail for a variety of reasons. If the failures are due to configuration errors, consult the documentation for the repository type that the snapshot policy is using. Refer to the [guide on managing repositories in ECE](/deploy-manage/tools/snapshot-and-restore/cloud-enterprise.md) if you are using an Elastic Cloud Enterprise deployment.

:::::::
One common failure scenario is repository corruption. This occurs most often when multiple instances of {{es}} write to the same repository location. There is a [separate troubleshooting guide](diagnosing-corrupted-repositories.md) to fix this problem.

:::{include} /deploy-manage/_snippets/autoops-callout-with-ech.md
:::
If snapshots are failing for other reasons check the logs on the elected master node during the snapshot execution period for more information.
32 changes: 9 additions & 23 deletions troubleshoot/elasticsearch/restore-from-snapshot.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,18 +3,13 @@ mapped_pages:
- https://www.elastic.co/guide/en/elasticsearch/reference/current/restore-from-snapshot.html
applies_to:
stack:
deployment:
eck:
ess:
ece:
self:
products:
- id: elasticsearch
---

# Restore from snapshot [restore-from-snapshot]

Elasticsearch is using snapshots to store a copy of your data outside a cluster. You can restore a snapshot to recover indices and data streams for which there are no copies of the shards in the cluster. This can happen if the data (indices or data streams) was deleted or if the cluster membership changed and the current nodes in the system do not contain a copy of the data anymore.
{{es}} is using snapshots to store a copy of your data outside a cluster. You can restore a snapshot to recover indices and data streams for which there are no copies of the shards in the cluster. This can happen if the data (indices or data streams) was deleted or if the cluster membership changed and the current nodes in the system do not contain a copy of the data anymore.

::::{important}
Restoring the missing data requires you to have a backup of the affected indices and data streams that is up-to-date enough for your use case. Don't proceed without confirming this.
Expand All @@ -23,26 +18,17 @@ Restoring the missing data requires you to have a backup of the affected indices

:::::::{tab-set}

::::::{tab-item} {{ech}}
In order to restore the indices and data streams that are missing data:
::::::{tab-item} Using {{kib}}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can remove this tab as it just repeats the same steps as in the API tab. It's sufficient to mention that you can run these steps in the console or as direct API calls, but the steps themselves are identical. (Actually there's tiny language differences which makes it look even worse, as it they're drifting over time).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I completely agree. I just didn't want to lose time on this, as the scope of this activity was to just fix the applies_to tag, and we have similar issues in multiple other docs on the troubleshooting area. But I'll change it :)

To restore the indices and data streams with missing data:

**Use {{kib}}**

1. Log in to the [{{ecloud}} console](https://cloud.elastic.co?page=docs&placement=docs-body).
2. On the **Hosted deployments** panel, click the name of your deployment.

::::{note}
If the name of your deployment is disabled your {{kib}} instances might be unhealthy, in which case contact [Elastic Support](https://support.elastic.co). If your deployment doesn’t include {{kib}}, all you need to do is [enable it first](../../deploy-manage/deploy/elastic-cloud/access-kibana.md).
::::

3. Open your deployment’s side navigation menu (placed under the Elastic logo in the upper left corner) and go to **Dev Tools > Console**.
1. Go to **Dev Tools > Console**. You can find the **Console** using the navigation menu or the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md).

:::{image} /troubleshoot/images/elasticsearch-reference-kibana-console.png
:alt: {{kib}} Console
:screenshot:
:::

4. To view the affected indices using the [cat indices API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cat-indices).
2. Review the affected indices using the [cat indices API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cat-indices).

```console
GET _cat/indices?v&health=red&h=index,status,health
Expand All @@ -58,7 +44,7 @@ In order to restore the indices and data streams that are missing data:

The `red` health of the indices above indicates that these indices are missing primary shards, meaning they are missing data.

5. In order to restore the data we need to find a snapshot that contains these two indices. To find such a snapshot use the [get snapshot API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-snapshot-get).
5. To restore the data we need to find a snapshot that contains these two indices. To find such a snapshot use the [get snapshot API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-snapshot-get).

```console
GET _snapshot/my_repository/*?verbose=false
Expand Down Expand Up @@ -227,10 +213,10 @@ In order to restore the indices and data streams that are missing data:
For more guidance on creating and restoring snapshots see [this guide](../../deploy-manage/tools/snapshot-and-restore.md).
::::::

::::::{tab-item} Self-managed
In order to restore the indices that are missing shards:
::::::{tab-item} Using the {{es}} API
To restore the indices and data streams with missing data:

1. View the affected indices using the [cat indices API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cat-indices).
1. Review the affected indices using the [cat indices API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cat-indices).

```console
GET _cat/indices?v&health=red&h=index,status,health
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading