Snapshot Restore doesn't restore all indices by default if there's one data stream

[lucabelluccini]

Elasticsearch version (bin/elasticsearch --version): 7.9.x onwards

Plugins installed: []

JVM version (java -version): included in Elasticsearch

OS version (uname -a if on a Unix-like system): Not relevant

Description of the problem including expected versus actual behavior:

When issuing a snapshot restore without any body, as explained in the documentation, I would expect all indices and data streams to be restored.

Since 7.9.x onwards, this is no more happening if there's at least a data stream in the cluster.

From the code, it seems we consider null, an empty list or _all as all indices. Reference:

elasticsearch/server/src/main/java/org/elasticsearch/cluster/metadata/IndexNameExpressionResolver.java

Line 709 in 91a1591

return aliasesOrIndices == null || aliasesOrIndices.isEmpty() || isExplicitAllPattern(aliasesOrIndices);

If we do not specify anything in the restore request, it defaults to empty array:

elasticsearch/server/src/main/java/org/elasticsearch/action/admin/cluster/snapshots/restore/RestoreSnapshotRequest.java

Line 43 in 91a1591

private String[] indices = Strings.EMPTY_ARRAY;

This is used in the RestoreService:

elasticsearch/server/src/main/java/org/elasticsearch/snapshots/RestoreService.java

Line 344 in 91a1591

final List<String> requestedIndicesInSnapshot = filterIndices(

I think the problematic section is at:

elasticsearch/server/src/main/java/org/elasticsearch/snapshots/RestoreService.java

Line 227 in a479a2a

requestIndices.removeAll(dataStreams.keySet());

But I would consider this as a breaking change.
I've tested it on 7.9.3 and 7.12.0.

If we want this to be the default behavior, then we need to update the docs and backport the docfix.

Workaround

Add the following body:

{ "indices": "*" }

Steps to reproduce:

1. Start an Elastic Cloud instance
2. Create 3 indices, even empty
3. Create a data stream following our docs - This step is optional worse in 7.12+ where the ILM & SLM history indices are using Data Streams by default
4. Take a snapshot
5. Delete all data streams (DELETE _data_stream/*?expand_wildcards=all), this is required as you cannot close them
6. Close all indices (POST *,.*/_close?expand_wildcards=all), or delete them...
7. Restore the snapshot without any body (just a POST _snapshot/found-snapshots/<sid>/_restore?wait_for_completion=true)

Provide logs (if relevant):

7.8.0

Request:

POST _snapshot/found-snapshots/cloud-snapshot-.../_restore?wait_for_completion=true

Response (abc, def, ghi):

{
  "snapshot": {
    "indices": [
      ".kibana-event-log-7.8.0-000001",
      ".security-tokens-7",
      ".security-7",
      ".slm-history-2-000001",
      "abc",
      ".apm-custom-link",
      "def",
      ".kibana_task_manager_1",
      "ilm-history-2-000001",
      ".apm-agent-configuration",
      "ghi",
      ".kibana_1"
    ],
    "snapshot": "cloud-snapshot...",
    "shards": {
      "successful": 12,
      "failed": 0,
      "total": 12
    }
  }
}

7.9.3 onwards

When there is no data stream.

Request:

POST _snapshot/found-snapshots/cloud-snapshot-.../_restore?wait_for_completion=true

Response:

{
  "snapshot": {
    "indices": [
      ".security-tokens-7",
      ".security-7",
      ".kibana-event-log-7.9.3-000001",
      ".slm-history-2-000001",
      "abc",
      ".apm-custom-link",
      "def",
      ".kibana_task_manager_1",
      ".apm-agent-configuration",
      "ilm-history-2-000001",
      "ghi",
      ".kibana_1"
    ],
    "snapshot": "cloud-snapshot-...",
    "shards": {
      "successful": 12,
      "failed": 0,
      "total": 12
    }
  }
}

With a data stream.

Request:

POST _snapshot/found-snapshots/cloud-snapshot-with-a-data-stream.../_restore?wait_for_completion=true

Response:

{
  "snapshot": {
    "indices": [
      ".ds-my-data-stream-000001"
    ],
    "snapshot": "cloud-snapshot-...",
    "shards": {
      "successful": 1,
      "failed": 0,
      "total": 1
    }
  }
}

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[elasticmachine]

Pinging @elastic/es-core-features (Team:Core/Features)