Snapshot and restore of system indices

[jaymode]

System indices are considered an implementation detail of stack components and a way should exist for these indices to be backed up. Snapshot and restore is our backup and recovery method for all indices and it is beyond the scope of system indices to consider any other method for backing up system data.

The thinking behind snapshot and restore of system indices is:

• System indices will indicate whether they should be backed up or not. (Their may be some that only contain data valid for a short period of time that do not need to be backed up like the .security-tokens index)
• System indices will be backed up when include_global_state is true
• Restoring a snapshot of a system index will replace the entire index
• Plugins provide a callback which will be called upon restoration of a system index it registered so that any in memory state can be brought in line with the data stored in the index

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Snapshot/Restore)

[dakrone]

System indices will be backed up when include_global_state is true

Will system indices be restored by default if include_global_state is true on the restoration?

[gwbrown]

@dakrone There's a few open questions and that's one of them - it's conceivable that a user might want to restore e.g. .security* but not .ml-state* (or vice-versa). However, that may be niche enough that we could get away with doing something like allowing negative index patterns to apply to system indices (e.g. "indices": "*,-.ml-state*" on the restore call)

[gwbrown]

@williamrandolph and I have been working on this. Here's an update with our current plan:

System indices will be included in both snapshot creation and restoration if include_global_state is true, and will not be included if it is false. In both cases, this can be overridden by specifying a new field on the create/restore snapshot request, feature_states (name is not necessarily final), an array of strings. Each SystemIndexPlugin will provide a name which can be used in this array, and if a feature/plugin's name is specified in a create/restore snapshot request, the System indices that plugin owns will be included in the snapshot or restored, as appropriate.

It's likely that there will be a way for a plugin to specify other index patterns that it owns to be included as well - for example, .ml-state* may be part of ML's "feature state", although it is not a System index. However, this part of the plan is still in early stages, and may change.

Finally, we will likely provide a way for plugins to specify that some of the System indices they own should not be part of snapshots. However, I'd like to confirm that this is necessary first - some candidates for this are .enrich* indices (as they are developed from other indices, and restoring these indices without restoring the entire cluster state may be problematic), and the .security_tokens* indices (I have seen it mentioned that this index tends to hold short-term data and may not be useful to snapshot, though my understanding may be incorrect). In both cases, there is a possibility that these will be handled in another way, and this opt-out may not be necessary.

Anything I haven't mentioned here is as described in the original issue - we're still planning to have callbacks for plugins to update in-memory state on restoration, etc.

[gwbrown]

Implemented in #63513.