Changing the ILM policy for a managed index is non-deterministic

[andreidan]

Currently, assigning a managed index to a different policy yields a non-deterministic behaviour due to the caching of the current phase an index is going through.

eg. An index managed by an ILM policy (policy A) is added to a new ILM policy (policy B) while still in the hot phase of policy A by using the update index settings API. Due to the cached phase definition in the index metadata, even if that phase is from a previous ILM policy (policy A), it will still need to be completed. Therefore the index will remain in the hot phase pending to roll over, having to wait for the rollover of the first policy to happen before it can go into a new phase (the new phase in this case will be the first phase of policy B)

This would not be the case if the ILM remove policy API would be used to remove the first policy, before assigning a new one, however, this api might be something our users don't know of (as opposed to the update index settings API)

I'm proposing to at least document the non-deterministic behaviour and/or react to the index.lifecycle.name configuration change and mimic the ILM remove policy API behaviour.

[elasticmachine]

Pinging @elastic/es-core-features (Team:Core/Features)

[elasticmachine]

Pinging @elastic/es-docs (Team:Docs)

[andreidan]

Now that #75296 is fixed the behaviour for changing the index.lifecycle.name setting for an index to a different policy is deterministic and behaves as follows: changing the index.lifecycle.name will have no effect on the currently executing phase (which we cache in the index metadata), but next phase will be read from the new policy.

eg. Let's say we have index1 managed by current_policy and is now in the warm phase. We now change the index.lifecycle.name for index1 to point to new_policy which has hot,warm, and delete phases.
index1 will finish executing the warm phase as defined by current_policy and then move into the delete phase defined in new_policy (subject to the min_age parameter defined in the delete phase of the new_policy)