Make xpack.ilm.enabled setting a no-op

[williamrandolph]

In #54816, we deprecated a group of toggle settings for basic license level features. In #55416 (in review), we are turning most of those settings into no-ops, i.e., having them in your settings file won't throw an error, but they won't do anything at all. I left monitoring and ILM for separate PRs because turning them into no-ops affects logic in other modules.

This PR turns xpack.ilm.enabled into a no-op. It removes logic and tests in xpack core, ML, and Watcher. The latter two modules in particular had to switch behavior based on the enablement or disablement of ILM, which is the kind of thing we're trying to eliminate with this effort; ILM should be a basic "building block" for higher level modules.

Meta issue: #54745

• documentation

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Plugins)

[williamrandolph]

@elasticmachine test this please

[williamrandolph]

@elasticmachine please run elasticsearch-ci/2

[williamrandolph]

@elasticmachine please run elasticsearch-ci/2

[williamrandolph]

I think I ought to add a new Watcher-specific setting that replaces how the old xpack.ilm.enabled setting was used in the watcher module. This would allow users who are managing Watcher indices with Curator to keep doing what they're already doing.

• Introduce Watcher setting to not use ILM.

[pugnascotia]

Can we remove ANOMALY_DETECTION_STATE_TEMPLATE_NO_ILM and STATS_TEMPLATE_NO_ILM now?

[williamrandolph]

I will reach out to someone from ML to see if this change is OK, but I would guess that we will be able to remove those templates.

• get feedback from someone on ML for removing NO_ILM templates

[williamrandolph]

@dimitris-athanasiou and @przemekwitek have confirmed that these templates can be removed.

[przemekwitek]

(Przemek from ML team): I think it is ok to remove the non-ILM versions.
Once only ILM versions exist, it would be good to inline the variables into the .json files but if you don't feel like doing it, I can do it as a followup PR.

[williamrandolph]

@przemekwitek Do you mean like this?

    private static IndexTemplateConfig stateTemplate() {
        Map<String, String> variables = new HashMap<>();
        variables.put(VERSION_ID_PATTERN, String.valueOf(Version.CURRENT.id));
        variables.put("xpack.ml.index.lifecycle.name", ML_SIZE_BASED_ILM_POLICY_NAME);
        variables.put("xpack.ml.index.rollover_alias.name", AnomalyDetectorsIndex.jobStateIndexWriteAlias());

        return new IndexTemplateConfig(AnomalyDetectorsIndexFields.STATE_INDEX_PREFIX,
            ANOMALY_DETECTION_PATH + "state_index_template.json",
            Version.CURRENT.id, VERSION_PATTERN,
            variables);
    }

…with this in the template json file?

  "settings" : {
    "index" : {
      "auto_expand_replicas" : "0-1",
      "hidden": true
    },
    "index.lifecycle.name": "${xpack.ml.index.lifecycle.name}",
    "index.lifecycle.rollover_alias": "${xpack.ml.index.rollover_alias.name}"
  },

Or should the actual values be inlined as well?

[przemekwitek]

Any of these 2 solutions work for me.
It's most important for me to get rid of syntax tokens (like comma ",") which were just a hack to allow enabled/disabled duality.
The first one is more flexible so my preference goes there.

[williamrandolph]

I've pushed a commit with the first, more flexible strategy. Thanks for the feedback!

[przemekwitek]

Could you make the variable name the same for both templates?
I.e. xpack.ml.index.lifecycle.name and xpack.ml.index.lifecycle.rollover_alias for both state and stats templates?

[williamrandolph]

Good catch. This is done.

[przemekwitek]

Thank you!

[przemekwitek]

LGTM

[przemekwitek]

Thank you!

[williamrandolph]

@elasticmachine please run elasticsearch-ci/bwc

[rjernst]

Thanks for adding the fallback to the old setting, so that users do not suddenly break if they were using curator externally for watcher management. LGTM.

[williamrandolph]

Rats. Github lost the commit message I wrote and used the generated commit summary message instead. Here's what I wanted to use:

This commit makes the 'xpack.ilm.enabled' setting into a no-op for all
purposes except for Watcher index management. ILM is an important building
block for higher level services, so, along with other basic license features,
we want it to always be available in the default distribution.

Since users might use a tool other than ILM to manage Watcher indices, we
introduce a new 'xpack.watcher.use_ilm_index_management' setting. If this
setting is not set explicitly, it will fall back to the value of
'xpack.ilm.enabled', meaning that this will not be a breaking change for our
users.

This commit *does* make ILM mandatory for ML by removing the "NO_ILM"
templates for ML indices.