[7.x] [ML] adds new trained model alias API to simplify trained model updates and deployments (#68922) (#69208)

* [ML] adds new trained model alias API to simplify trained model updates and deployments (#68922)

A `model_alias` allows trained models to be referred by a user defined moniker.

This not only improves the readability and simplicity of numerous API calls, but it allows for simpler deployment and upgrade procedures for trained models.

Previously, if you referenced a model ID directly within an ingest pipeline, when you have a new model that performs better than an earlier referenced model, you have to update the pipeline itself. If this model was used in numerous pipelines, ALL those pipelines would have to be updated.

When using a `model_alias` in an ingest pipeline, only that `model_alias` needs to be updated. Then, the underlying referenced model will change in place for all ingest pipelines automatically.

An additional benefit is that the model referenced is not changed until it is fully loaded into cache, this way throughput is not hampered by changing models.

Author: benwtrent

docs/reference/ml/df-analytics/apis/get-trained-models-stats.asciidoc

@@ -47,7 +47,7 @@ request by using a comma-separated list of model IDs or a wildcard expression.
 
 `<model_id>`::
 (Optional, string)
-include::{es-repo-dir}/ml/ml-shared.asciidoc[tag=model-id]
+include::{es-repo-dir}/ml/ml-shared.asciidoc[tag=model-id-or-alias]
 
 
 [[ml-get-trained-models-stats-query-params]]

docs/reference/ml/df-analytics/apis/get-trained-models.asciidoc

@@ -50,7 +50,7 @@ using a comma-separated list of model IDs or a wildcard expression.
 
 `<model_id>`::
 (Optional, string)
-include::{es-repo-dir}/ml/ml-shared.asciidoc[tag=model-id]
+include::{es-repo-dir}/ml/ml-shared.asciidoc[tag=model-id-or-alias]
 
 
 [[ml-get-trained-models-query-params]]

docs/reference/ml/df-analytics/apis/index.asciidoc

@@ -2,6 +2,7 @@ include::ml-df-analytics-apis.asciidoc[leveloffset=+1]
 //CREATE
 include::put-dfanalytics.asciidoc[leveloffset=+2]
 include::put-trained-models.asciidoc[leveloffset=+2]
+include::put-trained-models-aliases.asciidoc[leveloffset=+2]
 //UPDATE
 include::update-dfanalytics.asciidoc[leveloffset=+2]
 //DELETE

docs/reference/ml/df-analytics/apis/ml-df-analytics-apis.asciidoc

@@ -22,8 +22,9 @@ You can use the following APIs to perform {infer} operations.
 * <<get-trained-models>>
 * <<get-trained-models-stats>>
 * <<delete-trained-models>>
+* <<put-trained-models-aliases>>
 
-You can deploy a trained model to make predictions in an ingest pipeline or in 
+You can deploy a trained model to make predictions in an ingest pipeline or in
 an aggregation. Refer to the following documentation to learn more.
 
 * <<inference-processor,{infer-cap} processor>>

docs/reference/ml/df-analytics/apis/put-trained-models-aliases.asciidoc

@@ -0,0 +1,89 @@
+[role="xpack"]
+[testenv="platinum"]
+[[put-trained-models-aliases]]
+= Put Trained Models Aliases API
+[subs="attributes"]
+++++
+<titleabbrev>Put Trained Models Aliases</titleabbrev>
+++++
+
+Creates a trained models alias. These model aliases can be used instead of the trained model ID
+when referencing the model in the stack. Model aliases must be unique, and a trained model can have
+more than one model alias referring to it. But a model alias can only refer to a single trained model.
+
+beta::[]
+
+[[ml-put-trained-models-aliases-request]]
+== {api-request-title}
+
+`PUT _ml/trained_models/<model_id>/model_aliases/<model_alias>`
+
+
+[[ml-put-trained-models-aliases-prereq]]
+== {api-prereq-title}
+
+If the {es} {security-features} are enabled, you must have the following
+built-in roles and privileges:
+
+* `machine_learning_admin`
+
+For more information, see <<built-in-roles>>, <<security-privileges>>, and
+{ml-docs-setup-privileges}.
+
+[[ml-put-trained-models-aliases-desc]]
+== {api-description-title}
+
+This API creates a new model alias to refer to trained models, or updates an existing
+trained model's alias.
+
+When updating an existing model alias to a new model ID, this API will return a error if the models
+are of different inference types. Example, if attempting to put the model alias
+`flights-delay-prediction` from a regression model to a classification model, the API will error.
+
+The API will return a warning if there are very few input fields in common between the old
+and new models for the model alias.
+
+[[ml-put-trained-models-aliases-path-params]]
+== {api-path-parms-title}
+
+`model_id`::
+(Required, string)
+The trained model ID to which the model alias should refer.
+
+`model_alias`::
+(Required, string)
+The model alias to create or update. The model_alias cannot end in numbers.
+
+[[ml-put-trained-models-aliases-query-params]]
+== {api-query-parms-title}
+
+`reassign`::
+(Optional, boolean)
+Should the `model_alias` get reassigned to the provided `model_id` if it is already
+assigned to a model. Defaults to false. The API will return an error if the `model_alias`
+is already assigned to a model but this parameter is `false`.
+
+[[ml-put-trained-models-aliases-example]]
+== {api-examples-title}
+
+[[ml-put-trained-models-aliases-example-new-alias]]
+=== Creating a new model alias
+
+The following example shows how to create a new model alias for a trained model ID.
+
+[source,console]
+--------------------------------------------------
+PUT _ml/trained_models/flight-delay-prediction-1574775339910/model_aliases/flight_delay_model
+--------------------------------------------------
+// TEST[skip:setup kibana sample data]
+
+[[ml-put-trained-models-aliases-example-put-alias]]
+=== Updating an existing model alias
+
+The following example shows how to reassign an existing model alias for a trained model ID.
+
+[source,console]
+--------------------------------------------------
+PUT _ml/trained_models/flight-delay-prediction-1580004349800/model_aliases/flight_delay_model?reassign=true
+--------------------------------------------------
+// TEST[skip:setup kibana sample data]

docs/reference/ml/ml-shared.asciidoc

@@ -1149,6 +1149,10 @@ tag::model-id[]
 The unique identifier of the trained model.
 end::model-id[]
 
+tag::model-id-or-alias[]
+The unique identifier of the trained model or a model alias.
+end::model-id-or-alias[]
+
 tag::model-memory-limit[]
 The approximate maximum amount of memory resources that are required for
 analytical processing. Once this limit is approached, data pruning becomes

server/src/main/java/org/elasticsearch/common/util/set/Sets.java

@@ -60,6 +60,12 @@ public static <T> boolean haveEmptyIntersection(Set<T> left, Set<T> right) {
         return left.stream().noneMatch(right::contains);
     }
 
+    public static <T> boolean haveNonEmptyIntersection(Set<T> left, Set<T> right) {
+        Objects.requireNonNull(left);
+        Objects.requireNonNull(right);
+        return left.stream().anyMatch(right::contains);
+    }
+
     /**
      * The relative complement, or difference, of the specified left and right set. Namely, the resulting set contains all the elements that
      * are in the left set but not in the right set. Neither input is mutated by this operation, an entirely new set is returned.

x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/XPackClientPlugin.java

@@ -108,6 +108,7 @@
 import org.elasticsearch.xpack.core.ml.action.EvaluateDataFrameAction;
 import org.elasticsearch.xpack.core.ml.action.ExplainDataFrameAnalyticsAction;
 import org.elasticsearch.xpack.core.ml.action.FinalizeJobExecutionAction;
+import org.elasticsearch.xpack.core.ml.inference.ModelAliasMetadata;
 import org.elasticsearch.xpack.core.rollup.action.RollupIndexerAction;
 import org.elasticsearch.xpack.core.ml.action.FlushJobAction;
 import org.elasticsearch.xpack.core.ml.action.ForecastJobAction;
@@ -534,6 +535,8 @@ public List<NamedWriteableRegistry.Entry> getNamedWriteables() {
                 // logstash
                 new NamedWriteableRegistry.Entry(XPackFeatureSet.Usage.class, XPackField.LOGSTASH, LogstashFeatureSetUsage::new),
                 // ML - Custom metadata
+                new NamedWriteableRegistry.Entry(Metadata.Custom.class, ModelAliasMetadata.NAME, ModelAliasMetadata::new),
+                new NamedWriteableRegistry.Entry(NamedDiff.class, ModelAliasMetadata.NAME, ModelAliasMetadata::readDiffFrom),
                 new NamedWriteableRegistry.Entry(Metadata.Custom.class, "ml", MlMetadata::new),
                 new NamedWriteableRegistry.Entry(NamedDiff.class, "ml", MlMetadata.MlMetadataDiff::new),
                 // ML - Persistent action requests
@@ -712,6 +715,11 @@ public List<NamedXContentRegistry.Entry> getNamedXContent() {
                 // ML - Custom metadata
                 new NamedXContentRegistry.Entry(Metadata.Custom.class, new ParseField("ml"),
                         parser -> MlMetadata.LENIENT_PARSER.parse(parser, null).build()),
+                new NamedXContentRegistry.Entry(
+                    Metadata.Custom.class,
+                    new ParseField(ModelAliasMetadata.NAME),
+                    ModelAliasMetadata::fromXContent
+                ),
                 // ML - Persistent action requests
                 new NamedXContentRegistry.Entry(PersistentTaskParams.class, new ParseField(MlTasks.DATAFEED_TASK_NAME),
                         StartDatafeedAction.DatafeedParams::fromXContent),

x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/ml/action/GetTrainedModelsStatsAction.java

@@ -191,7 +191,7 @@ protected Reader<Response.TrainedModelStats> getReader() {
         public static class Builder {
 
             private long totalModelCount;
-            private Set<String> expandedIds;
+            private Map<String, Set<String>> expandedIdsWithAliases;
             private Map<String, IngestStats> ingestStatsMap;
             private Map<String, InferenceStats> inferenceStatsMap;
 
@@ -200,13 +200,13 @@ public Builder setTotalModelCount(long totalModelCount) {
                 return this;
             }
 
-            public Builder setExpandedIds(Set<String> expandedIds) {
-                this.expandedIds = expandedIds;
+            public Builder setExpandedIdsWithAliases(Map<String, Set<String>> expandedIdsWithAliases) {
+                this.expandedIdsWithAliases = expandedIdsWithAliases;
                 return this;
             }
 
-            public Set<String> getExpandedIds() {
-                return this.expandedIds;
+            public Map<String, Set<String>> getExpandedIdsWithAliases() {
+                return this.expandedIdsWithAliases;
             }
 
             public Builder setIngestStatsByModelId(Map<String, IngestStats> ingestStatsByModelId) {
@@ -220,8 +220,8 @@ public Builder setInferenceStatsByModelId(Map<String, InferenceStats> infereceSt
             }
 
             public Response build() {
-                List<TrainedModelStats> trainedModelStats = new ArrayList<>(expandedIds.size());
-                expandedIds.forEach(id -> {
+                List<TrainedModelStats> trainedModelStats = new ArrayList<>(expandedIdsWithAliases.size());
+                expandedIdsWithAliases.keySet().forEach(id -> {
                     IngestStats ingestStats = ingestStatsMap.get(id);
                     InferenceStats inferenceStats = inferenceStatsMap.get(id);
                     trainedModelStats.add(new TrainedModelStats(

x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/ml/action/InternalInferModelAction.java

@@ -160,18 +160,25 @@ public String toString() {
     public static class Response extends ActionResponse {
 
         private final List<InferenceResults> inferenceResults;
+        private final String modelId;
         private final boolean isLicensed;
 
-        public Response(List<InferenceResults> inferenceResults, boolean isLicensed) {
+        public Response(List<InferenceResults> inferenceResults, String modelId, boolean isLicensed) {
             super();
             this.inferenceResults = Collections.unmodifiableList(ExceptionsHelper.requireNonNull(inferenceResults, "inferenceResults"));
             this.isLicensed = isLicensed;
+            this.modelId = modelId;
         }
 
         public Response(StreamInput in) throws IOException {
             super(in);
             this.inferenceResults = Collections.unmodifiableList(in.readNamedWriteableList(InferenceResults.class));
             this.isLicensed = in.readBoolean();
+            if (in.getVersion().onOrAfter(Version.V_7_13_0)) {
+                this.modelId = in.readOptionalString();
+            } else {
+                this.modelId = null;
+            }
         }
 
         public List<InferenceResults> getInferenceResults() {
@@ -182,23 +189,32 @@ public boolean isLicensed() {
             return isLicensed;
         }
 
+        public String getModelId() {
+            return modelId;
+        }
+
         @Override
         public void writeTo(StreamOutput out) throws IOException {
             out.writeNamedWriteableList(inferenceResults);
             out.writeBoolean(isLicensed);
+            if (out.getVersion().onOrAfter(Version.V_7_13_0)) {
+                out.writeOptionalString(modelId);
+            }
         }
 
         @Override
         public boolean equals(Object o) {
             if (this == o) return true;
             if (o == null || getClass() != o.getClass()) return false;
             InternalInferModelAction.Response that = (InternalInferModelAction.Response) o;
-            return isLicensed == that.isLicensed && Objects.equals(inferenceResults, that.inferenceResults);
+            return isLicensed == that.isLicensed
+                && Objects.equals(inferenceResults, that.inferenceResults)
+                && Objects.equals(modelId, that.modelId);
         }
 
         @Override
         public int hashCode() {
-            return Objects.hash(inferenceResults, isLicensed);
+            return Objects.hash(inferenceResults, isLicensed, modelId);
         }
 
         public static Builder builder() {
@@ -207,6 +223,7 @@ public static Builder builder() {
 
         public static class Builder {
             private List<InferenceResults> inferenceResults;
+            private String modelId;
             private boolean isLicensed;
 
             public Builder setInferenceResults(List<InferenceResults> inferenceResults) {
@@ -219,8 +236,13 @@ public Builder setLicensed(boolean licensed) {
                 return this;
             }
 
+            public Builder setModelId(String modelId) {
+                this.modelId = modelId;
+                return this;
+            }
+
             public Response build() {
-                return new Response(inferenceResults, isLicensed);
+                return new Response(inferenceResults, modelId, isLicensed);
             }
         }