[ML] Enable built-in Inference Endpoints and default for Semantic Text

[davidkyle]

Adds built-in inference endpoints for the ELSER (.elser-2-elasticsearch) and multilingual-e5-small models (.multilingual-e5-small-elasticsearch). These endpoints will always appear in the GET _inference API, the models are automatically downloaded and deployed on a call to POST _inference. The endpoint use adaptive allocations with scale to 0 enabled (min_number_of_allocations: 0), after 15 minutes of inactivity the model deployment will scale down to 0 allocations at which point they use 0 resources and the node they are running on may scale down. Model deployments scale up again when the models are used. The built-in endpoints start with a . prefix and are suffixed with the name of the service that hosts them, in this case elasticsearch.

The Semantic Text field mapping defaults the inference_id option to the built-in ELSER inference endpoint. Indexing a document with a semantic text field mapping with trigger the download and deployment of the model.

GET _inference/_all
{
  "endpoints": [
    {
      "inference_id": ".elser-2-elasticsearch",
      "task_type": "sparse_embedding",
      "service": "elasticsearch",
      "service_settings": {
        "num_threads": 1,
        "model_id": ".elser_model_2",
        "adaptive_allocations": {
          "enabled": true,
          "min_number_of_allocations": 0,
          "max_number_of_allocations": 32
        }
      },
      "chunking_settings": {
        "strategy": "word",
        "max_chunk_size": 250,
        "overlap": 100
      }
    },
    {
      "inference_id": ".multilingual-e5-small-elasticsearch",
      "task_type": "text_embedding",
      "service": "elasticsearch",
      "service_settings": {
        "num_threads": 1,
        "model_id": ".multilingual-e5-small",
        "adaptive_allocations": {
          "enabled": true,
          "min_number_of_allocations": 0,
          "max_number_of_allocations": 32
        }
      },
      "chunking_settings": {
        "strategy": "word",
        "max_chunk_size": 250,
        "overlap": 100
      }
    }
  ]

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[elasticsearchmachine]

Pinging @elastic/es-search-relevance (Team:Search Relevance)

[elasticsearchmachine]

Hi @davidkyle, I've created a changelog YAML for you.

[elasticsearchmachine]

💔 Backport failed

The backport operation could not be completed due to the following error:

An unexpected error occurred when attempting to backport this PR.

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 116931