[Inference API] Add mapping and model changes for exposing endpoint heuristics

[jonathan-buttner]

This PR is the first stage to implement exposing the endpoint heuristics. To facility this the following changes were made:

• Modified the index schema to support the following document format:

Example document

{
  "_index": ".inference",
  "_id": "model_.elser-2-elastic",
  "_source": {
    "model_id": ".elser-2-elastic",
    "task_type": "sparse_embedding",
    "service": "elastic",
    "service_settings": {
      "model_id": "elser_model_2"
    },
    "task_settings": {},
    "chunking_settings": {
      "strategy": "sentence",
      "max_chunk_size": 250,
      "sentence_overlap": 1
    },
    "metadata": {
      "heuristics": {
        "properties": [
          "multilingual",
          "english"
        ],
        "status": "active",
        "release_date": "2024-01-15",
        "end_of_life_date": null
      },
      "display": {
        "name": "abc"
      },
      "internal": {
        "fingerprint": "1234567890",
        "version": 1
      }
    }
  }
}

• Modified the ModelConfigurations to accept the EndpointMetadata and store it
• It filters out the internal object when using the filtered method
• Modified InferenceService so the parsePersistedConfig* methods can also take an UnparsedModel
  • Eventually it'd be good to move all services to this implementation because it gives us flexibility with adding new fields
  • I needed this because the EndpointMetadata is parsed within the ModelRegistry at the same place as the task type and service name
• Modified the ElasticInferenceService models to accept EndpointMetadata within their constructors
  • This is necessary so we can store it within the ModelConfigurations
• To avoid the new metadata field being written and potentially causing unanticipated validation/mapping issues, the authorization polling logic will skip sending the authorization GET request until all nodes in the cluster can handle the new metadata field

The endpoint metadata isn't used at the moment. The cluster state and inference index will contain the metadata (for all endpoints). For EIS endpoints that already existed in the index, the metadata field will default to the EMPTY implementation. For new EIS endpoints, the metadata will be populated with the fields that EIS currently returns (properties, status, release_date, end_of_life_date). The other fields like fingerprint and kibana_connector_name will remain null. The version field will be set to 0.

This PR doesn't implement the logic to update the old endpoints. That logic will be added in a follow up PR.

User created endpoints will have metadata set to the EMPTY state.

The empty metadata state is not serialized to avoid a mixed cluster mapping failure where older nodes' mapping hasn't been updated yet.

Testing

Our rolling upgrade tests initially caught the mapping issue. Those will fail if we persist the new field during an upgrade.

[elasticsearchmachine]

Hi @jonathan-buttner, I've created a changelog YAML for you.

[dimitris-athanasiou]

LGTM