Support custom metadata on Data Streams

[cjcenizal]

The intention behind this feature is to enable the UI to identify the data streams created by Ingest Manager. We'll need to implement a solution that allows index templates to define the metadata that is applied to the data streams that they create.

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/Data streams)

[afharo]

We need this as well for the changes in elastic/kibana#71419.
Ideally, the _meta info in the Data Streams should be copied over to the underlying indices it creates.

[dakrone]

Ideally, the _meta info in the Data Streams should be copied over to the underlying indices it creates.

We first need to support _meta on indices themselves, which is probably a separate issue

[afharo]

@dakrone what do you mean with support _meta on indices themselves?

I thought it is already supported. Kibana uses that feature in the .kibana index and AFAIK, Beats does as well. I'm talking about mappings._meta though. Maybe that's my confusion?

[dakrone]

Mappings metadata is more thought of as metadata about the mappings themselves, not the index (at least that's they way I think about it). It would work for now, but I don't think we should use it as a general purpose metadata storage for the index.

[afharo]

@dakrone I agree! Thank you for explaining!

[yuliacech]

@dakrone Hi Lee, I was wondering if there is any work planned on adding metadata to data streams? I see now that we have

"_meta": {
    "package": {
      "name": "system"
    },
    "managed_by": "ingest-manager",
    "managed": true
  }

in both index templates mappings and index mappings if they were created by ingest manager.

[dakrone]

@yuliacech it's still something we'd like to do, but I don't think it has anyone currently working on adding it.

[martijnvg]

I think this is something that we can pickup in the near future.

When a data stream is created then custom data stream metadata should be sourced from the matching composable index template and be added to the data stream under the _meta key. This custom metadata can then be retrieved via the get data stream api.

I do have a question about from where the data stream custom metadata is sourced. Should all the keys under the current _meta json object on a composable index template be added to a data stream's custom metadata? Or only a subset of keys? Or should the data_stream object on a composable template have its own _meta object?

Also is there a need to update a data stream's custom metadata after its creation? Currently there isn't a data stream update api or something similar.