[Telemetry] Data: Report dataset info only if there is known metadata

[afharo]

Summary

On top of #64935, this PR adds the collection of the constant_keywords dataset.name and dataset.type only if Elastic-known meta fields are found in the index's mappings.

We'll collect the information about the dataset fields mentioned above only if _meta.beat (reported as shipper) exists, or the index is maintained by the new Ingest Manager (it should contain _meta.managed_by === 'ingest-manager' and/or the _meta.package.name` field).

Checklist

Delete any items that are not applicable to this PR.

• Unit or functional tests were updated or added to match the most common scenarios

For maintainers

• This was checked for breaking API changes and was labeled appropriately

[elasticmachine]

Pinging @elastic/kibana-telemetry (Team:KibanaTelemetry)

[Bamieh]

lgtm

[TinaHeiligers]

@afharo it's FF today. Please merge if possible, otherwise we'll have to bump to 7.10.

[afharo]

Update: This PR is currently blocked until elastic/elasticsearch#59195 is done.

There is no point on checking the metadata to retrieve the dataset.* info if data-streams won't have _meta information just yet.

[afharo]

Replanning for 7.10 as tentative (but it still depends on the ETA of elastic/elasticsearch#59195)

[afharo]

@elasticmachine merge upstream

[afharo]

@TinaHeiligers and @Bamieh! After testing the new Ingest Manager, I noticed we had to add an exception for the way the data-streams actually work and seen by Kibana:

• Data Stream name: logs-nginx.access-default
• Actual index name: .ds-logs-nginx.access-default-000001

Funny enough: the data-stream works as an alias, so you can get the index .ds-logs-nginx.access-default-000001 with the request GET {DATA_STREAM_NAME} or, obviously, GET {INDEX_NAME}.

That's why in my last commit I had to change it to GET *-*-* (instead of GET *-*-*-*) and allow indices starting with .ds- when filtering out the system indices.

I'll wait for you to confirm if you are happy with these latest additions :)
Thank you!

[afharo]

@elasticmachine merge upstream

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 8d2a19c

Build metrics

✅ unchanged

History

• 💚 Build #63639 succeeded 2e66c5e
• 💚 Build #63093 succeeded 8d0b450
• 💚 Build #60810 succeeded 313c747

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)