[elasticsearch] Extension of the Elasticsearch integration with datastream-centric stats

[3kt]

Proposed commit message

This adds a couple new assets to the elasticsearch integration (which natively aims to monitor Elasticsearch clusters):

• A transform job that pivots index stats (either provided by this same integration, or previously shipped with Metricbeat).
• An ingest pipeline that cleans up the documents created by the transform job.
• A dashboard consuming the resulting data to provide datastream and tier centric consumption information.

This aims to provide stack-native capabilities that are otherwise fulfilled through custom scripting (known as "consumption framework" within Elastic).

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• [ ]

How to test this PR locally

Related issues

Requires multiple dependencies:

• Index stats enhancement: creation date and tier_preference elasticsearch#116339 to be merged for index creation date and data tier to be exposed in the index stats API.
• The new fields to be correctly collected by Metricbeat or the Elasticsearch monitoring integration.
• The new fields to be correctly mapped in Elasticsearch.

Screenshots

[consulthys]

Fantastic job!! Here are a few initial comments.

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
94.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: bfa8c4925e5a276302790595a3f7caab7b43203e

History

• 💔 Build #20063 failed fbd122eda5e1bcf097c07bf0ee2272e5e15e3332
• 💔 Build #20059 failed 6f8db89434d9b1fa15541158a55626d5b2d97642
• 💔 Build #20051 failed 62fbe72df38e7705e82377c7c28db112c6146afb
• 💔 Build #20046 failed 3d261174a2172a468f1aff3808434f6330b70720
• 💔 Build #20015 failed 6f31e48a5c6ee086e05c715a7e5678aae4c587ad

[consulthys]

LGT Stack Monitoring

[elastic-vault-github-plugin-prod]

Package elasticsearch - 1.16.0 containing this change is available at https://epr.elastic.co/package/elasticsearch/1.16.0/

[IanLee1521]

Hi there! I've been using the tech preview dashboards for Elasticsearch for a bit now, and think this will be huge, I was literally making a transform for myself yesterday to try to create a latest index of stats that I could use for my indices (but ran into a bug and had to pause to open a support ticket).

A question as I just updated the Elasticsearch integration to add these new assets.. It looks like the new transform (logs-elasticsearch.index_pivot-default-0.1.0 ) did not start automatically. Is that expected? Is it normal to have to go and manually start up the transforms when integrations update and add this new functionality? -- Looking forward to getting this data available and useable for our monitoring!

[consulthys]

@IanLee1521 Thanks for your feedback!
Yes, it is on purpose as explained in the integration documentation

This transform isn’t started by default (Stack management > Transforms), but will perform the following once activated:

@3kt can probably fill you in on why it was decided to do it that way.

[3kt]

hi @IanLee1521, thanks for the feedback!

did not start automatically. Is that expected?

Yes - until we take the dashboards out of tech preview, I'd rather they have to be explicitly enabled by the users. I've tested performance and impact on large scale clusters, but I'm keen on getting wider field feedback.

The dashboard mentions that the transform has to be enabled manually, and gives a direct link to stack management for this purpose.

Also bear in mind that the full feature set will only be available once 8.17.1 hits, once this is live.

[IanLee1521]

@3kt - Happy to provide feedback! The short version for me at this point is that the target transform index (monitoring-indices) is not being populated. I thought it might just take a bit, but I gave it a few hours, and still nothing, despite no errors on the transform itself:

Happy to communicate in an appropriate venue (I'm in the Elastic Community Slack) to discuss what might be going on.

[IanLee1521]

The dashboard mentions that the transform has to be enabled manually, and gives a direct link to stack management for this purpose.

Just want to say that after I posted here I did find that message on the dashboard, I just hadn't seen it when first loading things.

[consulthys]

@IanLee1521

The short version for me at this point is that the target transform index (monitoring-indices) is not being populated. I thought it might just take a bit, but I gave it a few hours, and still nothing, despite no errors on the transform itself:

As mentioned by @3kt this is only going to work once Metricbeat 8.17.1 gets released.

[IanLee1521]

Thank you yes, 3kt pointed that out separately that I'd missed that line in their message. Looking forward to reporting back once that is out!

[amemkdm]

@3kt Just tried this and its looks great, the only thing is I am not able to see the elasticsearch.index.age field being populated in the index created by the transform. I also checked the transform configuration and couldn't find any transforms which would populate the field, so is this something which is still not added in the current version? Due to this the cosumption part of the dashboard is not getting filled. I have upgraded all the components to 8.17.1, we have a self managed on prem cluster.

[3kt]

hi @amemkdm, can you check if your documents have the elasticsearch.index.creation_date field set properly?
You will need to filter on event.dataset: elasticsearch.index or event.dataset: elasticsearch.stack_monitoring.index, depending on your collection mean.

The elasticsearch.index.age field is generated by a script in the ingest pipeline. Having a document to test for would allow me to check if there's any bug there :)

        ZonedDateTime currentDate = ZonedDateTime.parse(ctx['@timestamp']);
        ZonedDateTime creationDate = ZonedDateTime.parse(ctx.elasticsearch.index.creation_date);
        long ageInMillis = ChronoUnit.MILLIS.between(creationDate, currentDate);
        ctx.elasticsearch.index.age = (ageInMillis / (1000 * 60 * 60 * 24)).intValue();

Side note, you can hit me up on the community Slack if you have an account there - my name is the same as in my Github profile ;)

[amemkdm]

Done. I have reached out to you on Slack.