[Infrastructure UI] Filter out null bucket items from average calculation

[crespocarlos]

Summary

Closes #152328

This PR fixes the average calculation in the Snapshot API, filtering out buckets with null values from it, which are more likely to appear with queries that use small data ranges.

The results after this change are equal to what Elasticsearch would calculate in the avg aggregation

How to test

• Make sure you have metrics data (either through enabling the system module in metricbeat or connecting your local kibana to an oblt-cli cluster)
• Navigate to Infrastructure > Hosts
• Filter the results to see a single host
• Change the data range filter and compare the KPIs against the table.
• Validate other pages that use the Snapshot API (Inventory UI and Metrics UI to see if the results there are still correct

[crespocarlos]

This is what the query returns:

[{ timestamp: 1562786355035, metric_0: null },
{ timestamp: 1562786415035, metric_0: null },
{ timestamp: 1562786475035, metric_0: null },
{ timestamp: 1562786535035, metric_0: null },
{ timestamp: 1562786595035, metric_0: null },
{ timestamp: 1562786655035, metric_0: 0.47105555555555556 },
{ timestamp: 1562786715035, metric_0: null }]

So, if we're filtering out null values from the average calculation, it makes sense for the expected result to be 0.47105555555555556

[elasticmachine]

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)

[tonyghiani]

doubt: not sure if this scenario is possible, but in case the metric value is 0, this will be filtered out and wrongly compute as the expected value.
For example

[0, 0, 9] // expected (0 + 0 + 9) / 3 === 3 ✅

but with the current .filter(Boolean) it would result in

[0, 0, 9].filter(Boolean) // expected 9 / 1 === 9 ❌

It may be safer to only filter out only non-numeric values

[0, 0, 9].filter(isFinite) 

[crespocarlos]

This is absolutely correct. Thanks for catching it.

[null,0, 0, 9].filter(Number.isFinite) -> [0,0,9]

[tonyghiani]

I'm not sure why, but it looks like the values between the KPI tiles and the table are still mismatching on different time-range selections, here are some examples:

Time range	Screenshot	Avg RX matches?	Avg TX matches?
Last 5 mins		✅	❌
Last 10 mins		✅	❌
Last 15 mins		❌	❌
Last 20 mins		❌	❌
Last 30 mins		✅	❌

[crespocarlos]

I'm not sure why, but it looks like the values between the KPI tiles and the table are still mismatching on different time-range selections, here are some examples:

Yeah. My tests were clearly biased and didn't catch these big differences. Thanks for thoroughly testing this.

Pushing another change to mitigate it:

I believe this fix won't solve all scenarios, though. e.g: querying for the last 15m in a real environment might possibly return different values because both KPI and table queries run at different times.

Absolute time ranges should always return the same values, in theory. But they run different queries (one with a date histogram and another without it), so it's possible that small differences can be seen.

[crespocarlos]

@elasticmachine merge upstream

[tonyghiani]

LGTM, thanks for applying the changes :)

[kibana-ci]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 0cd0b59

Failed CI Steps

• FTR Configs #33

Test Failures

• [job] [logs] FTR Configs #33 / saved objects tagging - functional tests visualize integration editing allows to assign tags to an existing visualization

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
infra	1.3MB	1.3MB	+170.0B

Unknown metric groups

ESLint disabled line counts

id	before	after	diff
securitySolution	428	430	+2

Total ESLint disabled count

id	before	after	diff
securitySolution	506	508	+2

History

• 💔 Build #111337 failed b242a5d
• 💚 Build #111065 succeeded 64090d4
• 💔 Build #111022 failed 61a3dd9

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[kibanamachine]

💚 All backports created successfully

Status	Branch	Result
✅	8.7

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation