[Security Solution] Data Quality dashboard storage metrics

[andrew-goldstein]

[Security Solution] Data Quality dashboard storage metrics

Above: The new storage metrics treemap updates as indices are checked

Above: Storage metrics in the Data Quality dashboard

Summary

This PR introduces storage metrics to the Data Quality dashboard

• Multiple views are enhanced to display the size of indices

• A new interactive treemap visualizes the relative sizes of indices

• Markdown reports include the size of indices

• The Data Quality dashboard Beta tag is removed

• Inline action buttons replace the Take action popover

• The Global stats panel remains visible when the Select one or more ILM phases help is displayed

• Code coverage is improved throughout the dashboard

Details

Multiple views enhanced to display the size of indices

The following views have been enhanced to display the Size of indices, per the screenshots below:

• The pattern table's Size column displays the size of a single index

• The pattern table's Size tooltip

• The pattern rollup's Size stat displays the total size of indices in a pattern

• The pattern rollup's Size stat tooltip

• The global stats rollup Size stat displays the total size of all the patterns

• The global stats rollup Size stat tooltip

New interactive treemap

A new interactive treemap visualizes the relative sizes of indices:

• The color of indices in the treemap and its legend update as the data is checked

• Clicking on an index in the treemap or the legend expands (and scrolls to) the index

Markdown reports include the Size of indices

Markdown reports are enhanced to include the new Size statistic in:

• Pattern markdown tables

Result	Index	Docs	Incompatible fields	ILM Phase	Size
❌	auditbeat-7.14.2-2023.04.09-000001	48,077 (4.3%)	12	hot	41.3MB
❌	auditbeat-7.3.2-2023.04.09-000001	48,068 (4.3%)	8	hot	31MB
❌	auditbeat-7.11.2-2023.04.09-000001	48,064 (4.3%)	12	hot	40.8MB

• Pattern rollup markdown tables

Incompatible fields	Indices checked	Indices	Size	Docs
164	26	26	899.3MB	1,118,155

• The global stats markdown table

Incompatible fields	Indices checked	Indices	Size	Docs
166	32	32	9.2GB	20,779,245

Data Quality dashboard Beta tag removed

The Data Quality dashboard Beta tag is removed from the following views:

• The Dashboards page

Before:

After:

• Security Solution side navigation

Before:

After:

• The Data Quality dashboard page header

Before:

After:

Inline action buttons replace the Take action popover

Inline Add to new case and Copy to clipboard action buttons replace the Take action popover, the previous home of these actions:

Before:

After:

Global stats panel remains visible when the Select one or more ILM phases help is displayed

The Global stats panel now remains visible when the Select one or more ILM phases help is displayed:

Before:

After:

Code coverage improvements

Code coverage is improved throughout the dashboard, as measured by running the following command:

node scripts/jest --watch x-pack/packages/kbn-ecs-data-quality-dashboard --coverage

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[andrew-goldstein]

Files by Code Owner

elastic/security-threat-hunting-explore

• x-pack/plugins/security_solution/public/overview/links.ts
• x-pack/plugins/security_solution/public/overview/pages/data_quality.tsx

[semd]

Explore changes LGTM!

[michaelolo24]

these inline comments for what's changed are helpful 👌🏾

[michaelolo24]

it's the same thing, but you could do patterns[i + 1] ? (<Spacer>) : null or maybe just patterns[i + 1] && <Spacer>

[michaelolo24]

Nit and def not necessary, but with discussions around alerts index name changes, may be better to use the ${DEFAULT_ALERTS_INDEX}-default here.

[andrew-goldstein]

i'm reluctant to use ${DEFAULT_ALERTS_INDEX}-default in the context of these tests (and others) because artifacts of the current value, .alerts-security.alerts-default, are ubiquitous in the mock (stats and ILM) data, and in most of the tests.

In most contexts, there's nothing magic about .alerts-security.alerts-default other than it's currently a "foreign key" in the mock data. If the value behind the ${DEFAULT_ALERTS_INDEX}-default changes, tests may start failing (with false positives) unless the mock data is manually updated via search & replace.

[michaelolo24]

Yea, took me a sec to realize all of these are just in test files, so doesn't really matter what it is as long as it aligns with the mocks. Ignore me...I've been seeing the index pattern everywhere so I'm very sensitive to it 😂

[michaelolo24]

You can use orderBy to sort in descending order without the additional reverse. Not sure which is more performant though

[michaelolo24]

nice catch

[michaelolo24]

you can use orderBy in this function as well

[michaelolo24]

SUPER random thought, but you can have a space-builder so it's easier to get an idea of the numerical spacing.

Something like ${addSpace(8)} and const addSpace = (spaces: number) => Array(spaces).fill('-').join('');. DEFINITELY over-engineered, but couldn't help myself

[michaelolo24]

I'll stop commenting on these after this.. re the orderBy vs sort().reverse(). And once again, not sure which would be more performant

[michaelolo24]

Do you need to pass the reason=""? or keep it the same as line 143?

[michaelolo24]

nit: you should be able to just pass grow

[michaelolo24]

nit: missed the 'n' in unallowed for the name of this file :) and file path :)

[michaelolo24]

Is there ever a scenario where we would want to show partial results if one of the indices didn't return a value? Not necessary now, but just a thought. We'll see how often users run into this scenario...

[michaelolo24]

The test coverage here is excellent. Thank you for adding them. Desk tested locally across multiple spaces with all ilm phases selected and de-selected and it works great! The tooltips all have the text as expected and there are no signs of the beta tag as far as I can tell. Nice work! 🚀 Just added some minor nits and questions.

[kibana-ci]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 177eed0

Failed CI Steps

• FTR Configs #32

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
securitySolution	3847	3853	+6

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
securitySolution	9.1MB	9.1MB	+11.6KB

Unknown metric groups

ESLint disabled line counts

id	before	after	diff
@kbn/ecs-data-quality-dashboard	2	6	+4
enterpriseSearch	17	19	+2
securitySolution	397	400	+3
total			+9

Total ESLint disabled count

id	before	after	diff
@kbn/ecs-data-quality-dashboard	2	6	+4
enterpriseSearch	18	20	+2
securitySolution	477	480	+3
total			+9

History

• 💔 Build #122504 failed 01b06fc3587c4b387926c51d1d8fc18d96f3d2ea
• 💚 Build #122328 succeeded 2413edd131d06f507fe78f5a055c8ca7d243a6d0
• 💔 Build #122264 failed ba5f1d24970483414342ecbd47cd29cd93b2afce

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @andrew-goldstein

[dhru42]

@andrew-goldstein could we use the following screenshot in the 8.8 release blog?

[andrew-goldstein]

@andrew-goldstein could we use the following screenshot in the 8.8 release blog?

yes @dhru42, thanks!