Add 'Generic' Entity Engine Definition

[romulets]

Summary

Introduce a new Entity Engine Definition called Generic. The larger context on why we are introducing a new entity definition is described on this private github issue.

The tldr; is that we would like to have an entity store with all the entities described by the entity ecs field. The decision to call generic entity definition comes from the fact that any entity can be described with the entity field - user, host, service, database, queue, subscription and so on. Therefore it makes sense to have the concept called generic entity, meanwhile the existent entity definitions will be called concrete entities, because they describe a very concrete type of entity (currently user, host, service).

Other changes included on this PR:

• Don't override entity.name with entity.id, only set if no value is found
• Migrate the usage of entity.type as the entity definition type to entity.EngineMetadata.Type
• Changes touching Entity Analytics code around getRiskEngineEntityTypes and getAssetCriticalityEntityTypes. There was a somewhat unnecessary and duplicated logic in these functions which essentially described the concrete entity definitions to be used by entity analytics flows. A new function called getEntityAnalyticsEntityTypes was introduced which unifies this logic and returns the entity types that Entity Analytics care about.

Video of a scroll through the entities processed by the generic entity store, source of the data is cloudbeat asset management integration.

no-sound.mov

How to test:

• In Advanced Settings (/app/management/kibana/settings), enable securitySolution:enableAssetInventory

• In Entity Store management (/security/entity_analytics_entity_store) enable entity store

• Verify Generic Engine Status

• Ingest documents with entity.id and entity.* fields. Personally I run cloudbeat asset discovery locally

• Verify ingested documents in .entities.v1.latest.security_generic_default

--

OBS: Also test enabling the store without the uiSetting enabled, so you can make sure that it doesn't enable

Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios
• If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
• This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The release_note:breaking label should be applied in these situations.
• Flaky Test Runner was used on any tests changed
• The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines

Identify risks

Does this PR introduce any risks? For example, consider risks like hard to test bugs, performance regression, potential of data loss.

Describe the risk, its severity, and mitigation for each identified risk. Invite stakeholders and evaluate how to proceed before merging.

• I see risk on performance, given the amount of aggregations the generated transform does
  • tested, although we see a higher spike in CPU than before, it's behind a feature flag and it's going to be used in controlled data sets (entity centric logs that contain entity.id field) we decided it's good enough to go.
• Enablement/disablement of entity store in a different uiSetting configuration.
  • Enable entity store with securitySolution:enableAssetInventory disabled. Then enable securitySolution:enableAssetInventory ==> No generic entity definition installed. You can manually install it in the EntityStore status page
  • Enable entity store with securitySolution:enableAssetInventory enabled. Then disable securitySolution:enableAssetInventory definition ==> hanging assets of generic entity store that can be deleted manually

[elasticmachine]

Pinging @elastic/kibana-cloud-security-posture (Team:Cloud Security)

[lgestc]

investigations changes lgtm!

[dplumlee]

Rule management changes lgtm

[machadoum]

Hey @romulets, great job! 👏 👏

I found something unexpected in the Entity Analytics dashboard. Are we intentionally displaying a panel for the Generic Entity Risk score?

Besides that, everything looks good. I also desk-tested the most common flows.

[romulets]

@machadoum it's not intentional! Awesome catch!!! I'll fix right away

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 2e3d4a8
• Cloud Deployment
• Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-211232-2e3d4a803fe1
• Security Deployment

Failed CI Steps

• Defend Workflows Cypress Tests on Serverless #10

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
securitySolution	7221	7219	-2

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
inventory	191.9KB	191.9KB	+71.0B
securitySolution	9.0MB	9.0MB	+495.0B
total			+566.0B

History

• 💛 Build #292598 was flaky b15f631
• 💔 Build #292536 failed e790c9a
• 💛 Build #292361 was flaky 688f784
• 💚 Build #292342 succeeded 810989e

cc @romulets