Skip to content

[Entity Analytics][9.0] Remove all legacy risk engine code and features#201810

Merged
hop-dev merged 53 commits intoelastic:mainfrom
hop-dev:delete-legacy-risk-engine
Jan 10, 2025
Merged

[Entity Analytics][9.0] Remove all legacy risk engine code and features#201810
hop-dev merged 53 commits intoelastic:mainfrom
hop-dev:delete-legacy-risk-engine

Conversation

@hop-dev
Copy link
Contributor

@hop-dev hop-dev commented Nov 26, 2024

Summary

Closes https://github.com/elastic/security-team/issues/11253
Breaking change proposal: https://github.com/elastic/dev/issues/2822

The host and user risk scoring modules or "legacy risk engine" as we often call it internally, has been superseded since v8.10.0 by the risk engine. We submitted a breaking change proposal for v9.0.0 to remove all support to this legacy approach which was approved.

In 8.18 users will be given a warning if they are still using the legacy risk engine and directed to upgrade, this is implemented in #202775.

Changes

  • we previously used the presence of the risk index to decide whether to show the "enable risk engine" button instead of the risk score table, now that we only have only one risk scoring approach, I have changed this to use the risk engine status API, if the risk engine has ever been installed we show the table.

Deletions

  • all code related to displaying legacy risk score
  • we no longer enrich alerts with legacy risk score
  • all code related to upgrading to the "new" risk engine
  • telemetry on the ml_risk_score* indices
  • all internal APIs related to the legacy risk engine

@hop-dev hop-dev changed the title Delete legacy risk engine [Entity Analytics][9.0] Remove all legacy risk engine code Dec 2, 2024
@hop-dev hop-dev self-assigned this Dec 2, 2024
@hop-dev hop-dev added release_note:deprecation backport:skip This PR does not require backporting v9.0.0 Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:Entity Analytics Security Entity Analytics Team labels Dec 2, 2024
@hop-dev
Copy link
Contributor Author

hop-dev commented Dec 2, 2024

/ci

@hop-dev
Copy link
Contributor Author

hop-dev commented Dec 2, 2024

/ci

@hop-dev
Copy link
Contributor Author

hop-dev commented Dec 3, 2024

/ci

@hop-dev
Copy link
Contributor Author

hop-dev commented Dec 3, 2024

/ci

@hop-dev hop-dev requested a review from MadameSheema January 8, 2025 20:13
cy.get(ENABLE_HOST_RISK_SCORE_BUTTON).should('be.visible');

cy.get(ENABLE_USER_RISK_SCORE_BUTTON).should('be.visible');
it('shows enable risk button', () => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be great if we can invest time to try to unskip this spec file :)

cy.task('esArchiverUnload', { archiveName: 'risk_users' });
});

describe('Custom query rule', () => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as before, it would be great to try to unskip the test :)

Copy link
Contributor

@nkhristinin nkhristinin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DE changes LGTM!

Copy link
Contributor

@MadameSheema MadameSheema left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for reviewing all the comments :)

@hop-dev
Copy link
Contributor Author

hop-dev commented Jan 10, 2025

@elasticmachine merge upstream

@elasticmachine
Copy link
Contributor

💚 Build Succeeded

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id before after diff
securitySolution 6541 6445 -96

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
securitySolution 22.2MB 21.1MB -1.1MB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
securitySolution 88.2KB 87.3KB -882.0B
Unknown metric groups

ESLint disabled line counts

id before after diff
securitySolution 575 573 -2

References to deprecated APIs

id before after diff
securitySolution 464 355 -109

Total ESLint disabled count

id before after diff
securitySolution 659 657 -2

History

cc @hop-dev

@hop-dev hop-dev merged commit 80baa2c into elastic:main Jan 10, 2025
@hop-dev hop-dev deleted the delete-legacy-risk-engine branch January 13, 2025 13:11
viduni94 pushed a commit to viduni94/kibana that referenced this pull request Jan 23, 2025
machadoum added a commit that referenced this pull request May 6, 2025
…219858)

## Summary

Closes #219490

Fix an error toast appearing when changing an entities asset criticality
from the entity flyout. See video on issue above for more detail.

We were seeing the risk score request be aborted and this was creating
the error toast.

- Do not toast if an abort error is thrown
- Do not abort the request unless the component receives `skip:true`

### Test Steps

1. Use the security document generator `yarn start entity-store` command
to load entities and enable the risk engine
2. Enable the entity store
3. from the entities table open the entity flyout
4. Assign/change asset criticality
5. observe the error is not present

### Why did the bug happen?
The risk score component stops rendering because the conditions for
aborting and searching differ. So we aborted the HTTP call but didn't
search again.

I believe this PR #201810
introduced the bug

---------

Co-authored-by: machadoum <pablo.nevesmachado@elastic.co>
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request May 6, 2025
…lastic#219858)

## Summary

Closes elastic#219490

Fix an error toast appearing when changing an entities asset criticality
from the entity flyout. See video on issue above for more detail.

We were seeing the risk score request be aborted and this was creating
the error toast.

- Do not toast if an abort error is thrown
- Do not abort the request unless the component receives `skip:true`

### Test Steps

1. Use the security document generator `yarn start entity-store` command
to load entities and enable the risk engine
2. Enable the entity store
3. from the entities table open the entity flyout
4. Assign/change asset criticality
5. observe the error is not present

### Why did the bug happen?
The risk score component stops rendering because the conditions for
aborting and searching differ. So we aborted the HTTP call but didn't
search again.

I believe this PR elastic#201810
introduced the bug

---------

Co-authored-by: machadoum <pablo.nevesmachado@elastic.co>
(cherry picked from commit 64d20bb)
akowalska622 pushed a commit to akowalska622/kibana that referenced this pull request May 29, 2025
…lastic#219858)

## Summary

Closes elastic#219490

Fix an error toast appearing when changing an entities asset criticality
from the entity flyout. See video on issue above for more detail.

We were seeing the risk score request be aborted and this was creating
the error toast.

- Do not toast if an abort error is thrown
- Do not abort the request unless the component receives `skip:true`

### Test Steps

1. Use the security document generator `yarn start entity-store` command
to load entities and enable the risk engine
2. Enable the entity store
3. from the entities table open the entity flyout
4. Assign/change asset criticality
5. observe the error is not present

### Why did the bug happen?
The risk score component stops rendering because the conditions for
aborting and searching differ. So we aborted the HTTP call but didn't
search again.

I believe this PR elastic#201810
introduced the bug

---------

Co-authored-by: machadoum <pablo.nevesmachado@elastic.co>
qn895 pushed a commit to qn895/kibana that referenced this pull request Jun 3, 2025
…lastic#219858)

## Summary

Closes elastic#219490

Fix an error toast appearing when changing an entities asset criticality
from the entity flyout. See video on issue above for more detail.

We were seeing the risk score request be aborted and this was creating
the error toast.

- Do not toast if an abort error is thrown
- Do not abort the request unless the component receives `skip:true`

### Test Steps

1. Use the security document generator `yarn start entity-store` command
to load entities and enable the risk engine
2. Enable the entity store
3. from the entities table open the entity flyout
4. Assign/change asset criticality
5. observe the error is not present

### Why did the bug happen?
The risk score component stops rendering because the conditions for
aborting and searching differ. So we aborted the HTTP call but didn't
search again.

I believe this PR elastic#201810
introduced the bug

---------

Co-authored-by: machadoum <pablo.nevesmachado@elastic.co>
MadameSheema added a commit that referenced this pull request Feb 10, 2026
… orphaned code (#252494)

## Summary

Deletes the `enrichments.cy.ts` Cypress test that has been skipped since
March 2024 (#176965) and removes all orphaned code that was only used by
this test.

**Why delete instead of fix:**
- The test has been skipped for ~2 years with no progress toward
re-enabling
- The key selector (`EnrichedDataRow`) no longer exists in the
application code — the alert flyout enrichment UI was redesigned
- The legacy risk engine code was removed in #201810, further
invalidating the test's assumptions
- Server-side alert enrichment with risk scores is already well-covered
by API integration tests across multiple rule types (custom query, ESQL,
EQL, threshold, new terms, ML)

**Deleted:**
- `cypress/e2e/entity_analytics/enrichments.cy.ts` — the skipped test
- `es_archives/risk_scores_new_updated/` — es_archive only used by this
test

**Cleaned up orphaned selectors:**
- `ENRICHED_DATA_ROW` from `cypress/screens/alerts_details.ts`
- `HOST_RISK_HEADER_COLUMN`, `USER_RISK_HEADER_COLUMN`,
`HOST_RISK_COLUMN`, `USER_RISK_COLUMN`, `ACTION_COLUMN` from
`cypress/screens/alerts.ts`
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Feb 10, 2026
… orphaned code (elastic#252494)

## Summary

Deletes the `enrichments.cy.ts` Cypress test that has been skipped since
March 2024 (elastic#176965) and removes all orphaned code that was only used by
this test.

**Why delete instead of fix:**
- The test has been skipped for ~2 years with no progress toward
re-enabling
- The key selector (`EnrichedDataRow`) no longer exists in the
application code — the alert flyout enrichment UI was redesigned
- The legacy risk engine code was removed in elastic#201810, further
invalidating the test's assumptions
- Server-side alert enrichment with risk scores is already well-covered
by API integration tests across multiple rule types (custom query, ESQL,
EQL, threshold, new terms, ML)

**Deleted:**
- `cypress/e2e/entity_analytics/enrichments.cy.ts` — the skipped test
- `es_archives/risk_scores_new_updated/` — es_archive only used by this
test

**Cleaned up orphaned selectors:**
- `ENRICHED_DATA_ROW` from `cypress/screens/alerts_details.ts`
- `HOST_RISK_HEADER_COLUMN`, `USER_RISK_HEADER_COLUMN`,
`HOST_RISK_COLUMN`, `USER_RISK_COLUMN`, `ACTION_COLUMN` from
`cypress/screens/alerts.ts`

(cherry picked from commit 9a7adf3)
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Feb 10, 2026
… orphaned code (elastic#252494)

## Summary

Deletes the `enrichments.cy.ts` Cypress test that has been skipped since
March 2024 (elastic#176965) and removes all orphaned code that was only used by
this test.

**Why delete instead of fix:**
- The test has been skipped for ~2 years with no progress toward
re-enabling
- The key selector (`EnrichedDataRow`) no longer exists in the
application code — the alert flyout enrichment UI was redesigned
- The legacy risk engine code was removed in elastic#201810, further
invalidating the test's assumptions
- Server-side alert enrichment with risk scores is already well-covered
by API integration tests across multiple rule types (custom query, ESQL,
EQL, threshold, new terms, ML)

**Deleted:**
- `cypress/e2e/entity_analytics/enrichments.cy.ts` — the skipped test
- `es_archives/risk_scores_new_updated/` — es_archive only used by this
test

**Cleaned up orphaned selectors:**
- `ENRICHED_DATA_ROW` from `cypress/screens/alerts_details.ts`
- `HOST_RISK_HEADER_COLUMN`, `USER_RISK_HEADER_COLUMN`,
`HOST_RISK_COLUMN`, `USER_RISK_COLUMN`, `ACTION_COLUMN` from
`cypress/screens/alerts.ts`

(cherry picked from commit 9a7adf3)
kibanamachine added a commit that referenced this pull request Feb 10, 2026
…st and orphaned code (#252494) (#252584)

# Backport

This will backport the following commits from `main` to `9.2`:
- [[Security Solution] Delete skipped enrichments.cy.ts Cypress test and
orphaned code (#252494)](#252494)

<!--- Backport version: 9.6.6 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sorenlouv/backport)

<!--BACKPORT [{"author":{"name":"Gloria
Hornero","email":"gloria.hornero@elastic.co"},"sourceCommit":{"committedDate":"2026-02-10T16:55:32Z","message":"[Security
Solution] Delete skipped enrichments.cy.ts Cypress test and orphaned
code (#252494)\n\n## Summary\n\nDeletes the `enrichments.cy.ts` Cypress
test that has been skipped since\nMarch 2024 (#176965) and removes all
orphaned code that was only used by\nthis test.\n\n**Why delete instead
of fix:**\n- The test has been skipped for ~2 years with no progress
toward\nre-enabling\n- The key selector (`EnrichedDataRow`) no longer
exists in the\napplication code — the alert flyout enrichment UI was
redesigned\n- The legacy risk engine code was removed in #201810,
further\ninvalidating the test's assumptions\n- Server-side alert
enrichment with risk scores is already well-covered\nby API integration
tests across multiple rule types (custom query, ESQL,\nEQL, threshold,
new terms, ML)\n\n**Deleted:**\n-
`cypress/e2e/entity_analytics/enrichments.cy.ts` — the skipped test\n-
`es_archives/risk_scores_new_updated/` — es_archive only used by
this\ntest\n\n**Cleaned up orphaned selectors:**\n- `ENRICHED_DATA_ROW`
from `cypress/screens/alerts_details.ts`\n- `HOST_RISK_HEADER_COLUMN`,
`USER_RISK_HEADER_COLUMN`,\n`HOST_RISK_COLUMN`, `USER_RISK_COLUMN`,
`ACTION_COLUMN`
from\n`cypress/screens/alerts.ts`","sha":"9a7adf37d740d8828419b6f4a982596a77fc1643","branchLabelMapping":{"^v9.4.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:
SecuritySolution","backport:all-open","Team:Entity
Analytics","v9.4.0"],"title":"[Security Solution] Delete skipped
enrichments.cy.ts Cypress test and orphaned
code","number":252494,"url":"https://github.com/elastic/kibana/pull/252494","mergeCommit":{"message":"[Security
Solution] Delete skipped enrichments.cy.ts Cypress test and orphaned
code (#252494)\n\n## Summary\n\nDeletes the `enrichments.cy.ts` Cypress
test that has been skipped since\nMarch 2024 (#176965) and removes all
orphaned code that was only used by\nthis test.\n\n**Why delete instead
of fix:**\n- The test has been skipped for ~2 years with no progress
toward\nre-enabling\n- The key selector (`EnrichedDataRow`) no longer
exists in the\napplication code — the alert flyout enrichment UI was
redesigned\n- The legacy risk engine code was removed in #201810,
further\ninvalidating the test's assumptions\n- Server-side alert
enrichment with risk scores is already well-covered\nby API integration
tests across multiple rule types (custom query, ESQL,\nEQL, threshold,
new terms, ML)\n\n**Deleted:**\n-
`cypress/e2e/entity_analytics/enrichments.cy.ts` — the skipped test\n-
`es_archives/risk_scores_new_updated/` — es_archive only used by
this\ntest\n\n**Cleaned up orphaned selectors:**\n- `ENRICHED_DATA_ROW`
from `cypress/screens/alerts_details.ts`\n- `HOST_RISK_HEADER_COLUMN`,
`USER_RISK_HEADER_COLUMN`,\n`HOST_RISK_COLUMN`, `USER_RISK_COLUMN`,
`ACTION_COLUMN`
from\n`cypress/screens/alerts.ts`","sha":"9a7adf37d740d8828419b6f4a982596a77fc1643"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.4.0","branchLabelMappingKey":"^v9.4.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/252494","number":252494,"mergeCommit":{"message":"[Security
Solution] Delete skipped enrichments.cy.ts Cypress test and orphaned
code (#252494)\n\n## Summary\n\nDeletes the `enrichments.cy.ts` Cypress
test that has been skipped since\nMarch 2024 (#176965) and removes all
orphaned code that was only used by\nthis test.\n\n**Why delete instead
of fix:**\n- The test has been skipped for ~2 years with no progress
toward\nre-enabling\n- The key selector (`EnrichedDataRow`) no longer
exists in the\napplication code — the alert flyout enrichment UI was
redesigned\n- The legacy risk engine code was removed in #201810,
further\ninvalidating the test's assumptions\n- Server-side alert
enrichment with risk scores is already well-covered\nby API integration
tests across multiple rule types (custom query, ESQL,\nEQL, threshold,
new terms, ML)\n\n**Deleted:**\n-
`cypress/e2e/entity_analytics/enrichments.cy.ts` — the skipped test\n-
`es_archives/risk_scores_new_updated/` — es_archive only used by
this\ntest\n\n**Cleaned up orphaned selectors:**\n- `ENRICHED_DATA_ROW`
from `cypress/screens/alerts_details.ts`\n- `HOST_RISK_HEADER_COLUMN`,
`USER_RISK_HEADER_COLUMN`,\n`HOST_RISK_COLUMN`, `USER_RISK_COLUMN`,
`ACTION_COLUMN`
from\n`cypress/screens/alerts.ts`","sha":"9a7adf37d740d8828419b6f4a982596a77fc1643"}}]}]
BACKPORT-->

Co-authored-by: Gloria Hornero <gloria.hornero@elastic.co>
kibanamachine added a commit that referenced this pull request Feb 10, 2026
…st and orphaned code (#252494) (#252585)

# Backport

This will backport the following commits from `main` to `9.3`:
- [[Security Solution] Delete skipped enrichments.cy.ts Cypress test and
orphaned code (#252494)](#252494)

<!--- Backport version: 9.6.6 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sorenlouv/backport)

<!--BACKPORT [{"author":{"name":"Gloria
Hornero","email":"gloria.hornero@elastic.co"},"sourceCommit":{"committedDate":"2026-02-10T16:55:32Z","message":"[Security
Solution] Delete skipped enrichments.cy.ts Cypress test and orphaned
code (#252494)\n\n## Summary\n\nDeletes the `enrichments.cy.ts` Cypress
test that has been skipped since\nMarch 2024 (#176965) and removes all
orphaned code that was only used by\nthis test.\n\n**Why delete instead
of fix:**\n- The test has been skipped for ~2 years with no progress
toward\nre-enabling\n- The key selector (`EnrichedDataRow`) no longer
exists in the\napplication code — the alert flyout enrichment UI was
redesigned\n- The legacy risk engine code was removed in #201810,
further\ninvalidating the test's assumptions\n- Server-side alert
enrichment with risk scores is already well-covered\nby API integration
tests across multiple rule types (custom query, ESQL,\nEQL, threshold,
new terms, ML)\n\n**Deleted:**\n-
`cypress/e2e/entity_analytics/enrichments.cy.ts` — the skipped test\n-
`es_archives/risk_scores_new_updated/` — es_archive only used by
this\ntest\n\n**Cleaned up orphaned selectors:**\n- `ENRICHED_DATA_ROW`
from `cypress/screens/alerts_details.ts`\n- `HOST_RISK_HEADER_COLUMN`,
`USER_RISK_HEADER_COLUMN`,\n`HOST_RISK_COLUMN`, `USER_RISK_COLUMN`,
`ACTION_COLUMN`
from\n`cypress/screens/alerts.ts`","sha":"9a7adf37d740d8828419b6f4a982596a77fc1643","branchLabelMapping":{"^v9.4.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:
SecuritySolution","backport:all-open","Team:Entity
Analytics","v9.4.0"],"title":"[Security Solution] Delete skipped
enrichments.cy.ts Cypress test and orphaned
code","number":252494,"url":"https://github.com/elastic/kibana/pull/252494","mergeCommit":{"message":"[Security
Solution] Delete skipped enrichments.cy.ts Cypress test and orphaned
code (#252494)\n\n## Summary\n\nDeletes the `enrichments.cy.ts` Cypress
test that has been skipped since\nMarch 2024 (#176965) and removes all
orphaned code that was only used by\nthis test.\n\n**Why delete instead
of fix:**\n- The test has been skipped for ~2 years with no progress
toward\nre-enabling\n- The key selector (`EnrichedDataRow`) no longer
exists in the\napplication code — the alert flyout enrichment UI was
redesigned\n- The legacy risk engine code was removed in #201810,
further\ninvalidating the test's assumptions\n- Server-side alert
enrichment with risk scores is already well-covered\nby API integration
tests across multiple rule types (custom query, ESQL,\nEQL, threshold,
new terms, ML)\n\n**Deleted:**\n-
`cypress/e2e/entity_analytics/enrichments.cy.ts` — the skipped test\n-
`es_archives/risk_scores_new_updated/` — es_archive only used by
this\ntest\n\n**Cleaned up orphaned selectors:**\n- `ENRICHED_DATA_ROW`
from `cypress/screens/alerts_details.ts`\n- `HOST_RISK_HEADER_COLUMN`,
`USER_RISK_HEADER_COLUMN`,\n`HOST_RISK_COLUMN`, `USER_RISK_COLUMN`,
`ACTION_COLUMN`
from\n`cypress/screens/alerts.ts`","sha":"9a7adf37d740d8828419b6f4a982596a77fc1643"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.4.0","branchLabelMappingKey":"^v9.4.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/252494","number":252494,"mergeCommit":{"message":"[Security
Solution] Delete skipped enrichments.cy.ts Cypress test and orphaned
code (#252494)\n\n## Summary\n\nDeletes the `enrichments.cy.ts` Cypress
test that has been skipped since\nMarch 2024 (#176965) and removes all
orphaned code that was only used by\nthis test.\n\n**Why delete instead
of fix:**\n- The test has been skipped for ~2 years with no progress
toward\nre-enabling\n- The key selector (`EnrichedDataRow`) no longer
exists in the\napplication code — the alert flyout enrichment UI was
redesigned\n- The legacy risk engine code was removed in #201810,
further\ninvalidating the test's assumptions\n- Server-side alert
enrichment with risk scores is already well-covered\nby API integration
tests across multiple rule types (custom query, ESQL,\nEQL, threshold,
new terms, ML)\n\n**Deleted:**\n-
`cypress/e2e/entity_analytics/enrichments.cy.ts` — the skipped test\n-
`es_archives/risk_scores_new_updated/` — es_archive only used by
this\ntest\n\n**Cleaned up orphaned selectors:**\n- `ENRICHED_DATA_ROW`
from `cypress/screens/alerts_details.ts`\n- `HOST_RISK_HEADER_COLUMN`,
`USER_RISK_HEADER_COLUMN`,\n`HOST_RISK_COLUMN`, `USER_RISK_COLUMN`,
`ACTION_COLUMN`
from\n`cypress/screens/alerts.ts`","sha":"9a7adf37d740d8828419b6f4a982596a77fc1643"}}]}]
BACKPORT-->

Co-authored-by: Gloria Hornero <gloria.hornero@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport:skip This PR does not require backporting release_note:deprecation Team:Entity Analytics Security Entity Analytics Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v9.0.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

10 participants