[Security Solution][Detection Engine] adds async ES|QL query by vitaliidm · Pull Request #216667 · elastic/kibana

vitaliidm · 2025-04-01T15:40:53Z

Summary

addresses https://github.com/elastic/security-team/issues/11116 (list item 2)

Introducing async query would allow to overcome ES request timeout for long running rules and queries.

Timeout for ES request is defined in alerting framework and is smaller value out of rule execution timeout or default ES request timeout(which is 5m and hardcoded here).

If ES|QL rule performs a single long-running ES query, it can time out after 5m due to this ES request timeout. This value can't be changed, unlike rule execution timeout. It can be overwritten in Kibana config

xpack.alerting.rules.run:
  timeout: '10m'
  ruleTypeOverrides:
    - id:  'siem.esqlRule'
      timeout: '15m'

So, we can encounter situations when rule fails execution after 5m due to ES request timeout, despite a fact it configured with longer timeout of 15m

By using async query, we can overcome this limitation and can poll async query results until it completes or rule timeouts

More details in internal issue

… into de_9_1/async_esql

…apper abort controller

elasticmachine · 2025-04-03T09:43:17Z

Pinging @elastic/security-solution (Team: SecuritySolution)

elasticmachine · 2025-04-03T09:43:18Z

Pinging @elastic/security-detection-engine (Team:Detection Engine)

marshallmain · 2025-04-03T18:07:22Z

...curity/plugins/security_solution/server/lib/detection_engine/rule_types/esql/esql_request.ts

+      description: i18n.ESQL_SEARCH_REQUEST_DESCRIPTION,
+    });
+    const asyncSearchStarted = performance.now();
+    const asyncEsqlResponse = await esClient.transport.request<AsyncEsqlResponse>({


why use transport.request instead of .esql?

I have noticed esql method returns not correct typings. So decided to stick to transport.request as in previous implmentation

marshallmain · 2025-04-03T18:40:26Z

...curity/plugins/security_solution/server/lib/detection_engine/rule_types/esql/esql_request.ts

-  requestBody: Record<string, unknown>;
+  requestBody: {
+    query: string;
+    filter: QueryDslQueryContainer;


wait_for_completion_timeout and keep_alive should probably be params here too

I haven't added them, since they were not used.
Added them now

turned out, these values applicable to body only, so I removed them from this type

marshallmain · 2025-04-03T18:42:39Z

...curity/plugins/security_solution/server/lib/detection_engine/rule_types/esql/esql_request.ts

+      if (isCancelled) {
+        throw new Error('Rule execution cancelled due to timeout');
+      }
+      await new Promise((resolve) => setTimeout(resolve, pollInterval));


probably want to wait at the beginning of the loop so we wait in between the initial response and the first poll

marshallmain · 2025-04-03T18:48:30Z

...s/security_solution/server/lib/detection_engine/rule_types/esql/build_esql_search_request.ts

        filter: requestFilter,
      },
    },
+    wait_for_completion_timeout: '4m', // hard limit request timeout is 5m set by ES proxy and alerting framework. So, we should be fine to wait 4m for async query completion. If rule execution is shorter than 4m and query was not completed, it will be aborted.


setting wait_for_completion_timeout this high makes this effectively a synchronous query. Should this be keep_alive instead? Some keep_alive value longer than the rule timeout would be sufficient. I think we want wait_for_completion_timeout to be some number of seconds, like 5 or 10 seconds.

Setting keep_alive will help ensure that the results are deleted quickly even if the cleanup DELETE request fails.

Yes, the idea was not to start polling until rule timeouts. If rule timeout is greater then 5m, when we can hit ES requests timeout limitation, we would stop waiting for query to complete(4m) and start polling.

Setting keep_alive will help ensure that the results are deleted quickly even if the cleanup DELETE request fails.

We don't have access to rule timeout within executor - only to shouldStopExecution. Which can't be used to set up keep_alive beforehand. I am not sure we want to expose this to executor and in future to rely on that value while running rule.

I see, that makes sense. Is 4m the right value for both ECH and serverless? The default rule timeout is different for serverless, right? Is the default connection timeout different?

I think it would be worth following up on this PR to see if we can get access to the rule timeout or maybe have the framework wrapper inject a keep_alive value depending on the rule timeout. There could be some use case eventually for async requests initiated by one rule execution and retrieved by a later one, so maybe not every async request needs to have keep_alive == rule timeout, but I think it's the typical scenario for us. It'll just help with system resilience to set it since there could be hundreds of ESQL rules running every few minutes and we'll never need those results for 5 days.

I see, that makes sense. Is 4m the right value for both ECH and serverless? The default rule timeout is different for serverless, right? Is the default connection timeout different?

Default rule timeout for serverless is 1m. If async query request runs longer that this, it would be aborted and query cancelled/deleted in ES.
It's 5m for ECH. If query takes longer than this to finish, it would be cancelled and deleted as well

@marshallmain , issue to expose rule timeout to rule executor: #218072

Sounds good, we'll just have to be aware of any timeout discrepancies in the future. If the rule timeout increases in serverless (which is a desirable change for us) but the connection timeout values are different (e.g. lower in serverless), we could start seeing environment specific failures.

I think errors would be in the same category - timeout related. Can be just for different requests(query or poll) and different times. But, we already have different timeout for different envs

rylnd

Reviewed previously, but gave it another pass since I was still listed as blocking merge.

LGTM.

...curity/plugins/security_solution/server/lib/detection_engine/rule_types/esql/esql_request.ts

rylnd · 2025-04-16T18:50:43Z

...curity/plugins/security_solution/server/lib/detection_engine/rule_types/esql/esql_request.ts

+import { logEsqlRequest } from '../utils/logged_requests';
+import * as i18n from '../translations';
+
+const logDuration = (startTime: number, loggedRequests: RulePreviewLoggedRequest[] | undefined) => {


Nit: would this more accurately be something like

Suggested change

const logDuration = (startTime: number, loggedRequests: RulePreviewLoggedRequest[] | undefined) => {

const setLatestRequestDuration = (startTime: number, loggedRequests: RulePreviewLoggedRequest[] | undefined) => {

Since this function seems to only permute loggedRequests, and doesn't actually "log" anything itself?

…/detection_engine/rule_types/esql/esql_request.ts Co-authored-by: Ryland Herrick <ryalnd@gmail.com>

elasticmachine · 2025-04-17T12:03:08Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: 4d37838

Failed CI Steps

Jest Tests #13

Test Failures

[job] [logs] Jest Tests #13 / TemplatesList renders template details correctly

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`alerting`	852	853	+1

Unknown metric groups

API count

id	before	after	diff
`alerting`	886	887	+1

History

💔 Build #294073 failed adc5ef8
💛 Build #293678 was flaky 236e9da
💔 Build #293610 failed 7e67924
💔 Build #293368 failed 066c9e4
💔 Build #293132 failed b53a8ef

cc @vitaliidm

kibanamachine · 2025-04-17T14:23:25Z

Starting backport for target branches: 8.18, 8.19, 9.0

https://github.com/elastic/kibana/actions/runs/14517920458

…#216667) ## Summary - addresses elastic/security-team#11116 (list item 2) Introducing async query would allow to overcome ES request timeout for long running rules and queries. Timeout for ES request is [defined in alerting framework](https://github.com/elastic/kibana/blob/8.18/x-pack/platform/plugins/shared/alerting/server/lib/get_es_request_timeout.ts#L21) and is smaller value out of rule execution timeout or default ES request timeout(which is 5m and hardcoded [here](https://github.com/elastic/kibana/blob/8.18/x-pack/platform/plugins/shared/alerting/server/lib/get_rule_task_timeout.ts)). If ES|QL rule performs a single long-running ES query, it can time out after 5m due to this ES request timeout. This value can't be changed, unlike rule execution timeout. It can be overwritten in Kibana config ``` xpack.alerting.rules.run: timeout: '10m' ruleTypeOverrides: - id: 'siem.esqlRule' timeout: '15m' ``` So, we can encounter situations when rule fails execution after 5m due to ES request timeout, despite a fact it configured with longer timeout of 15m By using async query, we can overcome this limitation and can poll async query results until it completes or rule timeouts More details in internal [issue](elastic/sdh-security-team#1224) --------- Co-authored-by: Ryland Herrick <ryalnd@gmail.com> (cherry picked from commit 3d7aac1)

kibanamachine · 2025-04-17T14:29:45Z

💔 Some backports could not be created

Status	Branch	Result
❌	8.18	Backport failed because of merge conflicts
✅	8.19
❌	9.0	Backport failed because of merge conflicts

Note: Successful backport PRs will be merged automatically after passing CI.

Manual backport

To create the backport manually run:

node scripts/backport --pr 216667

Questions ?

Please refer to the Backport tool documentation

…#216667) ## Summary - addresses elastic/security-team#11116 (list item 2) Introducing async query would allow to overcome ES request timeout for long running rules and queries. Timeout for ES request is [defined in alerting framework](https://github.com/elastic/kibana/blob/8.18/x-pack/platform/plugins/shared/alerting/server/lib/get_es_request_timeout.ts#L21) and is smaller value out of rule execution timeout or default ES request timeout(which is 5m and hardcoded [here](https://github.com/elastic/kibana/blob/8.18/x-pack/platform/plugins/shared/alerting/server/lib/get_rule_task_timeout.ts)). If ES|QL rule performs a single long-running ES query, it can time out after 5m due to this ES request timeout. This value can't be changed, unlike rule execution timeout. It can be overwritten in Kibana config ``` xpack.alerting.rules.run: timeout: '10m' ruleTypeOverrides: - id: 'siem.esqlRule' timeout: '15m' ``` So, we can encounter situations when rule fails execution after 5m due to ES request timeout, despite a fact it configured with longer timeout of 15m By using async query, we can overcome this limitation and can poll async query results until it completes or rule timeouts More details in internal [issue](elastic/sdh-security-team#1224) --------- Co-authored-by: Ryland Herrick <ryalnd@gmail.com> (cherry picked from commit 3d7aac1) # Conflicts: # x-pack/solutions/security/plugins/security_solution/server/lib/detection_engine/rule_preview/api/preview_rules/route.ts # x-pack/solutions/security/plugins/security_solution/server/lib/detection_engine/rule_types/esql/esql.ts

vitaliidm · 2025-04-17T15:49:49Z

💚 All backports created successfully

Status	Branch	Result
✅	9.0
✅	8.18

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

…#216667) ## Summary - addresses elastic/security-team#11116 (list item 2) Introducing async query would allow to overcome ES request timeout for long running rules and queries. Timeout for ES request is [defined in alerting framework](https://github.com/elastic/kibana/blob/8.18/x-pack/platform/plugins/shared/alerting/server/lib/get_es_request_timeout.ts#L21) and is smaller value out of rule execution timeout or default ES request timeout(which is 5m and hardcoded [here](https://github.com/elastic/kibana/blob/8.18/x-pack/platform/plugins/shared/alerting/server/lib/get_rule_task_timeout.ts)). If ES|QL rule performs a single long-running ES query, it can time out after 5m due to this ES request timeout. This value can't be changed, unlike rule execution timeout. It can be overwritten in Kibana config ``` xpack.alerting.rules.run: timeout: '10m' ruleTypeOverrides: - id: 'siem.esqlRule' timeout: '15m' ``` So, we can encounter situations when rule fails execution after 5m due to ES request timeout, despite a fact it configured with longer timeout of 15m By using async query, we can overcome this limitation and can poll async query results until it completes or rule timeouts More details in internal [issue](elastic/sdh-security-team#1224) --------- Co-authored-by: Ryland Herrick <ryalnd@gmail.com> (cherry picked from commit 3d7aac1) # Conflicts: # x-pack/solutions/security/plugins/security_solution/server/lib/detection_engine/rule_preview/api/preview_rules/route.ts # x-pack/solutions/security/plugins/security_solution/server/lib/detection_engine/rule_types/esql/esql.ts

…216667) (#218567) # Backport This will backport the following commits from `main` to `8.19`: - [[Security Solution][Detection Engine] adds async ES|QL query (#216667)](#216667)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)  Co-authored-by: Vitalii Dmyterko <92328789+vitaliidm@users.noreply.github.com>

…216667) (#218583) # Backport This will backport the following commits from `main` to `9.0`: - [[Security Solution][Detection Engine] adds async ES|QL query (#216667)](#216667)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)

…216667) (#218585) # Backport This will backport the following commits from `main` to `8.18`: - [[Security Solution][Detection Engine] adds async ES|QL query (#216667)](#216667)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)  --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

…#216667) ## Summary - addresses elastic/security-team#11116 (list item 2) Introducing async query would allow to overcome ES request timeout for long running rules and queries. Timeout for ES request is [defined in alerting framework](https://github.com/elastic/kibana/blob/8.18/x-pack/platform/plugins/shared/alerting/server/lib/get_es_request_timeout.ts#L21) and is smaller value out of rule execution timeout or default ES request timeout(which is 5m and hardcoded [here](https://github.com/elastic/kibana/blob/8.18/x-pack/platform/plugins/shared/alerting/server/lib/get_rule_task_timeout.ts)). If ES|QL rule performs a single long-running ES query, it can time out after 5m due to this ES request timeout. This value can't be changed, unlike rule execution timeout. It can be overwritten in Kibana config ``` xpack.alerting.rules.run: timeout: '10m' ruleTypeOverrides: - id: 'siem.esqlRule' timeout: '15m' ``` So, we can encounter situations when rule fails execution after 5m due to ES request timeout, despite a fact it configured with longer timeout of 15m By using async query, we can overcome this limitation and can poll async query results until it completes or rule timeouts More details in internal [issue](elastic/sdh-security-team#1224) --------- Co-authored-by: Ryland Herrick <ryalnd@gmail.com>

[Security Solution][Detection Engine] adds async ES|QL query

8fd25d5

vitaliidm self-assigned this Apr 1, 2025

vitaliidm added release_note:skip Skip the PR/issue when compiling release notes Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:Detection Engine Security Solution Detection Engine Area labels Apr 1, 2025

vitaliidm added 13 commits April 2, 2025 10:44

Merge branch 'main' into de_9_1/async_esql

a7724d9

[Security Solution][Detection Engine] test

957e85b

[Security Solution][Detection Engine] more informative message

1ee4962

[Security Solution][Detection Engine] alerting changes

8a02d3c

Update build_esql_search_request.ts

0d3c168

[Security Solution][Detection Engine] refacotr

ff2ab3b

Merge branch 'de_9_1/async_esql' of https://github.com/vitaliidm/kibana…

4ac1d30

… into de_9_1/async_esql

[Security Solution][Detection Engine] logged requests refactor

de0c6d2

Merge branch 'main' into de_9_1/async_esql

df83947

[Security Solution][Detection Engine] alerting framewrok

79bc275

Merge branch 'de_9_1/async_esql' of https://github.com/vitaliidm/kibana…

fa753e4

… into de_9_1/async_esql

[Security Solution][Detection Engine] exclude DELETE from alerting wr…

46f74e6

…apper abort controller

Update build_esql_search_request.ts

27e7f84

vitaliidm marked this pull request as ready for review April 3, 2025 09:43

vitaliidm requested review from a team as code owners April 3, 2025 09:43

vitaliidm requested a review from rylnd April 3, 2025 09:43

vitaliidm added v9.1.0 v8.19.0 backport:version Backport to applied version labels labels Apr 3, 2025

yctercero requested a review from marshallmain April 3, 2025 18:02

marshallmain reviewed Apr 3, 2025

View reviewed changes

Merge branch 'main' into de_9_1/async_esql

c681fd5

vitaliidm enabled auto-merge (squash) April 15, 2025 16:25

vitaliidm disabled auto-merge April 15, 2025 17:17

vitaliidm added 2 commits April 16, 2025 10:39

Merge branch 'main' into de_9_1/async_esql

7e67924

Merge branch 'main' into de_9_1/async_esql

236e9da

rylnd approved these changes Apr 16, 2025

View reviewed changes

vitaliidm and others added 4 commits April 17, 2025 10:22

Update x-pack/solutions/security/plugins/security_solution/server/lib…

e849de0

…/detection_engine/rule_types/esql/esql_request.ts Co-authored-by: Ryland Herrick <ryalnd@gmail.com>

Update esql_request.ts

8b75f69

Merge branch 'main' into de_9_1/async_esql

adc5ef8

Update esql_request.ts

4d37838

vitaliidm added the v9.0.1 label Apr 17, 2025

vitaliidm merged commit 3d7aac1 into elastic:main Apr 17, 2025
11 checks passed

kibanamachine mentioned this pull request Apr 17, 2025

[8.19] [Security Solution][Detection Engine] adds async ES|QL query (#216667) #218567

Merged

vitaliidm mentioned this pull request Apr 17, 2025

[9.0] [Security Solution][Detection Engine] adds async ES|QL query (#216667) #218583

Merged

vitaliidm mentioned this pull request Apr 17, 2025

[8.18] [Security Solution][Detection Engine] adds async ES|QL query (#216667) #218585

Merged

vitaliidm mentioned this pull request Apr 22, 2025

[Security Solution][Detection Engine] expose rule timeout value in Detection Engine executor #218072

Closed

This was referenced May 1, 2025

8.18.1 release notes elastic/security-docs#6802

Merged

[Security] 9.0.1 release notes elastic/docs-content#1309

Merged

yctercero mentioned this pull request Jul 22, 2025

[Security] 8.19.0 release notes elastic/security-docs#6925

Merged

	const logDuration = (startTime: number, loggedRequests: RulePreviewLoggedRequest[] \| undefined) => {
	const setLatestRequestDuration = (startTime: number, loggedRequests: RulePreviewLoggedRequest[] \| undefined) => {

Comments

Conversation

vitaliidm commented Apr 1, 2025 • edited by kibanamachine Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

elasticmachine commented Apr 3, 2025

Uh oh!

elasticmachine commented Apr 3, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rylnd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Apr 17, 2025

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

Public APIs missing comments

API count

History

Uh oh!

Uh oh!

kibanamachine commented Apr 17, 2025

Uh oh!

kibanamachine commented Apr 17, 2025

💔 Some backports could not be created

Manual backport

Questions ?

Uh oh!

vitaliidm commented Apr 17, 2025

💚 All backports created successfully

Questions ?

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

vitaliidm commented Apr 1, 2025 •

edited by kibanamachine

Loading