[Security Solution] [Detection Engine] Logs shard failures for eql event queries on rule details page and in event log by dhurley14 · Pull Request #207396 · elastic/kibana

dhurley14 · 2025-01-21T16:29:53Z

Summary

Adds support for shard failures for EQL event queries in the detection engine.

How to test:

start auditbeat and packetbeat
Add the following "broken" runtime field to ONLY ONE of those index mappings

PUT auditbeat-8.13.2/_mapping
{
  "runtime": {
    "broken": {
      "type": "long",
      "script": {
        "lang": "painless",
        "source": "emit(doc['non_existing'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ENGLISH))"
      }
    }
  }
}

Create an EQL rule with the following query: any where agent.type == "auditbeat" or broken == 1
Create and enable the rule.
Warning message should appear on the rule details page

Similarly, you can create an eql sequence query and see the error appear for shard failures, as we don't support partial results for sequence queries.

Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines

…ial failure

elasticmachine · 2025-01-22T17:07:44Z

Pinging @elastic/security-detection-engine (Team:Detection Engine)

...ins/security_solution/server/lib/detection_engine/rule_types/eql/build_eql_search_request.ts

rylnd · 2025-01-23T22:39:19Z

...ins/security_solution/server/lib/detection_engine/rule_types/eql/build_eql_search_request.ts

      },
+      // the allow_partial$ query parameters supersede xpack settings on cluster
+      // @ts-expect-error unknown property allow_partial_search_results
+      // TODO: remove this ts-expect when 8.18 elasticsearch client is released.


Is there a PR/Issue we can reference, here?

Good question.. Maybe this? elastic/elasticsearch-specification#3342

v8.16.0 package.json:

kibana/package.json

Line 118 in d4cc532

"@elastic/elasticsearch": "^8.16.0",

@yctercero @rylnd I just opened this ticket to track the missing type support. Wondering when the 8.18 client will be added to the codebase 🤔

#208760 (comment)

Asking operations: https://elastic.slack.com/archives/C5UDAFZQU/p1738167614946439

...lutions/security/plugins/security_solution/server/lib/detection_engine/rule_types/eql/eql.ts

yctercero · 2025-01-27T19:47:43Z

...lutions/security/plugins/security_solution/server/lib/detection_engine/rule_types/eql/eql.ts

+
      const { events, sequences } = response.hits;

+      if (shardFailures) {


Chatted on Zoom, but to summarize - shardFailures is returned for both sequence and non-sequence queries, so we can't rely on this alone to determine what error message to display.

Testing it out, it looks great for non-sequence queries, but the same error displays for sequence queries. Still super helpful to show the shard failures for both types of queries, but for sequence queries we'll want to tweak the language just a bit to make it clear that no search was done, even on available shards.

…y vs eql event query

…, add comment ref to issue to track es client upgrade

rylnd

I was able to reproduce both the non-sequence warning:

As well as the error for sequence queries:

NB that you need the data necessary to find a sequence hit in order to reproduce the error.

rylnd · 2025-01-29T20:35:14Z

...lutions/security/plugins/security_solution/server/lib/detection_engine/rule_types/eql/eql.ts

 } from '@kbn/alerting-plugin/server';
 import type * as estypes from '@elastic/elasticsearch/lib/api/typesWithBodyKey';
 import type { Filter } from '@kbn/es-query';
+


I appreciate the addition of meaningful whitespace 👍

...lutions/security/plugins/security_solution/server/lib/detection_engine/rule_types/eql/eql.ts

approksiu · 2025-01-30T17:02:44Z

@dhurley14 what is the error message for EQL sequence failure with unavailable shards?

dhurley14 · 2025-01-30T18:03:06Z

~~@approksiu the error message for shard failures with sequence queries is: The EQL query failed to run successfully on the following shards: {shardFailures}~~

~~I can update it to something else if that's not clear.~~

Edit: Missed the suggested text in the slack message. I will update now. Sorry!

…o eql-partial-results

dhurley14 · 2025-01-30T18:21:38Z

@approksiu Updated it to match suggested text in slack: The EQL query failed to run successfully due to unavailable shards: {shardFailures}

elasticmachine · 2025-01-30T19:53:09Z

⏳ Build in-progress, with failures

Buildkite Build
Commit: d2f90b4

Failed CI Steps

History

💔 Build #272052 failed f13453d
💔 Build #271981 failed 8885889
💔 Build #271850 failed 0be99f5
💔 Build #271804 failed 8a5b3d6
💔 Build #271774 failed ab43c6f
💔 Build #271593 failed 7cc62ad

kibanamachine · 2025-01-30T19:56:09Z

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/13060174669

…ent queries on rule details page and in event log (elastic#207396) ## Summary Related: elastic/elasticsearch#116388 Adds support for shard failures for EQL event queries in the detection engine. (cherry picked from commit 4419390)

kibanamachine · 2025-01-30T20:01:32Z

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

…eql event queries on rule details page and in event log (#207396) (#209019) # Backport This will backport the following commits from `main` to `8.x`: - [[Security Solution] [Detection Engine] Logs shard failures for eql event queries on rule details page and in event log (#207396)](#207396)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)  Co-authored-by: Devin W. Hurley <devin.hurley@elastic.co>

marshallmain · 2025-03-07T18:23:44Z

💚 All backports created successfully

Status	Branch	Result
✅	9.0

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

…ent queries on rule details page and in event log (elastic#207396) ## Summary Related: elastic/elasticsearch#116388 Adds support for shard failures for EQL event queries in the detection engine. (cherry picked from commit 4419390)

…eql event queries on rule details page and in event log (#207396) (#213616) # Backport This will backport the following commits from `main` to `9.0`: - [[Security Solution] [Detection Engine] Logs shard failures for eql event queries on rule details page and in event log (#207396)](#207396)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)  Co-authored-by: Devin W. Hurley <devin.hurley@elastic.co>

log shard failures on eql event queries as partial failures

630a354

dhurley14 changed the title ~~[Security Solution] [Detection Engine] log shard failures on eql event queries as partial failures~~ [Security Solution] [Detection Engine] Logs shard failures for eql event queries on rule details page and in event log Jan 21, 2025

updates error message and write entry as error in event log, not part…

9c20a89

…ial failure

dhurley14 added review release_note:fix v9.0.0 Team:Detection Engine Security Solution Detection Engine Area v8.18.0 labels Jan 21, 2025

update snapshot and tests for new query param for eql queries

c46ed02

dhurley14 marked this pull request as ready for review January 22, 2025 17:07

dhurley14 requested a review from a team as a code owner January 22, 2025 17:07

dhurley14 requested a review from rylnd January 22, 2025 17:07

dhurley14 added the backport:version Backport to applied version labels label Jan 22, 2025

Merge branch 'main' into eql-partial-results

4522903

rylnd reviewed Jan 23, 2025

View reviewed changes

dhurley14 added 2 commits January 24, 2025 10:34

fix comments and i18n text

ea358e3

fix i18n

2b588f2

dhurley14 requested a review from rylnd January 24, 2025 16:42

dhurley14 added 3 commits January 24, 2025 13:15

fixes i18n id format

df9bd54

Merge branch 'main' into eql-partial-results

00c2a8c

Merge branch 'main' into eql-partial-results

6aaf331

yctercero added ci:cloud-deploy Create or update a Cloud deployment ci:cloud-redeploy Always create a new Cloud deployment labels Jan 27, 2025

yctercero reviewed Jan 27, 2025

View reviewed changes

dhurley14 linked an issue Jan 27, 2025 that may be closed by this pull request

[Detection Engine] Expose EQL partial results #205277

Closed

3 tasks

dhurley14 added 4 commits January 28, 2025 10:22

Merge remote-tracking branch 'upstream/main' into eql-partial-results

46d51df

differentiates error message for shard failures for eql sequence quer…

c2a13f6

…y vs eql event query

show partial failure for non-sequence, move logic to utility function…

7cc62ad

…, add comment ref to issue to track es client upgrade

merge with main

85bdadb

rylnd approved these changes Jan 29, 2025

View reviewed changes

dhurley14 added 3 commits January 29, 2025 16:39

only call logShardFailures if shard failures are not empty

09caabe

add missing types

8a5b3d6

copy packetbeat into serverless e2e test path for security solution

0be99f5

dhurley14 requested a review from a team as a code owner January 30, 2025 02:15

don't copy all packetbeat data over

8885889

dhurley14 enabled auto-merge (squash) January 30, 2025 13:34

Merge branch 'main' into eql-partial-results

f13453d

dhurley14 disabled auto-merge January 30, 2025 17:40

dhurley14 added 2 commits January 30, 2025 13:18

update error message when sequence query has shard failures

0a8e435

Merge branch 'eql-partial-results' of github.com:dhurley14/kibana int…

8747b21

…o eql-partial-results

dhurley14 enabled auto-merge (squash) January 30, 2025 18:20

capitalize T

d2f90b4

dhurley14 merged commit 4419390 into elastic:main Jan 30, 2025
9 checks passed

kibanamachine mentioned this pull request Jan 30, 2025

[8.x] [Security Solution] [Detection Engine] Logs shard failures for eql event queries on rule details page and in event log (#207396) #209019

Merged

dhurley14 mentioned this pull request Feb 26, 2025

[REQUEST]: EQL event queries allow for partial results elastic/docs-content#610

Closed

marshallmain mentioned this pull request Mar 7, 2025

[9.0] [Security Solution] [Detection Engine] Logs shard failures for eql event queries on rule details page and in event log (#207396) #213616

Merged

dhurley14 self-assigned this Mar 26, 2025

nkhristinin mentioned this pull request Jun 5, 2025

[Detection Engine][ES|QL] - ES|QL rule support for partial results #211622

Closed

2 tasks


		const { events, sequences } = response.hits;

		if (shardFailures) {

Conversation

dhurley14 commented Jan 21, 2025 • edited by kibanamachine Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Checklist

Uh oh!

elasticmachine commented Jan 22, 2025

Uh oh!

Uh oh!

rylnd Jan 23, 2025

Choose a reason for hiding this comment

Uh oh!

dhurley14 Jan 24, 2025

Choose a reason for hiding this comment

Uh oh!

dhurley14 Jan 24, 2025

Choose a reason for hiding this comment

Uh oh!

dhurley14 Jan 29, 2025

Choose a reason for hiding this comment

Uh oh!

yctercero Jan 29, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

yctercero Jan 27, 2025

Choose a reason for hiding this comment

Uh oh!

rylnd left a comment

Choose a reason for hiding this comment

Uh oh!

rylnd Jan 29, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

approksiu commented Jan 30, 2025

Uh oh!

dhurley14 commented Jan 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dhurley14 commented Jan 30, 2025

Uh oh!

elasticmachine commented Jan 30, 2025

⏳ Build in-progress, with failures

Failed CI Steps

History

Uh oh!

Uh oh!

kibanamachine commented Jan 30, 2025

Uh oh!

kibanamachine commented Jan 30, 2025

💚 All backports created successfully

Questions ?

Uh oh!

marshallmain commented Mar 7, 2025

💚 All backports created successfully

Questions ?

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

dhurley14 commented Jan 21, 2025 •

edited by kibanamachine

Loading

dhurley14 commented Jan 30, 2025 •

edited

Loading