[Security Solution] Add EQL query editable component with EQL options fields #199115

maximpn · 2024-11-06T12:19:41Z

Partially addresses: #171520

Summary

This PR adds is built on top of #193828 and #196948 and adds an EQL Query editable component with EQL Options fields (event_category_override, timestamp_field and tiebreaker_field) for Three Way Diff tab's final edit side of the upgrade prebuilt rule workflow.

Details

This PR make a set of changes to make existing EQL Query bar component easily reusable and type safe when used in forms. In particular the following was done

EQL query bar was wrapped in EqlQueryEdit component with UseField inside. It helps to make it type safe avoiding issues like passing invalid types to EqlQueryBar. UseField types component properties as Record<string, any> so potentially any refactoring can break some functionality. For example code in Timeline passes DataViewSpec where DataViewBase is expected while these two types aren't fully compatible.
Validation was added directly to EqlQueryEdit. Passing field configuration to UseField rewrites field configuration defined in from schema. It leads to cases when validation is defined in both form schema and as a field configuration for UseFields. Additionally we can reduce reusing complexity by incapsulating absolutely required validation in EqlQueryEdit component.
Empty string tiebreakerField was removed in Timelines. tiebreakerField is part of EQL options used for EQL validation. EQL validation endpoint /internal/search/eql returns an error when an empty string provided for tiebreakerField. This problem didn't surface earlier since It looks like EQL options weren't provided correctly before this PR. Timeline EQL validation requests were sent without EQL options.

How to test

The simplest way to test is via patching installed prebuilt rules via Rule Patch API. Please follow steps below

Run Kibana locally
Install an EQL prebuilt rule, e.g. Potential Code Execution via Postgresql with rule_id 2a692072-d78d-42f3-a48a-775677d79c4e
Patch the installed rule by running a query below

curl -X PATCH --user elastic:changeme  -H 'Content-Type: application/json' -H 'kbn-xsrf: 123' -H "elastic-api-version: 2023-10-31" -d '{"rule_id":"2a692072-d78d-42f3-a48a-775677d79c4e","version":1,"query":"process where process.name == \"cmd.exe\"","language":"eql","event_category_override": "test","timestamp_field": "@timestamp","tiebreaker_field": "tiebreaker"}' http://localhost:5601/kbn/api/detection_engine/rules

Open Detection Rules (SIEM) Page -> Rule Updates -> click on Potential Code Execution via Postgresql rule -> expand EQL Query to see EQL Query -> press Edit button

Screenshots

EQL Query in Prebuilt Rules Update workflow

event_category_override + tiebreaker_field + timestamp_field (aka EQL options) in Prebuilt Rules Update workflow

Examples of invalid EQL

maximpn · 2024-11-07T07:32:26Z

x-pack/plugins/security_solution/public/timelines/store/defaults.ts

@@ -33,7 +33,6 @@ export const timelineDefaults: SubsetTimelineModel &
  description: '',
  eqlOptions: {
    eventCategoryField: 'event.category',
-    tiebreakerField: '',


tiebreakerField as an empty doesn't look correct for the EQL validator. It sends requests to /internal/search/eql to validate the request. It returns an error when a tiebreaker field has an empty string.

Additionally it looks like EQL options weren't provided correctly before this PR. Timeline EQL validation requests were sent without EQL options.

elasticmachine · 2024-11-07T07:49:48Z

Pinging @elastic/security-detections-response (Team:Detections and Resp)

elasticmachine · 2024-11-07T07:49:49Z

Pinging @elastic/security-solution (Team: SecuritySolution)

elasticmachine · 2024-11-07T07:49:51Z

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

yctercero · 2024-11-11T07:25:14Z

Thanks for all the work here! Pulled down and tested in rule creation/edit flow. So far LGTM. Confirmed that logic around disabling suppression for sequence queries remains. cc @dhurley14

nikitaindik · 2024-11-11T15:44:59Z

...t/components/rule_details/three_way_diff/final_edit/fields/eql_query/eql_query_edit_form.tsx

+  QueryLanguageEnum,
+} from '../../../../../../../../../common/api/detection_engine';
+import { EqlQueryEditAdapter } from './eql_query_edit_adapter';
+// import {


Commented out code. Can be removed.

nikitaindik · 2024-11-11T15:45:05Z

...t/components/rule_details/three_way_diff/final_edit/fields/eql_query/eql_query_edit_form.tsx

+  eqlQuery: {
+    validations: [
+      // {
+      //   validator: debounceAsync(eqlValidator, 300),


Commented out code. Can be removed.

nikitaindik · 2024-11-11T15:46:02Z

...t/components/rule_details/three_way_diff/final_edit/fields/eql_query/eql_query_edit_form.tsx

+  );
+}
+
+const kqlQuerySchema = {


Since there are no validations we can probably remove the schema

RuleFieldEditFormWrapper's ruleFieldFormSchema property is required. We still should pass an empty object. So removing it completely it's possible.

Got it. Then let's rename it to eqlQuerySchema.

nikitaindik · 2024-11-11T15:48:29Z

For some reason getting an EQL validation error in the upgrade flyout, but on the the Rule Editing page.

The error is shown right away, after opening the edit mode. Prevents me from saving.

EQL query I used

api where host.os.type == "win" and 
  process.Ext.api.name in ("OpenProcess", "OpenThread") and Target.process.name : "lsass.exe" and 
  not 
  (
    process.executable : (
        "?:\\ProgramData\\GetSupportService*\\Updates\\Update_*.exe",
        "?:\\ProgramData\\Microsoft\\Windows Defender\\Platform\\*\\MsMpEng.exe",
        "?:\\Program Files (x86)\\Asiainfo Security\\OfficeScan Client\\NTRTScan.exe",
        "?:\\Program Files (x86)\\Blackpoint\\SnapAgent\\SnapAgent.exe",
        "?:\\Program Files (x86)\\CheckPoint\\Endpoint Security\\EFR\\EFRService.exe",
        "?:\\Program Files (x86)\\CyberCNSAgent\\osqueryi.exe",
        "?:\\Program Files (x86)\\cisco\\cisco anyconnect secure mobility client\\vpnagent.exe",
        "?:\\Program Files (x86)\\cisco\\cisco anyconnect secure mobility client\\aciseagent.exe",
        "?:\\Program Files (x86)\\cisco\\cisco anyconnect secure mobility client\\vpndownloader.exe",
        "?:\\Program Files (x86)\\eScan\\reload.exe",
        "?:\\Program Files (x86)\\Google\\Update\\GoogleUpdate.exe",
        "?:\\Program Files (x86)\\Kaspersky Lab\\*\\avp.exe",
        "?:\\Program Files (x86)\\microsoft intune management extension\\microsoft.management.services.intunewindowsagent.exe",
        "?:\\Program Files (x86)\\N-able Technologies\\Reactive\\bin\\NableReactiveManagement.exe",
        "?:\\Program Files (x86)\\N-able Technologies\\Windows Agent\\bin\\agent.exe",
        "?:\\Program Files (x86)\\Tanium\\Tanium Client\\TaniumClient.exe",
        "?:\\Program Files (x86)\\Trend Micro\\*\\CCSF\\TmCCSF.exe",
        "?:\\Program Files (x86)\\Trend Micro\\Security Agent\\TMASutility.exe",
        "?:\\Program Files*\\Windows Defender\\MsMpEng.exe",
        "?:\\Program Files\\Bitdefender\\Endpoint Security\\EPSecurityService.exe",
        "?:\\Program Files\\Cisco\\AMP\\*\\sfc.exe",
        "?:\\Program Files\\Common Files\\McAfee\\AVSolution\\mcshield.exe",
        "?:\\Program Files\\EA\\AC\\EAAntiCheat.GameService.exe",
        "?:\\Program Files\\Elastic\\Agent\\data\\elastic-agent-*\\components\\agentbeat.exe",
        "?:\\Program Files\\Elastic\\Agent\\data\\elastic-agent-*\\components\\metricbeat.exe",
        "?:\\Program Files\\Elastic\\Agent\\data\\elastic-agent-*\\components\\osqueryd.exe",
        "?:\\Program Files\\Elastic\\Agent\\data\\elastic-agent-*\\components\\packetbeat.exe",
        "?:\\Program Files\\ESET\\ESET Security\\ekrn.exe",
        "?:\\Program Files\\Fortinet\\FortiClient\\FortiProxy.exe",
        "?:\\Program Files\\Fortinet\\FortiClient\\FortiSSLVPNdaemon.exe",
        "?:\\Program Files\\Goverlan Inc\\GoverlanAgent\\GovAgentx64.exe",
        "?:\\Program Files\\Huntress\\HuntressAgent.exe",
        "?:\\Program Files\\LogicMonitor\\Agent\\bin\\sbshutdown.exe",
        "?:\\Program Files\\Malwarebytes\\Anti-Malware\\MBAMService.exe",
        "?:\\Program Files\\Microsoft Monitoring Agent\\Agent\\Health Service State\\*\\pmfexe.exe", 
        "?:\\Program Files\\Microsoft Security Client\\MsMpEng.exe",
        "?:\\Program Files\\Qualys\\QualysAgent\\QualysAgent.exe",
        "?:\\Program Files\\smart-x\\controlupagent\\version*\\cuagent.exe",
        "?:\\Program Files\\TDAgent\\ossec-agent\\ossec-agent.exe",
        "?:\\Program Files\\Topaz OFD\\Warsaw\\core.exe",
        "?:\\Program Files\\Trend Micro\\Deep Security Agent\\netagent\\tm_netagent.exe",
        "?:\\Program Files\\VMware\\VMware Tools\\vmtoolsd.exe",
        "?:\\Program Files\\Windows Defender Advanced Threat Protection\\MsSense.exe",
        "?:\\Program Files\\Wise\\Wise Memory Optimizer\\WiseMemoryOptimzer.exe",
        "?:\\Windows\\AdminArsenal\\PDQDeployRunner\\*\\exec\\Sysmon64.exe",
        "?:\\Windows\\Sysmon.exe",
        "?:\\Windows\\Sysmon64.exe",
        "?:\\Windows\\System32\\csrss.exe",
        "?:\\Windows\\System32\\MRT.exe",
        "?:\\Windows\\System32\\msiexec.exe",
        "?:\\Windows\\System32\\taskhostw.exe",
        "?:\\Windows\\System32\\RtkAudUService64.exe",
        "?:\\Windows\\System32\\wbem\\WmiPrvSE.exe",
        "?:\\Windows\\SysWOW64\\wbem\\WmiPrvSE.exe",
        "?:\\Windows\\tenable_mw_scan_142a90001fb65e0beb1751cc8c63edd0.exe"
    ) and not ?process.code_signature.trusted == false
  )

maximpn · 2024-11-12T10:28:48Z

Hi @nkhristinin, thanks for testing the PR.

Indeed event_category_override, tiebreaker_field and timestamp_field (aka EQL options) aren't shown as a part of EQL Query bar. These fields are displayed as separate fields (see screenshots in the PR description). This is intentional to make changes to these fields prominent for users.

Also, if I change tiebreakerField field in my rule update request I didn't see it in diff.

Could you elaborate on that? A screenshot or a video recording would be nice.

nkhristinin · 2024-11-12T12:10:42Z

Could you elaborate on that? A screenshot or a video recording would be nice.

Finally was able to see that in diff, thanks!

nikitaindik · 2024-11-12T12:28:02Z

...lution/public/detection_engine/rule_creation_ui/components/eql_query_edit/eql_query_edit.tsx

+  disabled,
+  onEqlOptionsChange,
+  onValidityChange,
+  onValiditingChange,


Looks like there's a typo. Did you mean onIsValidatingChange?

Thanks for noticing. I copies this prop name from the existing EqlQueryBar component but it definitely makes sense to fix this typo.

nikitaindik

Thanks, @maximpn! I have manually tested the EQL field in both the flyout and the Rule Creation and Editing pages. I also re-tested the query bar validation with other rule types since it was affected by this PR.

I haven't found any additional issues except the one I mentioned in #199115 (comment). As we decided, it should be addressed separately after discussing it with the product.

also left a few minor comments about the code. Overall, the PR looks good to me now.

PhilippeOberti

The code LGTM and I did some smoke testing and everything seems to be working normally.
Though I'm a little worried about the changes made to Timeline. I don't have enough knowledge about how things work internally related to this tiebreakerField (I don't even what it is or what it does).
I would feel more comfortable if someone with more experience (@michaelolo24 @kqualters-elastic @janmonschke) was having a second look at this PR before approving.

elasticmachine · 2024-11-13T10:19:31Z

💚 Build Succeeded

Buildkite Build
Commit: 1156ef3

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`securitySolution`	6200	6220	+20

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`securitySolution`	13.4MB	13.4MB	+9.0KB

Unknown metric groups

References to deprecated APIs

id	before	after	diff
`securitySolution`	448	447	-1

History

💛 Build #250531 was flaky 7408045
💔 Build #250523 failed e77ca51
💔 Build #250510 failed 9e6c8d1
💔 Build #250351 failed b2e14ff
💛 Build #249452 was flaky db856ba
💔 Build #249385 failed db856ba

cc @maximpn

maximpn self-assigned this Nov 6, 2024

maximpn requested a review from nikitaindik November 6, 2024 14:28

maximpn commented Nov 7, 2024

View reviewed changes

maximpn marked this pull request as ready for review November 7, 2024 07:49

maximpn requested review from a team as code owners November 7, 2024 07:49

maximpn requested a review from nkhristinin November 7, 2024 07:49

maximpn mentioned this pull request Nov 7, 2024

[Security Solution] Rework Update flyout to display all field updates and build Three-Way-Diff field component #171520

Open

58 tasks

banderror added backport:version Backport to applied version labels and removed backport:prev-minor Backport to (8.x) the previous minor version (i.e. one version back from main) labels Nov 7, 2024

maximpn force-pushed the add-eql-query-edit-component branch from 2291ad0 to db856ba Compare November 7, 2024 21:15

nikitaindik reviewed Nov 11, 2024

View reviewed changes

maximpn force-pushed the add-eql-query-edit-component branch from 326eeb0 to 64b371f Compare November 11, 2024 17:00

nkhristinin self-requested a review November 12, 2024 12:10

nkhristinin approved these changes Nov 12, 2024

View reviewed changes

nikitaindik reviewed Nov 12, 2024

View reviewed changes

nikitaindik approved these changes Nov 12, 2024

View reviewed changes

maximpn force-pushed the add-eql-query-edit-component branch 4 times, most recently from e77ca51 to 7408045 Compare November 12, 2024 17:33

PhilippeOberti reviewed Nov 12, 2024

View reviewed changes

maximpn added 17 commits November 13, 2024 09:26

add eql query editable component

23e55fb

wrap EqlQueryBar into EqlQueryEdit component with reasonable defaults

6c1c4e1

add eql options fields

17533a2

remove unused translation keys

a15cbff

properly run first EQL Query validation in Three Way Diff

d9e1cbd

shorten long paths

d9a61c0

remove empty tiebreakerField's default value

66557e1

remove unused EQL validation

fca784c

expose showEqlSizeOption property

3f97367

expose showFilterBar prop

76d3b79

fix unit tests

5c623a0

remove commented code

9ff7ce4

use comma delimiter while joining index patterns

2b01aa7

fix a typo

408cbb1

add assertUnreachable for rules having all editable fields implemented

2144b10

fix post-rebase issues

a81d091

add fieldsToValidateOnChange to EqlQueryEdit

1156ef3

maximpn force-pushed the add-eql-query-edit-component branch from 7408045 to 1156ef3 Compare November 13, 2024 08:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Solution] Add EQL query editable component with EQL options fields #199115

[Security Solution] Add EQL query editable component with EQL options fields #199115

maximpn commented Nov 6, 2024 •

edited

Loading

maximpn Nov 7, 2024

elasticmachine commented Nov 7, 2024

elasticmachine commented Nov 7, 2024

elasticmachine commented Nov 7, 2024

yctercero commented Nov 11, 2024

nikitaindik Nov 11, 2024

nikitaindik Nov 11, 2024

nikitaindik Nov 11, 2024

maximpn Nov 11, 2024

nikitaindik Nov 12, 2024

nikitaindik commented Nov 11, 2024 •

edited

Loading

maximpn commented Nov 12, 2024

nkhristinin commented Nov 12, 2024

nikitaindik Nov 12, 2024

maximpn Nov 12, 2024

nikitaindik left a comment

PhilippeOberti left a comment

elasticmachine commented Nov 13, 2024

References to deprecated APIs

[Security Solution] Add EQL query editable component with EQL options fields #199115

Are you sure you want to change the base?

[Security Solution] Add EQL query editable component with EQL options fields #199115

Conversation

maximpn commented Nov 6, 2024 • edited Loading

Summary

Details

How to test

Screenshots

Choose a reason for hiding this comment

elasticmachine commented Nov 7, 2024

elasticmachine commented Nov 7, 2024

elasticmachine commented Nov 7, 2024

yctercero commented Nov 11, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nikitaindik commented Nov 11, 2024 • edited Loading

maximpn commented Nov 12, 2024

nkhristinin commented Nov 12, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nikitaindik left a comment

Choose a reason for hiding this comment

PhilippeOberti left a comment

Choose a reason for hiding this comment

elasticmachine commented Nov 13, 2024

💚 Build Succeeded

Metrics [docs]

Module Count

Async chunks

References to deprecated APIs

History

maximpn commented Nov 6, 2024 •

edited

Loading

nikitaindik commented Nov 11, 2024 •

edited

Loading