[Security Solution][Detection Engine] adds DOES NOT MATCH capability to IM rule type by vitaliidm · Pull Request #227084 · elastic/kibana

vitaliidm · 2025-07-08T15:56:36Z

Summary

addresses https://github.com/elastic/security-team/issues/13022
allows selecting DOES NOT MATCH condition, it would create alert for any document that has at least one matching field and not matching. Single DOES NOT MATCH is not allowed, to prevent creation of large number of alerts. User would be able to select match first and then tune it with DOES NOT MATCH if needed
added negate: boolean value to ThreatMap entry schema to indicate this mapping should use as DOES NOT MATCH.
Field name negate was chosen as it already used in Kibana filters to indicate field does not match some value.
Enrichments for DOES NOT MATCH field are omitted
If one of the DOES NOT MATCH fields is empty, we create an alert. If both empty - no alert should be created
I fixed issue within validateCompleteThreatMatches, where False Positive alert can be created if there partial matches across AND group in multiple threats
Removed x-pack/solutions/security/packages/kbn-securitysolution-io-ts-alerting-types/src/threat_mapping/index.ts in favour of OpenAPI schema

Feature Flag

xpack.securitySolution.enableExperimental:
  - doesNotMatchForIndicatorMatchRuleEnabled

UI

Having help text under label and for the first entry, DOES NOT MATCH option should be disabled

User can add DOES NOT MATCH as second AND condition

If user deletes first MATCH we would show validation error

user cannot setup a match and not match condition for the same field, validation error

…IM rule type

…apping'

# Conflicts: # x-pack/solutions/security/plugins/security_solution/server/lib/detection_engine/rule_types/indicator_match/threat_mapping/build_threat_mapping_filter.ts # x-pack/solutions/security/plugins/security_solution/server/lib/detection_engine/rule_types/indicator_match/threat_mapping/get_signals_map_from_threat_index.ts

… corner case scenario

…kibana into de_9_2/im-not-matches

…/detection_engine/rule_types/indicator_match/threat_mapping/utils.test.ts Co-authored-by: Marshall Main <55718608+marshallmain@users.noreply.github.com>

vitaliidm · 2025-08-05T11:39:19Z

@marshallmain, thanks for the review. I have addressed the comments

…no-cache --fix'

…mon/components/threat_match/translations.ts Co-authored-by: Marshall Main <55718608+marshallmain@users.noreply.github.com>

nastasha-solomon

Just the one small tweak. Otherwise, lgtm for now!

nastasha-solomon · 2025-08-05T13:22:31Z

...on/components/threat_match_mapping_edit/validators/threat_match_mapping_validator_factory.ts

+        message: i18n.translate(
+          'xpack.securitySolution.detectionEngine.ruleManagement.threatMappingField.singleNotMatchClauseError',
+          {
+            defaultMessage: 'Conditions in AND clauses must have at least one MATCHES entry.',


Suggested change

defaultMessage: 'Conditions in AND clauses must have at least one MATCHES entry.',

defaultMessage: 'Conditions with AND clauses must have at least one MATCHES entry.',

…kibana into de_9_2/im-not-matches

marshallmain

LGTM, thanks for addressing my comments 🚀

Understanding the changes to validateCompleteThreatMatches was by far the hardest part, and I hope we can drastically simplify that logic by adjusting the way we name query clauses. The remaining unresolved comments are for future (soon) work.

marshallmain · 2025-08-05T18:23:22Z

...tection_engine/rule_types/indicator_match/threat_mapping/validate_complete_threat_matches.ts

-        return false;
-      });
+    Object.values(threatQueriesMap).forEach((threatQueriesPerId) => {
+      const matchedThreatQueriesForAndGroup: ThreatMatchNamedQuery[] = [];


This const can go inside the inner loop as well?

…kibana into de_9_2/im-not-matches

elasticmachine · 2025-08-06T11:34:54Z

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`@kbn/securitysolution-io-ts-alerting-types`	125	87	-38

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`securitySolution`	10.3MB	10.3MB	+1.8KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`securitySolution`	94.5KB	94.5KB	+45.0B

Unknown metric groups

API count

id	before	after	diff
`@kbn/securitysolution-io-ts-alerting-types`	147	109	-38

ESLint disabled in files

id	before	after	diff
`@kbn/securitysolution-io-ts-alerting-types`	9	8	-1

Total ESLint disabled count

id	before	after	diff
`@kbn/securitysolution-io-ts-alerting-types`	9	8	-1

History

💔 Build #326826 failed e6d68c9
💚 Build #326520 succeeded e66f486
💔 Build #326432 failed 52b79da
💚 Build #326026 succeeded bbdd39a
💔 Build #325992 failed 4b5a4d2

cc @vitaliidm

…to IM rule type (elastic#227084) ## Summary - addresses elastic/security-team#13022 - allows selecting DOES NOT MATCH condition, it would create alert for any document that has at least one matching field and not matching. Single DOES NOT MATCH is **not allowed**, to prevent creation of large number of alerts. User would be able to select match first and then tune it with DOES NOT MATCH if needed - added `negate: boolean` value to ThreatMap entry schema to indicate this mapping should use as DOES NOT MATCH. Field name `negate` was chosen as it already used in Kibana filters to indicate field does not match some value. - Enrichments for DOES NOT MATCH field are omitted - If one of the DOES NOT MATCH fields is empty, we create an alert. If both empty - no alert should be created - I fixed issue within `validateCompleteThreatMatches`, where False Positive alert can be created if there partial matches across AND group in multiple threats - Removed `x-pack/solutions/security/packages/kbn-securitysolution-io-ts-alerting-types/src/threat_mapping/index.ts` in favour of OpenAPI schema ### Feature Flag ```yml xpack.securitySolution.enableExperimental: - doesNotMatchForIndicatorMatchRuleEnabled ``` ### UI 1. Having help text under label and for the first entry, DOES NOT MATCH option should be disabled <img width="1112" height="214" alt="Screenshot 2025-07-28 at 10 47 26" src="https://github.com/user-attachments/assets/740c8170-45ae-4773-a93b-ac2e0bf12c9c" /> 2. User can add DOES NOT MATCH as second AND condition <img width="1050" height="320" alt="Screenshot 2025-07-28 at 10 47 08" src="https://github.com/user-attachments/assets/56b1f26e-6951-47e7-b7a6-4ff62f5b44a5" /> 3. If user deletes first MATCH we would show validation error <img width="1040" height="241" alt="Screenshot 2025-07-28 at 10 47 43" src="https://github.com/user-attachments/assets/a61274b3-087d-450d-97cc-5ac2b40079db" /> 4. user cannot setup a match and not match condition for the same field, validation error <img width="1064" height="282" alt="Screenshot 2025-07-28 at 10 48 02" src="https://github.com/user-attachments/assets/0ae790e1-7e3d-4fb7-8550-fb3d92b9d678" /> ### Flaky test runner Cypress: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8886 FTR: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8885 ### Docs issue - elastic/docs-content#2295 ### Test plan - elastic/security-team#13367 --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: florent-leborgne <florent.leborgne@elastic.co> Co-authored-by: Marshall Main <55718608+marshallmain@users.noreply.github.com>

…H operator (#232996) ## Summary - follow up for #227084 - enables FF `doesNotMatchForIndicatorMatchRuleEnabled`

…H operator (elastic#232996) ## Summary - follow up for elastic#227084 - enables FF `doesNotMatchForIndicatorMatchRuleEnabled`

…imental feature flag (#249550) ## Summary - removes Indicator match **DOES NOT MATCH** experimental feature flag`doesNotMatchForIndicatorMatchRuleEnabled` and corresponding feature tour - feature flag was introduced in #227084

…imental feature flag (elastic#249550) ## Summary - removes Indicator match **DOES NOT MATCH** experimental feature flag`doesNotMatchForIndicatorMatchRuleEnabled` and corresponding feature tour - feature flag was introduced in elastic#227084 (cherry picked from commit 7a7da04) # Conflicts: # x-pack/solutions/security/test/security_solution_api_integration/config/ess/config.base.ts # x-pack/solutions/security/test/security_solution_api_integration/config/serverless/config.base.ts # x-pack/solutions/security/test/security_solution_cypress/config.ts # x-pack/solutions/security/test/security_solution_cypress/cypress/e2e/entity_analytics/threat_hunting/threat_hunting_page.cy.ts # x-pack/solutions/security/test/security_solution_cypress/serverless_config.ts

… experimental feature flag (#249550) (#250888) # Backport This will backport the following commits from `main` to `9.3`: - [[Security Solution][Detection Engine] removes IM DOES NOT MATCH experimental feature flag (#249550)](#249550)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)

[Security Solution][Detection Engine] adds NOT MATCHES capability to …

10d574b

…IM rule type

vitaliidm self-assigned this Jul 8, 2025

vitaliidm and others added 23 commits July 9, 2025 12:26

[Security Solution][Detection Engine] changes

6133711

[Security Solution][Detection Engine] reanme

acbc430

Merge branch 'main' into de_9_2/im-not-matches

7be7308

[Security Solution][Detection Engine] adds basic functionality

f0584c6

[Security Solution][Detection Engine] adjustments

65c8a0a

[CI] Auto-commit changed files from 'node scripts/styled_components_m…

0dc84dd

…apping'

[CI] Auto-commit changed files from 'yarn openapi:bundle'

963ac33

Merge branch 'main' into de_9_2/im-not-matches

b38a11f

[CI] Auto-commit changed files from 'make api-docs'

c1ed0da

Merge branch 'main' into de_9_2/im-not-matches

cd9550c

[Security Solution][Detection Engine] resolve merge conflicts

94e3799

[Security Solution][Detection Engine] add very complicated OR fix for…

efbfd1e

… corner case scenario

[Security Solution][Detection Engine] add tests

794daea

[Security Solution][Detection Engine] fix more tests

d3e9f9a

Merge branch 'main' into de_9_2/im-not-matches

fd80994

[Security Solution][Detection Engine] fix ine more test

520002b

Merge branch 'main' into de_9_2/im-not-matches

72a11e1

[Security Solution][Detection Engine]

feff854

[Security Solution][Detection Engine] fix another test

cdfee8c

Merge branch 'main' into de_9_2/im-not-matches

af9ae5a

[Security Solution][Detection Engine] typings

a7efe30

Merge branch 'de_9_2/im-not-matches' of https://github.com/vitaliidm/…

e8614b4

…kibana into de_9_2/im-not-matches

vitaliidm and others added 2 commits August 5, 2025 12:12

[Security Solution][Detection Engine] invert loops

3906bc5

Update x-pack/solutions/security/plugins/security_solution/server/lib…

8e7885d

…/detection_engine/rule_types/indicator_match/threat_mapping/utils.test.ts Co-authored-by: Marshall Main <55718608+marshallmain@users.noreply.github.com>

[CI] Auto-commit changed files from 'node scripts/eslint_all_files --…

24aedc2

…no-cache --fix'

vitaliidm requested a review from marshallmain August 5, 2025 11:39

vitaliidm and others added 2 commits August 5, 2025 12:40

Merge branch 'main' into de_9_2/im-not-matches

52b79da

Update x-pack/solutions/security/plugins/security_solution/public/com…

233e971

…mon/components/threat_match/translations.ts Co-authored-by: Marshall Main <55718608+marshallmain@users.noreply.github.com>

vitaliidm requested a review from nastasha-solomon August 5, 2025 13:09

nastasha-solomon approved these changes Aug 5, 2025

View reviewed changes

vitaliidm added 3 commits August 5, 2025 14:47

[Security Solution][Detection Engine] wording

574d6ab

Merge branch 'de_9_2/im-not-matches' of https://github.com/vitaliidm/…

6842663

…kibana into de_9_2/im-not-matches

Merge branch 'main' into de_9_2/im-not-matches

e66f486

marshallmain approved these changes Aug 5, 2025

View reviewed changes

vitaliidm added 2 commits August 6, 2025 09:42

[Security Solution][Detection Engine] loop

e35c4b7

Merge branch 'de_9_2/im-not-matches' of https://github.com/vitaliidm/…

e6d68c9

…kibana into de_9_2/im-not-matches

vitaliidm enabled auto-merge (squash) August 6, 2025 10:29

Merge branch 'main' into de_9_2/im-not-matches

17b7371

vitaliidm merged commit 36d8883 into elastic:main Aug 6, 2025
12 checks passed

wildemat mentioned this pull request Aug 7, 2025

pr 230826 #231022

Closed

10 tasks

vitaliidm mentioned this pull request Aug 26, 2025

[Security Solution][Detection Engine] enables FF for IM DOES NOT MATCH operator #232996

Merged

vitaliidm added a commit that referenced this pull request Aug 28, 2025

[Security Solution][Detection Engine] enables FF for IM DOES NOT MATC…

99984c1

…H operator (#232996) ## Summary - follow up for #227084 - enables FF `doesNotMatchForIndicatorMatchRuleEnabled`

Mikaayenson mentioned this pull request Aug 29, 2025

[FR] Add negate DOES NOT MATCH capability to IM rule type (>=9.2) elastic/detection-rules#5041

Merged

5 tasks

qn895 pushed a commit to qn895/kibana that referenced this pull request Sep 2, 2025

[Security Solution][Detection Engine] enables FF for IM DOES NOT MATC…

8ed57d0

…H operator (elastic#232996) ## Summary - follow up for elastic#227084 - enables FF `doesNotMatchForIndicatorMatchRuleEnabled`

vitaliidm mentioned this pull request Jan 27, 2026

[Security Solution][Detection Engine] removes IM DOES NOT MATCH experimental feature flag #249550

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Solution][Detection Engine] adds DOES NOT MATCH capability to IM rule type#227084

[Security Solution][Detection Engine] adds DOES NOT MATCH capability to IM rule type#227084
vitaliidm merged 86 commits intoelastic:mainfrom
vitaliidm:de_9_2/im-not-matches

vitaliidm commented Jul 8, 2025 •

edited

Loading

Uh oh!

vitaliidm commented Aug 5, 2025

Uh oh!

nastasha-solomon left a comment

Uh oh!

nastasha-solomon Aug 5, 2025

Uh oh!

marshallmain left a comment

Uh oh!

marshallmain Aug 5, 2025

Uh oh!

elasticmachine commented Aug 6, 2025 •

edited

Loading

API count

ESLint disabled in files

Total ESLint disabled count

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

	defaultMessage: 'Conditions in AND clauses must have at least one MATCHES entry.',
	defaultMessage: 'Conditions with AND clauses must have at least one MATCHES entry.',

Conversation

vitaliidm commented Jul 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Feature Flag

UI

Flaky test runner

Docs issue

Test plan

Uh oh!

vitaliidm commented Aug 5, 2025

Uh oh!

nastasha-solomon left a comment

Choose a reason for hiding this comment

Uh oh!

nastasha-solomon Aug 5, 2025

Choose a reason for hiding this comment

Uh oh!

marshallmain left a comment

Choose a reason for hiding this comment

Uh oh!

marshallmain Aug 5, 2025

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Aug 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Async chunks

Page load bundle

API count

ESLint disabled in files

Total ESLint disabled count

History

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

vitaliidm commented Jul 8, 2025 •

edited

Loading

elasticmachine commented Aug 6, 2025 •

edited

Loading