Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Security Solution] Implement query fields diff algorithms #187658

Closed
8 tasks done
Tracked by #174168
banderror opened this issue Jul 5, 2024 · 4 comments
Closed
8 tasks done
Tracked by #174168

[Security Solution] Implement query fields diff algorithms #187658

banderror opened this issue Jul 5, 2024 · 4 comments
Assignees
Labels
8.16 candidate enhancement New value added to drive a business result Feature:Prebuilt Detection Rules Security Solution Prebuilt Detection Rules Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v8.16.0

Comments

@banderror
Copy link
Contributor

banderror commented Jul 5, 2024

Epics: https://github.com/elastic/security-team/issues/1974 (internal), #174168

Summary

Implement algorithms for diffing and merging changes in RuleKqlQuery, RuleEqlQuery, and RuleEsqlQuery types of fields. It should be applied to:

Context from the Rule Customization RFC:

To do

@banderror banderror added triage_needed Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:Detection Rule Management Security Detection Rule Management Team Feature:Prebuilt Detection Rules Security Solution Prebuilt Detection Rules labels Jul 5, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detections-response (Team:Detections and Resp)

@banderror banderror changed the title [Security Solution] Implement query fields diff algorithm (DRAFT) [Security Solution] Implement query fields diff algorithm Jul 5, 2024
@banderror banderror changed the title [Security Solution] Implement query fields diff algorithm [Security Solution] Implement query fields diff algorithms Jul 5, 2024
@banderror banderror added the enhancement New value added to drive a business result label Jul 5, 2024
dplumlee added a commit that referenced this issue Sep 9, 2024
…elds (#190179)

## Summary

Related ticket: #187658

Adds diff algorithm for all the grouped `query` fields we have in the
`DiffableRule` type and prebuilt rule upgrade workflow. These include
`kql_query` (both inline and saved query), `eql_query`, and
`esql_query`. Also adds unit tests for all three algorithms.


### Checklist

Delete any items that are not applicable to this PR.

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios


### For maintainers

- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
dplumlee added a commit that referenced this issue Sep 16, 2024
)

## Summary

Related ticket: #187658

Adds test plan for diff algorithm for arrays of scalar values
implemented here: #190179


### For maintainers

- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Sep 16, 2024
…tic#192529)

## Summary

Related ticket: elastic#187658

Adds test plan for diff algorithm for arrays of scalar values
implemented here: elastic#190179

### For maintainers

- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)

(cherry picked from commit 6680f35)
kibanamachine referenced this issue Sep 16, 2024
… algorithm (#192529) (#193062)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[Security Solution] Test plan for `query` fields diff
algorithm (#192529)](#192529)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Davis
Plumlee","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-09-16T17:19:39Z","message":"[Security
Solution] Test plan for `query` fields diff algorithm (#192529)\n\n##
Summary\r\n\r\nRelated ticket:
https://github.com/elastic/kibana/issues/187658\r\n\r\nAdds test plan
for diff algorithm for arrays of scalar values\r\nimplemented here:
https://github.com/elastic/kibana/pull/190179\r\n\r\n\r\n### For
maintainers\r\n\r\n- [ ] This was checked for breaking API changes and
was
[labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"6680f35ef94286d34dd5c56e28575b31eab70836","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","test-plan","v9.0.0","Team:Detections
and Resp","Team: SecuritySolution","Team:Detection Rule
Management","Feature:Prebuilt Detection
Rules","v8.16.0"],"title":"[Security Solution] Test plan for `query`
fields diff
algorithm","number":192529,"url":"https://github.com/elastic/kibana/pull/192529","mergeCommit":{"message":"[Security
Solution] Test plan for `query` fields diff algorithm (#192529)\n\n##
Summary\r\n\r\nRelated ticket:
https://github.com/elastic/kibana/issues/187658\r\n\r\nAdds test plan
for diff algorithm for arrays of scalar values\r\nimplemented here:
https://github.com/elastic/kibana/pull/190179\r\n\r\n\r\n### For
maintainers\r\n\r\n- [ ] This was checked for breaking API changes and
was
[labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"6680f35ef94286d34dd5c56e28575b31eab70836"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/192529","number":192529,"mergeCommit":{"message":"[Security
Solution] Test plan for `query` fields diff algorithm (#192529)\n\n##
Summary\r\n\r\nRelated ticket:
https://github.com/elastic/kibana/issues/187658\r\n\r\nAdds test plan
for diff algorithm for arrays of scalar values\r\nimplemented here:
https://github.com/elastic/kibana/pull/190179\r\n\r\n\r\n### For
maintainers\r\n\r\n- [ ] This was checked for breaking API changes and
was
[labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"6680f35ef94286d34dd5c56e28575b31eab70836"}},{"branch":"8.x","label":"v8.16.0","branchLabelMappingKey":"^v8.16.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

Co-authored-by: Davis Plumlee <[email protected]>
dplumlee added a commit that referenced this issue Sep 16, 2024
…92655)

## Summary

Completes #187658


Switches `kql_query`, `eql_query`, and `esql_query` fields to use the
implemented diff algorithms assigned to them in
#190179


Adds integration tests in accordance to
#192529 for the `upgrade/_review`
API endpoint for the `query` field diff algorithms.

### Checklist

Delete any items that are not applicable to this PR.

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed


### For maintainers

- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Sep 17, 2024
…astic#192655)

## Summary

Completes elastic#187658

Switches `kql_query`, `eql_query`, and `esql_query` fields to use the
implemented diff algorithms assigned to them in
elastic#190179

Adds integration tests in accordance to
elastic#192529 for the `upgrade/_review`
API endpoint for the `query` field diff algorithms.

### Checklist

Delete any items that are not applicable to this PR.

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed

### For maintainers

- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)

(cherry picked from commit ceb1b1a)
kibanamachine referenced this issue Sep 17, 2024
…f algorithms (#192655) (#193108)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[Security Solution] Integration tests for &#x60;query&#x60; diff
algorithms (#192655)](#192655)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Davis
Plumlee","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-09-16T23:58:55Z","message":"[Security
Solution] Integration tests for `query` diff algorithms (#192655)\n\n##
Summary\r\n\r\nCompletes
https://github.com/elastic/kibana/issues/187658\r\n\r\n\r\nSwitches
`kql_query`, `eql_query`, and `esql_query` fields to use
the\r\nimplemented diff algorithms assigned to them
in\r\nhttps://github.com//pull/190179\r\n\r\n\r\nAdds
integration tests in accordance
to\r\nhttps://github.com//pull/192529 for the
`upgrade/_review`\r\nAPI endpoint for the `query` field diff
algorithms.\r\n\r\n### Checklist\r\n\r\nDelete any items that are not
applicable to this PR.\r\n\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common scenarios\r\n- [x] [Flaky
Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)
was\r\nused on any tests changed\r\n\r\n\r\n### For maintainers\r\n\r\n-
[ ] This was checked for breaking API changes and was
[labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"ceb1b1a4bf253ac94f9ba0ba649e9a4908a76c51","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["test","release_note:skip","v9.0.0","Team:Detections
and Resp","Team: SecuritySolution","Team:Detection Rule
Management","Feature:Prebuilt Detection
Rules","v8.16.0"],"title":"[Security Solution] Integration tests for
`query` diff
algorithms","number":192655,"url":"https://github.com/elastic/kibana/pull/192655","mergeCommit":{"message":"[Security
Solution] Integration tests for `query` diff algorithms (#192655)\n\n##
Summary\r\n\r\nCompletes
https://github.com/elastic/kibana/issues/187658\r\n\r\n\r\nSwitches
`kql_query`, `eql_query`, and `esql_query` fields to use
the\r\nimplemented diff algorithms assigned to them
in\r\nhttps://github.com//pull/190179\r\n\r\n\r\nAdds
integration tests in accordance
to\r\nhttps://github.com//pull/192529 for the
`upgrade/_review`\r\nAPI endpoint for the `query` field diff
algorithms.\r\n\r\n### Checklist\r\n\r\nDelete any items that are not
applicable to this PR.\r\n\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common scenarios\r\n- [x] [Flaky
Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)
was\r\nused on any tests changed\r\n\r\n\r\n### For maintainers\r\n\r\n-
[ ] This was checked for breaking API changes and was
[labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"ceb1b1a4bf253ac94f9ba0ba649e9a4908a76c51"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/192655","number":192655,"mergeCommit":{"message":"[Security
Solution] Integration tests for `query` diff algorithms (#192655)\n\n##
Summary\r\n\r\nCompletes
https://github.com/elastic/kibana/issues/187658\r\n\r\n\r\nSwitches
`kql_query`, `eql_query`, and `esql_query` fields to use
the\r\nimplemented diff algorithms assigned to them
in\r\nhttps://github.com//pull/190179\r\n\r\n\r\nAdds
integration tests in accordance
to\r\nhttps://github.com//pull/192529 for the
`upgrade/_review`\r\nAPI endpoint for the `query` field diff
algorithms.\r\n\r\n### Checklist\r\n\r\nDelete any items that are not
applicable to this PR.\r\n\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common scenarios\r\n- [x] [Flaky
Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)
was\r\nused on any tests changed\r\n\r\n\r\n### For maintainers\r\n\r\n-
[ ] This was checked for breaking API changes and was
[labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"ceb1b1a4bf253ac94f9ba0ba649e9a4908a76c51"}},{"branch":"8.x","label":"v8.16.0","branchLabelMappingKey":"^v8.16.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

Co-authored-by: Davis Plumlee <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
8.16 candidate enhancement New value added to drive a business result Feature:Prebuilt Detection Rules Security Solution Prebuilt Detection Rules Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v8.16.0
Projects
None yet
Development

No branches or pull requests

3 participants