[Security Solution][Exceptions] - Tie server and client code together #70918

yctercero · 2020-07-07T07:03:01Z

Summary

This PR tries to start to tie together the server and client changes for exceptions lists.

Updates graphql types to allow UI access to a rule's exceptions_list property
Updates the exception viewer component to now dynamically take the rule exceptions_list, up until now we just had an empty array in it's place
Updates the viewer logic to check if a rule has an endpoint list associated with it. If it does, then it displays both detections and endpoint UIs (in the viewer), if it does not, then it only displays the detections UI
Updates the viewer UI to better deal with spacing when an exception list item only has one or two entries (before the and badge with the antennas was stretching passed the exception items to fill the space)
Updates the detections engine exceptions logic to fetch list items using an exception list's id as opposed to it's list_id, this now aligns with the UI using the same params on its end
Adds exception list type to information kept by the rule for exception lists
Updates the exception list type from string to endpoint | detection
Updates the exception list item type from string to simple
Adds unit tests for the detection engine server side util that fetches the exception list items

With this PR, you should now be able to use the API to create rules with exception lists and see the alerts filtered in the UI as well as seeing the exceptions show now in the exceptions viewer. See #69715 TO DO section for an explanation on the remaining updates needed to the filtering logic.

Note: This was originally #69939 , but opened new clean PR here.

How to test

To turn on lists plugin - in kibana.dev.yml

# Enable lists feature
xpack.lists.enabled: true
xpack.lists.listIndex: '.lists-yara'
xpack.lists.listItemIndex: '.items-yara'

Add export ELASTIC_XPACK_SIEM_LISTS_FEATURE=true to your bash file.

Use the scripts in x-pack/plugins/lists/server/scripts to create some sample exception lists and items. You can use the following:

If you've previously played around with lists (or if you've never, and need to create the index), run ./hard_reset.sh (this will delete any lists you've created).

Create large value list:

Create large value list ./post_list.sh
Create large value list item ./post_list_item.sh (I modified the value to be "value": "10.4.3.11")

Create exception list:

Create exception list ./post_exception_list.sh
Create exception list item ./post_exception_list_item.sh ./exception_lists/new/exception_list_item_with_list.json. This makes reference to the large value list created above.

Use the scripts in x-pack/plugins/security_solution/server/lib/detection_engine/scripts to create rule:
Before running script, you'll need to update the referenced exceptions_list id to the one you created

Run ./post_rule.sh ./rules/queries/query_with_list.json (Makes reference to the exception list created in step above)

In the Alerts table, you should see something like the following where you only see events where the event.module is suricata and source.ip is 10.4.3.11 (or whatever ip you specified).

Go to your newly created rule details and on the Exceptions tab you should see something like this:

What to test

If you follow the steps outlined above, do you see exception list items show up in the rule's details exception tab? (You should! 😬 )
Are signals being created as you expect?
- Remember that as these are exceptions, it is double negative logic
- If you aren't seeing signals created check to make sure that your exceptions aren't too narrow
- Keep in mind that the detection engine logic needs updating - see [SIEM][Detection Engine] - Update DE to work with new exceptions schema #69715 TO DO section for an explanation on the remaining updates needed to the filtering logic.
If you associate both a detection and an endpoint list, are you able to see both and toggle between the lists in the viewer?(You should! 😬 )
- Create detection type list ./post_exception_list.sh ./exception_lists/new/exception_list_detection.json
- Create detection type list items ./post_exception_list_item.sh ./exception_lists/new/exception_list_item_detection_auto_id.json - this script auto generates the item_id so you can run it as many times as you like to create multiple items
- In x-pack/plugins/security_solution/server/lib/detection_engine/scripts update /rules/patches/update_list.json to include your newly created exception list and run ./patch_rule.sh ./rules/patches/update_list.json
Using API, are you able to create an exception list with a type other than endpoint or detection? (Hopefully not! 🛑 )
Using API, are you able to create an exception list item with a type other than simple? (Hopefully not! 🛑 )
If you update the exceptions_list on the rule to just have a detections list, do you see the viewer display a modified view with no toggle options? (You should! 😬 ) Something like:

Checklist

Unit or functional tests were updated or added to match the most common scenarios
This was checked for keyboard-only and screenreader accessibility
This renders correctly on smaller devices using a responsive layout. (You can test this in your browser
This was checked for cross-browser compatibility, including a check against IE11

…d tests

…ons_ui

elasticmachine · 2020-07-07T07:03:04Z

Pinging @elastic/siem (Team:SIEM)

yctercero · 2020-07-07T07:03:44Z

x-pack/plugins/lists/server/services/exception_lists/create_exception_list_item.ts

  tags: Tags;
  tieBreaker?: string;
-  type: ExceptionListType;
+  type: ExceptionListItemType;


Before both the list and item types were just t.string so it didn't scream at us, but now that the list type and item type differ, had to update.

yctercero · 2020-07-07T07:03:57Z

x-pack/plugins/lists/server/services/exception_lists/exception_list_client_types.ts

  meta: MetaOrUndefined;
  tags: TagsOrUndefined;
-  type: ExceptionListTypeOrUndefined;
+  type: ExceptionListItemTypeOrUndefined;


Before both the list and item types were just t.string so it didn't scream at us, but now that the list type and item type differ, had to update.

yctercero · 2020-07-07T07:04:08Z

x-pack/plugins/lists/server/services/exception_lists/update_exception_list_item.ts

  tags: TagsOrUndefined;
  tieBreaker?: string;
-  type: ExceptionListTypeOrUndefined;
+  type: ExceptionListItemTypeOrUndefined;


Before both the list and item types were just t.string so it didn't scream at us, but now that the list type and item type differ, had to update.

yctercero · 2020-07-07T07:05:47Z

x-pack/plugins/lists/server/services/exception_lists/utils.ts

    tags,
    tie_breaker_id,
-    type,
+    type: exceptionListType.is(type) ? type : 'detection',


This is a result of both the list and item type being mapped to the same type property in the SO. We can discuss whether to keep this as is (@FrankHassanabad are there downsides to separating these out?) or change.

yctercero · 2020-07-07T07:06:32Z

x-pack/plugins/security_solution/common/detection_engine/build_exceptions_query.ts

  const and = getLanguageBooleanOperator({ language, value: 'and' });
  const exceptionItem = lists
-    .filter((t) => !entriesList.is(t))
+    .filter(({ type }) => type !== 'list')


Starting to try to not rely on io-ts check where I can given talk of not optimal performance.

yctercero · 2020-07-07T07:12:55Z

@rylnd this is a new PR I created off of #69939 . Per your feedback, moved to not change namespaceType to namespace_type in the UI.

…ons_ui

kibanamachine · 2020-07-07T09:01:34Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 293a12b

Build metrics

@kbn/optimizer bundle module count

id	value	diff	baseline
securitySolution	809	+24	785

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

peluja1012

Tested by integrating with the Exceptions Modal branch. Thank you the hard work you put in to tie it all together!

…elastic#70918) ## Summary This PR tries to start to tie together the server and client changes for exceptions lists. - Updates graphql types to allow UI access to a rule's `exceptions_list` property - Updates the exception viewer component to now dynamically take the rule `exceptions_list`, up until now we just had an empty array in it's place - Updates the viewer logic to check if a rule has an endpoint list associated with it. If it does, then it displays both detections and endpoint UIs (in the viewer), if it does not, then it only displays the detections UI - Updates the viewer UI to better deal with spacing when an exception list item only has one or two entries (before the and badge with the antennas was stretching passed the exception items to fill the space) - Updates the detections engine exceptions logic to fetch list items using an exception list's `id` as opposed to it's `list_id`, this now aligns with the UI using the same params on its end - Adds exception list `type` to information kept by the rule for exception lists - Updates the exception list type from `string` to `endpoint | detection` - Updates the exception list _item_ type from `string` to `simple` - Adds unit tests for the detection engine server side util that fetches the exception list items

…#70918) (#71012) ## Summary This PR tries to start to tie together the server and client changes for exceptions lists. - Updates graphql types to allow UI access to a rule's `exceptions_list` property - Updates the exception viewer component to now dynamically take the rule `exceptions_list`, up until now we just had an empty array in it's place - Updates the viewer logic to check if a rule has an endpoint list associated with it. If it does, then it displays both detections and endpoint UIs (in the viewer), if it does not, then it only displays the detections UI - Updates the viewer UI to better deal with spacing when an exception list item only has one or two entries (before the and badge with the antennas was stretching passed the exception items to fill the space) - Updates the detections engine exceptions logic to fetch list items using an exception list's `id` as opposed to it's `list_id`, this now aligns with the UI using the same params on its end - Adds exception list `type` to information kept by the rule for exception lists - Updates the exception list type from `string` to `endpoint | detection` - Updates the exception list _item_ type from `string` to `simple` - Adds unit tests for the detection engine server side util that fetches the exception list items

* master: (36 commits) fixed api url in example plugin (elastic#70934) [data.search.aggs]: Remove remaining client dependencies (elastic#70251) [Security Solution][Endpoint] Fix base64 download bug and adopt new user artifact/manifest format (elastic#70998) [Security Solution][Exceptions] - Exception Modal Part I (elastic#70639) [SIEM][Detection Engine][Lists] Adds additional data types to value based lists [SIEM][Detection Engine][Lists] Removes feature flag for lists [APM] Show license callout in ML settings (elastic#70959) Migrate service settings test to jest (elastic#70992) [APM] Add cloud attributes to data telemetry (elastic#71008) Fix breadcrumb on panels for visibility / round corners (elastic#71010) Improve search typescript (elastic#69333) [savedObjects field count] run in baseline job (elastic#70999) [Security Solution] [Timeline] Timeline manager tweaks (elastic#69988) [Endpoint] Support redirect from Policy Details to Ingest when user initiates Edit Policy from Datasource Edit page (elastic#70874) [APM] Add API tests (elastic#70740) [Security Solution][Exceptions] - Tie server and client code together (elastic#70918) [Audit Logging] Add AuditTrail service (elastic#69278) [Usage Collection] Ensure no type duplicates (elastic#70946) [Security Solution] [Timeline] Bugfix for timeline row actions disappear sometimes (elastic#70958) [CI] Add pipeline task queue framework and merge workers into one (elastic#64011) ...

elasticmachine · 2021-09-23T14:31:55Z

Pinging @elastic/security-solution (Team: SecuritySolution)

yctercero added 6 commits July 6, 2020 12:16

update graphql types to include exception lists

070b3d3

updated rule exceptions_list property to include list type and update…

3507d77

…d tests

updated lists plugin tests

48497d7

Merge branch 'master' of github.com:yctercero/kibana into tie_excepti…

a2a0a9d

…ons_ui

updated types in engine

03fc086

updated exceptions viewer and types

f4a21f0

yctercero added release_note:enhancement Team:SIEM v8.0.0 v7.9.0 labels Jul 7, 2020

yctercero requested review from a team as code owners July 7, 2020 07:03

yctercero self-assigned this Jul 7, 2020

yctercero commented Jul 7, 2020

View reviewed changes

yctercero added 2 commits July 7, 2020 03:16

Merge branch 'master' of github.com:yctercero/kibana into tie_excepti…

472730b

…ons_ui

updated scripts

293a12b

peluja1012 approved these changes Jul 7, 2020

View reviewed changes

yctercero merged commit 37c2c92 into elastic:master Jul 7, 2020

yctercero mentioned this pull request Jul 7, 2020

[7.x] [Security Solution][Exceptions] - Tie server and client code together (#70918) #71012

Merged

yctercero deleted the tie_exceptions_ui branch October 14, 2020 12:01

MindyRS added the Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. label Sep 23, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Security Solution][Exceptions] - Tie server and client code together #70918

[Security Solution][Exceptions] - Tie server and client code together #70918

Uh oh!

yctercero commented Jul 7, 2020

Uh oh!

elasticmachine commented Jul 7, 2020

Uh oh!

yctercero Jul 7, 2020

Uh oh!

yctercero Jul 7, 2020

Uh oh!

yctercero Jul 7, 2020

Uh oh!

yctercero Jul 7, 2020

Uh oh!

yctercero Jul 7, 2020

Uh oh!

yctercero commented Jul 7, 2020

Uh oh!

kibanamachine commented Jul 7, 2020

Uh oh!

peluja1012 left a comment

Uh oh!

elasticmachine commented Sep 23, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

[Security Solution][Exceptions] - Tie server and client code together #70918

[Security Solution][Exceptions] - Tie server and client code together #70918

Uh oh!

Conversation

yctercero commented Jul 7, 2020

Summary

How to test

What to test

Checklist

Uh oh!

elasticmachine commented Jul 7, 2020

Uh oh!

yctercero Jul 7, 2020

Choose a reason for hiding this comment

Uh oh!

yctercero Jul 7, 2020

Choose a reason for hiding this comment

Uh oh!

yctercero Jul 7, 2020

Choose a reason for hiding this comment

Uh oh!

yctercero Jul 7, 2020

Choose a reason for hiding this comment

Uh oh!

yctercero Jul 7, 2020

Choose a reason for hiding this comment

Uh oh!

yctercero commented Jul 7, 2020

Uh oh!

kibanamachine commented Jul 7, 2020

💚 Build Succeeded

Build metrics

@kbn/optimizer bundle module count

Uh oh!

peluja1012 left a comment

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Sep 23, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants