Skip to content

[Security Solution][Detections] Update detection alert mappings to ECS v1.10.0#101680

Merged
FrankHassanabad merged 6 commits intoelastic:masterfrom
FrankHassanabad:update-ecs-mapping
Jun 9, 2021
Merged

[Security Solution][Detections] Update detection alert mappings to ECS v1.10.0#101680
FrankHassanabad merged 6 commits intoelastic:masterfrom
FrankHassanabad:update-ecs-mapping

Conversation

@FrankHassanabad
Copy link
Contributor

@FrankHassanabad FrankHassanabad commented Jun 8, 2021 •
edited
Loading

Summary

  • Grabbed the ECS mappings from v1.10.0 tag
  • Updated the fields that had constant_keyword to keyword since we do many to 1 of source to signals index
  • Wrote a unit tests which tests to ensure we don't have any constant_keyword fields
  • Updated the SIGNALS_TEMPLATE_VERSION version by an increment of 10.

This should mostly fix:
#101572

Since agents add their data into _source even though they have a constant_keyword. When agents do not include the values in _source we will have to merge fields into _source before copying which are still planning on doing before release.

Checklist

@FrankHassanabad FrankHassanabad requested a review from a team as a code owner June 8, 2021 22:20
@FrankHassanabad FrankHassanabad self-assigned this Jun 8, 2021
@FrankHassanabad FrankHassanabad linked an issue Jun 8, 2021 that may be closed by this pull request
[Security Solution] For data_stream fields "Filter for value" and "Filter out value" are working opposite to their functionality. #101572
Closed
@FrankHassanabad FrankHassanabad added v8.0.0 v7.14.0 release_note:fix auto-backport Deprecated - use backport:version if exact versions are needed Team:Detections and Resp Security Detection Response Team labels Jun 8, 2021
@elasticmachine
Copy link
Contributor

elasticmachine commented Jun 8, 2021

Pinging @elastic/security-detections-response (Team:Detections and Resp)

@FrankHassanabad FrankHassanabad requested a review from rylnd June 8, 2021 22:26
rylnd
rylnd approved these changes Jun 9, 2021
View reviewed changes
Copy link
Contributor

@rylnd rylnd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thank you for removing some hardcoded versions in there, as well 👍

@FrankHassanabad FrankHassanabad enabled auto-merge (squash) June 9, 2021 21:04
@kibanamachine
Copy link
Contributor

kibanamachine commented Jun 9, 2021

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @FrankHassanabad

@FrankHassanabad FrankHassanabad merged commit 1cb17d0 into elastic:master Jun 9, 2021
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Jun 9, 2021
@FrankHassanabad @kibanamachine
[Security Solution][Detections] Update detection alert mappings to EC…
7ce7a9e 
…S v1.10.0 (elastic#101680)

## Summary

* Grabbed the ECS mappings from [v1.10.0 tag]( https://github.com/elastic/ecs/blob/v1.10.0/generated/elasticsearch/7/template.json)
* Updated the fields that had `constant_keyword` to `keyword` since we do many to 1 of source to signals index
* Wrote a unit tests which tests to ensure we don't have any `constant_keyword` fields
* Updated the `SIGNALS_TEMPLATE_VERSION` version by an increment of 10.

This should mostly fix:
elastic#101572

Since agents add their data into `_source` even though they have a `constant_keyword`. When agents do not include the values in `_source` we will have to merge `fields` into `_source` before copying which are still planning on doing before release.

### Checklist

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
@kibanamachine kibanamachine mentioned this pull request Jun 9, 2021
Merged
@kibanamachine
Copy link
Contributor

kibanamachine commented Jun 9, 2021

💚 Backport successful

Status Branch Result
✅ 7.x

This backport PR will be merged automatically after passing CI.

kibanamachine added a commit that referenced this pull request Jun 10, 2021
@kibanamachine @FrankHassanabad
[Security Solution][Detections] Update detection alert mappings to EC…
5f5cd72 
…S v1.10.0 (#101680) (#101847)

## Summary

* Grabbed the ECS mappings from [v1.10.0 tag]( https://github.com/elastic/ecs/blob/v1.10.0/generated/elasticsearch/7/template.json)
* Updated the fields that had `constant_keyword` to `keyword` since we do many to 1 of source to signals index
* Wrote a unit tests which tests to ensure we don't have any `constant_keyword` fields
* Updated the `SIGNALS_TEMPLATE_VERSION` version by an increment of 10.

This should mostly fix:
#101572

Since agents add their data into `_source` even though they have a `constant_keyword`. When agents do not include the values in `_source` we will have to merge `fields` into `_source` before copying which are still planning on doing before release.

### Checklist

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios

Co-authored-by: Frank Hassanabad <frank.hassanabad@elastic.co>
jloleysens added a commit to jloleysens/kibana that referenced this pull request Jun 10, 2021
@jloleysens
Merge branch 'master' of github.com:elastic/kibana into fleet/update-…
eb54fc0 
…add-agent-flyout

* 'master' of github.com:elastic/kibana: (35 commits)
  [Cases] Improve connectors mapping (elastic#101145)
  [ML] Fixes display of job group badges in recognizer wizard (elastic#101775)
  Fix es_archives path (elastic#101737)
  [kbnArchiver] convert archive names to root-relative paths (elastic#101839)
  [Reporting] Make "ScreenCapturePanel" shareable for Canvas (elastic#100623)
  [Alerting UI] Converted Rules and Connectors management pages to new layout. (elastic#101697)
  [Fleet] Support granular integrations in policy editor (elastic#101531)
  [Security Solution][Detections] Update detection alert mappings to ECS v1.10.0 (elastic#101680)
  [Fleet] Integrations UI: Adjust policies list UI (elastic#101600)
  chore(NA): moving @kbn/server-route-repository into bazel (elastic#101484)
  Support owner and description attributes inside the Manifest file, use in API docs (elastic#101786)
  [Security Solution] fix security empty overview links (elastic#101536)
  Unskips migration tests now that elastic search is fixed (elastic#101682)
  Fix endpoint -> integrations onboarding link (elastic#101804)
  [Alerting] Log warning when rules are not rescheduled due to Saved Object not found error (elastic#101591)
  Update datafeed_high_count_network_denies.json (elastic#101681)
  [Index patterns] Field editor example app (elastic#100524)
  [DOCS] Adding file upload to add data page (elastic#101674)
  [Security Solution][Endpoint] Adds Endpoint Host Isolation Status common component (elastic#101782)
  Upgrade ws v7.3.1->v7.4.2 and v6.2.1->v6.2.2 (elastic#101402)
  ...

# Conflicts:
#	x-pack/plugins/fleet/public/components/agent_enrollment_flyout/agent_policy_selection.tsx
#	x-pack/plugins/fleet/public/components/agent_enrollment_flyout/index.tsx
#	x-pack/plugins/fleet/public/components/agent_enrollment_flyout/managed_instructions.tsx
#	x-pack/plugins/fleet/public/components/agent_enrollment_flyout/standalone_instructions.tsx
jloleysens added a commit to jloleysens/kibana that referenced this pull request Jun 10, 2021
@jloleysens
Merge branch 'master' of github.com:elastic/kibana into fleet/adjust-…
96e51d9 
…add-integrations-redirect

* 'master' of github.com:elastic/kibana: (44 commits)
  Allow navigating discover flyout via arrow keys (elastic#101772)
  [Cases] Improve connectors mapping (elastic#101145)
  [ML] Fixes display of job group badges in recognizer wizard (elastic#101775)
  Fix es_archives path (elastic#101737)
  [kbnArchiver] convert archive names to root-relative paths (elastic#101839)
  [Reporting] Make "ScreenCapturePanel" shareable for Canvas (elastic#100623)
  [Alerting UI] Converted Rules and Connectors management pages to new layout. (elastic#101697)
  [Fleet] Support granular integrations in policy editor (elastic#101531)
  [Security Solution][Detections] Update detection alert mappings to ECS v1.10.0 (elastic#101680)
  [Fleet] Integrations UI: Adjust policies list UI (elastic#101600)
  chore(NA): moving @kbn/server-route-repository into bazel (elastic#101484)
  Support owner and description attributes inside the Manifest file, use in API docs (elastic#101786)
  [Security Solution] fix security empty overview links (elastic#101536)
  Unskips migration tests now that elastic search is fixed (elastic#101682)
  Fix endpoint -> integrations onboarding link (elastic#101804)
  [Alerting] Log warning when rules are not rescheduled due to Saved Object not found error (elastic#101591)
  Update datafeed_high_count_network_denies.json (elastic#101681)
  [Index patterns] Field editor example app (elastic#100524)
  [DOCS] Adding file upload to add data page (elastic#101674)
  [Security Solution][Endpoint] Adds Endpoint Host Isolation Status common component (elastic#101782)
  ...

# Conflicts:
#	x-pack/plugins/fleet/public/applications/fleet/sections/agent_policy/create_package_policy_page/index.tsx
#	x-pack/plugins/fleet/public/applications/fleet/sections/agent_policy/details_page/components/package_policies/package_policies_table.tsx
semd pushed a commit that referenced this pull request Jun 10, 2021
@FrankHassanabad @semd
[Security Solution][Detections] Update detection alert mappings to EC…
88a329e 
…S v1.10.0 (#101680)

## Summary

* Grabbed the ECS mappings from [v1.10.0 tag]( https://github.com/elastic/ecs/blob/v1.10.0/generated/elasticsearch/7/template.json)
* Updated the fields that had `constant_keyword` to `keyword` since we do many to 1 of source to signals index
* Wrote a unit tests which tests to ensure we don't have any `constant_keyword` fields
* Updated the `SIGNALS_TEMPLATE_VERSION` version by an increment of 10.

This should mostly fix:
#101572

Since agents add their data into `_source` even though they have a `constant_keyword`. When agents do not include the values in `_source` we will have to merge `fields` into `_source` before copying which are still planning on doing before release.

### Checklist

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rylnd rylnd rylnd approved these changes

Assignees

@FrankHassanabad FrankHassanabad

Labels

auto-backport Deprecated - use backport:version if exact versions are needed release_note:fix Team:Detections and Resp Security Detection Response Team v7.14.0 v8.0.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Security Solution] For data_stream fields "Filter for value" and "Filter out value" are working opposite to their functionality.

4 participants

@FrankHassanabad @elasticmachine @kibanamachine @rylnd