Skip to content

[Security Solution][Detection Engine] Fixes critical date time format issues #79911

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Oct 8, 2020
Merged

[Security Solution][Detection Engine] Fixes critical date time format issues #79911

merged 2 commits into from
Oct 8, 2020

Conversation

@FrankHassanabad
Copy link
Contributor

@FrankHassanabad FrankHassanabad commented Oct 7, 2020 •
edited
Loading

Summary

Fixes #79865

Also fixes:

  • Timestamp override not being pushed down into threshold rules to use
  • Timestamp override not being used for lastValidDate
  • The return format of the date time might have been different depending on the customer mapping for both the override and the regular @timestamp so this fixes that as well.
  • Fixes one small type issue with fields.

Checklist

Delete any items that are not applicable to this PR.

@FrankHassanabad FrankHassanabad requested review from a team as code owners October 7, 2020 19:50
@FrankHassanabad FrankHassanabad self-assigned this Oct 7, 2020
@elasticmachine
Copy link
Contributor

elasticmachine commented Oct 7, 2020

Pinging @elastic/siem (Team:SIEM)

@FrankHassanabad FrankHassanabad added bug Fixes for quality problems that affect the customer experience v7.10.0 v7.11.0 v8.0.0 labels Oct 7, 2020
@FrankHassanabad FrankHassanabad changed the title [Security Solution][Detection Engine] Fixes date time format issues [Security Solution][Detection Engine] fixes critical date time format issues Oct 7, 2020
spong
spong approved these changes Oct 8, 2020
View reviewed changes
Copy link
Member

@spong spong left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked out, tested locally with multiple test records (some with @timestamp, some with only specific time fields like event.ingested (in epoch as well)), and all cases outlined in the PR description appear to be functioning as intended.

In testing I did find a few issues, but not related to this PR, so LGTM! 👍 😉

Related issues:

  • Threshold rules allow non-aggregate fields to be selected #79948

  • Timeline is including the docvalue_fields in event details which makes it look like these fields are part of the record: cc @XavierM @andrew-goldstein

  • Threshold rules are looking like they just generate an id for signal.parent.id and signal.parents[].id. We should verify this implementation, and consider not setting a parent id as there isn't a single event that the alert corresponds to (but rather a bucket of events). cc @marshallmain

@FrankHassanabad FrankHassanabad merged commit 7732a21 into elastic:master Oct 8, 2020
@FrankHassanabad FrankHassanabad deleted the add-time-format branch October 8, 2020 00:53
FrankHassanabad added a commit to FrankHassanabad/kibana that referenced this pull request Oct 8, 2020
@FrankHassanabad
Adds date time query and return fields for timestamps and overrides (e…
7b7d931 
…lastic#79911)

## Summary

Fixes elastic#79865

Also fixes:
* Timestamp override not being pushed down into threshold rules to use
* Timestamp override not being used for lastValidDate
* The return format of the date time might have been different depending on the customer mapping for both the override and the regular @timestamp so this fixes that as well.
* Fixes one small type issue with fields.


### Checklist

Delete any items that are not applicable to this PR.

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
@FrankHassanabad FrankHassanabad mentioned this pull request Oct 8, 2020
Merged
FrankHassanabad added a commit to FrankHassanabad/kibana that referenced this pull request Oct 8, 2020
@FrankHassanabad
Adds date time query and return fields for timestamps and overrides (e…
d44e04b 
…lastic#79911)

## Summary

Fixes elastic#79865

Also fixes:
* Timestamp override not being pushed down into threshold rules to use
* Timestamp override not being used for lastValidDate
* The return format of the date time might have been different depending on the customer mapping for both the override and the regular @timestamp so this fixes that as well.
* Fixes one small type issue with fields.


### Checklist

Delete any items that are not applicable to this PR.

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
@FrankHassanabad FrankHassanabad mentioned this pull request Oct 8, 2020
Merged
@FrankHassanabad FrankHassanabad added the Feature:Detection Rules Security Solution rules and Detection Engine label Oct 8, 2020
@FrankHassanabad FrankHassanabad changed the title [Security Solution][Detection Engine] fixes critical date time format issues [Security Solution][Detection Engine] Fixes critical date time format issues Oct 8, 2020
@kibanamachine
Copy link
Contributor

kibanamachine commented Oct 8, 2020

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

FrankHassanabad added a commit that referenced this pull request Oct 8, 2020
@FrankHassanabad
Adds date time query and return fields for timestamps and overrides (#…
88400cb 
…79911) (#79965)

## Summary

Fixes #79865

Also fixes:
* Timestamp override not being pushed down into threshold rules to use
* Timestamp override not being used for lastValidDate
* The return format of the date time might have been different depending on the customer mapping for both the override and the regular @timestamp so this fixes that as well.
* Fixes one small type issue with fields.


### Checklist

Delete any items that are not applicable to this PR.

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
FrankHassanabad added a commit that referenced this pull request Oct 8, 2020
@FrankHassanabad
Adds date time query and return fields for timestamps and overrides (#…
72114b4 
…79911) (#79964)

## Summary

Fixes #79865

Also fixes:
* Timestamp override not being pushed down into threshold rules to use
* Timestamp override not being used for lastValidDate
* The return format of the date time might have been different depending on the customer mapping for both the override and the regular @timestamp so this fixes that as well.
* Fixes one small type issue with fields.


### Checklist

Delete any items that are not applicable to this PR.

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
gmmorris added a commit to gmmorris/kibana that referenced this pull request Oct 8, 2020
@gmmorris
Merge branch 'master' into task-manager/health
2f12de5 
* master: (217 commits)
  Fix dashboard "snapshot share" is not sharing panel state in view mode (elastic#79837)
  fix can't edit a scripted field with special char (elastic#79842)
  [ML] clear selection action (elastic#79834)
  [TSVB] Show tooltip on external pointer events (elastic#77306)
  Fixes bug where the same index was being passed in (elastic#79949)
  Adds date time query and return fields for timestamps and overrides (elastic#79911)
  [Security Solution][Detections] Reverts rules table tag filter to use AND operator (elastic#79920)
  add the correct class to truncate the names (elastic#79921)
  [kbn/optimizer] report limits with ci metrics (elastic#78205)
  [release notes] extract "dev docs" comment too (elastic#79351)
  Revert "skips test failing promotion (elastic#79777)" (elastic#79904)
  share tslib across bundles (elastic#79915)
  remove entire suite as partial skips aren't doing the trick
  skip flaky suite (elastic#78689)
  Skip failing suite (elastic#79522)
  skip flaky suite (elastic#79910)
  [es/mappings] remove doc_values from text fields (elastic#79869)
  remove skipped snapshots
  skip flaky tests (elastic#79891)
  chore(NA): add missing branches into backportrc configuration file (elastic#79848)
  ...
@MindyRS MindyRS added the Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. label Sep 23, 2021
@elasticmachine
Copy link
Contributor

elasticmachine commented Sep 23, 2021

Pinging @elastic/security-solution (Team: SecuritySolution)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@spong spong spong approved these changes

Assignees

@FrankHassanabad FrankHassanabad

Labels

bug Fixes for quality problems that affect the customer experience Feature:Detection Rules Security Solution rules and Detection Engine release_note:fix Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:SIEM v7.10.0 v7.11.0 v8.0.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Security Solutions][Detection Engine] Throws date time format errors on custom mappings

5 participants

@FrankHassanabad @elasticmachine @kibanamachine @spong @MindyRS