add zeek system tests #448

leehinman · 2020-12-04T21:20:23Z

What does this PR do?

Add system tests to zeek package and bump version to 0.3.5

Specific changes to data_streams were:

capture_loss
connection
dce_rpc, update ecs.yml
dhcp, update ecs.yml
dnp3, update ecs.yml
dns, update ecs.yml, fix type mismatch
dpd
files
http, update ecs.yml, fix path configuration
intel
irc
kerberos, update ecs.yml
modbus, update ecs.yml
mysql, update ecs.yml
notice
ntlm
pe
radius
rdp, update ecs.yml
rfb
sip
smb_cmd
smb_files
smb_mapping
smtp
snmp
socks
ssh
ssl, update ecs.yml
stats
traceroute
tunnel
weird
x509, update ecs.yml

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all datasets collect metrics or logs.

How to test this PR locally

elastic-package clean && \
elastic-package build && \
elastic-package stack up -d && \
$(elastic-package stack shellinit) && \
elastic-package test system -v

Related issues

Relates Add automated tests for security packages #377

elasticmachine · 2020-12-04T21:21:18Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

elasticmachine · 2020-12-04T21:33:27Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #448 updated
Start Time: 2020-12-10T19:25:43.512+0000
Duration: 33 min 3 sec

Test stats 🧪

Test	Results
Failed	0
Passed	124
Skipped	0
Total	124

andrewkroh · 2020-12-08T01:20:23Z

The CI error is

[2020-12-07T22:37:23.341Z] Error: checking package failed: formatting the integration failed (path: /var/lib/jenkins/workspace/gest-manager_integrations_PR-448/src/github.com/elastic/integrations/packages/zeek, failFast: true): walking through the integration files failed: formatting file failed (path: /var/lib/jenkins/workspace/gest-manager_integrations_PR-448/src/github.com/elastic/integrations/packages/zeek/data_stream/dns/fields/ecs.yml): file is not formatted (path: /var/lib/jenkins/workspace/gest-manager_integrations_PR-448/src/github.com/elastic/integrations/packages/zeek/data_stream/dns/fields/ecs.yml)

Try elastic-package format.

andrewkroh

LGTM. Just one minor issue that I apologize for naming the way I did.

andrewkroh · 2020-12-08T01:43:36Z

packages/zeek/data_stream/dns/agent/stream/log.yml.hbs

        - {from: zeek.dns.query, to: dns.question.name}
        - {from: zeek.dns.qtype_name, to: dns.question.type}
        - {from: zeek.dns.rcode_name, to: dns.response_code}
+  - convert:
+      ignore_missing: true
+      ignore_failure: true


Suggested change

ignore_failure: true

fail_on_error: false

- update version to 0.3.5 - capture_loss - connection - dce_rpc, update ecs.yml - dhcp, update ecs.yml - dnp3, update ecs.yml - dns, update ecs.yml, fix type mismatch - dpd - files - http, update ecs.yml, fix path configuration - intel - irc - kerberos, update ecs.yml - modbus, update ecs.yml - mysql, update ecs.yml - notice - ntlm - pe - radius - rdp, update ecs.yml - rfb - sip - smb_cmd - smb_files - smb_mapping - smtp - snmp - socks - ssh - ssl, update ecs.yml - stats - traceroute - tunnel - weird - x509, update ecs.yml

- fix indentation in fields files - fix "fail_on_error" option in dns

andrewkroh · 2021-01-11T18:44:29Z

packages/zeek/data_stream/dns/agent/stream/log.yml.hbs

@@ -21,7 +21,7 @@ processors:
      target: zeek.dns
  - registered_domain:
      ignore_missing: true
-      ignore_failure: true
+      fail_on_error: false


This one was correct with the use of ignore_failure: true.

Fix usages of ignore_failure with convert processor. Make DNS transaction ID a string. elastic/integrations#448

* Sync changes to AWS CloudTrail elastic/integrations#408 * Sync changes to CheckPoint Firewall Change type of event.severity. elastic/integrations#409 * Sync changes from Cisco ASA / FTD elastic/integrations#414 * Sync changes from Cisco IOS Make icmp and igmp fields strings because they are keywords. elastic/integrations#416 * Sync changes to CrowdStrike Falcon Fix some field types. elastic/integrations#377 * Sync changes to Fortinet Firewall Drop assignip if the value is "N/A". elastic/integrations#437 * Sync changes to Juniper SRX Convert event.risk values to float Protect against missing event.timezone Convert event.severity to long. elastic/integrations#443 * Sync changes to Suricata EVE Convert suricata.eve.flow_id to string because the field is a keyword in the mapping. elastic/integrations#457 * Sync changes to Zeek DNS Fix usages of ignore_failure with convert processor. Make DNS transaction ID a string. elastic/integrations#448 * Add changelog

* Sync changes to AWS CloudTrail elastic/integrations#408 * Sync changes to CheckPoint Firewall Change type of event.severity. elastic/integrations#409 * Sync changes from Cisco ASA / FTD elastic/integrations#414 * Sync changes from Cisco IOS Make icmp and igmp fields strings because they are keywords. elastic/integrations#416 * Sync changes to CrowdStrike Falcon Fix some field types. elastic/integrations#377 * Sync changes to Fortinet Firewall Drop assignip if the value is "N/A". elastic/integrations#437 * Sync changes to Juniper SRX Convert event.risk values to float Protect against missing event.timezone Convert event.severity to long. elastic/integrations#443 * Sync changes to Suricata EVE Convert suricata.eve.flow_id to string because the field is a keyword in the mapping. elastic/integrations#457 * Sync changes to Zeek DNS Fix usages of ignore_failure with convert processor. Make DNS transaction ID a string. elastic/integrations#448 * Add changelog (cherry picked from commit bf46572)

#24077) * Sync fixes from Integration Package Testing (#23424) * Sync changes to AWS CloudTrail elastic/integrations#408 * Sync changes to CheckPoint Firewall Change type of event.severity. elastic/integrations#409 * Sync changes from Cisco ASA / FTD elastic/integrations#414 * Sync changes from Cisco IOS Make icmp and igmp fields strings because they are keywords. elastic/integrations#416 * Sync changes to CrowdStrike Falcon Fix some field types. elastic/integrations#377 * Sync changes to Fortinet Firewall Drop assignip if the value is "N/A". elastic/integrations#437 * Sync changes to Juniper SRX Convert event.risk values to float Protect against missing event.timezone Convert event.severity to long. elastic/integrations#443 * Sync changes to Suricata EVE Convert suricata.eve.flow_id to string because the field is a keyword in the mapping. elastic/integrations#457 * Sync changes to Zeek DNS Fix usages of ignore_failure with convert processor. Make DNS transaction ID a string. elastic/integrations#448 * Add changelog (cherry picked from commit bf46572)

* add zeek system tests - update version to 0.3.5 - capture_loss - connection - dce_rpc, update ecs.yml - dhcp, update ecs.yml - dnp3, update ecs.yml - dns, update ecs.yml, fix type mismatch - dpd - files - http, update ecs.yml, fix path configuration - intel - irc - kerberos, update ecs.yml - modbus, update ecs.yml - mysql, update ecs.yml - notice - ntlm - pe - radius - rdp, update ecs.yml - rfb - sip - smb_cmd - smb_files - smb_mapping - smtp - snmp - socks - ssh - ssl, update ecs.yml - stats - traceroute - tunnel - weird - x509, update ecs.yml * incorporate feedback - fix indentation in fields files - fix "fail_on_error" option in dns

leehinman added enhancement New feature or request Team:Security-External Integrations labels Dec 4, 2020

leehinman mentioned this pull request Dec 4, 2020

Add automated tests for security packages #377

Closed

andrewkroh approved these changes Dec 8, 2020

View reviewed changes

leehinman added 2 commits December 10, 2020 13:25

incorporate feedback

ffe02cb

- fix indentation in fields files - fix "fail_on_error" option in dns

leehinman force-pushed the zeek_system_test branch from c28ea46 to ffe02cb Compare December 10, 2020 19:25

leehinman merged commit d864401 into elastic:master Dec 10, 2020

leehinman deleted the zeek_system_test branch December 10, 2020 20:39

andrewkroh reviewed Jan 11, 2021

View reviewed changes

andrewkroh added a commit to andrewkroh/beats that referenced this pull request Jan 12, 2021

Sync changes to Zeek DNS

c80e180

Fix usages of ignore_failure with convert processor. Make DNS transaction ID a string. elastic/integrations#448

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add zeek system tests #448

add zeek system tests #448

leehinman commented Dec 4, 2020

elasticmachine commented Dec 4, 2020

elasticmachine commented Dec 4, 2020 •

edited

Loading

Build stats

Test stats 🧪

andrewkroh commented Dec 8, 2020

andrewkroh left a comment

andrewkroh Dec 8, 2020

andrewkroh Jan 11, 2021

add zeek system tests #448

add zeek system tests #448

Conversation

leehinman commented Dec 4, 2020

What does this PR do?

Checklist

How to test this PR locally

Related issues

elasticmachine commented Dec 4, 2020

elasticmachine commented Dec 4, 2020 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

andrewkroh commented Dec 8, 2020

andrewkroh left a comment

Choose a reason for hiding this comment

andrewkroh Dec 8, 2020

Choose a reason for hiding this comment

andrewkroh Jan 11, 2021

Choose a reason for hiding this comment

elasticmachine commented Dec 4, 2020 •

edited

Loading