Add pipeline test for Fortinet Firewall #437

andrewkroh · 2020-12-02T22:53:18Z

What does this PR do?

This adds a pipeline test for Fortinet Firewall.

These are the original failures after enabling the test.

FAILURE DETAILS:

fortinet/firewall test-fortinet.log:
[0] field "_temp.time" is undefined
[1] field "fortinet.firewall.devid" is undefined
[2] field "fortinet.firewall.devname" is undefined
[3] field "fortinet.firewall.dir" is undefined
[4] field "fortinet.firewall.group" is undefined
[5] field "fortinet.firewall.level" is undefined
[6] field "fortinet.firewall.locip" is undefined
[7] field "fortinet.firewall.locport" is undefined
[8] field "fortinet.firewall.logdesc" is undefined
[9] field "fortinet.firewall.logid" is undefined
[10] field "fortinet.firewall.msg" is undefined
[11] field "fortinet.firewall.remip" is undefined
[12] field "fortinet.firewall.remport" is undefined
[13] field "fortinet.firewall.user" is undefined
[14] field "syslog5424_pri" is undefined
[15] field "syslog5424_sd" is undefined
[16] parsing field value failed: field "fortinet.firewall.disklograte"''s Go type, string, does not match the expected field type: long
[17] parsing field value failed: field "fortinet.firewall.fazlograte"''s Go type, string, does not match the expected field type: long
[18] parsing field value failed: field "fortinet.firewall.lanin"''s Go type, string, does not match the expected field type: long
[19] parsing field value failed: field "fortinet.firewall.lanout"''s Go type, string, does not match the expected field type: long
[20] parsing field value failed: field "fortinet.firewall.setuprate"''s Go type, string, does not match the expected field type: long
[21] parsing field value failed: field "fortinet.firewall.wanin"''s Go type, string, does not match the expected field type: long
[22] parsing field value failed: field "fortinet.firewall.wanout"''s Go type, string, does not match the expected field type: long
--- Test results for package: fortinet - END   ---

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all datasets collect metrics or logs.

Related issues

Relates Add automated tests for security packages #377

These are the original failures after enabling the test. FAILURE DETAILS: fortinet/firewall test-fortinet.log: [0] field "_temp.time" is undefined [1] field "fortinet.firewall.devid" is undefined [2] field "fortinet.firewall.devname" is undefined [3] field "fortinet.firewall.dir" is undefined [4] field "fortinet.firewall.group" is undefined [5] field "fortinet.firewall.level" is undefined [6] field "fortinet.firewall.locip" is undefined [7] field "fortinet.firewall.locport" is undefined [8] field "fortinet.firewall.logdesc" is undefined [9] field "fortinet.firewall.logid" is undefined [10] field "fortinet.firewall.msg" is undefined [11] field "fortinet.firewall.remip" is undefined [12] field "fortinet.firewall.remport" is undefined [13] field "fortinet.firewall.user" is undefined [14] field "syslog5424_pri" is undefined [15] field "syslog5424_sd" is undefined [16] parsing field value failed: field "fortinet.firewall.disklograte"''s Go type, string, does not match the expected field type: long [17] parsing field value failed: field "fortinet.firewall.fazlograte"''s Go type, string, does not match the expected field type: long [18] parsing field value failed: field "fortinet.firewall.lanin"''s Go type, string, does not match the expected field type: long [19] parsing field value failed: field "fortinet.firewall.lanout"''s Go type, string, does not match the expected field type: long [20] parsing field value failed: field "fortinet.firewall.setuprate"''s Go type, string, does not match the expected field type: long [21] parsing field value failed: field "fortinet.firewall.wanin"''s Go type, string, does not match the expected field type: long [22] parsing field value failed: field "fortinet.firewall.wanout"''s Go type, string, does not match the expected field type: long --- Test results for package: fortinet - END ---

andrewkroh · 2020-12-02T22:54:32Z

This PR needs fixes for the mappings and incorrect data types in the JSON (needs convert processors).

elasticmachine · 2020-12-02T23:07:29Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Branch indexing
Start Time: 2020-12-08T22:28:35.941+0000
Duration: 17 min 16 sec

Test stats 🧪

Test	Results
Failed	0
Passed	73
Skipped	0
Total	73

elasticmachine · 2020-12-03T15:58:39Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

andrewkroh · 2020-12-03T19:55:13Z

packages/fortinet/data_stream/firewall/elasticsearch/ingest_pipeline/default.yml

-    - fortinet.firewall.duration
-    - host
+      - _temp.time
+      - _temp


Does this only work on leaf fields? I'm curious since it has both _temp.time and _temp.

andrewkroh

@leehinman LGTM. I cannot approve since I'm the original author. 😆

andrewkroh · 2020-12-08T05:51:27Z

run tests

Drop assignip if the value is "N/A". elastic/integrations#437

* Sync changes to AWS CloudTrail elastic/integrations#408 * Sync changes to CheckPoint Firewall Change type of event.severity. elastic/integrations#409 * Sync changes from Cisco ASA / FTD elastic/integrations#414 * Sync changes from Cisco IOS Make icmp and igmp fields strings because they are keywords. elastic/integrations#416 * Sync changes to CrowdStrike Falcon Fix some field types. elastic/integrations#377 * Sync changes to Fortinet Firewall Drop assignip if the value is "N/A". elastic/integrations#437 * Sync changes to Juniper SRX Convert event.risk values to float Protect against missing event.timezone Convert event.severity to long. elastic/integrations#443 * Sync changes to Suricata EVE Convert suricata.eve.flow_id to string because the field is a keyword in the mapping. elastic/integrations#457 * Sync changes to Zeek DNS Fix usages of ignore_failure with convert processor. Make DNS transaction ID a string. elastic/integrations#448 * Add changelog

* Sync changes to AWS CloudTrail elastic/integrations#408 * Sync changes to CheckPoint Firewall Change type of event.severity. elastic/integrations#409 * Sync changes from Cisco ASA / FTD elastic/integrations#414 * Sync changes from Cisco IOS Make icmp and igmp fields strings because they are keywords. elastic/integrations#416 * Sync changes to CrowdStrike Falcon Fix some field types. elastic/integrations#377 * Sync changes to Fortinet Firewall Drop assignip if the value is "N/A". elastic/integrations#437 * Sync changes to Juniper SRX Convert event.risk values to float Protect against missing event.timezone Convert event.severity to long. elastic/integrations#443 * Sync changes to Suricata EVE Convert suricata.eve.flow_id to string because the field is a keyword in the mapping. elastic/integrations#457 * Sync changes to Zeek DNS Fix usages of ignore_failure with convert processor. Make DNS transaction ID a string. elastic/integrations#448 * Add changelog (cherry picked from commit bf46572)

#24077) * Sync fixes from Integration Package Testing (#23424) * Sync changes to AWS CloudTrail elastic/integrations#408 * Sync changes to CheckPoint Firewall Change type of event.severity. elastic/integrations#409 * Sync changes from Cisco ASA / FTD elastic/integrations#414 * Sync changes from Cisco IOS Make icmp and igmp fields strings because they are keywords. elastic/integrations#416 * Sync changes to CrowdStrike Falcon Fix some field types. elastic/integrations#377 * Sync changes to Fortinet Firewall Drop assignip if the value is "N/A". elastic/integrations#437 * Sync changes to Juniper SRX Convert event.risk values to float Protect against missing event.timezone Convert event.severity to long. elastic/integrations#443 * Sync changes to Suricata EVE Convert suricata.eve.flow_id to string because the field is a keyword in the mapping. elastic/integrations#457 * Sync changes to Zeek DNS Fix usages of ignore_failure with convert processor. Make DNS transaction ID a string. elastic/integrations#448 * Add changelog (cherry picked from commit bf46572)

* Add pipeline test for Fortinet Firewall These are the original failures after enabling the test. FAILURE DETAILS: fortinet/firewall test-fortinet.log: [0] field "_temp.time" is undefined [1] field "fortinet.firewall.devid" is undefined [2] field "fortinet.firewall.devname" is undefined [3] field "fortinet.firewall.dir" is undefined [4] field "fortinet.firewall.group" is undefined [5] field "fortinet.firewall.level" is undefined [6] field "fortinet.firewall.locip" is undefined [7] field "fortinet.firewall.locport" is undefined [8] field "fortinet.firewall.logdesc" is undefined [9] field "fortinet.firewall.logid" is undefined [10] field "fortinet.firewall.msg" is undefined [11] field "fortinet.firewall.remip" is undefined [12] field "fortinet.firewall.remport" is undefined [13] field "fortinet.firewall.user" is undefined [14] field "syslog5424_pri" is undefined [15] field "syslog5424_sd" is undefined [16] parsing field value failed: field "fortinet.firewall.disklograte"''s Go type, string, does not match the expected field type: long [17] parsing field value failed: field "fortinet.firewall.fazlograte"''s Go type, string, does not match the expected field type: long [18] parsing field value failed: field "fortinet.firewall.lanin"''s Go type, string, does not match the expected field type: long [19] parsing field value failed: field "fortinet.firewall.lanout"''s Go type, string, does not match the expected field type: long [20] parsing field value failed: field "fortinet.firewall.setuprate"''s Go type, string, does not match the expected field type: long [21] parsing field value failed: field "fortinet.firewall.wanin"''s Go type, string, does not match the expected field type: long [22] parsing field value failed: field "fortinet.firewall.wanout"''s Go type, string, does not match the expected field type: long --- Test results for package: fortinet - END --- * fix errors in pipeline Co-authored-by: Lee E. Hinman <[email protected]>

andrewkroh added the Team:Security-External Integrations label Dec 2, 2020

andrewkroh added the enhancement New feature or request label Dec 2, 2020

andrewkroh mentioned this pull request Dec 2, 2020

Add automated tests for security packages #377

Closed

fix errors in pipeline

402813f

leehinman marked this pull request as ready for review December 3, 2020 15:58

andrewkroh commented Dec 3, 2020

View reviewed changes

andrewkroh commented Dec 4, 2020

View reviewed changes

andrewstucki approved these changes Dec 9, 2020

View reviewed changes

andrewkroh merged commit 3a8589d into elastic:master Dec 9, 2020

andrewkroh added a commit to andrewkroh/beats that referenced this pull request Jan 12, 2021

Sync changes to Fortinet Firewall

84a7abe

Drop assignip if the value is "N/A". elastic/integrations#437

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add pipeline test for Fortinet Firewall #437

Add pipeline test for Fortinet Firewall #437

andrewkroh commented Dec 2, 2020

andrewkroh commented Dec 2, 2020

elasticmachine commented Dec 2, 2020 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented Dec 3, 2020

andrewkroh Dec 3, 2020

andrewkroh left a comment

andrewkroh commented Dec 8, 2020

Add pipeline test for Fortinet Firewall #437

Add pipeline test for Fortinet Firewall #437

Conversation

andrewkroh commented Dec 2, 2020

What does this PR do?

Checklist

Related issues

andrewkroh commented Dec 2, 2020

elasticmachine commented Dec 2, 2020 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

elasticmachine commented Dec 3, 2020

andrewkroh Dec 3, 2020

Choose a reason for hiding this comment

andrewkroh left a comment

Choose a reason for hiding this comment

andrewkroh commented Dec 8, 2020

elasticmachine commented Dec 2, 2020 •

edited

Loading