Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add pipeline tests for AWS CloudTrail #408

Merged
merged 1 commit into from
Dec 2, 2020

Conversation

andrewkroh
Copy link
Member

@andrewkroh andrewkroh commented Nov 19, 2020

What does this PR do?

The tests revealed a few issues. There was an error in the pipeline for update-user-json.log because
serviceEventDetails was not present. This was the error

        "error": {
            "message": "Cannot invoke \\\"Object.getClass()\\\" because \\\"receiver\\\" is null"
        }

The aws.cloudtrail.read_only field was mapped as keyword but was actual a JSON boolean.
I changed the type to boolean, but do not plan to backport this change to Filebeat.

And lastly some ECS user_agent fields were missing.

This depends on elastic/elastic-package#177 to make the flattened fields pass
test validation.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all datasets collect metrics or logs.

Related issues

@elasticmachine
Copy link

elasticmachine commented Nov 19, 2020

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Pull request #408 updated

  • Start Time: 2020-12-01T23:05:17.679+0000

  • Duration: 24 min 57 sec

Test stats 🧪

Test Results
Failed 0
Passed 57
Skipped 0
Total 57

@elasticmachine
Copy link

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@andrewkroh andrewkroh force-pushed the feature/aws/cloudtrail-tests branch from bedca2b to 304f5c1 Compare November 20, 2020 00:13
@andrewkroh andrewkroh force-pushed the feature/aws/cloudtrail-tests branch 2 times, most recently from 4f32459 to be9f816 Compare November 20, 2020 16:33
@mtojek mtojek self-requested a review November 23, 2020 08:19
Copy link
Contributor

@mtojek mtojek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As there were incompatible changes pushed to the master branch (PR: #410), please rebase this PR against the master branch.

You can fix go.sum with mage ModTidy.

@andrewkroh andrewkroh force-pushed the feature/aws/cloudtrail-tests branch 2 times, most recently from 7221f0c to eec40b6 Compare November 23, 2020 14:52
The tests revealed a few issues. There was an error in the pipeline for update-user-json.log because
serviceEventDetails was not present. This was the error

            "error": {
                "message": "Cannot invoke \\\"Object.getClass()\\\" because \\\"receiver\\\" is null"
            }

The aws.cloudtrail.read_only field was mapped as keyword but was actual a JSON boolean.
I changed the type to boolean, but do not plan to backport this change to Filebeat.

And lastly some ECS user_agent fields were missing.

This depends on elastic/elastic-package#177 to make the flattened fields pass
test validation.
@andrewkroh andrewkroh force-pushed the feature/aws/cloudtrail-tests branch from eec40b6 to e13a28f Compare December 1, 2020 23:05
Copy link
Contributor

@leehinman leehinman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great.

@andrewkroh andrewkroh merged commit 270be37 into elastic:master Dec 2, 2020
andrewkroh added a commit to andrewkroh/beats that referenced this pull request Jan 11, 2021
andrewkroh added a commit to elastic/beats that referenced this pull request Jan 25, 2021
* Sync changes to AWS CloudTrail

elastic/integrations#408

* Sync changes to CheckPoint Firewall

Change type of event.severity.

elastic/integrations#409

* Sync changes from Cisco ASA / FTD

elastic/integrations#414

* Sync changes from Cisco IOS

Make icmp and igmp fields strings because they are keywords.

elastic/integrations#416

* Sync changes to CrowdStrike Falcon

Fix some field types.

elastic/integrations#377

* Sync changes to Fortinet Firewall

Drop assignip if the value is "N/A".

elastic/integrations#437

* Sync changes to Juniper SRX

Convert event.risk values to float
Protect against missing event.timezone
Convert event.severity to long.

elastic/integrations#443

* Sync changes to Suricata EVE

Convert suricata.eve.flow_id to string because the field is a keyword in the mapping.

elastic/integrations#457

* Sync changes to Zeek DNS

Fix usages of ignore_failure with convert processor.
Make DNS transaction ID a string.

elastic/integrations#448

* Add changelog
andrewkroh added a commit to andrewkroh/beats that referenced this pull request Feb 16, 2021
* Sync changes to AWS CloudTrail

elastic/integrations#408

* Sync changes to CheckPoint Firewall

Change type of event.severity.

elastic/integrations#409

* Sync changes from Cisco ASA / FTD

elastic/integrations#414

* Sync changes from Cisco IOS

Make icmp and igmp fields strings because they are keywords.

elastic/integrations#416

* Sync changes to CrowdStrike Falcon

Fix some field types.

elastic/integrations#377

* Sync changes to Fortinet Firewall

Drop assignip if the value is "N/A".

elastic/integrations#437

* Sync changes to Juniper SRX

Convert event.risk values to float
Protect against missing event.timezone
Convert event.severity to long.

elastic/integrations#443

* Sync changes to Suricata EVE

Convert suricata.eve.flow_id to string because the field is a keyword in the mapping.

elastic/integrations#457

* Sync changes to Zeek DNS

Fix usages of ignore_failure with convert processor.
Make DNS transaction ID a string.

elastic/integrations#448

* Add changelog

(cherry picked from commit bf46572)
adriansr pushed a commit to elastic/beats that referenced this pull request Feb 17, 2021
#24077)

* Sync fixes from Integration Package Testing (#23424)

* Sync changes to AWS CloudTrail

elastic/integrations#408

* Sync changes to CheckPoint Firewall

Change type of event.severity.

elastic/integrations#409

* Sync changes from Cisco ASA / FTD

elastic/integrations#414

* Sync changes from Cisco IOS

Make icmp and igmp fields strings because they are keywords.

elastic/integrations#416

* Sync changes to CrowdStrike Falcon

Fix some field types.

elastic/integrations#377

* Sync changes to Fortinet Firewall

Drop assignip if the value is "N/A".

elastic/integrations#437

* Sync changes to Juniper SRX

Convert event.risk values to float
Protect against missing event.timezone
Convert event.severity to long.

elastic/integrations#443

* Sync changes to Suricata EVE

Convert suricata.eve.flow_id to string because the field is a keyword in the mapping.

elastic/integrations#457

* Sync changes to Zeek DNS

Fix usages of ignore_failure with convert processor.
Make DNS transaction ID a string.

elastic/integrations#448

* Add changelog

(cherry picked from commit bf46572)
eyalkraft pushed a commit to build-security/integrations that referenced this pull request Mar 30, 2022
The tests revealed a few issues. There was an error in the pipeline for update-user-json.log because
serviceEventDetails was not present. This was the error

            "error": {
                "message": "Cannot invoke \\\"Object.getClass()\\\" because \\\"receiver\\\" is null"
            }

The aws.cloudtrail.read_only field was mapped as keyword but was actual a JSON boolean.
I changed the type to boolean, but do not plan to backport this change to Filebeat.

And lastly some ECS user_agent fields were missing.

This depends on elastic/elastic-package#177 to make the flattened fields pass
test validation.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants