Generate geo ip test results with the new db by marc-gr · Pull Request #2339 · elastic/integrations

marc-gr · 2021-12-14T15:25:48Z

What does this PR do?

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

elastic-package version is updated
all integrations use IPs of the allowed set for testing
all test files are generated to use the new geoip database

elasticmachine · 2021-12-14T15:25:50Z

Pinging @elastic/integrations (Team:Integrations)

elasticmachine · 2021-12-14T15:25:50Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

elasticmachine · 2021-12-14T15:41:23Z

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-12-15T09:04:51.721+0000
Duration: 83 min 29 sec
Commit: 296efbc

Test stats 🧪

Test	Results
Failed	1
Passed	3683
Skipped	5
Total	3689

Test errors

Expand to view the tests failures

`Check integrations / akamai / akamai: check / pipeline test: test-http-json.log – akamai.siem`

no error details

Expand to view the stacktrace

 test case failed: Expected results are different from actual ones:  {
     "expected": [
         {
             "@timestamp": "2017-04-04T10:57:02.000Z",
             "akamai": {
                 "siem": {
                     "bot": {
                         "response_segment": 3,
                         "score": 100
                     },
                     "client_data": {
                         "app_bundle_id": "com.mydomain.myapp",
                         "app_version": "1.23",
                         "sdk_version": "4.7.1",
                         "telemetry_type": 2
                     },
                     "config_id": "14227",
                     "policy_id": "qik1_26545",
                     "request": {
                         "headers": {
                             "Accept": "text/html,application/xhtml xml",
                             "User-Agent": "BOT/0.1 (BOT for JCE)"
                         }
                     },
                     "response": {
                         "headers": {
                             "Content-Length": "150",
                             "Content-Type": "text/html",
                             "Mime-Version": "1.0",
                             "Server": "AkamaiGHost"
                         }
                     },
                     "rules": [
                         {
                             "ruleActions": "alert",
                             "ruleData": "telnet.exe",
                             "ruleMessages": "System Command Access",
                             "ruleSelectors": "ARGS:option",
                             "ruleTags": "OWASP_CRS/WEB_ATTACK/FILE_INJECTION",
                             "ruleVersions": "4",
                             "rules": "950002"
                         },
                         {
                             "ruleActions": "alert",
                             "ruleData": "telnet.exe",
                             "ruleMessages": "System Command Injection",
                             "ruleSelectors": "ARGS:option",
                             "ruleTags": "OWASP_CRS/WEB_ATTACK/COMMAND_INJECT",
                             "ruleVersions": "4",
                             "rules": "950006"
                         },
                         {
                             "ruleActions": "deny",
                             "ruleData": "Vector Score: 10, DENY threshold: 9, Ale",
                             "ruleMessages": "Anomaly Score Exceeded fo",
                             "ruleVersions": "1",
                             "rules": "CMD-INJECTION-ANOMALY"
                         }
                     ],
                     "user_risk": {
                         "allow": 0,
                         "general": {
                             "duc_1d": "30",
                             "duc_1h": "10"
                         },
                         "risk": {
                             "udfp": "1325gdg4g4343g/M",
                             "unp": "74256/H"
                         },
                         "score": 75,
                         "status": 0,
                         "trust": {
                             "ugp": "US"
                         },
                         "uuid": "964d54b7-0821-413a-a4d6-8131770ec8d5"
                     }
                 }
             },
             "client": {
                 "address": "52.91.36.10",
                 "as": {
-                    "number": 14618,
-                    "organization": {
-                        "name": "Amazon.com, Inc."
-                    }
+                    "number": 14618
                 },
                 "geo": {
-                    "city_name": "Ashburn",
-                    "continent_name": "North America",
+                    "city_name": "ASHBURN",
                     "country_iso_code": "US",
-                    "country_name": "United States",
-                    "location": {
-                        "lat": 39.0481,
-                        "lon": -77.4728
-                    },
-                    "region_iso_code": "US-VA",
-                    "region_name": "Virginia"
+                    "region_iso_code": "-VA"
                 },
                 "ip": "52.91.36.10"
             },
             "ecs": {
                 "version": "1.12.0"
             },
             "event": {
                 "category": "network",
                 "id": "1158db1758e37bfe67b7c09",
                 "kind": "event",
                 "original": "{\"format\":\"json\",\"type\":\"akamai_siem\",\"version\":\"1.0\",\"attackData\":{\"clientIP\":\"52.91.36.10\",\"configId\":\"14227\",\"policyId\":\"qik1_26545\",\"ruleActions\":\"YWxlcnQ%3d%3bYWxlcnQ%3d%3bZGVueQ%3d%3d\",\"ruleData\":\"dGVsbmV0LmV4ZQ%3d%3d%3bdGVsbmV0LmV4ZQ%3d%3d%3bVmVjdG9yIFNjb3JlOiAxMCwgREVOWSB0aHJlc2hvbGQ6IDksIEFsZX \",\"ruleMessages\":\"U3lzdGVtIENvbW1hbmQgQWNjZXNz%3bU3lzdGVtIENvbW1hbmQgSW5qZWN0aW9u%3bQW5vbWFseSBTY29yZSBFeGNlZWRlZCBmb3 \",\"ruleSelectors\":\"QVJHUzpvcHRpb24%3d%3bQVJHUzpvcHRpb24%3d%3b\",\"ruleTags\":\"T1dBU1BfQ1JTL1dFQl9BVFRBQ0svRklMRV9JTkpFQ1RJT04%3d%3bT1dBU1BfQ1JTL1dFQl9BVFRBQ0svQ09NTUFORF9JTkpFQ1R \",\"ruleVersions\":\"NA%3d%3d%3bNA%3d%3d%3bMQ%3d%3d\",\"rules\":\"OTUwMDAy%3bOTUwMDA2%3bQ01ELUlOSkVDVElPTi1BTk9NQUxZ\"},\"geo\":{\"asn\":\"14618\",\"city\":\"ASHBURN\",\"continent\":\"288\",\"country\":\"US\",\"regionCode\":\"VA\"},\"httpMessage\":{\"bytes\":\"266\",\"host\":\"www.hmapi.com\",\"method\":\"GET\",\"path\":\"/\",\"port\":\"80\",\"protocol\":\"HTTP/1.1\",\"query\":\"option=com_jce%20telnet.exe\",\"requestHeaders\":\"User-Agent%3a%20BOT%2f0.1%20(BOT%20for%20JCE)%0d%0aAccept%3a%20text%2fhtml,application%2fxhtml+xml\",\"requestId\":\"1158db1758e37bfe67b7c09\",\"responseHeaders\":\"Server%3a%20AkamaiGHost%0d%0aMime-Version%3a%201.0%0d%0aContent-Type%3a%20text%2fhtml%0d%0aContent-Length%3a%20150\",\"start\":\"1491303422\",\"status\":\"200\"},\"userRiskData\":{\"uuid\":\"964d54b7-0821-413a-a4d6-8131770ec8d5\",\"status\":\"0\",\"score\":\"75\",\"risk\":\"udfp:1325gdg4g4343g/M|unp:74256/H\",\"trust\":\"ugp:US\",\"general\":\"duc_1h:10|duc_1d:30\",\"allow\":\"0\"},\"clientData\":{\"appBundleId\":\"com.mydomain.myapp\",\"appVersion\":\"1.23\",\"sdkVersion\":\"4.7.1\",\"telemetryType\":\"2\"},\"botData\":{\"botScore\":\"100\",\"responseSegment\":\"3\"}}",
                 "start": "2017-04-04T10:57:02.000Z"
             },
             "http": {
                 "request": {
                     "id": "1158db1758e37bfe67b7c09",
                     "method": "GET"
                 },
                 "response": {
                     "bytes": 266,
                     "status_code": 200
                 },
                 "version": "1.1"
             },
             "network": {
                 "protocol": "http",
                 "transport": "tcp"
             },
             "observer": {
                 "type": "proxy",
                 "vendor": "akamai"
             },
             "related": {
                 "ip": [
                     "52.91.36.10"
                 ]
             },
             "source": {
                 "address": "52.91.36.10",
                 "as": {
-                    "number": 14618,
-                    "organization": {
-                        "name": "Amazon.com, Inc."
-                    }
+                    "number": 14618
                 },
                 "geo": {
-                    "city_name": "Ashburn",
-                    "continent_name": "North America",
+                    "city_name": "ASHBURN",
                     "country_iso_code": "US",
-                    "country_name": "United States",
-                    "location": {
-                        "lat": 39.0481,
-                        "lon": -77.4728
-                    },
-                    "region_iso_code": "US-VA",
-                    "region_name": "Virginia"
+                    "region_iso_code": "-VA"
                 },
                 "ip": "52.91.36.10"
             },
             "tags": [
                 "preserve_original_event"
             ],
             "url": {
                 "domain": "www.hmapi.com",
                 "full": "www.hmapi.com/?option=com_jce%20telnet.exe",
                 "path": "/",
                 "port": 80,
                 "query": "option=com_jce telnet.exe"
             }
         },
         {
             "@timestamp": "2016-08-11T13:45:33.026Z",
             "akamai": {
                 "siem": {
                     "bot": {
                         "response_segment": 3,
                         "score": 100
                     },
                     "client_data": {
                         "app_bundle_id": "com.mydomain.myapp",
                         "app_version": "1.23",
                         "sdk_version": "4.7.1",
                         "telemetry_type": 2
                     },
                     "config_id": "6724",
                     "policy_id": "scoe_5426",
                     "request": {
                         "headers": {
                             "Accept": "text/html,application/xhtml xml",
                             "User-Agent": "BOT/0.1 (BOT for JCE)"
                         }
                     },
                     "response": {
                         "headers": {
                             "Content-Type": "text/html",
                             "Mime-Version": "1.0",
                             "Server": "AkamaiGHost"
                         }
                     },
                     "rules": [
                         {
                             "ruleActions": "ALERT",
                             "ruleData": "alert(",
                             "ruleMessages": "Cross-site Scripting (XSS) Attack",
                             "ruleSelectors": "ARGS:a",
                             "ruleTags": "WEB_ATTACK/XSS",
                             "rules": "950004"
                         },
                         {
                             "ruleActions": "DENY",
                             "ruleData": "curl",
                             "ruleMessages": "Request Indicates an automated program explored the site",
                             "ruleSelectors": "REQUEST_HEADERS:User-Agent",
                             "ruleTags": "AUTOMATION/MISC",
                             "rules": "990011"
                         }
                     ],
                     "user_risk": {
                         "allow": 0,
                         "general": {
                             "duc_1d": "30",
                             "duc_1h": "10"
                         },
                         "risk": {
                             "udfp": "1325gdg4g4343g/M",
                             "unp": "74256/H"
                         },
                         "score": 75,
                         "status": 0,
                         "trust": {
                             "ugp": "US"
                         },
                         "uuid": "964d54b7-0821-413a-a4d6-8131770ec8d5"
                     }
                 }
             },
             "client": {
                 "address": "52.91.36.10",
                 "as": {
-                    "number": 14618,
-                    "organization": {
-                        "name": "Amazon.com, Inc."
-                    }
+                    "number": 12271
                 },
                 "geo": {
-                    "city_name": "Ashburn",
-                    "continent_name": "North America",
+                    "city_name": "NEWYORK",
                     "country_iso_code": "US",
-                    "country_name": "United States",
-                    "location": {
-                        "lat": 39.0481,
-                        "lon": -77.4728
-                    },
-                    "region_iso_code": "US-VA",
-                    "region_name": "Virginia"
+                    "region_iso_code": "-NY"
                 },
                 "ip": "52.91.36.10"
             },
             "ecs": {
                 "version": "1.12.0"
             },
             "event": {
                 "category": "network",
                 "id": "2ab418ac8515f33",
                 "kind": "event",
                 "original": "{\"format\":\"json\",\"type\":\"akamai_siem\",\"version\":\"1.0\",\"attackData\":{\"clientIP\":\"52.91.36.10\",\"configId\":\"6724\",\"policyId\":\"scoe_5426\",\"ruleActions\":\"QUxFUlQ;REVOWQ==\",\"ruleData\":\"YWxlcnQo;Y3VybA==\",\"ruleMessages\":\"Q3Jvc3Mtc2l0ZSBTY3 JpcHRpbmcgKFhTUykgQXR0YWNr; UmVxdWVzdCBJbmRpY2F0ZXMgYW4 gYXV0b21hdGVkIHByb2 dyYW0gZXhwbG9yZWQgdGhlIHNpdGU=\",\"ruleSelectors\":\"QVJHUzph;UkVRVUVTVF9IRU FERVJTOlVzZXItQWdlbnQ=\",\"ruleTags\":\"V0VCX0FUVEFDSy9YU1M=;QV VUT01BVElPTi9NSVND\",\"ruleVersions\":\";\",\"rules\":\"OTUwMDA0;OTkwMDEx\"},\"geo\":{\"asn\":\"12271\",\"city\":\"NEWYORK\",\"continent\":\"NA\",\"country\":\"US\",\"regionCode\":\"NY\"},\"httpMessage\":{\"bytes\":\"34523\",\"host\":\"www.example.com\",\"method\":\"POST\",\"path\":\"/examples/1/\",\"port\":\"80\",\"protocol\":\"http/2\",\"query\":\"a%3D..%2F..%2F..%2Fetc%2Fpasswd\",\"requestHeaders\":\"User-Agent%3a%20BOT%2f0.1%20(BOT%20for%20JCE)%0d%0aAccept%3a%20text%2fhtml,application%2fxhtml+xml\",\"requestId\":\"2ab418ac8515f33\",\"responseHeaders\":\"Server%3a%20AkamaiGHost%0d%0aMime-Version%3a%201.0%0d%0aContent-Type%3a%20text%2fhtml\",\"start\":\"1470923133.026\",\"status\":\"301\",\"tls\": \"TLSv1.2\"},\"userRiskData\":{\"uuid\":\"964d54b7-0821-413a-a4d6-8131770ec8d5\",\"status\":\"0\",\"score\":\"75\",\"risk\":\"udfp:1325gdg4g4343g/M|unp:74256/H\",\"trust\":\"ugp:US\",\"general\":\"duc_1h:10|duc_1d:30\",\"allow\":\"0\"},\"clientData\":{\"appBundleId\":\"com.mydomain.myapp\",\"appVersion\":\"1.23\",\"sdkVersion\":\"4.7.1\",\"telemetryType\":\"2\"},\"botData\":{\"botScore\":\"100\",\"responseSegment\":\"3\"}}",
                 "start": "2016-08-11T13:45:33.026Z"
             },
             "http": {
                 "request": {
                     "id": "2ab418ac8515f33",
                     "method": "POST"
                 },
                 "response": {
                     "bytes": 34523,
                     "status_code": 301
                 },
                 "version": "2"
             },
             "network": {
                 "protocol": "http",
                 "transport": "tcp"
             },
             "observer": {
                 "type": "proxy",
                 "vendor": "akamai"
             },
             "related": {
                 "ip": [
                     "52.91.36.10"
                 ]
             },
             "source": {
                 "address": "52.91.36.10",
                 "as": {
-                    "number": 14618,
-                    "organization": {
-                        "name": "Amazon.com, Inc."
-                    }
+                    "number": 12271
                 },
                 "geo": {
-                    "city_name": "Ashburn",
-                    "continent_name": "North America",
+                    "city_name": "NEWYORK",
                     "country_iso_code": "US",
-                    "country_name": "United States",
-                    "location": {
-                        "lat": 39.0481,
-                        "lon": -77.4728
-                    },
-                    "region_iso_code": "US-VA",
-                    "region_name": "Virginia"
+                    "region_iso_code": "-NY"
                 },
                 "ip": "52.91.36.10"
             },
             "tags": [
                 "preserve_original_event"
             ],
             "tls": {
                 "version": "1.2",
                 "version_protocol": "tls"
             },
             "url": {
                 "domain": "www.example.com",
                 "full": "www.example.com/examples/1/?a%3D..%2F..%2F..%2Fetc%2Fpasswd",
                 "path": "/examples/1/",
                 "port": 80,
                 "query": "a=../../../etc/passwd"
             }
         },
         null
     ]
 }

Steps errors

Expand to view the steps failures

`Test integration: akamai`

Took 1 min 24 sec . View more details here
Description: eval "$(../../build/elastic-package stack shellinit)" ../../build/elastic-package test -v --report-format xUnit --report-output file --test-coverage

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2021-12-15T11:08:18Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-12-15T09:45:27.784+0000
Duration: 83 min 34 sec
Commit: 3c78cae

Test stats 🧪

Test	Results
Failed	0
Passed	3685
Skipped	5
Total	3690

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

Generate geo ip test results with the new db

56b113f

marc-gr added enhancement New feature or request Team:Integrations Label for the Integrations team Team:Security-External Integrations labels Dec 14, 2021

marc-gr requested a review from mtojek December 14, 2021 15:25

marc-gr added 2 commits December 14, 2021 16:26

Add changelog entries

e3e42df

Update elastic-package

6b12a54

marc-gr added 5 commits December 14, 2021 17:05

Cleanup go.sum

e45d750

Merge remote-tracking branch 'upstream/master' into update-geoip

b33bcbf

Fix test files for mattermost, netflow, atlassian_jira, and tenable_sc

296efbc

Fix tests for akamai

d77ce4b

Merge remote-tracking branch 'upstream/master' into update-geoip

3c78cae

mtojek approved these changes Dec 15, 2021

View reviewed changes

mtojek merged commit a01caea into elastic:master Dec 15, 2021

marc-gr deleted the update-geoip branch December 15, 2021 10:33

andrewkroh mentioned this pull request May 2, 2022

[production] Promote many packages from snapshot elastic/package-storage#4302

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generate geo ip test results with the new db#2339

Generate geo ip test results with the new db#2339
mtojek merged 8 commits intoelastic:masterfrom
marc-gr:update-geoip

marc-gr commented Dec 14, 2021 •

edited

Loading

Uh oh!

elasticmachine commented Dec 14, 2021

Uh oh!

elasticmachine commented Dec 14, 2021

Uh oh!

elasticmachine commented Dec 14, 2021 •

edited

Loading

Build stats

Test stats 🧪

`Check integrations / akamai / akamai: check / pipeline test: test-http-json.log – akamai.siem`

`Test integration: akamai`

Uh oh!

elasticmachine commented Dec 15, 2021

Build stats

Test stats 🧪

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

marc-gr commented Dec 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Checklist

Author's Checklist

Uh oh!

elasticmachine commented Dec 14, 2021

Uh oh!

elasticmachine commented Dec 14, 2021

Uh oh!

elasticmachine commented Dec 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💔 Tests Failed

Build stats

Test stats 🧪

Test errors

Check integrations / akamai / akamai: check / pipeline test: test-http-json.log – akamai.siem

Steps errors

Test integration: akamai

🤖 GitHub comments

Uh oh!

elasticmachine commented Dec 15, 2021

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

marc-gr commented Dec 14, 2021 •

edited

Loading

elasticmachine commented Dec 14, 2021 •

edited

Loading

`Check integrations / akamai / akamai: check / pipeline test: test-http-json.log – akamai.siem`

`Test integration: akamai`