[AWS] Add Route 53 Public Zone Datastream by legoguy1000 · Pull Request #2316 · elastic/integrations

legoguy1000 · 2021-12-06T02:54:03Z

What does this PR do?

Adds a new datastream for Route 53 Public Zone logs

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Screenshots

elasticmachine · 2021-12-06T02:56:54Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-12-21T23:00:35.672+0000
Duration: 23 min 4 sec
Commit: 9942aa6

Test stats 🧪

Test	Results
Failed	0
Passed	319
Skipped	0
Total	319

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

legoguy1000 · 2021-12-09T17:55:40Z

@kaiyan-sheng when u get a chance, could u take a look at this? I will fix the conflicts later today. I plan to create another PR for the private route 53 logging after this is merged.

kaiyan-sheng · 2021-12-09T18:09:27Z

@legoguy1000 Will do! Thanks for adding this and also using aws-cloudwatch input. I will read into the PR a bit more after lunch, just a quick note here: I also see this PR replacing all IPs. Maybe we can also change the IPs here in this PR?

legoguy1000 · 2021-12-09T18:11:34Z

Tracking the GeoIP stuff. I will swap them out to make full use of the processor.

packages/aws/data_stream/route53_public_logs/agent/stream/aws-cloudwatch.yml.hbs

legoguy1000 · 2021-12-09T23:29:52Z

@kaiyan-sheng should be updated per ur comments. Also if u want to merge #2323 first then I can just update this to match how you did the closest cloud watch input

legoguy1000 · 2021-12-21T14:24:57Z

@kaiyan-sheng When u get a chance, could you review?

kaiyan-sheng

@legoguy1000 Sorry for the delay!! Let's merge your PR first and then I can work on mine later!

kaiyan-sheng · 2021-12-21T16:32:27Z

/test

kaiyan-sheng · 2021-12-21T17:00:29Z

@legoguy1000 Could you rerun the test for test-route53.log please? Thanks!!

legoguy1000 · 2021-12-21T19:04:31Z

@kaiyan-sheng Should be good now.

kaiyan-sheng · 2021-12-21T21:01:16Z

Hmm seems like the expected file is still a little off:

test case failed: Expected results are different from actual ones:  {
     "expected": [
         {
             "@timestamp": "2017-12-13T08:16:02.130Z",
             "aws": {
                 "route53": {
                     "edge_location": "FRA6",
                     "hosted_zone_id": "Z123412341234"
                 }
             },
             "cloud": {
                 "provider": "aws"
             },
             "dns": {
                 "question": {
                     "name": "example.com",
                     "registered_domain": "example.com",
                     "top_level_domain": "com",
                     "type": "A"
                 },
                 "response_code": "NOERROR"
             },
             "ecs": {
                 "version": "1.12.0"
             },
             "event": {
                 "category": [
                     "network"
                 ],
                 "kind": "event",
                 "original": "1.0 2017-12-13T08:16:02.130Z Z123412341234 example.com A NOERROR UDP FRA6 89.160.20.112 -",
                 "outcome": "success",
                 "type": [
                     "protocol"
                 ]
             },
             "network": {
                 "iana_number": "17",
                 "protocol": "dns",
                 "transport": "udp",
                 "type": "ipv4"
             },
             "related": {
                 "hosts": [
                     "example.com"
                 ],
                 "ip": [
                     "89.160.20.112"
                 ]
             },
             "source": {
                 "address": "89.160.20.112",
                 "as": {
                     "number": 29518,
                     "organization": {
                         "name": "Bredband2 AB"
                     }
                 },
                 "geo": {
-                    "city_name": "Tumba",
+                    "city_name": "Linköping",
                     "continent_name": "Europe",
                     "country_iso_code": "SE",
                     "country_name": "Sweden",
                     "location": {
-                        "lat": 59.2,
-                        "lon": 17.8167
+                        "lat": 58.4167,
+                        "lon": 15.6167
                     },
-                    "region_iso_code": "SE-AB",
-                    "region_name": "Stockholm"
+                    "region_iso_code": "SE-E",
+                    "region_name": "Östergötland County"
                 },
                 "ip": "89.160.20.112"
             },
             "tags": [
                 "preserve_original_event"
             ]
         },
         {
             "@timestamp": "2017-12-13T08:15:50.235Z",
             "aws": {
                 "route53": {
                     "edge_location": "IAD12",
                     "edns_client_subnet": "192.168.222.0/24",
                     "hosted_zone_id": "Z123412341234"
                 }
             },
             "cloud": {
                 "provider": "aws"
             },
             "dns": {
                 "question": {
                     "name": "example.com",
                     "registered_domain": "example.com",
                     "top_level_domain": "com",
                     "type": "AAAA"
                 },
                 "response_code": "NOERROR"
             },
             "ecs": {
                 "version": "1.12.0"
             },
             "event": {
                 "category": [
                     "network"
                 ],
                 "kind": "event",
                 "original": "1.0 2017-12-13T08:15:50.235Z Z123412341234 example.com AAAA NOERROR TCP IAD12 89.160.20.112 192.168.222.0/24",
                 "outcome": "success",
                 "type": [
                     "protocol"
                 ]
             },
             "network": {
                 "iana_number": "6",
                 "protocol": "dns",
                 "transport": "tcp",
                 "type": "ipv4"
             },
             "related": {
                 "hosts": [
                     "example.com"
                 ],
                 "ip": [
                     "89.160.20.112"
                 ]
             },
             "source": {
                 "address": "89.160.20.112",
                 "as": {
                     "number": 29518,
                     "organization": {
                         "name": "Bredband2 AB"
                     }
                 },
                 "geo": {
-                    "city_name": "Tumba",
+                    "city_name": "Linköping",
                     "continent_name": "Europe",
                     "country_iso_code": "SE",
                     "country_name": "Sweden",
                     "location": {
-                        "lat": 59.2,
-                        "lon": 17.8167
+                        "lat": 58.4167,
+                        "lon": 15.6167
                     },
-                    "region_iso_code": "SE-AB",
-                    "region_name": "Stockholm"
+                    "region_iso_code": "SE-E",
+                    "region_name": "Östergötland County"
                 },
                 "ip": "89.160.20.112"
             },
             "tags": [
                 "preserve_original_event"
             ]
         },
         {
             "@timestamp": "2017-12-13T08:16:03.983Z",
             "aws": {
                 "route53": {
                     "edge_location": "FRA6",
                     "edns_client_subnet": "2001:db8:abcd::/48",
                     "hosted_zone_id": "Z123412341234"
                 }
             },
             "cloud": {
                 "provider": "aws"
             },
             "dns": {
                 "question": {
                     "name": "example.com",
                     "registered_domain": "example.com",
                     "top_level_domain": "com",
                     "type": "ANY"
                 },
                 "response_code": "NOERROR"
             },
             "ecs": {
                 "version": "1.12.0"
             },
             "event": {
                 "category": [
                     "network"
                 ],
                 "kind": "event",
                 "original": "1.0 2017-12-13T08:16:03.983Z Z123412341234 example.com ANY NOERROR UDP FRA6 2a02:cf40:add:4002:91f2:a9b2:e09a:6fc6 2001:db8:abcd::/48",
                 "outcome": "success",
                 "type": [
                     "protocol"
                 ]
             },
             "network": {
                 "iana_number": "17",
                 "protocol": "dns",
                 "transport": "udp",
                 "type": "ipv6"
             },
             "related": {
                 "hosts": [
                     "example.com"
                 ],
                 "ip": [
                     "2a02:cf40:add:4002:91f2:a9b2:e09a:6fc6"
                 ]
             },
             "source": {
                 "address": "2a02:cf40:add:4002:91f2:a9b2:e09a:6fc6",
-                "as": {
-                    "number": 62121,
-                    "organization": {
-                        "name": "Christian Ebsen ApS"
-                    }
-                },
                 "geo": {
                     "continent_name": "Europe",
-                    "country_iso_code": "DK",
-                    "country_name": "Denmark",
+                    "country_iso_code": "NO",
+                    "country_name": "Norway",
                     "location": {
-                        "lat": 56,
+                        "lat": 62,
                         "lon": 10
                     }
                 },
                 "ip": "2a02:cf40:add:4002:91f2:a9b2:e09a:6fc6"
             },
             "tags": [
                 "preserve_original_event"
             ]
         },
         {
             "@timestamp": "2017-12-13T08:15:50.342Z",
             "aws": {
                 "route53": {
                     "edge_location": "IAD12",
                     "edns_client_subnet": "192.168.111.0/24",
                     "hosted_zone_id": "Z123412341234"
                 }
             },
             "cloud": {
                 "provider": "aws"
             },
             "dns": {
                 "question": {
                     "name": "bad.example.com",
                     "registered_domain": "example.com",
                     "subdomain": "bad",
                     "top_level_domain": "com",
                     "type": "A"
                 },
                 "response_code": "NXDOMAIN"
             },
             "ecs": {
                 "version": "1.12.0"
             },
             "event": {
                 "category": [
                     "network"
                 ],
                 "kind": "event",
                 "original": "1.0 2017-12-13T08:15:50.342Z Z123412341234 bad.example.com A NXDOMAIN UDP IAD12 89.160.20.112 192.168.111.0/24",
                 "outcome": "failure",
                 "type": [
                     "protocol"
                 ]
             },
             "network": {
                 "iana_number": "17",
                 "protocol": "dns",
                 "transport": "udp",
                 "type": "ipv4"
             },
             "related": {
                 "hosts": [
                     "bad.example.com"
                 ],
                 "ip": [
                     "89.160.20.112"
                 ]
             },
             "source": {
                 "address": "89.160.20.112",
                 "as": {
                     "number": 29518,
                     "organization": {
                         "name": "Bredband2 AB"
                     }
                 },
                 "geo": {
-                    "city_name": "Tumba",
+                    "city_name": "Linköping",
                     "continent_name": "Europe",
                     "country_iso_code": "SE",
                     "country_name": "Sweden",
                     "location": {
-                        "lat": 59.2,
-                        "lon": 17.8167
+                        "lat": 58.4167,
+                        "lon": 15.6167
                     },
-                    "region_iso_code": "SE-AB",
-                    "region_name": "Stockholm"
+                    "region_iso_code": "SE-E",
+                    "region_name": "Östergötland County"
                 },
                 "ip": "89.160.20.112"
             },
             "tags": [
                 "preserve_original_event"
             ]
         },
         {
             "@timestamp": "2017-12-13T08:16:05.744Z",
             "aws": {
                 "route53": {
                     "edge_location": "JFK5",
                     "hosted_zone_id": "Z123412341234"
                 }
             },
             "cloud": {
                 "provider": "aws"
             },
             "dns": {
                 "question": {
                     "name": "txt.example.com",
                     "registered_domain": "example.com",
                     "subdomain": "txt",
                     "top_level_domain": "com",
                     "type": "TXT"
                 },
                 "response_code": "NOERROR"
             },
             "ecs": {
                 "version": "1.12.0"
             },
             "event": {
                 "category": [
                     "network"
                 ],
                 "kind": "event",
                 "original": "1.0 2017-12-13T08:16:05.744Z Z123412341234 txt.example.com TXT NOERROR UDP JFK5 89.160.20.112 -",
                 "outcome": "success",
                 "type": [
                     "protocol"
                 ]
             },
             "network": {
                 "iana_number": "17",
                 "protocol": "dns",
                 "transport": "udp",
                 "type": "ipv4"
             },
             "related": {
                 "hosts": [
                     "txt.example.com"
                 ],
                 "ip": [
                     "89.160.20.112"
                 ]
             },
             "source": {
                 "address": "89.160.20.112",
                 "as": {
                     "number": 29518,
                     "organization": {
                         "name": "Bredband2 AB"
                     }
                 },
                 "geo": {
-                    "city_name": "Tumba",
+                    "city_name": "Linköping",
                     "continent_name": "Europe",
                     "country_iso_code": "SE",
                     "country_name": "Sweden",
                     "location": {
-                        "lat": 59.2,
-                        "lon": 17.8167
+                        "lat": 58.4167,
+                        "lon": 15.6167
                     },
-                    "region_iso_code": "SE-AB",
-                    "region_name": "Stockholm"
+                    "region_iso_code": "SE-E",
+                    "region_name": "Östergötland County"
                 },
                 "ip": "89.160.20.112"
             },
             "tags": [
                 "preserve_original_event"
             ]
         }
     ]
 }

legoguy1000 · 2021-12-21T21:04:06Z

Could it be a stack version mismatch? I think I generated the data with a 7.16.x stack but it looks like Jenkins spun up 7.15. I can see if that makes a difference.

kaiyan-sheng · 2021-12-21T21:15:30Z

@legoguy1000 I think that's the problem hmm

legoguy1000 · 2021-12-21T22:26:37Z

7.15.0 stack didn't produce any different data for my locally run test, trying 8.0.0-SNAPSHOT.

legoguy1000 · 2021-12-21T23:06:57Z

7.15.0 stack didn't produce any different data for my locally run test, trying 8.0.0-SNAPSHOT.

Looks like I just needed to run elastic-package stack update --version 7.16.2 and now the geoIP value are consistent between 7.x and 8.0

legoguy1000 force-pushed the aws-route53-public branch from e02bae2 to 9dbe3c1 Compare December 8, 2021 17:21

legoguy1000 marked this pull request as ready for review December 8, 2021 17:21

legoguy1000 force-pushed the aws-route53-public branch from 9dbe3c1 to 70450fc Compare December 9, 2021 18:00

kaiyan-sheng reviewed Dec 9, 2021

View reviewed changes

packages/aws/data_stream/route53_public_logs/agent/stream/aws-cloudwatch.yml.hbs Outdated Show resolved Hide resolved

legoguy1000 force-pushed the aws-route53-public branch from cddc3cf to 3f0489f Compare December 9, 2021 22:10

legoguy1000 mentioned this pull request Dec 15, 2021

[AWS] Add Route 53 Resolver logs #2341

Merged

4 tasks

legoguy1000 requested a review from kaiyan-sheng December 15, 2021 21:21

legoguy1000 added 3 commits December 21, 2021 14:23

Add Route 53 Public Zone Dataset

5373318

replaces IPs

6ff225d

Add additional cloudwatch input options

b2cef6b

legoguy1000 force-pushed the aws-route53-public branch from 3f0489f to b2cef6b Compare December 21, 2021 14:23

kaiyan-sheng approved these changes Dec 21, 2021

View reviewed changes

fixed cloudwatch fields

8fa40cf

regenerate data

9942aa6

kaiyan-sheng merged commit 2065c06 into elastic:master Dec 22, 2021

legoguy1000 deleted the aws-route53-public branch December 22, 2021 14:37

Conversation

legoguy1000 commented Dec 6, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

elasticmachine commented Dec 6, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

Uh oh!

legoguy1000 commented Dec 9, 2021

Uh oh!

kaiyan-sheng commented Dec 9, 2021

Uh oh!

legoguy1000 commented Dec 9, 2021

Uh oh!

Uh oh!

legoguy1000 commented Dec 9, 2021

Uh oh!

legoguy1000 commented Dec 21, 2021

Uh oh!

kaiyan-sheng left a comment

Choose a reason for hiding this comment

Uh oh!

kaiyan-sheng commented Dec 21, 2021

Uh oh!

kaiyan-sheng commented Dec 21, 2021

Uh oh!

legoguy1000 commented Dec 21, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kaiyan-sheng commented Dec 21, 2021

Uh oh!

legoguy1000 commented Dec 21, 2021

Uh oh!

kaiyan-sheng commented Dec 21, 2021

Uh oh!

legoguy1000 commented Dec 21, 2021

Uh oh!

legoguy1000 commented Dec 21, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

legoguy1000 commented Dec 6, 2021 •

edited

Loading

elasticmachine commented Dec 6, 2021 •

edited

Loading

legoguy1000 commented Dec 21, 2021 •

edited

Loading