Skip to content

[GCP] Add GCP Load Balancer Logs Datastream #3493

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
endorama merged 14 commits into from
Aug 23, 2022
Merged

[GCP] Add GCP Load Balancer Logs Datastream #3493

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
aef6d7d
Add GCP Load Balancer Datastream
legoguy1000 Jun 8, 2022
c1619ae
update changelog
legoguy1000 Jun 8, 2022
19b9832
add additional fields, fix tests
legoguy1000 Jun 9, 2022
f94ebbe
fix sample event
legoguy1000 Jun 9, 2022
b7b276d
fix formatting
legoguy1000 Jul 11, 2022
b37e536
rebuild docs
legoguy1000 Jul 12, 2022
6ae406b
change to Load Balancing
legoguy1000 Jul 19, 2022
2fc443c
rebase
legoguy1000 Jul 19, 2022
2d17789
Merge branch 'main' into gcp-load-balancers
endorama Jul 25, 2022
e0953af
Merge branch 'main' into gcp-load-balancers
endorama Aug 1, 2022
c634df6
Merge remote-tracking branch 'upstream/main' into gcp-load-balancers
legoguy1000 Aug 9, 2022
b7c9b43
update docs
legoguy1000 Aug 9, 2022
b49ce00
update manifest
legoguy1000 Aug 9, 2022
cf4bf7d
resolve comments
legoguy1000 Aug 12, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions packages/gcp/_dev/build/docs/loadbalancing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,13 @@
# Load Balancing

## Logs

The `loadbalancing_logs` dataset collects logs of the requests sent to and handled by GCP Load Balancers.

{{event "loadbalancing_logs"}}

{{fields "loadbalancing_logs"}}

## Metrics

The `loadbalancing_metrics` dataset fetches HTTPS, HTTP, and Layer 3 metrics from [Load Balancing](https://cloud.google.com/load-balancing/) in Google Cloud Platform. It contains all metrics exported from the [GCP Load Balancing Monitoring API](https://cloud.google.com/monitoring/api/metrics_gcp#gcp-loadbalancing).
Expand Down
14 changes: 14 additions & 0 deletions packages/gcp/_dev/deploy/docker/docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,3 +61,17 @@ services:
- /sample_logs/vpcflow.log
depends_on:
- gcppubsub-emulator
gcppubsub-load-balancer:
image: docker.elastic.co/observability/stream:v0.7.0
volumes:
- ./sample_logs:/sample_logs:ro
command:
- log
- --retry=30
- --addr=gcppubsub-emulator:8681
- -p=gcppubsub
- --gcppubsub-clear=true
- --gcppubsub-project=load_balancer
- /sample_logs/load_balancer.log
depends_on:
- gcppubsub-emulator
3 changes: 3 additions & 0 deletions packages/gcp/_dev/deploy/docker/sample_logs/load_balancer.log
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{"insertId":"1oek5rg3l3fxj7","jsonPayload":{"@type":"type.googleapis.com/google.cloud.loadbalancin,g.type.LoadBalancerLogEntry","cacheId":"SFO-fbae48ad","statusDetails":"response_from_cache"},"httpRequest":{"requestMethod":"GET","requestUrl":"http://81.2.69.193/static/us/three-cats.jpg","requestSize":"577","status":304,"responseSize":"157","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36","remoteIp":"89.160.20.156","cacheHit":true,"cacheLookup":true},"resource":{"type":"http_load_balancer","labels":{"zone":"global","url_map_name":"URL_MAP_NAME","forwarding_rule_name":"FORWARDING_RULE_NAME","target_proxy_name":"TARGET_PROXY_NAME","backend_service_name":"","project_id":"PROJECT_ID"}},"timestamp":"2020-06-08T23:41:30.078651Z","severity":"INFO","logName":"projects/PROJECT_ID/logs/requests","trace":"projects/PROJECT_ID/traces/241d69833e64b3bf83fabac8c873d992","receiveTimestamp":"2020-06-08T23:41:30.588272510Z","spanId":"7b6537d3672e08e1"}
{"insertId":"1oek5rg3l3fxj7","jsonPayload":{"@type":"type.googleapis.com/google.cloud.loadbalancin,g.type.LoadBalancerLogEntry","cacheId":"SFO-fbae48ad","statusDetails":"response_from_cache"},"httpRequest":{"requestMethod":"GET","requestUrl":"http://pictures.example.com/static/us/three-cats.jpg","requestSize":"577","status":304,"responseSize":"157","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36","remoteIp":"89.160.20.156","cacheHit":true,"cacheLookup":true},"resource":{"type":"http_load_balancer","labels":{"zone":"global","url_map_name":"URL_MAP_NAME","forwarding_rule_name":"FORWARDING_RULE_NAME","target_proxy_name":"TARGET_PROXY_NAME","backend_service_name":"","project_id":"PROJECT_ID"}},"timestamp":"2020-06-08T23:41:30.078651Z","severity":"INFO","logName":"projects/PROJECT_ID/logs/requests","trace":"projects/PROJECT_ID/traces/241d69833e64b3bf83fabac8c873d992","receiveTimestamp":"2020-06-08T23:41:30.588272510Z","spanId":"7b6537d3672e08e1"}
{"insertId":"1oek5rg3l3fxj7","jsonPayload":{"@type":"type.googleapis.com/google.cloud.loadbalancin,g.type.LoadBalancerLogEntry","cacheId":"SFO-fbae48ad","statusDetails":"response_from_cache"},"httpRequest":{"requestMethod":"GET","requestUrl":"http://81.2.69.193:8080/static/us/three-cats.jpg","requestSize":"577","status":304,"responseSize":"157","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36","remoteIp":"89.160.20.156:9989","cacheHit":true,"cacheLookup":true,"serverIp":"10.5.3.1:9090","protocol":"HTTP/2.0","referer":"https://developer.mozilla.org/en-US/docs/Web/JavaScript"},"resource":{"type":"http_load_balancer","labels":{"zone":"global","url_map_name":"URL_MAP_NAME","forwarding_rule_name":"FORWARDING_RULE_NAME","target_proxy_name":"TARGET_PROXY_NAME","backend_service_name":"","project_id":"PROJECT_ID"}},"timestamp":"2020-06-08T23:41:30.078651Z","severity":"INFO","logName":"projects/PROJECT_ID/logs/requests","trace":"projects/PROJECT_ID/traces/241d69833e64b3bf83fabac8c873d992","receiveTimestamp":"2020-06-08T23:41:30.588272510Z","spanId":"7b6537d3672e08e1"}
5 changes: 5 additions & 0 deletions packages/gcp/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "2.6.0"
changes:
- description: Add Load Balancing logs datastream
type: enhancement
link: https://github.com/elastic/integrations/pull/3493
- version: "2.5.0"
changes:
- description: Add GCP Load Balancing Metricset
Expand Down
3 changes: 3 additions & 0 deletions packages/gcp/data_stream/loadbalancing_logs/_dev/test/pipeline/test-common-config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
fields:
tags:
- preserve_original_event
3 changes: 3 additions & 0 deletions packages/gcp/data_stream/loadbalancing_logs/_dev/test/pipeline/test-load_balancer.log
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{"insertId":"1oek5rg3l3fxj7","jsonPayload":{"@type":"type.googleapis.com/google.cloud.loadbalancin,g.type.LoadBalancerLogEntry","cacheId":"SFO-fbae48ad","statusDetails":"response_from_cache"},"httpRequest":{"requestMethod":"GET","requestUrl":"http://81.2.69.193/static/us/three-cats.jpg","requestSize":"577","status":304,"responseSize":"157","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36","remoteIp":"89.160.20.156","cacheHit":true,"cacheLookup":true},"resource":{"type":"http_load_balancer","labels":{"zone":"global","url_map_name":"URL_MAP_NAME","forwarding_rule_name":"FORWARDING_RULE_NAME","target_proxy_name":"TARGET_PROXY_NAME","backend_service_name":"","project_id":"PROJECT_ID"}},"timestamp":"2020-06-08T23:41:30.078651Z","severity":"INFO","logName":"projects/PROJECT_ID/logs/requests","trace":"projects/PROJECT_ID/traces/241d69833e64b3bf83fabac8c873d992","receiveTimestamp":"2020-06-08T23:41:30.588272510Z","spanId":"7b6537d3672e08e1"}
{"insertId":"1oek5rg3l3fxj7","jsonPayload":{"@type":"type.googleapis.com/google.cloud.loadbalancin,g.type.LoadBalancerLogEntry","cacheId":"SFO-fbae48ad","statusDetails":"response_from_cache"},"httpRequest":{"requestMethod":"GET","requestUrl":"http://pictures.example.com/static/us/three-cats.jpg","requestSize":"577","status":304,"responseSize":"157","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36","remoteIp":"89.160.20.156","cacheHit":true,"cacheLookup":true},"resource":{"type":"http_load_balancer","labels":{"zone":"global","url_map_name":"URL_MAP_NAME","forwarding_rule_name":"FORWARDING_RULE_NAME","target_proxy_name":"TARGET_PROXY_NAME","backend_service_name":"","project_id":"PROJECT_ID"}},"timestamp":"2020-06-08T23:41:30.078651Z","severity":"INFO","logName":"projects/PROJECT_ID/logs/requests","trace":"projects/PROJECT_ID/traces/241d69833e64b3bf83fabac8c873d992","receiveTimestamp":"2020-06-08T23:41:30.588272510Z","spanId":"7b6537d3672e08e1"}
{"insertId":"1oek5rg3l3fxj7","jsonPayload":{"@type":"type.googleapis.com/google.cloud.loadbalancin,g.type.LoadBalancerLogEntry","cacheId":"SFO-fbae48ad","statusDetails":"response_from_cache"},"httpRequest":{"requestMethod":"GET","requestUrl":"http://81.2.69.193:8080/static/us/three-cats.jpg","requestSize":"577","status":304,"responseSize":"157","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36","remoteIp":"89.160.20.156:9989","cacheHit":true,"cacheLookup":true,"serverIp":"10.5.3.1:9090","protocol":"HTTP/2.0","referer":"https://developer.mozilla.org/en-US/docs/Web/JavaScript"},"resource":{"type":"http_load_balancer","labels":{"zone":"global","url_map_name":"URL_MAP_NAME","forwarding_rule_name":"FORWARDING_RULE_NAME","target_proxy_name":"TARGET_PROXY_NAME","backend_service_name":"","project_id":"PROJECT_ID"}},"timestamp":"2020-06-08T23:41:30.078651Z","severity":"INFO","logName":"projects/PROJECT_ID/logs/requests","trace":"projects/PROJECT_ID/traces/241d69833e64b3bf83fabac8c873d992","receiveTimestamp":"2020-06-08T23:41:30.588272510Z","spanId":"7b6537d3672e08e1"}
Loading