Skip to content

Add routing rules for WAF logs based on log format #7836

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kaiyan-sheng merged 5 commits into from
Sep 21, 2023
Merged

Add routing rules for WAF logs based on log format #7836

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
5d8789d
Add routing rules for WAF logs based on log format and remove namespace
Sep 14, 2023
2f43741
add changelog
Sep 14, 2023
7475312
simplify routing rule when checking for waf logs
Sep 18, 2023
87308eb
update kibana version
Sep 20, 2023
0247088
put namespace back
Sep 20, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions packages/awsfirehose/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,12 @@
# newer versions go on top
- version: 0.2.1
changes:
- description: Route WAF logs based on log format
type: enhancement
link: https://github.com/elastic/integrations/pull/7836
- description: Remove namespaces in routing rules
type: bugfix
Comment thread
tommyers-elastic marked this conversation as resolved.
Outdated
link: https://github.com/elastic/integrations/pull/7836
- version: 0.2.0
changes:
- description: Add support for routing api gateway logs
Expand Down
35 changes: 13 additions & 22 deletions packages/awsfirehose/data_stream/logs/routing_rules.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,14 +2,8 @@
rules:
- target_dataset: aws.firewall_logs
if: ctx.message != null && ctx.message.contains('firewall_name') && ctx.message.contains('availability_zone') && ctx.message.contains('event_timestamp') && ctx.message.contains('event')
namespace:
- "{{data_stream.namespace}}"
- default
- target_dataset: aws.cloudtrail
if: ctx['aws.cloudwatch.log_stream'] != null && ctx['aws.cloudwatch.log_stream'].contains('CloudTrail')
namespace:
- "{{data_stream.namespace}}"
- default
- target_dataset: aws.route53_public_logs
if: >-
if (ctx['aws.cloudwatch.log_stream'] == null) {
Expand All @@ -25,19 +19,13 @@
return true;
}
return false;
namespace:
- "{{data_stream.namespace}}"
- default
- target_dataset: aws.route53_resolver_logs
if: >-
ctx.message != null && ctx.message.contains('version') && ctx.message.contains('account_id') && ctx.message.contains('region') &&
ctx.message.contains('vpc_id') && ctx.message.contains('query_timestamp') && ctx.message.contains('query_name')
&& ctx.message.contains('query_type') && ctx.message.contains('query_class') && ctx.message.contains('rcode')
&& ctx.message.contains('answers') && ctx.message.contains('srcaddr') && ctx.message.contains('srcport')
&& ctx.message.contains('transport') && ctx.message.contains('srcids')
namespace:
- "{{data_stream.namespace}}"
- default
- target_dataset: aws.vpcflow
if: >-
if (ctx.message!= null) {
Expand All @@ -47,15 +35,21 @@
}
}
return false;
namespace:
- "{{data_stream.namespace}}"
- default
- target_dataset: aws.waf
# Kinesis Data Firehose stream name begins with `aws-waf-logs-`
# CloudWatch log group name begins with `aws-waf-logs-`
# Log fields:
# timestamp formatVersion webaclld terminatingRuleId terminatingRuleType action
# terminatingRuleMatchDetails httpSourceName httpSourceId ruleGroupList rateBasedRuleList
# nonTerminatingMatchingRules httpRequest labels
if: >-
(ctx['aws.kinesis.name'] != null && ctx['aws.kinesis.name'].contains('aws-waf-logs-')) || (ctx['aws.cloudwatch.log_group'] != null && ctx['aws.cloudwatch.log_group'].contains('aws-waf-logs-'))
namespace:
- "{{data_stream.namespace}}"
- default
(ctx['aws.kinesis.name'] != null && ctx['aws.kinesis.name'].contains('aws-waf-logs-'))
Comment thread
kaiyan-sheng marked this conversation as resolved.
Outdated
|| (ctx['aws.cloudwatch.log_group'] != null && ctx['aws.cloudwatch.log_group'].contains('aws-waf-logs-'))
|| (ctx.message != null && ctx.message.contains('timestamp') && ctx.message.contains('formatVersion')
&& ctx.message.contains('webaclld') && ctx.message.contains('terminatingRuleId') && ctx.message.contains('terminatingRuleType')
&& ctx.message.contains('action') && ctx.message.contains('terminatingRuleMatchDetails') && ctx.message.contains('httpSourceName')
&& ctx.message.contains('httpSourceId') && ctx.message.contains('ruleGroupList') && ctx.message.contains('rateBasedRuleList')
&& ctx.message.contains('nonTerminatingMatchingRules') && ctx.message.contains('httpRequest') && ctx.message.contains('labels'))
- target_dataset: aws.apigateway_logs
# Supported API Gateway log format by API type
# HTTP: requestId ip requestTime httpMethod routeKey status protocol responseLength
Expand All @@ -72,6 +66,3 @@
|| (ctx.message != null && ctx.message.contains('requestId') && ctx.message.contains('ip') && ctx.message.contains('caller')
&& ctx.message.contains('user') && ctx.message.contains('requestTime') && ctx.message.contains('eventType')
&& ctx.message.contains('routeKey') && ctx.message.contains('status') && ctx.message.contains('connectionId'))
namespace:
- "{{data_stream.namespace}}"
- default
2 changes: 1 addition & 1 deletion packages/awsfirehose/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
format_version: 2.9.0
name: awsfirehose
title: Amazon Kinesis Data Firehose
version: 0.2.0
version: 0.2.1
description: Stream logs from Amazon Kinesis Data Firehose into Elastic Cloud.
type: integration
categories:
Expand Down