Skip to content

Commit ad4597c

Browse files
kaiyan-shengkaiyan-sheng
authored
[Filebeat] Add cloudwatch fileset in aws module (#16579)
* Add cloudwatch fileset in aws module
1 parent 8cb8e1d commit ad4597c

File tree

22 files changed

+517
-78
lines changed

22 files changed

+517
-78
lines changed

‎CHANGELOG.next.asciidoc‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -164,6 +164,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
164164
- Improve ECS field mappings in aws module. {issue}16154[16154] {pull}16307[16307]
165165
- Improve ECS categorization field mappings in googlecloud module. {issue}16030[16030] {pull}16500[16500]
166166
- Improve ECS field mappings in haproxy module. {issue}16162[16162] {pull}16529[16529]
167+
- Add cloudwatch fileset and ec2 fileset in aws module. {issue}13716[13716] {pull}16579[16579]
167168
- Improve ECS categorization field mappings in kibana module. {issue}16168[16168] {pull}16652[16652]
168169
- Improve the decode_cef processor by reducing the number of memory allocations. {pull}16587[16587]
169170
- Add `cloudfoundry` input to send events from Cloud Foundry. {pull}16586[16586]

‎filebeat/docs/fields.asciidoc‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1309,6 +1309,29 @@ type: keyword
13091309
--
13101310
Identifies the VPC endpoint in which requests were made from a VPC to another AWS service, such as Amazon S3.
13111311
1312+
type: keyword
1313+
1314+
--
1315+
1316+
[float]
1317+
=== cloudwatch
1318+
1319+
Fields for AWS CloudWatch logs.
1320+
1321+
1322+
[float]
1323+
=== ec2
1324+
1325+
Fields for AWS EC2 logs in CloudWatch.
1326+
1327+
1328+
1329+
*`aws.ec2.ip_address`*::
1330+
+
1331+
--
1332+
The internet address of the requester.
1333+
1334+
13121335
type: keyword
13131336
13141337
--

‎filebeat/docs/modules/aws.asciidoc‎

Lines changed: 51 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ Example config:
3333
[source,yaml]
3434
----
3535
- module: aws
36-
s3access:
36+
cloudtrail:
3737
enabled: false
3838
#var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue
3939
#var.shared_credential_file: /etc/filebeat/aws_credentials
@@ -42,50 +42,51 @@ Example config:
4242
#var.api_timeout: 120s
4343
#var.endpoint: amazonaws.com
4444
45-
elb:
45+
cloudwatch:
4646
enabled: false
47-
48-
# AWS SQS queue url
4947
#var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue
48+
#var.shared_credential_file: /etc/filebeat/aws_credentials
49+
#var.credential_profile_name: fb-aws
50+
#var.visibility_timeout: 300s
51+
#var.api_timeout: 120s
52+
#var.endpoint: amazonaws.com
5053
51-
# Filename of AWS credential file
52-
# If not set "$HOME/.aws/credentials" is used on Linux/Mac
53-
# "%UserProfile%\.aws\credentials" is used on Windows
54-
# var.shared_credential_file: /etc/filebeat/aws_credentials
55-
56-
# Profile name for aws credential
57-
# If not set the default profile is used
58-
# var.credential_profile_name: fb-aws
59-
60-
vpcflow:
54+
ec2:
6155
enabled: false
62-
63-
# AWS SQS queue url
6456
#var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue
57+
#var.shared_credential_file: /etc/filebeat/aws_credentials
58+
#var.credential_profile_name: fb-aws
59+
#var.visibility_timeout: 300s
60+
#var.api_timeout: 120s
61+
#var.endpoint: amazonaws.com
6562
66-
# Filename of AWS credential file
67-
# If not set "$HOME/.aws/credentials" is used on Linux/Mac
68-
# "%UserProfile%\.aws\credentials" is used on Windows
69-
# var.shared_credential_file: /etc/filebeat/aws_credentials
70-
71-
# Profile name for aws credential
72-
# If not set the default profile is used
73-
# var.credential_profile_name: fb-aws
74-
75-
cloudtrail:
63+
elb:
7664
enabled: false
65+
#var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue
66+
#var.shared_credential_file: /etc/filebeat/aws_credentials
67+
#var.credential_profile_name: fb-aws
68+
#var.visibility_timeout: 300s
69+
#var.api_timeout: 120s
70+
#var.endpoint: amazonaws.com
7771
78-
# AWS SQS queue url
72+
s3access:
73+
enabled: false
7974
#var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue
75+
#var.shared_credential_file: /etc/filebeat/aws_credentials
76+
#var.credential_profile_name: fb-aws
77+
#var.visibility_timeout: 300s
78+
#var.api_timeout: 120s
79+
#var.endpoint: amazonaws.com
8080
81-
# Filename of AWS credential file
82-
# If not set "$HOME/.aws/credentials" is used on Linux/Mac
83-
# "%UserProfile%\.aws\credentials" is used on Windows
84-
# var.shared_credential_file: /etc/filebeat/aws_credentials
81+
vpcflow:
82+
enabled: false
83+
#var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue
84+
#var.shared_credential_file: /etc/filebeat/aws_credentials
85+
#var.credential_profile_name: fb-aws
86+
#var.visibility_timeout: 300s
87+
#var.api_timeout: 120s
88+
#var.endpoint: amazonaws.com
8589
86-
# Profile name for aws credential
87-
# If not set the default profile is used
88-
# var.credential_profile_name: fb-aws
8990
----
9091

9192
*`var.queue_url`*::
@@ -122,6 +123,22 @@ The `cloudtrail` fileset does not read the CloudTrail Digest files
122123
that are delivered to the S3 bucket when Log File Integrity is turned
123124
on, it only reads the CloudTrail logs.
124125

126+
[float]
127+
=== cloudwatch fileset
128+
129+
Users can use Amazon CloudWatch Logs to monitor, store, and access log files
130+
from different sources. Export logs from log groups to an Amazon S3 bucket which
131+
has SQS notification setup already. This fileset will parse these logs into
132+
`timestamp` and `message` field.
133+
134+
[float]
135+
=== ec2 fileset
136+
137+
This fileset is specifically for EC2 logs stored in AWS CloudWatch. Export logs
138+
from log groups to Amazon S3 bucket which has SQS notification setup already.
139+
With this fileset, EC2 logs will be parsed into fields like `ip`
140+
and `program_name`. For logs from other services, please use `cloudwatch` fileset.
141+
125142
[float]
126143
=== elb fileset
127144

‎x-pack/filebeat/filebeat.reference.yml‎

Lines changed: 29 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -96,7 +96,33 @@ filebeat.modules:
9696

9797
#--------------------------------- AWS Module ---------------------------------
9898
- module: aws
99-
s3access:
99+
cloudtrail:
100+
enabled: false
101+
102+
# AWS SQS queue url
103+
#var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue
104+
105+
# Filename of AWS credential file
106+
# If not set "$HOME/.aws/credentials" is used on Linux/Mac
107+
# "%UserProfile%\.aws\credentials" is used on Windows
108+
#var.shared_credential_file: /etc/filebeat/aws_credentials
109+
110+
# Profile name for aws credential
111+
# If not set the default profile is used
112+
#var.credential_profile_name: fb-aws
113+
114+
# The duration that the received messages are hidden from ReceiveMessage request
115+
# Default to be 300s
116+
#var.visibility_timeout: 300s
117+
118+
# Maximum duration before AWS API request will be interrupted
119+
# Default to be 120s
120+
#var.api_timeout: 120s
121+
122+
# Custom endpoint used to access AWS APIs
123+
#var.endpoint: amazonaws.com
124+
125+
cloudwatch:
100126
enabled: false
101127

102128
# AWS SQS queue url
@@ -148,7 +174,7 @@ filebeat.modules:
148174
# Custom endpoint used to access AWS APIs
149175
#var.endpoint: amazonaws.com
150176

151-
vpcflow:
177+
s3access:
152178
enabled: false
153179

154180
# AWS SQS queue url
@@ -174,7 +200,7 @@ filebeat.modules:
174200
# Custom endpoint used to access AWS APIs
175201
#var.endpoint: amazonaws.com
176202

177-
cloudtrail:
203+
vpcflow:
178204
enabled: false
179205

180206
# AWS SQS queue url

‎x-pack/filebeat/module/aws/_meta/config.yml‎

Lines changed: 29 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,31 @@
11
- module: aws
2-
s3access:
2+
cloudtrail:
3+
enabled: false
4+
5+
# AWS SQS queue url
6+
#var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue
7+
8+
# Filename of AWS credential file
9+
# If not set "$HOME/.aws/credentials" is used on Linux/Mac
10+
# "%UserProfile%\.aws\credentials" is used on Windows
11+
#var.shared_credential_file: /etc/filebeat/aws_credentials
12+
13+
# Profile name for aws credential
14+
# If not set the default profile is used
15+
#var.credential_profile_name: fb-aws
16+
17+
# The duration that the received messages are hidden from ReceiveMessage request
18+
# Default to be 300s
19+
#var.visibility_timeout: 300s
20+
21+
# Maximum duration before AWS API request will be interrupted
22+
# Default to be 120s
23+
#var.api_timeout: 120s
24+
25+
# Custom endpoint used to access AWS APIs
26+
#var.endpoint: amazonaws.com
27+
28+
cloudwatch:
329
enabled: false
430

531
# AWS SQS queue url
@@ -51,7 +77,7 @@
5177
# Custom endpoint used to access AWS APIs
5278
#var.endpoint: amazonaws.com
5379

54-
vpcflow:
80+
s3access:
5581
enabled: false
5682

5783
# AWS SQS queue url
@@ -77,7 +103,7 @@
77103
# Custom endpoint used to access AWS APIs
78104
#var.endpoint: amazonaws.com
79105

80-
cloudtrail:
106+
vpcflow:
81107
enabled: false
82108

83109
# AWS SQS queue url

‎x-pack/filebeat/module/aws/_meta/docs.asciidoc‎

Lines changed: 51 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ Example config:
2828
[source,yaml]
2929
----
3030
- module: aws
31-
s3access:
31+
cloudtrail:
3232
enabled: false
3333
#var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue
3434
#var.shared_credential_file: /etc/filebeat/aws_credentials
@@ -37,50 +37,51 @@ Example config:
3737
#var.api_timeout: 120s
3838
#var.endpoint: amazonaws.com
3939
40-
elb:
40+
cloudwatch:
4141
enabled: false
42-
43-
# AWS SQS queue url
4442
#var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue
43+
#var.shared_credential_file: /etc/filebeat/aws_credentials
44+
#var.credential_profile_name: fb-aws
45+
#var.visibility_timeout: 300s
46+
#var.api_timeout: 120s
47+
#var.endpoint: amazonaws.com
4548
46-
# Filename of AWS credential file
47-
# If not set "$HOME/.aws/credentials" is used on Linux/Mac
48-
# "%UserProfile%\.aws\credentials" is used on Windows
49-
# var.shared_credential_file: /etc/filebeat/aws_credentials
50-
51-
# Profile name for aws credential
52-
# If not set the default profile is used
53-
# var.credential_profile_name: fb-aws
54-
55-
vpcflow:
49+
ec2:
5650
enabled: false
57-
58-
# AWS SQS queue url
5951
#var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue
52+
#var.shared_credential_file: /etc/filebeat/aws_credentials
53+
#var.credential_profile_name: fb-aws
54+
#var.visibility_timeout: 300s
55+
#var.api_timeout: 120s
56+
#var.endpoint: amazonaws.com
6057
61-
# Filename of AWS credential file
62-
# If not set "$HOME/.aws/credentials" is used on Linux/Mac
63-
# "%UserProfile%\.aws\credentials" is used on Windows
64-
# var.shared_credential_file: /etc/filebeat/aws_credentials
65-
66-
# Profile name for aws credential
67-
# If not set the default profile is used
68-
# var.credential_profile_name: fb-aws
69-
70-
cloudtrail:
58+
elb:
7159
enabled: false
60+
#var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue
61+
#var.shared_credential_file: /etc/filebeat/aws_credentials
62+
#var.credential_profile_name: fb-aws
63+
#var.visibility_timeout: 300s
64+
#var.api_timeout: 120s
65+
#var.endpoint: amazonaws.com
7266
73-
# AWS SQS queue url
67+
s3access:
68+
enabled: false
7469
#var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue
70+
#var.shared_credential_file: /etc/filebeat/aws_credentials
71+
#var.credential_profile_name: fb-aws
72+
#var.visibility_timeout: 300s
73+
#var.api_timeout: 120s
74+
#var.endpoint: amazonaws.com
7575
76-
# Filename of AWS credential file
77-
# If not set "$HOME/.aws/credentials" is used on Linux/Mac
78-
# "%UserProfile%\.aws\credentials" is used on Windows
79-
# var.shared_credential_file: /etc/filebeat/aws_credentials
76+
vpcflow:
77+
enabled: false
78+
#var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue
79+
#var.shared_credential_file: /etc/filebeat/aws_credentials
80+
#var.credential_profile_name: fb-aws
81+
#var.visibility_timeout: 300s
82+
#var.api_timeout: 120s
83+
#var.endpoint: amazonaws.com
8084
81-
# Profile name for aws credential
82-
# If not set the default profile is used
83-
# var.credential_profile_name: fb-aws
8485
----
8586

8687
*`var.queue_url`*::
@@ -117,6 +118,22 @@ The `cloudtrail` fileset does not read the CloudTrail Digest files
117118
that are delivered to the S3 bucket when Log File Integrity is turned
118119
on, it only reads the CloudTrail logs.
119120

121+
[float]
122+
=== cloudwatch fileset
123+
124+
Users can use Amazon CloudWatch Logs to monitor, store, and access log files
125+
from different sources. Export logs from log groups to an Amazon S3 bucket which
126+
has SQS notification setup already. This fileset will parse these logs into
127+
`timestamp` and `message` field.
128+
129+
[float]
130+
=== ec2 fileset
131+
132+
This fileset is specifically for EC2 logs stored in AWS CloudWatch. Export logs
133+
from log groups to Amazon S3 bucket which has SQS notification setup already.
134+
With this fileset, EC2 logs will be parsed into fields like `ip`
135+
and `program_name`. For logs from other services, please use `cloudwatch` fileset.
136+
120137
[float]
121138
=== elb fileset
122139

‎x-pack/filebeat/module/aws/cloudwatch/_meta/fields.yml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
- name: cloudwatch
2+
type: group
3+
release: beta
4+
default_field: false
5+
description: >
6+
Fields for AWS CloudWatch logs.
7+
fields:

0 commit comments

Comments
 (0)