[Filebeat] Adding wildcard support for log_group_name in cloudwatch input #21528
Description
Find awscloudwatch log groups by discovery(pattern)
For example I have ~400 log_groups and I should to parse only ~70
My log_group contain some similar names like:
/development/application/api-south
/development/application/api-west
/development/application/web-south
.....
My filebeat.yml configuration:
- type: awscloudwatch
access_key_id: 0000000000000000000
secret_access_key: 0000000000000000000
log_group_name: /development/application/
region_name: us-east-1
scan_frequency: 10s
start_position: beginning
Got an error:
2020-10-05T14:36:28.816Z INFO [awscloudwatch] awscloudwatch/input.go:125 awscloudwatch input worker for log group: '/development/application/' has started
2020-10-05T14:36:28.816Z INFO [crawler] beater/crawler.go:108 Loading and starting Inputs completed. Enabled inputs: 1
2020-10-05T14:36:28.816Z INFO cfgfile/reload.go:164 Config reloader started
2020-10-05T14:36:28.816Z INFO cfgfile/reload.go:224 Loading of config files completed.
2020-10-05T14:36:29.337Z ERROR [awscloudwatch] awscloudwatch/input.go:189 failed FilterLogEventsRequestResourceNotFoundException: The specified log group does not exist.
status code: 400, request id: 6fb32149-c2d1-49e3-9f41-f5a30810fae3
If I set: log_group_name: /development/application/* have similar issue:
getLogEventsFromCloudWatch failed: InvalidParameterException: 1 validation error detected: Value ':/development/application/*' at 'logGroupName' failed to satisfy constraint: Member must satisfy regular expression pattern: [\.\-_/#A-Za-z0-9]+
So for my configuration, I must set up 70 input configurations (for each log_group)? Or I lost something and I may use some special pattern?