feat(lambda-promtail): Adding S3 log parser support for AWS GuardDuty

[samuelebstein]

This pull request introduces the following changes to the lambda-promtail module:

1. Adding Terraform Variable:

   • Introduces a new Terraform variable to allow a different filter suffix for the S3 bucket notification resource. This enhancement provides more flexibility in configuring S3 bucket notifications.

2. Adding GuardDuty Log Type:

   • Adds support for parsing GuardDuty log types in the S3 log parser. This ensures that Promtail can push GuardDuty findings logs to Loki for monitoring and analysis.

Which issue(s) this PR fixes:
Fixes # #13129

Why this PR is needed

• The addition of the filter suffix variable allows users to customize the suffix used for S3 bucket notifications, which is particularly useful for different logging requirements and setups.
• Including GuardDuty log types in the S3 log parser expands Promtail's capabilities, enabling it to handle and forward GuardDuty findings logs, which are crucial for security monitoring.

Checklist

• Added the new Terraform variable to the lambda-promtail module.
• Updated the S3 log parser to include GuardDuty log types.
• Tested the changes to ensure they work as expected.
• Updated documentation.

Upgrading Steps

If these changes affect the default configuration, metrics names, log lines used in dashboards or alerts, configuration parameters, or API endpoints, please document what has changed and what needs to be done in the upgrade guide.

• Default configuration values: None affected.
• Metric names or label names: None affected.
• Changes to existing log lines: None affected.
• Configuration parameters: Added new variable for filter suffix with default value that is the same as the previously hardcoded value.
• Breaking changes to HTTP or gRPC API endpoints: None.

Please review the changes and let me know if there are any questions or concerns. Thank you!

[CLAassistant]

All committers have signed the CLA.

[evilr00t]

Bump, any update on this? GuardDuty findings support would be great to have!

edit: I've made a fork and applied this patch, so far it looks good and guardduty logs in loki seems to be fine! Ruler also does not complain.

I can step in and fix issues in the comments - would that be ok and have chances to be merged?

[evilr00t]

I've noticed that for some logs, wrong timestamp is being used - in our case launchTime from Instance resource.

Reading the code I see promtail is already aware of such behaviour but for CloudTrail logs, I've added this parser for GuardDuty log types and so far it looks ok.

diff --git a/tools/lambda-promtail/lambda-promtail/s3.go b/tools/lambda-promtail/lambda-promtail/s3.go
index 1ddac3500..c4fdd21ca 100644
--- a/tools/lambda-promtail/lambda-promtail/s3.go
+++ b/tools/lambda-promtail/lambda-promtail/s3.go
@@ -179,7 +179,7 @@ func parseS3Log(ctx context.Context, b *batch, labels map[string]string, obj io.
        ls = applyLabels(ls)

        // extract the timestamp of the nested event and sends the rest as raw json
-       if labels["type"] == CLOUDTRAIL_LOG_TYPE {
+       if labels["type"] == CLOUDTRAIL_LOG_TYPE || labels["type"] == GUARDDUTY_LOG_TYPE {
                records := make(chan Record)
                jsonStream := NewJSONStream(records)
                go jsonStream.Start(gzreader, parser.skipHeaderCount)
(END)

looks like quick fix for now - FYI.

[samuelebstein]

@evilr00t

Bump, any update on this? GuardDuty findings support would be great to have!

edit: I've made a fork and applied this patch, so far it looks good and guardduty logs in loki seems to be fine! Ruler also does not complain.

I can step in and fix issues in the comments - would that be ok and have chances to be merged?

That would be great. Please do. I created a fork and used it and then had to prioritize other work. But would love this to get merged

[cstyan]

@samuelebstein it looks like @evilr00t has an additional fix we should add in as well? after that I think we're probably good to go

[cstyan]

I think you might need to also rebase on main to get CI to run properly

[samuelebstein]

@cstyan all comments resolved. Feel free to merge now!

[cstyan]

@samuelebstein in the future please try and avoid force pushing during reviews, it makes it harder to see what has changed since the last review

[evilr00t]

Sorry guys for being silent, didn't notice your conversation! Kudos on getting this merged, will check that soon!