From 684c3a9352bab3140079a0058e4bea7cf39318ff Mon Sep 17 00:00:00 2001 From: Michalis Katsoulis Date: Wed, 17 Sep 2025 12:08:14 +0300 Subject: [PATCH 1/9] Chore: Update edot cloud forwarder for AWS documentation --- docs/reference/edot-cloud-forwarder/aws.md | 146 ++++++++++++++++++++- 1 file changed, 144 insertions(+), 2 deletions(-) diff --git a/docs/reference/edot-cloud-forwarder/aws.md b/docs/reference/edot-cloud-forwarder/aws.md index 20b4caad..5e5cafef 100644 --- a/docs/reference/edot-cloud-forwarder/aws.md +++ b/docs/reference/edot-cloud-forwarder/aws.md @@ -18,8 +18,12 @@ products: | Log source | Description | | --- | --- | -| VPC Flow | Logs generated by a Virtual Private Cloud (VPC) | +| VPC Flow | Logs generated by a Virtual Private Cloud (VPC) | | ELB Access | Logs generated by an Elastic Load Balancer (ELB) | +| S3 Access | Logs generated by Amazon S3 access requests | +| Cloudtrail | Logs generated by AWS CloudTrail | +| WAF | Logs generated by AWS Web Application Firewall (WAF) | +| JSON | Logs in JSON format from various AWS services | % | CloudWatch {applies_to}`product: planned` | Logs generated by AWS CloudWatch | Read on to learn how to set up {{edot-cf}} for AWS. @@ -57,6 +61,47 @@ To collect Elastic Load Balancer (ELB) Access logs, you need: - An S3 bucket to store the access logs - Access logging enabled, with the bucket as the destination +::: + +:::{tab-item} S3 Access + +To collect S3 Access logs, you need: + +- An S3 bucket with server access logging enabled +- A destination S3 bucket to store the access logs +- S3 server access logging configured to write logs to the destination bucket + +::: + +:::{tab-item} CloudTrail + +To collect CloudTrail logs, you need: + +- AWS CloudTrail enabled for your AWS account +- An S3 bucket for storing CloudTrail log files +- CloudTrail configured to deliver log files to the S3 bucket + +::: + +:::{tab-item} WAF + +To collect AWS WAF logs, you need: + +- AWS WAF web ACL (protection pack) configured +- An S3 bucket in the same AWS account as your web ACL for storing WAF logs +- The S3 bucket name must start with `aws-waf-logs-` (for example, `aws-waf-logs-my-web-acl-logs`) +- WAF logging enabled in the AWS WAF console with the S3 bucket specified as the destination + +::: + +:::{tab-item} JSON + +To collect JSON logs, you need: + +- JSON-formatted log files stored in an S3 bucket +- Logs must be valid JSON objects (one per line or array format) +- An S3 bucket where the JSON log files are uploaded + ::: +## Kibana integration setup + +After {{edot-cf}} for AWS is successfully running and forwarding logs to Elastic Observability, you can install pre-built integrations in Kibana to visualize your data with out-of-the-box dashboards and visualizations. + +### Install integrations + +To set up data visualization for your AWS logs: + +1. **Navigate to Kibana**: Log into your Elastic Cloud deployment and open Kibana. + +2. **Access Integrations**: Go to **Management** → **Integrations** in the Kibana navigation menu. + +3. **Search and install**: Search for the appropriate integration based on your log type and install it: + +| **AWS Log Type** | **Integration Name** | **Description** | +|------------------|---------------------|-----------------| +| ELB Access Logs | **AWS ELB OpenTelemetry Assets** | Dashboards and visualizations for Elastic Load Balancer logs | +| VPC Flow Logs | **AWS VPC Flow Logs OpenTelemetry Assets** | Dashboards and visualizations for VPC flow log data | + +4. **Access dashboards**: Once installed, navigate to **Dashboard** to view the pre-built dashboards for your AWS log data. + +### Benefits + +This allows you to immediately start analyzing your AWS infrastructure without building dashboards from scratch. + ## **Delete a CloudFormation stack** If you no longer need a deployed stack and want to clean up all associated resources, you can delete it using the following command: From e659654756b9f50b30327dd57d4bec27fffe3e94 Mon Sep 17 00:00:00 2001 From: Kavindu Dodanduwa Date: Wed, 24 Sep 2025 15:12:18 -0700 Subject: [PATCH 3/9] refine documentation and add error replay section Signed-off-by: Kavindu Dodanduwa --- docs/reference/edot-cloud-forwarder/aws.md | 74 ++++++++++++++++++---- 1 file changed, 63 insertions(+), 11 deletions(-) diff --git a/docs/reference/edot-cloud-forwarder/aws.md b/docs/reference/edot-cloud-forwarder/aws.md index a42320f9..1193b2b1 100644 --- a/docs/reference/edot-cloud-forwarder/aws.md +++ b/docs/reference/edot-cloud-forwarder/aws.md @@ -130,12 +130,13 @@ For S3 logs, use the following settings: | Setting | Description | | ------------------ | --- | -| `EdotCloudForwarderS3LogsType` | The encoding format for logs in the S3 bucket. Supported options:
- `vpc_flow_log`: VPC Flow logs
- `elb_access_log`: Elastic Load Balancer (ELB) Access logs
- `s3_access_log`: S3 Access logs
- `cloudtrail_log`: CloudTrail logs
- `waf_log`: AWS WAF logs
- `json`: JSON-formatted logs | -| `S3LogsJsonEncodingMode` | _(Required if `EdotCloudForwarderS3LogsType` is `json`)_
Defines how JSON logs are structured:
- `body` _(default)_: Stores logs in the request body
- `body_with_inline_attributes`: Logs include inline attributes | +| `EdotCloudForwarderS3LogsType` | The encoding format for logs in the S3 bucket. Supported options:
- `vpc_flow_log`: VPC Flow logs
- `elb_access_log`: Elastic Load Balancer (ELB) Access logs| | `SourceS3BucketARN` | Amazon Resource Name (ARN) of the S3 bucket where logs are stored. This bucket will trigger the `edot-cloud-forwarder` Lambda function automatically. | +% | `S3LogsJsonEncodingMode` | _(Required if `EdotCloudForwarderS3LogsType` is `json`)_
Defines how JSON logs are structured:
- `body` _(default)_: Stores logs in the request body
- `body_with_inline_attributes`: Logs include inline attributes | :::: - + ::::: ### Optional settings These are optional settings you can set in the CloudFormation template: -| Setting | Description | -| ------------------- | --- | -| `EdotCloudForwarderTimeout` | Maximum execution time for the Lambda function, measured in seconds. Default value is `300` seconds. Minimum value is `1` second. Maximum value is `900` seconds. | +| Setting | Description | +| ------------------- |--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `EdotCloudForwarderTimeout` | Maximum execution time for the Lambda function, measured in seconds. Default value is `300` seconds. Minimum value is `1` second. Maximum value is `900` seconds. | | `EdotCloudForwarderVersion` | Version of the EDOT Cloud Forwarder. Expected format is semantic versioning, for example `1.0.0`. Defaults to the latest available patch version. Don't change this value unless advised by Elastic Support. | -| `EdotCloudForwarderMemorySize` | Set the allocated memory for the Lambda function, measured in megabytes. Default value is `1024` MB. Minimum value is `128` MB. Maximum value is `10240` MB. | -| `EdotCloudForwarderConcurrentExecutions` | Set the maximum number of reserved concurrent executions for the Lambda function. Default value is `50`. Make sure this value doesn't exceed your AWS account's concurrency limit. | +| `EdotCloudForwarderMemorySize` | Set the allocated memory for the Lambda function, measured in megabytes. Default value is `512` MB. Minimum value is `128` MB. Maximum value is `10240` MB. | +| `EdotCloudForwarderConcurrentExecutions` | Set the maximum number of reserved concurrent executions for the Lambda function. Default value is `5`. Make sure this value doesn't exceed your AWS account's concurrency limit. | +| `EdotCloudForwarderExporterMaxQueueSize` | Set the internal OTLP exporter queue size. Default is `50` MB. You may incease this based on the data volume. | +Default values of `EdotCloudForwarderMemorySize` and `EdotCloudForwarderConcurrentExecutions` should be sufficient for most use cases. +However, depending on your data volumes (individual Signal size such as size of S3 object per VPC log entry), you may need to finetune them. +Key indications for the need of tuning these parameters are Lambda throttling and Lambda timeouts. +Along with these, you may also need to adjust `EdotCloudForwarderExporterMaxQueueSize` to export higher data volumes. ## Deployment examples @@ -210,6 +218,8 @@ aws cloudformation create-stack \ ``` :::: + + :::{note} The `--capabilities CAPABILITY_NAMED_IAM` flag is required because this CloudFormation template creates AWS Identity and Access Management (IAM) resources. More specifically, it creates a named IAM role (`LambdaExecutionRole`) for the Lambda function. To acknowledge that AWS CloudFormation might create or modify IAM resources with custom names, you must specify the `CAPABILITY_NAMED_IAM` capability. ::: ---> + :::: ::::: @@ -431,7 +443,10 @@ This is a list of resources created by the stack when processing S3 logs. The main Lambda function, `LambdaFunction`, is the core component for processing S3 logs. S3 event notifications are handled dynamically using `CustomNotificationUpdater` and `NotificationUpdaterLambda`. CloudWatch logs ensure detailed monitoring of Lambda executions. IAM roles and permissions control access between S3 and Lambda functions, while `S3FailureBucketARN` prevents data loss by capturing unprocessed logs. - ## Kibana integration setup @@ -475,6 +491,42 @@ To set up data visualization for your AWS logs: This allows you to immediately start analyzing your AWS infrastructure without building dashboards from scratch. +## Error handling and retrying + +{{edot-cf}} store Lambda invocation events related to retryable errors at the S3 bucket specified by `S3FailureBucketARN` parameter. +Retryable errors here include, + - Network errors when attempting to forward to OTLPEndpoint + - Invalid or expired ElasticApiKey + - Lambda triggered by events that mismatch EdotCloudForwarderS3LogsType selection + +These errors can be replayed manually to back-fill any gaps in your data. + +### Replay failed events + +To replay errors simply invoke the Lambda with manual trigger type `replayFailedEvents`, + +```sh +aws lambda invoke \ + --function-name \ + --payload '{ "replayFailedEvents": {"replayFailedEvents":{"dryrun":false,"removeOnSuccess":true}}}' \ + --cli-binary-format raw-in-base64-out /dev/null +``` +Replace `` with the name from your deployment. + +Table below explains supported configuration options, + +| Option | Description | Default | +|-----------------|-----------------------------------------------------------------------------------------------------------------------------------------------|---------| +| dryrun | Run the command without processing actual backup events. Useful to understand details about replaying error files from Lambda CloudWatch logs | false | +| removeOnSuccess | Configure whether to remove error event from S3 error destination, if processing is successful | true | + +When successful, you should see `"StatusCode": 200` as an output. You may check CloudWatch logs (resource `LambdaLogGroup`) for detailed logs. + +:::{note} +With AWS CLI, you can use `--timeout` option to increase currently configured Lambda timeout for custom invocations. +However, if a timeout occur, you will need to run the custom event multiple times to fully process all error events from the bucket. +::: + ## **Delete a CloudFormation stack** If you no longer need a deployed stack and want to clean up all associated resources, you can delete it using the following command: From dd8c217e66407518cb197bd17edcf0be18b1223a Mon Sep 17 00:00:00 2001 From: Michalis Katsoulis Date: Fri, 26 Sep 2025 10:56:21 +0300 Subject: [PATCH 4/9] Set version of ga to v1 --- docs/reference/edot-cloud-forwarder/aws.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/reference/edot-cloud-forwarder/aws.md b/docs/reference/edot-cloud-forwarder/aws.md index 1193b2b1..62180084 100644 --- a/docs/reference/edot-cloud-forwarder/aws.md +++ b/docs/reference/edot-cloud-forwarder/aws.md @@ -98,7 +98,7 @@ Download the CloudFormation template to deploy the appropriate stack based on yo | Log Source | CloudFormation template | | --- | ------------------------------------------------ | -| S3 logs | `https://edot-cloud-forwarder.s3.amazonaws.com/v0/latest/cloudformation/s3_logs-cloudformation.yaml` | +| S3 logs | `https://edot-cloud-forwarder.s3.amazonaws.com/v1/latest/cloudformation/s3_logs-cloudformation.yaml` | % | CloudWatch logs | `https://edot-cloud-forwarder.s3.amazonaws.com/v0/latest/cloudformation/cloudwatch_logs-cloudformation.yaml` | For specific versions, edit `latest` in the URL to the required version in the format `vX.Y.Z`. @@ -189,7 +189,7 @@ This example deploys a CloudFormation stack to collect VPC Flow logs stored in a ```sh aws cloudformation create-stack \ --stack-name edot-cloud-forwarder-vpc \ - --template-url https://edot-cloud-forwarder.s3.amazonaws.com/v0/latest/cloudformation/s3_logs-cloudformation.yaml \ + --template-url https://edot-cloud-forwarder.s3.amazonaws.com/v1/latest/cloudformation/s3_logs-cloudformation.yaml \ --capabilities CAPABILITY_NAMED_IAM \ --region eu-central-1 \ --parameters \ @@ -207,7 +207,7 @@ This example deploys a CloudFormation stack to collect ALB Access logs stored in ```sh aws cloudformation create-stack \ --stack-name edot-cloud-forwarder-alb \ - --template-url https://edot-cloud-forwarder.s3.amazonaws.com/v0/latest/cloudformation/s3_logs-cloudformation.yaml \ + --template-url https://edot-cloud-forwarder.s3.amazonaws.com/v1/latest/cloudformation/s3_logs-cloudformation.yaml \ --capabilities CAPABILITY_NAMED_IAM \ --region eu-central-1 \ --parameters \ @@ -348,7 +348,7 @@ Run the command with the following parameters: ```sh aws cloudformation update-stack \ - --template-url https://edot-cloud-forwarder.s3.amazonaws.com/v0/latest/cloudformation/.yaml \ + --template-url https://edot-cloud-forwarder.s3.amazonaws.com/v1/latest/cloudformation/.yaml \ --stack-name \ --capabilities CAPABILITY_NAMED_IAM \ --region eu-central-1 \ @@ -363,7 +363,7 @@ For example, to modify the S3 bucket ARN for the `edot-cloud-forwarder-vpc` stac ```sh aws cloudformation update-stack \ - --template-url https://edot-cloud-forwarder.s3.amazonaws.com/v0/latest/cloudformation/s3_logs-cloudformation.yaml \ + --template-url https://edot-cloud-forwarder.s3.amazonaws.com/v1/latest/cloudformation/s3_logs-cloudformation.yaml \ --stack-name edot-cloud-forwarder-vpc \ --capabilities CAPABILITY_NAMED_IAM \ --region eu-central-1 \ From b1735f4b787a46e09098c475ac9e72f67632ad86 Mon Sep 17 00:00:00 2001 From: Michalis Katsoulis Date: Mon, 29 Sep 2025 16:03:57 +0300 Subject: [PATCH 5/9] Update format names --- docs/reference/edot-cloud-forwarder/aws.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/reference/edot-cloud-forwarder/aws.md b/docs/reference/edot-cloud-forwarder/aws.md index 62180084..ec7e51e9 100644 --- a/docs/reference/edot-cloud-forwarder/aws.md +++ b/docs/reference/edot-cloud-forwarder/aws.md @@ -130,7 +130,7 @@ For S3 logs, use the following settings: | Setting | Description | | ------------------ | --- | -| `EdotCloudForwarderS3LogsType` | The encoding format for logs in the S3 bucket. Supported options:
- `vpc_flow_log`: VPC Flow logs
- `elb_access_log`: Elastic Load Balancer (ELB) Access logs| +| `EdotCloudForwarderS3LogsType` | The encoding format for logs in the S3 bucket. Supported options:
- `vpcflow`: VPC Flow logs
- `elbaccess`: Elastic Load Balancer (ELB) Access logs| | `SourceS3BucketARN` | Amazon Resource Name (ARN) of the S3 bucket where logs are stored. This bucket will trigger the `edot-cloud-forwarder` Lambda function automatically. | % | `S3LogsJsonEncodingMode` | _(Required if `EdotCloudForwarderS3LogsType` is `json`)_
Defines how JSON logs are structured:
- `body` _(default)_: Stores logs in the request body
- `body_with_inline_attributes`: Logs include inline attributes | @@ -196,7 +196,7 @@ aws cloudformation create-stack \ ParameterKey=SourceS3BucketARN,ParameterValue=your-s3-vpc-bucket-arn \ ParameterKey=OTLPEndpoint,ParameterValue="" \ ParameterKey=ElasticAPIKey,ParameterValue="" \ - ParameterKey=EdotCloudForwarderS3LogsType,ParameterValue="vpc_flow_log" + ParameterKey=EdotCloudForwarderS3LogsType,ParameterValue="vpcflow" ``` :::: @@ -214,7 +214,7 @@ aws cloudformation create-stack \ ParameterKey=SourceS3BucketARN,ParameterValue=your-s3-alb-bucket-arn \ ParameterKey=OTLPEndpoint,ParameterValue="" \ ParameterKey=ElasticAPIKey,ParameterValue="" \ - ParameterKey=EdotCloudForwarderS3LogsType,ParameterValue="elb_access_log" + ParameterKey=EdotCloudForwarderS3LogsType,ParameterValue="elbaccess" ``` :::: From beb9548bbec7047c457d65deeb509c9e9ea7078c Mon Sep 17 00:00:00 2001 From: Michalis Katsoulis Date: Thu, 23 Oct 2025 16:21:05 +0300 Subject: [PATCH 6/9] Update versions and comment out sar --- docs/reference/edot-cloud-forwarder/aws.md | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/docs/reference/edot-cloud-forwarder/aws.md b/docs/reference/edot-cloud-forwarder/aws.md index ec7e51e9..d47a127f 100644 --- a/docs/reference/edot-cloud-forwarder/aws.md +++ b/docs/reference/edot-cloud-forwarder/aws.md @@ -98,7 +98,7 @@ Download the CloudFormation template to deploy the appropriate stack based on yo | Log Source | CloudFormation template | | --- | ------------------------------------------------ | -| S3 logs | `https://edot-cloud-forwarder.s3.amazonaws.com/v1/latest/cloudformation/s3_logs-cloudformation.yaml` | +| S3 logs | `https://edot-cloud-forwarder.s3.amazonaws.com/v0/latest/cloudformation/s3_logs-cloudformation.yaml` | % | CloudWatch logs | `https://edot-cloud-forwarder.s3.amazonaws.com/v0/latest/cloudformation/cloudwatch_logs-cloudformation.yaml` | For specific versions, edit `latest` in the URL to the required version in the format `vX.Y.Z`. @@ -161,7 +161,7 @@ These are optional settings you can set in the CloudFormation template: | Setting | Description | | ------------------- |--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `EdotCloudForwarderTimeout` | Maximum execution time for the Lambda function, measured in seconds. Default value is `300` seconds. Minimum value is `1` second. Maximum value is `900` seconds. | -| `EdotCloudForwarderVersion` | Version of the EDOT Cloud Forwarder. Expected format is semantic versioning, for example `1.0.0`. Defaults to the latest available patch version. Don't change this value unless advised by Elastic Support. | +| `EdotCloudForwarderVersion` | Version of the EDOT Cloud Forwarder. Expected format is semantic versioning, for example `0.2.4`. Defaults to the latest available patch version. Don't change this value unless advised by Elastic Support. | | `EdotCloudForwarderMemorySize` | Set the allocated memory for the Lambda function, measured in megabytes. Default value is `512` MB. Minimum value is `128` MB. Maximum value is `10240` MB. | | `EdotCloudForwarderConcurrentExecutions` | Set the maximum number of reserved concurrent executions for the Lambda function. Default value is `5`. Make sure this value doesn't exceed your AWS account's concurrency limit. | | `EdotCloudForwarderExporterMaxQueueSize` | Set the internal OTLP exporter queue size. Default is `50` MB. You may incease this based on the data volume. | @@ -176,8 +176,8 @@ Along with these, you may also need to adjust `EdotCloudForwarderExporterMaxQueu The following examples use the CloudFormation template files hosted in the [public S3 bucket](#download-templates). - Use the `--template-url` flag to reference a template hosted on S3. -- To always use the most recent stable templates, use the `latest` path. For example, `v1/latest`. -- To pin a specific version, replace `latest` with the desired version tag. For example, `v1/v1.0.0`. +- To always use the most recent stable templates, use the `latest` path. For example, `v0/latest`. +- To pin a specific version, replace `latest` with the desired version tag. For example, `v0/v0.2.4`. Alternatively, if you have downloaded the template file, you can use the `--template-body file://` option with a local template file. @@ -189,7 +189,7 @@ This example deploys a CloudFormation stack to collect VPC Flow logs stored in a ```sh aws cloudformation create-stack \ --stack-name edot-cloud-forwarder-vpc \ - --template-url https://edot-cloud-forwarder.s3.amazonaws.com/v1/latest/cloudformation/s3_logs-cloudformation.yaml \ + --template-url https://edot-cloud-forwarder.s3.amazonaws.com/v0/latest/cloudformation/s3_logs-cloudformation.yaml \ --capabilities CAPABILITY_NAMED_IAM \ --region eu-central-1 \ --parameters \ @@ -207,7 +207,7 @@ This example deploys a CloudFormation stack to collect ALB Access logs stored in ```sh aws cloudformation create-stack \ --stack-name edot-cloud-forwarder-alb \ - --template-url https://edot-cloud-forwarder.s3.amazonaws.com/v1/latest/cloudformation/s3_logs-cloudformation.yaml \ + --template-url https://edot-cloud-forwarder.s3.amazonaws.com/v0/latest/cloudformation/s3_logs-cloudformation.yaml \ --capabilities CAPABILITY_NAMED_IAM \ --region eu-central-1 \ --parameters \ @@ -348,7 +348,7 @@ Run the command with the following parameters: ```sh aws cloudformation update-stack \ - --template-url https://edot-cloud-forwarder.s3.amazonaws.com/v1/latest/cloudformation/.yaml \ + --template-url https://edot-cloud-forwarder.s3.amazonaws.com/v0/latest/cloudformation/.yaml \ --stack-name \ --capabilities CAPABILITY_NAMED_IAM \ --region eu-central-1 \ @@ -363,7 +363,7 @@ For example, to modify the S3 bucket ARN for the `edot-cloud-forwarder-vpc` stac ```sh aws cloudformation update-stack \ - --template-url https://edot-cloud-forwarder.s3.amazonaws.com/v1/latest/cloudformation/s3_logs-cloudformation.yaml \ + --template-url https://edot-cloud-forwarder.s3.amazonaws.com/v0/latest/cloudformation/s3_logs-cloudformation.yaml \ --stack-name edot-cloud-forwarder-vpc \ --capabilities CAPABILITY_NAMED_IAM \ --region eu-central-1 \ @@ -400,6 +400,7 @@ You can deploy the stack manually using the AWS Management Console by following 6. Review your configuration and select **Submit** to deploy the stack. 7. Monitor the progress until the stack reaches the `CREATE_COMPLETE` state. + ## CloudFormation stack resources From 4148000d4f23160080f8234d13c6415eb72e7484 Mon Sep 17 00:00:00 2001 From: Fabrizio Ferri-Benedetti Date: Thu, 23 Oct 2025 15:47:54 +0200 Subject: [PATCH 7/9] Apply suggestions from code review Co-authored-by: Miguel Luna <39376769+mlunadia@users.noreply.github.com> --- docs/reference/edot-cloud-forwarder/aws.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/reference/edot-cloud-forwarder/aws.md b/docs/reference/edot-cloud-forwarder/aws.md index d47a127f..55f407f1 100644 --- a/docs/reference/edot-cloud-forwarder/aws.md +++ b/docs/reference/edot-cloud-forwarder/aws.md @@ -16,10 +16,10 @@ products: {{edot-cf}} for AWS provides the EDOT Collector as a Lambda function that collects and forwards logs to Elastic Observability on {{serverless-full}}. {{edot-cf}} for AWS supports the following log sources: -| Log source | Description | +| AWS Service | Telemetry Description | | --- | --- | -| VPC Flow | Logs generated by a Virtual Private Cloud (VPC) | -| ELB Access | Logs generated by an Elastic Load Balancer (ELB) | +| Virtual Private Cloud (VPC) | [VPC Flow Logs](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs.html) to capture information about IP traffic | +| Elastic Load Balancer (ELB) | [Access logs](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html) for your Application Load Balancer | % | CloudWatch {applies_to}`product: planned` | Logs generated by AWS CloudWatch | Read on to learn how to set up {{edot-cf}} for AWS. @@ -88,7 +88,7 @@ Trim the API key from `Authorization=ApiKey MYKEYVALUE...` to just `MYKEYVALUE.. Before deploying {{edot-cf}} for AWS, keep these points in mind: -- Deploy a separate CloudFormation stack for each log type, for example VPC Flow Logs or ELB Logs. Each CloudFormation stack can only process one log source and format at a time. +- Deploy a separate CloudFormation stack for each log type, for example VPC Flow Logs or ELB Access Logs. Each CloudFormation stack can only process one log type and format at a time. - Logs stored in S3 must be placed in separate buckets. Each log type should reside in its own dedicated bucket. - The CloudFormation stack deployment region must match the region of the S3 bucket. @@ -130,7 +130,7 @@ For S3 logs, use the following settings: | Setting | Description | | ------------------ | --- | -| `EdotCloudForwarderS3LogsType` | The encoding format for logs in the S3 bucket. Supported options:
- `vpcflow`: VPC Flow logs
- `elbaccess`: Elastic Load Balancer (ELB) Access logs| +| `EdotCloudForwarderS3LogsType` | The encoding format for logs in the S3 bucket. Supported options:
- `vpcflow`: VPC Flow Logs
- `elbaccess`: ELB Access logs| | `SourceS3BucketARN` | Amazon Resource Name (ARN) of the S3 bucket where logs are stored. This bucket will trigger the `edot-cloud-forwarder` Lambda function automatically. | % | `S3LogsJsonEncodingMode` | _(Required if `EdotCloudForwarderS3LogsType` is `json`)_
Defines how JSON logs are structured:
- `body` _(default)_: Stores logs in the request body
- `body_with_inline_attributes`: Logs include inline attributes | From 6d8df49b6fe2069aa5c9867bd03e4a73ce839e42 Mon Sep 17 00:00:00 2001 From: Fabrizio Ferri-Benedetti Date: Thu, 23 Oct 2025 18:27:25 +0200 Subject: [PATCH 8/9] Update aws.md --- docs/reference/edot-cloud-forwarder/aws.md | 30 ++++++++++------------ 1 file changed, 13 insertions(+), 17 deletions(-) diff --git a/docs/reference/edot-cloud-forwarder/aws.md b/docs/reference/edot-cloud-forwarder/aws.md index 55f407f1..8e8961e9 100644 --- a/docs/reference/edot-cloud-forwarder/aws.md +++ b/docs/reference/edot-cloud-forwarder/aws.md @@ -166,20 +166,17 @@ These are optional settings you can set in the CloudFormation template: | `EdotCloudForwarderConcurrentExecutions` | Set the maximum number of reserved concurrent executions for the Lambda function. Default value is `5`. Make sure this value doesn't exceed your AWS account's concurrency limit. | | `EdotCloudForwarderExporterMaxQueueSize` | Set the internal OTLP exporter queue size. Default is `50` MB. You may incease this based on the data volume. | -Default values of `EdotCloudForwarderMemorySize` and `EdotCloudForwarderConcurrentExecutions` should be sufficient for most use cases. -However, depending on your data volumes (individual Signal size such as size of S3 object per VPC log entry), you may need to finetune them. -Key indications for the need of tuning these parameters are Lambda throttling and Lambda timeouts. -Along with these, you may also need to adjust `EdotCloudForwarderExporterMaxQueueSize` to export higher data volumes. +Default values of `EdotCloudForwarderMemorySize` and `EdotCloudForwarderConcurrentExecutions` are sufficient for most use cases. Key indications for the need of tuning these parameters are Lambda throttling and Lambda timeouts. Along with these, you might also need to adjust `EdotCloudForwarderExporterMaxQueueSize` to export higher data volumes. ## Deployment examples -The following examples use the CloudFormation template files hosted in the [public S3 bucket](#download-templates). +The following examples show how to deploy the ECF Cloud Forwarder using AWS CloudFormation. Copy and paste these commands after replacing the placeholder values with your actual configuration. - Use the `--template-url` flag to reference a template hosted on S3. - To always use the most recent stable templates, use the `latest` path. For example, `v0/latest`. -- To pin a specific version, replace `latest` with the desired version tag. For example, `v0/v0.2.4`. +- To pin a specific version, replace `latest` with the desired version tag. For example, `v0/v{{version.edot-cf-aws}}`. -Alternatively, if you have downloaded the template file, you can use the `--template-body file://` option with a local template file. +Alternatively, if you have downloaded the template file, use the `--template-body file://` option with a local template file. :::::{tab-set} ::::{tab-item} VPC Flow logs @@ -470,7 +467,7 @@ CloudWatch Log Groups help monitor execution performance and debug issues. IAM p --> ## Kibana integration setup -After {{edot-cf}} for AWS is successfully running and forwarding logs to Elastic Observability, you can install pre-built integrations in Kibana to visualize your data with out-of-the-box dashboards and visualizations. +After {{edot-cf}} for AWS is successfully running and forwarding logs to Elastic Observability, install the {{kib}} integrations to visualize your data with out-of-the-box dashboards and visualizations. ### Install integrations @@ -505,7 +502,7 @@ These errors can be replayed manually to back-fill any gaps in your data. ### Replay failed events -To replay errors simply invoke the Lambda with manual trigger type `replayFailedEvents`, +To replay errors invoke the Lambda with manual trigger type `replayFailedEvents`. Replace `` with the name from your deployment. ```sh aws lambda invoke \ @@ -513,23 +510,22 @@ aws lambda invoke \ --payload '{ "replayFailedEvents": {"replayFailedEvents":{"dryrun":false,"removeOnSuccess":true}}}' \ --cli-binary-format raw-in-base64-out /dev/null ``` -Replace `` with the name from your deployment. -Table below explains supported configuration options, +The following settings are available: | Option | Description | Default | |-----------------|-----------------------------------------------------------------------------------------------------------------------------------------------|---------| -| dryrun | Run the command without processing actual backup events. Useful to understand details about replaying error files from Lambda CloudWatch logs | false | -| removeOnSuccess | Configure whether to remove error event from S3 error destination, if processing is successful | true | +| dryrun | Run the command without processing actual backup events. Useful to understand details about replaying error files from Lambda CloudWatch logs. | false | +| removeOnSuccess | Configure whether to remove error event from S3 error destination, if processing is successful. | true | -When successful, you should see `"StatusCode": 200` as an output. You may check CloudWatch logs (resource `LambdaLogGroup`) for detailed logs. +When successful, you should get `"StatusCode": 200` as the output. Check CloudWatch logs (resource `LambdaLogGroup`) for detailed logs. :::{note} -With AWS CLI, you can use `--timeout` option to increase currently configured Lambda timeout for custom invocations. -However, if a timeout occur, you will need to run the custom event multiple times to fully process all error events from the bucket. +With AWS CLI, you can use `--timeout` to increase currently configured Lambda timeout for custom invocations. +However, if a timeout occurs, you need to run the custom event multiple times to fully process all error events from the bucket. ::: -## **Delete a CloudFormation stack** +## Delete a CloudFormation stack If you no longer need a deployed stack and want to clean up all associated resources, you can delete it using the following command: From fdfecb1fc2ac70615970e798f55af7541b858098 Mon Sep 17 00:00:00 2001 From: Fabrizio Ferri-Benedetti Date: Thu, 23 Oct 2025 18:30:24 +0200 Subject: [PATCH 9/9] Update aws.md --- docs/reference/edot-cloud-forwarder/aws.md | 18 +++++------------- 1 file changed, 5 insertions(+), 13 deletions(-) diff --git a/docs/reference/edot-cloud-forwarder/aws.md b/docs/reference/edot-cloud-forwarder/aws.md index 87dcf93e..fd4f436c 100644 --- a/docs/reference/edot-cloud-forwarder/aws.md +++ b/docs/reference/edot-cloud-forwarder/aws.md @@ -475,26 +475,18 @@ CloudWatch Log Groups help monitor execution performance and debug issues. IAM p After {{edot-cf}} for AWS is successfully running and forwarding logs to Elastic Observability, install the {{kib}} integrations to visualize your data with out-of-the-box dashboards and visualizations. -### Install integrations +To set up data visualization in {{kib}}: -To set up data visualization for your AWS logs: - -1. **Navigate to Kibana**: Log into your Elastic Cloud deployment and open Kibana. - -2. **Access Integrations**: Go to **Management** → **Integrations** in the Kibana navigation menu. - -3. **Search and install**: Search for the appropriate integration based on your log type and install it: +1.Log into your Elastic Cloud deployment and open Kibana. +2. Go to **Management** → **Integrations** in the Kibana navigation menu. +3. Search for the appropriate integration based on your log type and install it: | **AWS Log Type** | **Integration Name** | **Description** | |------------------|---------------------|-----------------| | ELB Access Logs | **AWS ELB OpenTelemetry Assets** | Dashboards and visualizations for Elastic Load Balancer logs | | VPC Flow Logs | **AWS VPC Flow Logs OpenTelemetry Assets** | Dashboards and visualizations for VPC flow log data | -4. **Access dashboards**: Once installed, navigate to **Dashboard** to view the pre-built dashboards for your AWS log data. - -### Benefits - -This allows you to immediately start analyzing your AWS infrastructure without building dashboards from scratch. +4. Once installed, navigate to **Dashboard** to view the pre-built dashboards for your AWS log data. ## Error handling and retrying