diff --git a/docs/reference/edot-cloud-forwarder/aws.md b/docs/reference/edot-cloud-forwarder/aws.md index 5549a958..fd4f436c 100644 --- a/docs/reference/edot-cloud-forwarder/aws.md +++ b/docs/reference/edot-cloud-forwarder/aws.md @@ -20,10 +20,10 @@ products: {{edot-cf}} for AWS supports the following log sources: -| Log source | Description | +| AWS Service | Telemetry Description | | --- | --- | -| VPC Flow | Logs generated by a Virtual Private Cloud (VPC) | -| ELB Access | Logs generated by an Elastic Load Balancer (ELB) | +| Virtual Private Cloud (VPC) | [VPC Flow Logs](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs.html) to capture information about IP traffic | +| Elastic Load Balancer (ELB) | [Access logs](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html) for your Application Load Balancer | % | CloudWatch {applies_to}`product: planned` | Logs generated by AWS CloudWatch | Read on to learn how to set up {{edot-cf}} for AWS. @@ -64,6 +64,7 @@ To collect Elastic Load Balancer (ELB) Access logs, you need: - Access logging enabled, with the bucket as the destination ::: + + ::::: ### Optional settings These are optional settings you can set in the CloudFormation template: -| Setting | Description | -| ------------------- | --- | -| `EdotCloudForwarderTimeout` | Maximum execution time for the Lambda function, measured in seconds. Default value is `300` seconds. Minimum value is `1` second. Maximum value is `900` seconds. | -| `EdotCloudForwarderVersion` | Version of the EDOT Cloud Forwarder. Expected format is semantic versioning, for example `0.1.5`. Defaults to the latest available patch version. Don't change this value unless advised by Elastic Support. | -| `EdotCloudForwarderMemorySize` | Set the allocated memory for the Lambda function, measured in megabytes. Default value is `1024` MB. Minimum value is `128` MB. Maximum value is `10240` MB. | -| `EdotCloudForwarderConcurrentExecutions` | Set the maximum number of reserved concurrent executions for the Lambda function. Default value is `5`. Make sure this value doesn't exceed your AWS account's concurrency limit. | +| Setting | Description | +| ------------------- |--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `EdotCloudForwarderTimeout` | Maximum execution time for the Lambda function, measured in seconds. Default value is `300` seconds. Minimum value is `1` second. Maximum value is `900` seconds. | +| `EdotCloudForwarderVersion` | Version of the EDOT Cloud Forwarder. Expected format is semantic versioning, for example `0.2.4`. Defaults to the latest available patch version. Don't change this value unless advised by Elastic Support. | +| `EdotCloudForwarderMemorySize` | Set the allocated memory for the Lambda function, measured in megabytes. Default value is `512` MB. Minimum value is `128` MB. Maximum value is `10240` MB. | +| `EdotCloudForwarderConcurrentExecutions` | Set the maximum number of reserved concurrent executions for the Lambda function. Default value is `5`. Make sure this value doesn't exceed your AWS account's concurrency limit. | +| `EdotCloudForwarderExporterMaxQueueSize` | Set the internal OTLP exporter queue size. Default is `50` MB. You may incease this based on the data volume. | +Default values of `EdotCloudForwarderMemorySize` and `EdotCloudForwarderConcurrentExecutions` are sufficient for most use cases. Key indications for the need of tuning these parameters are Lambda throttling and Lambda timeouts. Along with these, you might also need to adjust `EdotCloudForwarderExporterMaxQueueSize` to export higher data volumes. ## Deployment examples -The following examples use the CloudFormation template files hosted in the [public S3 bucket](#download-templates). +The following examples show how to deploy the ECF Cloud Forwarder using AWS CloudFormation. Copy and paste these commands after replacing the placeholder values with your actual configuration. - Use the `--template-url` flag to reference a template hosted on S3. - To always use the most recent stable templates, use the `latest` path. For example, `v0/latest`. -- To pin a specific version, replace `latest` with the desired version tag. For example, `v0/v0.1.5`. +- To pin a specific version, replace `latest` with the desired version tag. For example, `v0/v{{version.edot-cf-aws}}`. -Alternatively, if you have downloaded the template file, you can use the `--template-body file://` option with a local template file. +Alternatively, if you have downloaded the template file, use the `--template-body file://` option with a local template file. :::::{tab-set} ::::{tab-item} VPC Flow logs @@ -192,7 +199,7 @@ aws cloudformation create-stack \ ParameterKey=SourceS3BucketARN,ParameterValue=your-s3-vpc-bucket-arn \ ParameterKey=OTLPEndpoint,ParameterValue="" \ ParameterKey=ElasticAPIKey,ParameterValue="" \ - ParameterKey=EdotCloudForwarderS3LogsType,ParameterValue="vpc_flow_log" + ParameterKey=EdotCloudForwarderS3LogsType,ParameterValue="vpcflow" ``` :::: @@ -210,10 +217,89 @@ aws cloudformation create-stack \ ParameterKey=SourceS3BucketARN,ParameterValue=your-s3-alb-bucket-arn \ ParameterKey=OTLPEndpoint,ParameterValue="" \ ParameterKey=ElasticAPIKey,ParameterValue="" \ - ParameterKey=EdotCloudForwarderS3LogsType,ParameterValue="elb_access_log" + ParameterKey=EdotCloudForwarderS3LogsType,ParameterValue="elbaccess" ``` :::: - + :::{note} The `--capabilities CAPABILITY_NAMED_IAM` flag is required because this CloudFormation template creates AWS Identity and Access Management (IAM) resources. More specifically, it creates a named IAM role (`LambdaExecutionRole`) for the Lambda function. To acknowledge that AWS CloudFormation might create or modify IAM resources with custom names, you must specify the `CAPABILITY_NAMED_IAM` capability. ::: ---> + :::: ::::: @@ -315,6 +403,27 @@ You can deploy the stack manually using the AWS Management Console by following 6. Review your configuration and select **Submit** to deploy the stack. 7. Monitor the progress until the stack reaches the `CREATE_COMPLETE` state. + + ## CloudFormation stack resources The CloudFormation templates create a number of resources to process logs from a specific log source. @@ -339,7 +448,10 @@ This is a list of resources created by the stack when processing S3 logs. The main Lambda function, `LambdaFunction`, is the core component for processing S3 logs. S3 event notifications are handled dynamically using `CustomNotificationUpdater` and `NotificationUpdaterLambda`. CloudWatch logs ensure detailed monitoring of Lambda executions. IAM roles and permissions control access between S3 and Lambda functions, while `S3FailureBucketARN` prevents data loss by capturing unprocessed logs. - -## **Delete a CloudFormation stack** +## Kibana integration setup + +After {{edot-cf}} for AWS is successfully running and forwarding logs to Elastic Observability, install the {{kib}} integrations to visualize your data with out-of-the-box dashboards and visualizations. + +To set up data visualization in {{kib}}: + +1.Log into your Elastic Cloud deployment and open Kibana. +2. Go to **Management** → **Integrations** in the Kibana navigation menu. +3. Search for the appropriate integration based on your log type and install it: + +| **AWS Log Type** | **Integration Name** | **Description** | +|------------------|---------------------|-----------------| +| ELB Access Logs | **AWS ELB OpenTelemetry Assets** | Dashboards and visualizations for Elastic Load Balancer logs | +| VPC Flow Logs | **AWS VPC Flow Logs OpenTelemetry Assets** | Dashboards and visualizations for VPC flow log data | + +4. Once installed, navigate to **Dashboard** to view the pre-built dashboards for your AWS log data. + +## Error handling and retrying + +{{edot-cf}} store Lambda invocation events related to retryable errors at the S3 bucket specified by `S3FailureBucketARN` parameter. +Retryable errors here include, + - Network errors when attempting to forward to OTLPEndpoint + - Invalid or expired ElasticApiKey + - Lambda triggered by events that mismatch EdotCloudForwarderS3LogsType selection + +These errors can be replayed manually to back-fill any gaps in your data. + +### Replay failed events + +To replay errors invoke the Lambda with manual trigger type `replayFailedEvents`. Replace `` with the name from your deployment. + +```sh +aws lambda invoke \ + --function-name \ + --payload '{ "replayFailedEvents": {"replayFailedEvents":{"dryrun":false,"removeOnSuccess":true}}}' \ + --cli-binary-format raw-in-base64-out /dev/null +``` + +The following settings are available: + +| Option | Description | Default | +|-----------------|-----------------------------------------------------------------------------------------------------------------------------------------------|---------| +| dryrun | Run the command without processing actual backup events. Useful to understand details about replaying error files from Lambda CloudWatch logs. | false | +| removeOnSuccess | Configure whether to remove error event from S3 error destination, if processing is successful. | true | + +When successful, you should get `"StatusCode": 200` as the output. Check CloudWatch logs (resource `LambdaLogGroup`) for detailed logs. + +:::{note} +With AWS CLI, you can use `--timeout` to increase currently configured Lambda timeout for custom invocations. +However, if a timeout occurs, you need to run the custom event multiple times to fully process all error events from the bucket. +::: + +## Delete a CloudFormation stack If you no longer need a deployed stack and want to clean up all associated resources, you can delete it using the following command: