Observe AWS Collection

This module assembles different methods of collecting data from AWS into Observe. It is intended as both a starting point and as a reference.

The module sets up the following forwarding methods:

• the Observe lambda
• an S3 bucket, subscribed to the aforementioned Lambda
• a Firehose stream

Given these egresses, we extract data from the following sources:

• Cloudwatch Metrics, via Firehose
• CloudTrail, via S3
• EventBridge, via Firehose
• AWS snapshot data, via Lambda

Usage

The following snippet installs the Observe AWS collection stack to a single region:

module "observe_collection" {
  source           = "github.com/observeinc/terraform-aws-collection"
  observe_customer = ""
  observe_token    = ""
}

Common Options

The snippet below installs the Observe AWS collection stack so that all supported CloudWatch Logs, CloudWatch metrics, CloudTrail records, and AWS resource updates are collected, except for some excluded items:

module "observe_collection" {
  source           = "github.com/observeinc/terraform-aws-collection"
  observe_customer = ""
  observe_token    = ""
  
  subscribed_log_group_matches = [".*"]
  subscribed_log_group_excludes = ["/aws/elasticbeanstalk/my-app.*"]
  snapshot_exclude = ["kms:Describe*"]
  cloudwatch_metrics_exclude_filters = ["AWS/KMS"]
}

Diagram

 ┌──────────────────┐                          ┌───────────────┐    ┌─────────────┐
 │cloudwatch metrics├──┐                       │   s3 bucket   │    │  cloudtrail │
 └──────────────────┘  │           ┌───────────►               ◄────┤             │
                       │           │           └────────┬──────┘    └─────────────┘
                       │           │                    │
                       │           │                    │
                       │     ┌─────┴──────┐             │
                       └─────►            │             │
                             │  Firehose  ├──────┐      │
         ┌───────────────────►            │      │      │
         │                   └───▲──┬─────┘      │      │
         │                       │  │            │      │
         │                       │  │        ┌───▼───┐  │
   ┌─────┴─────┐                 │  │        │       │  │
   │eventbridge│                 │  │        │observe│  │
   └─────┬─────┘                 │  │        │       │  │
         │            ┌──────────┴──▼─┐      └────▲──┘  │
         │            │cloudwatch logs│           │     │
         │            └──────────┬──┬─┘           │     │
         │                       │  │             │     │
         │                       │  │             │     │
         │                   ┌───┴──▼─────┐       │     │
         └───────────────────►            ├───────┘     │
                             │   Lambda   │             │
         ┌───────────────────►            ◄─────────────┘
         │                   └────────────┘
┌────────┴─────────┐
│ cloudwatch logs  │
└──────────────────┘

Requirements

Name	Version
terraform	>= 1.2
aws	>= 5.0
random	>= 3.0.0

Providers

Name	Version
aws	>= 5.0
random	>= 3.0.0

Modules

Name	Source	Version
lambda_log_subscription	observeinc/kinesis-firehose/aws//modules/cloudwatch_logs_subscription	2.3.0
observe_cloudwatch_logs_subscription	observeinc/cloudwatch-logs-subscription/aws	0.5.0
observe_cloudwatch_metrics	observeinc/kinesis-firehose/aws//modules/cloudwatch_metrics	2.3.0
observe_firehose_eventbridge	observeinc/kinesis-firehose/aws//modules/eventbridge	2.3.0
observe_kinesis_firehose	observeinc/kinesis-firehose/aws	2.3.0
observe_lambda	observeinc/lambda/aws	3.6.0
observe_lambda_s3_bucket_subscription	observeinc/lambda/aws//modules/s3_bucket_subscription	3.6.0
observe_lambda_snapshot	observeinc/lambda/aws//modules/snapshot	3.6.0
s3_bucket	terraform-aws-modules/s3-bucket/aws	~> 4.0

Resources

Name	Type
aws_cloudtrail.trail	resource
aws_cloudwatch_event_rule.rules	resource
aws_cloudwatch_log_group.group	resource
random_string.this	resource
aws_caller_identity.current	data source
aws_iam_policy_document.bucket	data source
aws_partition.current	data source
aws_region.current	data source

Inputs

Name	Description	Type	Default	Required
cloudtrail_enable	Whether to create a CloudTrail trail. Useful for avoiding the 'trails per region' quota of 5, such as when testing.	bool	true	no
cloudtrail_enable_log_file_validation	Whether log file integrity validation is enabled for CloudTrail. Defalults to false.	bool	false	no
cloudtrail_exclude_management_event_sources	A list of management event sources to exclude. See the following link for more info: https://docs.aws.amazon.com/awscloudtrail/latest/userguide/logging-management-events-with-cloudtrail.html	set(string)	[ "kms.amazonaws.com", "rdsdata.amazonaws.com" ]	no
cloudtrail_is_multi_region_trail	Whether to enable multi region trail export	bool	true	no
cloudwatch_metrics_exclude_filters	Namespaces to exclude. Mutually exclusive with cloudwatch_metrics_include_filters. To disable Cloudwatch Metrics Stream entirely, use ["*"].	set(string)	[]	no
cloudwatch_metrics_include_filters	Namespaces to include. Mutually exclusive with cloudwatch_metrics_exclude_filters.	set(string)	[]	no
dead_letter_queue_destination	Send failed events/function executions to a dead letter queue arn sns or sqs	string	null	no
enable_s3_bucket_eventbridge	Enable sending bucket notifications to EventBridge	bool	false	no
eventbridge_rules	Eventbridge events matching these rules will be forwarded to Observe. Map keys are only used to provide stable resource addresses. If null, a default set of rules will be used.	map(object({ description = string event_pattern = string }))	null	no
invoke_snapshot_on_start_enabled	Toggle invocation of snapshot from Cloudformation. This can be useful for debug purposes if the lambda fails to complete successfully.	bool	false	no
kms_key_id	KMS key ARN to use to encrypt the logs delivered by CloudTrail.	string	""	no
lambda_envvars	Environment variables	map(any)	{}	no
lambda_kms_key	KMS key to encrypt environment variables	object({ arn = string })	null	no
lambda_memory_size	The amount of memory that your function has access to. Increasing the function's memory also increases its CPU allocation. The default value is 256 MB. The value must be a multiple of 64 MB.	number	256	no
lambda_reserved_concurrent_executions	The number of simultaneous executions to reserve for the function.	number	100	no
lambda_s3_custom_rules	List of rules to evaluate how to upload a given S3 object to Observe.	list(object({ pattern = string headers = map(string) }))	[]	no
lambda_subscribe_logs	Whether to subscribe to the Lambda function's logs and deliver them from CloudWatch to Observe via Kinesis Firehose.	bool	true	no
lambda_timeout	The amount of time that Lambda allows a function to run before stopping it. The maximum allowed value is 900 seconds.	number	120	no
lambda_version	Lambda version	string	"arm64/latest"	no
log_subscription_name	Name for log subscription resources to be created	string	null	no
name	Name for resources to be created	string	"observe-collection"	no
observe_customer	Observe Customer ID	string	n/a	yes
observe_domain	Observe Domain	string	"observeinc.com"	no
observe_token	Observe Token	string	n/a	yes
retention_in_days	Retention in days of cloudwatch log group	number	365	no
s3_bucket	Override S3 bucket used to to stage data to be sent to Observe.	object({ id = string arn = string })	null	no
s3_exported_prefix	Key prefix which is subscribed to be sent to Observe Lambda	string	""	no
s3_lifecycle_rule	List of maps containing configuration of object lifecycle management.	any	[]	no
s3_logging	Enable S3 access log collection	bool	false	no
snapshot_action	List of actions triggered by snapshot. Set to null to inherit all actions supported by the lambda.	set(string)	[ "autoscaling:Describe*", "cloudformation:Describe*", "cloudformation:List*", "cloudfront:List*", "dynamodb:Describe*", "dynamodb:List*", "ec2:Describe*", "ecs:Describe*", "ecs:List*", "eks:Describe*", "eks:List*", "elasticache:Describe*", "elasticloadbalancing:Describe*", "events:List*", "firehose:Describe*", "firehose:List*", "iam:Get*", "iam:List*", "kinesis:Describe*", "kinesis:List*", "lambda:List*", "logs:Describe*", "rds:Describe*", "route53:List*", "s3:GetBucket*", "s3:List*", "sns:Get*", "sns:List*", "sqs:Get*", "sqs:List*" ]	no
snapshot_exclude	List of actions to exclude from being executed on snapshot request.	list(string)	[]	no
snapshot_include	List of actions to include in snapshot request.	list(string)	[]	no
snapshot_schedule_expression	Rate at which snapshot is triggered. Must be valid EventBridge expression	string	"rate(1 hour)"	no
subscribed_log_group_excludes	A list of regex patterns describing CloudWatch log groups to NOT subscribe to. See https://github.com/observeinc/terraform-aws-cloudwatch-logs-subscription#input_log_group_excludes for more info"	list(string)	[]	no
subscribed_log_group_filter_pattern	A filter pattern for a CloudWatch Logs subscription filter. See https://github.com/observeinc/terraform-aws-cloudwatch-logs-subscription#input_filter_pattern or https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/FilterAndPatternSyntax.html for more info"	string	""	no
subscribed_log_group_matches	A list of regex patterns describing CloudWatch log groups to subscribe to. See https://github.com/observeinc/terraform-aws-cloudwatch-logs-subscription#input_log_group_matches for more info"	list(string)	[]	no
subscribed_s3_bucket_arns	List of additional S3 bucket ARNs to subscribe lambda to.	list(string)	[]	no
tags	A map of tags to add to all resources	map(string)	{}	no

Outputs

Name	Description
bucket	S3 bucket subscribed to Observe Lambda
observe_kinesis_firehose	Observe Kinesis Firehose module
observe_lambda	Observe Lambda module

License

Apache 2 Licensed. See LICENSE for full details.