Amazon Managed Workflows for Apache Airflow(MWAA) Module

This terraform module can be used to deploy Amazon Managed Workflows for Apache Airflow(MWAA) environment.

✅ Deployment examples can be found under examples folder.

✅ Amazon MWAA documentation for more details about Amazon MWAA

✅ Amazon MWAA for Analytics Workshop

Amazon MWAA Architecture

Usage

The example below builds Amazon MWAA environment with existing VPC and Private Subnets. Amazon MWAA supporting resources S3 bucket, IAM role and Security groups created by this module by default. This module allows you to bring your own S3 bucket, IAM role and Security group.

module "mwaa" {
  source = "aws-ia/mwaa/aws"

  name                 = "basic-mwaa"
  airflow_version      = "2.2.2"
  environment_class    = "mw1.medium"

  vpc_id                = "<ENTER_VPC_ID>"
  private_subnet_ids    = ["<ENTER_SUBNET_ID1>","<ENTER_SUBNET_ID2>"]

  min_workers           = 1
  max_workers           = 25
  webserver_access_mode = "PUBLIC_ONLY" # Default PRIVATE_ONLY for production environments

  input_iam_role_additional_policies = {
    "additional-policy-1" = "<ENTER_POLICY_ARN1>"
    "additional-policy-2" = "<ENTER_POLICY_ARN2>"
  }

  logging_configuration = {
    dag_processing_logs = {
      enabled   = true
      log_level = "INFO"
    }

    scheduler_logs = {
      enabled   = true
      log_level = "INFO"
    }

    task_logs = {
      enabled   = true
      log_level = "INFO"
    }

    webserver_logs = {
      enabled   = true
      log_level = "INFO"
    }

    worker_logs = {
      enabled   = true
      log_level = "INFO"
    }
  }

  airflow_configuration_options = {
    "core.load_default_connections" = "false"
    "core.load_examples"            = "false"
    "webserver.dag_default_view"    = "tree"
    "webserver.dag_orientation"     = "TB"
    "logging.logging_level"         = "INFO"
  }
}

Security

See CONTRIBUTING for more information.

License

Apache-2.0 Licensed. See LICENSE.

Requirements

Name	Version
terraform	>= 1.0.0
aws	>= 4.63.0

Providers

Name	Version
aws	>= 4.63.0

Modules

No modules.

Resources

Name	Type
aws_iam_role.mwaa	resource
aws_iam_role_policy.mwaa	resource
aws_iam_role_policy_attachment.mwaa	resource
aws_mwaa_environment.mwaa	resource
aws_s3_bucket.mwaa	resource
aws_s3_bucket_acl.mwaa	resource
aws_s3_bucket_public_access_block.mwaa	resource
aws_s3_bucket_server_side_encryption_configuration.mwaa	resource
aws_s3_bucket_versioning.mwaa	resource
aws_security_group.mwaa	resource
aws_security_group_rule.mwaa_sg_inbound	resource
aws_security_group_rule.mwaa_sg_inbound_vpn	resource
aws_security_group_rule.mwaa_sg_outbound	resource
aws_caller_identity.current	data source
aws_iam_policy_document.mwaa	data source
aws_iam_policy_document.mwaa_assume	data source
aws_partition.current	data source
aws_region.current	data source

Inputs

Name	Description	Type	Default	Required
airflow_configuration_options	(Optional) The airflow_configuration_options parameter specifies airflow override options.	any	null	no
airflow_version	(Optional) Airflow version of your environment, will be set by default to the latest version that MWAA supports.	string	null	no
create_iam_role	Create IAM role for MWAA	bool	true	no
create_s3_bucket	Create new S3 bucket for MWAA.	string	true	no
create_security_group	Create security group for MWAA	bool	true	no
dag_s3_path	(Required) The relative path to the DAG folder on your Amazon S3 storage bucket. For example, dags.	string	"dags"	no
environment_class	(Optional) Environment class for the cluster. Possible options are mw1.small, mw1.medium, mw1.large. Will be set by default to mw1.small. Please check the AWS Pricing for more information about the environment classes.	string	"mw1.small"	no
execution_role_arn	(Required) The Amazon Resource Name (ARN) of the task execution role that the Amazon MWAA and its environment can assume Mandatory if create_iam_role=false	string	null	no
force_detach_policies	IAM role Force detach policies	bool	false	no
iam_role_additional_policies	A map of additional policy ARNs to be added to the IAM role, with an arbitary key name	map(string)	{}	no
iam_role_name	IAM Role Name to be created if execution_role_arn is null	string	null	no
iam_role_path	IAM role path	string	"/"	no
iam_role_permissions_boundary	IAM role Permission boundary	string	null	no
kms_key	(Optional) The Amazon Resource Name (ARN) of your KMS key that you want to use for encryption. Will be set to the ARN of the managed KMS key aws/airflow by default.	string	null	no
logging_configuration	(Optional) The Apache Airflow logs which will be send to Amazon CloudWatch Logs.	any	null	no
max_workers	(Optional) The maximum number of workers that can be automatically scaled up. Value need to be between 1 and 25. Will be 10 by default	number	10	no
min_workers	(Optional) The minimum number of workers that you want to run in your environment. Will be 1 by default.	number	1	no
name	(Required) The name of the Apache Airflow MWAA Environment	string	n/a	yes
plugins_s3_object_version	(Optional) The plugins.zip file version you want to use.	string	null	no
plugins_s3_path	(Optional) The relative path to the plugins.zip file on your Amazon S3 storage bucket. For example, plugins.zip. If a relative path is provided in the request, then plugins_s3_object_version is required.	string	null	no
private_subnet_ids	(Required) The private subnet IDs in which the environment should be created. MWAA requires two subnets.	list(string)	n/a	yes
requirements_s3_object_version	(Optional) The requirements.txt file version you want to use.	string	null	no
requirements_s3_path	(Optional) The relative path to the requirements.txt file on your Amazon S3 storage bucket. For example, requirements.txt. If a relative path is provided in the request, then requirements_s3_object_version is required.	string	null	no
schedulers	(Optional) The number of schedulers that you want to run in your environment.	string	null	no
security_group_ids	Security group IDs for MWAA	list(string)	[]	no
source_bucket_arn	(Required) The Amazon Resource Name (ARN) of your Amazon S3 storage bucket. For example, arn:aws:s3:::airflow-mybucketname	string	null	no
source_bucket_name	New bucket will be created with the given name for MWAA when create_s3_bucket=true	string	null	no
source_cidr	(Required) Source CIDR block which will be allowed on MWAA SG to access Airflow UI Used only if create_security_group=true	list(string)	[]	no
startup_script_s3_object_version	(Optional) The version of the startup shell script you want to use. You must specify the version ID that Amazon S3 assigns to the file every time you update the script.	string	null	no
startup_script_s3_path	(Optional) The relative path to the script hosted in your bucket. The script runs as your environment starts before starting the Apache Airflow process. Use this script to install dependencies, modify configuration options, and set environment variables.	string	null	no
tags	(Optional) A map of resource tags to associate with the resource	map(string)	{}	no
vpc_id	(Required) VPC ID to deploy the MWAA Environment. Mandatory if create_security_group=true	string	""	no
webserver_access_mode	(Optional) Specifies whether the webserver should be accessible over the internet or via your specified VPC. Possible options: PRIVATE_ONLY (default) and PUBLIC_ONLY	string	"PRIVATE_ONLY"	no
weekly_maintenance_window_start	(Optional) Specifies the start date for the weekly maintenance window	string	null	no

Outputs

Name	Description
aws_s3_bucket_name	S3 bucket Name of the MWAA Environment
mwaa_arn	The ARN of the MWAA Environment
mwaa_role_arn	IAM Role ARN of the MWAA Environment
mwaa_role_name	IAM role name of the MWAA Environment
mwaa_security_group_id	Security group id of the MWAA Environment
mwaa_service_role_arn	The Service Role ARN of the Amazon MWAA Environment
mwaa_status	The status of the Amazon MWAA Environment
mwaa_webserver_url	The webserver URL of the MWAA Environment