Cyclic dependency error when creating buckets with two way replication

[hashibot]

This issue was originally opened by @chetandeshmukh as hashicorp/terraform#14295. It was migrated here as part of the provider split. The original body of the issue is below.

---

provider "aws" {
    alias   = "east"
    region  = "${var.aws_region_east}"
    profile = "${var.profile}"
}

provider "aws" {
    alias   = "west"
    region  = "${var.aws_region_west}"
    profile = "${var.profile}"
}

resource "aws_s3_bucket" "bucket_west" {
    provider = "aws.west"
    bucket   = "${var.bucket_west}"
    region   = "${var.aws_region_west}"
    acl      = "private"

    versioning {
    enabled = true
    }
    replication_configuration {
        role = "${data.aws_iam_role.cove_iam_role.arn}"
        rules {
            id     = "bucket_west"
            prefix = ""
            status = "Enabled"

            destination {
            bucket        = "${aws_s3_bucket.bucket_east.arn}"
            storage_class = "STANDARD"
            }
        }
    }
}

resource "aws_s3_bucket" "bucket_east" {
    provider = "aws.east"
    bucket   = "${var.bucket_east}"
    region   = "${var.aws_region_east}"
    acl      = "private"

    versioning {
    enabled = true
    }

    replication_configuration {
    role = "${data.aws_iam_role.cove_iam_role.arn}"
        rules {
            id     = "bucket_east"
            prefix = ""
            status = "Enabled"

            destination {
                bucket        = "${aws_s3_bucket.bucket_west.arn}"
                storage_class = "STANDARD"
            }
        }
    }
}

[jmdots]

Any word on this? I too have a need for S3 bidirectional replication via Terraform.

[apparentlymart]

This sort of thing is historically rather tricky to do in Terraform, but we have some patterns.

Probably the most straightforward way would be for the replication configuration to become a separate resource that refers to both buckets, which I think is theoretically possible since replication configuration seems to be a separate endpoint in the S3 API:

resource "aws_s3_bucket" "bucket_west" {
    provider = "aws.west"
    bucket   = "${var.bucket_west}"
    region   = "${var.aws_region_west}"
    acl      = "private"

    versioning {
    enabled = true
    }
}

resource "aws_s3_bucket" "bucket_east" {
    provider = "aws.east"
    bucket   = "${var.bucket_east}"
    region   = "${var.aws_region_east}"
    acl      = "private"

    versioning {
    enabled = true
    }
}

resource "aws_bucket_replication" "eastward" {
  provider = "aws.west"
  source_bucket_name = "${aws_s3_bucket.bucket_west.name}"
  role = "${data.aws_iam_role.cove_iam_role.arn}"
  rules {
    id     = "bucket_west"
    prefix = ""
    status = "Enabled"

    destination {
      bucket        = "${aws_s3_bucket.bucket_east.arn}"
      storage_class = "STANDARD"
    }
  }
}

resource "aws_bucket_replication" "westward" {
  provider = "aws.east"
  source_bucket_name = "${aws_s3_bucket.bucket_east.name}"
  role = "${data.aws_iam_role.cove_iam_role.arn}"
  rules {
    id     = "bucket_west"
    prefix = ""
    status = "Enabled"

    destination {
      bucket        = "${aws_s3_bucket.bucket_west.arn}"
      storage_class = "STANDARD"
    }
  }
}

By splitting this out, the cycle is avoided and instead the replication resources just both depend on both of the buckets.

Unfortunately implementing this separate resource is not straightforward -- it'd require some care to avoid breaking compatibility with the old inline form -- so it's not something we'll be able to jump on immediately 😖 but if someone else has the time and motivation to work on it we could perhaps work with them to figure out the backward compatibility path and then review a PR.

[mkohn]

Not the best solution but one that works is to manually build the arn into the replication, this will break the cycle issue.

Since S3 is an old service, its only arn:aws:s3:::my_corporate_bucket.

Here is a snippet of my working code.

resource "aws_s3_bucket" "s3-primary" {
  provider = "aws.source_region"
  bucket   = "{var.source_bucket}"

  versioning {
    enabled = true
  }

  replication_configuration {
    role = "${aws_iam_role.replication_role_source_to_dr.arn}"

    rules {
      id     = "replicate_all"
      prefix = ""
      status = "Enabled"

      destination {
        bucket        = "arn:aws:s3:::{var.dest_bucket}"
        storage_class = "STANDARD"
      }
    }
  }
}

resource "aws_s3_bucket" "s3-dr" {
  provider = "aws.dr_region"
  bucket   = "{var.dest_bucket}"

  versioning {
    enabled = true
  }


  replication_configuration {
    role = "${aws_iam_role.replication_role_dr_to_source.arn}"

    rules {
      id     = "replicate_all"
      prefix = ""
      status = "Enabled"

      destination {
        bucket        = "arn:aws:s3:::{var.source_bucket}"
        storage_class = "STANDARD"
      }
    }
  }
}

[CyrilDevOps]

Any news on that ? I had to fake the ARN too, and I don't like that.

[CyrilDevOps]

Today I was creating two new buckets with bidirectionnal replication, but the the fake arn didn't work,
i hit some timing issue, sometime I get the error

• aws_s3_bucket.s3-dr: Error putting S3 replication configuration: InvalidRequest: Destination bucket must exist.
  or
• aws_s3_bucket.s3-dr: Error putting S3 replication configuration: InvalidRequest: Destination bucket must have versioning enabled.

I had to comment the replication block, create my two buckets, then uncomment the
replication bloick and re-plan and apply.

We need the replication by itself outside of the aws_s3_bucket so terraform can manage the dependency.

[ewbankkit]

Related:

• [Discuss/Feature] Additional S3 Bucket Resources #4418

[ionosphere80]

Still an issue in provider v3.2.0. The first application run usually taints the S3 bucket in one of the two regions. A second application run usually resolves the issue.

[williamcodes]

Any news on this front? I'm migrating from cloudformation to terraform and have bidirectional s3 bucket replication in production already, so this is a blocker for me.

[networkhorse]

Blocker for me also it seems. I have each region deployed as a module, but then I handle replication between those regions outside of my module. It seems that when you are doing bidirectional replication/circular dependencies, you hit a wall.

I like @apparentlymart's solution since it will completely alleviate the troubles of having to know the bucket ARN of other regions, potentially before they're even made.

In my case, I need it to work as simple as:

1. Per region, create S3 bucket
2. Globally, connect all those regions via replication rules

[anGie44]

Hi all, I believe this should be fixed with the addition of the aws_s3_bucket_replication_configuration resource added in v3.66.0 of the provider (PR: #20777) so I'm going to close this issue but if you believe it's been done by mistake, please reach out!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.