Add support for S3 existing object replication field

[borrell]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

S3 object replication can be enabled to replicate all objects created or modified (matching a given filter) after the replication rule is created. A support request to AWS can optionally enable S3 object replication for existing objects in an S3 bucket (that can retroactively replicate objects). In the console, this appears as a checkbox in the Object Replication Configuration screen.

Currently TF does not support this field in the S3 Replication configuration.

New or Affected Resource(s)

• aws_s3_bucket

Potential Terraform Configuration

I propose a new block in the replication_configuration block, to match the struct that is passed to this object in the SDK:

resource "aws_s3_bucket" "bucket" {
  provider = "aws.central"
  bucket   = "tf-test-bucket-12345"
  acl      = "private"
  region   = "eu-central-1"

  versioning {
    enabled = true
  }

  replication_configuration {
    role = "${aws_iam_role.replication.arn}"

    rules {
      id     = "foobar"
      prefix = "foo"
      status = "Enabled"

      destination {
        bucket        = "${aws_s3_bucket.destination.arn}"
        storage_class = "STANDARD"
      }

      existing_object_replication {
        status = "Enabled"
      }
    }
  }
}

References

https://docs.aws.amazon.com/AmazonS3/latest/API/API_ReplicationRule.html#AmazonS3-Type-ReplicationRule-ExistingObjectReplication

AWS Go SDK supports this (ExistingObjectReplication)
https://docs.aws.amazon.com/sdk-for-go/api/service/s3/#ReplicationRule

[ewbankkit]

@borrell Thanks for opening the issue. As you mention, this functionality is not yet publicly available and I'm not sure how public the fact that you can get this enabled per-account via AWS support is. This will probably be hard to write an acceptance test for right now.

[borrell]

Yes I was thinking about this before I raised the issue. Ultimately I think it'd be worthwhile to include:

1. The feature is publicly available, just not enabled by default. AWS don't hide the fact that you can enable this with a support request - they mention it throughout their replication documentation https://docs.aws.amazon.com/AmazonS3/latest/dev/replication-what-is-isnot-replicated.html#existing-object-replication.

2. Once enabled on an account, it's available through various other interfaces, including the Go SDK and boto3. Including it in Terraform will help ensure feature parity, leading to my next point...

3. Without this in the TF configuration, it's a step that needs to be done separately (either through the SDK or the console) after a deployment or update. My concern with this is how this will interact with TF. Without this configuration being supported natively, we can't guarantee that this configuration item won't be 'reset' whenever the replication configuration is updated in TF (or that a diff will permanently be detected), leading to possible inconsistent states and requirements to build out-of-band configuration checking.

[ewbankkit]

@borrell Thanks for the additional info. I'll investigate what happens with an account where this feature isn't enabled.

[ewbankkit]

When I try and set ExistingObjectReplication.Status = Enabled I get an error MalformedXML: The XML you provided was not well-formed or did not validate against our published schema.

[borrell]

Yes I was just trying in boto3 on an account without the option enabled, and have the following message

botocore.exceptions.ParamValidationError: Parameter validation failed:
Unknown parameter in ReplicationConfiguration.Rules[0]: "ExistingObjectReplication", must be one of: ID, Priority, Prefix, Filter, Status, SourceSelectionCriteria, Destination, DeleteMarkerReplication

I'm going to do some more testing to see what the behavior is when omitting the option on an account with Existing Object Replication enabled.

[tdmalone]

Possibly relevant, this feature now has a blog post: https://aws.amazon.com/blogs/storage/replicating-existing-objects-between-s3-buckets

[rdshadab]

@ewbankkit Facing with same issue.
There is no way to enable the option like existing_object_replication from the terraform script in Resource: aws_s3_bucket.
Any update on whether this option would be enabled in near future.
This would really help lot of users who are trying to create the replication rule with huge data already in the source bucket.

[daroga0002]

have the same issue/requirement

[drmaciej]

I'm seeing the same when using s3api put-bucket-replication with the AWS CLI; all other fields seem to be processed properly.
A similar problem is described here: https://stackoverflow.com/questions/67004590/xml-provided-is-not-well-formed-for-boto-script.

[daroga0002]

I dont know does replicating existing objects is available in AWS as I dont see this on current S3 console implementation, additionally docs are stating clearly:

Replicating existing objects
To enable existing object replication for your account, you must contact AWS Support. To prevent your request from being delayed, title your AWS Support case "Replication for Existing Objects" and be sure to include the following information:

Source bucket

Destination buckets

Estimated storage volume to replicate (in terabytes)

Estimated storage object count to replicate

Ref:
https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication-what-is-isnot-replicated.html

[drmaciej]

@daroga0002

You're right, that is exactly the case. Support need to allow-list the bucket for existing account replication; until that happens requests will fail with MalformedXML.

[theophilechevalier]

FYI, after opening a support case to request activation of existing object replication feature, it takes a few.. weeks (!) before it is enabled (official answer from AWS Support).

[lnader]

👍

[tamsky]

Also mentioned here:

• https://aws.amazon.com/blogs/storage/replicating-existing-objects-between-s3-buckets/

Can also confirm it takes a few... weeks to enable.

[cbijeta]

Amazon S3 just launched Batch Replication: https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-batch-replication-batch.html You will not need to open a support ticket anymore to replicate existing objects.

[cbijeta]

Yes I was thinking about this before I raised the issue. Ultimately I think it'd be worthwhile to include:

1. The feature is publicly available, just not enabled by default. AWS don't hide the fact that you can enable this with a support request - they mention it throughout their replication documentation https://docs.aws.amazon.com/AmazonS3/latest/dev/replication-what-is-isnot-replicated.html#existing-object-replication.
2. Once enabled on an account, it's available through various other interfaces, including the Go SDK and boto3. Including it in Terraform will help ensure feature parity, leading to my next point...
3. Without this in the TF configuration, it's a step that needs to be done separately (either through the SDK or the console) after a deployment or update. My concern with this is how this will interact with TF. Without this configuration being supported natively, we can't guarantee that this configuration item won't be 'reset' whenever the replication configuration is updated in TF (or that a diff will permanently be detected), leading to possible inconsistent states and requirements to build out-of-band configuration checking.

We just launched S3 Batch Replication that lets you replicate existing objects. https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-batch-replication-batch.html

[daroga0002]

thx, so I think this issue can be closed

[anGie44]

Closing this as AWS also confirmed setting existing object replication is now only available via Batch Jobs as commented here: #23803 (comment)

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.