ECS resources don't handle OOB deploys well

[wr0ngway]

I use aws_ecs_cluster/aws_ecs_service/aws_ecs_task_definition to define a ECS cluster with a service and task definition. This all works great. If I taint the task definition, then apply, it causes the service to redeploy, and since I'm using a latest tag for my image, it picks up the new image during that redeploy.

However, we have a CI (travis) process in place for deploying. This process uses a helper ecs-deploy script to trigger the creating of a new task definition revision and update the service. This also works great.

The problem I'm having is that after a CI deploy, the tfstate is understandably out of sync with the state of the running system, and subsequent plans/applys want to revert the service's revision to the one in tfstate. If I taint the task def, and re-apply, things redeploy and everything is ok. However, its an extra deploy, and when I forget to do it, there is the danger of terraform doing a redeploy which reverts my running system to an old revision.

Am I missing something? Is there a better way to do this? I'd like to avoid having my terraform keys (full admin!) exposed to the CI environment, so don't want to do the deploy through terraform. It'd be nice if there was a way for the task def to always just refresh to the latest revision (maybe a flag on aws_ecs_task_definition?)

[apparentlymart]

You could try applying the ignore_changes lifecycle flag to the fields you're intentionally changing. This should cause Terraform to not include the affected fields in diffs.

Another (admittedly more fiddly) option would be to make a second set of credentials for Terraform when run as part of the CI system, whose IAM policy only allows a very targeted set of permissions needed for this task. Since Terraform likes to refresh everything before it plans, it would need access to read all of the resources that exist in the Terraform state, but it should only need access to write the relevant ECS APIs you'd use when running ecs-deploy.

In your case, where deployment seems specifically limited to ECS, the former is probably the easiest option. I use the latter option in some cases although I must admit I've never actually put into practice my idea of giving Terraform broad read access and only very focused write access, since in our case we're using Terraform to manage quite a broad set of resources when applications are deployed.

[radeksimko]

I will close this issue as I don't think there's much we can do on the Terraform's side regarding this (except what is being discussed in other issues already).

The mechanism for interacting with other tools (ignore_changes) is supported and if that doesn't work as expected, that should be submitted as a separate bug/feature request. There are some known issues already.

Besides what @apparentlymart mentioned already, some improvements could be made by introducing new option to the lifecycle block so you can actually let Terraform behave in "create and forget" way which is what you may be looking for. Keep watching #2159 and/or #3874

Feel free to reopen or submit new issue if you have more specific suggestions on how the current behaviour should be changed (if it's not covered already in the linked issues).

[wr0ngway]

Nice, ignore_changes did the trick, thanks!

[eric-johnson]

With the release of 0.7 and the state import feature, would it be possible to accomplish this without the ignore_changes lifecycle option and instead import the new task version into Terraform after deploying?

aws ecs ...  # Create task-dev, update service
terraform import aws_ecs_task_definition.app <new_task_arn>
terraform import aws_ecs_service.app <new_service_arn>

I think it would work if your revision was a fixed tag like latest, but I can't think of a way to get this to work if you instead tag your images by version number or commit SHA. If the tag changes with each task, then Terraform would somehow need to know what the desired tag is, and perhaps it could pull it from its own remote_state.

[gservat]

Don't think aws_ecs_service supports import :(

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.