"atlantis apply" also applies terragrunt cache directories

[ldormoy]

A picture is worth thousand words:

atlantis plan

atlantis apply

I suppose there is some internal setting to ignore /.terragrunt-cache/ pathes?

Running atlantis v0.4.13

[lkysow]

Are you using a custom workflow? Can you paste it here?

Atlantis uses this algorithm to find pending plans: https://github.com/runatlantis/atlantis/blob/master/server/events/pending_plan_finder.go#L38

It might be erroneously picking up .tfplan files from the terragrunt cache.

[ldormoy]

Yes, my workflow is:

workflows:
  terragrunt:
    apply:
      steps:
      - run: terragrunt apply -no-color $PLANFILE
    plan:
      steps:
      - run: terragrunt plan -no-color -out $PLANFILE

[sryabkov]

I am experiencing the same problem

[sryabkov]

So, I think I might be able to provide a little bit more info

In my case, the problem happens when you run atlantis plan (by commenting on the PR) more than once.

observations/evidence

Below are the tree structures from /atlantis-data/repos/redacted/infrastructure-tf/1/default/live/dev/atlantis-test on the atlantis server (produced with tree -a) produced at different times.

1. before the first atlantis plan we have

.
└── terraform.tfvars

0 directories, 1 file

2. after the first terraform apply, we get:

.
├── .terragrunt-cache
│   └── dG3hsPNMS2sD5wEkCih16cZY51k
│       └── 2dxXFM8zyFsxxe32_z8tSazNCFU
│           ├── .terragrunt-source-version
│           ├── README.md
│           ├── atlantis.yaml
│           ├── live
│           │   ├── dev
│           │   │   └── atlantis-test
│           │   │       └── terraform.tfvars
│           │   ├── org.tfvars
│           │   └── terraform-state.tfvars
│           └── modules
│               └── gcp
│                   └── project
│                       ├── .terraform
│                       │   ├── plugins
│                       │   │   └── linux_amd64
│                       │   │       ├── lock.json
│                       │   │       ├── terraform-provider-google-beta_v2.1.0_x4
│                       │   │       └── terraform-provider-google_v2.1.0_x4
│                       │   └── terraform.tfstate
│                       ├── main.tf
│                       ├── output.tf
│                       ├── terraform.tfvars
│                       └── variables.tf
├── default.tfplan
└── terraform.tfvars

12 directories, 16 files

Notice there are 16 files and there is only one default.tfplan. Everything looks normal.

If I add the atlantis apply comment at this point, atlantis executes the plan without a problem.

3. after the second terraform apply, we get:

.
├── .terragrunt-cache
│   └── dG3hsPNMS2sD5wEkCih16cZY51k
│       └── 2dxXFM8zyFsxxe32_z8tSazNCFU
│           ├── .terragrunt-source-version
│           ├── README.md
│           ├── atlantis.yaml
│           ├── live
│           │   ├── dev
│           │   │   └── atlantis-test
│           │   │       ├── default.tfplan
│           │   │       └── terraform.tfvars
│           │   ├── org.tfvars
│           │   └── terraform-state.tfvars
│           └── modules
│               └── gcp
│                   └── project
│                       ├── .terraform
│                       │   ├── plugins
│                       │   │   └── linux_amd64
│                       │   │       ├── lock.json
│                       │   │       ├── terraform-provider-google-beta_v2.1.0_x4
│                       │   │       └── terraform-provider-google_v2.1.0_x4
│                       │   └── terraform.tfstate
│                       ├── default.tfplan
│                       ├── main.tf
│                       ├── output.tf
│                       ├── terraform.tfvars
│                       └── variables.tf
├── default.tfplan
└── terraform.tfvars

12 directories, 18 files

Notice that there are 18 files now and three copies of default.tfplan

---

hypothesis

My hypothesis is that terragrunt copies all the files from the current working directory to multiple locations under .terragrunt-cache, and if atlantis had previously placed the generated plan file into this directory, it will be copied too. This causes atlantis to incorrectly think there are multiple projects to be applied.

solution proposal

Interestingly, I have **/.terragrunt-cache/* in .gitignore in the root of the repo, but it not respected by git ls-files . --others, which is what atlantis does to find pending plans, according to the code link @lkysow posted above. And atlantis doesn't seem to be doing a separate check for git ignored files either.

Can atlantis call git check-ignore on the candidate plans and reject them if they are supposed to be ignored?

git check-ignore doesn't return anything when called on ./default.tfplan, but returns path/to/file for the other two copies of the plan file (which are located under .terragrunt-cache)

E.g.

bash-4.4# find . -name default.tfplan
./default.tfplan
./.terragrunt-cache/dG3hsPNMS2sD5wEkCih16cZY51k/2dxXFM8zyFsxxe32_z8tSazNCFU/modules/gcp/project/default.tfplan
./.terragrunt-cache/dG3hsPNMS2sD5wEkCih16cZY51k/2dxXFM8zyFsxxe32_z8tSazNCFU/live/dev/atlantis-test/default.tfplan
bash-4.4# git check-ignore ./default.tfplan
bash-4.4# git check-ignore ./.terragrunt-cache/dG3hsPNMS2sD5wEkCih16cZY51k/2dxXFM8zyFsxxe32_z8tSazNCFU/modules/gcp/project/default.tfplan
./.terragrunt-cache/dG3hsPNMS2sD5wEkCih16cZY51k/2dxXFM8zyFsxxe32_z8tSazNCFU/modules/gcp/project/default.tfplan
bash-4.4# git check-ignore ./.terragrunt-cache/dG3hsPNMS2sD5wEkCih16cZY51k/2dxXFM8zyFsxxe32_z8tSazNCFU/live/dev/atlantis-test/default.tfplan
./.terragrunt-cache/dG3hsPNMS2sD5wEkCih16cZY51k/2dxXFM8zyFsxxe32_z8tSazNCFU/live/dev/atlantis-test/default.tfplan

@lkysow Would the above be an acceptable solution?

[sryabkov]

My timing seems to be off today as @lkysow just opened a PR (#517) to address this issue (but I think the solution I proposed is more generic).

[lkysow]

Thanks for the in-depth analysis Sergei! Yeah I agree that your solution of running check-ignore after filtering is a good one but it requires everyone add .terragrunt-cache to their .gitignore files.

I think let's just go with #517 for now and if there comes up more issues relating to this then we can add the check-ignore feature.

[sryabkov]

@lkysow Sounds good. By the way, another solution, even more generic, could be just deleting the previously created plan, if any exists, before executing atlantis plan.