Atlantis apply all after a failed apply; outputs Ran Apply for 0 projects #773

mlehner616 · 2019-09-11T01:41:29Z

I have a repo that uses the default workspace but there are a number of different project folders.

Atlantis version: 0.8.3
Terraform version: v0.12.8

version: 3
projects:
  - name: qa
    dir: qa_acct/qa_env
    terraform_version: v0.12.8
    autoplan:
      when_modified: ["../../projects/*", "*.tf*", "../../modules/*"]
      enabled: false
  - name: staging
    dir: prod_acct/staging_env
    terraform_version: v0.12.8
    autoplan:
      when_modified: ["../../projects/*", "*.tf*", "../../modules/*"]
      enabled: false
  - name: prod
    dir: prod_acct/prod_env
    terraform_version: v0.12.8
    autoplan:
      when_modified: ["../../projects/*", "*.tf*", "../../modules/*"]
      enabled: false

Plans are generated for all three projects as normal after commenting exactly atlantis plan.
Immediately afterword, commenting atlantis apply attempts to apply all three environments as expected. In this case, there was an apply error due to an AWS IAM policy being misconfigured and the plans were not successfully applied. A commit was pushed to fix this issue and another atlantis apply was submitted. Note, there was not another atlantis plan after the fix commit was pushed. Atlantis behaved as if it had forgotten about the failed plans and assumed they had been applied successfully when, in fact, they had not been. I believe the expected behavior should be to reject the apply since new commits were made and force another plan be run, correct?

The result was the following:

Ran Apply for 0 projects:

Automatically merging because all plans have been successfully applied.

Locks and plans deleted for the projects and workspaces modified in this pull request:

* dir: `prod_acct/prod_env` workspace: `default`
* dir: `prod_acct/staging_env` workspace: `default`
* dir: `qa_acct/qa_env` workspace: `default`

The text was updated successfully, but these errors were encountered:

lkysow · 2019-09-14T00:07:25Z

Yeah it's a bug. If autoplan had been enabled then there would have been new plans generated and the apply wouldn't have worked.

mlehner616 · 2019-09-16T21:34:50Z

@lkysow Thanks for the confirmation. This bug is killing us right now. We want people to be able see non-locking plans being run (in our normal CI pipleine) before Approvals are submitted so they can actually validate their code before blocking other development. If we wanted to dig into solving this, where would be a good place to start looking? I took a really quick glance through the repo and nothing jumped out at me.

Thank you for building this tool by the way, I really appreciate the work that went into this.

lkysow · 2019-09-19T19:40:17Z

After re-reading the ticket, this isn't technically a bug (although for your use-case it may as well be). Atlantis is just doing what you told it, it's up to you to run atlantis plan if you've pushed a new commit and don't have autoplan running. Coupled with automerge is the real issue here. If you didn't have automerge you'd quickly realize that you didn't re-run plan and there wouldn't be an issue.

Also if you were running with the -d or -p flags you'd get an error that "the plan doesn't exist for that project, please run plan". When we added the apply-all command (i.e. atlantis apply) we didn't replicate the behaviour. I'm not sure if it ever makes sense to not give an error in this case but I'd at least like to add a flag that lets you keep the old behaviour in case people were relying on it.

If we were to add some functionality to detect this case, it would be here: https://github.com/runatlantis/atlantis/blob/master/server/events/project_command_builder.go#L204 after Atlantis has found no pending plans. It could then exit with an error in this case.

I think a path forward may be:

new flag --allow-no-plan-apply which defaults to false now (breaking change)
thread that flag through and then check it at the line above

mlehner616 · 2019-09-19T20:29:00Z

Well, I actually think your original interpretation made sense to me. To clarify, we would never want atlantis to apply without having the most up to date plan saved and locked.

What we’re doing instead is just running validate, fmt, lint andterraform plan --lock=false in vanilla gitlab CI. Devs open an MR and need to fix any issues there, and get all approvals first, before the atlantis plan. The problem we were solving by doing it this way was autoplan opening locks too early in the process and thus blocking other MRs that were ready to be applied.

I still think this is a bug. Yes I wanted autoplan disabled but that just means I want the developers to run it if and only if all the pre-apply requirements are met. I would expect the apply step to run the same validation that the plan is locked and up to date and apply based on that. Turning off autoplan shouldn’t affect those checks. What seems to be happening with autoplan disabled is the apply is ignoring the plans and ultimately just applies nothing.

I can confirm there are plans and locks are created when they are supposed to be. It appears that the atlantis apply step is just ignoring those if a second apply is run after this first one fails. Expected behavior would be for the apply step to either force a replan if the MR was updated, or attempt to re-apply the original plan. It’s doing neither of these right now.

mlehner616 · 2019-09-19T20:35:02Z

One thing I did notice was that if the apply does fail, the saved plans are deleted but the locks are left open (this may be the actual bug here). If we removed those after a failed apply, that would basically force the plan step. I don’t know if that’s the best solution but I think it would work.

kipkoan · 2019-09-20T17:12:13Z

it's up to you to run atlantis plan if you've pushed a new commit and don't have autoplan running.

Our team has autoplan on, but pushing a new commit doesn't cause Atlantis to redo the plan (because Bitbucket).

One thing I did notice was that if the apply does fail, the saved plans are deleted but the locks are left open (this may be the actual bug here). If we removed those after a failed apply, that would basically force the plan step. I don’t know if that’s the best solution but I think it would work.

I agree we should either have Atlantis not delete the plans, or error if an apply is attempted without any plans.

@lkysow - what's the reason for Atlantis to delete the plans after a failed apply? It could have failed because a transient provider issue, and re-running apply on the same plan would later succeed.

ishallbethat · 2020-12-31T09:26:01Z

HI everyone I met this issue too.
Any work in progress to fix this bug ?
I removed the locks as mentioned above and redo "atlantis plan". It still shows "Ran Plan for 0 projects:"

jamengual · 2020-12-31T20:13:15Z

Running plan on the same PR after a failed to apply should not be any different than if atlantis does not delete the plan, it is just an extra step.

But if someone else in another PR modify the environment you are running plan against you will have a problem no matter what but by re-running a plan you could actually find the drift.

I do not think this is a bug, it is a bit annoying to run plan again but since terraform is idempotent it should only apply the difference.

evanstachowiak · 2022-04-06T09:11:05Z

I can run atlantis plan again and I am still getting the output "Ran Plan for 0 projects:"

If I run with atlantis plan -p *-production it will apply.

jamengual · 2022-04-06T18:16:40Z

with autoplan, you need to define every directory you want autoplan on/off in your atlantis.yaml otherwise it does not work, is what you guys are doing?

if this was a bug, no one will be using atlantis so I want to make sure if this is specific to multi-dir structure etc. For that, we need to see the altlantis.yaml files and dir structure so we can have a better idea.

This could be as simple as better documentation of autoplan with some examples.

evanstachowiak · 2022-04-14T08:06:49Z

@jamengual I am using an atlantis.yaml that was previously working. I think around v0.19.* this started breaking. It is about 50 projects, each with its own project name so that the -p wildcard flag can be used. The pattern for the naming is ${service_name}-${environment}.

I discovered that if i run atlantis apply -p *-environment, then the command will run, but it will run for ALL projects, regardless of what files have changed.

I have autoplan on, but if I run atlantis plan manually, it doesn't seem to make a difference.

Also of note, I am using custom workflows, not sure if that makes a difference.

jamengual · 2022-04-14T15:27:05Z

@evanstachowiak Please test with the pre-release image, we did some bug fixes there and I wonder if that could be the issue:

docker pull ghcr.io/runatlantis/atlantis:v0.19.3-pre.20220408

jamengual · 2022-08-26T02:41:09Z

is this still an issue with v0.19.8?

GusAntoniassi · 2022-09-22T17:03:16Z

Hello @jamengual I was able to reproduce this issue on v0.19.8, using the testdrive repository.

It only happened when using pre workflow hooks, such as the following:

---
repos:
  - id: /.*/
    pre_workflow_hooks:
    - run: echo "hello world"

The server logs for the execution:

{"level":"info","ts":"2022-09-22T13:58:58.502-0300","caller":"server/server.go:869","msg":"Atlantis started - listening on port 4141","json":{}}
{"level":"info","ts":"2022-09-22T13:58:58.502-0300","caller":"scheduled/executor_service.go:46","msg":"Scheduled Executor Service started","json":{}}
{"level":"info","ts":"2022-09-22T13:59:09.305-0300","caller":"events/events_controller.go:533","msg":"parsed comment as command=\"apply\" verbose=false dir=\"\" workspace=\"\" project=\"\" flags=\"\"","json":{"gh-request-id":"X-Github-Delivery=dfb30ec0-3a97-11ed-9f80-6ecf217e25c6"}}
{"level":"info","ts":"2022-09-22T13:59:14.712-0300","caller":"events/working_dir.go:225","msg":"creating dir \"/home/gus/workspace/opensource/apply-for-0-projects-test/atlantis_linux_amd64/data/repos/GusAntoniassi/atlantis-example/1/default\"","json":{"repo":"GusAntoniassi/atlantis-example","pull":"1"}}
{"level":"info","ts":"2022-09-22T13:59:15.360-0300","caller":"runtime/pre_workflow_hook_runner.go:50","msg":"successfully ran \"echo \\\"hello world\\\"\" in \"/home/gus/workspace/opensource/apply-for-0-projects-test/atlantis_linux_amd64/data/repos/GusAntoniassi/atlantis-example/1/default\"","json":{"repo":"GusAntoniassi/atlantis-example","pull":"1"}}

evanstachowiak · 2022-09-23T21:36:37Z

yes it's still an issue @jamengual

jamengual · 2022-09-23T21:38:46Z

I wonder if this is related to this : #1633

jamengual · 2022-09-26T19:25:43Z

Hello @jamengual I was able to reproduce this issue on v0.19.8, using the testdrive repository.

It only happened when using pre workflow hooks, such as the following:

---
repos:
  - id: /.*/
    pre_workflow_hooks:
    - run: echo "hello world"

The server logs for the execution:

{"level":"info","ts":"2022-09-22T13:58:58.502-0300","caller":"server/server.go:869","msg":"Atlantis started - listening on port 4141","json":{}}
{"level":"info","ts":"2022-09-22T13:58:58.502-0300","caller":"scheduled/executor_service.go:46","msg":"Scheduled Executor Service started","json":{}}
{"level":"info","ts":"2022-09-22T13:59:09.305-0300","caller":"events/events_controller.go:533","msg":"parsed comment as command=\"apply\" verbose=false dir=\"\" workspace=\"\" project=\"\" flags=\"\"","json":{"gh-request-id":"X-Github-Delivery=dfb30ec0-3a97-11ed-9f80-6ecf217e25c6"}}
{"level":"info","ts":"2022-09-22T13:59:14.712-0300","caller":"events/working_dir.go:225","msg":"creating dir \"/home/gus/workspace/opensource/apply-for-0-projects-test/atlantis_linux_amd64/data/repos/GusAntoniassi/atlantis-example/1/default\"","json":{"repo":"GusAntoniassi/atlantis-example","pull":"1"}}
{"level":"info","ts":"2022-09-22T13:59:15.360-0300","caller":"runtime/pre_workflow_hook_runner.go:50","msg":"successfully ran \"echo \\\"hello world\\\"\" in \"/home/gus/workspace/opensource/apply-for-0-projects-test/atlantis_linux_amd64/data/repos/GusAntoniassi/atlantis-example/1/default\"","json":{"repo":"GusAntoniassi/atlantis-example","pull":"1"}}

pre_workflow_hooks run before any atlantis.yaml file is parsed.

after that if no atlantis.yaml is defined it it will do nothing.

github-actions · 2023-03-18T01:58:15Z

This issue is stale because it has been open for 1 month with no activity. Remove stale label or comment or this will be closed in 1 month.'

scytem · 2023-09-06T13:51:25Z

@jamengual Hello! Recently I reproduced that problem on v0.25.0
Also, I'm using pre-workflow hooks as described above. Is it possible to reopen this issue to fix this bug?

jamengual · 2023-09-06T14:31:32Z

can you describe the steps you took to reproduce it?

scytem · 2023-09-06T14:59:56Z

Sure!
atlantis-0:/$ atlantis version atlantis v0.25.0 (commit: a12823e) (build date: 2023-08-11T20:51:19.440Z)

Repos config:

repos:
  - id: "/.*/"
    branch: "/.*/"
    workflow: check
    allow_custom_workflows: true
    allowed_overrides: [workflow, delete_source_branch_on_merge]
    apply_requirements: [approved]
    pre_workflow_hooks:
      - run: python3 code/atlantis_config_merge.py # script for generating atlantis.yaml
  workflows:
    check:
      plan:
        steps:
        - run: echo "check passed"
    terragrunt-tst:
      plan:
        steps:
        - env:
          ...
        - run: |
            if [ ! -d "/tmp/$BASE_REPO_OWNER-$BASE_REPO_NAME-$PULL_NUM" ]; then
              mkdir -p /tmp/$BASE_REPO_OWNER-$BASE_REPO_NAME-$PULL_NUM
            fi
        - run: terragrunt run-all plan -out ./plan.tfplan --terragrunt-non-interactive &> /tmp/$BASE_REPO_OWNER-$BASE_REPO_NAME-$PULL_NUM/output.txt || cat /tmp/$BASE_REPO_OWNER-$BASE_REPO_NAME-$PULL_NUM/output.txt
        - run: terragrunt run-all show -json ./plan.tfplan --terragrunt-non-interactive 2> /tmp/$BASE_REPO_OWNER-$BASE_REPO_NAME-$PULL_NUM/show_stderr.txt 1> ./plan.json || cat /tmp/$BASE_REPO_OWNER-$BASE_REPO_NAME-$PULL_NUM/show_stderr.txt
        - run: /tmp/infracost breakdown --path=. --format=json --log-level=info --out-file=./infracost.json --project-name=$REPO_REL_DIR 2>> /tmp/$BASE_REPO_OWNER-$BASE_REPO_NAME-$PULL_NUM/output.txt 1>> /tmp/$BASE_REPO_OWNER-$BASE_REPO_NAME-$PULL_NUM/output.txt || cat /tmp/$BASE_REPO_OWNER-$BASE_REPO_NAME-$PULL_NUM/output.txt
        - run: /tmp/infracost output --path=./infracost.json --format=json --out-file=./infracost-report.json 2>> /tmp/$BASE_REPO_OWNER-$BASE_REPO_NAME-$PULL_NUM/output.txt 1>> /tmp/$BASE_REPO_OWNER-$BASE_REPO_NAME-$PULL_NUM/output.txt || cat /tmp/$BASE_REPO_OWNER-$BASE_REPO_NAME-$PULL_NUM/output.txt
        - run: |
            /tmp/infracost comment gitlab --repo $BASE_REPO_OWNER/$BASE_REPO_NAME \
              --merge-request $PULL_NUM \
              --path ./infracost-report.json \
              --gitlab-token $ATLANTIS_GITLAB_TOKEN \
              --behavior new \
              --show-all-projects
        # script for output formatting. Not sure if it's relevant for this issue. Just to share
        - run: python3 /opt/terragrunt_output_formatter.py --file /tmp/$BASE_REPO_OWNER-$BASE_REPO_NAME-$PULL_NUM/output.txt --output-file /tmp/$BASE_REPO_OWNER-$BASE_REPO_NAME-$PULL_NUM/fmt_output.txt && cat /tmp/$BASE_REPO_OWNER-$BASE_REPO_NAME-$PULL_NUM/fmt_output.txt
        - run: rm -rf /tmp/$BASE_REPO_OWNER-$BASE_REPO_NAME-$PULL_NUM
      apply:
        steps:
        - env:
          ...
        - run: terragrunt run-all apply ./plan.tfplan --terragrunt-non-interactive

atlantis.yaml example:

projects:
- autoplan:
    when_modified:
    - '**/*.hcl'
    - '*.hcl'
  dir: accounts/...
  name: ...
  workflow: terragrunt-tst

As a result, I have an MR message:

Ran Apply for 0 projects:

atlantis apply -p ... solves the problem, but it's not comfortable to use it every time

github-actions · 2023-10-08T01:48:09Z

This issue is stale because it has been open for 1 month with no activity. Remove stale label or comment or this will be closed in 1 month.'

valentindeaconu · 2024-08-30T09:02:01Z

I also face this issue on v0.28.1. I have a multi-dir structure (mono-repository) with a lot of projects (> 300). I recently introduced a CI stage that has to run before Atlantis. Because there's no other way of pipelining this (running the CI stage before Atlantis), I disabled the autoplan feature from atlantis.yaml everywhere and added a step in our pipeline that comments atlantis plan if everything is fine.

The issue occurred after Atlantis successfully generated the plans, then the PR was rebased and until Atlantis finished planning, another user commented atlantis apply. With auto planning enabled, Atlantis would've commented back something like another command is already running for this PR and error out the atlantis apply command, but now, it just commented Ran Apply for 0 projects and then merged and deleted the source branch, because that's how we have it configured, so in our case, manually commenting again atlantis plan and atlantis apply doesn't work.

Is there any chance this issue will be reopened?

LE: I found a workaround for now, at least for our use-case: I added a pre-workflow-hook with commands: plan that checks if the CI pipeline is finished. If the pre-workflow-hook fails, no comment will be posted. When the CI finishes, it comments atlantis plan and the normal flow begins. With this setup, I was able to re-enable the auto-plan feature and avoid this bug.

github-actions · 2024-10-19T02:00:43Z

This issue is stale because it has been open for 1 month with no activity. Remove stale label or comment or this will be closed in 1 month.'

raghib-goto · 2024-11-10T20:37:53Z

is there any permanent solution for this?

github-actions · 2024-12-12T02:07:49Z

This issue is stale because it has been open for 1 month with no activity. Remove stale label or comment or this will be closed in 1 month.'

NP-compete · 2024-12-26T19:39:28Z

Did anyone figure out a workaround for this issue?

github-actions · 2025-01-28T01:59:01Z

This issue is stale because it has been open for 1 month with no activity. Remove stale label or comment or this will be closed in 1 month.'

lkysow added the bug Something isn't working label Sep 14, 2019

evanstachowiak mentioned this issue Apr 14, 2022

Locked Workspace ignores previously generated plan file #2193

Closed

jamengual added the waiting-on-response Waiting for a response from the user label Aug 26, 2022

github-actions bot added the Stale label Mar 18, 2023

github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Apr 19, 2023

jamengual reopened this Sep 6, 2023

github-actions bot removed the Stale label Sep 7, 2023

github-actions bot added the Stale label Oct 8, 2023

github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Nov 9, 2023

carlosaml mentioned this issue Aug 2, 2024

apply command on top of failed plans gets reported as successful when --silence-no-projects is enabled #4807

Open

jamengual reopened this Aug 30, 2024

github-actions bot added the Stale label Oct 19, 2024

github-actions bot removed the Stale label Nov 11, 2024

github-actions bot added the Stale label Dec 12, 2024

github-actions bot removed the Stale label Dec 27, 2024

github-actions bot added the Stale label Jan 28, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Atlantis apply all after a failed apply; outputs Ran Apply for 0 projects #773

Atlantis apply all after a failed apply; outputs Ran Apply for 0 projects #773

mlehner616 commented Sep 11, 2019 •

edited

Loading

lkysow commented Sep 14, 2019

mlehner616 commented Sep 16, 2019 •

edited

Loading

lkysow commented Sep 19, 2019

mlehner616 commented Sep 19, 2019 •

edited

Loading

mlehner616 commented Sep 19, 2019

kipkoan commented Sep 20, 2019

ishallbethat commented Dec 31, 2020 •

edited

Loading

jamengual commented Dec 31, 2020

evanstachowiak commented Apr 6, 2022 •

edited

Loading

jamengual commented Apr 6, 2022

evanstachowiak commented Apr 14, 2022

jamengual commented Apr 14, 2022

jamengual commented Aug 26, 2022

GusAntoniassi commented Sep 22, 2022

evanstachowiak commented Sep 23, 2022

jamengual commented Sep 23, 2022

jamengual commented Sep 26, 2022

github-actions bot commented Mar 18, 2023

scytem commented Sep 6, 2023

jamengual commented Sep 6, 2023

scytem commented Sep 6, 2023 •

edited

Loading

github-actions bot commented Oct 8, 2023

valentindeaconu commented Aug 30, 2024 •

edited

Loading

github-actions bot commented Oct 19, 2024

raghib-goto commented Nov 10, 2024

github-actions bot commented Dec 12, 2024

NP-compete commented Dec 26, 2024

github-actions bot commented Jan 28, 2025

Atlantis apply all after a failed apply; outputs Ran Apply for 0 projects #773

Atlantis apply all after a failed apply; outputs Ran Apply for 0 projects #773

Comments

mlehner616 commented Sep 11, 2019 • edited Loading

lkysow commented Sep 14, 2019

mlehner616 commented Sep 16, 2019 • edited Loading

lkysow commented Sep 19, 2019

mlehner616 commented Sep 19, 2019 • edited Loading

mlehner616 commented Sep 19, 2019

kipkoan commented Sep 20, 2019

ishallbethat commented Dec 31, 2020 • edited Loading

jamengual commented Dec 31, 2020

evanstachowiak commented Apr 6, 2022 • edited Loading

jamengual commented Apr 6, 2022

evanstachowiak commented Apr 14, 2022

jamengual commented Apr 14, 2022

jamengual commented Aug 26, 2022

GusAntoniassi commented Sep 22, 2022

evanstachowiak commented Sep 23, 2022

jamengual commented Sep 23, 2022

jamengual commented Sep 26, 2022

github-actions bot commented Mar 18, 2023

scytem commented Sep 6, 2023

jamengual commented Sep 6, 2023

scytem commented Sep 6, 2023 • edited Loading

github-actions bot commented Oct 8, 2023

valentindeaconu commented Aug 30, 2024 • edited Loading

github-actions bot commented Oct 19, 2024

raghib-goto commented Nov 10, 2024

github-actions bot commented Dec 12, 2024

NP-compete commented Dec 26, 2024

github-actions bot commented Jan 28, 2025

mlehner616 commented Sep 11, 2019 •

edited

Loading

mlehner616 commented Sep 16, 2019 •

edited

Loading

mlehner616 commented Sep 19, 2019 •

edited

Loading

ishallbethat commented Dec 31, 2020 •

edited

Loading

evanstachowiak commented Apr 6, 2022 •

edited

Loading

scytem commented Sep 6, 2023 •

edited

Loading

valentindeaconu commented Aug 30, 2024 •

edited

Loading