Add support for parallel plans #926

Fauzyy · 2020-02-07T22:37:45Z

Issue: #260

We've been running a simple repo config implementation of parallel plans for a couple weeks in our fork and it's working well

Caveat: because Atlantis has internal locks on the workspace name, this only works if all projects in the repo use Terraform Workspaces.

This PR also adds parallel apply as well.

Example config:

version: 3
automerge: true
parallel_plan: true (default false)
parallel_apply: true (default false)
...

lkysow

Hey sorry it's taken so long to get to this. I've left a couple of comments, but I know it's been a while so no worries if you can't get back to me.

I was thinking about having this be on automatically if the plans are in separate workspaces and skip the config option.

cmd/server.go

server/events/command_runner.go

server/events/yaml/raw/repo_cfg.go

Fauzyy · 2020-04-06T17:26:52Z

@lkysow thanks! I'll address the comments this week

lkysow · 2020-04-06T19:21:39Z

@Fauzyy FYI I was looking at enabling this for all projects, not just workspaces. If we add a queuing system to the working dir locker then we could lock it while it's doing the clone but otherwise we'd be fine to run plan in separate directories. What do you think?

You wouldn't have to do it in this PR, I could pick it up once you've got the per-workspace code ready.

Fauzyy · 2020-04-06T22:22:05Z

@lkysow that sounds great 👍. The only case I'd be concerned about is the one where multiple atlantis projects point to the same directory (through some kind of .tfvars setup with custom workflows)

adnankobir · 2020-04-08T21:47:51Z

@Fauzyy @lkysow we would love to have the ability to run apply's in parallel as well. Given that there's a risk of unintended side effects, could we perhaps have it default to disable and expose a flag to enable?

…l-plans-upstream

Fauzyy · 2020-05-06T00:36:15Z

@adnankobir that sounds reasonable yeah, I can make plans default parallel but add a flag for applies. Let me know what you think, @lkysow

+ Default parallel plans breaks in a lot of autoplan scenarios i.e. with shared workspaces + Added a set of e2e tests to check parallel plans and applies + Also added an option to the Terraform client to turn off the plugin cache. This was breaking the e2e test suite (never seen this error in practice)

Fauzyy · 2020-05-08T05:48:23Z

For some reason the e2e test is running when it shouldn't be:

- run: if [ -z "${CIRCLE_PR_REPONAME}" ]; then ./scripts/e2e.sh; fi

I also added back the flag to disable parallel plans (off by default), for now. It's breaking a lot of e2e tests when you're not using workspaces. Feel free to change this after!

lkysow · 2020-05-08T16:30:41Z

It looks like your segmentio circleci instance is running on this PR and somehow reporting checks back.

Fauzyy · 2020-05-08T17:12:32Z

Oops, should be fixed now 👍

codecov · 2020-05-08T17:18:33Z

Codecov Report

Merging #926 into master will decrease coverage by 0.45%.
The diff coverage is 52.74%.

@@            Coverage Diff             @@
##           master     #926      +/-   ##
==========================================
- Coverage   71.96%   71.50%   -0.46%     
==========================================
  Files          65       65              
  Lines        5429     5489      +60     
==========================================
+ Hits         3907     3925      +18     
- Misses       1215     1254      +39     
- Partials      307      310       +3

Impacted Files	Coverage Δ
server/events/models/models.go	`75.00% <ø> (ø)`
server/events/yaml/valid/repo_cfg.go	`19.04% <ø> (ø)`
server/events/command_runner.go	`42.95% <8.88%> (-6.42%)`	⬇️
server/events/yaml/raw/repo_cfg.go	`91.66% <83.33%> (-4.77%)`	⬇️
server/events/project_command_builder.go	`82.57% <100.00%> (+0.44%)`	⬆️
server/events/terraform/terraform_client.go	`78.36% <100.00%> (+0.21%)`	⬆️
server/server.go	`63.88% <100.00%> (+0.10%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 541d25d...67ef5fc. Read the comment docs.

rverma-jm · 2020-05-13T09:31:41Z

How do we enable parallel plan using repos.yaml

lkysow · 2020-05-25T19:15:16Z

server/events/terraform/terraform_client.go

@@ -65,6 +65,9 @@ type DefaultClient struct {

 	// versionsLock is used to ensure versions isn't being concurrently written to.
 	versionsLock *sync.Mutex
+
+	// usePluginCache determines whether or not to set the TF_PLUGIN_CACHE_DIR env var
+	usePluginCache bool


Are you going to expose this via a config file flag? Curious as to your use case for why you want to be able to turn this off.

I could definitely expose this as a config file flag, this was just to fix the e2e test suite for parallel plans. They would transiently fail with the plugin cache enabled, we never saw this error in production however. The error mentions that the null_resource provider is missing when a plan is run, I wonder if the nature of these tests exacerbates the problem

lkysow · 2020-05-25T19:25:39Z

Amazing work, sorry this took so long. I'm planning to take steps to avoid this long a delay in the future. We will need to add docs for this.

ghostsquad · 2020-10-12T22:42:49Z

Has the documentation not been updated for this feature? I don't see it here: https://www.runatlantis.io/docs/repo-level-atlantis-yaml.html#example-using-all-keys

lkysow · 2020-10-23T04:18:38Z

Has the documentation not been updated for this feature? I don't see it here: https://www.runatlantis.io/docs/repo-level-atlantis-yaml.html#example-using-all-keys

No it has not. PRs welcome!

nwsparks · 2021-05-04T19:14:13Z

Is this only valid if using terraform workspaces?

mwarkentin · 2021-05-04T19:40:09Z

@nwsparks yes this is terraform workspaces only I believe.

dmattia · 2021-05-04T23:02:28Z

We do not use terraform workspaces, but use parallelization. There is a difference between Atlantis workspaces and terraform workspaces

Fauzyy added 3 commits February 7, 2020 14:21

Add parallel plans via repo config

f2609fd

Add wait group for parallel plans

f0c483d

Use default not repoCfg

643fbd4

lkysow reviewed Mar 30, 2020

View reviewed changes

cmd/server.go Outdated Show resolved Hide resolved

server/events/command_runner.go Show resolved Hide resolved

server/events/yaml/raw/repo_cfg.go Outdated Show resolved Hide resolved

Fauzyy added 4 commits May 4, 2020 17:19

Merge branch 'master' of github.com:runatlantis/atlantis into paralle…

d078357

…l-plans-upstream

Fix mod vendor

d0940b6

Nope, go 1.14

526b801

Remove pool size option, set default to 15

4d1899f

Fauzyy force-pushed the parallel-plans-upstream branch from 741c892 to 4d1899f Compare May 6, 2020 00:34

Fauzyy added 2 commits May 5, 2020 18:15

Make parallel plans the default. Add flag for parallel apply

7452d6a

Fauzyy requested a review from lkysow May 8, 2020 05:47

gofmt -s

0e22d2b

Fauzyy force-pushed the parallel-plans-upstream branch from 3060bb0 to 0e22d2b Compare May 8, 2020 17:02

Bump CI

67ef5fc

lkysow approved these changes May 25, 2020

View reviewed changes

lkysow merged commit 6418b57 into runatlantis:master May 25, 2020

Lowess mentioned this pull request Jun 2, 2020

Run plan/apply for multiple projects in parallel #260

Open

Fauzyy deleted the parallel-plans-upstream branch May 4, 2021 23:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for parallel plans #926

Add support for parallel plans #926

Fauzyy commented Feb 7, 2020 •

edited

Loading

lkysow left a comment

Fauzyy commented Apr 6, 2020

lkysow commented Apr 6, 2020

Fauzyy commented Apr 6, 2020

adnankobir commented Apr 8, 2020

Fauzyy commented May 6, 2020

Fauzyy commented May 8, 2020 •

edited

Loading

lkysow commented May 8, 2020

Fauzyy commented May 8, 2020

codecov bot commented May 8, 2020

rverma-jm commented May 13, 2020

lkysow May 25, 2020

Fauzyy May 26, 2020

lkysow May 26, 2020

lkysow commented May 25, 2020

ghostsquad commented Oct 12, 2020

lkysow commented Oct 23, 2020

nwsparks commented May 4, 2021

mwarkentin commented May 4, 2021

dmattia commented May 4, 2021

Add support for parallel plans #926

Add support for parallel plans #926

Conversation

Fauzyy commented Feb 7, 2020 • edited Loading

lkysow left a comment

Choose a reason for hiding this comment

Fauzyy commented Apr 6, 2020

lkysow commented Apr 6, 2020

Fauzyy commented Apr 6, 2020

adnankobir commented Apr 8, 2020

Fauzyy commented May 6, 2020

Fauzyy commented May 8, 2020 • edited Loading

lkysow commented May 8, 2020

Fauzyy commented May 8, 2020

codecov bot commented May 8, 2020

Codecov Report

rverma-jm commented May 13, 2020

lkysow May 25, 2020

Choose a reason for hiding this comment

Fauzyy May 26, 2020

Choose a reason for hiding this comment

lkysow May 26, 2020

Choose a reason for hiding this comment

lkysow commented May 25, 2020

ghostsquad commented Oct 12, 2020

lkysow commented Oct 23, 2020

nwsparks commented May 4, 2021

mwarkentin commented May 4, 2021

dmattia commented May 4, 2021

Fauzyy commented Feb 7, 2020 •

edited

Loading

Fauzyy commented May 8, 2020 •

edited

Loading