refactor: use `hc-install` for TF downloads + constraints #4494

james0209 · 2024-04-30T02:15:07Z

What

Replace go-getter usage with hc-install usage for both default + custom URL TF downloads.

Why

Currently, a URL based on system architecture etc is put together to produce a download link. go-getter's GetFile is then used to get the binary. This is broken with TF >= 1.8.2 as a License.txt file is also included. TF maintainer suggested looking into hc-install instead.

Changes

URL Downloads

This refactor uses hc-install to:

Get a list of Versions based on a constraint
Install the binary (installs as terraform by default so we rename the binary to terraform{version} to match current convention)

Custom URL's

For custom URL usage, the root URL must have the same structure as the default releases.hashicorp.com layout. e.g. it must have 2 index.json files; 1 in the project root and 1 in the version root. e.g.
https://releases.hashicorp.com/terraform/index.json
https://releases.hashicorp.com/terraform/1.8.2/index.json

This is because hc-install parses these json files behind the scenes for it's logic.

Example setup can be seen in this commit: 39443b3

The URL's in the json can be kept as the default Hashicorp releases url's - if a custom ApiBaseUrl is set, hc-install will use that instead as the base URL - the URL in the JSON is ignored.

Can be done via mirroring the website, or an approach similar to this to make one locally

# URL of the root directory
root_url="https://releases.hashicorp.com/terraform"

# Download the root index.json file
curl -s "${root_url}/index.json" -o "index.json"


# Download the version-specific index.json file into the version directory
curl -s "${root_url}/${version}/index.json" -o "${version}/index.json"

# Extract the file names from the version-specific index.json file and download each file
jq -r '.builds[].url' index.json | while read file_url; do
  curl -O "${file_url}"
done

Removing usage of other packages

Usage of warrensbox/terraform-switcher is removed

Was used to scrape the API to get a list of TF versions (ref) and to ensure ValidVersionFormat (ref)
Both of these use-cases are covered by hc-install

Usage of go-getter's GetFile is also removed.

The usage has been replaced by hc-install, and it will not be required by eventual addition of OpemTofu support, as "GetAny" will be used for that to accommodate extra files in the ZIP.
e.g. The current WIP PR to add Tofu support uses GetAny.

Future support for OpenTofu

Whilst hc-install cannot be used to install OpenTofu, I personally don't see that as a good reason not to use it for Terraform.

Due to possible divergences in the future (look at the License File in the zip change, unexpected), the same method of go-getter etc. may not be viable for both anyway - there may end up being a lot of conditionals for vendor-specific changes.

I think a better approach is to have different "retrievers" for the different softwares. hc-install for Terraform, go-getter or some other approach for OpenTofu etc.

Tests

Seems to pass Unit Tests and Integration Tests.

Local Testing

Ran Atlants + ngrok locally

terraform {
  required_providers {
  }

  required_version = ">= 1.5.0"
}

{"level":"debug","ts":"2024-06-12T19:29:51.162+0100","caller":"terraform/terraform_client.go:309","msg":"Found required_version setting of \"= 1.5.0\"","json":{"repo":"james0209/test-terraform","pull":"1"}}
2024/06/12 19:29:51 [DEBUG] GET https://releases.hashicorp.com/terraform/index.json
{"level":"info","ts":"2024-06-12T19:29:51.829+0100","caller":"terraform/terraform_client.go:553","msg":"could not find terraform version 1.5.0 in PATH or /home/jamesbrookes/.atlantis/bin","json":{"repo":"james0209/test-terraform","pull":"1"}}
{"level":"info","ts":"2024-06-12T19:29:51.829+0100","caller":"terraform/terraform_client.go:555","msg":"using Hashicorp's 'hc-install' to download Terraform version 1.5.0 from download URL https://releases.hashicorp.com","json":{"repo":"james0209/test-terraform","pull":"1"}}
2024/06/12 19:29:51 [DEBUG] GET https://releases.hashicorp.com/terraform/1.5.0/index.json
2024/06/12 19:29:52 [DEBUG] GET https://releases.hashicorp.com/terraform/1.5.0/terraform_1.5.0_SHA256SUMS.72D7468F.sig
2024/06/12 19:29:52 [DEBUG] GET https://releases.hashicorp.com/terraform/1.5.0/terraform_1.5.0_SHA256SUMS
2024/06/12 19:29:52 [DEBUG] GET https://releases.hashicorp.com/terraform/1.5.0/terraform_1.5.0_linux_amd64.zip
{"level":"info","ts":"2024-06-12T19:29:53.545+0100","caller":"terraform/terraform_client.go:562","msg":"Downloaded terraform 1.5.0 to /home/jamesbrookes/.atlantis/bin/terraform1.5.0","json":{"repo":"james0209/test-terraform","pull":"1"}}
{"level":"debug","ts":"2024-06-12T19:29:53.545+0100","caller":"models/shell_command_runner.go:95","msg":"starting \"/home/jamesbrookes/.atlantis/bin/terraform1.5.0 init -input=false -upgrade\" in \"/home/jamesbrookes/.atlantis/repos/james0209/test-terraform/1/default\"","json":{"repo":"james0209/test-terraform","pull":"1"}}
{"level":"info","ts":"2024-06-12T19:29:53.560+0100","caller":"models/shell_command_runner.go:161","msg":"successfully ran \"/home/jamesbrookes/.atlantis/bin/terraform1.5.0 init -input=false -upgrade\" in \"/home/jamesbrookes/.atlantis/repos/james0209/test-terraform/1/default\"","json":{"repo":"james0209/test-terraform","pull":"1","duration":0.014672532}}
{"level":"info","ts":"2024-06-12T19:29:53.575+0100","caller":"terraform/terraform_client.go:412","msg":"successfully ran \"/home/jamesbrookes/.atlantis/bin/terraform1.5.0 workspace show\" in \"/home/jamesbrookes/.atlantis/repos/james0209/test-terraform/1/default\"","json":{"repo":"james0209/test-terraform","pull":"1","duration":0.015204647}}
{"level":"debug","ts":"2024-06-12T19:29:53.576+0100","caller":"models/shell_command_runner.go:95","msg":"starting \"/home/jamesbrookes/.atlantis/bin/terraform1.5.0 plan -input=false -refresh -out \\\"/home/jamesbrookes/.atlantis/repos/james0209/test-terraform/1/default/default.tfplan\\\"\" in \"/home/jamesbrookes/.atlantis/repos/james0209/test-terraform/1/default\"","json":{"repo":"james0209/test-terraform","pull":"1"}}
{"level":"info","ts":"2024-06-12T19:29:53.594+0100","caller":"models/shell_command_runner.go:161","msg":"successfully ran \"/home/jamesbrookes/.atlantis/bin/terraform1.5.0 plan -input=false -refresh -out \\\"/home/jamesbrookes/.atlantis/repos/james0209/test-terraform/1/default/default.tfplan\\\"\" in \"/home/jamesbrookes/.atlantis/repos/james0209/test-terraform/1/default\"","json":{"repo":"james0209/test-terraform","pull":"1","duration":0.018726578}}

terraform {
  required_providers {
  }

  required_version = "= 1.7.5"
}

{"level":"debug","ts":"2024-06-12T19:26:01.126+0100","caller":"terraform/terraform_client.go:309","msg":"Found required_version setting of \"= 1.7.5\"","json":{"repo":"james0209/test-terraform","pull":"1"}}
2024/06/12 19:26:01 [DEBUG] GET https://releases.hashicorp.com/terraform/index.json
{"level":"info","ts":"2024-06-12T19:26:01.861+0100","caller":"terraform/terraform_client.go:553","msg":"could not find terraform version 1.7.5 in PATH or /home/jamesbrookes/.atlantis/bin","json":{"repo":"james0209/test-terraform","pull":"1"}}
{"level":"info","ts":"2024-06-12T19:26:01.861+0100","caller":"terraform/terraform_client.go:555","msg":"using Hashicorp's 'hc-install' to download Terraform version 1.7.5 from download URL https://releases.hashicorp.com","json":{"repo":"james0209/test-terraform","pull":"1"}}
2024/06/12 19:26:01 [DEBUG] GET https://releases.hashicorp.com/terraform/1.7.5/index.json
2024/06/12 19:26:01 [DEBUG] GET https://releases.hashicorp.com/terraform/1.7.5/terraform_1.7.5_SHA256SUMS.72D7468F.sig
2024/06/12 19:26:02 [DEBUG] GET https://releases.hashicorp.com/terraform/1.7.5/terraform_1.7.5_SHA256SUMS
2024/06/12 19:26:02 [DEBUG] GET https://releases.hashicorp.com/terraform/1.7.5/terraform_1.7.5_linux_amd64.zip
{"level":"info","ts":"2024-06-12T19:26:03.144+0100","caller":"terraform/terraform_client.go:562","msg":"Downloaded terraform 1.7.5 to /home/jamesbrookes/.atlantis/bin/terraform1.7.5","json":{"repo":"james0209/test-terraform","pull":"1"}}
{"level":"debug","ts":"2024-06-12T19:26:03.144+0100","caller":"models/shell_command_runner.go:95","msg":"starting \"/home/jamesbrookes/.atlantis/bin/terraform1.7.5 init -input=false -upgrade\" in \"/home/jamesbrookes/.atlantis/repos/james0209/test-terraform/1/default\"","json":{"repo":"james0209/test-terraform","pull":"1"}}
{"level":"info","ts":"2024-06-12T19:26:03.167+0100","caller":"models/shell_command_runner.go:161","msg":"successfully ran \"/home/jamesbrookes/.atlantis/bin/terraform1.7.5 init -input=false -upgrade\" in \"/home/jamesbrookes/.atlantis/repos/james0209/test-terraform/1/default\"","json":{"repo":"james0209/test-terraform","pull":"1","duration":0.023179423}}
{"level":"info","ts":"2024-06-12T19:26:03.187+0100","caller":"terraform/terraform_client.go:412","msg":"successfully ran \"/home/jamesbrookes/.atlantis/bin/terraform1.7.5 workspace show\" in \"/home/jamesbrookes/.atlantis/repos/james0209/test-terraform/1/default\"","json":{"repo":"james0209/test-terraform","pull":"1","duration":0.019830465}}
{"level":"debug","ts":"2024-06-12T19:26:03.187+0100","caller":"models/shell_command_runner.go:95","msg":"starting \"/home/jamesbrookes/.atlantis/bin/terraform1.7.5 plan -input=false -refresh -out \\\"/home/jamesbrookes/.atlantis/repos/james0209/test-terraform/1/default/default.tfplan\\\"\" in \"/home/jamesbrookes/.atlantis/repos/james0209/test-terraform/1/default\"","json":{"repo":"james0209/test-terraform","pull":"1"}}
{"level":"info","ts":"2024-06-12T19:26:03.206+0100","caller":"models/shell_command_runner.go:161","msg":"successfully ran \"/home/jamesbrookes/.atlantis/bin/terraform1.7.5 plan -input=false -refresh -out \\\"/home/jamesbrookes/.atlantis/repos/james0209/test-terraform/1/default/default.tfplan\\\"\" in \"/home/jamesbrookes/.atlantis/repos/james0209/test-terraform/1/default\"","json":{"repo":"james0209/test-terraform","pull":"1","duration":0.018946915}}

Test: Remove <=1.8.2 constraint, test that hc-install correctly downloads it.

terraform {
  required_providers {
  }

  required_version = "= 1.8.3"
}

{"level":"info","ts":"2024-06-12T19:10:14.647+0100","caller":"terraform/terraform_client.go:553","msg":"could not find terraform version 1.8.3 in PATH or /home/jamesbrookes/.atlantis/bin","json":{"repo":"james0209/test-terraform","pull":"1"}}
{"level":"info","ts":"2024-06-12T19:10:14.647+0100","caller":"terraform/terraform_client.go:555","msg":"using Hashicorp's 'hc-install' to download Terraform version 1.8.3 from download URL https://releases.hashicorp.com","json":{"repo":"james0209/test-terraform","pull":"1"}}
2024/06/12 19:10:14 [DEBUG] GET https://releases.hashicorp.com/terraform/1.8.3/index.json
2024/06/12 19:10:15 [DEBUG] GET https://releases.hashicorp.com/terraform/1.8.3/terraform_1.8.3_SHA256SUMS.72D7468F.sig
2024/06/12 19:10:15 [DEBUG] GET https://releases.hashicorp.com/terraform/1.8.3/terraform_1.8.3_SHA256SUMS
2024/06/12 19:10:15 [DEBUG] GET https://releases.hashicorp.com/terraform/1.8.3/terraform_1.8.3_linux_amd64.zip
{"level":"info","ts":"2024-06-12T19:10:16.363+0100","caller":"terraform/terraform_client.go:562","msg":"Downloaded terraform 1.8.3 to /home/jamesbrookes/.atlantis/bin/terraform1.8.3","json":{"repo":"james0209/test-terraform","pull":"1"}}
{"level":"debug","ts":"2024-06-12T19:10:16.363+0100","caller":"models/shell_command_runner.go:95","msg":"starting \"/home/jamesbrookes/.atlantis/bin/terraform1.8.3 init -input=false -upgrade\" in \"/home/jamesbrookes/.atlantis/repos/james0209/test-terraform/1/default\"","json":{"repo":"james0209/test-terraform","pull":"1"}}
{"level":"info","ts":"2024-06-12T19:10:16.385+0100","caller":"models/shell_command_runner.go:161","msg":"successfully ran \"/home/jamesbrookes/.atlantis/bin/terraform1.8.3 init -input=false -upgrade\" in \"/home/jamesbrookes/.atlantis/repos/james0209/test-terraform/1/default\"","json":{"repo":"james0209/test-terraform","pull":"1","duration":0.022444605}}
{"level":"info","ts":"2024-06-12T19:10:16.407+0100","caller":"terraform/terraform_client.go:412","msg":"successfully ran \"/home/jamesbrookes/.atlantis/bin/terraform1.8.3 workspace show\" in \"/home/jamesbrookes/.atlantis/repos/james0209/test-terraform/1/default\"","json":{"repo":"james0209/test-terraform","pull":"1","duration":0.02133343}}
{"level":"debug","ts":"2024-06-12T19:10:16.407+0100","caller":"models/shell_command_runner.go:95","msg":"starting \"/home/jamesbrookes/.atlantis/bin/terraform1.8.3 plan -input=false -refresh -out \\\"/home/jamesbrookes/.atlantis/repos/james0209/test-terraform/1/default/default.tfplan\\\"\" in \"/home/jamesbrookes/.atlantis/repos/james0209/test-terraform/1/default\"","json":{"repo":"james0209/test-terraform","pull":"1"}}
{"level":"info","ts":"2024-06-12T19:10:16.429+0100","caller":"models/shell_command_runner.go:161","msg":"successfully ran \"/home/jamesbrookes/.atlantis/bin/terraform1.8.3 plan -input=false -refresh -out \\\"/home/jamesbrookes/.atlantis/repos/james0209/test-terraform/1/default/default.tfplan\\\"\" in \"/home/jamesbrookes/.atlantis/repos/james0209/test-terraform/1/default\"","json":{"repo":"james0209/test-terraform","pull":"1","duration":0.022609425}}

terraform {
  required_providers {
  }

  required_version = "= 1.8.4"
}

{"level":"info","ts":"2024-06-12T19:09:07.645+0100","caller":"terraform/terraform_client.go:553","msg":"could not find terraform version 1.8.4 in PATH or /home/jamesbrookes/.atlantis/bin","json":{"repo":"james0209/test-terraform","pull":"1"}}
{"level":"info","ts":"2024-06-12T19:09:07.645+0100","caller":"terraform/terraform_client.go:555","msg":"using Hashicorp's 'hc-install' to download Terraform version 1.8.4 from download URL https://releases.hashicorp.com","json":{"repo":"james0209/test-terraform","pull":"1"}}
2024/06/12 19:09:07 [DEBUG] GET https://releases.hashicorp.com/terraform/1.8.4/index.json
2024/06/12 19:09:07 [DEBUG] GET https://releases.hashicorp.com/terraform/1.8.4/terraform_1.8.4_SHA256SUMS.72D7468F.sig
2024/06/12 19:09:07 [DEBUG] GET https://releases.hashicorp.com/terraform/1.8.4/terraform_1.8.4_SHA256SUMS
2024/06/12 19:09:07 [DEBUG] GET https://releases.hashicorp.com/terraform/1.8.4/terraform_1.8.4_linux_amd64.zip
{"level":"info","ts":"2024-06-12T19:09:08.955+0100","caller":"terraform/terraform_client.go:562","msg":"Downloaded terraform 1.8.4 to /home/jamesbrookes/.atlantis/bin/terraform1.8.4","json":{"repo":"james0209/test-terraform","pull":"1"}}
{"level":"debug","ts":"2024-06-12T19:09:08.955+0100","caller":"models/shell_command_runner.go:95","msg":"starting \"/home/jamesbrookes/.atlantis/bin/terraform1.8.4 init -input=false -upgrade\" in \"/home/jamesbrookes/.atlantis/repos/james0209/test-terraform/1/default\"","json":{"repo":"james0209/test-terraform","pull":"1"}}
{"level":"info","ts":"2024-06-12T19:09:08.979+0100","caller":"models/shell_command_runner.go:161","msg":"successfully ran \"/home/jamesbrookes/.atlantis/bin/terraform1.8.4 init -input=false -upgrade\" in \"/home/jamesbrookes/.atlantis/repos/james0209/test-terraform/1/default\"","json":{"repo":"james0209/test-terraform","pull":"1","duration":0.024192705}}
{"level":"info","ts":"2024-06-12T19:09:09.000+0100","caller":"terraform/terraform_client.go:412","msg":"successfully ran \"/home/jamesbrookes/.atlantis/bin/terraform1.8.4 workspace show\" in \"/home/jamesbrookes/.atlantis/repos/james0209/test-terraform/1/default\"","json":{"repo":"james0209/test-terraform","pull":"1","duration":0.02076715}}
{"level":"debug","ts":"2024-06-12T19:09:09.001+0100","caller":"models/shell_command_runner.go:95","msg":"starting \"/home/jamesbrookes/.atlantis/bin/terraform1.8.4 plan -input=false -refresh -out \\\"/home/jamesbrookes/.atlantis/repos/james0209/test-terraform/1/default/default.tfplan\\\"\" in \"/home/jamesbrookes/.atlantis/repos/james0209/test-terraform/1/default\"","json":{"repo":"james0209/test-terraform","pull":"1"}}
{"level":"info","ts":"2024-06-12T19:09:09.023+0100","caller":"models/shell_command_runner.go:161","msg":"successfully ran \"/home/jamesbrookes/.atlantis/bin/terraform1.8.4 plan -input=false -refresh -out \\\"/home/jamesbrookes/.atlantis/repos/james0209/test-terraform/1/default/default.tfplan\\\"\" in \"/home/jamesbrookes/.atlantis/repos/james0209/test-terraform/1/default\"","json":{"repo":"james0209/test-terraform","pull":"1","duration":0.022099695}}

Notes

Atlantis will just store the LICENSE.txt file alongside the downloaded bin file (the default when downloading via hc-install, although there is a LicenseDir arg that we can customise.
- Atlantis does not currently have seperation of TF version executables - there is a bin dir in which they are all placed with the name terraform<version>.
- Because of this, LICENSE.txt is overwritten each time a version >= TF 1.8.2 is downloaded, as it is always saved to the same location.
- This is currently not an issue, as the LICENSE.txt file stays the same.
- Follow-up PR could maybe change the "bin" dir structure so each tf vertsion is downloaded into it's own dir, so the LICENSE can be placed alongside it?
  - We would not need to change the name of the downloaded files that way either. Instead, just have dirs such as 1.8.1, 1.8.2, and the executables at 1.8.1/terraform, 1.8.2/terraform etc.

References

closes Consider using hc-install instead of go-getter to install terraform binary #4483

…ethod for custom download URL's

nitrocode · 2024-04-30T04:20:44Z

One downside of swapping out warrensbox for hc-install is that we lose the potential to reuse the same library to download opentofu versions.

Is opentofu possible with hc-install or would we need to add additional code to support it? If additional code, would swapping to hc-install be worth it in the long run?

server/core/terraform/terraform_client.go

james0209 · 2024-04-30T17:27:34Z

One downside of swapping out warrensbox for hc-install is that we lose the potential to reuse the same library to download opentofu versions.

Is opentofu possible with hc-install or would we need to add additional code to support it? If additional code, would swapping to hc-install be worth it in the long run?

@nitrocode I do not think installing OpenTofu with hc-install is possible unfortunately. There are a couple of things that come to my mind:

Atlantis got caught out by the Terraform license file change as there wasn't much forewarning; the current method is brittle; no thought that it would break anything etc. I think using their first-party tooling which they also use internally would be the best bet to not run into any other issues regarding this?
I don't see any issue with using hc-install for Terraform and a different approach for OpenTofu? i.e. when it comes to supporting OpenTofu, we could have a terraformretriever.go and a tofuretriever.go for example. Similar to how tenv has different retrievers for Terraform here and Tofu here

I believe hc-install is also open license, so from my understanding it's not out of the realm of possibility for OpenTofu to fork/adapt a verson for themselves.

It also looks as though warrensbox/terraform-switcher doesn't yet support OpenTofu and there isn't a timeline yet for it's support - based on the open issue I see over there anyway - I haven't dived into it's code yet.

I don't think Atlantis should pigeon-hole itself into using the same methods for obtaining Terraform and it's forks given the divergences that could come in the future - if there's a tool designed for obtaining Terraform and it is the best way forward, why not use it, and use a more suited approach for OpenTofu?

From what I can gather, Atlantis doesn't even use terraform-switcher libs that much. It's used to scrape the API to get a list of TF versions (ref) and to ensure ValidVersionFormat (ref), but in-house Atlantis code deals with constraining the list, selecting the version, and downloading the binary via go-getter

There could be a discussion about using tenv, tfenv, terraform-switcher etc. to actually download Terraform?

This is my first PR for Atlantis and I don't know the in's and out's too well yet so I'm more than happy to hear other opinions/views!

jamengual · 2024-04-30T18:54:52Z

Atlantis got caught out by the Terraform license file change as there wasn't much forewarning; the current method is brittle; no thought that it would break anything etc. I think using their first-party tooling which they also use internally would be the best bet to not run into any other issues regarding this?

no, we got Hashicorp approval to use terraform with Atlantis, we do not have that problem.

james0209 · 2024-04-30T18:59:12Z

Atlantis got caught out by the Terraform license file change as there wasn't much forewarning; the current method is brittle; no thought that it would break anything etc. I think using their first-party tooling which they also use internally would be the best bet to not run into any other issues regarding this?

no, we got Hashicorp approval to use terraform with Atlantis, we do not have that problem.

Sorry if I wasn't clear - I don't mean the Hashicorp License change - I mean the inclusion of the license file in the packaged TF that made Atlantis have to constrain below TF 1.8.2. I'm not questioning Atlantis' perms to use TF, merely saying that the current system broke because of the inclusion of a License File. If anyones tooling is most likely to not break from changes to TF packaging, it's Hashicorps own installer that they use internally

…n't have to do s many conversions

nitrocode · 2024-06-13T05:58:54Z

For what it's worth, the warrensbox package now supports opentofu

warrensbox/terraform-switcher#435
warrensbox/terraform-switcher#315

It's pending an official release.

The downside of supporting a tf specific package (hc-getter) is that we need to reimplement it for opentofu.

Since the upstream package is planning to support it, then we might as well wait, upgrade the dependency, and expose the inputs. This would reduce maintenance for us.

caiodelgadonew · 2024-06-13T10:16:05Z

@nitrocode have you looked into tfenv?
I use it and I think it's amazing..

https://github.com/tfutils/tfenv

nitrocode · 2024-06-13T11:28:03Z

I think youre suggesting to use a different library that supports both opentofu and terraform which would be additional work to fit in into atlantis. I haven't evaluated that dependency.

If a single dependency is chosen, it supports both opentofu and terraform, it's well maintained, easy to implement, then that can be used.

Either way (warrensbox or tfenv), using hc-install would be more effort to maintain when it may be easier to defer that to a separate library (warrensbox or tfenv).

james0209 · 2024-06-13T11:49:47Z

For what it's worth, the warrensbox package now supports opentofu

warrensbox/terraform-switcher#435 warrensbox/terraform-switcher#315

It's pending an official release.

The downside of supporting a tf specific package (hc-getter) is that we need to reimplement it for opentofu.

Since the upstream package is planning to support it, then we might as well wait, upgrade the dependency, and expose the inputs. This would reduce maintenance for us.

@nitrocode The way Atlantis currently uses terraform-switcher lib currently is not for the actual downloading part of the process.

It uses a GetTFList method from the lib to return all available versions of TF (ref). That is currently the only implementation of terraform-switcher in Atlantis (that function has since been made private upstream (ref).

Are you talking about fully-implementing terraform-switcher to do the downloading of Terraform? I think it would be possible - looks like you could make use GetSemver and InstallProductVersion.

Although if the tfswitch route is to be taken, you should be mindful of:

Add error return values to signatures of public methods to allow migrating to returning errors rather than Fatals in future warrensbox/terraform-switcher#457
What do do about public methods warrensbox/terraform-switcher#452
There is a PR as you can see - but tfswitch is only just maybe moving to "officially" supporting other libs importing it (e.g. until that PR is merged, no errors are thrown from public funcs, just os.Exit()

I don't mind which route is chosen; I made this PR a month and a half ago to try and fix something that is still broken.

nitrocode · 2024-06-13T11:56:14Z

Are you talking about fully-implementing terraform-switcher to do the downloading of Terraform?

Yes that's exactly what I mean.

Since the library added support for opentofu, we can use the library to also download the binary.

I made this PR a month and a half ago to try and fix something that is still broken.

What is broken? I thought this was originally fixing the issue where 1.8.2+ could not be installed but that has since been resolved.

After that was fixed, i thought this pr was being pushed to insulate against future terraform zip changes.

james0209 · 2024-06-13T12:00:42Z

Are you talking about fully-implementing terraform-switcher to do the downloading of Terraform?

Yes that's exactly what I mean.

Since the library added support for opentofu, we can use the library to also download the binary.

I made this PR a month and a half ago to try and fix something that is still broken.

What is broken? I thought this was originally fixing the issue where 1.8.2+ could not be installed but that has since been resolved.

After that was fixed, i thought this pr was being pushed to insulate against future terraform zip changes.

Nope. >= 1.8.2 download is still not possible; the temporary fix implemented was a constraint preventing Atlantis from downloading greater than that. The only version greater than that available to Atlantis is the one that is pre-packaged into the Docker image as the default (1.8.4 or 1.8.5 now I think?). Which is why the lack of reviews/discussion around this PR has been a bit annoying.

atlantis/server/core/terraform/terraform_client.go

Lines 341 to 345 in 854c287

    
           // Since terraform version 1.8.2, terraform is not a single file download anymore and 
        
           // Atlantis fails to download version 1.8.2 and higher. So, as a short-term fix, 
        
           // we need to block any version higher than 1.8.1 until proper solution is implemented. 
        
           // More details on the issue here - https://github.com/runatlantis/atlantis/issues/4471 
        
           highestSupportedConstraint, _ := version.NewConstraint("<= 1.8.1")

@nitrocode

GenPage · 2024-06-13T17:03:41Z

I will go ahead and weigh in here. @james0209, I apologize for the lack of review from the team. We are stretched very thin, and my focus has been on growing the community and our team so that we can adequately handle reviews promptly. @nitrocode I appreciate your driving the discussion thus far.

I want us all to align on a simple solution agnostic to the workloads users wish to run, whether it be OpenTofu, Terraform, etc. However, given the nature of the broken user experience, I think it would be prudent to proceed with this PR that has a solution ready and revisit this in a future refactor once we've decided on an approach for our long-term goals to support Terraform and OpenTofu.

@nitrocode @jamengual @lukemassa

jamengual · 2024-06-13T19:02:33Z

sounds good to me.

nitrocode · 2024-06-14T06:24:37Z

I didn't realize it was still broken, thanks for the comment. My apologies.

GenPage · 2024-06-15T23:41:58Z

@james0209 its approved and automerged is enabled. Just need to resolve the merge conflicts.

james0209 · 2024-06-15T23:53:08Z

@james0209 its approved and automerged is enabled. Just need to resolve the merge conflicts.

Fixing the conflict unfortunately disabled auto-merge because I don't have write access @GenPage

WIP: Use hc-install for default downloading - still currently old m…

051535b

…ethod for custom download URL's

james0209 requested review from a team as code owners April 30, 2024 02:15

james0209 requested review from jamengual, lukemassa and X-Guardian and removed request for a team April 30, 2024 02:15

github-actions bot added dependencies PRs that update a dependency file go Pull requests that update Go code labels Apr 30, 2024

james0209 changed the title ~~WIP: Use hc-install for default downloading - still currently old method for custom download URL's~~ feat: use hc-install for default URL TF downloads Apr 30, 2024

james0209 changed the title ~~feat: use hc-install for default URL TF downloads~~ refactor: use hc-install for default URL TF downloads Apr 30, 2024

Lowercase logs

5bc8d18

Little golines fix

c32ccae

jamengual added the waiting-on-response Waiting for a response from the user label Apr 30, 2024

radeksimko reviewed Apr 30, 2024

View reviewed changes

server/core/terraform/terraform_client.go Show resolved Hide resolved

server/core/terraform/terraform_client.go Outdated Show resolved Hide resolved

nitrocode mentioned this pull request Apr 30, 2024

Autodownloading Terraform 1.8.2 breaks Atlantis entirely #4471

Closed

Merge branch 'main' into use-hc-install

06a0c5c

james0209 added 7 commits April 30, 2024 21:18

Use *version.Version instead of string as arg to Install so we do…

1ae96b6

…n't have to do s many conversions

Update comment based on local testing of hc-install

cf7e829

Add some testing of separate functions

3ee7692

Expand regex test

e8faad4

Move custom download URL logic to it's own func

871d79c

Merge branch 'main' into use-hc-install

2982a03

Add extra log message for extracted version

eb56d06

james0209 mentioned this pull request May 1, 2024

Consider using hc-install instead of go-getter to install terraform binary #4483

Closed

1 task

jamengual and others added 2 commits June 12, 2024 14:12

Merge branch 'main' into use-hc-install

49ce10a

Merge branch 'main' into use-hc-install

1753090

jamengual previously approved these changes Jun 13, 2024

View reviewed changes

nitrocode removed needs discussion Large change that needs review from community/maintainers waiting-on-review Waiting for a review from a maintainer labels Jun 14, 2024

Luay-Sol approved these changes Jun 14, 2024

View reviewed changes

GenPage enabled auto-merge (squash) June 15, 2024 23:40

Merge branch 'main' into use-hc-install

0ce27d6

auto-merge was automatically disabled June 15, 2024 23:51
Head branch was pushed to by a user without write access

james0209 dismissed jamengual’s stale review via 0ce27d6 June 15, 2024 23:51

Merge branch 'main' into use-hc-install

9841851

jamengual enabled auto-merge (squash) June 16, 2024 01:34

jamengual approved these changes Jun 16, 2024

View reviewed changes

jamengual merged commit a6a3b21 into runatlantis:main Jun 16, 2024
28 checks passed

BrewTestBot mentioned this pull request Jun 18, 2024

atlantis 0.28.2 Homebrew/homebrew-core#175011

Closed

1 task

james0209 mentioned this pull request Jun 20, 2024

fix: adapt terraform download due to new folder zip structure #4478

Closed

lukemassa mentioned this pull request Jul 9, 2024

expected a single file: /tmp/getter3359610012/archive: Downloading terraform Version 1.8.2 hashicorp/terraform#35075

Closed

james0209 mentioned this pull request Aug 9, 2024

[Bug]: Error: configmaps "aws-auth" is forbidden: User "system:anonymous" cannot get resource "configmaps" in API group "" in the namespace "kube-system" hashicorp/terraform-provider-aws#38638

Closed

plyr4 mentioned this pull request Aug 19, 2024

fix: binary installation, checksums, cleanup go-vela/vela-terraform#181

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor: use `hc-install` for TF downloads + constraints #4494

refactor: use `hc-install` for TF downloads + constraints #4494

james0209 commented Apr 30, 2024 •

edited

Loading

nitrocode commented Apr 30, 2024

james0209 commented Apr 30, 2024 •

edited

Loading

jamengual commented Apr 30, 2024

james0209 commented Apr 30, 2024 •

edited

Loading

nitrocode commented Jun 13, 2024 •

edited

Loading

caiodelgadonew commented Jun 13, 2024

nitrocode commented Jun 13, 2024

james0209 commented Jun 13, 2024 •

edited

Loading

nitrocode commented Jun 13, 2024

james0209 commented Jun 13, 2024 •

edited

Loading

GenPage commented Jun 13, 2024 •

edited

Loading

jamengual commented Jun 13, 2024

nitrocode commented Jun 14, 2024

GenPage commented Jun 15, 2024

james0209 commented Jun 15, 2024 •

edited

Loading

refactor: use hc-install for TF downloads + constraints #4494

refactor: use hc-install for TF downloads + constraints #4494

Conversation

james0209 commented Apr 30, 2024 • edited Loading

What

Why

Changes

URL Downloads

Custom URL's

Removing usage of other packages

Future support for OpenTofu

Tests

Local Testing

Notes

References

nitrocode commented Apr 30, 2024

james0209 commented Apr 30, 2024 • edited Loading

jamengual commented Apr 30, 2024

james0209 commented Apr 30, 2024 • edited Loading

nitrocode commented Jun 13, 2024 • edited Loading

caiodelgadonew commented Jun 13, 2024

nitrocode commented Jun 13, 2024

james0209 commented Jun 13, 2024 • edited Loading

nitrocode commented Jun 13, 2024

james0209 commented Jun 13, 2024 • edited Loading

GenPage commented Jun 13, 2024 • edited Loading

jamengual commented Jun 13, 2024

nitrocode commented Jun 14, 2024

GenPage commented Jun 15, 2024

james0209 commented Jun 15, 2024 • edited Loading

refactor: use `hc-install` for TF downloads + constraints #4494

refactor: use `hc-install` for TF downloads + constraints #4494

james0209 commented Apr 30, 2024 •

edited

Loading

james0209 commented Apr 30, 2024 •

edited

Loading

james0209 commented Apr 30, 2024 •

edited

Loading

nitrocode commented Jun 13, 2024 •

edited

Loading

james0209 commented Jun 13, 2024 •

edited

Loading

james0209 commented Jun 13, 2024 •

edited

Loading

GenPage commented Jun 13, 2024 •

edited

Loading

james0209 commented Jun 15, 2024 •

edited

Loading