Cloud Composer connection resource

[TV4Fun]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment. If the issue is assigned to the "modular-magician" user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If the issue is assigned to a user, that user is claiming responsibility for the issue. If the issue is assigned to "hashibot", a community member has claimed the issue already.

Description

In addition to being able to create Cloud Composer Environments through Terraform, it would be nice if we could set up connections, and perhaps some other internal config as well.

New or Affected Resource(s)

• google_composer_environment_connection

Potential Terraform Configuration

resource "google_composer_environment" "my-environment" {
  ...
}

# One possible example
resource "google_composer_environment_connection" "my-environment-postgres" {
  composer_environment = google_composer_environment.my-enviroment.name
  conn_id = "my-postgres-db"
  conn_type = "postgres"
  conn_host = "db.example.com"
  conn_port = 1234
  conn_login = username
  conn_password = file("password.txt")
  conn_extra = {
    sslmode = "prefer"
    keepalives_idle = 60
  }
}

This can be implemented via the gcloud utility e.g. gcloud composer environments run my-environment --location us-central1 connections -- --add, but there doesn't appear to be an equivalent REST API command. Still I'm sure there must be a way to do it. Just do whatever gcloud does.

b/299443256

[irvifa]

Hi @danawillow when will this implemented in terraform?

[danawillow]

We don't have a timeline for this feature yet. We're happy to accept PRs (see https://github.com/terraform-providers/terraform-provider-google/blob/master/.github/CONTRIBUTING.md; we generally add new resources viaMagic Modules instead of in TF directly). Otherwise, if you have an account manager, customer engineer, or other contact at Google or HashiCorp, you can ask them to reach out to our team to prioritize it higher.

[oslobodian]

Helo @danawillow, any chance for making this happen in 2020?

[danawillow]

Same comment as before applies!

[irvifa]

@danawillow I was wondering if there's specific API in golang equivalent with run command. From what I've seen in the API. Note that we're using this for the Terraform Provider there's several utility for ListCall, CreateCall, and DeleteCall. However there's no specific API to run an environment.

[danawillow]

I don't see anything at the link you posted or at https://cloud.google.com/composer/docs/reference/rest. If you run gcloud with --log-http, it'll tell you what requests it's making. Maybe there's a clue there, but it's also possible that it's calling an endpoint that isn't accessible from tools that aren't gcloud. For new resources we'd want them generated with magic modules anyway, so it doesn't matter whether it's in the golang SDK or not, as long as we can call the API publicly.

[franviera92]

If necessary this feature

[raphaelauv]

Do not use terraform for everything !

airflow connections
airflow pools
airflow users
airflow variables

store them in JSON files that you load with the airflow cli or airflow REST API ( since airflow v2)

Do that in your CD pipelines the same way you publish your airflow DAGS and airflow plugins with your CD and not terraform

[DrFaust92]

FYI I forked the airflow tf provider and added support for this and more:
https://registry.terraform.io/providers/DrFaust92/airflow/latest

[maguro]

Do not use terraform for everything !

airflow connections airflow pools airflow users airflow variables

store them in JSON files that you load with the airflow cli or airflow REST API ( since airflow v2)

Do that in your CD pipelines the same way you publish your airflow DAGS and airflow plugins with your CD and not terraform

We use Terraform to publish DAGs etc.

[rileykarson]

Moving into Goals to mark that we'd like to support this, if possible. We're not sure that this is possible through the Composer API today (although gcloud seems to be able to, as folks suggest). We'll reach out to Cloud Composer engineers to figure out whether we're just missing something.

If this is not possible through their API, we'd want to recommend using other providers, most likely.

[michalmodras]

That would require exposing native Airflow configuration directly in Terraform - something we don't do at the moment and do not have plans in the imminent future.