Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

google_bigquery_dataset - infinite diff if access is specified with non-legacy IAM roles #8370

Open
jcanseco opened this issue Feb 2, 2021 · 7 comments

Comments

@jcanseco
Copy link
Contributor

jcanseco commented Feb 2, 2021

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
  • Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment.
  • If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

Terraform v0.14.0
+ provider registry.terraform.io/hashicorp/google v5.0.0

Affected Resource(s)

  • google_bigquery_dataset

Terraform Configuration Files

provider "google" {
  project = "cnrm-jcanseco-2"
}

resource "google_bigquery_dataset" "dataset" {
  dataset_id = "bigquerydatasetsampletf"
  access {
    role = "roles/bigquery.dataOwner"
    user_by_email = google_service_account.bqowner.email
  }
}

resource "google_service_account" "bqowner" {
  account_id = "bqowner"
}

Debug Output

https://gist.github.com/jcanseco/f51d951db4a30378a0183cef0d21de08

Expected Behavior

Running terraform plan after terraform apply should return an empty diff.

Actual Behavior

Running terraform plan after terraform apply returned a non-empty diff:

$ terraform plan 
google_service_account.bqowner: Refreshing state... [id=projects/cnrm-jcanseco-2/serviceAccounts/[email protected]]
google_bigquery_dataset.dataset: Refreshing state... [id=projects/cnrm-jcanseco-2/datasets/bigquerydatasetsampletf]

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # google_bigquery_dataset.dataset will be updated in-place
  ~ resource "google_bigquery_dataset" "dataset" {
        id                              = "projects/cnrm-jcanseco-2/datasets/bigquerydatasetsampletf"
        # (11 unchanged attributes hidden)

      - access {
          - role          = "OWNER" -> null
          - user_by_email = "[email protected]" -> null
        }
      + access {
          + role          = "roles/bigquery.dataOwner"
          + user_by_email = "[email protected]"
        }
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Steps to Reproduce

  1. terraform apply the above configuration
  2. terraform plan the above configuration
  3. Notice that the diff is non-empty.

It seems the issue is that Terraform's set hashing function for access doesn't account for the fact the BigQuery API only returns legacy IAM roles (e.g. OWNER) even if users set access[].role to modern IAM roles (e.g. roles/bigquery.dataOwner). See the API documentation for access[].role for more info.

So it seems like one idea for a fix is to implement a set hashing function that accounts for the conversion done by the BigQuery API.

References

  • b/179080823

b/359595039

@ghost ghost added the bug label Feb 2, 2021
@venkykuberan venkykuberan self-assigned this Feb 3, 2021
@venkykuberan
Copy link
Contributor

This is a known behavior, Predefined roles that have equivalent basic roles are swapped by the API to their basic counterparts. refer the doc here

You can change your config to use Basic role OWNER or you include life_cycle block like below to avoid the permadiff

resource "google_bigquery_dataset" "dataset" {
  dataset_id                  = "sink_test"
  project                     = var.project_id
  location                    = "US"
  default_table_expiration_ms = 3600000
  access {
    role = "roles/bigquery.dataOwner"
    user_by_email = google_service_account.bqowner.email
  }
   lifecycle {
      ignore_changes = [access]
  }
}

@jcanseco
Copy link
Contributor Author

jcanseco commented Feb 5, 2021

Hey @venkykuberan, I understand that the API converts predefined roles to their basic counterparts. I'm asking if it's possible to avoid a diff in the case where the predefined role on the config already matches the basic role in the live state, since there isn't actually a semantic difference in this case.

@ghost ghost removed waiting-response labels Feb 5, 2021
@venkykuberan
Copy link
Contributor

@rileykarson what are you thoughts ?

@rileykarson
Copy link
Collaborator

It's currently expected behaviour that only legacy roles can be specified: GoogleCloudPlatform/magic-modules#4191

I could see a case for allowing it, but at the moment that would be an enhancement and not a bug.

@rileykarson rileykarson removed their assignment Feb 5, 2021
@jcanseco
Copy link
Contributor Author

jcanseco commented Feb 5, 2021

Gotcha I see, is there anything I need to do here to change this to an enhancement and not a bug then?

Also, I would not consider this high priority, though it has caused confusion for the user since we didn't have a warning like GoogleCloudPlatform/magic-modules#4191 (and even if we did, I'm sure users will still run into the issue anyway since the underlying API does accept the new roles). We can add a similar warning on our side in the meantime though.

@kemalizing
Copy link

Hi,
I know this is kind of an old issue but I have a similar problem. the only difference is I'm already using the legacy roles and it is still showing diff after applying.

Here is the resource:

terraform {
  required_version = "1.3.6"
  required_providers {
    google      = "4.48.0"
  }
}

resource "google_bigquery_dataset" "self_service" {
  project                     = var.self_service_project
  dataset_id                  = "${var.tenant}_${var.stage}_self_service"
  friendly_name               = "Views for ${var.tenant} agent tool"
  description                 = "Views for ${var.tenant} agent tool"
  location                    = var.dataset_location
  default_table_expiration_ms = null
  delete_contents_on_destroy  = false
  labels                      = null

  access {
    role           = "READER"
    group_by_email = "gcp-${var.self_service_project}-${var.tenant}${var.stage}[email protected]"
  }
  access {
    role          = "OWNER"
    special_group = "projectOwners"
  }
}

and here is the plan:

  ~ resource "google_bigquery_dataset" "self_service" {
        id                              = "projects/<project-id>/datasets/test_pre_self_service"
        # (13 unchanged attributes hidden)

      - access {
          - role          = "OWNER" -> null
          - special_group = "projectOwners" -> null
        }
      + access {
          + group_by_email = "[email protected]"
          + role           = "READER"
        }
      - access {
          - group_by_email = "[email protected]" -> null
          - role           = "READER" -> null
        }
      + access {
          + role          = "OWNER"
          + special_group = "projectOwners"
        }
    }

Any help will be highly appreciated.
Thanks

modular-magician added a commit to modular-magician/terraform-provider-google that referenced this issue Jul 19, 2023
@github-actions github-actions bot added service/bigquery forward/review In review; remove label to forward labels Aug 17, 2023
@ggtisc
Copy link
Collaborator

ggtisc commented Aug 13, 2024

This issue still persists, even without making any changes to the resources there is a permadiff when it is attempting to run a terraform plan or terraform apply

@ggtisc ggtisc assigned ggtisc and unassigned ggtisc Aug 13, 2024
@ggtisc ggtisc removed the forward/review In review; remove label to forward label Aug 13, 2024
sachinpro added a commit to sachinpro/magic-modules that referenced this issue Oct 15, 2024
sachinpro added a commit to sachinpro/magic-modules that referenced this issue Oct 22, 2024
sachinpro added a commit to sachinpro/magic-modules that referenced this issue Oct 22, 2024
sachinpro added a commit to sachinpro/magic-modules that referenced this issue Oct 22, 2024
sachinpro added a commit to sachinpro/magic-modules that referenced this issue Oct 22, 2024
sachinpro added a commit to sachinpro/magic-modules that referenced this issue Oct 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

6 participants