google_bigquery_dataset - infinite diff if access is specified with non-legacy IAM roles

[jcanseco]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

Terraform v0.14.0
+ provider registry.terraform.io/hashicorp/google v5.0.0

Affected Resource(s)

• google_bigquery_dataset

Terraform Configuration Files

provider "google" {
  project = "cnrm-jcanseco-2"
}

resource "google_bigquery_dataset" "dataset" {
  dataset_id = "bigquerydatasetsampletf"
  access {
    role = "roles/bigquery.dataOwner"
    user_by_email = google_service_account.bqowner.email
  }
}

resource "google_service_account" "bqowner" {
  account_id = "bqowner"
}

Debug Output

https://gist.github.com/jcanseco/f51d951db4a30378a0183cef0d21de08

Expected Behavior

Running terraform plan after terraform apply should return an empty diff.

Actual Behavior

Running terraform plan after terraform apply returned a non-empty diff:

$ terraform plan 
google_service_account.bqowner: Refreshing state... [id=projects/cnrm-jcanseco-2/serviceAccounts/[email protected]]
google_bigquery_dataset.dataset: Refreshing state... [id=projects/cnrm-jcanseco-2/datasets/bigquerydatasetsampletf]

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # google_bigquery_dataset.dataset will be updated in-place
  ~ resource "google_bigquery_dataset" "dataset" {
        id                              = "projects/cnrm-jcanseco-2/datasets/bigquerydatasetsampletf"
        # (11 unchanged attributes hidden)

      - access {
          - role          = "OWNER" -> null
          - user_by_email = "[email protected]" -> null
        }
      + access {
          + role          = "roles/bigquery.dataOwner"
          + user_by_email = "[email protected]"
        }
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Steps to Reproduce

1. terraform apply the above configuration
2. terraform plan the above configuration
3. Notice that the diff is non-empty.

It seems the issue is that Terraform's set hashing function for access doesn't account for the fact the BigQuery API only returns legacy IAM roles (e.g. OWNER) even if users set access[].role to modern IAM roles (e.g. roles/bigquery.dataOwner). See the API documentation for access[].role for more info.

So it seems like one idea for a fix is to implement a set hashing function that accounts for the conversion done by the BigQuery API.

References

• b/179080823

b/359595039

[venkykuberan]

This is a known behavior, Predefined roles that have equivalent basic roles are swapped by the API to their basic counterparts. refer the doc here

You can change your config to use Basic role OWNER or you include life_cycle block like below to avoid the permadiff

resource "google_bigquery_dataset" "dataset" {
  dataset_id                  = "sink_test"
  project                     = var.project_id
  location                    = "US"
  default_table_expiration_ms = 3600000
  access {
    role = "roles/bigquery.dataOwner"
    user_by_email = google_service_account.bqowner.email
  }
   lifecycle {
      ignore_changes = [access]
  }
}

[jcanseco]

Hey @venkykuberan, I understand that the API converts predefined roles to their basic counterparts. I'm asking if it's possible to avoid a diff in the case where the predefined role on the config already matches the basic role in the live state, since there isn't actually a semantic difference in this case.

[venkykuberan]

@rileykarson what are you thoughts ?

[rileykarson]

It's currently expected behaviour that only legacy roles can be specified: GoogleCloudPlatform/magic-modules#4191

I could see a case for allowing it, but at the moment that would be an enhancement and not a bug.

[jcanseco]

Gotcha I see, is there anything I need to do here to change this to an enhancement and not a bug then?

Also, I would not consider this high priority, though it has caused confusion for the user since we didn't have a warning like GoogleCloudPlatform/magic-modules#4191 (and even if we did, I'm sure users will still run into the issue anyway since the underlying API does accept the new roles). We can add a similar warning on our side in the meantime though.

[kemalizing]

Hi,
I know this is kind of an old issue but I have a similar problem. the only difference is I'm already using the legacy roles and it is still showing diff after applying.

Here is the resource:

terraform {
  required_version = "1.3.6"
  required_providers {
    google      = "4.48.0"
  }
}

resource "google_bigquery_dataset" "self_service" {
  project                     = var.self_service_project
  dataset_id                  = "${var.tenant}_${var.stage}_self_service"
  friendly_name               = "Views for ${var.tenant} agent tool"
  description                 = "Views for ${var.tenant} agent tool"
  location                    = var.dataset_location
  default_table_expiration_ms = null
  delete_contents_on_destroy  = false
  labels                      = null

  access {
    role           = "READER"
    group_by_email = "gcp-${var.self_service_project}-${var.tenant}${var.stage}[email protected]"
  }
  access {
    role          = "OWNER"
    special_group = "projectOwners"
  }
}

and here is the plan:

  ~ resource "google_bigquery_dataset" "self_service" {
        id                              = "projects/<project-id>/datasets/test_pre_self_service"
        # (13 unchanged attributes hidden)

      - access {
          - role          = "OWNER" -> null
          - special_group = "projectOwners" -> null
        }
      + access {
          + group_by_email = "[email protected]"
          + role           = "READER"
        }
      - access {
          - group_by_email = "[email protected]" -> null
          - role           = "READER" -> null
        }
      + access {
          + role          = "OWNER"
          + special_group = "projectOwners"
        }
    }

Any help will be highly appreciated.
Thanks

[ggtisc]

This issue still persists, even without making any changes to the resources there is a permadiff when it is attempting to run a terraform plan or terraform apply