Provider produced inconsistent result after apply with resource google_bigquery_dataset_access

[Clausewitz45]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

# terraform -v
Terraform v0.14.4
+ provider registry.terraform.io/hashicorp/external v2.0.0
+ provider registry.terraform.io/hashicorp/google v3.52.0
+ provider registry.terraform.io/hashicorp/google-beta v3.52.0
+ provider registry.terraform.io/hashicorp/local v2.0.0
+ provider registry.terraform.io/hashicorp/null v3.0.0
+ provider registry.terraform.io/hashicorp/random v3.0.0
+ provider registry.terraform.io/hashicorp/vault v2.15.0

The issue is reproducible with terraform version 0.13.5, and provider versions 3.51.1 and 3.51.0 also.

Affected Resource(s)

• google_bigquery_dataset_access

Terraform Configuration Files

resource "google_bigquery_dataset_access" "project1_access_logs_controlling" {
  project = var.projectname

  dataset_id     = google_bigquery_dataset.bigquery_logs.dataset_id
  role           = "READER"
  group_by_email = "[email protected]"

}

resource "google_bigquery_dataset_access" "project2_access_logs_controlling" {
  project = var.projectname

  dataset_id     = google_bigquery_dataset.bigquery_logs.dataset_id
  role           = "READER"
  group_by_email = "[email protected]"

}

Debug Output

GitHub Gist with debug.log output

Expected Behavior

Providing BigQuery Data Reader permission to the specified email group.

Actual Behavior

# TF_LOG=DEBUG TF_LOG_PATH=~/terraform_debug.log terraform apply plan.out
2021/01/12 12:25:47 [WARN] Log levels other than TRACE are currently unreliable, and are supported only for backward compatibility.
  Use TF_LOG=TRACE to see Terraform's internal logs.
  ----
google_bigquery_dataset_access.project1_access_logs_controlling: Creating...
google_bigquery_dataset_access.project2_access_logs_controlling: Creating...

Error: Provider produced inconsistent result after apply

When applying changes to
google_bigquery_dataset_access.project1_access_logs_controlling,
provider "registry.terraform.io/hashicorp/google" produced an unexpected new
value: Root resource was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.


Error: Provider produced inconsistent result after apply

When applying changes to
google_bigquery_dataset_access.project2_access_logs_controlling,
provider "registry.terraform.io/hashicorp/google" produced an unexpected new
value: Root resource was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Steps to Reproduce

1. terraform apply

[venkykuberan]

@Clausewitz45 Can you clarify why you want to create the same resource twice in the config ?

[Clausewitz45]

Hi @venkykuberan , sorry it seems only because of the redacted information. In the end, I'm managing multiple projects from a single Terraform repository, and these two resources belong to two different projects. Both projects are containing two different BigQuery datasets with the same name (logs dataset for internal statistics). The resources were provisioned ~2-3 months earlier with different terraform versions, and during the provisioning, I didn't have this issue. Now I'm trying to extend the permissions with a new group as a new resource, and I'm having this error.

[rileykarson]

@Clausewitz45 can you check line 312 in the original log and confirm whether the groups are identical or not? I see Skipping item with groupByEmail= "[email protected]", looking for "[email protected]") which both look like redactions, and whether they match or not is a likely cause here.

Additionally, do you use a provider-default project? And if so, does it match the one of that resource? I suspect that Terraform may be messing up the project(s) in use.

[rileykarson]

I get a success with the following, which leads me to believe the issue is not a project thing and it's more likely something weird is happening with the returned emails. Can you inspect the matching section in the unobfuscated logs and ensure that there are no differences between the values? We expect them to match exactly-- even a minor capitalization difference could be the cause.

GOOGLE_PROJECT=project3 terraform apply

resource "google_bigquery_dataset" "default" {
  project = "project1"
  dataset_id                  = "foo"
  friendly_name               = "test"
  description                 = "This is a test description"
  location                    = "EU"
  default_table_expiration_ms = 3600000

  labels = {
    env = "default"
  }
}

resource "google_bigquery_dataset" "default2" {
  project = "project2"
  dataset_id                  = "foo"
  friendly_name               = "test"
  description                 = "This is a test description"
  location                    = "EU"
  default_table_expiration_ms = 3600000

  labels = {
    env = "default"
  }
}

resource "google_bigquery_dataset_access" "project1_access_logs_controlling" {
  project = "project1"

  dataset_id     = google_bigquery_dataset.default.dataset_id
  role           = "READER"
  group_by_email = "[email protected]"
}

resource "google_bigquery_dataset_access" "project2_access_logs_controlling" {
  project = "project2"

  dataset_id     = google_bigquery_dataset.default2.dataset_id
  role           = "READER"
  group_by_email = "[email protected]"

}

[Clausewitz45]

Hi @rileykarson , thanks for coming back to me. Unfortunately I don't have the old logs - lost in time somewhere, but I was able to reproduce the same issue with terraform version 0.14.5 and provider version 3.54.0 - so everything is the latest.

Background: terraform has it's own project with a single service account, and has the permission to manage ~24 other projects. So the resources are not in the provider default project, and referenced with project = var.projectname1 and project = var.projectname2 at the resources.

The user backend is Google Workspace / Suite with Cloud Identity. I checked first our Cloud Identity if the email address is matching what is in the logs - it was matching, so no issue here. I also checked the capitalization of the characters, it is fine now, but it turned out, previously the group email address was created like this:
[email protected], and now it looks like [email protected] (please note the "g/G" is different).

Then I was running a full plan, and apply with debug outputs - this is a huge TF repository (please don't judge - there are multiple reasons, and one of them is a Security Officer) with more than 4900 resources there.

I checked the new logs and you were right. If I take the old logs as an example, and checking the line 312/313 the email addresses are not matching there. Line 312 contains a capitalization issue, in 313 is weird because the looking for is the same for both cases and the Skipping item with groupByEmail is different like this (please also note the g/G difference there):

2021-01-12T12:26:06.893+0100 [INFO]  plugin.terraform-provider-google_v3.52.0_x5: 2021/01/12 12:26:06 [DEBUG] Skipping item with groupByEmail= "[email protected]", looking for "[email protected]"): timestamp=2021-01-12T12:26:06.893+0100
2021-01-12T12:26:06.893+0100 [INFO]  plugin.terraform-provider-google_v3.52.0_x5: 2021/01/12 12:26:06 [DEBUG] Skipping item with groupByEmail= "[email protected]", looking for "[email protected]"): timestamp=2021-01-12T12:26:06.893+0100

I did a grep for both in the code - but the capitalized version is coming from nowhere - no sign of it in the code. As I mentioned at the beginning, previously it was capitalized, but by checking the audit logs, it was changed roughly a month ago.

What can I do with this thingy?

P.s.: if you can tell me how to share with you securely the non-obfuscated logs, I would be happy to do that.

[rileykarson]

Great! This seems like a pretty likely cause.

The value in your config right now is [email protected] right? Can you check which value appears for the same field if you open the state file or run terraform state show? The value that appears in a PATCH request and in a response from the API would also be helpful. I expect both to be [email protected] or for the request to be [email protected] and the response [email protected].

[Clausewitz45]

Hi @rileykarson . The value in my config is with the small capital: [email protected] , not with the big one - basically it is a variable. I checked the state (downloaded the JSON, and did a grep on it) - but the email address cannot be found there: none of the exists with big/small capital in the state.

Here comes the fun part. When I was adding the email address manually on the Cloud Console, I entered it with the small capital: like [email protected], but when I did a save and refresh, it is changed back to [email protected] as it was created originally! So I updated the variable in Terraform to have the email address with big capital, run the plan/apply - the access was create without any problem.

This is the reason why I think there must be some "discrepancy" between Cloud Identity (where the email was originally created with big capital) and GCP - during the information exchange - and this is why I think that this is not a Terraform bug, but inconsistency between the two other actor.

Many thanks for the help and if you agree we can close the ticket.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!