Format of the datasetId in single_target_dataset as described in the Datastream documentation is incorrect.

[Ko1103]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

Terraform v1.3.9
on linux_arm64

Affected Resource(s)

• google_datastream_stream

Terraform Configuration Files

resource "google_bigquery_dataset" "postgres" {
  dataset_id    = "postgres"
  friendly_name = "postgres"
  description   = "Database of postgres"
  location      = "asia-northeast1"
}

resource "google_datastream_stream" "cdc" {
  display_name  = "postgres to bigQuery"
  location      = "asia-northeast1"
  stream_id     = "postgres-to-big-query"
  desired_state = "RUNNING"

  source_config {}

  destination_config {
    destination_connection_profile = google_datastream_connection_profile.destination.id
    bigquery_destination_config {
      data_freshness = "900s"
      single_target_dataset {
        dataset_id = google_bigquery_dataset.postgres.id
      }
    }
  }

  backfill_all {
  }

}

Debug Output

Expected Behavior

No error will be shown.

Actual Behavior

╷
│ Error: Error creating Stream: googleapi: Error 400: Request contains an invalid argument.
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.BadRequest",
│     "fieldViolations": [
│       {
│         "description": "Invalid format, must be in the format: projectId:datasetId",
│         "field": "destination_config.bigquery_destination_config.single_target_dataset.dataset_id"
│       }
│     ]
│   }
│ ]

References

• https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/datastream_stream#nested_single_target_dataset

[edwardmedia]

Error complains on the field datasetId. Instead of "projects/myproject/datasets/issue13974", the resource expects "myproject:issue13974". It is different from the doc. Has API updated its format?

POST /v1/projects/myproject/locations/asia-northeast1/streams?alt=json&streamId=postgres-to-big-query HTTP/1.1
Host: datastream.googleapis.com
{
 "backfillAll": {},
 "backfillNone": null,
 "destinationConfig": {
  "bigqueryDestinationConfig": {
   "dataFreshness": "900s",
   "singleTargetDataset": {
    "datasetId": "projects/myproject/datasets/issue13974"
   }
  },
  "destinationConnectionProfile": "projects/myproject/locations/asia-northeast1/connectionProfiles/issue13974"
 },
 "displayName": "issue13974",
 "sourceConfig": {
  "mysqlSourceConfig": {
   "maxConcurrentCdcTasks": 0
  },
  "oracleSourceConfig": null,
  "postgresqlSourceConfig": null,
  "sourceConnectionProfile": "projects/myproject/locations/asia-northeast1/connectionProfiles/source_connection_profile"
 },
 "state": "RUNNING"
}

[Ko1103]

Here is the documentation for the corresponding API for your reference.

• https://cloud.google.com/datastream/docs/reference/rest/v1/projects.locations.streams#SingleTargetDataset

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.