Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Format of the datasetId in single_target_dataset as described in the Datastream documentation is incorrect. #13974

Assignees
Labels

Comments

@Ko1103
Copy link

Ko1103 commented Mar 12, 2023

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
  • Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment.
  • If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

Terraform v1.3.9
on linux_arm64

Affected Resource(s)

  • google_datastream_stream

Terraform Configuration Files

resource "google_bigquery_dataset" "postgres" {
  dataset_id    = "postgres"
  friendly_name = "postgres"
  description   = "Database of postgres"
  location      = "asia-northeast1"
}

resource "google_datastream_stream" "cdc" {
  display_name  = "postgres to bigQuery"
  location      = "asia-northeast1"
  stream_id     = "postgres-to-big-query"
  desired_state = "RUNNING"

  source_config {}

  destination_config {
    destination_connection_profile = google_datastream_connection_profile.destination.id
    bigquery_destination_config {
      data_freshness = "900s"
      single_target_dataset {
        dataset_id = google_bigquery_dataset.postgres.id
      }
    }
  }

  backfill_all {
  }

}

Debug Output

Expected Behavior

No error will be shown.

Actual Behavior

╷
│ Error: Error creating Stream: googleapi: Error 400: Request contains an invalid argument.
│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.BadRequest",
│     "fieldViolations": [
│       {
│         "description": "Invalid format, must be in the format: projectId:datasetId",
│         "field": "destination_config.bigquery_destination_config.single_target_dataset.dataset_id"
│       }
│     ]
│   }
│ ]

References

@Ko1103 Ko1103 added the bug label Mar 12, 2023
@edwardmedia edwardmedia self-assigned this Mar 13, 2023
@edwardmedia
Copy link
Contributor

Error complains on the field datasetId. Instead of "projects/myproject/datasets/issue13974", the resource expects "myproject:issue13974". It is different from the doc. Has API updated its format?

POST /v1/projects/myproject/locations/asia-northeast1/streams?alt=json&streamId=postgres-to-big-query HTTP/1.1
Host: datastream.googleapis.com
{
 "backfillAll": {},
 "backfillNone": null,
 "destinationConfig": {
  "bigqueryDestinationConfig": {
   "dataFreshness": "900s",
   "singleTargetDataset": {
    "datasetId": "projects/myproject/datasets/issue13974"
   }
  },
  "destinationConnectionProfile": "projects/myproject/locations/asia-northeast1/connectionProfiles/issue13974"
 },
 "displayName": "issue13974",
 "sourceConfig": {
  "mysqlSourceConfig": {
   "maxConcurrentCdcTasks": 0
  },
  "oracleSourceConfig": null,
  "postgresqlSourceConfig": null,
  "sourceConnectionProfile": "projects/myproject/locations/asia-northeast1/connectionProfiles/source_connection_profile"
 },
 "state": "RUNNING"
}

@Ko1103
Copy link
Author

Ko1103 commented Mar 13, 2023

Here is the documentation for the corresponding API for your reference.

@github-actions
Copy link

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Apr 15, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.