kubernetes-public: Enable BigQuery Data Transfer

[ameukam]

Part of : #1308
Ref: #787
Followup of : #2747

Enable GCP required service for data transfer

Signed-off-by: Arnaud Meukam [email protected]

[ameukam]

This is already deployed.

   enabled:
     - bigquery.googleapis.com
  +  - bigquerydatatransfer.googleapis.com
     - bigquerystorage.googleapis.com
     - billingbudgets.googleapis.com
     - cloudasset.googleapis.com
  @@ -58,8 +59,7 @@ expected:
     - secretmanager.googleapis.com
     - storage-api.googleapis.com
     - storage-component.googleapis.com
  -to_enable:
  -  - bigquerydatatransfer.googleapis.com
  +to_enable: []
   to_disable:
     - bigquerystorage.googleapis.com
     - cloudresourcemanager.googleapis.com

[ameukam]

cc @dims @spiffxp

[ameukam]

I enabled the job and did a manually sync.

 ~ resource "google_bigquery_data_transfer_config" "bq_data_transfer_kettle" {
      ~ disabled                 = true -> false
        id                       = "projects/127754664067/locations/us/transferConfigs/617ffed3-0000-274f-bb87-d4f547e654cc"
        name                     = "projects/127754664067/locations/us/transferConfigs/617ffed3-0000-274f-bb87-d4f547e654cc"
        # (8 unchanged attributes hidden)

      + email_preferences {
          + enable_failure_email = false
        }
    }

[ameukam]

Transfer was successful for tables

gcloud alpha bq tables list --dataset=k8s_infra_kettle --project kubernetes-public
DATASET_ID        TABLE_ID        CREATION_TIME        EXPIRATION_TIME  TYPE
k8s_infra_kettle  all             2021-09-17T08:06:18                   TABLE
k8s_infra_kettle  all_build_logs  2021-09-17T08:05:58                   TABLE
k8s_infra_kettle  day             2021-09-17T08:05:58                   TABLE
k8s_infra_kettle  staging         2021-09-17T08:06:18                   TABLE
k8s_infra_kettle  week            2021-09-17T08:05:58                   TABLE

but not for the BigQuery views:

Cannot copy table presubmit_times. Copying table is not supported for views, materialized views and external tables.
Cannot copy table week_flakes. Copying table is not supported for views, materialized views and external tables.

Ref: https://cloud.google.com/bigquery/docs/copying-datasets#table_limitations

[spiffxp]

I wouldn't worry about the views not being transferred, AFAIK those are either things humans added, or can be rebuilt by things like kettle, gubernator or triage

[spiffxp]

/approve
/lgtm

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ameukam, spiffxp

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• infra/gcp/OWNERS [ameukam,spiffxp]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment