Migrate k8s-triage, k8s-metrics to terraform, setup k8s-triage dataset #2461

spiffxp · 2021-08-04T18:21:27Z

Part of: Migrate triage dashboard to wg-k8s-infra #1305
Followup to: infra/gcp: setup k8s-project-triage special-case #2454
Depends on: config/jobs: setup canary triage job, mv other triage jobs test-infra#23126

Since converting go.k8s.io/triage to triage.k8s.io is going to be more involved than I thought, setup to migrate gs://k8s-gubernator/triage to gs://k8s-triage. The bucket is currently populated with whatever it contained a while ago (which is basically as fresh as it's going to get since kettle has been down this whole time)

The dependency test-infra PR sets up a job that runs in k8s-infra-prow-build-trusted to update gs://k8s-triage

While I initially did all the provisioning for this via ensure-main-project.sh, I have since updated the PR to do all of this via terraform. I migrated all of the setup code for gs://k8s-metrics while doing so.

This also adds a bigquery dataset kubernetes-public:k8s-triage for the ci-test-infra-triage-canary job to be able to write to. A future test-infra PR will configure the job to write to it instead of currently failing to write to k8s-gubernator:temp.triage

In setting up the dataset, I'm trying out use of IAM roles instead of basic access (ref: https://cloud.google.com/bigquery/docs/access-control-basic-roles)

spiffxp · 2021-08-04T19:19:54Z

Ran ./infra/gcp/ensure-main-project.sh and manually removed gs://k8s-project-triage

this is a trial in using IAM to control access to dataset resources instead of the legacy access roles based off of https://cloud.google.com/bigquery/docs/access-control-basic-roles

k8s-ci-robot · 2021-08-12T20:29:38Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: spiffxp

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~infra/gcp/OWNERS~~ [spiffxp]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

spiffxp · 2021-08-12T20:39:18Z

/assign @ameukam
/hold
I'm going to deploy this now but anticipate I may need to edit the terraform state slightly, so don't want to merge until I've confirmed everything is as expected.

spiffxp · 2021-08-12T20:43:27Z

infra/gcp/terraform/kubernetes-public/k8s-metrics.tf

+This file defines:
+- bigquery dataset for triage to store temp results
+- GCS bucket to serve go.k8s.io/triage results
+- IAM bindings


Copy-pasta comments need updating

I also think we need to put boilerplate license in our tf files...

spiffxp · 2021-08-12T21:27:41Z

Out of excessive paranoia I'm backing up the buckets first:

gsutil mb -p kubernetes-public gs://k8s-triage-bak
gsutil mb -p kubernetes-public gs://k8s-metrics-bak
gsutil -m rsync -r gs://k8s-triage gs://k8s-triage-bak # Operation completed over 1.5k objects/7.6 GiB.
gsutil -m rsync -r gs://k8s-metrics gs://k8s-metrics-bak # Operation completed over 7.7k objects/5.1 GiB.

ameukam · 2021-08-12T21:30:37Z

/lgtm

The buckets were created as STANDARD storage class in US, so let's not delete them unless we really need to. The bigquery dataset can't have `-` in its title The bigquery iam policy binding needs to include project in its fields, apparently it won't default to the project the dataset is in

spiffxp · 2021-08-12T22:17:49Z

In terms of manual stuff I had to do to get this to work, this was basically it...

terraform import google_storage_bucket.triage_bucket kubernetes-public/k8s-triage
terraform import google_storage_bucket.metrics_bucket kubernetes-public/k8s-metrics

The rest was terraform apply, read error, do what it said, repeat...

It's maybe worth noting that after my first successful terraform apply, I ran again and saw the following

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the last "terraform apply":

  # google_bigquery_dataset.triage_dataset has been changed
  ~ resource "google_bigquery_dataset" "triage_dataset" {
      ~ etag                            = "ObAE6ASzv6jJuAl9fPwfqg==" -> "cyljn5Sy1j+fDr/MKBXOag=="
        id                              = "projects/kubernetes-public/datasets/k8s_triage"
      ~ last_modified_time              = 1628805757044 -> 1628806369623
        # (10 unchanged attributes hidden)

      - access {
          - role          = "OWNER" -> null
          - user_by_email = "[email protected]" -> null
        }
      - access {
          - role          = "OWNER" -> null
          - special_group = "projectOwners" -> null
        }
      - access {
          - role          = "READER" -> null
          - special_group = "projectReaders" -> null
        }
      + access {
          + role          = "WRITER"
          + user_by_email = "[email protected]"
        }
      - access {
          - role          = "WRITER" -> null
          - special_group = "projectWriters" -> null
        }
      + access {
          + group_by_email = "[email protected]"
          + role           = "OWNER"
        }
    }

Unless you have made equivalent changes to your configuration, or ignored the relevant attributes using ignore_changes, the following plan may include actions to undo or respond to these changes.

─────────────────────────────────────────────────────────────

No changes. Your infrastructure matches the configuration.

spiffxp · 2021-08-12T22:18:38Z

/hold cancel
Changes in here have been deployed

spiffxp · 2021-08-12T22:23:23Z

Some manual cleanup...

gsutil rm -r gs://k8s-project-triage # what existed prior to this PR

ameukam · 2021-08-12T22:43:14Z

/lgtm

sftim

I spotted 2 typos. Nothing serious!

sftim · 2021-08-13T09:56:39Z

infra/gcp/terraform/kubernetes-public/k8s-triage.tf

+resource "google_bigquery_dataset" "triage_dataset" {
+  dataset_id  = "k8s_triage"
+  project     = data.google_project.project.project_id
+  description = "Dataset for kubernetes/test-infra/triage to store temprorary results"


post-merge nit:

Suggested change

description = "Dataset for kubernetes/test-infra/triage to store temprorary results"

description = "Dataset for kubernetes/test-infra/triage to store temporary results"

sftim · 2021-08-13T09:57:45Z

infra/gcp/terraform/kubernetes-public/k8s-triage.tf

+    ]
+    role = "roles/storage.admin"
+  }
+  // Preserve legacy storage bindings, give storage.admim members legacy bucket owner


post-merge nit:

Suggested change

// Preserve legacy storage bindings, give storage.admim members legacy bucket owner

// Preserve legacy storage bindings, give storage.admin members legacy bucket owner

spiffxp · 2021-08-16T16:39:25Z

This broke https://testgrid.k8s.io/wg-k8s-infra-prow#metrics-bigquery

#2548 should address

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 4, 2021

k8s-ci-robot requested review from aojea and thockin August 4, 2021 18:21

This was referenced Aug 4, 2021

Migrate triage dashboard to wg-k8s-infra #1305

Closed

infra/gcp: setup k8s-project-triage special-case #2454

Merged

spiffxp force-pushed the k8s-triage branch from 56c5c5c to 8958486 Compare August 5, 2021 21:24

spiffxp mentioned this pull request Aug 5, 2021

audit: update as of 2021-08-05 #2456

Merged

spiffxp force-pushed the k8s-triage branch from 8958486 to 5245285 Compare August 6, 2021 19:37

spiffxp mentioned this pull request Aug 9, 2021

audit: update as of 2021-08-09 #2477

Merged

spiffxp added 5 commits August 12, 2021 09:01

infra/gcp/main: mv k8s-project-triage k8s-triage

7bc5324

infra/gcp: mv k8s-triage special case to terraform

51dbc9a

infra/gcp: mv k8s-metrics special case to terraform

942fcdb

terraform/kubernetes-public: dedup prow-oncall

8447c62

terraform/kubernetes-public: add k8s-triage bq dataset

41288c3

this is a trial in using IAM to control access to dataset resources instead of the legacy access roles based off of https://cloud.google.com/bigquery/docs/access-control-basic-roles

spiffxp force-pushed the k8s-triage branch from 5245285 to 41288c3 Compare August 12, 2021 20:29

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. area/terraform Terraform modules, testing them, writing more of them, code in infra/gcp/clusters/ and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Aug 12, 2021

spiffxp changed the title ~~[wip] migrate go.k8s.io/triage to gs://k8s-triage~~ Migrate k8s-triage, k8s-metrics to terraform, setup k8s-triage dataset Aug 12, 2021

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 12, 2021

k8s-ci-robot assigned ameukam Aug 12, 2021

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 12, 2021

spiffxp commented Aug 12, 2021

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 12, 2021

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 12, 2021

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 12, 2021

spiffxp mentioned this pull request Aug 12, 2021

config/jobs: update ci-test-infra-triage-canary kubernetes/test-infra#23235

Merged

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 12, 2021

k8s-ci-robot merged commit 4ea13f5 into kubernetes:main Aug 12, 2021

k8s-ci-robot added this to the v1.23 milestone Aug 12, 2021

sftim reviewed Aug 13, 2021

View reviewed changes

spiffxp deleted the k8s-triage branch August 16, 2021 14:27

This was referenced Aug 16, 2021

apps/k8s.io: flip go.k8s.io/triage to community-owned bucket #2544

Merged

audit: update as of 2021-08-16 #2533

Merged

terraform/kubernetes-public: fix copy-pasta #2548

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate k8s-triage, k8s-metrics to terraform, setup k8s-triage dataset #2461

Migrate k8s-triage, k8s-metrics to terraform, setup k8s-triage dataset #2461

spiffxp commented Aug 4, 2021 •

edited

Loading

spiffxp commented Aug 4, 2021

k8s-ci-robot commented Aug 12, 2021

spiffxp commented Aug 12, 2021

spiffxp Aug 12, 2021

spiffxp commented Aug 12, 2021

ameukam commented Aug 12, 2021

spiffxp commented Aug 12, 2021

spiffxp commented Aug 12, 2021

spiffxp commented Aug 12, 2021

ameukam commented Aug 12, 2021

sftim left a comment

sftim Aug 13, 2021

sftim Aug 13, 2021

spiffxp commented Aug 16, 2021

	description = "Dataset for kubernetes/test-infra/triage to store temprorary results"
	description = "Dataset for kubernetes/test-infra/triage to store temporary results"

	// Preserve legacy storage bindings, give storage.admim members legacy bucket owner
	// Preserve legacy storage bindings, give storage.admin members legacy bucket owner

Migrate k8s-triage, k8s-metrics to terraform, setup k8s-triage dataset #2461

Migrate k8s-triage, k8s-metrics to terraform, setup k8s-triage dataset #2461

Conversation

spiffxp commented Aug 4, 2021 • edited Loading

spiffxp commented Aug 4, 2021

k8s-ci-robot commented Aug 12, 2021

spiffxp commented Aug 12, 2021

spiffxp Aug 12, 2021

Choose a reason for hiding this comment

spiffxp commented Aug 12, 2021

ameukam commented Aug 12, 2021

spiffxp commented Aug 12, 2021

spiffxp commented Aug 12, 2021

spiffxp commented Aug 12, 2021

ameukam commented Aug 12, 2021

sftim left a comment

Choose a reason for hiding this comment

sftim Aug 13, 2021

Choose a reason for hiding this comment

sftim Aug 13, 2021

Choose a reason for hiding this comment

spiffxp commented Aug 16, 2021

spiffxp commented Aug 4, 2021 •

edited

Loading