Skip to content

Initial implementation of metrics -> BigQuery loader#2042

Closed
justinsb wants to merge 1 commit intokubernetes:mainfrom
justinsb:metrics_spike
Closed

Initial implementation of metrics -> BigQuery loader#2042
justinsb wants to merge 1 commit intokubernetes:mainfrom
justinsb:metrics_spike

Conversation

@justinsb
Copy link
Copy Markdown
Member

@justinsb justinsb commented May 13, 2021

No description provided.

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label May 13, 2021
@k8s-ci-robot
Copy link
Copy Markdown
Contributor

k8s-ci-robot commented May 13, 2021

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: justinsb
To complete the pull request process, please assign thockin after the PR has been reviewed.
You can assign the PR to them by writing /assign @thockin in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot requested review from dims and thockin May 13, 2021 16:40
@k8s-ci-robot k8s-ci-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label May 13, 2021
@k8s-ci-robot
Copy link
Copy Markdown
Contributor

k8s-ci-robot commented May 13, 2021 •
edited
Loading

@justinsb: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Rerun command
pull-k8sio-verify 9f64ea7 link /test pull-k8sio-verify

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

ameukam
ameukam reviewed May 13, 2021
View reviewed changes
metrics/access-logs/create-schema.sh

# Must generally create in same location as GCS buckets, doesn't apply to US but still a good idea!
# https://cloud.google.com/bigquery/docs/batch-loading-data#data-locations
bq mk --location=US --dataset ${PROJECT}:kubernetes_public_logs
Copy link
Copy Markdown
Member

@ameukam ameukam May 13, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@justinsb can we use this project and this dataset : #2031 ?

@spiffxp
Copy link
Copy Markdown
Contributor

spiffxp commented Sep 3, 2021

Is this used or intended for merge? I'm going to close next week if not.

ref: #1834

@spiffxp
Copy link
Copy Markdown
Contributor

spiffxp commented Sep 3, 2021

/cc @BobyMCbobs @Riaankl @hh

@BobyMCbobs
Copy link
Copy Markdown
Member

BobyMCbobs commented Sep 7, 2021

This was a good bunch of work for loading the data in in.
In the end, we took the route of building out a job that uses Postgres that pulls in ASN data from this repo and several other sources.

https://github.com/ii/org/blob/afc91d7/research/asn-data-pipeline/asn_pipeline_docker_file.org

@spiffxp
Copy link
Copy Markdown
Contributor

spiffxp commented Sep 8, 2021

/close

@k8s-ci-robot
Copy link
Copy Markdown
Contributor

k8s-ci-robot commented Sep 8, 2021

@spiffxp: Closed this PR.

Details

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ameukam ameukam ameukam left review comments

@dims dims Awaiting requested review from dims

@thockin thockin Awaiting requested review from thockin

@BobyMCbobs BobyMCbobs Awaiting requested review from BobyMCbobs

@hh hh Awaiting requested review from hh

@riaankleinhans riaankleinhans Awaiting requested review from riaankleinhans

Assignees

No one assigned

Labels

cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@justinsb @k8s-ci-robot @spiffxp @BobyMCbobs @ameukam