Update k8s.io/test-infra dependency and fix GCP Mason by ixdy · Pull Request #2447 · istio/test-infra

ixdy · 2020-02-29T02:23:40Z

Some recent changes to Boskos changed some of the CRDs slightly, which is now causing older Mason implementations to fail to register:

  error: "no kind "DRLCObjectList" is registered for version "boskos.k8s.io/v1" in scheme "pkg/runtime/scheme.go:101""

Thankfully, the changes are fairly straightforward, and we can easily use a caching kubernetes client now, too. (The benefits of caching are probably fairly low, since DRLCs don't change that often.)

Relevant upstream changes for context:
kubernetes/test-infra#16206
kubernetes/test-infra#16249
kubernetes/test-infra#16367

istio-policy-bot · 2020-02-29T02:23:44Z

😊 Welcome @ixdy! This is either your first contribution to the Istio test-infra repo, or it's been
awhile since you've been here.

You can learn more about the Istio working groups, code of conduct, and contributing guidelines
by referring to Contributing to Istio.

Thanks for contributing!

Courtesy of your friendly welcome wagon.

istio-testing · 2020-02-29T02:23:56Z

Hi @ixdy. Thanks for your PR.

I'm waiting for a istio member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

clarketm · 2020-02-29T03:15:51Z

/ok-to-test

go.mod

ixdy · 2020-03-05T03:41:53Z

I additionally needed to update some unrelated files, since some Prow internal data structures have changed.

clarketm · 2020-03-05T03:49:29Z

I additionally needed to update some unrelated files, since some Prow internal data structures have changed.

Thanks! Additionally, can you run make gen then commit the changes.

ixdy · 2020-03-05T04:05:18Z

it looks like it's unhappy about licenses, and I have no idea how to fix this:

ERROR: Some modules have unrecognized licenses:
  github.com/DataDog/zstd: similar to NCSA, 63.67% confidence, path '/home/prow/go/pkg/mod/github.com/!data!dog/zstd@v1.4.1/LICENSE'
  github.com/emirpasic/gods: path '/home/prow/go/pkg/mod/github.com/emirpasic/gods@v1.12.0/LICENSE'
  github.com/kevinburke/ssh_config: path '/home/prow/go/pkg/mod/github.com/kevinburke/ssh_config@v0.0.0-20190725054713-01f96b0aa0cd/LICENSE'
  github.com/tektoncd/pipeline: path '/home/prow/go/pkg/mod/github.com/tektoncd/pipeline@v0.10.1/third_party/github.com/ghodss/yaml/LICENSE'
  github.com/tektoncd/pipeline: similar to BSD-3-Clause-Clear, 89.66% confidence, path '/home/prow/go/pkg/mod/github.com/tektoncd/pipeline@v0.10.1/third_party/github.com/gogo/protobuf/LICENSE'
  github.com/tektoncd/pipeline: path '/home/prow/go/pkg/mod/github.com/tektoncd/pipeline@v0.10.1/third_party/github.com/jmespath/go-jmespath/LICENSE'
  github.com/tektoncd/pipeline: path '/home/prow/go/pkg/mod/github.com/tektoncd/pipeline@v0.10.1/third_party/sigs.k8s.io/yaml/LICENSE'
ERROR: Some modules have no discernible license:
  github.com/otiai10/curr
  github.com/pelletier/go-buffruneio
common/Makefile.common.mk:64: recipe for target 'lint-licenses' failed

clarketm · 2020-03-05T04:53:35Z

it looks like it's unhappy about licenses, and I have no idea how to fix this:

ERROR: Some modules have unrecognized licenses:
  github.com/DataDog/zstd: similar to NCSA, 63.67% confidence, path '/home/prow/go/pkg/mod/github.com/!data!dog/zstd@v1.4.1/LICENSE'
  github.com/emirpasic/gods: path '/home/prow/go/pkg/mod/github.com/emirpasic/gods@v1.12.0/LICENSE'
  github.com/kevinburke/ssh_config: path '/home/prow/go/pkg/mod/github.com/kevinburke/ssh_config@v0.0.0-20190725054713-01f96b0aa0cd/LICENSE'
  github.com/tektoncd/pipeline: path '/home/prow/go/pkg/mod/github.com/tektoncd/pipeline@v0.10.1/third_party/github.com/ghodss/yaml/LICENSE'
  github.com/tektoncd/pipeline: similar to BSD-3-Clause-Clear, 89.66% confidence, path '/home/prow/go/pkg/mod/github.com/tektoncd/pipeline@v0.10.1/third_party/github.com/gogo/protobuf/LICENSE'
  github.com/tektoncd/pipeline: path '/home/prow/go/pkg/mod/github.com/tektoncd/pipeline@v0.10.1/third_party/github.com/jmespath/go-jmespath/LICENSE'
  github.com/tektoncd/pipeline: path '/home/prow/go/pkg/mod/github.com/tektoncd/pipeline@v0.10.1/third_party/sigs.k8s.io/yaml/LICENSE'
ERROR: Some modules have no discernible license:
  github.com/otiai10/curr
  github.com/pelletier/go-buffruneio
common/Makefile.common.mk:64: recipe for target 'lint-licenses' failed

I am not entirely certain either. It is either a bug with the license resolution logic or an issue with the dependencies. Related discussion: istio/istio#20379. @howardjohn or @sdake any idea how to resolve?

ixdy · 2020-03-06T01:38:56Z

ok, I think I've fixed some of these issues by pinning to newer versions (which include the LICENSE files), though a bunch are still failing; I suspect they'll need to be whitelisted:

ERROR: Some modules have unrecognized licenses:
  github.com/DataDog/zstd: similar to NCSA, 63.67% confidence, path '/go/pkg/mod/github.com/!data!dog/zstd@v1.4.1/LICENSE'
  github.com/emirpasic/gods: path '/go/pkg/mod/github.com/emirpasic/gods@v1.12.0/LICENSE'
  github.com/kevinburke/ssh_config: path '/go/pkg/mod/github.com/kevinburke/ssh_config@v0.0.0-20190725054713-01f96b0aa0cd/LICENSE'
  github.com/otiai10/curr: path '/go/pkg/mod/github.com/otiai10/curr@v0.0.0-20190513014714-f5a3d24e5776/LICENSE'
  github.com/tektoncd/pipeline: path '/go/pkg/mod/github.com/tektoncd/pipeline@v0.10.1/third_party/github.com/ghodss/yaml/LICENSE'
  github.com/tektoncd/pipeline: similar to BSD-3-Clause-Clear, 89.66% confidence, path '/go/pkg/mod/github.com/tektoncd/pipeline@v0.10.1/third_party/github.com/gogo/protobuf/LICENSE'
  github.com/tektoncd/pipeline: path '/go/pkg/mod/github.com/tektoncd/pipeline@v0.10.1/third_party/github.com/jmespath/go-jmespath/LICENSE'
  github.com/tektoncd/pipeline: path '/go/pkg/mod/github.com/tektoncd/pipeline@v0.10.1/third_party/sigs.k8s.io/yaml/LICENSE'

Going one-by-one:

github.com/DataDog/zstd looks like a weird BSD?
github.com/emirpasic/gods has multiple licenses
github.com/kevinburke/ssh_config has multiple licenses
github.com/otiai10/curr uses a DWTFYWTP license (annoyingly)
github.com/tektoncd/pipeline/third_party/github.com/ghodss/yaml has multiple licenses
github.com/tektoncd/pipeline/third_party/github.com/gogo/protobuf has a slightly strange BSD
github.com/tektoncd/pipeline/third_party/github.com/jmespath/go-jmespath has an abbreviated Apache2
github.com/tektoncd/pipeline/third_party/sigs.k8s.io/yaml/LICENSE has multiple licenses

ixdy · 2020-03-06T20:46:06Z

All of the third-party dependencies under tektoncd/pipeline are already whitelisted, just not with their third-party paths. I'm not sure whether the tool should be taught to recognize this, or if we should whitelist them explicitly.

Any guidance on what I should do about the other dependencies?

clarketm · 2020-03-07T00:40:53Z

All of the third-party dependencies under tektoncd/pipeline are already whitelisted, just not with their third-party paths. I'm not sure whether the tool should be taught to recognize this, or if we should whitelist them explicitly.

Any guidance on what I should do about the other dependencies?

Whitelisting those specific modules should be fine.

licenses/github.com/otiai10/curr/LICENSE

howardjohn · 2020-03-07T00:58:04Z

I am no expert, but I don't think k8s can/should depend on it either... Unless it's different due to CNCF

…

On Fri, Mar 6, 2020 at 4:55 PM Travis Clarke ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In licenses/github.com/otiai10/curr/LICENSE <#2447 (comment)>: > @@ -0,0 +1,5 @@ +Copyright © 2015 Hiromu OCHIAI ***@***.***> + +This work is free. You can redistribute it and/or modify it under the +terms of the Do What The Fuck You Want To Public License, Version 2, Can we drop the dependency? This if from kubernetes/test-infra. I am not sure removing it is an (easy) option. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2447?email_source=notifications&email_token=AAEYGXPZMED4GN44EBED2W3RGGLOZA5CNFSM4K6H7NBKYY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOCYMICLI#discussion_r389208821>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEYGXLFONWB7VORI6AODK3RGGLOZANCNFSM4K6H7NBA> .

ixdy · 2020-03-07T00:59:33Z

I'm trying to understand where this dependency is coming from, but I'm failing.

Nothing in k8s.io/test-infra directly depends on it. If I try downloading the deps into vendor/, nothing there depends on it. go mod why isn't helping either.

clarketm · 2020-03-07T01:06:12Z

I have not traced it all the way but it is from github.com/tektoncd/pipeline v0.10.1

howardjohn · 2020-03-07T01:06:24Z

Ahh thats right... we have had this problem before. K8s license checker checks code it uses from vendor, which is pruned, whereas istio's is checking all transitive dependencies by go mod graph.

github.com/tektoncd/plumbing > github.com/google/go-licenses > github.com/otiai10/copy > github.com/otiai10/mint > github.com/otiai10/curr

I don't know what we do now.

ixdy · 2020-03-07T01:17:09Z

I opened otiai10/curr#2. We'll see if the maintainer is willing to switch the license.

go.mod

ixdy · 2020-03-07T18:32:25Z

oh, this is going to need me to re-run make gen, right.

Additionally, update (and pin) - github.com/otiai10/curr - github.com/otiai10/mint - github.com/pelletier/go-buffruneio

ixdy · 2020-03-07T18:44:55Z

ok, looks like this is hopefully set once istio/common-files#206 gets in.

howardjohn · 2020-03-07T18:50:40Z

thanks! sorry for the license issues. the common files update should automatically get merged in here in a bit

howardjohn · 2020-03-07T19:32:05Z

/retest should work without rebase

…

On Sat, Mar 7, 2020, 11:31 AM Travis Clarke ***@***.***> wrote: @ixdy <https://github.com/ixdy> - rebase to pull in the new common/config/license-lint.yml, then I think all is set. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2447?email_source=notifications&email_token=AAEYGXNAJAIX4NLQ24FMNPTRGKOH5A5CNFSM4K6H7NBKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOEDQ7Q#issuecomment-596129918>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEYGXNXSLESYYVNXXBVIM3RGKOH5ANCNFSM4K6H7NBA> .

ixdy · 2020-03-07T20:06:20Z

ok, the last remaining question: does anything automatically build and publish gcr.io/istio-testing/mason? or is that a manual process?

clarketm · 2020-03-07T20:14:29Z

ok, the last remaining question: does anything automatically build and publish gcr.io/istio-testing/mason? or is that a manual process?

It is currently manual. make -C boskos mason-image should do the trick. I can automate this for the next time (#2484).

ixdy · 2020-03-07T20:18:45Z

can you build and push when convenient? (doesn't need to happen this weekend.) I don't have access to the istio-testing project.

clarketm · 2020-03-07T20:28:39Z

can you build and push when convenient? (doesn't need to happen this weekend.) I don't have access to the istio-testing project.

gcr.io/istio-testing/mason:v20200307-1.5.0-19-gc06282b0

ixdy requested a review from a team as a code owner February 29, 2020 02:23

googlebot added the cla: yes Set by the Google CLA bot to indicate the author of a PR has signed the Google CLA. label Feb 29, 2020

istio-testing added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Feb 29, 2020

istio-testing added the needs-ok-to-test label Feb 29, 2020

ixdy force-pushed the update-gcp-mason branch from 5d6d287 to 89478f8 Compare February 29, 2020 02:24

istio-testing added ok-to-test Set this label allow normal testing to take place for a PR not submitted by an Istio org member. and removed needs-ok-to-test labels Feb 29, 2020

clarketm assigned sebastienvas and fejta Feb 29, 2020

fejta reviewed Mar 2, 2020

View reviewed changes

go.mod Show resolved Hide resolved

ixdy force-pushed the update-gcp-mason branch from 89478f8 to 1c1268c Compare March 5, 2020 03:40

istio-testing added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Mar 5, 2020

ixdy force-pushed the update-gcp-mason branch from 4747f13 to 05a745b Compare March 6, 2020 01:29

clarketm mentioned this pull request Mar 7, 2020

whitelist modules for k/t-i bump istio/tools#782

Closed

howardjohn reviewed Mar 7, 2020

View reviewed changes

licenses/github.com/otiai10/curr/LICENSE Outdated Show resolved Hide resolved

otiai10 mentioned this pull request Mar 7, 2020

Change LICENSE to MIT otiai10/curr#3

Merged

clarketm mentioned this pull request Mar 7, 2020

whitelist modules for k8s.io/test-infra dep istio/common-files#206

Merged

clarketm approved these changes Mar 7, 2020

View reviewed changes

go.mod Outdated Show resolved Hide resolved

clarketm added the do-not-merge/hold Block automatic merging of a PR. label Mar 7, 2020

Update k8s.io/test-infra dependency and fix GCP Mason

f82e961

Additionally, update (and pin) - github.com/otiai10/curr - github.com/otiai10/mint - github.com/pelletier/go-buffruneio

ixdy force-pushed the update-gcp-mason branch from 7ba40df to f82e961 Compare March 7, 2020 18:37

howardjohn approved these changes Mar 7, 2020

View reviewed changes

clarketm removed the do-not-merge/hold Block automatic merging of a PR. label Mar 7, 2020

istio-testing merged commit c06282b into istio:master Mar 7, 2020

clarketm mentioned this pull request Mar 9, 2020

fix install of istio/test-infra dependencies in build-tools istio/tools#792

Merged

Comments

Conversation

ixdy commented Feb 29, 2020

Uh oh!

istio-policy-bot commented Feb 29, 2020

Uh oh!

istio-testing commented Feb 29, 2020

Uh oh!

clarketm commented Feb 29, 2020

Uh oh!

Uh oh!

ixdy commented Mar 5, 2020

Uh oh!

clarketm commented Mar 5, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ixdy commented Mar 5, 2020

Uh oh!

clarketm commented Mar 5, 2020

Uh oh!

ixdy commented Mar 6, 2020

Uh oh!

ixdy commented Mar 6, 2020

Uh oh!

clarketm commented Mar 7, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

howardjohn commented Mar 7, 2020 via email

Uh oh!

ixdy commented Mar 7, 2020

Uh oh!

clarketm commented Mar 7, 2020

Uh oh!

howardjohn commented Mar 7, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ixdy commented Mar 7, 2020

Uh oh!

Uh oh!

ixdy commented Mar 7, 2020

Uh oh!

ixdy commented Mar 7, 2020

Uh oh!

howardjohn commented Mar 7, 2020

Uh oh!

howardjohn commented Mar 7, 2020 via email

Uh oh!

ixdy commented Mar 7, 2020

Uh oh!

clarketm commented Mar 7, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ixdy commented Mar 7, 2020

Uh oh!

clarketm commented Mar 7, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

clarketm commented Mar 5, 2020 •

edited

Loading

clarketm commented Mar 7, 2020 •

edited

Loading

howardjohn commented Mar 7, 2020 •

edited

Loading

clarketm commented Mar 7, 2020 •

edited

Loading