Replace custom `go` helper with `go list` #7045

jeffwidman · 2023-04-10T19:56:31Z

Over time, we've managed to thin out our custom go helper in favor of native package manager tooling via go list, go get, and go mod.

The final remaining usage was for translating package import URLs into the actual repository URL for fetching metadata.

The original impetus for the github.com/Masterminds/vcs library was that it was a copy/paste of some code in core go:

Further context:

I think the code is trying to map a Go import path back to a repository so we can e.g. link to the repo in the Dependabot PR, find the CHANGELOG, link to the diff between versions, etc, etc.
It’s not totally trivial with Go because:

they’ve got that whole vanity URL thing going on, where you have to pass go-get=1 in to make it act like a fancy redirect to a repo

a Go import path doesn’t necessarily actually match a repo URL. For instance, I might be able to import https://github.com/go-kit/kit/tracing, but that’s not a valid URL on GitHub. The repo is github.com/go-kit/kit, and the package is browsable at https://github.com/go-kit/kit/tree/master/tracing. So Go bakes in some logic to handle that for a series of SCMs, and the Masterminds/vcs package does a similar thing too.

Ideally, we’d just use whatever golang/go does, but IIRC that wasn’t importable so the next best thing was the Masterminds/vcs implementation. The Ruby code you linked to predates our usage of that Go library, and I’d guess is just a really incomplete Ruby implementation of the same thing.
Perhaps these days a) there’s a good Ruby implementation, or b) we think the VCS list is stable enough we’d be happy to translate it to Ruby, or c) we could somehow codegen the Ruby from the canonical Go implementation? Or we just stick with that one-off Go helper?

However, I discovered after doing some digging in #6938 that there is both an upstream copy (golang/go#18387 (comment), which may potentially get deprecated golang/go#57051) and an actual go list command that provides enough of what we need:

cmd/go: include vcs revision in module .info files golang/go#44742

$ go list -m -f '{{.Origin}}' golang.org/x/tools@latest
{git https://go.googlesource.com/tools    refs/tags/v0.3.0 502c634771c4ba335286d55fc24eeded1704f592 }

While this doesn't fully resolve the upstream issue:

cmd/go: provide a way to resolve an import path to a vcs, url, etc. golang/go#18387

For our purposes it provides enough of what we need since we already have the module path extracted from go.mod (golang/go#18387 (comment)).

Performing this switch allows us to completely delete the native go helper altogether.

I debated leaving the framework in place in case we later wish to extend it for other features, but decided for now to rip it out as it's easy enough to add back later. And in general we're trying to move toward leveraging native package manager tooling whenever possible. So if we later needed an additional feature, we're probably start with a discussion with the go team to see if they'd build it into the go cmd tooling.

Fix #6938

jeffwidman · 2023-04-10T19:58:10Z

go_modules/spec/dependabot/go_modules/metadata_finder_spec.rb

@@ -63,4 +63,46 @@
      it { is_expected.to eq("https://github.com/satori/go.uuid") }
    end
  end
+
+  describe "#look_up_source_using_go_list" do


These test cases are moved here via copy/paste from the deleted go_modules/spec/dependabot/go_modules/path_converter_spec.rb.

go_modules/lib/dependabot/go_modules/update_checker.rb

go_modules/lib/dependabot/go_modules/file_updater/go_mod_updater.rb

Over time, we've managed to thin out our custom `go` helper in favor of native package manager tooling via `go list`, `go get`, and `go mod`. The final remaining usage was for translating package import URLs into the actual repository URL for fetching metadata. The original impetus for the `github.com/Masterminds/vcs` library was that it was a copy/paste of some code in core `go`: [Further context](#4448 (comment)): > I think the code is trying to map a Go import path back to a repository so we can e.g. link to the repo in the Dependabot PR, find the CHANGELOG, link to the diff between versions, etc, etc. > It’s not totally trivial with Go because: > 1. they’ve got that whole vanity URL thing going on, where you have to pass `go-get=1` in to make it act like a fancy redirect to a repo > 2. a Go import path doesn’t necessarily actually match a repo URL. For instance, I might be able to import https://github.com/go-kit/kit/tracing, but that’s not a valid URL on GitHub. The repo is [github.com/go-kit/kit](https://github.com/go-kit/kit), and the package is browsable at https://github.com/go-kit/kit/tree/master/tracing. So Go bakes in some [logic](https://github.com/golang/go/blob/95c125a44ad1a0c3e441c3214160cd7b4483e79c/src/cmd/go/internal/vcs/vcs.go#L1437) to handle that for a series of SCMs, and the Masterminds/vcs package does a [similar thing](https://github.com/Masterminds/vcs/blob/master/vcs_remote_lookup.go) too. > > Ideally, we’d just use whatever golang/go does, but IIRC that wasn’t importable so the next best thing was the Masterminds/vcs implementation. The Ruby code you linked to predates our usage of that Go library, and I’d guess is just a really incomplete Ruby implementation of the same thing. > Perhaps these days a) there’s a good Ruby implementation, or b) we think the VCS list is stable enough we’d be happy to translate it to Ruby, or c) we could somehow codegen the Ruby from the canonical Go implementation? Or we just stick with that one-off Go helper? However, I discovered after doing some digging in #6938 that there _is both_ an upstream copy (golang/go#18387 (comment), which may potentially get deprecated golang/go#57051) and an actual `go list` command that provides enough of what we need: * golang/go#44742 ``` $ go list -m -f '{{.Origin}}' golang.org/x/tools@latest {git https://go.googlesource.com/tools refs/tags/v0.3.0 502c634771c4ba335286d55fc24eeded1704f592 } ``` While this doesn't fully resolve the upstream issue: * golang/go#18387 For our purposes it provides enough of what we need since we already have the module path extracted from `go.mod` (golang/go#18387 (comment)). Performing this switch allows us to completely delete the native go helper altogether. I debated leaving the framework in place in case we later wish to extend it for other features, but decided for now to rip it out as it's easy enough to add back later. And in general we're trying to move toward leveraging native package manager tooling whenever possible. So if we later needed an additional feature, we're probably start with a discussion with the `go` team to see if they'd build it into the `go` cmd tooling.

jakecoffman · 2023-04-11T12:56:32Z

The Go smoke tests hint at a problem. Take this for example:

$ curl https://golang.org/x/text\?go-get\=1
<!DOCTYPE html>
<html lang="en">
<title>The Go Programming Language</title>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
<meta name="go-import" content="golang.org/x/text git https://go.googlesource.com/text">
<meta name="go-source" content="golang.org/x/text https://github.com/golang/text/ https://github.com/golang/text/tree/master{/dir} https://github.com/golang/text/blob/master{/dir}/{file}#L{line}">
<meta http-equiv="refresh" content="0; url=https://pkg.go.dev/golang.org/x/text">
</head>
<body>
<a href="https://pkg.go.dev/golang.org/x/text">Redirecting to documentation...</a>
</body>
</html>

The Origin only lists the go-import meta tag content:

$ go list -m -f '{{.Origin.URL}}' golang.org/x/text@latest
https://go.googlesource.com/text

Our current solution returns the go-source meta tag content, so we'd lose the PR content for these common dependencies and any other set up in this way.

jeffwidman · 2023-04-12T08:28:24Z

Thanks for pointing that out @jakecoffman ... I'd been only working on this locally so didn't even notice the smoke tests were failing!

I poked around the interwebs a bit, but this one looks tricky... the most straightforward thing would be if go list starts exposing the go-source meta tag content as well.

jeffwidman · 2023-12-16T01:39:43Z

Given that the go team declined my feature request to expose the go-source meta tag content (golang/go#59566), I don't see a path forward here unfortunately until golang/go#18387 is resolved.

jeffwidman commented Apr 10, 2023

View reviewed changes

go_modules/lib/dependabot/go_modules/update_checker.rb Show resolved Hide resolved

jeffwidman commented Apr 10, 2023

View reviewed changes

go_modules/lib/dependabot/go_modules/file_updater/go_mod_updater.rb Show resolved Hide resolved

jeffwidman force-pushed the switch-metadata-to-use-go-list-rather-than-custom-native-helper branch from 2e830ba to e43685c Compare April 10, 2023 20:23

jeffwidman force-pushed the switch-metadata-to-use-go-list-rather-than-custom-native-helper branch from e43685c to feced2c Compare April 10, 2023 20:46

jeffwidman mentioned this pull request Apr 12, 2023

proposal: cmd/go: go list: report go-source tag information golang/go#59566

Closed

jeffwidman closed this Dec 16, 2023

jeffwidman deleted the switch-metadata-to-use-go-list-rather-than-custom-native-helper branch June 27, 2024 22:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace custom `go` helper with `go list` #7045

Replace custom `go` helper with `go list` #7045

jeffwidman commented Apr 10, 2023 •

edited

Loading

jeffwidman Apr 10, 2023 •

edited

Loading

jakecoffman commented Apr 11, 2023

jeffwidman commented Apr 12, 2023 •

edited

Loading

jeffwidman commented Dec 16, 2023 •

edited

Loading

Replace custom go helper with go list #7045

Replace custom go helper with go list #7045

Conversation

jeffwidman commented Apr 10, 2023 • edited Loading

jeffwidman Apr 10, 2023 • edited Loading

Choose a reason for hiding this comment

jakecoffman commented Apr 11, 2023

jeffwidman commented Apr 12, 2023 • edited Loading

jeffwidman commented Dec 16, 2023 • edited Loading

Replace custom `go` helper with `go list` #7045

Replace custom `go` helper with `go list` #7045

jeffwidman commented Apr 10, 2023 •

edited

Loading

jeffwidman Apr 10, 2023 •

edited

Loading

jeffwidman commented Apr 12, 2023 •

edited

Loading

jeffwidman commented Dec 16, 2023 •

edited

Loading