cmd/go: track tool dependencies in go.mod

[mtibben]

UPDATE: 2024-07-29: the latest proposal can be found here.

---

Background

The current best-practice to track tool dependencies for a module is to add a tools.go file to your module that includes import statements for the tools of interest. This has been extensively discussed in #25922 and is the recommended approach in the Modules FAQ

This approach works, but managing the tool dependencies still feels like a missing piece in the go mod toolchain. For example, the instructions for getting a user set up with a new project using gqlgen (a codegen tool) looks like this

# Initialise a new go module
mkdir example
cd example
go mod init example

# Add gqlgen as a tool
printf '// +build tools\npackage tools\nimport _ "github.com/99designs/gqlgen"' | gofmt > tools.go
go mod tidy

# Initialise gqlgen config and generate models
go run github.com/99designs/gqlgen init

The printf line above really stands out as an arbitrary command to "add a tool" and reflects a poor developer experience when managing tools. For example, an immediate problem is that the printf line will only work on unix systems and not windows. And what happens if tools.go already exists?

So while we have some excellent tools for managing dependencies within the go.mod file using go get and go mod edit, there is no such equivalent for managing tools in the tools.go file.

Proposed Solution

The go.mod file uses the // indirect comment to track some dependencies. An // indirect comment indicates that no package from the required module is directly imported by any package in the main module (source).

I propose that this same mechanism be used to add tool dependencies, using a // tool comment.

Users could add a tool with something like

go get -tool github.com/99designs/[email protected]

or

go mod edit -require=github.com/99designs/gqlgen -tool

A go.mod would then look something like

module example

go 1.17

require (
	github.com/99designs/gqlgen v0.14.0 // tool
)

And would allow users to subsequently run the tool with go run github.com/99designs/gqlgen

This would mean a separate tools.go file would no longer be required as the tool dependency is tracked in the go.mod file.

Go modules would be "tool" aware. For example:

• go mod tidy would not remove the // tool dependency, even though it is not referenced directly in the module
• Perhaps if a module with a // tool dependency is imported by another module, Go modules understands that the // tool dependency is not required as an indirect dependency. Currently when using tools.go, go modules does not have that context and the tool is treated like any other indirect dependency
• go get -tool [packages] would only add a dependency with a main package

[fsouza]

I like this, I find it annoying to use the tools.go solution, though I'll admit I don't have a better complaint than it being annoying/weird.

If this proposal moves forward, where does the dependency go in the go.mod file? (assuming the 1.17 format with multiple require blocks). Will it have a dedicated block for tools? Or are tools treated like // indirect and placed in the same block?

[ianlancetaylor]

CC @bcmills @jayconrod

[mtibben]

If this proposal moves forward, where does the dependency go in the go.mod file? (assuming the 1.17 format with multiple require blocks). Will it have a dedicated block for tools? Or are tools treated like // indirect and placed in the same block?

Good question! I'm not so familiar with the reasoning behind the multiple blocks... something to do with lazy loading? I'd defer to those with more experience in this area

[mvdan]

Personally, I think #42088 is already a pretty good solution. With it, one can write go generate lines like:

//go:generate go run golang.org/x/tools/cmd/stringer@1a7ca93429 -type=Foo

Similarly, go run pkg@version can be used in scripts, makefiles, and so on. Plus, it doesn't even require a go.mod file to be used; you can use this method anywhere, just like go install pkg@version.

Another big advantage is that you can pick specific versions of tools, and they won't interfere with your main go.mod module dependency graph. Perhaps I want to use a generator that pulls in an unstable master version of a library that my project also uses, and I don't want my project to be forced into using the same newer unstable version.

[mvdan]

The only downside to #42088 is that, if you repeat the same go run pkg@version commands across multiple files, it can get a bit repetitive. Luckily, you have multiple solutions at hand: sed scripts to keep the versions in sync, short script files to deduplicate the commands, or even a module-aware tool that could sync go run pkg@version strings with a go.mod file, if you wanted to do that.

[seankhliao]

Or GOBIN=local-dir go install pkg@version, always run from the local directory and not clobber whatever version the user may have globally installed.
I think it would be a mistake for modules to implicitly rely on shared mutable global bin dir for a first class workflow

[mtibben]

Oh interesting, thanks @mvdan I wasn't aware of that solution. 🤔

A few concerns immediately come to mind...

1. You mean go run hack.me/[email protected] will just download and run some random go code 😱 That is slightly unexpected to me, equivalent to a curl | bash command. My assumption was always that go run ran local code or modules already specified in go.mod, but seems that assumption is incorrect

2. Should gqlgen instructions always be to specify version with go run github.com/99designs/[email protected]? That seems verbose

3. Repetition across multiple files, keeping version in sync, yep your comment above nails it

[mtibben]

Also this go run solution should probably be added to the Go Modules FAQ if this is now considered best-practice for go:generate tools

[mvdan]

In module mode, go run can always download, build, and run arbitrary code. The difference between go run pkg relying on go.mod and go run pkg@version is how you specify the version and how it's verified. With a go.mod, you are forced into a specific version recorded in go.mod and go.sum. Without one, it's up to you what version you specify; @master is obviously risky, @full-commit-hash is safest, and @v1.2.3 is a middle ground that would probably be best for most people. Even if a malicious upstream rewrites a tag to inject code, GOPROXY and GOSUMDB should protect you from that.

[mvdan]

Also this go run solution should probably be added to the Go Modules FAQ if this is now considered best-practice for go:generate tools

It certainly warrants a mention. I'm not sure we should bless it as the only best practice, though, because there can be legitimate reasons for versioning, downloading, and running tools some other way. Perhaps some of your tools aren't written in Go, such as protoc, so you use a "tool bundler" that's entirely separate to Go. Or perhaps you do need your tools to share the same MVS graph with your main module for proper compatibility, so you want them to share a go.mod file.

[mtibben]

Gotta say though... go run pkg@version seems like a massive security footgun to me.

go install I understand well that it can download code from a remote location and build a binary. It's not obvious at all that go run directly executes code from a remote location, and I wonder how widely that is understood.

[mtibben]

So even with the go run pkg@version approach, I still think this proposal has value for specifying tool dependency versions in the context of a module. This approach avoids requiring a tools.go file (as with the existing best-practice), and avoids specifying the tool version for every file that uses it (with the go run approach)

[lwc]

Also worth noting: codegen tools like gqlgen and protobuf are often comprised of a generator command and a runtime, both of which typically need to be versioned in lock-step.

This proposal solves that case rather neatly, allowing go.mod to manage both generator and runtime versions.

[fsouza]

Personally, I think #42088 is already a pretty good solution. With it, one can write go generate lines like:

//go:generate go run golang.org/x/tools/cmd/stringer@1a7ca93429 -type=Foo

Similarly, go run pkg@version can be used in scripts, makefiles, and so on. Plus, it doesn't even require a go.mod file to be used; you can use this method anywhere, just like go install pkg@version.

We used to do that. Then people would have that replicated across different files and the version wouldn't always match, and we wanted to automate tool updating, so we figured that migrating to tools.go + having everything in go.mod would be better for compatibility with the ecosystem built around go modules (vs rolling our own tool to keep modules used directly in //go:generate up to date).

Again, tools.go works, but it's weird (not very scientific, I know 🙈). I think this proposal makes version management of tools better because it enables people to manage them using solely go commands (vs things like the bash oneliner shared by the OP).

[bcmills]

@jayconrod has previously suggested something similar, using a new directive (perhaps tool?) instead of a // tool comment.

Personally, I prefer the approach of adding a new directive — today we do treat requirements with // indirect comments a bit specially in terms of syntax, but they are semantically still just comments, and I would rather keep them that way at least to the extent possible.

A new tool directive, on the other hand, would allow us to preserve the existing semantics of go mod tidy without special treatment for // tool comments.

[mvdan]

@bcmills would such tool requirements be part of the same MVS module graph?

[bcmills]

The tool directive would list package paths (not module requirements), and the named packages would be treated as if imported in a .go source file in the main module.

In particular:

• go mod tidy would ensure that the packages transitively imported by the named package (and its test) can be resolved from the module graph.
• go mod vendor would copy the packages transitively imported by the name package into the vendor directory (but would omit its test code and dependencies as usual).
• go list direct (cmd/go: enable listing direct dependency updates #40364) would report the named packages as direct imports.

[carldunham]

Or go list tools

[jayconrod]

I like this proposal. I've had something similar in my drafts folder for a while. @bcmills touched on the main difference. go.mod would have a tool directive that would name the full package path for the tool. You'd still need a separate require directive for the containing module, and that would be treated like a normal require directive by MVS.

module example.com/use

go 1.18

require golang.org/x/tools v0.1.6

tool golang.org/x/tools/cmd/stringer

I don't think go run tool@version and go install tool@version completely replace go run tool and go install tool. When the @version suffix is used, it ignores the go.mod file for the current module. That's useful most of the time, but not if you want to track your tool dependencies together with other dependencies, or if you want to use a patched version of a tool (applying replace directives).

[mtibben]

Yeah I like the tool directive. There might be a couple of tradeoffs with compatibility with older go versions. A tool directive wouldn't be recognised by older go versions, and presumably ignored. A require directive with // tool would be recognised, but would be removed by a go mod tidy.

A tool directive would keep the dependency tree separate - as they should be. For example, I don't think indirect dependencies would need to be tracked for tools, or shared by the module. Essentially a tool directive would specify a version when running go run tool instead of needing go run tool@version

[mtibben]

Or have I got that wrong? Is sharing indirect dependencies between tools and other dependencies a desirable feature?

[jayconrod]

A tool directive wouldn't be recognised by older go versions, and presumably ignored. A require directive with // tool would be recognised, but would be removed by a go mod tidy.

Right. The go command reports errors for unknown directives in the main module's go.mod file, but it ignores unknown directives in dependencies' go.mod files. So everyone working on a module that used this would need to upgrade to a version of Go that supports it (same as most other new features), but their users would be unaffected.

A tool directive would keep the dependency tree separate - as they should be. For example, I don't think indirect dependencies would need to be tracked for tools, or shared by the module. Essentially a tool directive would specify a version when running go run tool instead of needing go run tool@version

Or have I got that wrong? Is sharing indirect dependencies between tools and other dependencies a desirable feature?

My suggestion is to have tool act as a disembodied import declaration: it's just in go.mod instead of tools.go. You'd still need a require directive for the module providing the tool, and it would be treated as a regular requirement by go mod tidy and everything else.

If you don't want to mix tool and library dependencies in go.mod, it's probably better to either use go run tool@version or to have a separate tools.mod file, then go run -modfile=tools.mod tool.

[mtibben]

Yep that makes a lot of sense @jayconrod

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[mtibben]

@jayconrod Did you want to write up the tool directive approach that we could incorporate as an option into this proposal? I'm happy to collaborate on it with you. Positive feedback on that approach so far in this thread, and it would be good to compare the options directly against each other, now that this proposal will be considered by the go-powers-that-be

[bcmills]

@ConradIrwin, unfortunately I won't be able to review these changes after all; I'm leaving Google (and the Go team) on this Friday, March 15.

[ConradIrwin]

@bcmills wow, big change! I hope that everything turns out well for you.

@rsc / @matloob is there anyone else that would be interested in taking this on?

[acramsay]

Hi all. I've been quietly watching this work for a few months now and I'm quite excited for this new functionality! That said, the last comments on this issue are a little discouraging. If possible, I would appreciate an update. Has anyone from the Go team taken over the code review? Is this progressing?

Thanks everyone for working toward this contribution!!

[matloob]

Unfortunately, at this point I'm not sure if we're going to have the time to review this before the freeze.

That said, I'll try to start reviewing the CLs at the bottom of the stack and we can submit them together if we are able to get the stack done in time.

[ConradIrwin]

@matloob I would love to have it reviewed before the next freeze if that's possible :D. Should I connect with you when the window opens again?

[matloob]

Yes that sounds good. I'll try to see if I can start during the freeze so we can maybe make progress before the window opens.

[ConradIrwin]

Great, thank you!

…

On Thu, Apr 18 2024 at 11:59, Michael Matloob ***@***.***> wrote: Yes that sounds good. I'll try to see if I can start during the freeze so we can maybe make progress before the window opens. — Reply to this email directly, view it on GitHub <#48429 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAXAQGM3MK3HNJZT2V6BCTY6ACY7AVCNFSM5EGC4OSKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMBWGQ3TMNBYHAYQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[meling]

Just pinging to check on the progress of this one; since there is an implementation that mainly requires review it seems like a relatively low-hanging fruit to get this done before the window opens ;-)

[matloob]

I've started the reviews of the CLs.

[pierrre]

With this proposal, does the go.mod file contain the list of transitive dependencies of the tools ?

[matloob]

@pierrre Yes. Tools will be treated similar to as if they were a package of the main module, so a tidy go.mod will contain all the dependencies needed to build it.

[stevenh]

Nice to see this being worked on.

[gopherbot]

Change https://go.dev/cl/614555 mentions this issue: cmd/mod/edit: disallow relative tool paths

[mcandre]

Reminder that buildtime dependencies installed via out of band methods are likely missing from SBOM's sent to SCA tools.

govulncheck, Snyk, Dependabot, etc. look primarily in go.mod for information about Go dependencies.

A few container SCA tools like Docker Scout, Red Hat Quay Container Security Operator, Snyk Container may happen to match CVE's for vulnerable buildtime Go dependency tools. But only for Go tools installed via OS package managers, and likely only for Go tools that are not removed from the final image using modern, multi stage images.