cmd/go: track tool dependencies in go.mod

[mtibben]

UPDATE: 2024-07-29: the latest proposal can be found here.

---

Background

The current best-practice to track tool dependencies for a module is to add a tools.go file to your module that includes import statements for the tools of interest. This has been extensively discussed in #25922 and is the recommended approach in the Modules FAQ

This approach works, but managing the tool dependencies still feels like a missing piece in the go mod toolchain. For example, the instructions for getting a user set up with a new project using gqlgen (a codegen tool) looks like this

# Initialise a new go module
mkdir example
cd example
go mod init example

# Add gqlgen as a tool
printf '// +build tools\npackage tools\nimport _ "github.com/99designs/gqlgen"' | gofmt > tools.go
go mod tidy

# Initialise gqlgen config and generate models
go run github.com/99designs/gqlgen init

The printf line above really stands out as an arbitrary command to "add a tool" and reflects a poor developer experience when managing tools. For example, an immediate problem is that the printf line will only work on unix systems and not windows. And what happens if tools.go already exists?

So while we have some excellent tools for managing dependencies within the go.mod file using go get and go mod edit, there is no such equivalent for managing tools in the tools.go file.

Proposed Solution

The go.mod file uses the // indirect comment to track some dependencies. An // indirect comment indicates that no package from the required module is directly imported by any package in the main module (source).

I propose that this same mechanism be used to add tool dependencies, using a // tool comment.

Users could add a tool with something like

go get -tool github.com/99designs/[email protected]

or

go mod edit -require=github.com/99designs/gqlgen -tool

A go.mod would then look something like

module example

go 1.17

require (
	github.com/99designs/gqlgen v0.14.0 // tool
)

And would allow users to subsequently run the tool with go run github.com/99designs/gqlgen

This would mean a separate tools.go file would no longer be required as the tool dependency is tracked in the go.mod file.

Go modules would be "tool" aware. For example:

• go mod tidy would not remove the // tool dependency, even though it is not referenced directly in the module
• Perhaps if a module with a // tool dependency is imported by another module, Go modules understands that the // tool dependency is not required as an indirect dependency. Currently when using tools.go, go modules does not have that context and the tool is treated like any other indirect dependency
• go get -tool [packages] would only add a dependency with a main package

[fsouza]

I like this, I find it annoying to use the tools.go solution, though I'll admit I don't have a better complaint than it being annoying/weird.

If this proposal moves forward, where does the dependency go in the go.mod file? (assuming the 1.17 format with multiple require blocks). Will it have a dedicated block for tools? Or are tools treated like // indirect and placed in the same block?

[ianlancetaylor]

CC @bcmills @jayconrod

[mtibben]

If this proposal moves forward, where does the dependency go in the go.mod file? (assuming the 1.17 format with multiple require blocks). Will it have a dedicated block for tools? Or are tools treated like // indirect and placed in the same block?

Good question! I'm not so familiar with the reasoning behind the multiple blocks... something to do with lazy loading? I'd defer to those with more experience in this area

[mvdan]

Personally, I think #42088 is already a pretty good solution. With it, one can write go generate lines like:

//go:generate go run golang.org/x/tools/cmd/stringer@1a7ca93429 -type=Foo

Similarly, go run pkg@version can be used in scripts, makefiles, and so on. Plus, it doesn't even require a go.mod file to be used; you can use this method anywhere, just like go install pkg@version.

Another big advantage is that you can pick specific versions of tools, and they won't interfere with your main go.mod module dependency graph. Perhaps I want to use a generator that pulls in an unstable master version of a library that my project also uses, and I don't want my project to be forced into using the same newer unstable version.

[mvdan]

The only downside to #42088 is that, if you repeat the same go run pkg@version commands across multiple files, it can get a bit repetitive. Luckily, you have multiple solutions at hand: sed scripts to keep the versions in sync, short script files to deduplicate the commands, or even a module-aware tool that could sync go run pkg@version strings with a go.mod file, if you wanted to do that.

[seankhliao]

Or GOBIN=local-dir go install pkg@version, always run from the local directory and not clobber whatever version the user may have globally installed.
I think it would be a mistake for modules to implicitly rely on shared mutable global bin dir for a first class workflow

[mtibben]

Oh interesting, thanks @mvdan I wasn't aware of that solution. 🤔

A few concerns immediately come to mind...

1. You mean go run hack.me/[email protected] will just download and run some random go code 😱 That is slightly unexpected to me, equivalent to a curl | bash command. My assumption was always that go run ran local code or modules already specified in go.mod, but seems that assumption is incorrect

2. Should gqlgen instructions always be to specify version with go run github.com/99designs/[email protected]? That seems verbose

3. Repetition across multiple files, keeping version in sync, yep your comment above nails it

[mtibben]

Also this go run solution should probably be added to the Go Modules FAQ if this is now considered best-practice for go:generate tools

[mvdan]

In module mode, go run can always download, build, and run arbitrary code. The difference between go run pkg relying on go.mod and go run pkg@version is how you specify the version and how it's verified. With a go.mod, you are forced into a specific version recorded in go.mod and go.sum. Without one, it's up to you what version you specify; @master is obviously risky, @full-commit-hash is safest, and @v1.2.3 is a middle ground that would probably be best for most people. Even if a malicious upstream rewrites a tag to inject code, GOPROXY and GOSUMDB should protect you from that.

[mvdan]

Also this go run solution should probably be added to the Go Modules FAQ if this is now considered best-practice for go:generate tools

It certainly warrants a mention. I'm not sure we should bless it as the only best practice, though, because there can be legitimate reasons for versioning, downloading, and running tools some other way. Perhaps some of your tools aren't written in Go, such as protoc, so you use a "tool bundler" that's entirely separate to Go. Or perhaps you do need your tools to share the same MVS graph with your main module for proper compatibility, so you want them to share a go.mod file.

[mtibben]

Gotta say though... go run pkg@version seems like a massive security footgun to me.

go install I understand well that it can download code from a remote location and build a binary. It's not obvious at all that go run directly executes code from a remote location, and I wonder how widely that is understood.

[mtibben]

So even with the go run pkg@version approach, I still think this proposal has value for specifying tool dependency versions in the context of a module. This approach avoids requiring a tools.go file (as with the existing best-practice), and avoids specifying the tool version for every file that uses it (with the go run approach)

[lwc]

Also worth noting: codegen tools like gqlgen and protobuf are often comprised of a generator command and a runtime, both of which typically need to be versioned in lock-step.

This proposal solves that case rather neatly, allowing go.mod to manage both generator and runtime versions.

[fsouza]

Personally, I think #42088 is already a pretty good solution. With it, one can write go generate lines like:

//go:generate go run golang.org/x/tools/cmd/stringer@1a7ca93429 -type=Foo

Similarly, go run pkg@version can be used in scripts, makefiles, and so on. Plus, it doesn't even require a go.mod file to be used; you can use this method anywhere, just like go install pkg@version.

We used to do that. Then people would have that replicated across different files and the version wouldn't always match, and we wanted to automate tool updating, so we figured that migrating to tools.go + having everything in go.mod would be better for compatibility with the ecosystem built around go modules (vs rolling our own tool to keep modules used directly in //go:generate up to date).

Again, tools.go works, but it's weird (not very scientific, I know 🙈). I think this proposal makes version management of tools better because it enables people to manage them using solely go commands (vs things like the bash oneliner shared by the OP).

[bcmills]

@jayconrod has previously suggested something similar, using a new directive (perhaps tool?) instead of a // tool comment.

Personally, I prefer the approach of adding a new directive — today we do treat requirements with // indirect comments a bit specially in terms of syntax, but they are semantically still just comments, and I would rather keep them that way at least to the extent possible.

A new tool directive, on the other hand, would allow us to preserve the existing semantics of go mod tidy without special treatment for // tool comments.

[mvdan]

@bcmills would such tool requirements be part of the same MVS module graph?

[bcmills]

The tool directive would list package paths (not module requirements), and the named packages would be treated as if imported in a .go source file in the main module.

In particular:

• go mod tidy would ensure that the packages transitively imported by the named package (and its test) can be resolved from the module graph.
• go mod vendor would copy the packages transitively imported by the name package into the vendor directory (but would omit its test code and dependencies as usual).
• go list direct (cmd/go: enable listing direct dependency updates #40364) would report the named packages as direct imports.

[carldunham]

Or go list tools

[jayconrod]

I like this proposal. I've had something similar in my drafts folder for a while. @bcmills touched on the main difference. go.mod would have a tool directive that would name the full package path for the tool. You'd still need a separate require directive for the containing module, and that would be treated like a normal require directive by MVS.

module example.com/use

go 1.18

require golang.org/x/tools v0.1.6

tool golang.org/x/tools/cmd/stringer

I don't think go run tool@version and go install tool@version completely replace go run tool and go install tool. When the @version suffix is used, it ignores the go.mod file for the current module. That's useful most of the time, but not if you want to track your tool dependencies together with other dependencies, or if you want to use a patched version of a tool (applying replace directives).

[mtibben]

Yeah I like the tool directive. There might be a couple of tradeoffs with compatibility with older go versions. A tool directive wouldn't be recognised by older go versions, and presumably ignored. A require directive with // tool would be recognised, but would be removed by a go mod tidy.

A tool directive would keep the dependency tree separate - as they should be. For example, I don't think indirect dependencies would need to be tracked for tools, or shared by the module. Essentially a tool directive would specify a version when running go run tool instead of needing go run tool@version

[mtibben]

Or have I got that wrong? Is sharing indirect dependencies between tools and other dependencies a desirable feature?

[jayconrod]

A tool directive wouldn't be recognised by older go versions, and presumably ignored. A require directive with // tool would be recognised, but would be removed by a go mod tidy.

Right. The go command reports errors for unknown directives in the main module's go.mod file, but it ignores unknown directives in dependencies' go.mod files. So everyone working on a module that used this would need to upgrade to a version of Go that supports it (same as most other new features), but their users would be unaffected.

A tool directive would keep the dependency tree separate - as they should be. For example, I don't think indirect dependencies would need to be tracked for tools, or shared by the module. Essentially a tool directive would specify a version when running go run tool instead of needing go run tool@version

Or have I got that wrong? Is sharing indirect dependencies between tools and other dependencies a desirable feature?

My suggestion is to have tool act as a disembodied import declaration: it's just in go.mod instead of tools.go. You'd still need a require directive for the module providing the tool, and it would be treated as a regular requirement by go mod tidy and everything else.

If you don't want to mix tool and library dependencies in go.mod, it's probably better to either use go run tool@version or to have a separate tools.mod file, then go run -modfile=tools.mod tool.

[mtibben]

Yep that makes a lot of sense @jayconrod

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[mtibben]

@jayconrod Did you want to write up the tool directive approach that we could incorporate as an option into this proposal? I'm happy to collaborate on it with you. Positive feedback on that approach so far in this thread, and it would be good to compare the options directly against each other, now that this proposal will be considered by the go-powers-that-be

[GoVeronicaGo]

@matloob @samthanawalla to confirm the code has been completed and only release notes are pending for 1.24.

[mtibben]

@dmitshur the Modules FAQ may also need to be updated

[gopherbot]

Change https://go.dev/cl/632555 mentions this issue: cmd/go: add tool meta-pattern to go help packages

[gopherbot]

Change https://go.dev/cl/632595 mentions this issue: _content/doc: document module tools

[gopherbot]

Change https://go.dev/cl/632556 mentions this issue: doc/next: introduce module tools

[gopherbot]

Change https://go.dev/cl/632596 mentions this issue: Modules: add go1.24 tools

[mknyszek]

The RC is planned for next week, and we need a full draft of the release notes before then. Please prioritize writing the release notes for this. Thanks!

[thepudds]

Using -modfile when getting a tool with the -tool flag seems to work:

$ go get -modfile=tool.mod -tool golang.org/x/tools/cmd/stringer
go: downloading golang.org/x/tools v0.28.0
[...]

But using -modfile when invoking a tool seems to not work currently:

$ go tool -modfile=tool.mod stringer
flag provided but not defined: -modfile

I don't know if that was a deliberate choice (including I didn't notice -modfile referenced in the design doc), but it would be nice if -modfile was supported when invoking a tool.

Checks above were using tip from earlier today (c46ba1f), and it might be I misunderstood the new capabilities or otherwise made a mistake.

[ConradIrwin]

@thepudds This will be fixed by https://go-review.googlesource.com/c/go/+/632575

[fredrikaverpil]

Is the intention that this should work?

$ gotip mod -modfile=tools.mod init foo
go mod: unknown command
Run 'go help mod' for usage

I'm on gotip version: go version devel go1.24-c46ba1f Wed Dec 4 22:20:08 2024 +0000 darwin/arm64

[thepudds]

Hi @ConradIrwin, great, thanks! 🚀🎉

@fredrikaverpil, if you want to use -modfile with go mod init, I think the proper spelling is with the -modfile flag after the init:

$ go mod init -modfile=tool.mod foo

[gopherbot]

Change https://go.dev/cl/632575 mentions this issue: cmd/go: add -modfile and -modcacherw to go tool

[ConradIrwin]

Now this is out in RC, I've put a blog post explaining some of the motivation for the change here: https://cirw.in/blog/go-tools

[mcandre]

For clarification:

• Where do the tool executables live, $GOPATH/bin?
• Do you have to invoke the tools with a go tool prefix, like npx and bundle exec, or nah?
• I assume that the same version identifier syntaxes, including VCS tags as well as commit ID replace directives, work for tools as they do for non-tool Go modules, right?

[meling]

Quick reply with this blog post. It doesn't answer all your questions, but shows (at the bottom of the post) that you can install tools both globally and locally for the current module.

[ConradIrwin]

@mcandre

• tool executables live in the build cache, and are built on demand (and expired when unused). This lets go tool select which version of a tool is used depending on the active module.
• you invoke them with go tool <toolname>. If you want to (and versioning is not a concern) you can install all tools for the current module to GOBIN with go install tool.
• Yes. Tool dependencies participate in the same module graph as imported dependencies, and require/replace/exclude directives apply in the same way.

@meling nice post! You might want to recommend go get -tool instead of go mod edit followed by go get -u tool` save people a command or two.

[gopherbot]

Change https://go.dev/cl/638296 mentions this issue: _content/doc/go1.24: fix meaning of -u in 'go get -u tool'