proposal: switch to using nested modules for -m mode resolution

[myitcv]

Building on the ideas started in #44.

Currently, the -m flag tells gobin to use the main module for resolution of dependencies. -m can also, therefore, be thought of as "non-global" mode. In -m mode, we use the guidance at golang/go#25922 (comment) for keeping track of the tools we require in a build-ignored tools.go file.

Under this behaviour however, -m mode ends up cluttering our main module.

#44 proposed an alternative means of capturing the list of tools we require in a go.tools file. This issue more precisely specifies the behaviour as follows:

go.tools will capture the import paths of tools we require, e.g.:

# go.tools
golang.org/x/tools/cmd/stringer
example.com/blah/v2

gobin will then use synthetic nested modules (in a .gobin directory) per main package for module resolution (including the main package's module). For example:

// $PWD/.gobin/golang.org/x/tools/cmd/stringer/go.mod
module mod

require (
    golang.org/x/tools v0.0.0-20190826060629-95c3470cfb70
)

Points to note:

• The filepath of nested modules will need to use the same escaping as seen in the cmd/go module cache
• All non-directory replace directives in the main package's main module will appear in the synthetic nested module for a main package

cc @rogpeppe @mvdan

[mvdan]

Do we need one go.mod file per tool dependency? For example, I see no value in different tools using different versions of x/tools/go/packages, and it's just going to make the builds slower and the setup more complex.

Going even further - why not add the go.mod/go.sum data right into go.tools? As a silly example, using txtar itself:

$ cat go.tools
golang.org/x/tools/cmd/stringer
example.com/blah/v2

-- go.mod --
module mod

require (
    golang.org/x/tools v0.0.0-20190826060629-95c3470cfb70
)
-- go.sum --
etc etc

[mvdan]

To clarify; my point about using a single go.tools file is to reduce the overhead that is adding more files or directories to the root of a module.

However, one disadvantage is that not having the plain files on disk somewhere would make it harder to use vanilla cmd/go to use or update the go.mod and go.sum files. But this is something that gobin could allow; for example:

$ gobin go mod graph
# extracts go.mod and go.sum into a temporary folder
# runs the cmd/go command in said folder (in this case 'go mod graph')
# saves go.mod and go.sum back into go.tools

[myitcv]

Do we need one go.mod file per tool dependency?

I would argue yes, and I suspect @rogpeppe would too.

Various reasons:

• brings versioned tool use within a project (via go.tools) inline with the global use, i.e. each tool is a distinct and separate entry point
• means tools' respective dependencies don't ever mix and influence each other, which ensures reproducibility over time, especially globally vs locally
• allows us to make more aggressive assertions about the install target. That was the intent of main: optimise "runtime" steps #80 but precisely because the tools' dependencies interact with each other (via the main module in this instance) we can't make these assertions
• allows use to share the same gobin cache target for a project's versioned (i.e. non main module and non directory-replaced) module packages, and global tools

Just to confirm, the "overhead" for modules requiring tools under this proposal would be:

• a go.tools
• a directory .gobin/ containing a tree of directories for each main package, each containing a go.{mod,sum} pair

Is your concern about the number of files in this .gobin directory?

[mvdan]

Gotcha, thanks. I hadn't thought about the side effects as much as you two have. Not convinced that this is a clear decision, but I see it's a tradeoff.

Is your concern about the number of files in this .gobin directory?

My concern is the number of files in the root of a module. Right now we already have go.mod and go.sum, and we've been thinking about go.tools for a while. .gobin/ (or whatever other directory) would be the fourth, which is already a non-trivial amount of clutter.

[myitcv]

Not convinced that this is a clear decision, but I see it's a tradeoff.

Agreed.

... which is already a non-trivial amount of clutter.

Indeed.

[rogpeppe]

note that you only need such a nested module for tools that don't contain a go.mod file. Hopefully that will become increasingly uncommon as time goes on.

[myitcv]

There are two reasons why I think we should still have a go.{mod,sum} per main package module:

• handles the case where the main package's module might not be tidy/complete
• allows us to bake version information into the target binary, which can then be revealed using go version -m /path/to/binary in Go 1.13. If we simply install from the main module we will only get a devel version identifier

[mvdan]

handles the case where the main package's module might not be tidy/complete

If we live in a world where proxy.golang.org is the standard for open source software, wouldn't (or couldn't) this already be covered?

If we simply install from the main module we will only get a devel version identifier

Are you positive about this? I understand why a go build ./some/tool doesn't contain version info at the moment, as it does no module version resolution. But surely a go install some-tool would contain the version info, as it would be pulled from the main go.mod.

[myitcv]

handles the case where the main package's module might not be tidy/complete

If we live in a world where proxy.golang.org is the standard for open source software, wouldn't (or couldn't) this already be covered?

I don't think so. Assume either the main module or a (transitive) dependency has incomplete dependencies. At time t1 missing dependency d1 resolves to v1.0.0 of; but some time later at t2, after v1.1.0 has been released, it resolves to v1.1.0. We don't have reproducible builds this way, and that's something I think we want and need.

If we simply install from the main module we will only get a devel version identifier

Are you positive about this? I understand why a go build ./some/tool doesn't contain version info at the moment, as it does no module version resolution. But surely a go install some-tool would contain the version info, as it would be pulled from the main go.mod.

But the go.mod itself has no version information. Perhaps I not quite getting what you're referring to; could you explain further with a cmd/go "equivalent" perhaps? i.e. the steps that would in effect be followed by gobin?