-
Notifications
You must be signed in to change notification settings - Fork 17.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
cmd/go: offer a consistent "global install" command #30515
Comments
CC @jayconrod |
cc @ianthehat given the recent golang-tools related conversations too. |
I think we should repurpose |
I agree that repurposing |
I think it is fairly easy to make a case for The harder things are at the edges, things like are a way of doing that work and then running the resulting binary (had to do will with go run, and I am not sure we should even try), or wether the behaviour of go install when not given a version should be changed to avoid any confusion. |
From what I have observed as well as for my personal usage, this would be very useful regardless of how it is spelled ( If it is implemented, I think it would be important to use the remote |
At this point, personally I would vote for Regarding this:
If A related question -- should |
I also support go install being for global binary installs to gopath and go get only for use inside a module path. Modules brings some breaking tooling changes and it seems like making some more here is the time to do it and worth the cost since this makes the tooling much easier to understand. |
I support this suggestion. One question: would this re-purposed To take one example, it wouldn't be possible to build a correctly working executable for |
I think the results should be identical to as if you had checked out the module at the supplied version to a temporary directory, changed to that directory and done |
Something that we might want to be careful about is |
Yes, this is one of the reasons why I am dubious about making any changes to non versioned forms, and thus don't agree that an implied @latest is the right choice. |
@mvdan I suspect I am missing something obvious… But could the rule be to use the |
Trying to distill the discussion so far into a proposal: We would change the behavior of We change behavior if:
If the conditions above are true,
Examples:
|
I think a key question is whether I'd argue the form with no version should be consistent with other module-related commands. For a new user, it would be strange to accidentally run the former command instead of the latter, then see it take your build constraints into account and possibly modify your |
I think |
@rogpeppe @ianthehat The trouble with applying That means, for example, that we wouldn't be able to fetch the module from a proxy. The same consideration holds for We could, at least in theory, apply the subset of replacements that specify module paths rather than filesystem paths. However, that would produce a third variation on the build configuration: one that doesn't necessarily match the clean-checkout configuration (because it doesn't apply filesystem replacements), but doesn't necessarily match the clean-external-build configuration either (because it does apply non-filesystem replacements). As another alternative, we could apply module replacements and emit an error of the module has any filesystem replacements. That would produce the same result as a build within the module, but at the cost of rejecting some otherwise-well-formed modules. I don't think that the benefit of enabling temporary fixes and workarounds offsets the complexity of adding either of those modes of operation. Users have enough trouble understanding module-mode behavior as it is. It seems much simpler to require that modules build successfully using unmodified dependencies, even if that results in a bit of extra work for the maintainers of large modules to upstream their modifications or maintain a complete fork. |
As I said before, the rule should be the tool is built exactly as if you had fetched it from a proxy, extracted it to a temporary directory, changed to the directory, and typed go install. I think possibly we should also specify the readonly flag, but that's a separate discussion. |
"Repurposing" |
Half-applying |
That's fair enough, but aren't This is where the idea of slightly changing the meaning of one of them comes from, to keep both commands useful. We could always add a flag like |
@bcmills and I talked a bit about various "auto-detect" kind of ways to shove this into the go command and didn't come up with anything palatable. I could see adding a short flag to
|
@mvdan Whether they are practically the same doesn't really matter. What matters is that they have established semantics that can't just be redefined. |
This is what I'd imagine.
Wouldn't the same have applied to |
Prevent `go get` from touching `go.mod` and `go.sum` when executing global binaries during the build process. Once golang/go#30515 is fixed, we should is whatever solution is provided there. Fixes: #12010 Co-authored-by: techknowlogick <[email protected]>
This seems correct to me as is. If I import a package that uses replace statements as a dependency, then failed tests might help alert me to this problem and I may reconsider the import. I certainly wouldn't ever want replace statements from another module to apply in one i'm using. This is a core principal of replace directives. I just categorize installing a global binary as being equivalent to building inside that binaries module.
This is a good point and I agree with it. I will say that many prefer pure go over cgo for this reason and more. Also why open the door for less reproducibility just because its not perfect now? Looking forward to proposal and summary. Thanks all. |
A few quick additional details that probably should be covered in an actual proposal (with one answer or another):
In any event, I'm not sure those are the best answers, and probably better to view those more as details to iron out in the proposed behavior... |
Prevent `go get` from touching `go.mod` and `go.sum` when executing global binaries during the build process. Once golang/go#30515 is fixed, we should is whatever solution is provided there. Fixes: go-gitea#12010 Co-authored-by: techknowlogick <[email protected]>
Prevent `go get` from touching `go.mod` and `go.sum` when executing global binaries during the build process. Once golang/go#30515 is fixed, we should is whatever solution is provided there. Fixes: #12010 Co-authored-by: techknowlogick <[email protected]> Co-authored-by: silverwind <[email protected]> Co-authored-by: techknowlogick <[email protected]>
Change https://golang.org/cl/243077 mentions this issue: |
Hey everyone, I've opened a new proposal issue, #40276, with a design doc at CL 243077. It'd be best if we could continue the discussion on that CL: Gerrit's threading and versioning will be more readable than a long GitHub issue. To concentrate discussion, I'd like to close and lock this issue. Before I do that, if you feel the proposal doesn't address an important use case and would like to continue this discussion separate from the proposal, please comment here. Also note, this will be on the agenda for the next golang-tools discussion on 2020-07-22 at 15:30 UTC, so feel free to join in then as well. |
I helped Jay with the new proposal above, so I agree that we should close this issue in favor of that newer one. This isue was more of a problem statement than a proposal for a specific solution that can be implemented. |
Prevent `go get` from touching `go.mod` and `go.sum` when executing global binaries during the build process. Once golang/go#30515 is fixed, we should is whatever solution is provided there. Fixes: go-gitea#12010 Co-authored-by: techknowlogick <[email protected]>
Modules prevent the installation of executables using the standard `go get` syntax. Using `GOMODULE111=off` should fix this until [#40276](golang/go#40276) is implemented. See also: [#27643](golang/go#27643) [#30515](golang/go#30515) [#171660042](https://www.pivotaltracker.com/story/show/171660042) Co-authored-by: Sebastian Vidrio <[email protected]>
Modules prevent the installation of executables using the standard `go get` syntax. Using `GOMODULE111=off` should fix this until [#40276](golang/go#40276) is implemented. See also: [#27643](golang/go#27643) [#30515](golang/go#30515) [#171660042](https://www.pivotaltracker.com/story/show/171660042) Co-authored-by: Sebastian Vidrio <[email protected]>
>>> Go Executable Installation When installing a Go executable from within a Go module [1] directory using the `go install` command [2], it is installed into the Go executable search path that is defined through the `GOBIN` environment variable [3] and can also be shown and modified using the `go env` command [4]. Even though the executable gets installed globally, the `go.mod` file [5] will be updated to include the installed packages since this is the default behavior of the `go get` command [6] running in "module mode" [7]. Next to this problem, the installed executable will also overwrite any executable of the same module/package that was installed already, but maybe from a different version. Therefore only one version of a executable can be installed at a time which makes it impossible to work on different projects that use the same tool but with different versions. >>>> History & Future The local installation of executables built from Go modules/packages has always been a somewhat controversial point which unfortunately, partly for historical reasons, does not offer an optimal and user-friendly solution up to now. The `go` command [8] is a fantastic toolchain that provides many great features one would expect to be provided out-of-the-box from a modern and well designed programming language without the requirement to use a third-party solution: from compiling code, running unit/integration/benchmark tests, quality and error analysis, debugging utilities and many more. Unfortunately the way the `go install` command [2] of Go versions less or equal to 1.15 handles the installation of an Go module/package executable is still not optimal. The general problem of tool dependencies is a long-time known issue/weak point of the current Go toolchain and is a highly rated change request from the Go community with discussions like golang/go#30515 [9], golang/go#25922 [10] and golang/go#27653 [11] to improve this essential feature, but they've been around for quite a long time without a solution that works without introducing breaking changes and most users and the Go team agree on. Luckily, this topic was finally picked up for the next upcoming Go release version 1.16 [12] and golang/go#40276 [13] introduces a way to install executables in module mode outside a module. The release note preview also already includes details about this change [14] and how installation of executables from Go modules will be handled in the future. >>>> The Workaround Beside the great news and anticipation about an official solution for the problem the usage of a workaround is almost inevitable until Go 1.16 is finally released. The official Go wiki [15] provides a section on "How can I track tool dependencies for a module?" [16] that describes a workaround that tracks tool dependencies. It allows to use the Go module logic by using a file like `tools.go` with a dedicated `tools` build tag that prevents the included module dependencies to be picked up included for normal executable builds. This approach works fine for non-main packages, but CLI tools that are only implemented in the `main` package can not be imported in such a file. In order to tackle this problem, a user from the community created "gobin" [17], an experimental, module-aware command to install/run main packages. It allows to install or run main-package commands without "polluting" the `go.mod` file by default. It downloads modules in version-aware mode into a binary cache path within the systems cache directory [18]. It prevents problems due to already globally installed executables by placing each version in its own directory. The decision to use a cache directory instead of sub-directories within the `GOBIN` path keeps the system clean. "gobin" is still in an early development state, but has already received a lot of positive feedback and is used in many projects. There are also members of the core Go team that have contributed to the project and the chance is high that the changes for Go 1.16 were influenced or partially ported from it. It is currently the best workaround to... 1. prevent the Go toolchain to pick up the `GOMOD` environment variable [4] (see [`go env GOMOD`][4]) that is initialized automatically with the path to the `go.mod` file in the current working directory. 2. install module/package executables globally without "polluting" the `go.mod` file. 3. install module/package executables globally without overriding already installed executables of different versions. See gobin's FAQ page [19] in the repository wiki for more details about the project. >>>> The Go Module Caster To allow to manage the tool dependency problem, wand now uses "gobin" through a new caster [20] `go.mod` file and allows to... 1. install `gobin` itself into `GOBIN` (`go env GOBIN` [4]). 2. cast any spell incantation [21] of kind `KindGoModule` [22] by installing the executable globally into the dedicated `gobin` cache. [1]: https://golang.org/ref/mod [10]: golang/go#25922 [11]: golang/go#27653 [12]: https://github.com/golang/go/milestone/145 [13]: golang/go#40276 [14]: https://tip.golang.org/doc/go1.16#modules [15]: https://github.com/golang/go/wiki [16]: https://github.com/golang/go/wiki/Modules#how-can-i-track-tool-dependencies-for-a-module [17]: https://github.com/myitcv/gobin [18]: https://golang.org/pkg/os/#UserCacheDir [19]: https://github.com/myitcv/gobin/wiki/FAQ [2]: https://golang.org/cmd/go#hdr-Compile_and_install_packages_and_dependencies [20]: https://pkg.go.dev/github.com/svengreb/wand/pkg/cast/gobin#Caster [21]: https://pkg.go.dev/github.com/svengreb/wand/pkg/spell#Incantation [22]: https://pkg.go.dev/github.com/svengreb/wand/pkg/spell#KindGoModule [3]: https://golang.org/cmd/go/#hdr-Environment_variables [4]: https://golang.org/cmd/go/#hdr-Print_Go_environment_information [5]: https://golang.org/ref/mod#go-mod-file [6]: https://golang.org/cmd/go/#hdr-Add_dependencies_to_current_module_and_install_them [7]: https://golang.org/ref/mod#mod-commands [8]: https://golang.org/cmd/go [9]: golang/go#30515 GH-22
>>> Go Executable Installation When installing a Go executable from within a Go module [1] directory using the `go install` command [2], it is installed into the Go executable search path that is defined through the `GOBIN` environment variable [3] and can also be shown and modified using the `go env` command [4]. Even though the executable gets installed globally, the `go.mod` file [5] will be updated to include the installed packages since this is the default behavior of the `go get` command [6] running in "module mode" [7]. Next to this problem, the installed executable will also overwrite any executable of the same module/package that was installed already, but maybe from a different version. Therefore only one version of a executable can be installed at a time which makes it impossible to work on different projects that use the same tool but with different versions. >>>> History & Future The local installation of executables built from Go modules/packages has always been a somewhat controversial point which unfortunately, partly for historical reasons, does not offer an optimal and user-friendly solution up to now. The `go` command [8] is a fantastic toolchain that provides many great features one would expect to be provided out-of-the-box from a modern and well designed programming language without the requirement to use a third-party solution: from compiling code, running unit/integration/benchmark tests, quality and error analysis, debugging utilities and many more. Unfortunately the way the `go install` command [2] of Go versions less or equal to 1.15 handles the installation of an Go module/package executable is still not optimal. The general problem of tool dependencies is a long-time known issue/weak point of the current Go toolchain and is a highly rated change request from the Go community with discussions like golang/go#30515 [9], golang/go#25922 [10] and golang/go#27653 [11] to improve this essential feature, but they've been around for quite a long time without a solution that works without introducing breaking changes and most users and the Go team agree on. Luckily, this topic was finally picked up for the next upcoming Go release version 1.16 [12] and golang/go#40276 [13] introduces a way to install executables in module mode outside a module. The release note preview also already includes details about this change [14] and how installation of executables from Go modules will be handled in the future. >>>> The Workaround Beside the great news and anticipation about an official solution for the problem the usage of a workaround is almost inevitable until Go 1.16 is finally released. The official Go wiki [15] provides a section on "How can I track tool dependencies for a module?" [16] that describes a workaround that tracks tool dependencies. It allows to use the Go module logic by using a file like `tools.go` with a dedicated `tools` build tag that prevents the included module dependencies to be picked up included for normal executable builds. This approach works fine for non-main packages, but CLI tools that are only implemented in the `main` package can not be imported in such a file. In order to tackle this problem, a user from the community created "gobin" [17], an experimental, module-aware command to install/run main packages. It allows to install or run main-package commands without "polluting" the `go.mod` file by default. It downloads modules in version-aware mode into a binary cache path within the systems cache directory [18]. It prevents problems due to already globally installed executables by placing each version in its own directory. The decision to use a cache directory instead of sub-directories within the `GOBIN` path keeps the system clean. "gobin" is still in an early development state, but has already received a lot of positive feedback and is used in many projects. There are also members of the core Go team that have contributed to the project and the chance is high that the changes for Go 1.16 were influenced or partially ported from it. It is currently the best workaround to... 1. prevent the Go toolchain to pick up the `GOMOD` environment variable [4] (see [`go env GOMOD`][4]) that is initialized automatically with the path to the `go.mod` file in the current working directory. 2. install module/package executables globally without "polluting" the `go.mod` file. 3. install module/package executables globally without overriding already installed executables of different versions. See gobin's FAQ page [19] in the repository wiki for more details about the project. >>>> The Go Module Caster To allow to manage the tool dependency problem, wand now uses "gobin" through a new caster [20] `go.mod` file and allows to... 1. install `gobin` itself into `GOBIN` (`go env GOBIN` [4]). 2. cast any spell incantation [21] of kind `KindGoModule` [22] by installing the executable globally into the dedicated `gobin` cache. [1]: https://golang.org/ref/mod [10]: golang/go#25922 [11]: golang/go#27653 [12]: https://github.com/golang/go/milestone/145 [13]: golang/go#40276 [14]: https://tip.golang.org/doc/go1.16#modules [15]: https://github.com/golang/go/wiki [16]: https://github.com/golang/go/wiki/Modules#how-can-i-track-tool-dependencies-for-a-module [17]: https://github.com/myitcv/gobin [18]: https://golang.org/pkg/os/#UserCacheDir [19]: https://github.com/myitcv/gobin/wiki/FAQ [2]: https://golang.org/cmd/go#hdr-Compile_and_install_packages_and_dependencies [20]: https://pkg.go.dev/github.com/svengreb/wand/pkg/cast/gobin#Caster [21]: https://pkg.go.dev/github.com/svengreb/wand/pkg/spell#Incantation [22]: https://pkg.go.dev/github.com/svengreb/wand/pkg/spell#KindGoModule [3]: https://golang.org/cmd/go/#hdr-Environment_variables [4]: https://golang.org/cmd/go/#hdr-Print_Go_environment_information [5]: https://golang.org/ref/mod#go-mod-file [6]: https://golang.org/cmd/go/#hdr-Add_dependencies_to_current_module_and_install_them [7]: https://golang.org/ref/mod#mod-commands [8]: https://golang.org/cmd/go [9]: golang/go#30515 Closes GH-22
Draft design for 'go install' functionality to install executables in module mode outside a module at a specific version. For golang/go#30515 For golang/go#40276 Change-Id: Iaec50eeae92148be51e6acb1d3a9488bb2587bc8 Reviewed-on: https://go-review.googlesource.com/c/proposal/+/243077 Trust: Jay Conrod <[email protected]> Reviewed-by: Jay Conrod <[email protected]>
As of 1.12, one can run the following from outside a Go module:
The same mechanism can be used to download a specific version instead of
@latest
, such as@v1.2.3
.We can even emulate very similar behavior in Go 1.11, with a one-liner like:
1.13 will likely make
GO111MODULE=on
a default, so it's likely that project READMEs will be able to justs tell users to rungo get foo.com/cmd/bar
on Go 1.13 and later.However, this has a problem - if the user runs the command from within a Go module, the command will add clutter to the module's
go.mod
andgo.sum
files. The binary will also be installed as usual, so the user might not even notice the unintended consequences until much later.This is a fairly common point of confusion among Go developers, particularly those new to modules. Now that most Go projects are modules, the chances of one running cmd/go commands within a module are quite high.
What we need is a "global install" command, which will ignore the current module. For example, imagine a
go get -global
, to ensure backwards compatibility. Or perhaps even repurposinggo install
to always mean global installs, sincego get
can be used for non-global ones.I think we should make a decision and implement it before the 1.13 release, so that READMEs can finally drop the problematic
go get -u foo.com/cmd/bar
line. We can almost do it now, minus this confusing edge case with$PWD
./cc @bcmills @rsc @myitcv @rogpeppe @thepudds
The text was updated successfully, but these errors were encountered: