cmd/go: updating go mod edit -replaced version is a nightmare

[tmm1]

What version of Go are you using (go version)?

$ go version
go version go1.16.2 darwin/amd64

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE="auto"
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/tmm1/Library/Caches/go-build"
GOENV="/Users/tmm1/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/tmm1/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/tmm1/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/Cellar/go/1.16.2/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.16.2/libexec/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.16.2"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/tmp/gotest/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/_2/hljyy_zj3912lv9qqpy70t5w0000gn/T/go-build2252094498=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Often I am using an open-source project, until I reach a point where I need to fork it.

Say for example I am using github.com/gin-gonic/gin very simply:

$ cat go.mod
module github.com/tmm1/gotest

go 1.16

require github.com/gin-gonic/gin v1.6.3

$ cat main.go
package main

import (
	_ "github.com/gin-gonic/gin"
)

func main() {
}

Now I need to make a change, so I fork a copy over to github.com/fancybits/gin and create a new branch fancybits. Next I use go mod edit -replace to swap over to my fork:

$ go mod edit -replace github.com/gin-gonic/gin=github.com/fancybits/gin@fancybits
$ git diff
--- a/go.mod
+++ b/go.mod
@@ -3,3 +3,5 @@ module github.com/tmm1/gotest
 go 1.16

 require github.com/gin-gonic/gin v1.6.3
+
+replace github.com/gin-gonic/gin => github.com/fancybits/gin fancybits

$ go mod tidy
$ git diff
diff --git a/go.mod b/go.mod
index 68c3950..c621608 100644
--- a/go.mod
+++ b/go.mod
@@ -3,3 +3,5 @@ module github.com/tmm1/gotest
 go 1.16

 require github.com/gin-gonic/gin v1.6.3
+
+replace github.com/gin-gonic/gin => github.com/fancybits/gin v1.6.3-0.20210107060700-dd6fd641b2f2

This is great and works as expected.

Next I start making changes to my fork. I make a new commit on the fancybits branch and push it up to github.

Now I just want to update my go.mod to the new commit I just pushed, but I cannot figure out how. If I use go get -u, it yells at me:

$ go get -u github.com/fancybits/gin@fancybits
go get: github.com/fancybits/[email protected] updating to
	github.com/fancybits/[email protected]: parsing go.mod:
	module declares its path as: github.com/gin-gonic/gin
	        but was required as: github.com/fancybits/gin

If I try go mod edit -replace again with the branch name, running go mod tidy again just uses the old cached resolution of that branch to the old commit.

If I try to use the actual commit sha instead of the branch name, it fetches the new commit information but then still fails and yells at me:

$ go get github.com/fancybits/gin@5ac6c37c50ff33bfaf8b492ef1cac02ac0232b88
go: downloading github.com/fancybits/gin v1.6.3-0.20210407001758-5ac6c37c50ff
go get: github.com/fancybits/[email protected] updating to
	github.com/fancybits/[email protected]: parsing go.mod:
	module declares its path as: github.com/gin-gonic/gin
	        but was required as: github.com/fancybits/gin

At this point I hoped maybe since it downloaded the new commit above, it would now have updated the cache. So I tried to run go mod edit -replace github.com/gin-gonic/gin=github.com/fancybits/gin@fancybits && go mod tidy but it didn't update my go.mod file.

I also tried go mod edit -dropreplace and then adding the module back with -replace but the old commit information is still cached somewhere.

The only way I could make it work was to edit the go.mod file manually and replace the version information using what's printed in the error message from go get -u github.com/fancybits/gin@fancybits.

Please make this easier.

[seankhliao]

1. set GOPRIVATE=github.com/fancybits/gin if you will be repeatedly pushing/pulling new commits, this will avoid the cached results
2. as you correctly inferred, go mod edit -replace github.com/gin-gonic/gin=github.com/fancybits/gin@fancybits && go mod tidy. (It didn't work for you because it was using the cached result of fancybits -> oldcommit, pulling in a new commit but without the association that the branch points to it doesn't help)

go get is for adding new dependencies, so it thinks you want to add a module called github.com/fancybits/gin

For questions please refer to https://github.com/golang/go/wiki/Questions

[bcmills]

#26904 is probably a better long-term approach for targeting a long-running fork: it would allow MVS to increase the replacement version, rather than hard-coding a specific version.

#32721 proposes another alternative, still under consideration.

[dkegel-fastly]

This causes me grief every time I need to update my fancybits branch.

I gather the idiom for pointing to a fork on a branch is:

1. Make sure the replacement branch has the new module path in go.mod
2. In the consuming project's go.mod, add a line "replace oldmod => newmod newbranch", or do go mod edit -replace oldmod=newmod@newbranch
3. Do go mod tidy to replace "newmod newbranch" with "newmod fancystrangeidentifierfornewbranchcommit"

(It is awkward that the first method for step 2 forbids the @, but the second method requires it.)

Where is all this clearly documented? https://golang.org/ref/mod#go-mod-file-replace probably should, but doesn't.