Problem with unstable go-sqlite3 version - 2.0.3

[Shriram-RP]

Hi,

We at the platform team of razorpay internally use bulk-insert which internally uses github.com/jinzhu/gorm v1.9.12 and creates an indirect dependency on github.com/mattn/go-sqlite3 v2.0.3.

Reference :- https://github.com/t-tiger/gorm-bulk-insert/blob/master/go.mod#L14

Now, this setup was working fine for us till last week and suddenly we had a failure in our docker build for github actions citing
github.com/mattn/[email protected]+incompatible: unknown revision v2.0.3

We also saw one of your issues where you mentioned versions 2.0+ were unstable - issue

Did you stop supporting 2.0.3 version suddenly in the last 7-10 days. We were wondering what caused the breakage of the build from our side. It would be helpful if you could throw some light on this issue.

Thanks,
Shriram

[rittneje]

What version of Go are you using? Also, what is the output of go env?

[ghzmhrm]

I have the same problem with @Shriram-RP, but everything is going well after changing the GOPROXY env from direct to proxy.golang.org

Before changing the GOPROXY to proxy.golang.org

foo@bar golang-dir % go mod tidy
go: github.com/mattn/[email protected]+incompatible: reading github.com/mattn/go-sqlite3/go.mod at revision v2.0.1: unknown revision v2.0.1
go: downloading github.com/mattn/go-sqlite3 v2.0.1+incompatible
go: github.com/mattn/[email protected]+incompatible: reading github.com/mattn/go-sqlite3/go.mod at revision v2.0.1: unknown revision v2.0.1

Changing the GOPROXY to proxy.golang.org

foo@bar golang-dir % export GOPROXY=https://proxy.golang.org

After changing the GOPROXY to proxy.golang.org

foo@bar golang-dir % go mod tidy
go: downloading github.com/mattn/go-sqlite3 v2.0.1+incompatible

[Shriram-RP]

What version of Go are you using? Also, what is the output of go env?

Version - 1.17

go env output :-

GO111MODULE=""
GOARCH="arm64"
GOBIN=""
GOCACHE="/Users/{name}/Library/Caches/go-build"
GOENV="/Users/{name}/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/{name}/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/{name}/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/opt/homebrew/Cellar/go/1.17/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/opt/homebrew/Cellar/go/1.17/libexec/pkg/tool/darwin_arm64"
GOVCS=""
GOVERSION="go1.17"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/{name}go/src/github.com/razorpay/reminders/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/qs/9hmssmm50p7djrwxsww6nfkm0000gq/T/go-build1013432823=/tmp/go-build -gno-record-gcc-switches -fno-common"

[shaswa]

One way to fix this is adding the below code to the go.mod file
exclude github.com/mattn/go-sqlite3 v2.0.3+incompatible

[ghost]

any fix ?

[mattn]

This is not an issue of go-sqlite3. v2 was an accident as written in README.md

NOTE: The increase to v2 was an accident. There were no major changes or features.

[ghost]

@mattn I wonder if this would be a reason to use a retraction and just retag v2: https://golang.org/ref/mod#go-mod-file-retract

The impact of this is a bit broad. Large projects seem to have been impacted by this as well as small random odds and ends. Notice that tekton/cli was affected when I was trying to import it into my project. Also searching my cache I see gorm managed to get a commit with a v2 tag.

Can't imagine what else this might have landed in. A retraction seems like it would prevent tons of people from having to stuff exclude and replace into their module files, if I understand the mechanics of it correctly.

---

[protosam@nullhost]$ go mod init placebo
go: creating new go.mod: module placebo
[protosam@nullhost]$ go get github.com/tektoncd/[email protected]
go get: github.com/tektoncd/[email protected] updating to
	github.com/tektoncd/[email protected] requires
	github.com/tektoncd/hub/[email protected] requires
	github.com/go-testfixtures/testfixtures/[email protected] requires
	github.com/mattn/[email protected]+incompatible: reading github.com/mattn/go-sqlite3/go.mod at revision v2.0.2: unknown revision v2.0.2

[protosam@nullhost]$ grep -ri 'mattn/go-sqlite3 v2' ~/go/pkg/mod/cache
~/go/pkg/mod/cache/download/github.com/go-testfixtures/testfixtures/v3/@v/v3.2.0.mod:	github.com/mattn/go-sqlite3 v2.0.2+incompatible
~/go/pkg/mod/cache/download/github.com/jinzhu/gorm/@v/v1.9.12.mod:	github.com/mattn/go-sqlite3 v2.0.1+incompatible

[mattn]

If once you clean module cache on your environment, vertion 1.14 shold be used in next time.

[ghost]

That's not the case. People will be stumbling onto this for a while. go-sqlite3 is a very wide spread dependency and gets added into other projects during both manual and automated tasks.

Commits like this and this are actively referenced in projects today.

There are 39,000 files that come up with I search to get an idea of how wide spread this is.

[rittneje]

@protosam From what I understand, retracting the v2 tags won't resolve any of the existing issues. It just stops the issue from continuing to grow. If I have a dependency on some other module that in turn has a dependency on one of the broken v2 tags of this library, then go mod will still attempt to download that version. And unless you are using proxy.golang.org (or similar), it will fail. The only time retraction matters is when someone is explicitly upgrading their dependencies.

In other words, even if a retraction is published, someone would still have to go to every one of the broken dependents and fix their go.mod to reference a valid release. Then all affected developers have to go and update to the newer versions of these modules.

@mattn I'm guessing the reason this just become an issue recently is because the v2 tags were deleted from this repo. Is that correct? If so, would you be able to re-create them for the same commits? That way people using "direct" at least won't get failed builds. (This is orthogonal to retracting the v2 releases.)

[mattn]

Anyone know which commit should i tag ?

[ghost]

@mattn I'm looking into this. I expect to find it in a go.sum file somewhere.

[ghost]

Google had them cached in sum.golang.org and also they're still listed at pkg.go.dev.

v2.0.0+incompatible
v2.0.1+incompatible
v2.0.2+incompatible
v2.0.3+incompatible

I'm actually a bit confused on the serialization. Saying what I got in case someone else is able to understand deserializing this faster than I can.

[mattn]

v2.0.0 was not a tag. It was branch name.

Last commit of v2.0.0 branch was

e904eb4

v2.0.1
https://web.archive.org/web/20201202062638/https://github.com/mattn/go-sqlite3/releases/tag/v2.0.1
590d44c

v2.0.2
https://web.archive.org/web/20201202062638/https://github.com/mattn/go-sqlite3/releases/tag/v2.0.2
53cff3f

v2.0.3
https://web.archive.org/web/20201202060439/https://github.com/mattn/go-sqlite3/releases/tag/v2.0.3
9bdaffc

v2.0.4
https://web.archive.org/web/20201202060440/https://github.com/mattn/go-sqlite3/releases/tag/v2.0.4
db4c942

v2.0.5
https://web.archive.org/web/20201202060440/https://github.com/mattn/go-sqlite3/releases/tag/v2.0.5
db4c942

v2.0.6
https://web.archive.org/web/20201202060441/https://github.com/mattn/go-sqlite3/releases/tag/v2.0.6
baaf8a9

[mattn]

Should all tags be restored?

[mattn]

I restored older v2.0.X tags. How goes good?

[mattn]

So you (who used v2.0.X) should modify go.mod to switch v1.4.X.

require github.com/mattn/go-sqlite3 v1.14.9

[ghost]

The thousands of forks and archive.org are pretty useful, these commits look good to me and below I've tested them. They work as I expect (they validated against sum.golang.org).

I did some additional testing to see how the retraction will work. I don't think it works against legacy versioning. Ideally if it worked, we would be capable of getting a message out to the end users so they know they're using bad versions and later these versions can be gracefully claimed for reuse.

No idea where to go with that discovery though. =/

So v2.0.0, v2.0.1, v2.0.2, and v2.0.3 work because the +incompatible tag and there is not a go.mod file (legacy versioning).

Tags v2.0.4, v2.0.5, and v2.0.6 do not seem to be needed. These have go.mod versioning and yield errors that people would have to do more work than necessary to work around. They also don't show up in sum.golang.org and are not listed on pkg.go.dev. Seems likely that no one would miss these tags. This leads me to believe they've not been in use.

---

Started with an empty project and tested getting a module that has a lower dependent that requires v2.0.2.

[protosam@nullhost]$ go mod init placebo
go: creating new go.mod: module placebo

[protosam@nullhost]$ go get github.com/tektoncd/[email protected]
go get: added github.com/tektoncd/cli v0.21.0

[protosam@nullhost]$ go mod verify
all modules verified

Below I tested against v2.0.0 - v2.0.6

[protosam@nullhost]$ go mod download github.com/mattn/[email protected]+incompatible
[protosam@nullhost]$ go mod tidy

[protosam@nullhost]$ go get github.com/mattn/[email protected]+incompatible
go get: upgraded github.com/mattn/go-sqlite3 v1.14.9 => v2.0.0+incompatible

[protosam@nullhost]$ go get github.com/mattn/[email protected]+incompatible
go get: upgraded github.com/mattn/go-sqlite3 v2.0.0+incompatible => v2.0.1+incompatible

[protosam@nullhost]$ go get github.com/mattn/[email protected]+incompatible
go get: upgraded github.com/mattn/go-sqlite3 v2.0.1+incompatible => v2.0.2+incompatible

[protosam@nullhost]$ go get github.com/mattn/[email protected]+incompatible
go: downloading github.com/mattn/go-sqlite3 v2.0.3+incompatible
go get: upgraded github.com/mattn/go-sqlite3 v2.0.2+incompatible => v2.0.3+incompatible

[protosam@nullhost]$ go get github.com/mattn/[email protected]+incompatible
go get: github.com/mattn/[email protected]+incompatible: invalid version: +incompatible suffix not allowed: module contains a go.mod file, so semantic import versioning is required

[protosam@nullhost]$ go get github.com/mattn/[email protected]
go get: github.com/mattn/[email protected]: invalid version: module contains a go.mod file, so major version must be compatible: should be v0 or v1, not v2

[protosam@nullhost]$ go get github.com/mattn/[email protected]
go get: github.com/mattn/[email protected]: invalid version: module contains a go.mod file, so major version must be compatible: should be v0 or v1, not v2

[protosam@nullhost]$ go get github.com/mattn/[email protected]
go get: github.com/mattn/[email protected]: invalid version: module contains a go.mod file, so major version must be compatible: should be v0 or v1, not v2

[sagikazarmark]

@mattn I'd probably pin this issue while it's affecting people otherwise they are going to open new ones as they rarely look at slightly older issues.

I'd probably reach out to the Go team at this point to see if they have any ideas to stop the spread and fix the problem (eg. remove these entries from sumdb? Dunno if that's possible/good idea)

Retracting would probably mean that you can't use these tags in the future.

[mattn]

This issue is already reported to Go team. golang/go#35732

[sagikazarmark]

@mattn as far as I can tell that issue was closed with working as intended. I was rather suggesting to contact them about this specific case, hoping that they could remove the offending entries from the sumdb/proxy.

[itmecho]

One way to fix this is adding the below code to the go.mod file exclude github.com/mattn/go-sqlite3 v2.0.3+incompatible

Thank you! This fixed my problem. When I ran go get github.com/mattn/[email protected], it downgraded another library I'm using (ory/fosite) from 0.40.2 to 0.35.1. I then added this line to my go.mod and ran go get github.com/ory/[email protected]. This upgraded the package without upgrading go-sqlite3 to the bad version.

[kvtb]

I restored older v2.0.X tags. How goes good?

The tags are disappeared again :(

[mattn]

The 2.0.X was retracted.

https://pkg.go.dev/github.com/mattn/go-sqlite3?tab=versions

You may have already had go-sqlite3 version 2.0.X. Please update manually to 1.4.X.

[rittneje]

@mattn Even though you retracted those versions, those tags should never be deleted, so that people can continue to download their dependencies, including for older versions of their projects.

[mattn]

@rittneje Yes. My understanding is that once people manually revert to 1.4.X, they will not be upgraded thereafter since 2.X was retracted. I believe this is a problem that cannot be recovered automatically.

[rittneje]

My point is that deleting those v2 tags breaks people's builds so we should not do that. We can only strongly discourage their use going forward (which the retraction helps).

[mattn]

I have no plans to release v2 any more.

[rittneje]

But those tags were already released. So there are other libraries and projects that reference them. By deleting them, those versions cannot be built properly anymore (without using a Go module proxy), which was the original stated issue. We can tell people not to use v2 in new versions of their code anymore, but we should not cause the older/existing versions to break.

[mattn]

v2 was an accidental release because the Go mod incorrectly identified the branch name as a tag. #998 is a PR to add retract as a way to fix these problems.

[rittneje]

Retract doesn't really "fix" the problem per se. In fact, the problem is not really fixable in the way you are hoping. Once a version has been released, it should not be deleted. The retraction fixes some latent issues with fetching/upgrading via go get github.com/mattn/go-sqlite3@latest. However, if a project already references, say, v2.0.3 in its go.sum file, the retraction does not (and should not) interfere with that.

Typically people are using the default Go module proxy at proxy.golang.org. By design, this proxy does not allow versions to be deleted, so that you have eternal build reproducibility. This is an important feature for most people, as they expect all previous versions of their code to always work.

However, in the event that you are not using a proxy (GOPROXY=direct) then it instead relies on the GitHub tag still existing. It is for this reason that the tags of this repo should be preserved.

My understanding is there were two issues in the history of this repo. The first is the existence of a branch named v2.0.0, which, due to an unfortunate bug in go mod, was misinterpreted as a release version. The second is a bunch of actual tags (v2.0.1, v2.0.2, etc.) that were released in error.

[mattn]

Yes, so I added note in README. https://github.com/mattn/go-sqlite3#go-sqlite3

I have not released v2 in other projects because I do not break compatibility. I also don't want to make it a hassle for the user. Fortunately, as far as I can see, many people still have v1. Because the bump to v2 is not automatic. So I decided to save the v1 users instead of the v2 users. Do you still think we should release v2.1.X or v3?

BTW, go get with fresh go.mod always get 1.4.X.

[rittneje]

Do you still think we should release v2.1.X or v3?

I think you should continue releasing v1.X.Y, as you have been. (In the event you do decide to make breaking changes, going to v3 is probably best to have a clean break from v2.) However, the v2 tags that were already published before should continue to exist indefinitely, you just won't be releasing any new v2.X.Y tags.

BTW, go get with fresh go.mod always get 1.4.X.

Yes this is due to fixes in go mod itself as well as the retraction you did. But if you specifically mention v2.0.3 in your go.mod then it works despite the retraction.

[mattn]

I figured that leaving the v2 tag could be confusing to new users.

Yes this is due to fixes in go mod itself as well as the retraction you did. But if you specifically mention v2.0.3 in your go.mod then it works despite the retraction.

Yes, it's same with my understanding. Thanks. BTW, I don't touch v2 tags recently.

[kvtb]

The 2.0.X was retracted.

https://pkg.go.dev/github.com/mattn/go-sqlite3?tab=versions

You may have already had go-sqlite3 version 2.0.X. Please update manually to 1.4.X.

Indeed, I can update my code, but there are tons of 3rd party libs which require go-sqlite v2.x.x
Should I fork them all ?

[kvtb]

I see that v2 was released by mistake and having legacy v2 and ongoing v1 is confusing.
But it seems that in order to fix that mistake you should keep v1 and v2 frozen and continue development as v3

[ntsd]

I got the same problem but after I did go env -w GOPROXY="https://proxy.golang.org,direct" now it can download.

[drgrib]

@mattn Deleting the v2.0.3 broke code that I hadn't needed to touch for years. Now when I try to use v1.14.9 I get this error for that code that worked fine before:

Binary was compiled with 'CGO_ENABLED=0', go-sqlite3 requires cgo to work. This is a stub

None of the logic I wrote works anymore and it used to. I just get empty values when I query now. How do I fix this?
https://github.com/drgrib/alfred-bear/blob/master/db/db.go#L172

I have CGO_ENABLED=1 in my .bash_profile. Can we get clearer instructions on how to fix #855? The README is not helpful, at least for someone on a Mac.

[rittneje]

@drgrib Are you getting an error, or an empty result set? How specifically are you compiling and executing your code? Have you tried with the latest release (1.14.16)? Also, what does go env say?

#855 is not fixable as the Go toolchain does not support that mode of operation. You must compile with CGO_ENABLED=1 every time.

[drgrib]

@rittneje I guess I'm technically not getting an error, just that message when I run, along with empty result sets.

My bad too. I had CGO_ENABLED=1 in my .bash_profile rather than export CGO_ENABLED=1. It does work now. Thanks for following up.