cmd/cgo: type confusion between CGO and CoreFoundation types on Go1.10

[akalin-keybase]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

1.10

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/akalin-keybase/Library/Caches/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/akalin-keybase/go"
GORACE=""
GOROOT="/Users/akalin-keybase/homebrew/Cellar/go/1.10/libexec"
GOTMPDIR=""
GOTOOLDIR="/Users/akalin-keybase/homebrew/Cellar/go/1.10/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/f7/k3hkmgcx71d2n0vz0fp2662r0000gq/T/go-build917425438=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

If possible, provide a recipe for reproducing the error.
A complete runnable program is good.
A link on play.golang.org is best.

What did you expect to see?

The following file should compile:

// +build darwin

package kext

/*
#cgo LDFLAGS: -framework CoreFoundation -framework IOKit

#include <CoreFoundation/CoreFoundation.h>
*/
import "C"

func bugRepro() {
	arr := returnsCFArrayRef()
	// If arr is replaced with C.CFArrayRef(arr), compilation succeeds.
	C.CFArrayCreateCopy(nil, arr)
}

returnsCFArrayRef is a function in another file in the same package that looks like

func returnsCFArrayRef() C.CFArrayRef {
  // returns C.CFArrayRef(0) or C.CFArrayRef(nil) depending on go version.
}

A full example is on https://gist.github.com/akalin-keybase/6d075a61e683a4b2e2424638308e95c5 .

What did you see instead?

go install ./...
# github.com/keybase/go-kext
./file2.go:22: cannot use arr (type _Ctype_CFArrayRef) as type *_Ctype_struct___CFArray in argument to func literal

Compilation succeeds with go 1.9.2.

This is likely related to the go 1.10 changes that made some C types map to uintptr.

[bradfitz]

Did you read https://golang.org/doc/go1.10#cgo ? Starting at "Cgo now translates some C types".

[akalin-keybase]

Yes, that's not the problem here. The problem is that a variable ostensibly of type C.CFArrayRef (since it was assigned from a function that returns C.CFArrayRef) gives a compile error when passed to a function that expect a parameter of type C.CFArrayRef. This should work regardless of whether C.CFArrayRef is mapped to uintptr or unsafe.Pointer.

In fact, merely mentioning the C.CFArrayRef somewhere else in the file suffices to make compilation succeed, which seems to suggest that the original compilation error isn't the fault of the code itself.

[bradfitz]

/cc @randall77 @ianlancetaylor

[randall77]

Question: is there an import "C" (and #include directive?) in the file in which returnsCFArrayRef is defined? Does it fix things to add that import and/or include?

[akalin-keybase]

@randall77 yes, both import "C" and #include are already present; see https://gist.github.com/akalin-keybase/6d075a61e683a4b2e2424638308e95c5 ; the bug is present even with it.

[randall77]

This one is tricky.
In file2.go, their is a call f(x, y) where y's type is one of the bad typedefs.
That bad typedef name is not mentioned anywhere in the file. (y gets its type from a := assignment.)
So when cgo rewrites the call f(x, y) to func(_x typeof(x), _y typeof(y)) { ..cgochecks.. f(_x, _y) }(x, y), the typeof(y) is wrong, particularly because of this, from cmd/cgo:gcc.go:FuncArg:

		// C has much more relaxed rules than Go for
		// implicit type conversions. When the parameter
		// is type T defined as *X, simulate a little of the
		// laxness of C by making the argument *X instead of T.

But more generally, introducing typeof(y) when we never properly probed its badness is problematic. Even if we disabled the T->*X relaxation, we still generate the type declaration:

type _Ctype_CFArrayRef *_Ctype_struct___CFArray

Whereas when cgo processes file1.go, we get

type _Ctype_CFArrayRef uintptr

It happens to build successfully like that, but it sounds dangerous in general.

We could somehow re-run the part that asks gcc for type info, once we extract all the argument types from functions whose types we obtained from the first gcc run.

[btoews]

Sorry to pile on here, but I noticed some additional weird behavior. The SecKeyAlgorithm type from Security.framework is defined as

typedef CFStringRef SecKeyAlgorithm CF_STRING_ENUM

Variables of this type can be passed to C functions without an issue. When you start using CFStringRefs though, you start getting errors about the SecKeyAlgorithm usage.

Here is a contrived example that doesn't really do anything.

• example0.go calls some Security.framework function with a SecKeyAlgorithm argument. The build succeeds.
• example1.go adds a function call that returns a CFStringRef which is then compared with an int. The compiler complains about this comparison.
• example2.go adds var _ C.CFStringRef. The compiler no longer complains about the CFStringRef vs. int comparison, but now complains about our SecKeyAlgorithm argument being a _Ctype_CFStringRef instead of a *_Ctype_struct___CFString.
• example3.go casts the SecKeyAlgorithm to an unsafe.Pointer and then to a *struct___CFString. The build succeeds.

[FiloSottile]

@gopherbot please file this for backport against 1.10. This is a regression.

[gopherbot]

Backport issue(s) opened: #25036 (for 1.10).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

[gopherbot]

Change https://golang.org/cl/122575 mentions this issue: cmd/cgo: check function argument types for bad C pointer types

[gopherbot]

Change https://golang.org/cl/123177 mentions this issue: cmd/cgo: fix cgo bad typedefs

[randall77]

@gopherbot open some more backport issues please.

[gopherbot]

Change https://golang.org/cl/123715 mentions this issue: misc/cgo/test: fix issue 24161 test for 1.11 and earlier

[gopherbot]

Change https://golang.org/cl/123959 mentions this issue: misc/cgo: fix test on iOS

[gopherbot]

Change https://golang.org/cl/124075 mentions this issue: misc/cgo: fix darwin test, again

[mvdan]

@randall77 it seems like the fix for this issue introduced a regression, so beware when backporting this for 1.10.4: #26517

[btoews]

Go 1.11 breaks the workaround for the behavior I described above. Here is the output from compiling the example workaround:

$ go build ./example3.go
# command-line-arguments
./example3.go:19:223: cannot use nil as type _Ctype_SecKeyRef in argument to func literal
./example3.go:19:223: cannot use algo (type *_Ctype_struct___CFString) as type _Ctype_CFStringRef in argument to func literal
./example3.go:19:223: cannot use nil as type _Ctype_CFDataRef in argument to func literal

I don't think it will be possible to fix this in a way that works with 1.9, 1.10, and 1.11 without writing version-specific code and using build tags.

[btoews]

I guess its not quite so dire. The only versions that would require special casing would be 1.10.0→1.10.3. I hadn't realized that there had been a patch release changing the behavior (1.10.4).

[lzhaoGitHub]

github.com/pchain/vendor/github.com/rjeczalik/notify

vendor/github.com/rjeczalik/notify/watcher_fsevents_cgo.go:51:216: cannot use nil as type _Ctype_CFAllocatorRef in argument to func literal
vendor/github.com/rjeczalik/notify/watcher_fsevents_cgo.go:162:47: cannot use nil as type _Ctype_CFAllocatorRef in argument to _Cfunc_CFStringCreateWithCStringNoCopy
vendor/github.com/rjeczalik/notify/watcher_fsevents_cgo.go:163:225: cannot use nil as type _Ctype_CFAllocatorRef in argument to func literal
make: *** [build] Error 2

[ianlancetaylor]

@lzhaoGitHub This issue is closed. If you are having a problem, please open a new issue with full details, including filling out the issue template. Thanks.