cmd/link: OS X codesign only works with -ldflags -s

[derekparker]

Using the Go 1.5 compiler toolchain and compiling a binary that uses CGO and then codesigning that binary, upon attempting to execute the program it will be killed immediately by the system. Codesigning a non-cgo binary works just fine, as expected.

Given the following small program:

package main

/*
int
cfunc() {
    return 2;
}
*/
import "C"
import "fmt"

func main() {
    fmt.Println("cfunc says:", C.cfunc())
}

and building simply with go build and running codesign -f -s my_cert ./myprog, when attempting to execute the program in a shell the response is "killed: ./myprog".

The system log reports:

$ taskgated[92]: no signature for pid=1991 (cannot make code: UNIX[No such process])

Tested against Go1.5beta2 and:

$ go version devel +765cea2 Mon Jul 27 18:03:45 2015 +0000 darwin/amd64

[ianlancetaylor]

Please run "go build -ldflags=-v" and attach the output. I'd like to confirm that the linker is invoking the external linker. Thanks.

[derekparker]

$ go build -ldflags=-v

# github.com/derekparker/testcgo
HEADER = -H1 -T0x2000 -D0x0 -R0x1000
searching for runtime.a in $WORK/runtime.a
searching for runtime.a in /usr/local/go/pkg/darwin_amd64/runtime.a
 0.00 deadcode
 0.02 pclntab=334118 bytes, funcdata total 69364 bytes
 0.02 dodata
 0.02 symsize = 0
 0.03 symsize = 0
 0.03 reloc
 0.04 reloc
 0.05 asmb
 0.05 codeblk
 0.07 datblk
 0.07 dwarf
 0.08 symsize = 0
 0.13 dwarf pass 2.
 0.14 sym
 0.16 headr
host link: "clang" "-m64" "-gdwarf-2" "-Wl,-no_pie,-headerpad,1144" "-Wl,-pagezero_size,4000000" "-o" "/var/folders/qk/sdqgc3hn4v7_zxpkbmms53k00000gn/T/go-build627095567/github.com/derekparker/testcgo/_obj/exe/a.out" "-Qunused-arguments" "/var/folders/qk/sdqgc3hn4v7_zxpkbmms53k00000gn/T/go-link-928391324/000000.o" "/var/folders/qk/sdqgc3hn4v7_zxpkbmms53k00000gn/T/go-link-928391324/000001.o" "/var/folders/qk/sdqgc3hn4v7_zxpkbmms53k00000gn/T/go-link-928391324/go.o" "-g" "-O2" "-g" "-O2" "-lpthread"
 0.26 cpu time
42397 symbols
30720 liveness data

[ianlancetaylor]

Thanks. We apparently have a program built by clang for which codesign does not work. Off hand I have no guess as to what the Go linker might be doing to cause this to fail. This will have to be tackled by somebody who understands how codesign works on Darwin.

[rsc]

If you build with -ldflags -s, which drops the DWARF information from the executable, then codesign produces a valid binary. I think this is mostly not Go's fault (except that Go works hard to put DWARF info in the binaries, unlike most OS X programs). Certainly for Go 1.5 but possibly long term we can suggest that people who need codesign can use -ldflags -s.

[derekparker]

Thanks, that does seem to work.

[ianlancetaylor]

Any thoughts on whether we should document this for Go 1.5, and where we should do it? Ideally we should put the docs in the place where someone who encounters the problem will look, but where would that be?

[stuartcarnie]

-ldflags -s no longer seems to work with go 1.5.2 and results in same error in logs

12/3/15 1:32:57.505 PM taskgated[59690]: no signature for pid=59709 (cannot make code: UNIX[No such process])

[fasterthanlime]

Issue still exists on 1.6, ldflags -s doesn't work either:

ld: warning: option -s is obsolete and being ignored
ld: internal error: atom not found in symbolIndex(__ZN6brotli10FindBlocksIhLi256EEEvPKT_mdRKNSt3__16vectorINS_9HistogramIXT0_EEENS4_9allocatorIS7_EEEEPh) for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Probably going to forgo codesigning for now, but eager to learn about another workaround or a real fix! 🌟

[rsc]

The ld: warning: option -s is obsolete... suggests you are somehow passing the -s to the host linker instead of to cmd/link.

[fasterthanlime]

The ld: warning: option -s is obsolete... suggests you are somehow passing the -s to the host linker instead of to cmd/link.

That would be really hard to do accidentally, one would have to run:

go build -ldflags "-extldflags -s"

instead of:

go build -ldflags "-s"

and I can assure you I was doing the latter. However, I suspect cmd/link is passing -s down to clang.

[rsc]

If the problem with cmd/link is just dwarf, then try -ldflags -w instead of
-ldflags -s.

On Tue, Mar 22, 2016 at 12:00 PM Amos Wenger [email protected]
wrote:

The ld: warning: option -s is obsolete... suggests you are somehow passing
the -s to the host linker instead of to cmd/link.

That would be really hard to do accidentally, one would have to run:

go build -ldflags "-extldflags -s"

instead of:

go build -ldflags "-s"

and I can assure you I was doing the latter. However, I suspect cmd/link
is passing -s down to clang.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#11887 (comment)

[fasterthanlime]

If the problem with cmd/link is just dwarf, then try -ldflags -w instead of
-ldflags -s.

With -w, the compilation & linking succeeds, a working binary is produced, but codesign-ing that binary breaks it, as in the original report.

[gazed]

I'm finding that ldflags -s does work with go 1.6 and codesign
based on the following test code in 11887.go

package main
import "C" // Need this line to break codesign without "ldflags -s".
func main() {
    println("Hello OSX.")
}

: go build -ldflags -s -o 11887s 11887.go
: go build -o 11887n 11887.go
: codesign -s "3rd Party Mac Developer Application: Galvanized Logic Inc." 11887s
: codesign -s "3rd Party Mac Developer Application: Galvanized Logic Inc." 11887n
: ./11887s
Hello OSX.
: ./11887n
Killed: 9
: go version
go version go1.6 darwin/amd64

[fasterthanlime]

I'm finding that ldflags -s does work with go 1.6 and codesign
based on the following test code in 11887.go

Following the exact same protocol, but with my program, I'm able to reproduce your results on my local MacbookPro, but not on my build server.

$ go build -ldflags -s -o butler-s .
# github.com/itchio/butler
/usr/local/Cellar/go/1.6/libexec/pkg/tool/darwin_amd64/link: running clang++ failed: exit status 1
ld: warning: option -s is obsolete and being ignored
ld: internal error: atom not found in symbolIndex(__ZN6brotli10FindBlocksIhLi256EEEvPKT_mdRKNSt3__16vectorINS_9HistogramIXT0_EEENS4_9allocatorIS7_EEEEPh) for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

The build server has an older version of clang & OSX:

$ clang --version
Apple LLVM version 7.0.2 (clang-700.1.81)
Target: x86_64-apple-darwin14.5.0
Thread model: posix

$ sw_vers -productVersion
10.10.5

My local computer has the latest stable of both:

$ clang --version
Apple LLVM version 7.3.0 (clang-703.0.29)
Target: x86_64-apple-darwin15.3.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

$ sw_vers -productVersion
10.11.3

Note that, even on my local machine, cmd/link does pass -s down to clang, which prints a warning — it just looks like cmd/link swallows linker warnings if the link step is successful:

$ go build -v -x -ldflags "-s -v" -o butler-s .
# (manually suppressed output for readability)
# note the "-s" below
host link: "clang++" "-m64" "-s" "-Wl,-no_pie,-headerpad,1144" "-Wl,-pagezero_size,4000000" "-o" "/var/folders/kj/vbzcvcf94hn0hkxpc6d97zcc0000gn/T/go-build649125102/github.com/itchio/butler/_obj/exe/a.out" "-Qunused-arguments" "/var/folders/kj/vbzcvcf94hn0hkxpc6d97zcc0000gn/T/go-link-896862525/go.o" "/var/folders/kj/vbzcvcf94hn0hkxpc6d97zcc0000gn/T/go-link-896862525/000000.o" "/var/folders/kj/vbzcvcf94hn0hkxpc6d97zcc0000gn/T/go-link-896862525/000001.o" "/var/folders/kj/vbzcvcf94hn0hkxpc6d97zcc0000gn/T/go-link-896862525/000002.o" "/var/folders/kj/vbzcvcf94hn0hkxpc6d97zcc0000gn/T/go-link-896862525/000003.o" "/var/folders/kj/vbzcvcf94hn0hkxpc6d97zcc0000gn/T/go-link-896862525/000004.o" "/var/folders/kj/vbzcvcf94hn0hkxpc6d97zcc0000gn/T/go-link-896862525/000005.o" "/var/folders/kj/vbzcvcf94hn0hkxpc6d97zcc0000gn/T/go-link-896862525/000006.o" "-g" "-O2" "-g" "-O2" "-g" "-O2" "-lpthread" "-g" "-O2" "-framework" "CoreFoundation" "-framework" "Security" "-g" "-O2" "-g" "-O2" "-g" "-O2"
# here's the warning
ld: warning: option -s is obsolete and being ignored

I'd understand if this bug doesn't receive a lot of attention, since recent clangs seem to handle it just fine. I'll just upgrade my build server, thanks for the help! 🌟

edit: if anyone's interested in reproducing the above, the program I'm building is https://github.com/itchio/butler

edit 2: note that the simple 11887.go test case posted by @gazed above isn't enough to reproduce the problem

[kujenga]

I'm currently running into this exact issue without performing any explicit code signing.

I updated my mac yesterday with the latest command line tools and Xcode, version 8.3, and I believe exactly then is when I started seeing this issue. I'm on macOS version 10.12.4 (16E195).

The failure happens even when doing a vanilla go run on a single file containing Cgo code. I created a repository with a minimal example of the failure here: https://github.com/kujenga/brokencgo

Here's a file causing failures when I go run it:

package main

import (
	"unsafe"
)

/*
#include <stdio.h>
#include <stdlib.h>

void myprint(char* s) {
	printf("%s\n", s);
}
*/
import "C"

func main() {
	cs := C.CString("Hello from stdio\n")
	C.myprint(cs)
	C.free(unsafe.Pointer(cs))
}

My theory is that something in the update to the command line tools is causing a breakage here.

[kujenga]

It looks like the new issues are now being tracked here: #19734

[gopherbot]

CL https://golang.org/cl/38853 mentions this issue.

[rsc]

I see now what happened with -s above: passing -s to the Go linker relays it to the macOS linker. And even though the macOS linker says it is ignoring -s, it seems to not be ignoring it quite enough. This was #19775 just now.

I expect the fix for that will be in Go 1.8.1, so that you can use -ldflags=-s successfully on binaries with external linking. I hope that will get codesign working again.

In fact codesign might start working out of the box without any special flags. At least on my machine right now:

$ cd go/src/runtime/testdata/testprogcgo
$ go build
$ codesign -f -s - testprogcgo
$ ./testprogcgo
usage: ./testprogcgo name-of-test
$ 

I don't know if using -s - instead of a real identity bypasses the problems others have seen. If anyone else who can reproduce a problem can check that codesign -f -s - yourbinary fails as well, that'd be great.

Thanks.

[gopherbot]

CL https://golang.org/cl/39602 mentions this issue.

[OSMeteor]

use go1.8.1 fixed it good