runtime: unexpected fault address in math/big

[mikioh]

See http://build.golang.org/log/1b98f2faca78eb97adbd218ad62fb20d3b62d8e8. But it happened only once on the freebsd-race buildbot.

unexpected fault address 0x5ec7a6
fatal error: fault
[signal 0xa code=0x3 addr=0x5ec7a6 pc=0x5ec7a6]

goroutine 23 [running]:
runtime.throw(0x7df110, 0x5)
    /tmp/workdir/go/src/runtime/panic.go:549 +0x8c fp=0xc820098cf0 sp=0xc820098cd8
runtime.sigpanic()
    /tmp/workdir/go/src/runtime/sigpanic_unix.go:21 +0x1ce fp=0xc820098d40 sp=0xc820098cf0
math/big.mulAddVWW(0xc8201b8070, 0x9, 0xe, 0xdeaddeaddeaddead, 0x9, 0x1e, 0x1, 0x0, 0xe, 0x9, ...)
    /tmp/workdir/go/src/math/big/arith_amd64.s:331 +0x26 fp=0xc820098d48 sp=0xc820098d40
math/big.nat.mulAddWW(0x0, 0x0, 0x0, 0xdeaddeaddeaddead, 0x9, 0x1e, 0x1, 0x0, 0x9, 0xe, ...)
    /tmp/workdir/go/src/math/big/nat.go:183 +0x184 fp=0xc820098db8 sp=0xc820098d48
math/big.nat.mul(0x0, 0x0, 0x0, 0xdeaddeaddeaddead, 0x9, 0x1e, 0xc8201d4060, 0x1, 0x6, 0x0, ...)
    /tmp/workdir/go/src/math/big/nat.go:408 +0xac8 fp=0xc820098ea8 sp=0xc820098db8
math/big.(*Int).Mul(0xc8200990d0, 0xc8202f8580, 0xc820099130, 0xc8200990f0)
    /tmp/workdir/go/src/math/big/int.go:156 +0xdf fp=0xc820098f30 sp=0xc820098ea8
crypto/elliptic.(*CurveParams).addJacobian(0xc820052800, 0xc82025dee0, 0xc82025df00, 0xc82025c000, 0xc8202f8580, 0xc8202f85a0, 0xc82025c020, 0x0, 0xc8201d5d70, 0x1)
    /tmp/workdir/go/src/crypto/elliptic/elliptic.go:133 +0x420 fp=0xc820099158 sp=0xc820098f30
crypto/elliptic.(*CurveParams).Add(0xc820052800, 0xc82025dee0, 0xc82025df00, 0xc8202f8580, 0xc8202f85a0, 0x42, 0xc8202f8580)
    /tmp/workdir/go/src/crypto/elliptic/elliptic.go:105 +0xaf fp=0xc8200991b8 sp=0xc820099158
crypto/ecdsa.Verify(0xc8201e90a0, 0xc8201e9580, 0x14, 0x20, 0xc8201e95a0, 0xc8201e95c0, 0x0)
    /tmp/workdir/go/src/crypto/ecdsa/ecdsa.go:253 +0x73b fp=0xc8200992e8 sp=0xc8200991b8
crypto/tls.(*ecdheKeyAgreement).processServerKeyExchange(0xc8200f1d40, 0xc820084280, 0xc820085680, 0xc820089e40, 0xc8201f6000, 0xc8201d4ff0, 0x0, 0x715360)
    /tmp/workdir/go/src/crypto/tls/key_agreement.go:366 +0xe82 fp=0xc8200994a0 sp=0xc8200992e8
crypto/tls.(*clientHandshakeState).doFullHandshake(0xc820099c10, 0xc8201d78c0, 0x59)
    /tmp/workdir/go/src/crypto/tls/handshake_client.go:316 +0x2569 fp=0xc820099928 sp=0xc8200994a0
crypto/tls.(*Conn).clientHandshake(0xc8201dc600, 0x8508c8, 0xc8201dc614)
    /tmp/workdir/go/src/crypto/tls/handshake_client.go:207 +0x19cc fp=0xc820099cc0 sp=0xc820099928
crypto/tls.(*Conn).Handshake(0xc8201dc600, 0x0, 0x0)
    /tmp/workdir/go/src/crypto/tls/conn.go:1033 +0x123 fp=0xc820099d00 sp=0xc820099cc0
crypto/tls.(*Conn).Write(0xc8201dc600, 0xc820099e50, 0x6, 0x6, 0x0, 0x0, 0x0)
    /tmp/workdir/go/src/crypto/tls/conn.go:880 +0xfc fp=0xc820099d90 sp=0xc820099d00
crypto/tls.(*clientTest).run.func1(0xc8201dc600, 0xc8201ea1b0, 0xc820103400, 0x800aae7b8, 0xc82010dc20, 0xc8201d7740)
    /tmp/workdir/go/src/crypto/tls/handshake_client_test.go:211 +0x9b fp=0xc820099f60 sp=0xc820099d90
runtime.goexit()
    /tmp/workdir/go/src/runtime/asm_amd64.s:2006 +0x1 fp=0xc820099f68 sp=0xc820099f60
created by crypto/tls.(*clientTest).run
    /tmp/workdir/go/src/crypto/tls/handshake_client_test.go:222 +0x298

[mikioh]

Happened on the openbsd build bot: http://build.golang.org/log/4125d6f78c1da04049cd603fea2a012e53849b76

unexpected fault address 0x55aaf0
fatal error: fault
[signal 0xa code=0x3 addr=0x55aaf0 pc=0x55aaf0]

goroutine 180 [running]:
runtime.throw(0x6e1ed4, 0x5)
    /tmp/workdir/go/src/runtime/panic.go:549 +0x8c fp=0xc8201a1620 sp=0xc8201a1608
runtime.sigpanic()
    /tmp/workdir/go/src/runtime/sigpanic_unix.go:21 +0x1ce fp=0xc8201a1670 sp=0xc8201a1620
math/big.nat.cmp(0xdeaddeaddeaddead, 0x9, 0x1e, 0xc82000e410, 0x9, 0xa, 0x9)
    /tmp/workdir/go/src/math/big/nat.go:162 +0x40 fp=0xc8201a1678 sp=0xc8201a1670
math/big.nat.div(0x0, 0x0, 0x0, 0xdeaddeaddeaddead, 0x9, 0x1e, 0xdeaddeaddeaddead, 0x9, 0x1e, 0xc82000e410, ...)
    /tmp/workdir/go/src/math/big/nat.go:525 +0x7b fp=0xc8201a1738 sp=0xc8201a1678
math/big.(*Int).QuoRem(0xc8201a1820, 0xc8203b39e0, 0xc820010860, 0xc8203b39e0, 0xc820012800, 0xc8203ac580)
    /tmp/workdir/go/src/math/big/int.go:227 +0xaf fp=0xc8201a17e8 sp=0xc8201a1738
math/big.(*Int).Mod(0xc8203b39e0, 0xc8203b39e0, 0xc820010860, 0xc8203ac500)
    /tmp/workdir/go/src/math/big/int.go:258 +0xcf fp=0xc8201a1868 sp=0xc8201a17e8
crypto/ecdsa.Verify(0xc820383ce0, 0xc8203643a0, 0x20, 0x20, 0xc820364360, 0xc820364380, 0x0)
    /tmp/workdir/go/src/crypto/ecdsa/ecdsa.go:259 +0x405 fp=0xc8201a1978 sp=0xc8201a1868
crypto/tls.(*serverHandshakeState).doFullHandshake(0xc8201a1c18, 0xc82001f400, 0x0)
    /tmp/workdir/go/src/crypto/tls/handshake_server.go:531 +0xef1 fp=0xc8201a1bd8 sp=0xc8201a1978
crypto/tls.(*Conn).serverHandshake(0xc8202d0f00, 0x746800, 0xc8202d0f14)
    /tmp/workdir/go/src/crypto/tls/handshake_server.go:80 +0x250 fp=0xc8201a1cf0 sp=0xc8201a1bd8
crypto/tls.(*Conn).Handshake(0xc8202d0f00, 0x0, 0x0)
    /tmp/workdir/go/src/crypto/tls/conn.go:1033 +0xff fp=0xc8201a1d18 sp=0xc8201a1cf0
crypto/tls.(*Conn).Write(0xc8202d0f00, 0xc8201a1e58, 0xd, 0xd, 0x0, 0x0, 0x0)
    /tmp/workdir/go/src/crypto/tls/conn.go:878 +0xdd fp=0xc8201a1da0 sp=0xc8201a1d18
crypto/tls.(*serverTest).run.func1(0xc8202d0f00, 0xc8202ebaa0, 0xc8200b0ab0, 0x204e307b8, 0xc82030d2c0, 0xc8202ebb00)
    /tmp/workdir/go/src/crypto/tls/handshake_server_test.go:524 +0x9a fp=0xc8201a1f70 sp=0xc8201a1da0
runtime.goexit()
    /tmp/workdir/go/src/runtime/asm_amd64.s:2006 +0x1 fp=0xc8201a1f78 sp=0xc8201a1f70
created by crypto/tls.(*serverTest).run
    /tmp/workdir/go/src/crypto/tls/handshake_server_test.go:539 +0x1ff

[bradfitz]

Another:
https://storage.googleapis.com/go-build-log/92d07aae/freebsd-amd64-gce101_89ff76d3.log

[randall77]

I have been able to reproduce this using:

$ cd crypto/tls
$ go test -c
$ GOGC=1 ./tls.test -test.count=100

My suspicions lie in Verify (crypto/ecdsa/ecdsa.go:238), I think something is not being kept live that should be. The relevant code:

var w *big.Int
if in, ok := c.(invertible); ok {
    w = in.Inverse(s)
} else {
    w = new(big.Int).ModInverse(s, N)
}

That new(big.Int) gets allocated on the stack. It's a tricky case to get right, as on the other branch of the if it never gets initialized. Is it marked as live at the next call? That's next up in the investigation.

[randall77]

My suspicion was wrong. Very painful binary search fingers SSA compilation of (*CurveParams).affineFromJacobian

Maybe related to #14725

[gopherbot]

CL https://golang.org/cl/20457 mentions this issue.