Getting better stack traces?

[tleyden]

I somehow managed to cause a panic (I'm still trying to dig into the root cause), and I'm seeing a stack trace:

fatal error: unexpected signal during runtime execution
[signal 0xb code=0x1 addr=0x15 pc=0x4ea78bb]

runtime stack:
runtime.gothrow(0x4847e70, 0x2a)
    /usr/local/go/src/runtime/panic.go:503 +0x8e
runtime.sigpanic()
    /usr/local/go/src/runtime/sigpanic_unix.go:14 +0x5e

goroutine 7 [syscall, locked to thread]:
runtime.cgocall_errno(0x4002df0, 0xc20819b900, 0xc200000000)
    /usr/local/go/src/runtime/cgocall.go:130 +0xf5 fp=0xc20819b8e0 sp=0xc20819b8b8
github.com/couchbaselabs/goforestdb._Cfunc_fdb_kvs_close(0x5500090, 0x0)
    /Users/tleyden/Development/sync_gateway/src/github.com/couchbaselabs/goforestdb/:251 +0x43 fp=0xc20819b900 sp=0xc20819b8e0
github.com/couchbaselabs/goforestdb.(*KVStore).Close(0xc20802bb10, 0x0, 0x0)
    /Users/tleyden/Development/sync_gateway/src/github.com/couchbaselabs/goforestdb/forestdb.go:26 +0x135 fp=0xc20819b978 sp=0xc20819b900
github.com/couchbaselabs/forestdb-bucket.(*forestdbBucket).Close(0xc208092480)
    /Users/tleyden/Development/sync_gateway/src/github.com/couchbaselabs/forestdb-bucket/forestdb_bucket.go:470 +0x71 fp=0xc20819b998 sp=0xc20819b978

it's a bit cryptic, and I was wondering if there was a way to get a better error message.

[tleyden]

I noticed in couchbase/indexing@969f73a it is setting #cgo CFLAGS: -O0, but I haven't tried that yet.

[mschoch]

Well stating the obvious it appears to be a null pointer exception in fdb_kvs_close().

Getting reliable stack trace or core file has been a major nightmare so far. As you observed the indexing team has some tricks, BUT last I heard they are not always working. One thing I heard was that using gccgo produces core dumps without any additional drama... But I recommend you talk to the indexing team to get the latest info.

[tleyden]

it appears to be a null pointer exception in fdb_kvs_close().

I believe you, but can you elaborate on how you arrived at that from the stacktrace?

[mschoch]

Sure that last readable line in the stack trace was:

/Users/tleyden/Development/sync_gateway/src/github.com/couchbaselabs/goforestdb/forestdb.go:26

Assuming you're using the latest commit, this is:

https://github.com/couchbase/goforestdb/blob/master/forestdb.go#L26

Also, you can see the text: goforestdb._Cfunc_fdb_kvs_close in the trace too.

[tleyden]

Thanks, makes sense. Looks like there is a way to add tracing, which will be useful for debugging.