json.Marshal results in "panic: runtime error: cgo argument has Go pointer to Go pointer"

[matthias-margush]

Hi there!

Changing the producer_example.go to marshal a struct (see below) results in this error:

» go run producer_example.go localhost:9092 foo
Created Producer rdkafka#producer-1
panic: runtime error: cgo argument has Go pointer to Go pointer

goroutine 1 [running]:
panic(0x40d59a0, 0xc420ff0090)
	/usr/local/Cellar/go/1.7.4_1/libexec/src/runtime/panic.go:500 +0x1a1
github.com/confluentinc/confluent-kafka-go/kafka._cgoCheckPointer0(0xc42009a024, 0x0, 0x0, 0x0, 0x0)
	??:0 +0x59
github.com/confluentinc/confluent-kafka-go/kafka.(*Producer).produce(0xc4200a0000, 0xc42004deb0, 0x0, 0xc420098120, 0x0, 0xc42009a000)
	/Users/matthias/code/src/github.com/confluentinc/confluent-kafka-go/kafka/producer.go:130 +0x31b
github.com/confluentinc/confluent-kafka-go/kafka.(*Producer).Produce(0xc4200a0000, 0xc42004deb0, 0xc420098120, 0x11, 0x40)
	/Users/matthias/code/src/github.com/confluentinc/confluent-kafka-go/kafka/producer.go:154 +0x48
main.main()
	/Users/matthias/code/src/github.com/confluentinc/confluent-kafka-go/examples/producer_example/producer_example.go:53 +0x454
exit status 2

Here are the changes to producer_example.go:

diff --git a/examples/producer_example/producer_example.go b/examples/producer_example/producer_example.go
index f81f98a..1bf4ab4 100644
--- a/examples/producer_example/producer_example.go
+++ b/examples/producer_example/producer_example.go
@@ -18,9 +18,11 @@ package main
  */

 import (
+       "encoding/json"
        "fmt"
-       "github.com/confluentinc/confluent-kafka-go/kafka"
        "os"
+
+       "github.com/confluentinc/confluent-kafka-go/kafka"
 )

 func main() {
@@ -47,7 +49,7 @@ func main() {
        // .Events channel is used.
        deliveryChan := make(chan kafka.Event)

-       value := "Hello Go!"
+       value, _ := json.Marshal(struct{ M string }{M: "Hello Go!"})
        err = p.Produce(&kafka.Message{TopicPartition: kafka.TopicPartition{Topic: &topic, Partition: kafka.PartitionAny}, Value: []byte(value)}, deliveryChan)

        e := <-deliveryChan

Should this work?

[mcandre]

Same error on my machine.

I think this is what is happening:

1. json.Marshal() may emit a slice rather than a static byte array. Go's []byte syntax hides this subtly.
2. confluent-kafka-go passes the slice through in a unsafe.Pointer() wrapper. If the data were always a static byte array, this would be okay. Instead, we're passing a pointer to a Go slice to librdkafka.
3. librdkafka's API offers void * for message value, failing to provide type safety. Woops!

To fix the bug, confluent-kafka-go just needs to explicitly convert the Go []byte into a uintptr[] before wrapping in an unsafe.Pointer. Go has a C.CBytes() convenience method for this, e.g.:

golang/go@5a34472#diff-6255934d72e047dd802d046c5748b331R1395

[terrywh]

is there some workaround available now ?

[edenhill]

This is fixed on the api_catchup branch now, please give it a spin to verify the fix.

[edenhill]

Now on master.