High memory usage when producing messages

[intfish123]

Description

I run a process to write messages to redpanda for pressure test, the memory usage is very high with kafka._Cfunc_GoBytes. It's so weird.

How to reproduce

Run redpanda with HDD that is lower write performance, then read messages and write to redpanda.

producer code:

p, err := kafka.NewProducer(kafkaconf)
if err != nil {
	return err
}

go func() {
	i := 0
	for {
		e, ok := <-ck.producer.Events()
		if !ok {
			zap.S().Infof("producer.Events() exited")
			return
		}
		switch ev := e.(type) {
		case *kafka.Message:
			m := ev
			if m.TopicPartition.Error != nil {
				zap.S().Errorf("Delivery %v failed: %v, data size: %vbytes\n", *m.TopicPartition.Topic, m.TopicPartition.Error, len(m.Value))
			}
		default:
			zap.S().Warnf("Ignored event: %v", ev)
		}
	}
}()

send message to chan

for m :=  range readChan {
    p.ProduceChannel() <- m
}

Client configuration:

        defaultBatchSize := 1 * 1024 * 1024           // in bytes
	defaultMessageMaxBytesSize := 2 * 1024 * 1024 // in bytes
	defaultBatchNumMessages := 10000
	defaultLingerMs := 1000
	
	kafkaconf := &kafka.ConfigMap{
		"bootstrap.servers":             strings.Join(address, ","),
		"compression.type":              compressType,
		"batch.size":                    defaultBatchSize,
		"message.max.bytes":             messageMaxBytesSize,
		"batch.num.messages":            batchNumMessages,
		"linger.ms":                     lingerMs,
		"sticky.partitioning.linger.ms": 2 * lingerMs,
		"acks":                          1,
	}

Checklist

Please provide the following information:

• confluent-kafka-go and librdkafka version (v1.8.2):
• Apache Kafka broker version: 2.2.0
• Client configuration

[yanmxa]

I encountered a similar issue with kafka._Cfunc_GoBytes.

[dingteddy]

I encountered a similar issue with kafka._Cfunc_GoBytes.

i got same issue, how do you solve this problem?