probable memory leak observed after deploying delivery.report.only.error=true

[Navgeet]

Description

We deployed a build last night that turned off success acks. Exactly since the deployment, we're seeing monotonically increasing memory usage, which points to a memory leak. We're attaching the difference of two heap dumps taken 30mins apart.

How to reproduce

This issue was observed in production. We're working on reproducing this in a load environment.

Checklist

Please provide the following information:

• confluent-kafka-go and librdkafka version (LibraryVersion()): both v0.11.6
• Apache Kafka broker version: confluent 4.1.2 (kafka 1.1.1-cp1)
• Client configuration:

		"statistics.interval.ms":     10000,
		"delivery.report.only.error": true,
		"compression.codec":          "gzip"

• Operating system: ubuntu 16.04
• Provide client logs (with "debug": ".." as necessary)
• Provide broker log excerpts
• Critical issue

[edenhill]

Thanks for the heap dumps, very useful!

For each message produced the Go client allocates a small state object (cgoDr) which is later freed when the delivery report is received.
When successful delivery reports are disabled these cgoDr memory objects are never freed, and that is what you are seeing here.

What we'll need to do is intercept the configuration property and not passing it on to librdkafka, but instead filter out the successful delivery reports on the Go side instead so that the cgoDr object can be freed.

Until that fix is in we recommend not to set delivery.report.only.error.

[Navgeet]

Thanks. Until this is fixed, we should also add a note to the README against setting this.

[edenhill]

I think we should fail NewProducer if this property is set.

[Navgeet]

Hi, is this getting fixed in the upcoming 1.0 release?

[edenhill]

Fixed as in support it, or reject it?
The upcoming v1.0 release (master branch) rejects the property, and I believe that is the best option a this point: your application can filter out non-errored deliveries.