Reason for setting max heap size in Strimzi

[maciej-tatarski]

What's the reason of selecting 5Gi as max heap size for kafka resource in Strimzi? Wouldn't it be better to bind it to container resources request or limits? For example with flags like:

-XX:+UseContainerSupport
-XX:MaxRAMPercentage=50.0

Related docs link

[scholzj]

I guess the short answer is best practices. Kafka itself does not require too much heap memory (well, I guess too much is a bit relative) but it relies quite heavily on the disk page cache. So when configuring it, you need to make sure that from the memory the container gets, there is a limit on what is used for heap and enough space is left for the disk page cache. If you don't do it, you risk that the JVM will use most of the memory for the heap and not enough will be left for the page cache and the performance of your cluster will suffer.

The general idea is that Kafka does not make any good use of more than 5Gi JVM and the rest can be left to the page cache. So you can normally see brokers with e.g. 64Gi of memory but with having only a few Gi for the heap.

I'm quite sure someone might say that it doesn't need more than 4Gi, someone else would say 8Gi, for whatever reason we settled on 5Gi. But the basic idea is the same. That said, there always might be one or other use-case where it might be best to configure it differently. So if needed, you should be able to customize it in the custom resource.

[maciej-tatarski]

Isn't strimzi running prometheus jmx_exporter alongside kafka as well? I noticed that when number of topics and partitions is growing, it is getting slower to scrape the metrics. Shouldn't the heap size be increased as well in this cases as exporter is also consuming part of the heap?

jmx exporter issue

[scholzj]

We are still using it currently if you enable metrics. I know it can be slow, but I do not have any exact experience with whether it needs additional gigs of heap memory. As I said, you can customize it.

[maciej-tatarski]

Sure, thanks for help. I noticed significant improvement (50% shorter scraping times) when increasing the heap size from default 5Gb to 8Gb for cluster with around 4000 partitions per broker.