kafka_group not working with kafka > 2.1

[splushii]

Using the kafka_group consumer with a Kafka 2.3 cluster results in lost messages, duplicated messages and abnormally high load and low throughput. Using a Kafka 2.1 cluster works well. I am not sure of the reason but I suspect that the consumer offset change (introduced in Kafka 2.2) is not supported by ruby-kafka (due to the compatibility chart). But these are just speculations that made me try out Kafka 2.1, nothing I have confirmed.

The complete plugin config for reference:

    <source>
      @type kafka_group

      brokers              log-kafka-bootstrap.log.svc.cluster.local:9093
      consumer_group       fluentd
      topics               test
      format               json
      time_source          record
      time_format          "%iso8601"
      start_from_beginning false
      max_wait_time        5
      max_bytes            1048576

      username        "#{ENV['KAFKA_USERNAME']}"
      password        "#{ENV['KAFKA_PASSWORD']}"
      scram_mechanism sha512
      ssl_ca_cert     /etc/ssl/certs/kafka/ca.crt
    </source>

I am running multiple fluentd instances with the same config.

[repeatedly]

Hmm... if ruby-kafka keeps limited support with recent versions, we need to consider rdkafka based consumers.

[wcollin]

i have the same issue. my version is td-agent-3.5.1-0.el7.x86_64 and kafka 2.12-2.3.0，when i fallback kafka to 2.12-2.2.0，everything is ok.

[pathikritmodak]

Having the same issue. I would upvote for rdkafka based input plugin too. With ruby_kafka, there have been some or the other issue with kafka upgrades, and their compatibility charts are so convoluted and doesn't give much details so you don't know what's going to break.

[robinp-tw]

As a mitigation in the README at least, see zendesk/ruby-kafka#839 - please add to that PR's comments if you have any more specifics.

[den-is]

what's the error message you are getting ?
Interesting if it is similar to mine #346

rdkafka binding sounds good. iirc fluent-bit is already using it

[repeatedly]

rdkafka input plugin will be handled in #350

[tkornai]

Related PR: #367

[github-actions]

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

[github-actions]

This issue was automatically closed because of stale in 30 days