rdkafka input plugin

[repeatedly]

rdkafka-ruby is now considering each_batch support.

karafka/rdkafka-ruby#124

We can implement in_rdkafka with this feature.

[repeatedly]

may I ask that when can be release?

It depends on rdkafka-ruby. When rdkafka-ruby merge above patch and release new version, we can test it.

[tkornai]

Hi @repeatedly,

may I ask what causes the hard dependency on the above PR? I think a similar batching method could be fairly easily implemented in this client lib with the existing functionality exposed by rdkafka-ruby, that could be replaced in the future once the feature is merged.

[repeatedly]

I'm not familiar with entire rdkafka's API, so I can't judge above unfinished implementation is good or not.
If you ok with current implemenation, could you send a patch?

[tkornai]

I smoke tested the above PR locally, but it is definitely not battle hardened yet.

[tkornai]

I have created a new PR which helps with tracking the changes against the in_kafka_group.rb implementation:
#367

[github-actions]

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

[kenhys]

released as fluent-plugin-kafka v0.15.0