Expose rd_kafka_position and rd_kafka_committed

[lucabrunox]

librdkafka has two methods that are very useful for comitting offsets manually when consuming.

rd_kafka_position: https://github.com/edenhill/librdkafka/blob/dca8b2ee91076f2ca72f00abb4d18102a4b7e19a/src/rdkafka.h#L2035

and

rd_kafka_committed:
https://github.com/edenhill/librdkafka/blob/dca8b2ee91076f2ca72f00abb4d18102a4b7e19a/src/rdkafka.h#L2016

That is, instead of relying on commit() without arguments which automatically commits the fetched messages, it's possible to manually provide the offsets to be committed.

The use case is when fetched messages are consumed partially. That is, if 10 messages are fetched but only the first 5 have been processed for one reason or another, it's wanted to only commit 5 messages, hence pass custom information to commit().
However, currently in php-rdkafka there's no way to know such offsets, if I'm not wrong.

rd_kafka_position is certainly the most interesting for such a use case, but given rd_kafka_committed has a similar API, it shouldn't be hard to also expose it.

[arnaud-lb]

Agreed 👍

This should probably be added to the KafkaConsumer class. Would accept a PR for that 😉

[arnaud-lb]

rd_kafka_committed has been exposed as RdKafka\KafkaConsumer::getCommittedOffsets(array $topics, int timeout_ms) in #208.

[nick-zh]

@arnaud-lb since we now also have rd_kafka_position exposed, we can close this 🎉

[roohitavaf]

What about metadata? I my application I need to write and read metadata, but unfortunately, it is not possible with C++ API.

[nick-zh]

@roohitavaf this should help:
https://arnaud.le-blanc.net/php-rdkafka-doc/phpdoc/rdkafka.getmetadata.html

[roohitavaf]

Thank, but I am not sure, if it is the metadata that I want. I want the metadata the you can write whenever you commit an offset. It should be for each topic-partition instead of per topic.

[nick-zh]

@roohitavaf can you elaborate more or give me the c function that you are referring to, i am not sure what kind of meta data we are talking about :)

[roohitavaf]

Sure, look at this test:
https://github.com/edenhill/librdkafka/blob/d8f24f9e1932988a9db526e9f97f267d5a544f0d/tests/0099-commit_metadata.c#L184

I want that "Hello word!" :)

[roohitavaf]

You can turn off automatic offset commit and do it manually. Kafka allows you write some metadata whenever you commit the offset and read it back. In our design we rely on this capability to write and read metadata with each offset commit, but C++ API does not have any API for that, so we have to use C API.

[nick-zh]

ok so just to be sure, you want to use this for the php extension as well?
Afaik we cannot set metadata yet in php for a TopicPartition, we would need to add this first. I assume it could be read back afterwards with the offsetCommitCb

[roohitavaf]

No, I need it for C++. Oh, I am so sorry, I didn't noticed the repo! I just googled and came here. I thought it is official librdkafka :)