KAFKA-6733: Support of printing additional ConsumerRecord fields in DefaultMessageFormatter

[Matzz]

New supported fields: offset, partition and headers. Code is backward compatible, so output will remain the same if user won't specify new configuration properties.
Added unit tests to DefaultMessageFormatter which cover previous and new properties.

Committer Checklist (excluded from commit message)

• Verify design and implementation
• Verify test coverage and CI build status
• Verify documentation (including upgrade notes)

[Matzz]

@huxihx @ijuma Could I ask for review?

[rajinisivaram]

@ijuma Do we need a KIP for this since it is adding new properties to the CLI of ConsoleConsumer?

[Matzz]

@rajinisivaram @ijuma I didn't found any documentation for current properties of DefaultMessageFormatter (provided by console consumer --property parameters). I'm happy to add some docs but I'm not sure where.

[Matzz]

@rajinisivaram @ijumai resolved conflicts with trunk. Could I ask for some guide what's next? I should just wait for review?

[rajinisivaram]

@Matzz Usually, when we add or change command line arguments to a tool, we need to write up a KIP (https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals) and go throught the KIP review process. This is not exactly adding new CLI arguments, but it is adding externally visible properties, so not sure if we still need a KIP. Will see what @ijuma says.

BTW, looks like there are still conflicts on ConsoleConsumer.scala in the PR.

[onzo-mateuszzakarczemny]

@rajinisivaram @ijuma A Kind nudge.
It takes a bit long for that PR. It doesn't brake any contract or any other backward comparability. It just adds couple more parameters for default message formatter (and therefore console consumer). I saw in other PRs that there was no KIP for such changes (eg. #4797)
If you look at the code you will see that the most of code is just bit of refactoring plus tests for the exiting functionality. Functionally there are just couple of if(shoidIPrintX) { print(x) }

[ijuma]

The KIP for #4797 was https://cwiki.apache.org/confluence/display/KAFKA/KIP-265%3A+Make+Windowed+Serde+to+public+APIs

[onzo-mateuszzakarczemny]

@ijuma Ok then. I'll create a KIP

[gwenshap]

Was a KIP ever created? Can you follow up with #?

[gAmUssA]

@onzo-mateuszzakarczemny Hi Mateusz. Any progress on KIP? Thank you

[Matzz]

Sorry I didn't have time to create a KIP. I'll try to work on that this week.

[gAmUssA]

@Matzz Thank you!

[Matzz]

KIP created - https://cwiki.apache.org/confluence/display/KAFKA/KIP-431%3A+Support+of+printing+additional+ConsumerRecord+fields+in+DefaultMessageFormatter
I need to create a email thread to now.

[Matzz]

Discussion thread - http://mail-archives.apache.org/mod_mbox/kafka-dev/201902.mbox/%3CCAHwv8Gb8Uf5MgDKawwc_OSaxfO916cJ-Mq5YJyCmfj_h3f_adw%40mail.gmail.com%3E

[Matzz]

retest this please

[badaiaqrandista]

Any update on this PR? Can this be merge? @Matzz @rajinisivaram

[stanislavkozlovski]

@badaiaqrandista we cannot merge this as the KIP was not voted in. The vote thread was started on April 12th but has not received any votes.

@Matzz would you be interested in ressurrecting the thread?

[Matzz]

@stanislavkozlovski
Yes sure. Just create new voting thread?

[stanislavkozlovski]

@Matzz yeah!

[badaiaqrandista]

This PR should be closed as it has been superseded by PR 9099 (#9099).

[dajac]

Closing as this was addressed by #9099.