Consider switching default Kafka version to at least 2.1

[ijuma]

The default Kafka version was set to 1.0 via #1787. This was a huge improvement over the previous default and the hope was that proper api versions support would be supported some time later.

2.5 years have passed and the default remains 1.0 if I understand correctly (please correct me if this is wrong).

I am working on a KIP to remove older protocol versions in Apache Kafka 4.0 and the proposed baseline is Apache Kafka 2.1:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-896%3A+Remove+old+client+protocol+API+versions+in+Kafka+4.0

The KIP is still under discussion (i.e. it hasn't been approved yet), but I wanted to:

1. Give an early heads up to this project since it would impact your users with the current default
2. Get your input on whether bumping the default Kafka version in the near term is something you would consider

Thanks!

[dnwe]

@ijuma thanks for getting in touch and the great writeup of the KIP and 2.1 sounds like a good baseline. Bumping the default KafkaVersion isn't a problem at all, but as you note our protocol version implementations aren't in sync with the Java client anyway

I'd suggest we sync this up with bumping Sarama to a 2.x and we can work on ensuring that we have implemented all of the protocol baselines to match the Java client at 2.1 (V7 MetadataRequest etc.) and to match what an ApiVersionRequest to a Kafka 2.1 broker returns

[ijuma]

@dnwe that sounds great if the project has the cycles to update all of the protocol baselines as well.

[ijuma]

Hi @dnwe. Is Sarama 2.x on the horizon or would it be better to do something more incremental in the meantime to avoid a situation where Sarama users with reasonably recent clients break by default with Apache Kafka 4.0?

[dnwe]

@ijuma whilst on holiday I spent a bit of time looking into the minimum we'd need to do in order to meet this i.e., if we change the default baked version to sarama.V2.1.0.0 then a) what default request versions do we support, but are not sending correctly atm and b) what protocol versions are not even implemented that need implementing so this is WIP

[ijuma]

Awesome to see the progress @dnwe!

[dnwe]

@ijuma 😀

Is there a branch or patchset anywhere for apache/kafka that has a draft implementation of KAFKA-14560? It would be useful if we could configure a broker and set the minimum APIKey versions somehow so we can test ourselves out against it

[ijuma]

@dnwe not yet, I'll let you know once that exists.

[github-actions]

Thank you for taking the time to raise this issue. However, it has not had any activity on it in the past 90 days and will be closed in 30 days if no updates occur.
Please check if the main branch has already resolved the issue since it was raised. If you believe the issue is still valid and you would like input from the maintainers then please comment to ask for it to be reviewed.

[ijuma]

Is there a branch or patchset anywhere for apache/kafka that has a draft implementation of KAFKA-14560? It would be useful if we could configure a broker and set the minimum APIKey versions somehow so we can test ourselves out against it

The 4.0 branch of Apache Kafka now contains the KIP-896 change:
apache/kafka#18291

[dnwe]

Experimenting with 4.0 in the FV under this draft PR: #3045

[dnwe]

@ijuma ok so everything looks good when doing the FV runs against Kafka 4.0 #3045 so from a producer + consumer perspective we're up-to-scratch

I think our FV test coverage on the admin client is much poorer by comparison, but I'll try and bolster that up to give us further confidence there before 4.0 final release

We'll probably use this milestone as a good point to put v1 of Sarama into maintenance mode and spinoff a /v2 branch where we can more freely make some major changes such as finally landing support for KIP-429 (incremental coop balance) in the consumer, drop the manual protocol configuration (sarama.Version) and actually use the ApiVersionsResponse values, rework the admin client so it doesn't expose protocol internals, rework the producer, etc. etc.

[ijuma]

Sounds good.

Replacing the manual protocol configuration with API versions would be a huge win. My experience is that a significant number or users (including users of beats and other software that uses Sarama) never changes sarama.Version and hence ends up using older protocol versions for no good reason. A secondary negative side effect is that client software name & version aren't sent to the server and hence users find it harder to manage old client versions and making sure they get upgraded.

[ijuma]

A secondary negative side effect is that client software name & version aren't sent to the server and hence users find it harder to manage old client versions and making sure they get upgraded.

One quick note on this, I think there's a quick fix for this that can be done in the existing branch:

#2028 (comment)