Sarama 1.41.0+ incompatibility with V2_4_0_0 and higher

[Manicben]

Version & Environment

Redpanda version: (use rpk version): 21.11.22 (also tested with 23.3.10)

IBM/Sarama version: 1.41.0+

What went wrong?

When upgrading from Sarama 1.40.1 to 1.41.0 or higher, with sarama.Config.Version set to anything higher than V2_3_0_0 (in our case we had it set to V2_8_0_0 as we use Admin API features available from Kafka 2.8 onwards), creating new clients against Redpanda no longer works. The error returned by Sarama is kafka: client has run out of available brokers to talk to: EOF.

Whilst Redpanda is API compatible with Kafka, we believe that it doesn't support all API versions. The change of behaviour in Sarama 1.41.0 is described as:

this version of Sarama has had a big overhaul in its adherence to the use of the right Kafka protocol versions for the given Config Version.
This means that in Sarama's eyes, Redpanda is only compatible with Kafka 2.3 API versions.

What should have happened instead?

Redpanda should ideally support newer API versions, along with older API versions (up until 2.1, if Redpanda plans on matching Apache Kafka as per KIP-896. We shouldn't need to change our application code or make API version compromises to be able to use Redpanda.

How to reproduce the issue?

1. Use Sarama 1.41.0 or higher
2. Set sarama.Config.Version to V2_4_0_0 or higher
3. Connect to Redpanda using any client, such as a sarama.NewClusterAdmin, observing the EOF error returned

JIRA Link: CORE-2022

[github-actions]

This issue hasn't seen activity in 3 months. If you want to keep it open, post a comment or remove the stale label – otherwise this will be closed in two weeks.

[github-actions]

This issue was closed due to lack of activity. Feel free to reopen if it's still relevant.

[stevenh]

This appears to still be an issue.

[dotnwat]

thanks @stevenh we'll take a look. we're tracking this internally in jira, but we'll leave this public ticket open.

[github-actions]

This issue hasn't seen activity in 3 months. If you want to keep it open, post a comment or remove the stale label – otherwise this will be closed in two weeks.

[piyushredpanda]

Let me chase this down internally now.