could not connect kafka with kerberos from v1.46.0

[wxldonnaBJ]

Description

From v1.46.0, when we use , we couldn't connect to broker using kerberos , sarama client return error
kafka: client has run out of available brokers to talk to
we can use v1.45.2 to connect kafka with the same connection info

Versions

Sarama	Kafka	Go
v.1.46.0	docker.io/bitnami/kafka:3.7	1.24.5

Configuration

Logs

logs: CLICK ME

2025-09-01 11:00:18.847471|+0000|DEBUG|[Sarama] 2025/09/01 11:00:18 client/metadata retrying after 250ms... (1 attempts remaining)
|com.sap.abap2|rms kafka housekeeper|371|Write|config.go(98)�
2025-09-01 11:00:18.847518|+0000|DEBUG|[Sarama] 2025/09/01 11:00:18 client/metadata fetching metadata for all topics from broker nonabap-kafka1.datahub.sapcloud.io:30097
|com.sap.abap2|rms kafka housekeeper|371|Write|config.go(98)�
2025-09-01 11:00:18.892942|+0000|DEBUG|[Sarama] 2025/09/01 11:00:18 Completed ApiVersions request to nonabap-kafka1.datahub.sapcloud.io:30097. Broker supports 57 APIs
|com.sap.abap2|rms kafka housekeeper|380|Write|config.go(98)�
2025-09-01 11:00:18.959127|+0000|DEBUG|[Sarama] 2025/09/01 11:00:18 Error while performing GSSAPI Kerberos Authentication: EOF
|com.sap.abap2|rms kafka housekeeper|380|Write|config.go(98)�
2025-09-01 11:00:18.959719|+0000|DEBUG|[Sarama] 2025/09/01 11:00:18 Closed connection to broker nonabap-kafka1.datahub.sapcloud.io:30097 due to SASL v0 auth error: EOF
|com.sap.abap2|rms kafka housekeeper|380|Write|config.go(98)�
2025-09-01 11:00:18.959773|+0000|DEBUG|[Sarama] 2025/09/01 11:00:18 client/metadata got error from broker -1 while fetching metadata: EOF
|com.sap.abap2|rms kafka housekeeper|371|Write|config.go(98)�
2025-09-01 11:00:18.959790|+0000|DEBUG|[Sarama] 2025/09/01 11:00:18 client/metadata no available broker to send metadata request to
|com.sap.abap2|rms kafka housekeeper|371|Write|config.go(98)�
2025-09-01 11:00:18.959796|+0000|DEBUG|[Sarama] 2025/09/01 11:00:18 client/brokers resurrecting 1 dead seed brokers
|com.sap.abap2|rms kafka housekeeper|371|Write|config.go(98)�
2025-09-01 11:00:19.209899|+0000|DEBUG|[Sarama] 2025/09/01 11:00:19 client/metadata retrying after 250ms... (0 attempts remaining)
|com.sap.abap2|rms kafka housekeeper|371|Write|config.go(98)�
2025-09-01 11:00:19.209967|+0000|DEBUG|[Sarama] 2025/09/01 11:00:19 client/metadata fetching metadata for all topics from broker nonabap-kafka1.datahub.sapcloud.io:30097
|com.sap.abap2|rms kafka housekeeper|371|Write|config.go(98)�
2025-09-01 11:00:19.252646|+0000|DEBUG|[Sarama] 2025/09/01 11:00:19 Completed ApiVersions request to nonabap-kafka1.datahub.sapcloud.io:30097. Broker supports 57 APIs
|com.sap.abap2|rms kafka housekeeper|32|Write|config.go(98)�
2025-09-01 11:00:19.317480|+0000|DEBUG|[Sarama] 2025/09/01 11:00:19 Error while performing GSSAPI Kerberos Authentication: EOF
|com.sap.abap2|rms kafka housekeeper|32|Write|config.go(98)�
2025-09-01 11:00:19.317594|+0000|DEBUG|[Sarama] 2025/09/01 11:00:19 Closed connection to broker nonabap-kafka1.datahub.sapcloud.io:30097 due to SASL v0 auth error: EOF
|com.sap.abap2|rms kafka housekeeper|32|Write|config.go(98)�
2025-09-01 11:00:19.317625|+0000|DEBUG|[Sarama] 2025/09/01 11:00:19 client/metadata got error from broker -1 while fetching metadata: EOF
|com.sap.abap2|rms kafka housekeeper|371|Write|config.go(98)�
2025-09-01 11:00:19.317632|+0000|DEBUG|[Sarama] 2025/09/01 11:00:19 client/metadata no available broker to send metadata request to
|com.sap.abap2|rms kafka housekeeper|371|Write|config.go(98)�
2025-09-01 11:00:19.317637|+0000|DEBUG|[Sarama] 2025/09/01 11:00:19 client/brokers resurrecting 1 dead seed brokers
|com.sap.abap2|rms kafka housekeeper|371|Write|config.go(98)�
2025-09-01 11:00:19.317670|+0000|DEBUG|[Sarama] 2025/09/01 11:00:19 Closing Client
|com.sap.abap2|rms kafka housekeeper|348|Write|config.go(98)�

Additional Context

[puellanivis]

What other messages could you maybe have? Usually there will be some sort of message about why the available brokers were not accepted?

[wxldonnaBJ]

Hi @puellanivis
i also find the error from

sarama/gssapi_kerberos.go

Line 254 in 107930b

broker.updateIncomingCommunicationMetrics(bytesRead, requestLatency)

Error while performing GSSAPI Kerberos Authentication: EOF, i l also update the logs in the above issue

[puellanivis]

Huh… intriguing. This could, I suppose, be an issue either with the metadata bundling somehow interacting improperly with GSSAPI authentication, or some other authentication incompatibility introduced by the API versioning… though, I would have suspected tests should have triggered this issue. 🤔

Sadly, I’m not sure I have enough domain knowledge to help out here, but I think there’s enough info for someone with better domain knowledge to dig into what broke.

[dnwe]

Yes Kerberos is a little unusual in Sarama as (for unknown reason) it is pinned to use the SASL v0 flow rather than v1

I think that might be related and the sending of ApiVersionsRequest before the v0 SASL flow is probably the issue here.

@wxldonnaBJ can you test the theory by putting ApiVersionsRequest: false in your sarama.Config struct?

[wxldonnaBJ]

Hi @dnwe, thanks for your suggestion 🙏.
I retested with ApiVersionsRequest: false, and it solved the issue.
However, this feels more like a workaround — since Kerberos was working fine before v1.46.0 without needing this flag.
Until the root cause is fixed, I think the safer approach is to stick with v1.45.2 for Kerberos environments.

[dnwe]

@wxldonnaBJ yes that was just a workaround to test my theory. The reason why it started failing after you upgraded to v1.46.0 is because (as per the release notes) we started always sending ApiVersionsRequest upon first connecting to a broker, but this is incompatible with the v1 SASL flow that Kerberos mode is currently attempting. We will fix this in the next release

[dnwe]

@wxldonnaBJ I put together an experimental branch to use the modern SASL v1 flows for sending GSSAPI auth to the broker, would you be able to give the branch from this PR a try #3279 ?

go get github.com/IBM/sarama@6dfca8b should work to make your go.mod point at this

[wxldonnaBJ]

Hi @dnwe , thanks for your support , the #3279 works well with kerberos in our case