Upgrading Kafka to 3.5.1 and issue KAFKA-15353

[jslusher]

• Currently we're on version 0.37.0 of the operator running Kafka 3.4.0
• In order to upgrade the operator past 0.37.0 we need to first upgrade Kafka to a version supported by later versions of the operator
• The available Kafka versions that are supported both by 0.37.0 and any later operator versions are 3.5.0 and 3.5.1
• Both of these versions have a bug (KAFKA-15353) that looks very dangerous if the situation described occurs

From the issue:

This will happen when users trying to upgrade from versions < 3.5.0 to 3.5.0 or later. During the rolling upgrade, there will be some nodes in v3.5.0, and some are not. So, for the node in v3.5.0 will try to build an old version of AlterPartitionRequest. And then, if it happen to have some transient error during the AlterPartitionRequest send, the ISR will be empty and no producers will be able to write data to the partitions.

If I understand correctly, in ZK mode, the leader can send AlterPartition request to empty the ISR. But the leader will continue to serve as the leader.
If the leader stays alive, the previous ISR members except itself will be added back. I guess the partition will stay in this wield "leader not in ISR" state until the leadership is changed.
If the leader fails before any followers can be added to the ISR, an unclean leader election will be required.

• The last part there is what I'm most concerned about. In our Kafka cluster, we're running a lot of CDC tables and their state is critical

I'm wondering if it might be possible to support version 3.5.2 in 0.37.0 or if it's too late and then if anyone might have any insight on that issue. The writeup in the ticket looks frightening, but it's hard to tell what our chances are that we'll run into it. Ideally we would not want to take the risk.

[scholzj]

You can upgrade the operator and Kafka at the same time in one big jump. So if you want to go directly to some newer Kafka version, you can do it that way. There are some more details in the docs. I also had a talk abotu it at the StrimziCon conference: https://youtu.be/5Ji4lFbnaYs

I don't think we have any plans to do new patch releases for Strimzi 0.37. Sorry.

[jslusher]

Thanks for the video. I had read about updating both the operator and Kafka at the same time but thought it safer to upgrade one at a time. Maybe I'm being overcautious with all this. It seems these later versions of Kafka are pretty safe. I would hate to be wrong about that of course. I have a staging test that seemed to have gone fine between 0.37/3.4.0 and 0.42.0/3.7.1 (after I removed some quota configuration that isn't supported between operator versions) so that goes some way in giving me assurance that we can jump versions. Our prod environment has a lot more factors to account for than our staging environment (which is technically not your concern :) ). We're going to attempt the upgrade in prod next and I'll report back one way or the other.

[jslusher]

Reporting back on this for anyone who might find it in the future. We upgraded from Strimzi/Kafka 0.37.0/3.4.0 directly to 0.42.0/3.7.1 in production without any issue whatsoever. We're at 9 brokers at the moment and we mostly use KafkaConnect plugins, but we have a handful of Python and Rust clients as well. The Strimzi documentation is excellent and complete.

For completeness, in staging I tested downgrading all the way back the original 0.37.0/3.4.0 and it took 5 steps:

3.7.1 --> 3.6.0
0.42.0 --> 0.38.0
3.6.0 --> 3.5.0
0.38.0 --> 0.37.0
3.5.0 --> 3.4.0

This was all while leaving inter.broker.protocol.version set at "3.4".