Segmentation fault on `rd_kafka_destroy` and `rd_kafka_consumer_poll`

[joshuay03]

Hello, we ran into some segfaults in production at @buildkite.

When this first happened, we were running:

karafka (2.5.0)
karafka-core (2.5.2)
karafka-rdkafka (0.19.5)

and our base image was: ruby:3.4.5-slim-bookworm@sha256:f89ae2834adb08a456be48ac9d7582230e23bfb7cb84b1ff014b206426a20570.

Stack trace: buildkite_karafka_segfault_1.txt

We thought it might have something to do with the recently reported issues with OpenSSL 3.0.17, which we found we were running in our final production image. So, we decided to bump karafka-rdkafka to use the newly released precompiled binaries, which are compiled against OpenSSL 3.0.16. We are now running:

karafka (2.5.0)
karafka-core (2.5.5)
karafka-rdkafka (0.20.1-x86_64-linux-gnu)

However, with these versions we run into a different segfault.

Stack trace: buildkite_karafka_segfault_2.txt

Please let me know if you need any further information to assist with debugging.

[mensfeld]

@joshuay03 downgrade openssl and libssl: karafka#667

i am investigating this meanwhile (but it is not an easy one)

[mensfeld]

Yeah the first segfault is exactly the null reference caused by openssl, about the second one, investigating.

[mensfeld]

also it was already ack by the Debian team: openssl/openssl#28171

[joshuay03]

Thank you for the ultra-quick response. It seems like downgrading both openssl and libssl should help, we'll do that and report back!

[mensfeld]

@joshuay03 it does help. I already confirmed that with other users. Since I do not know the nature of the second bug (may be related in which case it would mean I did not manage to fully isolate my openssl build from the OS) you may resolve to ruby platform in your Gemfile to build librdkafka (like in 0.19).

[mensfeld]

FYI if I can reproduce this locally (which I cannot at the moment 😓 ) I will add this to the integration test suite. Also bookworm as a platform is already partially enrolled in the testing matrix since Sunday.

[joshuay03]

@joshuay03 it does help.

And indeed it did, no more segfaults 😄 Thank you for your help, I'm happy if you want to close this off 👍🏽

[mensfeld]

@joshuay03:

1. did it help with 0.20 or 0.19?
2. can you share your cluster setup (here or in private) and auth details so I can reproduce it better?
3. can you tell me a bit about your cluster?

[joshuay03]

1. did it help with 0.20 or 0.19?

0.20.1. We opted not to downgrade karafka and instead downgrade and pin libssl and openssl.

2. can you share your cluster setup (here or in private) and auth details so I can reproduce it better?

3. can you tell me a bit about your cluster?

MSK Cluster running Kafka 3.7.x with 6 brokers across 3 AZs, each a kafka.m7g.2xlarge. No custom configuration. Authentication is via SASL/SCRAM + Secrets Manager using SASL_SSL security protocol with SCRAM-SHA-512 mechanism.

Production Karafka config details:

• Using cooperative-sticky partition assignment strategy
• Snappy compression
• Concurrency: 10 workers
• Consumer persistence enabled
• Connection pool of 16 transactional producers
• Multiple consumer groups consuming from the same topics (e.g. 1 topic has 3 different consumer groups)
• Using Avro deserialization with schema registry
• Dead letter queues configured for some topics
• Topics have varying partition counts (2-32 partitions)

[mensfeld]

Going to close this one now. The openssl update should be released, mitigation is known and the monitoring CI setup has been added.