Segmentation faults in SSL/TLS connections on Debian Bookworm with OpenSSL 3.0.17

[mensfeld]

Note: There is no action to be taken by librdkafka team. I just wanted to highlight this issue to anyone whom librdkafka would suddenly start crashing (like mine did).

Problem Description

Applications using librdkafka with SSL/TLS connections are experiencing segmentation faults when running on Debian Bookworm systems that have been upgraded to OpenSSL 3.0.17. This affects multiple language bindings and applications that depend on librdkafka for Kafka connectivity.

The issue comes from a regression introduced in OpenSSL 3.0.17 that affects concurrent TLS connections. This is not a librdkafka bug, but an upstream OpenSSL issue that impacts any software using OpenSSL for SSL/TLS connections under high concurrency.

Upstream References:

• OpenSSL issue: concurrent TLS connection segfault in x509 storage (regression on 3.0.17) openssl/openssl#28171
• Debian bug report: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1110254
• Debian will issue a fix: OpenSSL 3.0.17-1~deb12u2 (reverts the problematic commits)

Affected Systems and Evidence

Systems:

• OS: Debian 12 (Bookworm)
• OpenSSL version: 3.0.17-1~deb12u1 (the problematic version)
• Impact: Applications using librdkafka with SSL/TLS connections experience segfaults
• Scope: Language bindings (Ruby, Python, Go, etc.) using librdkafka dynamically linking to openssl

Evidence from the ecosystem:

• Ruby (rdkafka-ruby): Segmentation fault with Debian Bookworm and OpenSSL 3.0.17 karafka/rdkafka-ruby#667
• Karafka Framework: Segmentation fault on rd_kafka_destroy and rd_kafka_consumer_poll karafka/karafka-rdkafka#229 (segfaults in rd_kafka_destroy and rd_kafka_consumer_poll)
• SSL stress testing added: SSL stress testing + testing matrix split into defaults + complementary karafka/karafka-rdkafka#233
• Other affected projects: Prisma (segfault with debian-bookworm and openssl 3.0.17 prisma/prisma#27785), MariaDB (https://jira.mariadb.org/browse/MDEV-37361)

The segfaults occur in libcrypto.so.3 during concurrent SSL/TLS operations with stack traces showing:

segfault at [address] ip [instruction_pointer] sp [stack_pointer] error 4 in libcrypto.so.3

Resolution and Workarounds

FIXED - Debian is releasing OpenSSL 3.0.17-1~deb12u2 which reverts the problematic commits.

Check if your system is affected:

dpkg -l | grep libssl3
# If you see: libssl3 3.0.17-1~deb12u1, you're affected
# If you see: libssl3 3.0.17-1~deb12u2 or later, you're safe

Workarounds until systems are updated:

1. Update OpenSSL (recommended):

   sudo apt update && sudo apt upgrade openssl libssl3

2. Pin to OpenSSL 3.0.16 (temporary):

3. sudo apt install openssl=3.0.16-1~deb12u1 libssl3=3.0.16-1~deb12u1
   sudo apt-mark hold openssl libssl3

4. Use precompiled binaries: If available for your language binding, use precompiled librdkafka binaries built against OpenSSL 3.0.16

[mensfeld]

Closing it. I created this for reference and to provide guidance 🙏