[BUG] libddprof on macOS links to CoreFoundation library, which creates problems for apps using fork

[ivoanjo]

Important note is that right now we don't officially support macOS, so fixing this is low-priority.

Currently we pull in the rustls-native-certs to use the system certificate store for reporting via HTTPS.

As of this writing, on macOS, this means linking to the CoreFoundation library:

│   │   ├── rustls-native-certs v0.6.1
│   │   │   ├── rustls-pemfile v0.2.1
│   │   │   │   └── base64 v0.13.0
│   │   │   └── security-framework v2.6.1
│   │   │       ├── bitflags v1.3.2
│   │   │       ├── core-foundation v0.9.3
│   │   │       │   ├── core-foundation-sys v0.8.3
│   │   │       │   └── libc v0.2.123
│   │   │       ├── core-foundation-sys v0.8.3
│   │   │       ├── libc v0.2.123
│   │   │       └── security-framework-sys v2.6.1
│   │   │           ├── core-foundation-sys v0.8.3
│   │   │           └── libc v0.2.123

(excerpt from cargo tree).

Unfortunately, it seems CoreFoundation, along with other "objective-c" libraries on macOS don't quite like applications that use fork, see https://bugs.ruby-lang.org/issues/14009 and https://stackoverflow.com/questions/52671926/rails-may-have-been-in-progress-in-another-thread-when-fork-was-called .

This manifested with the dd-trace-rb profiler test suite failing and crashing in tests that made use of fork, with errors such as:

objc[90367]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called.
objc[90367]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug

Sometimes the error presented itself differently, as a crash inside the macOS system libraries, e.g.:

       -- C level backtrace information -------------------------------------------
       /Users/ivo.anjo/.rvm/rubies/ruby-2.7.5/lib/libruby.2.7.dylib(rb_vm_bugreport+0x96) [0x1013ce4f6]
       /Users/ivo.anjo/.rvm/rubies/ruby-2.7.5/lib/libruby.2.7.dylib(rb_bug_for_fatal_signal+0x136) [0x101226526]
       /Users/ivo.anjo/.rvm/rubies/ruby-2.7.5/lib/libruby.2.7.dylib(sigsegv+0x52) [0x10133bb32]
       /usr/lib/system/libsystem_platform.dylib(_sigtramp+0x1d) [0x7fff205ddd7d]
       /usr/lib/system/libdispatch.dylib(_dispatch_mgr_queue_push+0x2d) [0x7fff203f946b]
       [0x7fff2268ab99]

I can confirm that disabling the use of rustls-native-certs makes the crashes go away:

diff --git a/ddprof-exporter/Cargo.toml b/ddprof-exporter/Cargo.toml
index e43de43..0485834 100644
--- a/ddprof-exporter/Cargo.toml
+++ b/ddprof-exporter/Cargo.toml
@@ -28,8 +28,8 @@ mime_guess = { version = "2.0", default-features = false }
 http-body = "0.4"
 pin-project = "1"
 rustls = { version = "0.20.4", default-features = false }
-rustls-native-certs = { version = "0.6" }
-hyper-rustls = { version = "0.23", default-features = false, features = ["native-tokio", "http1", "tls12"] }
+# rustls-native-certs = { version = "0.6" }
+hyper-rustls = { version = "0.23", default-features = false, features = ["webpki-tokio", "http1", "tls12"] }
 hex = "0.4"
 hyper-multipart-rfc7578 = "0.7.0"
 ddcommon = { path = "../ddcommon" }
diff --git a/ddprof-exporter/src/connector/mod.rs b/ddprof-exporter/src/connector/mod.rs
index b104e1d..fabd8a4 100644
--- a/ddprof-exporter/src/connector/mod.rs
+++ b/ddprof-exporter/src/connector/mod.rs
@@ -69,12 +69,12 @@ fn build_https_connector(
 fn load_root_certs() -> anyhow::Result<rustls::RootCertStore> {
     let mut roots = rustls::RootCertStore::empty();

-    for cert in rustls_native_certs::load_native_certs()? {
-        let cert = rustls::Certificate(cert.0);
+    // for cert in rustls_native_certs::load_native_certs()? {
+    //     let cert = rustls::Certificate(cert.0);

-        //TODO: log when invalid cert is loaded
-        roots.add(&cert).ok();
-    }
+    //     //TODO: log when invalid cert is loaded
+    //     roots.add(&cert).ok();
+    // }
     if roots.is_empty() {
         return Err(crate::errors::Error::NoValidCertifacteRootsFound.into());
     }

As, again, we don't officially support macOS for now, I've stopped my investigation here, since I can disable this feature manually on the builds I use for the Ruby profiler development. But this is something we need to tackle if we ever want to support macOS (or at least if we want to support macOS on apps that use fork...).

@pawelchcki suggested

I think that leaves a feature flag to enable disable HTTPs as the easiest option. Other than debugging stuff 🙂 and seeing what calls this