Allow force enabling jemalloc on macOS #27067
Conversation
| @@ -53,7 +61,9 @@ function deps() { | |||
| # output if there is no dependency to any of the specified packages. | |||
| for crate in "${crates[@]}"; do | |||
| # Gather data for the current target, reverse lines, find the dependency hierarchy to the root, reverse lines. | |||
| output=$(cargo tree --workspace --format ':{lib}:{f}' \ | |||
| output=$(cargo tree \ | |||
| $(printf -- "-p %s " "${entrypoints[@]}") \ | |||
There was a problem hiding this comment.
Oh neat, this didn't work when I wrote the script!
There was a problem hiding this comment.
Since Cargo doesn't explicitly support target-specific features, the trick is to introduce a crate of indirection, as described in [https://github.com/rust-lang/cargo/issues/1197#issuecomment-1837113787].
i thought about this but couldnt figure out how it'd work! neat!!
This is awesome, thanks!!
| [target.'cfg(not(target_os = "macos"))'.dependencies] | ||
| mz-alloc = { path = "../alloc", features = ["jemalloc"] } |
There was a problem hiding this comment.
fascinating that cargo correctly overrides the default dep here!
There was a problem hiding this comment.
Yeah, when they implemented target-specific dependencies, they actually did all the hard work to support target-specific features, too. This is why they rolled out that resolver = "2" thing, if you remember. The second version of the resolver is the good one that gets this right.
What they didn't do is the last mile to allow saying [target.'cfg(...)'.features]. Which is a little tragic. All that hard work just to miss the last little bit of plumbing.
The good news is that I'm convinced every target-specific feature can be represented as a target-specific dependency with one or more indirection crates. Just can be annoyingly tricky to figure out how to do the transformation.
Twiddle our crates and Cargo configuration for activating jemalloc to
achieve the following ideal developer experience:
* --no-default-features unequivocally disables jemalloc, regardless of
platform.
* --features=jemalloc unequivocally enables jemalloc, regardless of
platform.
* --default-features chooses the best allocator for the platform: the
system allocator on macOS and jemalloc on Linux.
Since Cargo doesn't explicitly support target-specific features, the
trick is to introduce a crate of indirection, as described in [0].
Here's how it works:
* The `mz-prof`, `mz-prof-http`, and `mz-alloc` crates have a
straightforward setup: they expose a `jemalloc` feature and
conditionally enable or disable jemalloc features based on whether
the `jemalloc` feature is provided or not.
* The `mz-alloc-default` crate depends on the `mz-alloc` crate and
enables the best features for the platform: `jemalloc` on Linux, and
no additional features on macOS. You can think of this crate as
unconditionally enabling the best allocator for the platform.
* The `mz-environmentd` and `mz-clusterd` crates unlock the
conditional behavior. Bothc crates:
1. depend on `mz-alloc-default` by default, and
2. expose a `jemalloc` feature that force enables the
`mz-alloc/jemalloc` feature.
The motivation for all of this is to allow macOS developers to enable
jemalloc upon request, which is useful for certain kinds of testing.
Alternative to MaterializeInc#27041.
Closes MaterializeInc#27041.
[0]: rust-lang/cargo#1197 (comment)
Twiddle our crates and Cargo configuration for activating jemalloc to achieve the following ideal developer experience:
--no-default-featuresunequivocally disables jemalloc, regardless of platform.--features=jemallocunequivocally enables jemalloc, regardless of platform.--default-featureschooses the best allocator for the platform: the system allocator on macOS and jemalloc on Linux.Since Cargo doesn't explicitly support target-specific features, the trick is to introduce a crate of indirection, as described in [0].
Here's how it works:
The
mz-prof,mz-prof-http, andmz-alloccrates have a straightforward setup: they expose ajemallocfeature and conditionally enable or disable jemalloc features based on whether thejemallocfeature is provided or not.The
mz-alloc-defaultcrate depends on themz-alloccrate and enables the best features for the platform:jemallocon Linux, and no additional features on macOS. You can think of this crate as unconditionally enabling the best allocator for the platform.The
mz-environmentdandmz-clusterdcrates unlock the conditional behavior. Both crates:mz-alloc-defaultby default, andjemallocfeature that force enables themz-alloc/jemallocfeature.The motivation for all of this is to allow macOS developers to enable jemalloc upon request, which is useful for certain kinds of testing.
Alternative to #27041.
Closes #27041.
Motivation
Checklist
$T ⇔ Proto$Tmapping (possibly in a backwards-incompatible way), then it is tagged with aT-protolabel.