-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Managing the propagation of features across a large workspace is infeasible without tooling support #9094
Comments
Hi! I understand that it can be difficult to manage features in a large workspace. However, this issue is a little unfocused, so it may be difficult for it to have any traction. Perhaps a discussion on one of the forums might be more well suited?
Unless I'm misunderstanding what you are saying, this is how Cargo has always worked. The features for a workspace are resolved for a particular execution of Cargo, and ignores packages not selected. The primary issue for tracking this is #4463. One workaround is something like Mutually exclusive features are tracked in #2980. They are an inherently tricky problem, and I usually recommend avoiding them if at all possible. (I realize that is not always possible, but finding creative solutions to avoiding that can help.) The only tooling I'm aware of for analyzing Cargo workspaces (and features) is cargo guppy which Facebook uses for things like Diem. As part of the v2 work, I also made a bunch of changes to Otherwise, if you have more specific ideas for something not already in the issue tracker here, I'd suggest trying to keep the feature request focused and with more concrete problem statements and suggestions. |
We have similar issues in Substrate(-based projects) and I have a particular idea that might fit here (without knowing the design of cargo well): |
What if we exposed dependency features for conditional compilation regardless of which crate enabled the feature, like See also |
It seems to me that we don't want to go with something automatic:
On the contrary, we want something explicit at the application level:
Here's an imperfect (see drawbacks below) attempt at this:
Here's how a library crate could look like: [dependencies]
serde = { version = "1.0.147", default-features = false, optional = true }
[features]
std = ["serde?/std"]
serde = ["dep:serde", "serde/derive"]
[global-features]
std = ["std"]
serde = ["serde"] If something like #4956 ends up being merged, then we could have a shortcut to map a regular feature to the homonymous global feature and avoid the [features]
std = { dependencies = ["serde?/std"], global = true }
serde = { dependencies = ["dep:serde", "serde/derive"], global = true } And if the feature only enables an optional dependency, then we could also rely on the implicit feature of optional dependencies and avoid the [dependencies]
# Note that it doesn't make sense to have `global = true` without `optional = true`.
# So maybe the latter could be implicit and omitted.
serde = { version = "1.0.147", features = ["derive"], optional = true, global = true } This solution has some drawbacks though:
Footnotes
|
I've created a Pre-RFC for mutually-exclusive, global features which might be pertinent to the discussion. |
I'd agree with discussion notes above about handling global propagating features would be difficult. I'd never expect that to be ever solved in a reasonable timeframe. But the issue title, currently, is propagation across a large workspace. Could the primary use case be solved by specifying the propagation logic in the workspace file, i.e., more or less a syntactic sugar? My use case is propagating a feature like [features]
fizzbuzz = ["dep:fizzbuzz"]
with-prost = [
"my-api/with-prost",
"fizzbuzz?/with-prost",
"dep:prost",
"dep:prost-types",
]
[dependencies]
fizzbuzz = { workspace = true, optional = true }
my-api = { workspace = true }
prost = { workspace = true }
prost-types = { workspace = true } with: [features]
fizzbuzz = ["dep:fizzbuzz"]
with-prost = [
"dep:prost",
"dep:prost-types",
]
[dependencies]
fizzbuzz = { workspace = true, optional = true}
my-api = { workspace = true }
prost = { workspace = true }
prost-types = { workspace = true } given my workspace file has something like: feature-aspects = [
"with-prost",
] The above just means flip on a feature named |
I created a cargo plugin prototype that offers this functionality: https://github.com/dflemstr/cargo-feature-aspect I'm not super happy with the implementation or the command-line API/UI so any feedback is much appreciated! |
Describe the problem you are trying to solve
TiKV is a big workspace.
It has several features that are propagated throughout most of the workspace.
The "pattern" for maintaining these features in each crate is identical,
but humans are unable to maintain the pattern consistently across the workspace.
As a result, the build for any individual crate, under any particular combination of features,
has a good chance of being broken.
This is much exacerbated in the v2 resolver,
which TiKV uses,
because cargo now does feature resolution only based on the crates passed to
-p
,not based on the default binary (or whatever it did before - I don't know exactly).
So the build of multiple crates is usually broken under various combinations of features
even though the main binary or
--all
may build correctly under those feature combinations.Maintaining a working build for each crate under its various feature combinations
is basically impossible without some kind of custom test scripting.
The script I have put together to maintain correct builds for TiKV's crates takes hours to run,
so are infeasible to put under CI.
Here is an example of the pattern we are using to maintain two mutually-exclusive protobuf backends:
Every crate has a default protobuf backend so that the crate compiles by default with
-p
under the v2 resolver.Each of those crates default features must be turned off by the crates that depend on them,
and re-activated by that crate's default features.
The list of crates that have protobuf features is huge, and each crate needs to follow this same pattern perfectly -
any mistakes and some crate in the workspace will stop compiling under some feature combination.
Describe the solution you'd like
Not sure!
My first thought was a static analysis that could understand this pattern across the workspace and tell me I've made a mistake. After reading https://github.com/rust-lang/rfcs/blob/master/text/2906-cargo-workspace-deduplicate.md via #6921 though I'm also wondering if deduplicating feature metadata in some way could help.
Notes
Here's the script I'm using to brute-force test every crate/feature combo:
https://github.com/tikv/tikv/blob/master/scripts/check-build-opts.py
cc @nrc
The text was updated successfully, but these errors were encountered: