Stack overflow killing build with SGX crates #209
Comments
sphw
changed the title
|
Another reproduction: using |
|
Whats the nix Version you are using? It might be worth checking against the latest master branch of Nix as that had some changes that enabled the compiler to do tail-call-optimization in a couple of places. |
|
I'm using |
|
I updated |
|
@andir I was unable to successfully run Nix against my repository for the following versions:
So this issue is not fixed, I guess. |
|
I have ran into this in the wild today. I made a reproducer. In the reproducer you I bundle a patched You can debug what's going on like so: diff --git a/nix/Cargo.nix b/nix/Cargo.nix
index 0edef57..ff00f34 100644
--- a/nix/Cargo.nix
+++ b/nix/Cargo.nix
@@ -5044,6 +5044,7 @@ rec {
mergePackageFeatures =
{ crateConfigs ? crates
, packageId
+ , depth ? 0
, rootPackageId ? packageId
, features ? rootFeatures
, dependencyPath ? [ crates.${packageId}.crateName ]
@@ -5063,8 +5064,12 @@ rec {
assert (builtins.isBool runTests);
let
crateConfig = crateConfigs."${packageId}" or (builtins.throw "Package not found: ${packageId}");
- expandedFeatures = expandFeatures (crateConfig.features or { }) features;
- enabledFeatures = enableFeatures (crateConfig.dependencies or [ ]) expandedFeatures;
+ dbgStr = d: s: if d == 0 then s else " ${dbgStr (d - 1) s}";
+ dbg = s: lib.debug.traceSeq (dbgStr depth s);
+ expandedFeatures_ = expandFeatures depth (crateConfig.features or { }) (dbg "Expanding for ${packageId} (${builtins.toString features})" features);
+ expandedFeatures = dbg "Expanded for ${packageId} (${builtins.toString (builtins.length expandedFeatures_)})" expandedFeatures_;
+ enabledFeatures_ = enableFeatures (crateConfig.dependencies or [ ]) (dbg "Enabling for ${packageId} (${builtins.toString (builtins.length expandedFeatures)})" expandedFeatures);
+ enabledFeatures = dbg "Enabled for ${packageId} (${builtins.toString (builtins.length enabledFeatures_)})" enabledFeatures_;
depWithResolvedFeatures = dependency:
let
packageId = dependency.packageId;
@@ -5093,6 +5098,7 @@ rec {
then cache
else
mergePackageFeatures {
+ depth = depth + 1;
features = combinedFeatures;
featuresByPackageId = cache;
inherit crateConfigs packageId target runTests rootPackageId;
@@ -5157,16 +5163,21 @@ rec {
featureMap is an attribute set which maps feature names to lists of further
feature names to enable in case this feature is selected.
*/
- expandFeatures = featureMap: inputFeatures:
+ expandFeatures = depth: featureMap: inputFeatures:
assert (builtins.isAttrs featureMap);
assert (builtins.isList inputFeatures);
let
+ dbgStr = d: s: if d == 0 then s else " ${dbgStr (d - 1) s}";
+ dbg = s: lib.debug.traceSeq (dbgStr depth s);
expandFeature = feature:
assert (builtins.isString feature);
- [ feature ] ++ (expandFeatures featureMap (featureMap."${feature}" or [ ]));
+ let
+ enables = featureMap."${feature}" or [ ];
+ in
+ (dbg "${feature} --> ${builtins.toString enables}") [ feature ] ++ (expandFeatures (depth + 1) featureMap enables);
outFeatures = lib.concatMap expandFeature inputFeatures;
in
- sortedUnique outFeatures;
+ /*dbg "Expanding ${builtins.toString inputFeatures} with ${builtins.toString (builtins.attrNames featureMap)}"*/ (sortedUnique outFeatures);
/* This function adds optional dependencies as features if they are enabled
indirectly by dependency features. This function mimics Cargo's behaviorIf you do this (being careful about nix laziness), you can spot a cycle like this:
|
|
The below patch should work I think. I'm able to apply this to our rather large tree and it fetches from the binary cache which implies that usual behaviour is completely unchanged. I'll try to cook up a patch. diff --git a/nix/Cargo.nix b/nix/Cargo.nix
index c89bae956..44f8c5a19 100644
--- a/nix/Cargo.nix
+++ b/nix/Cargo.nix
@@ -36514,12 +36514,19 @@ rec {
assert (builtins.isAttrs featureMap);
assert (builtins.isList inputFeatures);
let
- expandFeature = feature:
- assert (builtins.isString feature);
- [ feature ] ++ (expandFeatures featureMap (featureMap."${feature}" or [ ]));
- outFeatures = lib.concatMap expandFeature inputFeatures;
- in
- sortedUnique outFeatures;
+ expandFeaturesNoCycle = oldSeen: inputFeatures:
+ if inputFeatures != []
+ then
+ let
+ feature = builtins.head inputFeatures;
+ tail = builtins.tail inputFeatures;
+ seen = oldSeen // { ${feature} = 1; };
+ enables = builtins.filter (f: !(seen ? "${f}")) (featureMap."${feature}" or []);
+ in [ feature ] ++ (expandFeaturesNoCycle seen (tail ++ enables))
+ # We're done.
+ else [];
+ outFeatures = expandFeaturesNoCycle { } inputFeatures;
+ in sortedUnique outFeatures;
/* This function adds optional dependencies as features if they are enabled
indirectly by dependency features. This function mimics Cargo's behavior |
Fixes nix-community#209. Hopefully. I'm not sure how to properly update all the tests and stuff. For now I manually re-generated the same set of files that was re-generated in some other, recent commit.
Fixes #209. Hopefully. I'm not sure how to properly update all the tests and stuff. For now I manually re-generated the same set of files that was re-generated in some other, recent commit.
Fixes #209. Hopefully. I'm not sure how to properly update all the tests and stuff. For now I manually re-generated the same set of files that was re-generated in some other, recent commit.
Fixes nix-community#209. Hopefully. I'm not sure how to properly update all the tests and stuff. For now I manually re-generated the same set of files that was re-generated in some other, recent commit.
I have a similar issue to the one described in #42, but I suspect the root cause is somewhat different. I am attempting to build a Rust project that makes extensive use of Intel SGX and the Rust SGX SDK. To use the Rust SGX SDK you have to use patched versions of common crates. These patched versions replace
stdwith the special SGX std version. When using some of these crates withcrate2nixI get stack-overflows.I can reproduce the issue in a few different ways, but I have put up the easiest minimal reproduction here: https://github.com/sphw/crate2nix-sgx-test/blob/main/Cargo.nix
When I run
nix-instantiate Cargo.nix -A rootCrate.build -vvvvvv --show-traceI get the following output:I'm not sure the root-cause of this issue. I have run the build with
ulimit -s unlimited, which causes the process to use all the remaining system memory.The text was updated successfully, but these errors were encountered: