Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix, fix OOM when ArtifactStorageError throws #9014

Merged
merged 1 commit into from
Jul 10, 2024

Conversation

davidfirst
Copy link
Member

This is one of the most bizarre things I have even seen in node. When an error of type ArtifactStorageError is thrown, node exits with out of memory. (I'll attach the stack trace in a comment).
However, if I make the following simple change to this class, it shows the error nicely.

- constructor(private originalError: Error, private component: Component) {  
+ constructor(private originalError: Error, component: Component) {

This change basically tells JS to ignore the component object and don't store it in this error class. The component object is indeed huge and has some circular references. Still, it doesn't explain the strange memory leak here.
I found out that the "factory" property of the component object is the culprit. If I only set a simple object with the component.factory it throws OOM. The factory is the Workspace object.
I stopped the investigation here as it's time consuming. It's probably a node.js bug, but will be very difficult to provide steps to reproduce, so I'll just leave it.

For this bug, the fix is simple, we don't really need the component object, so I'm only passing the ID.

@davidfirst
Copy link
Member Author

The stack trace just in case it'll be useful down the road:


<--- Last few GCs --->

[31187:0x130008000]    34895 ms: Scavenge 4091.5 (4105.6) -> 4089.9 (4105.6) MB, 0.75 / 0.00 ms  (average mu = 0.317, current mu = 0.298) allocation failure;
[31187:0x130008000]    34897 ms: Scavenge 4091.6 (4105.6) -> 4089.9 (4105.9) MB, 1.00 / 0.00 ms  (average mu = 0.317, current mu = 0.298) allocation failure;


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

 1: 0x1004d559c node::Abort() [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
 2: 0x1004d579c node::ModifyCodeGenerationFromStrings(v8::Local<v8::Context>, v8::Local<v8::Value>, bool) [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
 3: 0x10065a95c v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
 4: 0x10082f050 v8::internal::Heap::GarbageCollectionReasonToString(v8::internal::GarbageCollectionReason) [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
 5: 0x100832f04 v8::internal::Heap::CollectGarbageShared(v8::internal::LocalHeap*, v8::internal::GarbageCollectionReason) [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
 6: 0x10082f968 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
 7: 0x10082d6f0 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
 8: 0x100824344 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
 9: 0x100824ba4 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
10: 0x100809c14 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
11: 0x100bf16dc v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
12: 0x100f50c44 Builtins_CEntry_Return1_ArgvOnStack_NoBuiltinExit [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
13: 0x100eca8a8 Builtins_GrowFastSmiOrObjectElements [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
14: 0x106af94c8
15: 0x106adac40
16: 0x106af91a0
17: 0x106af91a0
18: 0x106af91a0
19: 0x106af91a0
20: 0x106af91a0
21: 0x106af91a0
22: 0x106af91a0
23: 0x1064a7e84
24: 0x100f68b3c Builtins_ArrayForEach [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
25: 0x106afb614
26: 0x106af85dc
27: 0x1064a7e84
28: 0x100f68b3c Builtins_ArrayForEach [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
29: 0x106afb614
30: 0x106af85dc
31: 0x10691cca8
32: 0x10691db7c
33: 0x106af85dc
34: 0x106af91a0
35: 0x106af91a0
36: 0x106af91a0
37: 0x106af91a0
38: 0x106af91a0
39: 0x106af91a0
40: 0x106af91a0
41: 0x106af91a0
42: 0x106af91a0
43: 0x1064a7e84
44: 0x100f68b3c Builtins_ArrayForEach [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
45: 0x106afb614
46: 0x106af85dc
47: 0x10691cca8
48: 0x100f73ee4 Builtins_ArrayMap [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
49: 0x1064a6e74
50: 0x1064a7b14
51: 0x106af5670
52: 0x1064a66b0
53: 0x100f68b3c Builtins_ArrayForEach [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
54: 0x1067163f0
55: 0x10673b308
56: 0x106716f1c
57: 0x100f68b3c Builtins_ArrayForEach [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
58: 0x1067163f0
59: 0x10673b308
60: 0x106716f1c
61: 0x100f68b3c Builtins_ArrayForEach [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
62: 0x1067163f0
63: 0x10673b308
64: 0x106716f1c
65: 0x100f68b3c Builtins_ArrayForEach [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
66: 0x100ec83e4 Builtins_InterpreterEntryTrampoline [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
67: 0x10673b308
68: 0x100ec83e4 Builtins_InterpreterEntryTrampoline [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
69: 0x100f68b3c Builtins_ArrayForEach [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
70: 0x100ec83e4 Builtins_InterpreterEntryTrampoline [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
71: 0x1064993cc
72: 0x10673b308
73: 0x100ec83e4 Builtins_InterpreterEntryTrampoline [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
74: 0x100f68b3c Builtins_ArrayForEach [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
75: 0x100ec83e4 Builtins_InterpreterEntryTrampoline [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
76: 0x1064993cc
77: 0x10673b308
78: 0x100ec83e4 Builtins_InterpreterEntryTrampoline [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
79: 0x100f73ee4 Builtins_ArrayMap [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
80: 0x100ec83e4 Builtins_InterpreterEntryTrampoline [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
81: 0x100ec83e4 Builtins_InterpreterEntryTrampoline [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
82: 0x1064993cc
83: 0x106499404
84: 0x106499404
85: 0x106499404
86: 0x106499404
87: 0x1064aea24
88: 0x100f68b3c Builtins_ArrayForEach [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
89: 0x1067163f0
90: 0x1064aea24
91: 0x100f68b3c Builtins_ArrayForEach [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
92: 0x1067163f0
93: 0x1064aea24
94: 0x100f68b3c Builtins_ArrayForEach [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
95: 0x100ec83e4 Builtins_InterpreterEntryTrampoline [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
96: 0x106ae0d78
97: 0x100ec83e4 Builtins_InterpreterEntryTrampoline [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
98: 0x1065c9c9c
99: 0x100ec83e4 Builtins_InterpreterEntryTrampoline [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
100: 0x100ec83e4 Builtins_InterpreterEntryTrampoline [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
101: 0x100f68b3c Builtins_ArrayForEach [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
102: 0x100ec83e4 Builtins_InterpreterEntryTrampoline [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
103: 0x100ec83e4 Builtins_InterpreterEntryTrampoline [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
104: 0x100ec83e4 Builtins_InterpreterEntryTrampoline [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
105: 0x100ec83e4 Builtins_InterpreterEntryTrampoline [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
106: 0x100ec83e4 Builtins_InterpreterEntryTrampoline [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
107: 0x100eff130 Builtins_AsyncFunctionAwaitRejectClosure [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
108: 0x100fad118 Builtins_PromiseRejectReactionJob [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
109: 0x100eeecac Builtins_RunMicrotasks [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
110: 0x100ec63f4 Builtins_JSRunMicrotasksEntry [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
111: 0x10079c4d0 v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
112: 0x10079c9bc v8::internal::(anonymous namespace)::InvokeWithTryCatch(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
113: 0x10079cb98 v8::internal::Execution::TryRunMicrotasks(v8::internal::Isolate*, v8::internal::MicrotaskQueue*) [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
114: 0x1007c3d64 v8::internal::MicrotaskQueue::RunMicrotasks(v8::internal::Isolate*) [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
115: 0x1007c4500 v8::internal::MicrotaskQueue::PerformCheckpoint(v8::Isolate*) [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
116: 0x100404c64 node::InternalCallbackScope::Close() [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
117: 0x10040501c node::InternalMakeCallback(node::Environment*, v8::Local<v8::Object>, v8::Local<v8::Object>, v8::Local<v8::Function>, int, v8::Local<v8::Value>*, node::async_context) [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
118: 0x10041b4b8 node::AsyncWrap::MakeCallback(v8::Local<v8::Function>, int, v8::Local<v8::Value>*) [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
119: 0x1004db00c node::fs::FSReqCallback::Resolve(v8::Local<v8::Value>) [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
120: 0x1004db75c node::fs::AfterNoArgs(uv_fs_s*) [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
121: 0x1004d04e8 node::MakeLibuvRequestCallback<uv_fs_s, void (*)(uv_fs_s*)>::Wrapper(uv_fs_s*) [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
122: 0x100ea239c uv__work_done [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
123: 0x100ea5dec uv__async_io [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
124: 0x100eb7ec4 uv__io_poll [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
125: 0x100ea63b0 uv_run [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
126: 0x100405754 node::SpinEventLoopInternal(node::Environment*) [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
127: 0x100515c6c node::NodeMainInstance::Run(node::ExitCode*, node::Environment*) [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
128: 0x100515a08 node::NodeMainInstance::Run() [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
129: 0x10049f718 node::Start(int, char**) [/Users/davidfirst/.nvm/versions/node/v20.11.1/bin/node]
130: 0x187a550e0 start [/usr/lib/dyld]
[1]    31187 abort      bit3 snap --unmodified --build

@davidfirst davidfirst merged commit c23ff97 into master Jul 10, 2024
11 checks passed
@davidfirst davidfirst deleted the oom-ArtifactStorageError branch July 10, 2024 15:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants