Unable to Catch V8 OOM Exceptions

[mattezell]

Good day! First off, thank you very much to the authors of this awesome project!

I am hoping that you can assist me, as we are encountering an issue that we aren't quite sure how to get around - and it's an issue that proving to be a showstopper for our desired design which leverages ClearScript+V8.

Essentially, we're seeking a way to 'secure' our V8 engine in a way that gracefully handles all potential crashes or excessive resource consumption when executing unsecure/unvalidated scripts.

As part of this effort, I've read nearly every discussion in the issues here, as well as the various documentation resources. These two discussions have proven particularly useful:
#101
#84

Currently, my main problem is not being able to catch exceptions bubbling up from the unmanaged V8 DLLs. I am also struggling to figure out the why with one particular bit of code works as expected (allowing us to interrupt execution) and another does not...

From one of your other examples, I found generally the following code as a demonstration for how to gracefully handle excessive memory consumption issues:

try
{
    var runtime = new V8Runtime(new V8RuntimeConstraints() { MaxOldSpaceSize = 2048 });
    runtime.MaxHeapSize = (UIntPtr)(1024UL * 1024 * 1024);
    var engine = runtime.CreateScriptEngine();
    engine.Evaluate("var leak = [];");
    for (; ;) 
    {
        engine.Execute("(function() {const x = []; for (let i = 0; i < 10000; i++) x[i] = Math.random(); leak.push(x);})();");
    }
}
catch (Exception ex)
{
    Console.WriteLine(ex.Message);
}

The above example works perfectly - we are able to halt script execution when we exceed ~1GB and handle that situation gracefully.

Now consider the following, where our looping has been moved out of the CLR and into the V8 engine:

try
{
    var runtime = new V8Runtime(new V8RuntimeConstraints() { MaxOldSpaceSize = 2048 });
    runtime.MaxHeapSize = (UIntPtr)(1024UL * 1024 * 1024);
    var engine = runtime.CreateScriptEngine();
    engine.Evaluate("var leak = [];");
    engine.Evaluate("while(true){leak.push('MORE DATA!')}");
    Console.WriteLine("Hello World!");
}
catch (Exception ex)
{
    //Never hit...
    Console.WriteLine(ex.Message);
}
catch 
{
    //Never hit...
    Console.Write("NON-CLR Exception!");
}

As commented, thus far I've found it impossible to trap the V8 exception - neither as a CLR exception nor as a NON-CLR... Instead, the following message is displayed in the parent console a second before the parent application itself terminates with a -2147483645 (0x80000003) code and the following message:

<--- Last few GCs --->

[55916:0000023543A12010]     3735 ms: Mark-sweep (reduce) 1434.8 (1436.5) -> 861.0 (862.6) MB, 891.4 / 0.0 ms  (average mu = 0.354, current mu = 0.345) low memory notification GC in old space requested
[55916:0000023543A12010]     4289 ms: Mark-sweep (reduce) 861.0 (862.6) -> 861.0 (862.6) MB, 554.0 / 0.0 ms  (average mu = 0.227, current mu = 0.000) low memory notification GC in old space requested


<--- JS stacktrace --->


#
# Fatal javascript OOM in invalid array length
#


C:\me\w\JSEngine\JSEngine\bin\Debug\netcoreapp3.1\JSEngine.exe (process 55916) exited with code -2147483645.
To automatically close the console when debugging stops, enable Tools->Options->Debugging->Automatically close the console when debugging stops.
Press any key to close this window . . .

I'm assuming that the 2nd code sample, where the loop has been moved into JavaScript, is preventing the V8 Garbage Collector from doing its job, which is why the graceful MaxOldSpaceSize+MaxHeapSize approach isn't working... Is this a correct assumption?

Do you know of a way that this can be worked around - so that an internal V8 exception may be caught and handled by the caller?

Any assistance is appreciated greatly!

Again, thank you very much for an awesome project!
-Matt

[ClearScriptLib]

Hi Matt,

Thank you very much for your kind words!

Unfortunately, a V8 OOM condition is fatal to the host process. Sorry if we didn't make that clear in earlier discussions. V8 itself, by design, can't recover from an OOM condition. Recovery requires checks and tracking code that would degrade V8's performance, however slightly, so the V8 team decided years ago to drop support for OOM recovery.

Instead, when V8 detects an OOM condition or exceeds one of its internal limits, it kills the host process immediately. No exception is thrown; the process is simply aborted as if memory corruption or a security breach had been detected.

So if you must run untrusted JavaScript code in V8 with perfect safety, running it in a dedicated process is the only way. That's why Chromium has a multi-process architecture. Early V8 versions aimed to provide support for in-process sandboxing, but the quest for performance supremacy pushed that goal out of scope long ago.

Because there can be no recovery from V8 OOM conditions, the best the host can do is try to avoid them. ClearScript offers heap and stack monitoring that can interrupt script execution before V8 decides to kill the process (see MaxHeapSize, MaxStackUsage). These facilities operate by sampling the heap and stack periodically. They work reasonably well when resource consumption grows steadily, but they can't always catch sudden spikes.

V8 also has numerous internal limits that crash the process when exceeded. Your array leak test is an example. It turns out that arrays in V8 are limited to a relatively low maximum length (roughly 268 million elements on a 64-bit system), and because V8 expands arrays by reallocating them to geometrically increasing internal lengths, it can exceed that limit long before it hits an actual OOM condition. That's why heap size monitoring is ineffective here, and why the error message mentions an "invalid array length".

Please send any additional questions or thoughts our way.

Thanks!

[mattezell]

Thank you very much for your quick response, @ClearScriptLib!

I have had success accomplishing most of what I'd set out to accomplish using the built-in heap/stack monitoring in combination with Interrupt(), largely thanks to your guidance provided in other threads on related topics - so, again, thank you for going above and beyond to help users out (as I understand this isn't a support forum / Stackoverflow)!

To the point of "other internal limits"... Thank you for clarifying that this OOM error is somewhat different than others - this exposes some of my ignorance with regards to other potential pitfalls when leveraging this V8 build...

Would you happen to know of a resource that calls some of these V8 limitations out, or perhaps otherwise have some guidance that you can share on this topic?

Do you know if these V8 limitations unique to this build/release of V8, or are these limitations universal to most/all V8 releases (as I suddenly find myself concerned about other unknowns, while struggling to nail down what I should be searching for)?

I've found the official V8 Blog (https://v8.dev/blog) to be a treasure trove of useful internal information, though I'm struggling to extract what feels to be even a semi-comprehensive list of potential gotchas such as this 'max array length' one that I encountered by pure chance.

Any help is greatly appreciated!

Again, thank you so much for this project and your support!
-Matt

[ClearScriptLib]

Hi Matt,

thank you for going above and beyond to help users out (as I understand this isn't a support forum / Stackoverflow)!

No problem at all. Being able to engage directly with users is one of the nice things about maintaining a small project.

Would you happen to know of a resource that calls some of these V8 limitations out, or perhaps otherwise have some guidance that you can share on this topic?

V8's top-notch performance and rapid evolution is a double-edged sword. There's a new stable release approximately every six weeks, the codebase typically having changed massively during that time.

In our V8 experience of several years, there have been huge changes in V8's API, build procedure, and implementation. Some of these have been announced in the official blog and/or the v8-users group, but details such as nonstandard internal limits aren't often mentioned. And even when a host is aware of a hidden limit, it usually has no way to work around it.

Recently the V8 team has been taking steps to make V8 a bit less draconian when it comes to limits. Specific string and array operations now throw exceptions instead of killing the process. There are new callbacks that allow hosts to relax heap constraints at the last possible moment in order to avoid a crash.

Unfortunately these measures don't provide full protection. Performance is still V8's top priority, and Chromium is still V8's primary consumer. And since Chromium is designed to withstand tab crashes, the V8 team isn't likely to sacrifice performance in favor of crash prevention.

Sorry we don't have a bulletproof solution for you. If you have additional questions about V8, consider posting them on v8-users or v8-dev. Another useful resource is V8's bug database.

Good luck!

[mattezell]

I completely understand! Thanks again for your assistance! Have a great weekend!

-Matt