Bound eval recursion with a catchable RangeError instead of a native stack overflow or an endless thread hop

[lahma]

Defect

Options.Constraints.StackOverflowGuard (on by default) is meant to turn deep native recursion into a catchable RangeError. Recursion through eval went around it. Each of these ended the host process with 0xC00000FD (native stack overflow) on net10.0, net8.0 and net472, on a 1 MB thread with default options:

var s = 'eval(s)'; eval(s);                        // direct eval
var s = '(0, eval)(s)'; (0, eval)(s);              // indirect eval
var s = 'eval?.(s)'; eval?.(s);                    // optional-call eval (indirect, dispatched like a direct one)
var s = '[s].forEach(eval)'; [s].forEach(eval);    // eval as a callback
var o = {}; Object.defineProperty(o, Symbol.hasInstance, { value: eval });
var s = 's instanceof o'; s instanceof o;          // eval reached with no call expression at all

On the MaxExecutionStackCount lane, direct eval and eval?.() recursion never threw. They hopped to a fresh thread-pool thread each time the stack ran low, left the thread below blocked, and never reached the limit. In the harness this was still running after 20 s with 57–66 threads on every TFM.

new Function(s)() and Function.prototype.constructor(s)() re-entry was already guarded on both lanes, because the constructed function is entered through a normal, probed call. CreateDynamicFunction evaluates nothing itself, so it gets no probe. A ShadowRealm.prototype.evaluate recursion also already raised a catchable error on main, so this PR does not change it.

Mechanism

Evaluating eval's source re-enters the interpreter (PerformEval → JintScript.Execute) without entering a function. None of the function-entry probes in ScriptFunction sees it. The routes that reach EvalFunction.Call have no dispatcher probe above them either: JintCallExpression's native branch, a callback dispatch, and InstanceofOperator's direct ICallable.Call.

On the count lane, the only check is TryEnterOnCurrentStack at the call expression, and it compares CallStack.Count with the limit. JintCallExpression.HandleEval is the one call a call expression dispatches without pushing a call-stack frame, so a recursion made only of direct evals never grew the count. That means hop, hop, hop.

Fix

1. EvalFunction.PerformEval calls StackGuard.EnsureStackHeadroom() before it parses. This is the same function-entry backstop the four ScriptFunction entries use, so it is off on the count lane. EvalFunction.Call is unchanged, so the IL pin ExactlyTheInteropAndForwardingFunctionsProbeTheNativeStack is untouched.
2. StackGuard._unframedEvalDepth counts the evals HandleEval dispatches without a frame (direct eval and eval?.()). TryEnterOnCurrentStack adds it to CallStack.Count, so on the count lane a direct eval counts as the call it is, the same way an indirect eval, whose EvalFunction the call expression pushes, always has. It is a counter rather than a pushed frame because a frame is observable in error.stack, the debugger's call stack and MaxRecursionDepth.

Why the function-entry gate and not EnsureNativeStackHeadroom (decided by measurement)

I prototyped the other option first: the native-backstop probe, which stays on under the count lane. It bounded the pathological rows, but only by racing the call expression's hop. The probe sits a few frames below the hop, so whichever finds the stack low first decides, and that moves with JIT frame sizes. In the harness, direct eval on the count lane ended with 8 threads (probe won, no hop) while eval?.() hopped to 16. It also broke what the lane exists for. A finite function f(n) { return n === 0 ? 0 : eval('f(n - 1)') + 1; } f(3000) with MaxExecutionStackCount = 100_000:

	main	native-gated probe	this PR
net10.0	3000 (hops)	RangeError	3000 (hops)
net472	3000	3000	3000

The function-entry gate plus the count is deterministic: the count lane hops until the count, which now includes unframed evals, passes the limit, and then throws.

Residual, documented rather than new. On the count lane, eval reached with no call expression in the loop (the @@hasInstance row) still overflows. That is the gap MaxExecutionStackCount's docs already state for every non-call route into a function body. The harness shows the same crash on main for an accessor recursion and for a script-function @@hasInstance recursion on that lane. ValidateSecurityConfiguration already flags the lane (StackOverflowGuardShadowed).

Fail-before / pass-after

New cases in Jint.Tests.PublicInterface/HostNativeRecursionGuardTests.cs:

• ARecursionThroughEvaluatedSourceRaisesACatchableErrorAndTheEngineRecovers: 7 rows on the default lane. Rows: direct, indirect, eval?.(), forEach(eval), @@hasInstance, new Function, Function.prototype.constructor.
  • Unfixed engine, net10.0: Zero tests ran … Exit code: -1073741571 Error output: Stack overflow. at Jint.Engine.EvalDeclarationInstantiation … at Jint.Native.Function.EvalFunction.PerformEval … at Jint.Runtime.Interpreter.Expressions.JintCallExpression.HandleEval
  • Unfixed engine, net472: Exit code: -1073741571 Process is terminated due to StackOverflowException.
• OnTheExecutionStackCountLaneARecursionThroughEvaluatedSourceStopsAtTheCount: 5 rows with MaxExecutionStackCount = 500 and TestBudgets.WedgeCeiling as the join ceiling. Unfixed, net10.0: failed … ("direct eval", …) (2m 00s 009ms) 'direct eval' did not stop at the configured count within 00:02:00; the lane is hopping threads without bound, the same for optional-call eval, and the other 3 rows passed (they were bounded already).
• OnTheExecutionStackCountLaneAFiniteRecursionThroughEvalDeeperThanTheThreadReturns: 2 rows. They pin the lane decision. I broke it on purpose by swapping the probe to EnsureNativeStackHeadroom, and the direct-eval row failed on net10.0 with JavaScriptException : Maximum call stack size exceeded.

With the fix, the class passes 55/55 on net10.0, net8.0 and net472.

Verification (Release, base 85571424b)

Suite	net10.0	net8.0	net472
Jint.Tests	12,802 total / 0 failed / 5 skipped	12,802 / 0 / 5	8,850 / 0 / 6
Jint.Tests.PublicInterface	3,833 / 0 / 21	3,823 / 0 / 21	3,040 / 0 / 21
Jint.Tests.Test262	102,740 total, 102,651 passed / 0 failed / 89 skipped (same as the control on main)

Cost, and the rows to gate

Per eval invocation there is one inlined bool test plus one out-of-line TryEnsureSufficientExecutionStack FCall, and on direct eval an int increment and decrement in a try/finally. Per eval, the existing work already includes a source-keyed cache lookup, ParserOptions equality, an environment, EvalDeclarationInstantiation and an execution-context push and pop. Nothing is added to calls, statements or new Function. I ran no benchmark. The rows that exercise the changed path are:

• DromaeoBenchmark dromaeo-core-eval
• EngineComparisonBenchmark dromaeo-core-eval-modern
• EvalCacheBenchmarks.EvalSameSource, EvalCacheBenchmarks.EvalDistinctSources (smallest eval bodies, so most sensitive)
• EvalExecutionBenchmarks.EvalHotReusedEngine, EvalExecutionBenchmarks.EvalFreshEngine

What could break

Only an engine on the MaxExecutionStackCount lane that recurses through direct eval sees a change. Each level now counts the function and the eval, as an indirect eval always did, so such a recursion reaches the configured count at about half the depth. The default lane only changes where the process used to die. I added no migration-guide section. Say if you want one.

Follow-up to #4076 / #4078 / #4165 / #4176.

🤖 Generated with Claude Code