Backport #4172, #4184, #4187 to 4.x: stop instanceof and eval recursions from killing the host

[lahma]

This backports three fixes from main to 4.x. Each one stops a recursion that script controls from ending the host process, or from running on new threads forever. Each commit is a cherry-pick of the main PR, adapted to 4.x. No public API changes and no default changes. The public API Verify snapshots are byte-identical.

4.x commit	main PR (commit)	What it fixes
7b0649bc5	#4172 (5a7534069)	instanceof over a bound function now asks every bound link for its own @@hasInstance, as OrdinaryHasInstance step 2 requires. Before, it jumped to the innermost target, so a bound class's static [Symbol.hasInstance] was never called, no lookup was observable, and a bound proxy target was refused as "not callable". The walk is still a loop. A link that inherits %Function.prototype[@@hasInstance]% is one step of the loop. Any other method is called behind a native stack probe.
df8395454	#4184 (e880b428a)	JsValue.InstanceofOperator probes the native stack before it calls a target's own @@hasInstance, unless that method is the intrinsic. Without the probe, class C { static [Symbol.hasInstance](v) { return v instanceof C; } }, Function.prototype.call used as the method, or eval used as the method ended the process.
49ccd5e23	#4187 (553cc7c45)	PerformEval probes before it parses. On the MaxExecutionStackCount lane, a direct eval(…) or eval?.(…) now counts as a call. Before, var s = 'eval(s)'; eval(s) and its indirect, optional-call and callback forms ended the process. On the count lane, the direct and optional forms hopped to a new thread at every exhaustion and never threw.

Adapted for 4.x

• StackOverflowGuard is opt-in on 4.x (main turns it on by default). Every new probe is gated on it, like every existing 4.x probe:

  • EnsureNativeStackHeadroom in the instanceof paths is gated on the guard flag alone.
  • EnsureStackHeadroom in PerformEval is gated on the guard, and is off on the count lane.

  So the fixes protect engines that ask for the guard, and a default 4.x engine behaves as before. Every test engine sets the guard. The MaxExecutionStackCount rows set both the guard and the count, which is the configuration a default engine on main has when that lane is chosen. The InstanceofOperator remarks say that the probe is gated on the guard. The StackOverflowGuard doc summary keeps 4.x's "Defaults to false" and adds only main's "or into eval source".

• The direct-eval count does not depend on the guard. It applies on the MaxExecutionStackCount lane whatever the guard setting. So the counted-eval rows run twice, once with the guard on and once with it off. This is the one behaviour change on the count lane: a recursion through direct eval now reaches the configured count, where before it was never counted. That is the fix itself; main documents it on MaxExecutionStackCount.

• BindFunction differs on 4.x. It derives from ObjectInstance, and the [[Call]] check is spelled IsCallable where main has HasCall. The loop uses IsCallable. Main's Walk bound-function chains in IsConstructor and GetFunctionRealm instead of recursing #4165 walk, already on 4.x, provides the IsConstructor loop that the new remarks refer to.

• bind builds names eagerly on 4.x. It writes "bound " + name on every bind, so a chain of 200,000 links would build quadratically long names. The new deep chains (in BoundFunctionChainWalkTests and the cross-realm FunctionTests row) delete each link's own name, as the existing 4.x BoundChain already does. The chains are otherwise unchanged.

• Order. On main, Bound eval recursion with a catchable RangeError instead of a native stack overflow or an endless thread hop #4187 landed before Probe the native stack before instanceof calls a target's own @@hasInstance method #4184. Here the order is Ask each bound target for its own Symbol.hasInstance in instanceof #4172, Probe the native stack before instanceof calls a target's own @@hasInstance method #4184, Bound eval recursion with a catchable RangeError instead of a native stack overflow or an endless thread hop #4187. So Probe the native stack before instanceof calls a target's own @@hasInstance method #4184's follow-up edit to a remark on Bound eval recursion with a catchable RangeError instead of a native stack overflow or an endless thread hop #4187's rows is taken in its final form in the Bound eval recursion with a catchable RangeError instead of a native stack overflow or an endless thread hop #4187 commit, and Bound eval recursion with a catchable RangeError instead of a native stack overflow or an endless thread hop #4187's "eval as @@hasInstance" row is already bounded by Probe the native stack before instanceof calls a target's own @@hasInstance method #4184's probe.

• Jint/Constraints/AGENTS.md does not exist on 4.x. Bound eval recursion with a catchable RangeError instead of a native stack overflow or an endless thread hop #4187's hunk to it is dropped.

• Tests are transcribed from NUnit to xUnit v3. Main's TestCases/TestCaseSource become TheoryData/MemberData. TestBudgets.WedgeCeiling becomes a local two-minute constant, as in HostModuleGraphDepthTests. Every deep row runs on a 1 MB DedicatedThread, as on main.

How the process-death tests are isolated

Main's tests run each body on a small dedicated thread, and this PR mirrors that. That thread only makes the overflow quick and deterministic: a native stack overflow still kills the whole test host. To keep one dying row from hiding the others, the "unfixed" evidence below runs each theory row in its own process. It uses -preEnumerateTheories -id <row id> against the test executable, and records the exit code and the repeated top frame. A regression in the committed suite would still show as a dead test host (TestPipelineException), the same as on main.

Evidence (Windows x64, Release)

Unfixed means: the new tests, run against the engine files as they were before the commit that ports each fix. The earlier backports in this PR are applied.

Test	net10.0 unfixed	net472 unfixed	fixed (both TFMs)
#4172 FunctionTests.Instanceof* (4 new)	4 fail: own @@hasInstance not called ("false"…), link reads "2,1,0" not observed, bound proxy refused, cross-realm bottom method skipped	same 4 fail	pass (FunctionTests 76/76)
#4172 BoundFunctionChainWalkTests (4 new rows)	2 fail ("false" where RangeError expected, both lanes); the 2 deep-chain Answers rows pass (they pin that the walk stays a loop)	same	7/7
#4184 AHasInstanceMethodThatAsksInstanceofOfItsOwnTarget… (6 rows)	4 kill the host (class static, count lane; built-in, count lane; eval, both lanes). Stack overflow. with JsValue.InstanceofOperator → ScriptFunction.CallOnce repeated ×916. 2 pass (guard-only lane, where the callee probes)	same 4 kill the host (Process is terminated due to StackOverflowException.)	6/6
#4187 ARecursionThroughEvaluatedSource… (7 rows)	4 kill the host (direct, indirect, optional-call, callback; exit 0xC00000FD, JintCallExpression.HandleEval ×531 for direct). 3 pass (eval as @@hasInstance, already bounded by #4184; both Function constructor rows, never broken)	same 4 kill the host	7/7
#4187 OnTheExecutionStackCountLane…StopsAtTheCount (10 rows)	4 fail at the 2-minute wedge ceiling, "the lane is hopping threads without bound" (direct and optional-call, guard on and off). 6 pass	same 4 fail	10/10
#4187 OnTheExecutionStackCountLaneAFiniteRecursionThroughEval… (2 rows)	2 pass (they pin that the fix did not turn the lane's hop into a throw)	2 pass	2/2

With all three commits, HostNativeRecursionGuardTests, BoundFunctionChainWalkTests, HostStackOverflowGuardTests and CallStackTests together pass 82/82 on both net10.0 and net472.

Skipped

None. All three source PRs apply to 4.x.

Test totals (all three commits, dotnet test -c Release, no --timeout)

Project	net10.0	net472
Jint.Tests	7659 passed, 4 skipped, 0 failed	7574 passed, 4 skipped, 0 failed
Jint.Tests.PublicInterface	1932 passed, 9 skipped, 0 failed	1924 passed, 9 skipped, 0 failed
Jint.Tests.Test262 (full suite)	102509 passed, 175 skipped, 0 failed	n/a (net10.0 only)

🤖 Generated with Claude Code