Probe the native stack before instanceof calls a target's own @@hasInstance method

[lahma]

Summary

JsValue.InstanceofOperator step 3 calls whatever GetMethod(target, @@hasInstance) finds. A method that asks instanceof of its own target is therefore a recursion that script controls, one native frame per level, with no call expression in it:

class C { static [Symbol.hasInstance](v) { return v instanceof C; } }
({}) instanceof C;

The operator made that call without probing the native stack and relied on the callee to probe for itself. That does not hold on two routes:

• The MaxExecutionStackCount lane. A script function does not probe on entry there, so the route above ends the process on net10.0, net8.0 and net472 (Stack overflow. / Process is terminated due to StackOverflowException). With the default StackOverflowGuard lane it was already a catchable RangeError.
• eval as the method, on both lanes. Object.defineProperty(C, Symbol.hasInstance, { value: eval }) with the source 's instanceof C' re-enters the operator without entering any function, so even the default lane ended the process.

The call is now made behind EnsureNativeStackHeadroom for every method except %Function.prototype[@@hasInstance]%. Every ordinary function inherits that one, so nearly every instanceof finds it, and it is still called unprobed: its whole behaviour is OrdinaryHasInstance. This is the same placement and the same intrinsic check that #4172 gave BindFunction's walk over the bound form of the same step, so the check moves to FunctionPrototype.IsIntrinsicHasInstance, where both call it.

Behaviour notes

• Every hostile route on both lanes now raises RangeError: Maximum call stack size exceeded and the engine recovers. That covers a class's static method, a defineProperty'd method, a plain-object target, Function.prototype.call as the method, a proxy apply trap, and eval. Ordinary instanceof answers are unchanged.
• Trade-off on the count lane: with MaxExecutionStackCount = 100000, a legitimate 3000-deep call recursion that runs a non-recursive custom @@hasInstance at every level used to return 3000, because the lane hops to a fresh thread. It now raises RangeError, because the probe fires on native-stack exhaustion before the hop. The existing forward probes from Walk the prototype chain in a loop instead of one native frame per link #4078 (trapless proxy) and Ask each bound target for its own Symbol.hasInstance in instanceof #4172 (bound link) already behave this way on the analogous shapes. Plain deep recursion and ordinary deep instanceof still return 3000.
• The one route still fatal on the count lane is a @@hasInstance getter that recurses. That is the documented accessor route (StackOverflowGuardTests.UnboundedRecursions, THREAT_MODEL TM-07) and is out of scope here.
• Hot path: ordinary instanceof gains only the discriminator: one isinst Function, a few field loads to the realm's cached intrinsic, and a ReferenceEquals. There is no probe on it. A paired measurement of ForInGuardBenchmark.InstanceofMixed follows before merge.

Tests

Jint.Tests.PublicInterface/HostNativeRecursionGuardTests.cs: six new depth rows covering the routes above on both lanes. On the unfixed engine the test host dies on all three TFMs (net10/net8 Stack overflow., exit 0xC00000FD; net472 Process is terminated due to StackOverflowException.). With the fix all 47 rows of the class pass on each TFM.

Full suites, Release, whole projects with no --timeout, 0 failures:

Project	net10.0	net8.0	net472
Jint.Tests	12,776	12,776	8,824
Jint.Tests.PublicInterface	3,823	3,813	3,030

test262 was not run locally, because nothing spec-visible changes: the probe fires only on stack exhaustion and the discriminator changes no answer. CI runs it.

Found separately and not in this PR: plain eval recursion (var s = 'eval(s)'; eval(s)) also ends the host on the default lane. It is being fixed in its own PR.

Follow-up to #4172. Base: upstream/main at the time of push.

🤖 Generated with Claude Code

[lahma]

Gate: no regression attributable to this change

Paired, alternating order, DefaultJob (measure-paired.ps1), idle box verified with a live per-process CPU sample. Base 7d9cdcc8f (this PR's merge base) vs a61ea852a; the two sides have identical Jint.Benchmark trees.

Stage 1: ForInGuardBenchmark, 6 rounds

row	median	95% CI	sign	verdict
InstanceofMixed (runs the discriminator on every instanceof)	+3.63%	[−8.50, +8.78]	4/6	no change
ForInProtoChain	+3.34%	[+1.88, +6.89]	6/6	SLOWER
the other 9 rows	−3.19% … +4.99%	every CI includes zero		no change

ForInProtoChain is for…in plus hasOwnProperty and contains no instanceof, so it cannot execute this change. The subject row's interval was too wide to decide either way, so both went to stage 2 along with a control.

Stage 2: 12 rounds

row	median	95% CI	sign	verdict
InstanceofMixed	+0.68%	[−1.23, +11.21]	6/12	no change
ForInProtoChain	+1.94%	[−1.49, +2.76]	7/12	no change
InOperatorMixed (control)	−0.91%	[−1.68, +0.94]	4/12	no change

InstanceofMixed reads slower in only half the rounds. The median is +0.68%, and the wide upper bound comes from a few outlier rounds rather than a shift. The stage-1 ForInProtoChain flag does not reproduce. The discriminator (an isinst plus a few field loads and a ReferenceEquals on the ordinary path) is below what this box can resolve.

A possible follow-up would repay it outright, and it is deliberately not in this PR: when the handler is the intrinsic, call target.OrdinaryHasInstance(this) directly, which is BindFunction's shape, and skip the dispatcher call and its params array.

🤖 Generated with Claude Code