Make the encoder stack-depth tests hold on a reused thread stack

[marcschier]

Description

EncoderStackDepthTests.DecodeWithUnboundedNestingLimitFailsBeforeTheStackOverflows(Json, VariantArray) and (Json, DataValue) fail intermittently on Linux CI, for example in run 37921301898, attempt 1, test-linux-net10.0 (12/15). The 450-level JSON input decodes without tripping the stack check, so the expected BadEncodingLimitsExceeded is missing. A re-run passes.

Cause: a thread that asks for a 256 KB stack can run on more. glibc reuses the cached stack of an exited thread when it is at most four times the requested size, and the fixture's own 1 MB threads leave such stacks behind. RuntimeHelpers.EnsureSufficientExecutionStack measures the stack the thread actually got, and 450 JSON levels fit a 1 MB stack. The product's stack check is correct; the test relied on the requested size.

Deepest nesting that fits before the check fires (net10.0, Release):

• On a 256 KB thread: about 100 JSON levels on Linux and 70 on Windows.
• On a 1 MB thread, Linux: 548 levels for a JSON Variant array, and up to 2477 for a JSON EncodeableTree. Encoding a Binary EncodeableTree fits 1899 levels, so the 2000 levels of the other codecs had no margin either.

Fix:

• The unbounded-limit decode and encode tests run on a 1 MB and a 256 KB stack: the largest stack glibc can hand a 256 KB request, and the requested size.
• Every codec nests 5000 levels, twice the deepest nesting that fits a 1 MB stack.
• The JsonEncoder cannot nest deeper than 1000 JSON levels, so the JSON input is written by hand. NestedJsonMatchesTheJsonEncoder checks it against the encoder's output for every shape.

Validation

Command	Result
Original test logic in the mcr.microsoft.com/dotnet/sdk:10.0 container, each decode right after a 1 MB thread exits, 20 times per shape	Decode succeeded 20 of 20 times for VariantArray and DataValue, the CI failure; 0 of 20 without the 1 MB thread, and 0 of 20 on Windows either way
dotnet test tests/Opc.Ua.Types.Tests -f net10.0 --filter FullyQualifiedName~EncoderStackDepthTests, Release and Debug on Windows, 2 runs each	80 passed
Same in the Linux container (Release), 3 runs	80 passed
Same with -f net48	80 passed
Linux, depth temporarily set back to 2000	EncodeWithUnboundedNestingLimitFailsBeforeTheStackOverflows(Binary,EncodeableTree) fails on the 1 MB stack
Linux, EncodingLimits.EnsureSufficientStack temporarily disabled	The test host crashes with a stack overflow, so the tests still catch a missing check

Related Issues

• No tracking issue. Seen on the CI of Use a 4-byte KeyNonce in the reference PubSub sample #4710.

Checklist

• I have signed the CLA and read the CONTRIBUTING doc.
• I have added tests that prove my fix is effective or that my feature works and increased code coverage.
• I have added all necessary documentation.
• I have verified that my changes do not introduce (new) build or analyzer warnings.
• I ran all tests locally using the UA.slnx solution against at least .net framework and .net 10, and all passed.
• I fixed all failing and flaky tests in the CI pipelines and all CodeQL warnings.
• I have addressed all PR feedback received.

[Copilot]

🟢 Approval recommended

The focused test changes correctly address reused thread stacks and validate the handcrafted JSON inputs.

0 open findings

What changed in this PR

Stabilizes encoder stack-depth tests against Linux thread-stack reuse behavior.

Changes:

• Tests both 256 KB and 1 MB thread stacks at 5,000 nesting levels.
• Generates deeply nested JSON manually and validates it against JsonEncoder output.

File	Description
tests/​Opc.Ua.Types.Tests/​Encoders/​EncoderStackDepthTests.cs	Strengthens stack-overflow regression coverage across codecs and shapes.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

Code coverage

✅ Coverage gate passed.

Check	Result	Threshold
✅ Project line rate	88.60% (306508/345943 lines)	>= 70.00%
✅ Project branch rate	79.04%	>= 60.00%
ℹ️ Patch coverage	no changed C# files	-
ℹ️ Baseline delta (advisory)	+15.00 pp	73.60% recorded

Coverage is above the recorded baseline - consider ratcheting coverage-thresholds.json.

_{Thresholds live in coverage-thresholds.json. Whole report before exclusions: line 87.74%, branch 78.25%.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.83%. Comparing base (60dc5ab) to head (d3d2517).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4722      +/-   ##
==========================================
- Coverage   82.84%   82.83%   -0.02%     
==========================================
  Files        2267     2267              
  Lines      345943   345943              
  Branches    60303    60303              
==========================================
- Hits       286609   286568      -41     
- Misses      39238    39269      +31     
- Partials    20096    20106      +10     

Flag	Coverage Δ
actions	82.83% <ø> (-0.02%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 30 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.