Enable out-of-process tests for CoreCLR browser WASM - #131110
Conversation
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 44f4b4ea-409c-4a52-89a0-f488a89488b9
|
Azure Pipelines: Successfully started running 3 pipeline(s). 13 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
|
Tagging subscribers to 'arch-wasm': @lewing, @pavelsavara |
There was a problem hiding this comment.
Pull request overview
This PR extends the CoreCLR browser-WASM test execution pipeline to support “out-of-process” (process-isolated) tests by adding a plan+execute orchestration flow: the merged managed runner can emit an OOP plan, host-side scripts run the existing per-test wrappers in fresh processes, and the merged runner then imports the precomputed wrapper results during the main test run.
Changes:
- Add a build-time marker file (
*.HasOutOfProcessTests) for CoreCLR browser merged runners that have OOP tests. - Update the merged runner codegen to support a “planning mode” (write OOP assembly paths instead of executing) and to route OOP tests through a precomputed-result import path on Browser.
- Update browser Windows (
.cmd) and Unix (.sh) execution scripts to run the plan phase, execute OOP wrappers, write versioned result files with a per-run token, then run the merged runner to import results.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| src/tests/Common/XUnitWrapperGenerator/XUnitWrapperGenerator.props | Writes a HasOutOfProcessTests marker for CoreCLR+browser merged runners that contain OOP tests. |
| src/tests/Common/XUnitWrapperGenerator/XUnitWrapperGenerator.cs | Emits new generated-runner behavior: per-test executors with an optional plan-writer parameter; planning mode for OOP selection; Browser guard when orchestration is missing. |
| src/tests/Common/XUnitWrapperGenerator/ITestInfo.cs | Adds plan-writing support to the reporting wrapper; exposes OOP assembly path for plan emission. |
| src/tests/Common/CoreCLRTestLibrary/OutOfProcessTest.cs | Adds env-var plumbing and precomputed-result import so Browser can “run” OOP tests without forking. |
| src/tests/Common/CLRTest.Execute.Batch.targets | Adds Windows browser orchestration: plan phase, wrapper execution, result-file generation, then merged runner import phase. |
| src/tests/Common/CLRTest.Execute.Bash.targets | Adds Unix browser orchestration mirroring the batch logic: plan, run wrappers, write result files, then run merged runner to import. |
Merged runners query each project reference for process isolation. Matching tests are moved into host-side OOP scheduling. Only return projects that generate a run script and are enabled for the current target. This prevents BuildOnly and SharedLibrary helpers, as well as disabled or unsupported projects, from producing OOP entries without runnable wrappers. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 44f4b4ea-409c-4a52-89a0-f488a89488b9
InterpreterTester normally uses Process.Start to launch Interpreter.dll under corerun. Browser WASM cannot create that child process. Keep the test enabled for CoreCLR browser WASM and make its generated wrapper launch Interpreter.dll directly. Enable interpreter mode and disable tiering so the payload remains interpreted, while preserving the existing desktop path. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 44f4b4ea-409c-4a52-89a0-f488a89488b9
Enabling OOP execution exposes tests that either cannot exercise their scenario in a browser or block the merged runner. These include NoGCRegion callbacks, the Android/Java GC bridge, Mono embedding APIs, and Windows-only PDB, APC, and type-equivalence tests. Mark these tests unsupported only for CoreCLR browser runs. Desktop CoreCLR, WASI, and Mono configurations remain unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 44f4b4ea-409c-4a52-89a0-f488a89488b9
Standalone test wrappers can exit successfully when every fact is skipped. The browser OOP orchestration previously preserved only the wrapper exit code and output, so merged runners reported those children as passed and lost ActiveIssue attribution. Add status and reason fields to the precomputed result protocol. Teach standalone RequiresProcessIsolation runners to emit a skip marker and map it back into the merged TestSummary. Use the same signal for direct local OOP execution, and regenerate incremental wrappers when their script targets change. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 44f4b4ea-409c-4a52-89a0-f488a89488b9
CoreCLR browser runtime test payloads do not yet relink per-test native assets, and UCO thunk relinking is tracked separately. OOP execution exposes tests that were previously absent and lets those paths fail at runtime. Apply the existing tracking issues to xUnit entry points so merged results retain ActiveIssue attribution. Mark the two non-xUnit projects unsupported on WASM, where no attribute-based skip can be generated. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 44f4b4ea-409c-4a52-89a0-f488a89488b9
CoreCLR's browser launcher invoked corerun.js without the -p arguments generated from RuntimeHostConfigurationOption. Tests using host properties therefore behaved differently from their native counterparts. Include the existing CoreRunArgs in the Node launch commands on Bash and Batch. This lets browser tests configure runtime properties such as the external type-map entry assembly. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 44f4b4ea-409c-4a52-89a0-f488a89488b9
The AssemblyDependencyResolver test requires a native mock hostpolicy and the SuperPMI collection test launches host RyuJIT tools. Neither dependency is available inside the browser runtime, so exclude those projects from the CoreCLR browser target. SystemCoreLibDirectory also cannot validate a physical CoreLib path when System.Private.CoreLib has no Assembly.Location. Check that assembly directly so the generated standalone runner records the test as skipped. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 44f4b4ea-409c-4a52-89a0-f488a89488b9
String.CompareTo guarantees a value less than zero when the first value sorts before the second, but it does not guarantee exactly -1. The browser runtime returns -15 for the smoke test inputs, which is valid behavior. Check the documented sign contract so the instance-method test remains portable while continuing to exercise the generic interface call. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 44f4b4ea-409c-4a52-89a0-f488a89488b9
CoreCLR browser does not provide the COM ABI hooks, dynamic native library loading, diagnostics IPC, child process creation, host ReadyToRun tools, or FileVersionInfo support required by these tests. Running their newly enabled out-of-process wrappers therefore produces only platform-not-supported failures. Mark the projects unsupported only for CoreCLR browser so native CoreCLR and other runtime targets retain their existing coverage. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 44f4b4ea-409c-4a52-89a0-f488a89488b9
CoreCLR browser-WASM does not support managed multithreading, but several newly enabled out-of-process tests also contain useful single-threaded coverage. Run the allocation and collectible-static checks that do not need worker threads, and condition only thread-dependent facts on the runtime capability. Use the writable temporary directory for the collectible assembly-on-disk test instead of the read-only browser application directory. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 44f4b4ea-409c-4a52-89a0-f488a89488b9
CoreCLR browser-WASM does not support managed multithreading. These tests exercise concurrent GC stress, JIT behavior under concurrent mutation, or unhandled-exception delivery from worker and native threads, so a single-threaded fallback would not preserve their purpose. Mark only the CoreCLR browser target unsupported. Native CoreCLR continues to build and run the existing thread coverage unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 44f4b4ea-409c-4a52-89a0-f488a89488b9
Core_Root staging assumed every cross-architecture build ran on an x64 host. On macOS arm64, browser-WASM builds therefore omitted the host Crossgen2 executable and its WASM JIT libraries even though the build had produced them under the arm64 host-artifact directory. Resolve the fallback directory from BuildArchitecture so cross-targeting and sanitizer layouts use the tools built for their actual host architecture. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 44f4b4ea-409c-4a52-89a0-f488a89488b9
Composite WASM Crossgen2 output contains a manifest plus one ReadyToRun image per input assembly. R2RDump rejected the manifest because it intentionally has no ReadyToRun header, causing otherwise successful browser-WASM compilations to fail before test execution. Validate each emitted component for composite WASM builds in both Bash and Batch scripts. Keep the existing single-output validation for other modes, and do not invoke R2RDump after a failed Crossgen2 command. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 44f4b4ea-409c-4a52-89a0-f488a89488b9
Composite WASM ReadyToRun execution does not populate Assembly.Location for the entry assembly. The smoke test used that value to reopen its IL image and failed before exercising its constrained-dispose check. Use argv[0] as the entry assembly path when Location is empty. Native and non-ReadyToRun execution continue to use Assembly.Location unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 44f4b4ea-409c-4a52-89a0-f488a89488b9
CoreCLR browser builds compile without BACKGROUND_GC, but GC.GetGCMemoryInfo(GCKind.Background) remains a valid API request. The native selector therefore fell through to the Any case and asserted in checked builds. Use a zero-initialized record for Background requests when background GC is not compiled. This preserves the documented Index == 0 result for a GC kind that has never occurred and leaves BACKGROUND_GC builds unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 44f4b4ea-409c-4a52-89a0-f488a89488b9
NativeLibraryTests is excluded from CoreCLR browser-WASM builds, but its AfterTargets="Build" setup target still ran under nobuild.targets and tried to copy an assembly that was intentionally not produced. Stale output masked this during incremental builds and made clean builds fail. Run the managed subdirectory setup only when the project itself is built. The normal macOS build continues copying the suffixed test assembly. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 44f4b4ea-409c-4a52-89a0-f488a89488b9
RelativeAssemblyPath is only consumed by wrappers within XUnitWrapperGenerator. Keep it internal instead of widening the public surface of the generator assembly. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 44f4b4ea-409c-4a52-89a0-f488a89488b9
Avoid the unsupported Process lookup used only to format the GC Collect test's usage text. Keep the ReadyToRun generics coverage while skipping only its cross-thread field checks when multithreading is unavailable. The Emscripten 3.1.56 to 5.0.6 update included upstream change emscripten-core/emscripten#25093, which moved preInit processing before WASM module creation. At that point ___funcs_on_exit is only an early-access placeholder, and later export binding replaces the browser shutdown wrapper installed from preInit. Without the wrapper, Environment.Exit proceeds through Emscripten shutdown without BrowserHost_ShutdownDotnet. A finalization callback retains the runtime, and later host activity changes the requested exit code 100 into generic failure status 1. Install the wrapper from preRun, after WASM exports are assigned but before managed Main runs. This shuts CoreCLR down before Emscripten exit callbacks and lets b65423 keep its original Environment.Exit behavior and process-isolation coverage. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 44f4b4ea-409c-4a52-89a0-f488a89488b9
CoreCLR browser-WASM is single-threaded and is built without background GC. Priority-one scenarios that require worker threads, finalizer threads, registered callbacks, or concurrent GC cannot exercise their intended behavior there. Mark only the CoreCLR browser target unsupported so these projects remain enabled on threaded runtimes and other WASM configurations. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 44f4b4ea-409c-4a52-89a0-f488a89488b9
Emit an annotations-only nullable context for merged runners with out-of-process tests and mark the optional plan writer parameter nullable. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 44f4b4ea-409c-4a52-89a0-f488a89488b9
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 44f4b4ea-409c-4a52-89a0-f488a89488b9
Remove the GC and ReadyToRun product fixes from this infrastructure change. Attribute affected browser CoreCLR tests to their tracking issues, and avoid composite loading before managed skips can run. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 44f4b4ea-409c-4a52-89a0-f488a89488b9
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 133 out of 134 changed files in this pull request and generated no new comments.
Suppressed comments (1)
src/tests/Common/XUnitWrapperGenerator/ITestInfo.cs:557
StandaloneTestReporting.WrapTestExecutionWithReportingsets theoutOfProcessTestExecutedflag totruebefore runningtestExecution. For wrapperITestInfotypes that callWrapTestExecutionWithReportingeven when the underlying test is skipped (e.g.,TestWithCustomDisplayNamewraps an innerConditionalTestand still callsWrapTestExecutionWithReportingeven on the skip path), this can incorrectly mark an all-skipped child as “executed”, preventing__TestOutOfProcessStatusFilefrom being written and causing the parent to report the child as passed instead of skipped.
public CodeBuilder WrapTestExecutionWithReporting(CodeBuilder testExecution, ITestInfo test)
{
CodeBuilder builder = new();
builder.AppendLine($"{_testExecutedLocalIdentifier} = true;");
builder.Append(testExecution);
|
Azure Pipelines: Successfully started running 3 pipeline(s). 13 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
| <RequiresProcessIsolation>true</RequiresProcessIsolation> | ||
| <AllowUnsafeBlocks>true</AllowUnsafeBlocks> | ||
| <CLRTestPriority>1</CLRTestPriority> | ||
| <CLRTestTargetUnsupported Condition="'$(FeatureMultithreading)' != 'true'">true</CLRTestTargetUnsupported> |
There was a problem hiding this comment.
Is there a reason for why we are using the csproj conditions and not PlatformDetection.IsMultithreadingSupported ? We prefer the PlatformDetection conditions where posssible.
There was a problem hiding this comment.
…se ConditionalFact for FeatureMultithreading test skips (#131844) Follow-up to #131110: Replaced all `Condition="'$(FeatureMultithreading)' != 'true'"` build-time skip conditions in test `.csproj`/`.ilproj` files with `ConditionalFact`s and runtime checks > [!NOTE] > This PR description and changes were generated with AI assistance (GitHub Copilot). --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: jkotas <6668460+jkotas@users.noreply.github.com>
…-1 tests (#131883) ## Summary PR #131110 enabled CoreCLR browser-WASM out-of-process runtime tests. The newly enabled priority-1 coverage exposed eight unique failures. Four were composite WebCIL/ReadyToRun failures tracked by #131767 and are fixed on `main` by #131354. This change removes their temporary `ActiveIssue` annotations and Crossgen2 bypasses, restoring the intended R2R coverage. The unrelated field-layout suppression for #131421 remains. The other four tests are narrowly gated: - `GC/API/GC/GetTotalAllocatedBytesServerGC` is excluded on CoreCLR browser, which does not provide Server GC. - `GC/API/GC/GetTotalMemoryConcurrent` uses `PlatformDetection.IsMultithreadingSupported` through `ConditionalFact`. - `baseservices/threading/regressions/2164/foreground-shutdown` retains its explicit `Main` and returns success before creating a thread when multithreading is unsupported. - `JIT/jit64/regress/ddb/113574` is excluded only on CoreCLR browser because its multi-billion-iteration optimizing-JIT workload exceeds the browser interpreter's CI timeout. The runtime capability checks skip single-threaded browser and WASI while preserving desktop and threaded-browser coverage. The Server GC and interpreter-workload exclusions are tracked in #131321. ## Testing Built the Checked browser runtime from `main`, including #131354, then built and ran the complete priority-1 browser-WASM Node suite: | Scope | Total | Passed | Failed | Skipped | | --- | ---: | ---: | ---: | ---: | | Full suite | 14,745 | 13,823 | 0 | 922 | | Out-of-process subset | 672 | 560 | 0 | 112 | The four former WebCIL/R2R failures now pass: - `Regressions/coreclr/GitHub_49826/test49826` - `Regressions/coreclr/GitHub_49982/test49982` - `readytorun/tests/genericsload/callgenericctor` - `readytorun/tests/genericsload/usegenericfield` Additional focused validation for the runtime multithreading gates: - 8 MSBuild evaluations across desktop, threaded browser, single-threaded browser, and single-threaded WASI; neither project is permanently marked unsupported. - 2 targeted Checked browser builds, both with 0 warnings and 0 errors. - 2 single-threaded browser wrapper runs, both passing with expected/actual exit code 100. - Native `foreground-shutdown` execution exited with 100 after 2.28 seconds, confirming that its foreground thread still keeps the process alive after `Main` returns. > [!NOTE] > This pull request was created with assistance from GitHub Copilot. Copilot-Session: 19e61792-38f9-4f80-8d38-3e3e12d1275e
Summary
CoreCLR browser-WASM cannot use
forkor managed process creation, so runtimetests marked
RequiresProcessIsolationwere omitted from merged runners. ThisPR enables those tests without adding Node-specific process APIs to managed
runtime code:
fresh child shell;
The PR also includes the runtime-test infrastructure, platform gating, and
focused runtime/test fixes needed to make priority-0 and priority-1 CoreCLR
browser-WASM coverage pass consistently.
This reuses the existing CoreCLR
corerunwrappers; it does not introduce aparallel
browserhostscheduler or duplicate per-test execution semantics.Execution model
The browser run has three phases:
assembly paths.
.shor.cmdwrapper and recordsits output, exit code, pass/skip state, and skip reason.
precomputed child outcomes through the existing xUnit summary.
Existing wrappers remain authoritative for test arguments, environment
variables, pre/post commands, expected exit codes, and per-test timeouts.
Merged-runner arguments and stripe state are cleared while child wrappers run
and restored afterward.
The result format is versioned and bound to a per-run token. Missing plans,
wrappers, results, malformed records, or stale tokens fail as infrastructure
errors. A child in which every test is skipped is reported as skipped rather
than passed. Empty plan/status environment settings are treated as unset.
The host orchestration marker is emitted only for CoreCLR browser-WASM merged
runners that contain out-of-process tests.
Browser-WASM enablement and fixes
merged-runner OOP plans.
InterpreterTesterrunsInterpreter.dlldirectly throughthe host OOP path with the required interpreter environment instead of
calling
Process.Startinside the browser.cross-architecture runs, and emitted WASM components, including composite
output, are validated.
ActiveIssue for
#131767. Their browser
projects do not force Crossgen2, allowing the managed wrapper to report the
skip before CoreCLR probes the unsupported composite WebCIL layout. Other
targets retain their existing composite R2R coverage.
operations while exposing Unix-style absolute paths to browser managed code.
FeatureMultithreadingis available during test projectevaluation. Tests whose only requirement is managed multithreading use that
global capability, so they remain excluded on single-threaded targets and
become buildable when threads are enabled.
threads, child processes, unsupported browser APIs, mutable host filesystem
behavior, or impractical interpreter workloads are gated or excluded at the
narrowest applicable level.
from Emscripten
preRun, after native exports are bound, so explicitEnvironment.Exitvalues are preserved and no finalization work isscheduled while the runtime exits.
GetGCMemoryInfotest carries an ActiveIssue for#131766, which tracks the
assertion from requesting
GCKind.Backgroundwhen background GC is notcompiled. This PR does not change the product behavior.
threading, filesystem, and host-tool assumptions while preserving the
behavior each test is intended to cover.
failures after newer
mainchanges activated more browser scenarios. Theeager-fixup lock-order failure is fixed on
mainby#131355. Composite WebCIL
loading and background-GC information remain tracked separately by [wasm][CoreCLR][R2R] Composite WebCIL images request an unsupported loaded PE layout #131767
and [browser][CoreCLR] GC.GetGCMemoryInfo(GCKind.Background) asserts when background GC is unavailable #131766. The
Int128field-layout correctness fix is proceeding in#131421; other unsupported
cases are gated only for CoreCLR browser-WASM with explicit re-enable
prerequisites.
No shipping public API is added.
Deferred scope
Per-test native assets, UCO thunk generation, and native relinking are
intentionally deferred. Tests that need them remain guarded by their existing
tracking issues.
The remaining non-multithreading/non-relinking exclusions were audited:
41 represent recoverable coverage and 16 exercise browser-inapplicable
contracts. The recoverable set and its prerequisites are tracked by
#131321, with links from each
suppression site. The newly exposed native-relinking test remains tracked by
#123946.
Validation
The following complete-suite results were recorded before the two product fixes
were moved to their dedicated tracking issues:
mainmergeCurrent-head validation after removing those fixes:
warnings or errors;
exited with the expected status and reported [browser][CoreCLR] GC.GetGCMemoryInfo(GCKind.Background) asserts when background GC is unavailable #131766 or [wasm][CoreCLR][R2R] Composite WebCIL images request an unsupported loaded PE layout #131767 through the
OOP skip-status protocol, without either runtime assertion;
BasicTestremains enabled;validates its composite image, and passes.
Additional earlier validation:
NativeLibraryTestsasset staging;records, environment-sensitive wrappers, and all-skipped children;
InterpreterTesterexecution under Node;layout after the
mainmerge;evaluations, preserving the pre-existing Mono ReadyToRun exclusions;
JIT, Loader, and tracing work items that failed in build 1528104.
System.Threading.Timer.TestsSystem.Runtime.InteropServices.JavaScript.TestsSystem.Reflection.Emit.Tests(Debug)System.Runtime.Tests(Debug)The generated Batch path has been inspected and generated successfully but
still requires execution on Windows CI.
Note
This pull request description was updated with GitHub Copilot.