Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[GCStress] coreroot_determinism test #50704

Closed
AaronRobinsonMSFT opened this issue Apr 5, 2021 · 15 comments · Fixed by #56770
Closed

[GCStress] coreroot_determinism test #50704

AaronRobinsonMSFT opened this issue Apr 5, 2021 · 15 comments · Fixed by #56770
Labels
area-crossgen2-coreclr disabled-test The test is disabled in source code against the issue GCStress
Milestone

Comments

@AaronRobinsonMSFT
Copy link
Member

Failing on most platforms architectures in different manners, but all are timeouts when GCStress is enabled.

Windows x86 Checked gcstress0xc

1 / 1 (100%, 1 failed): timed out in 603151 msecs: corerun.exe C:\h\w\B4B30A30\p\crossgen2\crossgen2.dll @C:\h\w\B4B30A30\w\AFC70989\e\readytorun\coreroot_determinism\coreroot_determinism\seed1\CPAOT-ret.out\System.Private.CoreLib.dll.rsp
Emitting R2R PE file: C:\h\w\B4B30A30\w\AFC70989\e\readytorun\coreroot_determinism\coreroot_determinism\seed1\CPAOT-ret.out\System.Private.CoreLib.dll

Return code:      1
Raw output file:      C:\h\w\B4B30A30\w\AFC70989\e\readytorun\coreroot_determinism\Reports\readytorun.coreroot_determinism\coreroot_determinism\coreroot_determinism.output.txt
Raw output:
BEGIN EXECUTION
 "C:\h\w\B4B30A30\p\corerun.exe" coreroot_determinism.dll ================================== Compiling with seed 1 ==================================
Locating output folder C:\h\w\B4B30A30\w\AFC70989\e\readytorun\coreroot_determinism\coreroot_determinism\seed1
Deleting 0 output folders
Successfully deleted 0 output folders in 480 msecs
Using dotnet: C:\h\w\B4B30A30\p\corerun.exe
1 / 1 (0%, 0 failed): launching: C:\h\w\B4B30A30\p\corerun.exe C:\h\w\B4B30A30\p\crossgen2\crossgen2.dll @C:\h\w\B4B30A30\w\AFC70989\e\readytorun\coreroot_determinism\coreroot_determinism\seed1\CPAOT-ret.out\System.Private.CoreLib.dll.rsp
Compilation failed. C:\h\w\B4B30A30\p\corerun.exe C:\h\w\B4B30A30\p\R2RTest\R2RTest.dll compile-directory -cr C:\h\w\B4B30A30\p -in TestAssemblies --nojit --noexe --large-bubble --release --nocleanup -out seed1 failed with exit code 1
Expected: 100
Actual: 1
END EXECUTION - FAILED
FAILED
Test Harness Exitcode is : 1
To run the test:
> set CORE_ROOT=C:\h\w\B4B30A30\p
> C:\h\w\B4B30A30\w\AFC70989\e\readytorun\coreroot_determinism\coreroot_determinism\coreroot_determinism.cmd
Expected: True
Actual:   False

Windows x64 Checked gcstress0x3

1 / 1 (100%, 1 failed): timed out in 610853 msecs: corerun.exe C:\h\w\A7D30987\p\crossgen2\crossgen2.dll @C:\h\w\A7D30987\w\B70F0A6C\e\readytorun\coreroot_determinism\coreroot_determinism\seed1\CPAOT-ret.out\System.Private.CoreLib.dll.rsp

Assert failure(PID 3288 [0x00000cd8], Thread: 4304 [0x10d0]): !CREATE_CHECK_STRING(pMT && pMT->Validate())

CORECLR! Object::ValidateInner + 0x120 (0x00007ffae36194b0)
CORECLR! Object::Validate + 0xB9 (0x00007ffae3619359)
CORECLR! WKS::GCHeap::Promote + 0xA0 (0x00007ffae3a78dd0)
CORECLR! TransitionFrame::PromoteCallerStackHelper + 0xF5 (0x00007ffae355fb0d)
CORECLR! TransitionFrame::PromoteCallerStack + 0x17F (0x00007ffae355f9ff)
CORECLR! GcStackCrawlCallBack + 0x379 (0x00007ffae3773999)
CORECLR! Thread::MakeStackwalkerCallback + 0x6B (0x00007ffae366fd8f)
CORECLR! Thread::StackWalkFramesEx + 0x164 (0x00007ffae3671658)
CORECLR! Thread::StackWalkFrames + 0x184 (0x00007ffae3671454)
CORECLR! ScanStackRoots + 0x22E (0x00007ffae37724be)
    File: D:\workspace\_work\1\s\src\coreclr\vm\object.cpp Line: 514
    Image: C:\h\w\A7D30987\p\corerun.exe


Return code:      1
Raw output file:      C:\h\w\A7D30987\w\B70F0A6C\e\readytorun\coreroot_determinism\Reports\readytorun.coreroot_determinism\coreroot_determinism\coreroot_determinism.output.txt
Raw output:
BEGIN EXECUTION
 "C:\h\w\A7D30987\p\corerun.exe" coreroot_determinism.dll 
================================== Compiling with seed 1 ==================================
Locating output folder C:\h\w\A7D30987\w\B70F0A6C\e\readytorun\coreroot_determinism\coreroot_determinism\seed1
Deleting 0 output folders
Successfully deleted 0 output folders in 2195 msecs
Using dotnet: C:\h\w\A7D30987\p\corerun.exe
1 / 1 (0%, 0 failed): launching: C:\h\w\A7D30987\p\corerun.exe C:\h\w\A7D30987\p\crossgen2\crossgen2.dll @C:\h\w\A7D30987\w\B70F0A6C\e\readytorun\coreroot_determinism\coreroot_determinism\seed1\CPAOT-ret.out\System.Private.CoreLib.dll.rsp
Compilation failed. C:\h\w\A7D30987\p\corerun.exe C:\h\w\A7D30987\p\R2RTest\R2RTest.dll compile-directory -cr C:\h\w\A7D30987\p -in TestAssemblies --nojit --noexe --large-bubble --release --nocleanup -out seed1 failed with exit code -1073740286
Expected: 100
Actual: 1
END EXECUTION - FAILED
FAILED
Test Harness Exitcode is : 1
To run the test:
> set CORE_ROOT=C:\h\w\A7D30987\p
> C:\h\w\A7D30987\w\B70F0A6C\e\readytorun\coreroot_determinism\coreroot_determinism\coreroot_determinism.cmd
Expected: True
Actual:   False

OSX arm64 Checked gcstress0x3

1 / 1 (100%, 1 failed): timed out in 748524 msecs: corerun /tmp/helix/working/B28A0A0D/p/crossgen2/crossgen2.dll @/private/tmp/helix/working/B28A0A0D/w/C3A50A8D/e/readytorun/coreroot_determinism/coreroot_determinism/seed1/CPAOT-ret.out/System.Private.CoreLib.dll.rsp

Return code:      1
Raw output file:      /private/tmp/helix/working/B28A0A0D/w/C3A50A8D/e/readytorun/coreroot_determinism/Reports/readytorun.coreroot_determinism/coreroot_determinism/coreroot_determinism.output.txt
Raw output:
BEGIN EXECUTION
/tmp/helix/working/B28A0A0D/p/corerun coreroot_determinism.dll ''
================================== Compiling with seed 1 ==================================
Locating output folder /private/tmp/helix/working/B28A0A0D/w/C3A50A8D/e/readytorun/coreroot_determinism/coreroot_determinism/seed1
Deleting 0 output folders
Successfully deleted 0 output folders in 20123 msecs
Using dotnet: /private/tmp/helix/working/B28A0A0D/p/corerun
1 / 1 (0%, 0 failed): launching: /private/tmp/helix/working/B28A0A0D/p/corerun /tmp/helix/working/B28A0A0D/p/crossgen2/crossgen2.dll @/private/tmp/helix/working/B28A0A0D/w/C3A50A8D/e/readytorun/coreroot_determinism/coreroot_determinism/seed1/CPAOT-ret.out/System.Private.CoreLib.dll.rsp
Compilation failed. /tmp/helix/working/B28A0A0D/p/corerun /tmp/helix/working/B28A0A0D/p/R2RTest/R2RTest.dll compile-directory -cr /tmp/helix/working/B28A0A0D/p -in TestAssemblies --nojit --noexe --large-bubble --release --nocleanup -out seed1 failed with exit code 1
Expected: 100
Actual: 1
END EXECUTION - FAILED
Test Harness Exitcode is : 1
To run the test:
> set CORE_ROOT=/tmp/helix/working/B28A0A0D/p
> /private/tmp/helix/working/B28A0A0D/w/C3A50A8D/e/readytorun/coreroot_determinism/coreroot_determinism/coreroot_determinism.sh
Expected: True
Actual:   False
@dotnet-issue-labeler dotnet-issue-labeler bot added the untriaged New issue has not been triaged by the area owner label Apr 5, 2021
@AaronRobinsonMSFT AaronRobinsonMSFT added the disabled-test The test is disabled in source code against the issue label Apr 5, 2021
@VincentBu
Copy link
Contributor

Failed again in runtime-coreclr r2r-extra 20210404.1

Failed test:

R2R Linux arm64 Checked gcstress0xf @ (Ubuntu.1804.Arm64.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-a45aeeb-20190620155855
 -readytorun/coreroot_determinism/coreroot_determinism/coreroot_determinism.sh

Error message:

1 / 1 (100%, 1 failed): timed out in 615154 msecs: corerun /root/helix/work/correlation/crossgen2/crossgen2.dll @/root/helix/work/workitem/readytorun/coreroot_determinism/coreroot_determinism/seed1/CPAOT-ret.out/System.Private.CoreLib.dll.rsp

Return code: 1
Raw output file: /root/helix/work/workitem/readytorun/coreroot_determinism/Reports/readytorun.coreroot_determinism/coreroot_determinism/coreroot_determinism.output.txt
Raw output:
BEGIN EXECUTION
in takeLock
/root/helix/work/correlation/corerun coreroot_determinism.dll ''
================================== Compiling with seed 1 ==================================
Locating output folder /root/helix/work/workitem/readytorun/coreroot_determinism/coreroot_determinism/seed1
Deleting 0 output folders
Successfully deleted 0 output folders in 6159 msecs
Using dotnet: /root/helix/work/correlation/corerun
1 / 1 (0%, 0 failed): launching: /root/helix/work/correlation/corerun /root/helix/work/correlation/crossgen2/crossgen2.dll @/root/helix/work/workitem/readytorun/coreroot_determinism/coreroot_determinism/seed1/CPAOT-ret.out/System.Private.CoreLib.dll.rsp
Compilation failed. /root/helix/work/correlation/corerun /root/helix/work/correlation/R2RTest/R2RTest.dll compile-directory -cr /root/helix/work/correlation -in TestAssemblies --nojit --noexe --large-bubble --release --nocleanup -out seed1 failed with exit code 1
in ReleaseLock
Expected: 100
Actual: 1
END EXECUTION - FAILED
Test Harness Exitcode is : 1
To run the test:
> set CORE_ROOT=/root/helix/work/correlation
> /root/helix/work/workitem/readytorun/coreroot_determinism/coreroot_determinism/coreroot_determinism.sh
Expected: True
Actual: False

Stack trace
   at readytorun_coreroot_determinism._coreroot_determinism_coreroot_determinism_._coreroot_determinism_coreroot_determinism_sh()

@trylek
Copy link
Member

trylek commented Apr 12, 2021

Please note that runtime-coreclr r2r-extra is a Crossgen1 test which seems to indicate that the problem is actually not Crossgen2-specific.

@trylek trylek removed the untriaged New issue has not been triaged by the area owner label Apr 12, 2021
@trylek trylek added this to the 6.0.0 milestone Apr 12, 2021
@VincentBu
Copy link
Contributor

VincentBu commented Apr 12, 2021

Historical failures of this test:

Queued OS Arch Pipeline
2021-05-30T04:02:18.598Z windows.10.amd64.open.rt x86 runtime-coreclr r2r-extra Checked-jitstress1_tiered
2021-05-26T06:40:28.977Z windows.10.arm64v8.open arm runtime-coreclr r2r Checked-no_tiered_compilation
2021-05-26T06:40:28.054Z windows.10.arm64v8.open arm runtime-coreclr r2r Checked
2021-04-24T21:31:18.517Z osx.1100.arm64.open arm64 runtime-coreclr jitstress-isas-arm Checked-jitstress_isas_nohwintrinsic
2021-04-20T06:43:19.907Z osx.1100.arm64.open arm64 runtime-coreclr jitstress Checked-jitstress1
2021-04-17T06:40:19.71Z ubuntu.1804.armarch.open arm runtime-coreclr jitstress Checked-jitstress1
2021-04-15T06:43:20.839Z osx.1100.arm64.open arm64 runtime-coreclr jitstress Checked-jitstress1
2021-04-05T05:47:19.044Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr r2r-extra Checked-gcstress0xf
2021-04-05T05:31:35.442Z ubuntu.1804.armarch.open arm64 runtime-coreclr r2r-extra Checked-gcstress0xf
2021-04-05T05:17:47.219Z ubuntu.1804.armarch.open arm runtime-coreclr r2r-extra Checked-gcstress0xf
2021-04-05T01:08:30.795Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_jitminopts_heapverify1
2021-04-05T00:55:48.686Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr gcstress-extra Checked-gcstress0xc_jitminopts_heapverify1
2021-04-05T00:52:00.514Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress-extra Checked-gcstress0xc_jitminopts_heapverify1
2021-04-05T00:42:00.629Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_tailcallstress
2021-04-05T00:41:37.53Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr gcstress-extra Checked-gcstress0xc_tailcallstress
2021-04-05T00:37:39.414Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress-extra Checked-gcstress0xc_tailcallstress
More failures
Queued OS Arch Pipeline
2021-04-05T00:36:34.803Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress2
2021-04-05T00:34:39.935Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress2
2021-04-05T00:25:17.563Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress2
2021-04-05T00:16:38.32Z osx.1100.arm64.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_jitminopts_heapverify1
2021-04-05T00:13:37.405Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress1
2021-04-05T00:12:24.16Z osx.1100.arm64.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_tailcallstress
2021-04-05T00:09:48.253Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress1
2021-04-04T23:55:36.978Z osx.1100.arm64.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress2
2021-04-04T23:50:04.715Z osx.1100.arm64.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress1
2021-04-04T23:48:58.526Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress1
2021-04-04T23:47:35.873Z osx.1100.arm64.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable_heapverify1
2021-04-04T23:24:04.748Z osx.1100.arm64.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable_jitstress2
2021-04-04T23:24:04.102Z osx.1100.arm64.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable
2021-04-04T23:24:00.529Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable_heapverify1
2021-04-04T23:23:59.652Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable_jitstress2
2021-04-04T23:23:58.98Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable
2021-04-04T23:23:43.076Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable
2021-04-04T23:23:42.383Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable_jitstress2
2021-04-04T23:23:41.664Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable_heapverify1
2021-04-04T23:23:36.147Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable_heapverify1
2021-04-04T23:23:35.314Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable_jitstress2
2021-04-04T23:23:34.403Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable
2021-04-04T15:49:20.494Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2021-04-04T15:49:19.541Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0xc
2021-04-04T15:47:24.154Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2021-04-04T15:47:23.478Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0xc
2021-04-04T15:46:46.957Z osx.1100.arm64.open arm64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0xc
2021-04-04T15:46:46.254Z osx.1100.arm64.open arm64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2021-04-04T15:45:53.934Z windows.10.arm64v8.open arm64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2021-04-04T15:45:53.09Z windows.10.arm64v8.open arm64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0xc
2021-04-04T15:43:19.452Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0xc
2021-04-04T15:43:17.653Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2021-04-04T15:40:23.999Z windows.10.amd64.open.rt x64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0xc
2021-04-04T15:40:22.712Z windows.10.amd64.open.rt x64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2021-04-04T15:38:49.392Z windows.10.amd64.open.rt x86 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0xc
2021-04-04T15:38:48.579Z windows.10.amd64.open.rt x86 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2021-04-04T10:54:04.9Z ubuntu.1804.armarch.open arm64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_jitminopts_heapverify1
2021-04-04T10:34:30.964Z ubuntu.1804.armarch.open arm64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_tailcallstress
2021-04-04T10:20:12.862Z windows.10.amd64.open.rt x64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_jitminopts_heapverify1
2021-04-04T10:01:12.105Z ubuntu.1804.armarch.open arm64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_jitstress2
2021-04-04T09:59:18.249Z windows.10.arm64v8.open arm64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_jitminopts_heapverify1
2021-04-04T09:56:26.75Z windows.10.amd64.open.rt x64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_tailcallstress
2021-04-04T09:47:52.052Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_jitminopts_heapverify1
2021-04-04T09:47:24.638Z windows.10.amd64.open.rt x64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_jitstress2
2021-04-04T09:45:52.635Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_tailcallstress
2021-04-04T09:44:34.401Z windows.10.arm64v8.open arm64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_tailcallstress
2021-04-04T09:40:12.523Z windows.10.arm64v8.open arm64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_jitstress2
2021-04-04T09:39:30.482Z ubuntu.1804.armarch.open arm64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_jitstress1
2021-04-04T09:35:12.543Z windows.10.amd64.open.rt x64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_jitstress1
2021-04-04T09:25:32.669Z windows.10.arm64v8.open arm64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_jitstress1
2021-04-04T09:19:22.753Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_jitstress2
2021-04-04T09:05:32.486Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_jitstress1
2021-04-04T08:50:15.173Z windows.10.amd64.open.rt x64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_zapdisable
2021-04-04T08:50:12.501Z windows.10.amd64.open.rt x64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_zapdisable_jitstress2
2021-04-04T08:50:11.486Z windows.10.amd64.open.rt x64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_zapdisable_heapverify1
2021-04-04T08:49:38.544Z windows.10.arm64v8.open arm64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_zapdisable_heapverify1
2021-04-04T08:49:36.342Z windows.10.arm64v8.open arm64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_zapdisable_jitstress2
2021-04-04T08:49:35.325Z windows.10.arm64v8.open arm64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_zapdisable
2021-04-04T08:23:49.086Z ubuntu.1804.armarch.open arm64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_zapdisable_jitstress2
2021-04-04T08:23:48.093Z ubuntu.1804.armarch.open arm64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_zapdisable_heapverify1
2021-04-04T08:23:47.47Z ubuntu.1804.armarch.open arm64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_zapdisable
2021-04-04T08:19:07.38Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_zapdisable
2021-04-04T08:19:06.44Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_zapdisable_jitstress2
2021-04-04T08:19:05.69Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_zapdisable_heapverify1
2021-04-04T06:38:02.646Z windows.10.amd64.open.rt x64 runtime-coreclr r2r-extra Checked-gcstress0xf
2021-04-04T06:31:37.341Z windows.10.amd64.open.rt x86 runtime-coreclr r2r-extra Checked-gcstress0xf
2021-04-04T05:38:21.939Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr r2r-extra Checked-gcstress0xf
2021-04-04T05:14:15.332Z ubuntu.1804.armarch.open arm64 runtime-coreclr r2r-extra Checked-gcstress0xf
2021-04-04T05:09:29.959Z windows.10.arm64v8.open arm64 runtime-coreclr r2r-extra Checked-gcstress0xf
2021-04-04T05:07:20.614Z ubuntu.1804.armarch.open arm runtime-coreclr r2r-extra Checked-gcstress0xf
2021-04-04T01:27:31.004Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_jitminopts_heapverify1
2021-04-04T01:11:50.013Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr gcstress-extra Checked-gcstress0xc_jitminopts_heapverify1
2021-04-04T01:10:08.617Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_tailcallstress
2021-04-04T01:08:31.307Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress-extra Checked-gcstress0xc_jitminopts_heapverify1
2021-04-04T01:07:34.143Z windows.10.amd64.open.rt x86 runtime-coreclr gcstress-extra Checked-gcstress0xc_jitminopts_heapverify1
2021-04-04T01:04:48.6Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr gcstress-extra Checked-gcstress0xc_tailcallstress
2021-04-04T01:01:22.868Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress2
2021-04-04T00:55:37.132Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress-extra Checked-gcstress0xc_tailcallstress
2021-04-04T00:50:52.077Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress2
2021-04-04T00:49:11.136Z windows.10.amd64.open.rt x64 runtime-coreclr gcstress-extra Checked-gcstress0xc_jitminopts_heapverify1
2021-04-04T00:49:04.149Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress2
2021-04-04T00:47:02.582Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress1
2021-04-04T00:34:23.995Z windows.10.amd64.open.rt x86 runtime-coreclr gcstress-extra Checked-gcstress0xc_tailcallstress
2021-04-04T00:33:27.978Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress1
2021-04-04T00:33:14.924Z osx.1100.arm64.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_jitminopts_heapverify1
2021-04-04T00:32:43.311Z windows.10.amd64.open.rt x86 runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress2
2021-04-04T00:31:33.211Z osx.1100.arm64.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_tailcallstress
2021-04-04T00:29:23.132Z windows.10.amd64.open.rt x86 runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress1
2021-04-04T00:25:15.593Z windows.10.amd64.open.rt x64 runtime-coreclr gcstress-extra Checked-gcstress0xc_tailcallstress
2021-04-04T00:18:47.786Z windows.10.arm64v8.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_jitminopts_heapverify1
2021-04-04T00:17:01.54Z windows.10.arm64v8.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_tailcallstress
2021-04-04T00:16:17.082Z windows.10.amd64.open.rt x64 runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress2
2021-04-04T00:15:38.076Z windows.10.amd64.open.rt x64 runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress1
2021-04-04T00:12:58.117Z osx.1100.arm64.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress2
2021-04-04T00:05:56.014Z osx.1100.arm64.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress1
2021-04-04T00:04:47.177Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress1
2021-04-04T00:03:37.325Z osx.1100.arm64.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable_heapverify1
2021-04-04T00:00:41.428Z windows.10.arm64v8.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress2
2021-04-03T23:59:52.828Z windows.10.arm64v8.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress1
2021-04-03T23:48:44.904Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable_jitstress2
2021-04-03T23:48:43.049Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable_heapverify1
2021-04-03T23:48:42.346Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable
2021-04-03T23:42:19.576Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable_heapverify1
2021-04-03T23:42:18.772Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable_jitstress2
2021-04-03T23:42:16.543Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable
2021-04-03T23:37:37.878Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable_jitstress2
2021-04-03T23:37:37.189Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable
2021-04-03T23:37:36.411Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable_heapverify1
2021-04-03T23:37:25.113Z osx.1100.arm64.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable_jitstress2
2021-04-03T23:37:24.024Z osx.1100.arm64.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable
2021-04-03T23:37:20.377Z windows.10.arm64v8.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable
2021-04-03T23:37:19.175Z windows.10.arm64v8.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable_heapverify1
2021-04-03T23:37:18.455Z windows.10.arm64v8.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable_jitstress2
2021-04-03T23:31:03.368Z windows.10.amd64.open.rt x86 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable_heapverify1
2021-04-03T23:31:01.523Z windows.10.amd64.open.rt x86 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable_jitstress2
2021-04-03T23:31:00.636Z windows.10.amd64.open.rt x86 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable
2021-04-03T23:28:44.014Z windows.10.amd64.open.rt x64 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable_heapverify1
2021-04-03T23:28:43.076Z windows.10.amd64.open.rt x64 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable_jitstress2
2021-04-03T23:28:41.342Z windows.10.amd64.open.rt x64 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable
2021-04-03T15:49:56.388Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0xc
2021-04-03T15:49:55.684Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2021-04-03T15:47:25.637Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2021-04-03T15:47:24.895Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0xc
2021-04-03T15:46:53.566Z osx.1100.arm64.open arm64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2021-04-03T15:46:52.592Z osx.1100.arm64.open arm64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0xc
2021-04-03T15:44:51.272Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2021-04-03T15:44:50.433Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0xc
2021-04-03T15:44:13.738Z windows.10.amd64.open.rt x64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2021-04-03T15:44:13.088Z windows.10.amd64.open.rt x64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0xc
2021-04-03T15:42:47.683Z windows.10.arm64v8.open arm64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0xc
2021-04-03T15:42:46.997Z windows.10.arm64v8.open arm64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2021-04-03T15:42:41.9Z windows.10.amd64.open.rt x86 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0xc
2021-04-03T15:42:40.775Z windows.10.amd64.open.rt x86 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2021-03-04T07:22:58.396Z windows.10.amd64.open.rt x86 runtime-coreclr r2r Checked-no_tiered_compilation

@AaronRobinsonMSFT
Copy link
Member Author

launching: /root/helix/work/correlation/corerun /root/helix/work/correlation/crossgen2/crossgen2.dll

@trylek Even with the above line? That is why I had originally tagged it crossgen2.

@trylek
Copy link
Member

trylek commented Apr 12, 2021

Hmm, sorry, you're right, while the r2r-extra is generally a Crossgen1 test, this is one of the few tests that have Crossgen2 hardcoded in their project script. One traditional problem we had several times in the past was that the COMPlus variable settings for GC stress got inadvertently applied to the managed Crossgen2 compiler itself, that is naturally 1-2 orders of magnitude slower and prone to timing out.

@AaronRobinsonMSFT
Copy link
Member Author

Perhaps these tests should be disabled when CrossGen2 is involved somehow? I guess these tests involve CrossGen2 so maybe they are not GCStress compatible.

@trylek
Copy link
Member

trylek commented Apr 12, 2021

Well, I think that the R2R code Crossgen2 produces is generally GC stress-compatible, it's just that Crossgen2 itself is a too big managed test app to run under GC stress and successfully compile something in ten minutes or so. Having said that, you're right that the determinism test is not about any runtime stress at all, the purpose of the test is to validate that parallelized Crossgen2 produces deterministic results. And I think it's due to the special nature of the test that runs Crossgen2 twice that it bypasses the provisions I added to CLRTest.Crossgen.targets to suppress the GC stress runtime variables for the duration of Crossgen2 compilation. I'll mark the test as GC stress-incompatible.

@trylek
Copy link
Member

trylek commented Apr 12, 2021

Hmmm. The test is actually marked as GCStressIncompatible. In such case I suspect it's most likely a test infra script issue - the YAML files have special provisions for dealing with GC stress tests and maybe in this particular case the fact that it's also a R2R test causes the stress flag not to propagate to the inner build script. I'll take a look what might be causing that.

@VincentBu
Copy link
Contributor

Failed again in runtime 20210418.3

Failed test:

CoreCLR OSX arm64 Checked @ OSX.1100.ARM64.Open
 -readytorun/coreroot_determinism/coreroot_determinism/coreroot_determinism.sh

Error message:

1 / 1 (100%, 1 failed): failed in 13404 msecs, exit code 134, expected 0: corerun /tmp/helix/working/A7920915/p/crossgen2/crossgen2.dll @/private/tmp/helix/working/A7920915/w/C4070A49/e/readytorun/coreroot_determinism/coreroot_determinism/seed2/CPAOT-ret.out/System.Private.CoreLib.dll.rsp
 Emitting R2R PE file: /private/tmp/helix/working/A7920915/w/C4070A49/e/readytorun/coreroot_determinism/coreroot_determinism/seed2/CPAOT-ret.out/System.Private.CoreLib.dll
!! Unhandled exception. System.AggregateException: One or more errors occurred. (Object reference not set to an instance of an object.)
!! ---> System.NullReferenceException: Object reference not set to an instance of an object.
!! at ILCompiler.Sorting.Implementation.MergeSortCore5.ParallelSort(TDataStructure arrayToSort, Int32 index, Int32 length, TComparer comparer) in ILCompiler.DependencyAnalysisFramework.dll:token 0x600001e+0x14
!! at ILCompiler.Sorting.Implementation.MergeSortCore5.ParallelSort(TDataStructure arrayToSort, Int32 index, Int32 length, TComparer comparer) in ILCompiler.DependencyAnalysisFramework.dll:token 0x600001e+0x1a2
!! --- End of inner exception stack trace ---
!! at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions) in System.Private.CoreLib.dll:token 0x6002ca2+0x21
!! at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken) in System.Private.CoreLib.dll:token 0x6002cbe+0x4
!! at System.Threading.Tasks.Task.Wait() in System.Private.CoreLib.dll:token 0x6002cba+0x1c
!! at ILCompiler.Sorting.Implementation.MergeSortCore`5.ParallelSortApi(TDataStructure arrayToSort, TComparer comparer) in ILCompiler.DependencyAnalysisFramework.dll:token 0x600001d+0x33
!! at ILCompiler.DependencyAnalysis.ReadyToRun.GCRefMapNode.GetData(NodeFactory factory, Boolean relocsOnly) in ILCompiler.ReadyToRun.dll:token 0x60004ca+0x38
!! at ILCompiler.DependencyAnalysis.ReadyToRunObjectWriter.EmitPortableExecutable() in ILCompiler.ReadyToRun.dll:token 0x600031a+0x174
!! at ILCompiler.ReadyToRunCodegenCompilation.Compile(String outputFile) in ILCompiler.ReadyToRun.dll:token 0x6000128+0x35
!! at ILCompiler.Program.Run(String[] args) in crossgen2.dll:token 0x6000154+0x93d
!! at ILCompiler.Program.Main(String[] args) in crossgen2.dll:token 0x600015a+0x5
!! task_for_pid(71458) FAILED 5 (os/kern) failure

Return code: 1
Raw output file: /private/tmp/helix/working/A7920915/w/C4070A49/e/readytorun/coreroot_determinism/Reports/readytorun.coreroot_determinism/coreroot_determinism/coreroot_determinism.output.txt
Raw output:
BEGIN EXECUTION
/tmp/helix/working/A7920915/p/corerun coreroot_determinism.dll ''
================================== Compiling with seed 1 ==================================
Locating output folder /private/tmp/helix/working/A7920915/w/C4070A49/e/readytorun/coreroot_determinism/coreroot_determinism/seed1
Deleting 0 output folders
Successfully deleted 0 output folders in 52 msecs
Using dotnet: /private/tmp/helix/working/A7920915/p/corerun
1 / 1 (0%, 0 failed): launching: /private/tmp/helix/working/A7920915/p/corerun /tmp/helix/working/A7920915/p/crossgen2/crossgen2.dll @/private/tmp/helix/working/A7920915/w/C4070A49/e/readytorun/coreroot_determinism/coreroot_determinism/seed1/CPAOT-ret.out/System.Private.CoreLib.dll.rsp
1 / 1 (100%, 0 failed): succeeded in 12535 msecs: corerun /tmp/helix/working/A7920915/p/crossgen2/crossgen2.dll @/private/tmp/helix/working/A7920915/w/C4070A49/e/readytorun/coreroot_determinism/coreroot_determinism/seed1/CPAOT-ret.out/System.Private.CoreLib.dll.rsp
================================== Compiling with seed 2 ==================================
Locating output folder /private/tmp/helix/working/A7920915/w/C4070A49/e/readytorun/coreroot_determinism/coreroot_determinism/seed2
Deleting 0 output folders
Successfully deleted 0 output folders in 51 msecs
Using dotnet: /private/tmp/helix/workin

Stack trace
   at readytorun_coreroot_determinism._coreroot_determinism_coreroot_determinism_._coreroot_determinism_coreroot_determinism_sh()

Historical failures of this test has updated in this comment

@VincentBu
Copy link
Contributor

Failed again in runtime-coreclr r2r-extra 20210529.1

Failed test:

R2R windows x86 Checked jitstress1_tiered @ Windows.10.Amd64.Open

- readytorun\\coreroot_determinism\\coreroot_determinism\\coreroot_determinism.cmd

Error message:

1 / 1 (100%, 1 failed): failed in 54588 msecs, exit code -1073740286 = 0xC0000602, expected 0: corerun.exe C:\h\w\9E7008B3\p\crossgen2\crossgen2.dll @C:\h\w\9E7008B3\w\AEFA093B\e\readytorun\coreroot_determinism\coreroot_determinism\seed2\CPAOT-ret.out\System.Private.CoreLib.dll.rsp
!! Assert failure(PID 8712 [0x00002208], Thread: 7588 [0x1da4]): !PreemptiveGCDisabled()
!! CORECLR! Thread::DetachThread + 0xCF (0x707985d7)
!! CORECLR! TlsDestructionMonitor::~TlsDestructionMonitor + 0x8A (0x70c33a94)
!! CORECLR! _dyn_tls_dtor + 0x86 (0x70c45606)
!! NTDLL! RtlDecompressBuffer + 0xDE (0x776eea4e)
!! NTDLL! LdrShutdownThread + 0x386 (0x776beea6)
!! NTDLL! LdrSetAppCompatDllRedirectionCallback + 0x1052A (0x77713c5a)
!! NTDLL! LdrShutdownProcess + 0x15F (0x776d2d7f)
!! NTDLL! RtlExitUserProcess + 0x96 (0x776d3886)
!! KERNEL32! ExitProcess + 0x13 (0x74e1b3e3)
!! CORECLR! exit_or_terminate_process + 0x38 (0x70c7ad68)
!!     File: D:\workspace_work\1\s\src\coreclr\vm\threads.cpp Line: 986
!!     Image: C:\h\w\9E7008B3\p\corerun.exe

Return code:      1
Raw output file:      C:\h\w\9E7008B3\w\AEFA093B\e\readytorun\coreroot_determinism\Reports\readytorun.coreroot_determinism\coreroot_determinism\coreroot_determinism.output.txt
Raw output:
BEGIN EXECUTION
"C:\h\w\9E7008B3\p\corerun.exe"  coreroot_determinism.dll
================================== Compiling with seed 1 ==================================
Locating output folder C:\h\w\9E7008B3\w\AEFA093B\e\readytorun\coreroot_determinism\coreroot_determinism\seed1
Deleting 0 output folders
Successfully deleted 0 output folders in 162 msecs
Using dotnet: C:\h\w\9E7008B3\p\corerun.exe
1 / 1 (0%, 0 failed): launching: C:\h\w\9E7008B3\p\corerun.exe C:\h\w\9E7008B3\p\crossgen2\crossgen2.dll @C:\h\w\9E7008B3\w\AEFA093B\e\readytorun\coreroot_determinism\coreroot_determinism\seed1\CPAOT-ret.out\System.Private.CoreLib.dll.rsp
1 / 1 (100%, 0 failed): succeeded in 48952 msecs: corerun.exe C:\h\w\9E7008B3\p\crossgen2\crossgen2.dll @C:\h\w\9E7008B3\w\AEFA093B\e\readytorun\coreroot_determinism\coreroot_determinism\seed1\CPAOT-ret.out\System.Private.CoreLib.dll.rsp
================================== Compiling with seed 2 ==================================
Locating output folder C:\h\w\9E7008B3\w\AEFA093B\e\readytorun\coreroot_determinism\coreroot_determinism\seed2
Deleting 0 output folders
Successfully deleted 0 output folders in 173 msecs
Using dotnet: C:\h\w\9E7008B3\p\corerun.exe
1 / 1 (0%, 0 failed): launching: C:\h\w\9E7008B3\p\corerun.exe C:\h\w\9E7008B3\p\crossgen2\crossgen2.dll @C:\h\w\9E7008B3\w\AEFA093B\e\readytorun\coreroot_determinism\coreroot_determinism\seed2\CPAOT-ret.out\System.Private.CoreLib.dll.rsp
Compilation failed. C:\h\w\9E7008B3\p\corerun.exe C:\h\w\9E7008B3\p\R2RTest\R2RTest.dll compile-directory -cr C:\h\w\9E7008B3\p -in TestAssemblies --nojit --noexe --large-bubble --release --nocleanup -out seed2 failed with exit code 1
Expected: 100
Actual: 1
END EXECUTION - FAILED
FAILED
Test Harness Exitcode is : 1
To run the test:

set CORE_ROOT=C:\h\w\9E7008B3\p
C:\h\w\9E7008B3\w\AEFA093B\e\readytorun\coreroot_determinism\coreroot_determinism\coreroot_determinism.cmd
Expected: True
Actual:   False


Stack trace
   at readytorun_coreroot_determinism._coreroot_determinism_coreroot_determinism_._coreroot_determinism_coreroot_determinism_cmd()

Historical failures of this test has updated in this comment.

@mateoatr
Copy link
Contributor

Failed again: runtime-coreclr gcstress-extra.

1 / 1 (100%, 1 failed): timed out in 600228 msecs: corerun /root/helix/work/correlation/crossgen2/crossgen2.dll @/root/helix/work/workitem/readytorun/coreroot_determinism/coreroot_determinism/seed1/CPAOT-ret.out/System.Private.CoreLib.dll.rsp
Emitting R2R PE file: /root/helix/work/workitem/readytorun/coreroot_determinism/coreroot_determinism/seed1/CPAOT-ret.out/System.Private.CoreLib.dll

Return code:      1
Raw output file:      /root/helix/work/workitem/uploads/Reports/readytorun.coreroot_determinism/coreroot_determinism/coreroot_determinism.output.txt
Raw output:
BEGIN EXECUTION
/root/helix/work/correlation/corerun coreroot_determinism.dll ''
================================== Compiling with seed 1 ==================================
Locating output folder /root/helix/work/workitem/readytorun/coreroot_determinism/coreroot_determinism/seed1
Deleting 0 output folders
Successfully deleted 0 output folders in 290 msecs
Using dotnet: /root/helix/work/correlation/corerun
1 / 1 (0%, 0 failed): launching: /root/helix/work/correlation/corerun /root/helix/work/correlation/crossgen2/crossgen2.dll @/root/helix/work/workitem/readytorun/coreroot_determinism/coreroot_determinism/seed1/CPAOT-ret.out/System.Private.CoreLib.dll.rsp
Compilation failed. /root/helix/work/correlation/corerun /root/helix/work/correlation/R2RTest/R2RTest.dll compile-directory -cr /root/helix/work/correlation -in TestAssemblies --nojit --noexe --large-bubble --release --nocleanup -out seed1 failed with exit code 1
Expected: 100
Actual: 1
END EXECUTION - FAILED
Test Harness Exitcode is : 1
To run the test:

set CORE_ROOT=/root/helix/work/correlation
/root/helix/work/workitem/readytorun/coreroot_determinism/coreroot_determinism/coreroot_determinism.sh
Expected: True
Actual:   False

Stack trace:

   at readytorun_coreroot_determinism._coreroot_determinism_coreroot_determinism_._coreroot_determinism_coreroot_determinism_sh() in readytorun.coreroot_determinism.XUnitWrapper.dll:token 0x6000004+0x295

@VincentBu
Copy link
Contributor

Failed again in runtime-coreclr gcstress-extra 20210801.1

Failed test:

CoreCLR Linux arm64 Checked heapverify1 @ (Ubuntu.1804.Arm64.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-20210531091519-97d8652
readytorun/coreroot_determinism/coreroot_determinism/coreroot_determinism.sh

Error message:

1 / 1 (100%, 1 failed): timed out in 600583 msecs: corerun /root/helix/work/correlation/crossgen2/crossgen2.dll @/root/helix/work/workitem/readytorun/coreroot_determinism/coreroot_determinism/seed2/CPAOT-ret.out/System.Private.CoreLib.dll.rsp
Emitting R2R PE file: /root/helix/work/workitem/readytorun/coreroot_determinism/coreroot_determinism/seed2/CPAOT-ret.out/System.Private.CoreLib.dll

Return code:      1
Raw output file:      /root/helix/work/workitem/uploads/Reports/readytorun.coreroot_determinism/coreroot_determinism/coreroot_determinism.output.txt
Raw output:
BEGIN EXECUTION
/root/helix/work/correlation/corerun coreroot_determinism.dll ''
================================== Compiling with seed 1 ==================================
Locating output folder /root/helix/work/workitem/readytorun/coreroot_determinism/coreroot_determinism/seed1
Deleting 0 output folders
Successfully deleted 0 output folders in 284 msecs
Using dotnet: /root/helix/work/correlation/corerun
1 / 1 (0%, 0 failed): launching: /root/helix/work/correlation/corerun /root/helix/work/correlation/crossgen2/crossgen2.dll @/root/helix/work/workitem/readytorun/coreroot_determinism/coreroot_determinism/seed1/CPAOT-ret.out/System.Private.CoreLib.dll.rsp
1 / 1 (100%, 0 failed): succeeded in 593473 msecs: corerun /root/helix/work/correlation/crossgen2/crossgen2.dll @/root/helix/work/workitem/readytorun/coreroot_determinism/coreroot_determinism/seed1/CPAOT-ret.out/System.Private.CoreLib.dll.rsp
================================== Compiling with seed 2 ==================================
Locating output folder /root/helix/work/workitem/readytorun/coreroot_determinism/coreroot_determinism/seed2
Deleting 0 output folders
Successfully deleted 0 output folders in 183 msecs
Using dotnet: /root/helix/work/correlation/corerun
1 / 1 (0%, 0 failed): launching: /root/helix/work/correlation/corerun /root/helix/work/correlation/crossgen2/crossgen2.dll @/root/helix/work/workitem/readytorun/coreroot_determinism/coreroot_determinism/seed2/CPAOT-ret.out/System.Private.CoreLib.dll.rsp
Compilation failed. /root/helix/work/correlation/corerun /root/helix/work/correlation/R2RTest/R2RTest.dll compile-directory -cr /root/helix/work/correlation -in TestAssemblies --nojit --noexe --large-bubble --release --nocleanup -out seed2 failed with exit code 1
Expected: 100
Actual: 1
END EXECUTION - FAILED
Test Harness Exitcode is : 1
To run the test:

set CORE_ROOT=/root/helix/work/correlation
/root/helix/work/workitem/readytorun/coreroot_determinism/coreroot_determinism/coreroot_determinism.sh
Expected: True
Actual:   False


Stack trace
   at readytorun_coreroot_determinism._coreroot_determinism_coreroot_determinism_._coreroot_determinism_coreroot_determinism_sh() in readytorun.coreroot_determinism.XUnitWrapper.dll:token 0x6000004+0x295

@trylek
Copy link
Member

trylek commented Aug 3, 2021

I have a new theory why this is happening. As @janvorli explained to me the other day, one subtle difference between Windows shell and bash is that, in bash, setting an environment variable to an empty string doesn't actually clear it, you need to unset it. I believe we've got it wrong here:

export COMPlus_GCStress=

Interestingly enough someone fixed the last case (or perhaps added it later) without noticing the pre-existing bug. Due to this fact, on Linux, we still run Crossgen2 itself under GC stress mode, slowing it down to the point of timing out systematically. I'll put out a PR to fix this.

Thanks

Tomas

/cc @dotnet/crossgen-contrib

@ghost ghost added the in-pr There is an active PR which will close this issue when it is merged label Aug 3, 2021
@mangod9
Copy link
Member

mangod9 commented Aug 3, 2021

yeah sorry I had added the GC and wanted to change the others too use unset too, but got distracted.

@ghost ghost removed the in-pr There is an active PR which will close this issue when it is merged label Aug 3, 2021
@AntonLapounov
Copy link
Member

I think export COMPlus_GCStress= worked as expected. All recent failures were in HeapVerify and JitStress legs, not GCStress ones.

@ghost ghost locked as resolved and limited conversation to collaborators Sep 7, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area-crossgen2-coreclr disabled-test The test is disabled in source code against the issue GCStress
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants