Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[pgo] Assertion failed '((-slop) <= diff) && (diff <= slop)' during 'Compute edge weights (2, false)' #77450

Closed
jakobbotsch opened this issue Oct 25, 2022 · 10 comments · Fixed by #77747 or #81377
Closed

[pgo] Assertion failed '((-slop) <= diff) && (diff <= slop)' during 'Compute edge weights (2, false)' #77450

jakobbotsch opened this issue Oct 25, 2022 · 10 comments · Fixed by #77747 or #81377
Assignees
@AndyAyersMS
Labels
area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI blocking-clean-ci-optional Blocking optional rolling runs
Milestone
8.0.0

Comments

@jakobbotsch
Copy link
Member

Pipeline: runtime-coreclr pgo
Config: win-arm64 Checked fullpgo_random_gdv_methodprofiling_only
Link: https://dev.azure.com/dnceng-public/public/_build/results?buildId=62149&view=ms.vss-test-web.build-test-results-tab&runId=1274480&resultId=108923&paneView=debug

Assert failure(PID 11476 [0x00002cd4], Thread: 7904 [0x1ee0]): Assertion failed '((-slop) <= diff) && (diff <= slop)' in 'System.RuntimeType:GetConstructorCandidates(System.String,int,int,System.Type[],bool):ListBuilder`1[System.Reflection.ConstructorInfo]:this' during 'Compute edge weights (2, false)' (IL size 103; hash 0xfc7100fa; Tier1)

File: D:\a\_work\1\s\src\coreclr\jit\fgprofile.cpp Line: 3941
Image: D:\h\w\BE4A09E7\p\corerun.exe

cc @AndyAyersMS
@dotnet-issue-labeler dotnet-issue-labeler bot added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Oct 25, 2022
@ghost ghost added the untriaged New issue has not been triaged by the area owner label Oct 25, 2022
@ghost
Copy link

ghost commented Oct 25, 2022

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

Pipeline: runtime-coreclr pgo
Config: win-arm64 Checked fullpgo_random_gdv_methodprofiling_only
Link: https://dev.azure.com/dnceng-public/public/_build/results?buildId=62149&view=ms.vss-test-web.build-test-results-tab&runId=1274480&resultId=108923&paneView=debug

Assert failure(PID 11476 [0x00002cd4], Thread: 7904 [0x1ee0]): Assertion failed '((-slop) <= diff) && (diff <= slop)' in 'System.RuntimeType:GetConstructorCandidates(System.String,int,int,System.Type[],bool):ListBuilder`1[System.Reflection.ConstructorInfo]:this' during 'Compute edge weights (2, false)' (IL size 103; hash 0xfc7100fa; Tier1)

File: D:\a\_work\1\s\src\coreclr\jit\fgprofile.cpp Line: 3941
Image: D:\h\w\BE4A09E7\p\corerun.exe

cc @AndyAyersMS

Author: jakobbotsch
Assignees: -
Labels:

area-CodeGen-coreclr, untriaged
Milestone: -

@jakobbotsch jakobbotsch added blocking-clean-ci-optional Blocking optional rolling runs and removed untriaged New issue has not been triaged by the area owner labels Oct 25, 2022
@AndyAyersMS
Copy link
Member

I just loosened the threshold recently: #76409. Looks like we'll have to loosen even more.

@runfoapp runfoapp bot mentioned this issue Oct 25, 2022
Closed
@JulieLeeMSFT JulieLeeMSFT added this to the 8.0.0 milestone Oct 27, 2022
@AndyAyersMS
Copy link
Member

AndyAyersMS commented Oct 31, 2022 •
edited
Loading

I'm not able to repro this so far. Probably timing dependen (or maybe harder to repro with the new pgo strategy?).

Will keep trying.

It's no longer happening in the CI runs. So, removing the blocking label.

@AndyAyersMS AndyAyersMS removed the blocking-clean-ci-optional Blocking optional rolling runs label Oct 31, 2022
@AndyAyersMS
Copy link
Member

I still can't repro, either with locally built bits or runfo downloaded bits, so am going to close this.

Would not be surprising if further tweaks are needed to this assert, but don't want to modify it blindly. So a fix will have to wait for a more consistent repro.

@jakobbotsch
Copy link
Member Author

It's no longer happening in the CI runs. So, removing the blocking label.

Looks like it happened again on Saturday's run:
https://dev.azure.com/dnceng-public/public/_build/results?buildId=67103&view=ms.vss-test-web.build-test-results-tab&runId=1365674&resultId=192095&paneView=dotnet-dnceng.dnceng-anon-build-release-tasks.helix-anon-test-information-tab
https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-7abbdb10ea47480a83/System.Collections.Concurrent.Tests/1/console.6aec2436.log?helixlogtype=result

@AndyAyersMS AndyAyersMS reopened this Nov 1, 2022
@AndyAyersMS
Copy link
Member

This one was also intermittent, but I now have it captured via SPMI.

@AndyAyersMS
Copy link
Member

In this latest case we just slightly over the tolerance limit:

BB78->BB80 (1) diff 1.040404 slop 1.040404 delta 2.220446e-16

AndyAyersMS added a commit to AndyAyersMS/runtime that referenced this issue Nov 1, 2022
@AndyAyersMS
JIT: increase edge weight slop tolerance some more
72a6482 
Allow up to 1/50 variance between the expected and computed values.

Fixes dotnet#77450.
@AndyAyersMS AndyAyersMS mentioned this issue Nov 1, 2022
Merged
@ghost ghost added the in-pr There is an active PR which will close this issue when it is merged label Nov 1, 2022
AndyAyersMS added a commit that referenced this issue Nov 2, 2022
@AndyAyersMS
JIT: increase edge weight slop tolerance some more (#77747)
82aa87f 
Allow up to 1/50 variance between the expected and computed values.

Fixes #77450.
@ghost ghost removed the in-pr There is an active PR which will close this issue when it is merged label Nov 2, 2022
@kunalspathak
Copy link
Member

This again happened lately. https://dev.azure.com/dnceng-public/public/_build/results?buildId=94427&view=ms.vss-test-web.build-test-results-tab&runId=1990586&resultId=195188&paneView=dotnet-dnceng.dnceng-build-release-tasks.helix-test-information-tab

@kunalspathak kunalspathak reopened this Dec 1, 2022
@BrianBohe
Copy link
Member

This happened again on 12/24 with pipeline runtime-coreclr libraries-pgo > "checked coreclr Linux arm64 Release". Full log.

@JulieLeeMSFT JulieLeeMSFT added the blocking-clean-ci-optional Blocking optional rolling runs label Jan 3, 2023
@AndyAyersMS
Copy link
Member

Might be time to just delete this assert; I don't think it provides any real value.

AndyAyersMS added a commit to AndyAyersMS/runtime that referenced this issue Jan 30, 2023
@AndyAyersMS
JIT: change profile slop assert to a jitdump note
3b63865 
Stop asserting if we see unusually large discrepancies in the outgoing profile
flow from a block. Instead just make a note in the jit dump.

Fixes dotnet#77450.
@AndyAyersMS AndyAyersMS mentioned this issue Jan 30, 2023
Merged
@ghost ghost added in-pr There is an active PR which will close this issue when it is merged and removed in-pr There is an active PR which will close this issue when it is merged labels Jan 30, 2023
AndyAyersMS added a commit that referenced this issue Feb 2, 2023
@AndyAyersMS
JIT: change profile slop assert to a jitdump note (#81377)
6c9190c 
Stop asserting if we see unusually large discrepancies in the outgoing profile
flow from a block. Instead just make a note in the jit dump.

Fixes #77450.
@ghost ghost removed the in-pr There is an active PR which will close this issue when it is merged label Feb 2, 2023
@ghost ghost locked as resolved and limited conversation to collaborators Mar 4, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@AndyAyersMS AndyAyersMS

Labels
area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI blocking-clean-ci-optional Blocking optional rolling runs
Projects
None yet
Milestone
8.0.0
5 participants
@jakobbotsch @BrianBohe @AndyAyersMS @kunalspathak @JulieLeeMSFT