Bundle_Is_Extracted test failing intermittently on Linux #43316
Assignees
Labels
area-Single-File
blocking-clean-ci
Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms'
Milestone
Comments
Dotnet-GitSync-Bot
added
area-Tracing-coreclr
untriaged
jkoritzinsky
added
area-Single-File
blocking-clean-ci
|
Tagging subscribers to this area: @agocke, @vitek-karas |
|
Failed again here: |
|
I'll take a look at this |
|
Thanks, Andy |
agocke
added a commit
to agocke/runtime
that referenced
this issue
Dec 3, 2020
Tests in the AppHost.Bundle.Tests assembly seem to randomly fail due to a race condition with the file system. They try to create separate '0','1','2'... subdirectories to isolate the published files for each test, but I think what's happening is that files may be marked for deletion, but then not deleted until a later write. For instance, files in '2' may be marked for deletion and some may fail a File.Exists check, which leads to '2' being recreated, at which point deletion may occur, which will cause the current test to fail due to a concurrent write operation. This change tries to simplify the system by sharing the test state across all the classes in the assembly, instead of per-class, and then cleaning up only when all of them are finished executing. Fixes dotnet#43316
agocke
added a commit
that referenced
this issue
Dec 8, 2020
Tests in the AppHost.Bundle.Tests assembly seem to randomly fail due to a race condition with the file system. They try to create separate '0','1','2'... subdirectories to isolate the published files for each test, but I think what's happening is that files may be marked for deletion, but then not deleted until a later write. For instance, files in '2' may be marked for deletion and some may fail a File.Exists check, which leads to '2' being recreated, at which point deletion may occur, which will cause the current test to fail due to a concurrent write operation. This change tries to avoid locking & contention by randomly generating folder names and using a (hopefully atomically created) lock file to indicate ownership of a particular name. Fixes #43316
|
Failed again: https://dev.azure.com/dnceng/public/_build/results?buildId=912265&view=ms.vss-test-web.build-test-results-tab&runId=29055812&resultId=100011&paneView=debug on |
|
Closing this until this happens again. FYI @agocke |
ghost
locked as resolved and limited conversation to collaborators
|
Failed again on https://github.com/dotnet/runtime/pull/53280/checks?check_run_id=2690948600, build: https://dev.azure.com/dnceng/public/_build/results?buildId=1160943&view=logs&jobId=fcbd3ad6-b1a4-55f2-40cc-7db598750d43&j=fcbd3ad6-b1a4-55f2-40cc-7db598750d43&t=0fc7de3b-e85e-5109-ff09-70d9e625d9be |
|
Still failing every day in CI but likely with different causes? @agocke can you please take a look? |
|
Yup, I'm actively investigating this. Currently trying to repro the latest version. It still seems like a race condition |
|
Dup of #53587 |
|
@agocke isn't that a different cause? In the log above (in my post) I don't see a text file is busy mention. |
|
I think there's some race condition in the publishing, and they might be the same one. ~~From that build in particular, there were dozens of different failures from different tests and I'm not sure that it should be even be categorized as the same failure. ~~ Nvm, was looking at the wrong build. For this one, I think it's still a race in publishing the test, since it looks like the file hasn't been completely written. I think the same thing is probably causing this failure as the others. We can re-asses if we can't find a similar root cause. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Runfo Tracking Issue: Bundle_Is_Extracted test failing intermittently on Linux
Build Result Summary
The text was updated successfully, but these errors were encountered: