Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Retry installer Process.Start if ETXTBSY #59765

Merged
merged 3 commits into from
Sep 29, 2021
Merged

Conversation

agocke
Copy link
Member

@agocke agocke commented Sep 29, 2021

Caused by #58964

Fixes #53587

@ghost
Copy link

ghost commented Sep 29, 2021

Tagging subscribers to this area: @vitek-karas, @agocke, @VSadov
See info in area-owners.md if you want to be subscribed.

Issue Details

Caused by #58964

Fixes #53587

Author: agocke
Assignees: -
Labels:

area-Host

Milestone: -

{
// 10 ms is short, but the race we're trying to avoid is in-between
// "fork" and "exec", so it should be fast
Thread.Sleep(10);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally this should print out some diagnostics - so that we know it happened. Not sure if Console.WriteLine is the correct solution here - but it might be.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Console.WriteLine is captured by XUnit so I don't think that would work.

Why would this code in particular be worthy of a diagnostic? I've found a few other places where we appear to retry without warning, e.g. in TestFileBackup:

                // Directory.Delete sometimes fails with error that the directory is not empty.
                // This is a known problem where the actual Delete call is not 100% synchronous
                // the OS reports a success but the file/folder is not fully removed yet.
                // So implement a simple retry with a short timeout.
                IOException exception = null;
                for (int retryCount = 5; retryCount > 0; retryCount--)
                {
                    try
                    {
                        Directory.Delete(_backupPath, recursive: true);
                        if (!Directory.Exists(_backupPath))
                        {
                            return;
                        }
                    }
                    catch (IOException ex)
                    {
                        exception = ex;
                    }

                    System.Threading.Thread.Sleep(200);
                }

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this has been giving us trouble, having enough info to diagnose potential issues sounds like a good idea.
I know that we don't have existing diagnostics, but ideally that should not block us from adding them 😉.

That said - I don't know how the diagnostics should look. Is there some existing mechanism for tests to output additional info to be used in case of failure (but honestly it would be great to get this even on passes if it's not too verbose).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately I don't know of a good logging mechanism at the moment. This code could be executed across many threads, so there would likely need to be some coordination for anything useful to come out.

catch (Win32Exception e) when (e.Message.Contains("Text file busy"))
{
// 10 ms is short, but the race we're trying to avoid is in-between
// "fork" and "exec", so it should be fast
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to balance latency and likelihood of waiting long enough, perhaps we should scale the backoff a little, like this could be Sleep(i * 10) ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's probably worth seeing if this solves the issue, but there's no reason we can't come back to it later.

@agocke agocke merged commit 8dd058a into dotnet:main Sep 29, 2021
@agocke agocke deleted the retry-text-busy branch September 29, 2021 23:22
@ghost ghost locked as resolved and limited conversation to collaborators Nov 3, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Host tests often fail on text file busy issues
3 participants