Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

azure win10 vs2019 machines have been timing out in debugger tests #3014

Closed
joyeecheung opened this issue Aug 22, 2022 · 10 comments
Closed

azure win10 vs2019 machines have been timing out in debugger tests #3014

joyeecheung opened this issue Aug 22, 2022 · 10 comments

Comments

@joyeecheung
Copy link
Member

See nodejs/reliability#354 for reference, it seems the earliest CI run that exhibited this was https://ci.nodejs.org/job/node-test-binary-windows-js-suites/16283/, which is for a PR that hasn't landed rebased on top of a doc commit nodejs/node@dcc9589 so it might be infra issues.

@joyeecheung
Copy link
Member Author

The daily master build on the commit prior to the doc commit was also green on those machines: nodejs/node@5e57d24

@joyeecheung
Copy link
Member Author

joyeecheung commented Aug 22, 2022

I do not know how to log into those machines, but I suspect that there might be some stale Node.js processes running there that block the port 9229. That could cause these tests to timeout.

@richardlau
Copy link
Member

I couldn't see any stale Node.js processes still running on test-azure_msft-win10_vs2019-x64-1 or test-azure_msft-win10_vs2019-x64-2 but I've just scheduled a reboot of both machines.

@joyeecheung
Copy link
Member Author

The failures still showed up with nodejs/node#44342, so there must be something else that is causing the time out. Also I noticed that this only happes with test-azure_msft-win10_vs2019-x64-1 and -2, the tests don't fail on -3.

@joyeecheung
Copy link
Member Author

Looking at the build history, this seems to happen more with test-azure_msft-win10_vs2019-x64-1, and somewhat less frequently with -2. -3 and -4 seem to be immune to that.

@richardlau
Copy link
Member

FWIW -1 and -2 did get rebooted (#3014 (comment)) so whatever the problem is it has persisted, or recreated itself, across the reboot.

@joyeecheung
Copy link
Member Author

joyeecheung commented Aug 22, 2022

Another observation: it seems sometimes these failures are accompanied by a few random failures from other tests that involve timeouts, e.g. sequential/test-heap-prof, parallel/test-child-process-exec-timeout, see nodejs/reliability#354

@tniessen
Copy link
Member

It's been failing almost every time, making it difficult to land PRs:

Windows build trend showing lots of failures

nodejs-github-bot pushed a commit to nodejs/node that referenced this issue Aug 25, 2022
sequential/test-child-process-execsync and
parallel/test-child-process-spawnsync-timeout are both flaky
on azure Windows machines, where it may take longer for Node.js
to launch and receive output from child processes. These tests
work by spawning a child processes that is supposed to sleep
for a long time, but the option is configured so that Node.js
would terminate them early when a shorter timeout is reached.
Then the tests assert that the time taken for the whole thing
is shorter than the specified sleep time (meaning the process
don't actually get to sleep for that long). To make the tests
less brittle on azure Windows, this patch raises the sleep
times in those tests on Windows platform, so that the overhead
can be taken into account there.

PR-URL: #44375
Refs: nodejs/build#3014
Reviewed-By: Moshe Atlow <[email protected]>
Reviewed-By: Nitzan Uziely <[email protected]>
sidwebworks pushed a commit to sidwebworks/node that referenced this issue Aug 26, 2022
sequential/test-child-process-execsync and
parallel/test-child-process-spawnsync-timeout are both flaky
on azure Windows machines, where it may take longer for Node.js
to launch and receive output from child processes. These tests
work by spawning a child processes that is supposed to sleep
for a long time, but the option is configured so that Node.js
would terminate them early when a shorter timeout is reached.
Then the tests assert that the time taken for the whole thing
is shorter than the specified sleep time (meaning the process
don't actually get to sleep for that long). To make the tests
less brittle on azure Windows, this patch raises the sleep
times in those tests on Windows platform, so that the overhead
can be taken into account there.

PR-URL: nodejs#44375
Refs: nodejs/build#3014
Reviewed-By: Moshe Atlow <[email protected]>
Reviewed-By: Nitzan Uziely <[email protected]>
@joyeecheung
Copy link
Member Author

joyeecheung commented Aug 29, 2022

Looks like the flakes are now gone, the only recent repros in the reliability issues came from resumed builds that weren't rebased. I'll close this issue.

@tniessen
Copy link
Member

Thank you for working on this @joyeecheung :)

joyeecheung added a commit to joyeecheung/node that referenced this issue Sep 1, 2022
sequential/test-child-process-execsync and
parallel/test-child-process-spawnsync-timeout are both flaky
on azure Windows machines, where it may take longer for Node.js
to launch and receive output from child processes. These tests
work by spawning a child processes that is supposed to sleep
for a long time, but the option is configured so that Node.js
would terminate them early when a shorter timeout is reached.
Then the tests assert that the time taken for the whole thing
is shorter than the specified sleep time (meaning the process
don't actually get to sleep for that long). To make the tests
less brittle on azure Windows, this patch raises the sleep
times in those tests on Windows platform, so that the overhead
can be taken into account there.

PR-URL: nodejs#44375
Refs: nodejs/build#3014
Reviewed-By: Moshe Atlow <[email protected]>
Reviewed-By: Nitzan Uziely <[email protected]>
joyeecheung added a commit to joyeecheung/node that referenced this issue Sep 1, 2022
sequential/test-child-process-execsync and
parallel/test-child-process-spawnsync-timeout are both flaky
on azure Windows machines, where it may take longer for Node.js
to launch and receive output from child processes. These tests
work by spawning a child processes that is supposed to sleep
for a long time, but the option is configured so that Node.js
would terminate them early when a shorter timeout is reached.
Then the tests assert that the time taken for the whole thing
is shorter than the specified sleep time (meaning the process
don't actually get to sleep for that long). To make the tests
less brittle on azure Windows, this patch raises the sleep
times in those tests on Windows platform, so that the overhead
can be taken into account there.

PR-URL: nodejs#44375
Refs: nodejs/build#3014
Reviewed-By: Moshe Atlow <[email protected]>
Reviewed-By: Nitzan Uziely <[email protected]>
RafaelGSS pushed a commit to nodejs/node that referenced this issue Sep 5, 2022
sequential/test-child-process-execsync and
parallel/test-child-process-spawnsync-timeout are both flaky
on azure Windows machines, where it may take longer for Node.js
to launch and receive output from child processes. These tests
work by spawning a child processes that is supposed to sleep
for a long time, but the option is configured so that Node.js
would terminate them early when a shorter timeout is reached.
Then the tests assert that the time taken for the whole thing
is shorter than the specified sleep time (meaning the process
don't actually get to sleep for that long). To make the tests
less brittle on azure Windows, this patch raises the sleep
times in those tests on Windows platform, so that the overhead
can be taken into account there.

PR-URL: #44375
Refs: nodejs/build#3014
Reviewed-By: Moshe Atlow <[email protected]>
Reviewed-By: Nitzan Uziely <[email protected]>
joyeecheung added a commit to joyeecheung/node that referenced this issue Sep 6, 2022
sequential/test-child-process-execsync and
parallel/test-child-process-spawnsync-timeout are both flaky
on azure Windows machines, where it may take longer for Node.js
to launch and receive output from child processes. These tests
work by spawning a child processes that is supposed to sleep
for a long time, but the option is configured so that Node.js
would terminate them early when a shorter timeout is reached.
Then the tests assert that the time taken for the whole thing
is shorter than the specified sleep time (meaning the process
don't actually get to sleep for that long). To make the tests
less brittle on azure Windows, this patch raises the sleep
times in those tests on Windows platform, so that the overhead
can be taken into account there.

PR-URL: nodejs#44375
Refs: nodejs/build#3014
Reviewed-By: Moshe Atlow <[email protected]>
Reviewed-By: Nitzan Uziely <[email protected]>
Fyko pushed a commit to Fyko/node that referenced this issue Sep 15, 2022
sequential/test-child-process-execsync and
parallel/test-child-process-spawnsync-timeout are both flaky
on azure Windows machines, where it may take longer for Node.js
to launch and receive output from child processes. These tests
work by spawning a child processes that is supposed to sleep
for a long time, but the option is configured so that Node.js
would terminate them early when a shorter timeout is reached.
Then the tests assert that the time taken for the whole thing
is shorter than the specified sleep time (meaning the process
don't actually get to sleep for that long). To make the tests
less brittle on azure Windows, this patch raises the sleep
times in those tests on Windows platform, so that the overhead
can be taken into account there.

PR-URL: nodejs#44375
Refs: nodejs/build#3014
Reviewed-By: Moshe Atlow <[email protected]>
Reviewed-By: Nitzan Uziely <[email protected]>
targos pushed a commit to nodejs/node that referenced this issue Sep 16, 2022
sequential/test-child-process-execsync and
parallel/test-child-process-spawnsync-timeout are both flaky
on azure Windows machines, where it may take longer for Node.js
to launch and receive output from child processes. These tests
work by spawning a child processes that is supposed to sleep
for a long time, but the option is configured so that Node.js
would terminate them early when a shorter timeout is reached.
Then the tests assert that the time taken for the whole thing
is shorter than the specified sleep time (meaning the process
don't actually get to sleep for that long). To make the tests
less brittle on azure Windows, this patch raises the sleep
times in those tests on Windows platform, so that the overhead
can be taken into account there.

PR-URL: #44375
Refs: nodejs/build#3014
Reviewed-By: Moshe Atlow <[email protected]>
Reviewed-By: Nitzan Uziely <[email protected]>
juanarbol pushed a commit to nodejs/node that referenced this issue Oct 10, 2022
sequential/test-child-process-execsync and
parallel/test-child-process-spawnsync-timeout are both flaky
on azure Windows machines, where it may take longer for Node.js
to launch and receive output from child processes. These tests
work by spawning a child processes that is supposed to sleep
for a long time, but the option is configured so that Node.js
would terminate them early when a shorter timeout is reached.
Then the tests assert that the time taken for the whole thing
is shorter than the specified sleep time (meaning the process
don't actually get to sleep for that long). To make the tests
less brittle on azure Windows, this patch raises the sleep
times in those tests on Windows platform, so that the overhead
can be taken into account there.

PR-URL: #44375
Refs: nodejs/build#3014
Reviewed-By: Moshe Atlow <[email protected]>
Reviewed-By: Nitzan Uziely <[email protected]>
juanarbol pushed a commit to nodejs/node that referenced this issue Oct 11, 2022
sequential/test-child-process-execsync and
parallel/test-child-process-spawnsync-timeout are both flaky
on azure Windows machines, where it may take longer for Node.js
to launch and receive output from child processes. These tests
work by spawning a child processes that is supposed to sleep
for a long time, but the option is configured so that Node.js
would terminate them early when a shorter timeout is reached.
Then the tests assert that the time taken for the whole thing
is shorter than the specified sleep time (meaning the process
don't actually get to sleep for that long). To make the tests
less brittle on azure Windows, this patch raises the sleep
times in those tests on Windows platform, so that the overhead
can be taken into account there.

PR-URL: #44375
Refs: nodejs/build#3014
Reviewed-By: Moshe Atlow <[email protected]>
Reviewed-By: Nitzan Uziely <[email protected]>
guangwong pushed a commit to noslate-project/node that referenced this issue Jan 3, 2023
sequential/test-child-process-execsync and
parallel/test-child-process-spawnsync-timeout are both flaky
on azure Windows machines, where it may take longer for Node.js
to launch and receive output from child processes. These tests
work by spawning a child processes that is supposed to sleep
for a long time, but the option is configured so that Node.js
would terminate them early when a shorter timeout is reached.
Then the tests assert that the time taken for the whole thing
is shorter than the specified sleep time (meaning the process
don't actually get to sleep for that long). To make the tests
less brittle on azure Windows, this patch raises the sleep
times in those tests on Windows platform, so that the overhead
can be taken into account there.

PR-URL: nodejs/node#44375
Refs: nodejs/build#3014
Reviewed-By: Moshe Atlow <[email protected]>
Reviewed-By: Nitzan Uziely <[email protected]>
guangwong pushed a commit to noslate-project/node that referenced this issue Jan 3, 2023
sequential/test-child-process-execsync and
parallel/test-child-process-spawnsync-timeout are both flaky
on azure Windows machines, where it may take longer for Node.js
to launch and receive output from child processes. These tests
work by spawning a child processes that is supposed to sleep
for a long time, but the option is configured so that Node.js
would terminate them early when a shorter timeout is reached.
Then the tests assert that the time taken for the whole thing
is shorter than the specified sleep time (meaning the process
don't actually get to sleep for that long). To make the tests
less brittle on azure Windows, this patch raises the sleep
times in those tests on Windows platform, so that the overhead
can be taken into account there.

PR-URL: nodejs/node#44375
Refs: nodejs/build#3014
Reviewed-By: Moshe Atlow <[email protected]>
Reviewed-By: Nitzan Uziely <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants