test_runner: add cwd option to run #54705

pmarchini · 2024-09-01T19:23:51Z

I'm opening this PR as a draft to have a place where we can discuss this implementation.

Some background context:

During #54225, we discussed the possibility of adding a new option to run.
This new option would be cwd.

This change could impact many other parts of the code, such as:
#54225 (comment)

I just pushed a new commit.

Nice. I realized that this new cwd option may introduce new bugs in at least three places:

With --experimental-test-isolation=none.

With code coverage reporting.

With test locations.

For this reason, we decided to work on this in a separate PR (#54225 (comment)).

nodejs-github-bot · 2024-09-01T19:23:56Z

Review requested:

@nodejs/test_runner

doc/api/test.md

cjihrig · 2024-09-03T18:50:41Z

With --experimental-test-isolation=none.

We need to turn the cwd option into a URL and get the href.

With code coverage reporting.

If we make the cwd a part of the globalOptions object, the coverage code will have access to it.

With test locations.

Same thing. The test object there will have a reference to the cwd.

pmarchini · 2024-09-03T19:03:04Z

@cjihrig thanks for the support, I'm gonna take a look ASAP!

doc/api/cli.md

src/node_options.cc

doc/api/cli.md

lib/internal/test_runner/utils.js

src/node_options.h

test/parallel/test-runner-run.mjs

cjihrig · 2024-09-11T13:59:52Z

src/node_options.cc

@@ -671,6 +671,9 @@ EnvironmentOptionsParser::EnvironmentOptionsParser() {
            &EnvironmentOptions::test_coverage_lines,
            kAllowedInEnvvar);

+  AddOption("--experimental-test-cwd", 


I don't think we should add this. The cwd option should be supported by the run() API only IMO.

Okay, makes sense! I was thinking about a way to set the entire test runner's cwd to allow this behavior to be configured via the CLI as well.

Btw, while working on this, I noticed an unexpected behavior related to isolation and watch mode.
I'm going to open a PR to ask for your feedback, @cjihrig

If the user wants to use the CLI, they could do something like cd path/to/tests && node --test

lib/internal/test_runner/coverage.js

cjihrig · 2024-09-12T14:07:39Z

lib/internal/test_runner/runner.js

+  if (existsSync(cwd) === false) {
+    throw new ERR_INVALID_ARG_VALUE('options.cwd', cwd, 'expects an existing directory');
+  } else if (!lstatSync(cwd).isDirectory()) {
+    throw new ERR_INVALID_ARG_VALUE('options.cwd', cwd, 'expects a directory, a file was provided');
+  }


I would rather not do this. existsSync() specifically is a race condition. Just assume that the cwd is a directory and handle any errors. For example, the child process APIs, which also take a cwd option don't perform this type of validation.

I removed the validation and added a couple of tests to check that run handles an incorrect cwd without throwing errors

cjihrig · 2024-09-12T19:55:56Z

lib/internal/test_runner/coverage.js

@@ -477,7 +477,7 @@ function sortCoverageFiles(a, b) {

 function setupCoverage(options) {
  let originalCoverageDirectory = process.env.NODE_V8_COVERAGE;
-  const cwd = process.cwd();
+  const cwd = options.cwd ?? process.cwd();


We should ensure that options.cwd always exists and remove the ?? process.cwd() fallback.

cjihrig · 2024-09-12T19:56:43Z

lib/internal/test_runner/harness.js

@@ -83,7 +83,7 @@ function createTestTree(rootTestOptions, globalOptions) {
  return globalRoot;
 }

-function createProcessEventHandler(eventName, rootTest) {
+function createProcessEventHandler(eventName, rootTest, cwd) {


I would prefer to get the cwd from the rootTest here instead of adding another argument.

In the meantime, I've seen that in harness.js we have:

node/lib/internal/test_runner/harness.js

Lines 80 to 90 in c1afd2c

harness.resetCounters();

globalRoot = new Test({

__proto__: null,

...rootTestOptions,

harness,

name: '<root>',

});

setupProcessState(globalRoot, globalOptions, harness);

globalRoot.startTime = hrtime();

return globalRoot;

}

While the method definition is:

node/lib/internal/test_runner/harness.js

Lines 182 to 184 in c1afd2c

function setupProcessState(root, globalOptions) {

const hook = createHook({

Can I remove harness from the setupProcessState method call, or should I do that in a separate PR?

lib/internal/test_runner/harness.js

src/node_options.cc

test/parallel/test-runner-no-isolation-different-cwd.mjs

Co-authored-by: Aviv Keller <[email protected]>

codecov · 2024-09-16T14:51:12Z

Codecov Report

Attention: Patch coverage is 85.71429% with 3 lines in your changes missing coverage. Please review.

Project coverage is 88.08%. Comparing base (53ede87) to head (46e63f3).
Report is 14 commits behind head on main.

Files with missing lines	Patch %	Lines
lib/internal/test_runner/runner.js	76.92%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main   #54705   +/-   ##
=======================================
  Coverage   88.07%   88.08%           
=======================================
  Files         652      652           
  Lines      183542   183550    +8     
  Branches    35862    35863    +1     
=======================================
+ Hits       161653   161672   +19     
+ Misses      15144    15123   -21     
- Partials     6745     6755   +10

Files with missing lines	Coverage Δ
lib/internal/test_runner/coverage.js	`66.06% <100.00%> (-0.06%)`	⬇️
lib/internal/test_runner/harness.js	`93.44% <100.00%> (+0.07%)`	⬆️
lib/internal/test_runner/runner.js	`89.73% <76.92%> (+1.87%)`	⬆️

... and 35 files with indirect coverage changes

pmarchini · 2024-09-18T08:06:37Z

test/parallel/test-runner-watch-mode.mjs

-    const interval = setInterval(() => renameSync(fileToRenamePath, newFileNamePath), common.platformTimeout(1000));
+    const interval = setInterval(() => {
+      renameSync(fileToRenamePath, newFileNamePath);
+      clearInterval(interval);
+    }, common.platformTimeout(1000));


Hey @cjihrig, regarding these tests, I was working on a GitHub codespace yesterday, and I noticed that this test was flaky there.
The issue was with the interval: if the system is delayed for any reason beyond the timeout, a second run of "rename" (and similarly for "delete") is triggered, causing an error because the file has already been renamed and can no longer be found under its previous name.

I'll take a look at the other similar tests

That makes sense. Timers in tests should be avoided like the plague if possible.

Do we need to use an interval for operations like that? For example, would it work to use a setTimeout() and then reschedule the callback again once the timer fires?

nodejs-github-bot added needs-ci PRs that need a full CI run. test_runner Issues and PRs related to the test runner subsystem. labels Sep 1, 2024

RedYetiDev reviewed Sep 1, 2024

View reviewed changes

doc/api/test.md Outdated Show resolved Hide resolved

doc/api/test.md Outdated Show resolved Hide resolved

pmarchini force-pushed the feat/cwd-option-run branch from 4fa3784 to 3419393 Compare September 11, 2024 09:35

RedYetiDev reviewed Sep 11, 2024

View reviewed changes

cjihrig reviewed Sep 11, 2024

View reviewed changes

pmarchini force-pushed the feat/cwd-option-run branch from fc95c78 to 518bb16 Compare September 11, 2024 14:48

RedYetiDev added the commit-queue-squash Add this label to instruct the Commit Queue to squash all the PR commits into the first one. label Sep 11, 2024

RedYetiDev reviewed Sep 12, 2024

View reviewed changes

lib/internal/test_runner/coverage.js Outdated Show resolved Hide resolved

cjihrig reviewed Sep 12, 2024

View reviewed changes

lib/internal/test_runner/harness.js Outdated Show resolved Hide resolved

cjihrig reviewed Sep 12, 2024

View reviewed changes

src/node_options.cc Outdated Show resolved Hide resolved

cjihrig reviewed Sep 12, 2024

View reviewed changes

test/parallel/test-runner-no-isolation-different-cwd.mjs Outdated Show resolved Hide resolved

pmarchini and others added 10 commits September 16, 2024 11:53

test_runner: add cwd option to run

d0dd103

test_runner: add --experimental-test-cwd option to test runner

43564be

Update doc/api/cli.md

79bffc6

Co-authored-by: Aviv Keller <[email protected]>

test_runner: refactor cwd handling in test runner

a90ba94

test: add isolation test cases to test runner run watch

c3ba4f2

test: add test cases

6c1f490

Update doc/api/test.md

4993e0e

Co-authored-by: Aviv Keller <[email protected]>

Update doc/api/test.md

1d76037

Co-authored-by: Aviv Keller <[email protected]>

test_runner: add validation for run cwd option

768c088

test_runner: remove cwd path validation

ec3ff3e

pmarchini force-pushed the feat/cwd-option-run branch from c0a6949 to ec3ff3e Compare September 16, 2024 09:57

test: update test case

b88718b

test_runner: revert new line

ac74667

pmarchini force-pushed the feat/cwd-option-run branch from 924c375 to ac74667 Compare September 16, 2024 10:06

pmarchini added 2 commits September 16, 2024 13:19

test_runner: remove cwd fallback

6141f7c

test_runner: remove additional param

6dd0915

pmarchini marked this pull request as ready for review September 16, 2024 13:21

pmarchini added 2 commits September 17, 2024 18:10

test_runner: fix unix failures

a3bea98

test: lint

46e63f3

pmarchini commented Sep 18, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test_runner: add cwd option to run #54705

test_runner: add cwd option to run #54705

pmarchini commented Sep 1, 2024

nodejs-github-bot commented Sep 1, 2024

cjihrig commented Sep 3, 2024

pmarchini commented Sep 3, 2024

cjihrig Sep 11, 2024

pmarchini Sep 11, 2024

RedYetiDev Sep 11, 2024

pmarchini Sep 11, 2024

cjihrig Sep 12, 2024

pmarchini Sep 12, 2024

cjihrig Sep 12, 2024

cjihrig Sep 12, 2024

pmarchini Sep 16, 2024

pmarchini Sep 16, 2024

codecov bot commented Sep 16, 2024 •

edited

Loading

pmarchini Sep 18, 2024

cjihrig Sep 18, 2024

	harness.resetCounters();
	globalRoot = new Test({
	__proto__: null,
	...rootTestOptions,
	harness,
	name: '<root>',
	});
	setupProcessState(globalRoot, globalOptions, harness);
	globalRoot.startTime = hrtime();
	return globalRoot;
	}


	function setupProcessState(root, globalOptions) {
	const hook = createHook({

test_runner: add cwd option to run #54705

Are you sure you want to change the base?

test_runner: add cwd option to run #54705

Conversation

pmarchini commented Sep 1, 2024

nodejs-github-bot commented Sep 1, 2024

cjihrig commented Sep 3, 2024

pmarchini commented Sep 3, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Sep 16, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Sep 16, 2024 •

edited

Loading