Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test_runner: fix support watch with run(), add globPatterns option #53866

Merged
merged 24 commits into from
Jul 24, 2024

Conversation

mcollina
Copy link
Member

This PR fixes one regression I found when using run({ watch: true }): a different test was run after a file rename.

This was caused by calling createTestFileList() from the changed event handler in

const updatedTestFiles = createTestFileList();
.

The problem is that createTestFileList() was loading the glob pattern from argv, which can be whatever. The result was but that was very hard to track.

This PR includes:

  • tests for run({ watch: true })
  • a new option called globPatterns that will allow users to control what run() will be globbing
  • avoid tracking "rename" changes if individual files were specified

@mcollina mcollina requested review from cjihrig and MoLow July 16, 2024 13:19
@nodejs-github-bot
Copy link
Collaborator

Review requested:

  • @nodejs/test_runner

@nodejs-github-bot nodejs-github-bot added needs-ci PRs that need a full CI run. test_runner Issues and PRs related to the test runner subsystem. labels Jul 16, 2024
@mcollina mcollina added the request-ci Add this label to start a Jenkins CI on a PR. label Jul 16, 2024
// This is needed, because calling run() with no files will
// load files matching from cwd, which includes all content of test/
tmpdir.refresh();
process.chdir(tmpdir.path);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Won't this break running the test suite in worker threads?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without this, I would need to fix #53867 as well in this PR.

Let's see if it actually breaks something in CI.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like it did break in the CI (https://ci.nodejs.org/job/node-test-commit-custom-suites-freestyle/37045/). The issue is that the Node test suite is run within a worker thread in the CI. Any test that uses an API that is unsupported in worker threads needs to be skipped there.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we still need to remove this.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok. I'll need to add a cwd parameter to run(), otherwise it won't work.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would hold off on adding a cwd parameter for now, but I do foresee us adding it in the future. I'm just not sure if it will break anything until the necessary refactoring happens. Another option is to spawn a child process with the correct cwd.

test/parallel/test-runner-run.mjs Outdated Show resolved Hide resolved
test/parallel/test-runner-run.mjs Outdated Show resolved Hide resolved
lib/internal/test_runner/runner.js Show resolved Hide resolved
test/fixtures/test-runner-watch.mjs Outdated Show resolved Hide resolved
lib/internal/test_runner/runner.js Outdated Show resolved Hide resolved
@github-actions github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Jul 16, 2024
@nodejs-github-bot
Copy link
Collaborator

@mcollina
Copy link
Member Author

@cjihrig @MoLow can you tell me why this is trying to load tools/github_reporter/index.js?

=== release test-runner-run ===
Path: parallel/test-runner-run
Error: --- stderr ---
node:internal/test_runner/harness:46
      throw err;
      ^

Error [ERR_MODULE_NOT_FOUND]: Cannot find module '/home/runner/work/node/node/test/.tmp.2615/tools/github_reporter/index.js' imported from /home/runner/work/node/node/test/.tmp.2615/
    at finalizeResolution (node:internal/modules/esm/resolve:260:11)
    at moduleResolve (node:internal/modules/esm/resolve:921:10)
    at defaultResolve (node:internal/modules/esm/resolve:1123:11)
    at ModuleLoader.defaultResolve (node:internal/modules/esm/loader:557:12)
    at ModuleLoader.resolve (node:internal/modules/esm/loader:526:25)
    at ModuleLoader.getModuleJob (node:internal/modules/esm/loader:249:38)
    at onImport.tracePromise.__proto__ (node:internal/modules/esm/loader:484:36)
    at TracingChannel.tracePromise (node:diagnostics_channel:337:14)
    at ModuleLoader.import (node:internal/modules/esm/loader:483:21)
    at node:internal/test_runner/utils:158:45 {
  code: 'ERR_MODULE_NOT_FOUND',
  url: 'file:///home/runner/work/node/node/test/.tmp.2615/tools/github_reporter/index.js'
}

@cjihrig
Copy link
Contributor

cjihrig commented Jul 16, 2024

My guess is because the Python test runner uses it and the process.chdir() call broke that logic.

@mcollina
Copy link
Member Author

I guess it's #53867 at play.

@cjihrig
Copy link
Contributor

cjihrig commented Jul 16, 2024

Yep. But also, the Python runner should probably not use a relative path like that.

doc/api/test.md Show resolved Hide resolved
doc/api/test.md Outdated Show resolved Hide resolved
@@ -72,6 +72,7 @@ const options = {
setup: setupTestReporters,
timeout,
shard,
globPatterns: process.argv.slice(1),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use primordials?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like this is not addressed.

test/parallel/test-runner-run-watch.mjs Show resolved Hide resolved
test/parallel/test-runner-run-watch.mjs Outdated Show resolved Hide resolved
test/parallel/test-runner-run-watch.mjs Outdated Show resolved Hide resolved
test/parallel/test-runner-run-watch.mjs Outdated Show resolved Hide resolved
@mcollina
Copy link
Member Author

@cjihrig @atlowChemi @MoLow PTAL, all reviews should have been handled.

@mcollina mcollina added request-ci Add this label to start a Jenkins CI on a PR. semver-minor PRs that contain new features and should be released in the next minor version. labels Jul 18, 2024
@github-actions github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Jul 18, 2024
@nodejs-github-bot
Copy link
Collaborator

// This is needed, because calling run() with no files will
// load files matching from cwd, which includes all content of test/
tmpdir.refresh();
process.chdir(tmpdir.path);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we still need to remove this.

@@ -72,6 +72,7 @@ const options = {
setup: setupTestReporters,
timeout,
shard,
globPatterns: process.argv.slice(1),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like this is not addressed.

@@ -503,6 +505,16 @@ function run(options = kEmptyObject) {
if (only != null) {
validateBoolean(only, 'options.only');
}
if (globPatterns != null) {
validateArray(globPatterns, 'options.globPatterns');
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should probably validate that the array is not empty. You can add the logic here. Otherwise, I will probably do it in a follow up.

const patterns = hasUserSuppliedPattern ? ArrayPrototypeSlice(process.argv, 1) : [kDefaultPattern];
const hasUserSuppliedPattern = patterns != null;
if (!patterns || patterns.length === 0) {
patterns = [kDefaultPattern];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would assign the default patterns in the run() validation. You can do that in this PR, otherwise, I will probably do it in a follow up.

@mcollina mcollina added the request-ci Add this label to start a Jenkins CI on a PR. label Jul 18, 2024
@github-actions github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Jul 18, 2024
@nodejs-github-bot
Copy link
Collaborator

@mcollina mcollina added the request-ci Add this label to start a Jenkins CI on a PR. label Jul 18, 2024
@github-actions github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Jul 18, 2024
@nodejs-github-bot
Copy link
Collaborator

@mcollina
Copy link
Member Author

mcollina commented Jul 18, 2024

Can someone help me with this? I see no conflicts locally but CI does: https://ci.nodejs.org/job/node-test-commit/72259/console.

Was by any chance a force push broke something?

@github-actions github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Jul 23, 2024
@nodejs-github-bot
Copy link
Collaborator

Signed-off-by: Matteo Collina <[email protected]>
@mcollina mcollina added the request-ci Add this label to start a Jenkins CI on a PR. label Jul 24, 2024
@github-actions github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Jul 24, 2024
@nodejs-github-bot
Copy link
Collaborator

Signed-off-by: Matteo Collina <[email protected]>
@mcollina mcollina added the request-ci Add this label to start a Jenkins CI on a PR. label Jul 24, 2024
@github-actions github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Jul 24, 2024
@nodejs-github-bot
Copy link
Collaborator

@mcollina mcollina added commit-queue-squash Add this label to instruct the Commit Queue to squash all the PR commits into the first one. commit-queue Add this label to land a pull request using GitHub Actions. labels Jul 24, 2024
@nodejs-github-bot nodejs-github-bot removed the commit-queue Add this label to land a pull request using GitHub Actions. label Jul 24, 2024
@nodejs-github-bot nodejs-github-bot merged commit 2208948 into nodejs:main Jul 24, 2024
59 checks passed
@nodejs-github-bot
Copy link
Collaborator

Landed in 2208948

@mcollina mcollina deleted the support-watch-in-run branch July 24, 2024 13:26
targos pushed a commit that referenced this pull request Jul 28, 2024
This commit updates the test runner's code coverage so that
coverage options are explicitly passed in instead of pulled
from command line options.

PR-URL: #53931
Refs: #53924
Refs: #53867
Refs: #53866
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Moshe Atlow <[email protected]>
Reviewed-By: Chemi Atlow <[email protected]>
Reviewed-By: James M Snell <[email protected]>
targos pushed a commit that referenced this pull request Jul 28, 2024
Signed-off-by: Matteo Collina <[email protected]>
PR-URL: #53866
Reviewed-By: Chemi Atlow <[email protected]>
Reviewed-By: Colin Ihrig <[email protected]>
Reviewed-By: Moshe Atlow <[email protected]>
RafaelGSS added a commit that referenced this pull request Jul 30, 2024
Notable changes:

deps:
  * (SEMVER-MINOR) V8: backport 7857eb34db42 (Stephen Belanger) #53997
http:
  * (SEMVER-MINOR) add diagnostics channel `http.client.request.error` (Kohei Ueno) #54054
inspector:
  * (SEMVER-MINOR) add initial support for network inspection (Kohei Ueno) #53593
lib,src:
  * drop --experimental-network-imports (Rafael Gonzaga) #53822
meta:
  * add jake to collaborators (jakecastelli) #54004
module:
  * (SEMVER-MINOR) add --experimental-strip-types (Marco Ippolito) #53725
  * (SEMVER-MINOR) unflag detect-module (Geoffrey Booth) #53619
stream:
  * (SEMVER-MINOR) expose DuplexPair API (Austin Wright) #34111
test_runner:
  * (SEMVER-MINOR) fix support watch with run(), add globPatterns option (Matteo Collina) #53866
  * (SEMVER-MINOR) refactor snapshots to get file from context (Colin Ihrig) #53853
  * (SEMVER-MINOR) add context.filePath (Colin Ihrig) #53853

PR-URL: TODO
@RafaelGSS RafaelGSS mentioned this pull request Jul 30, 2024
RafaelGSS added a commit that referenced this pull request Jul 30, 2024
Notable changes:

deps:
  * (SEMVER-MINOR) V8: backport 7857eb34db42 (Stephen Belanger) #53997
http:
  * (SEMVER-MINOR) add diagnostics channel `http.client.request.error` (Kohei Ueno) #54054
inspector:
  * (SEMVER-MINOR) add initial support for network inspection (Kohei Ueno) #53593
lib,src:
  * drop --experimental-network-imports (Rafael Gonzaga) #53822
meta:
  * add jake to collaborators (jakecastelli) #54004
module:
  * (SEMVER-MINOR) add --experimental-strip-types (Marco Ippolito) #53725
  * (SEMVER-MINOR) unflag detect-module (Geoffrey Booth) #53619
stream:
  * (SEMVER-MINOR) expose DuplexPair API (Austin Wright) #34111
test_runner:
  * (SEMVER-MINOR) fix support watch with run(), add globPatterns option (Matteo Collina) #53866
  * (SEMVER-MINOR) refactor snapshots to get file from context (Colin Ihrig) #53853
  * (SEMVER-MINOR) add context.filePath (Colin Ihrig) #53853

PR-URL: #54123
RafaelGSS pushed a commit that referenced this pull request Aug 5, 2024
This commit updates the test runner's code coverage so that
coverage options are explicitly passed in instead of pulled
from command line options.

PR-URL: #53931
Refs: #53924
Refs: #53867
Refs: #53866
Reviewed-By: Matteo Collina <[email protected]>
Reviewed-By: Moshe Atlow <[email protected]>
Reviewed-By: Chemi Atlow <[email protected]>
Reviewed-By: James M Snell <[email protected]>
RafaelGSS pushed a commit that referenced this pull request Aug 5, 2024
Signed-off-by: Matteo Collina <[email protected]>
PR-URL: #53866
Reviewed-By: Chemi Atlow <[email protected]>
Reviewed-By: Colin Ihrig <[email protected]>
Reviewed-By: Moshe Atlow <[email protected]>
RafaelGSS added a commit that referenced this pull request Aug 5, 2024
Notable changes:

deps:
  * (SEMVER-MINOR) V8: backport 7857eb34db42 (Stephen Belanger) #53997
http:
  * (SEMVER-MINOR) add diagnostics channel `http.client.request.error` (Kohei Ueno) #54054
inspector:
  * (SEMVER-MINOR) add initial support for network inspection (Kohei Ueno) #53593
lib,src:
  * drop --experimental-network-imports (Rafael Gonzaga) #53822
meta:
  * add jake to collaborators (jakecastelli) #54004
module:
  * (SEMVER-MINOR) add --experimental-strip-types (Marco Ippolito) #53725
stream:
  * (SEMVER-MINOR) expose DuplexPair API (Austin Wright) #34111
test_runner:
  * (SEMVER-MINOR) fix support watch with run(), add globPatterns option (Matteo Collina) #53866
  * (SEMVER-MINOR) refactor snapshots to get file from context (Colin Ihrig) #53853
  * (SEMVER-MINOR) add context.filePath (Colin Ihrig) #53853

PR-URL: #54123
RafaelGSS added a commit that referenced this pull request Aug 5, 2024
Notable changes:

deps:
  * (SEMVER-MINOR) V8: backport 7857eb34db42 (Stephen Belanger) #53997
http:
  * (SEMVER-MINOR) add diagnostics channel `http.client.request.error` (Kohei Ueno) #54054
inspector:
  * (SEMVER-MINOR) add initial support for network inspection (Kohei Ueno) #53593
lib,src:
  * drop --experimental-network-imports (Rafael Gonzaga) #53822
meta:
  * add jake to collaborators (jakecastelli) #54004
module:
  * (SEMVER-MINOR) add --experimental-strip-types (Marco Ippolito) #53725
stream:
  * (SEMVER-MINOR) expose DuplexPair API (Austin Wright) #34111
test_runner:
  * (SEMVER-MINOR) fix support watch with run(), add globPatterns option (Matteo Collina) #53866
  * (SEMVER-MINOR) refactor snapshots to get file from context (Colin Ihrig) #53853
  * (SEMVER-MINOR) add context.filePath (Colin Ihrig) #53853

PR-URL: #54123
RafaelGSS added a commit that referenced this pull request Aug 6, 2024
Notable changes:

deps:
  * (SEMVER-MINOR) V8: backport 7857eb34db42 (Stephen Belanger) #53997
http:
  * (SEMVER-MINOR) add diagnostics channel `http.client.request.error` (Kohei Ueno) #54054
inspector:
  * (SEMVER-MINOR) add initial support for network inspection (Kohei Ueno) #53593
lib,src:
  * drop --experimental-network-imports (Rafael Gonzaga) #53822
meta:
  * add jake to collaborators (jakecastelli) #54004
module:
  * (SEMVER-MINOR) add --experimental-strip-types (Marco Ippolito) #53725
stream:
  * (SEMVER-MINOR) expose DuplexPair API (Austin Wright) #34111
test_runner:
  * (SEMVER-MINOR) fix support watch with run(), add globPatterns option (Matteo Collina) #53866
  * (SEMVER-MINOR) refactor snapshots to get file from context (Colin Ihrig) #53853
  * (SEMVER-MINOR) add context.filePath (Colin Ihrig) #53853

PR-URL: #54123
@targos targos added the dont-land-on-v20.x PRs that should not land on the v20.x-staging branch and should not be released in v20.x. label Sep 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
commit-queue-squash Add this label to instruct the Commit Queue to squash all the PR commits into the first one. dont-land-on-v20.x PRs that should not land on the v20.x-staging branch and should not be released in v20.x. needs-ci PRs that need a full CI run. semver-minor PRs that contain new features and should be released in the next minor version. test_runner Issues and PRs related to the test runner subsystem.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants