fs: buffer dir entries in opendir() #29893

addaleax · 2019-10-08T23:26:46Z

Read up to 32 directory entries in one batch when dir.readSync()
or dir.read() are called.

This increases performance significantly, although it introduces
quite a bit of edge case complexity.

                                                             confidence improvement accuracy (*)    (**)    (***)
 fs/bench-opendir.js mode='async' dir='lib' n=100                  ***    155.93 %      ±30.05% ±40.34%  ±53.21%
 fs/bench-opendir.js mode='async' dir='test/parallel' n=100        ***    479.65 %      ±56.81% ±76.47% ±101.32%
 fs/bench-opendir.js mode='sync' dir='lib' n=100                           10.38 %      ±14.39% ±19.16%  ±24.96%
 fs/bench-opendir.js mode='sync' dir='test/parallel' n=100         ***     63.13 %      ±12.84% ±17.18%  ±22.58%

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
commit message follows commit guidelines

nodejs-github-bot · 2019-10-08T23:27:46Z

CI: https://ci.nodejs.org/job/node-test-pull-request/25915/

mscdex · 2019-10-09T00:27:44Z

So this means that instead of 1 Dirent the APIs will now return an array of Dirents? If so, doesn't the documentation need to be updated? Also, wouldn't it be a good idea to have the number be configurable (via options passed to dir.read()/dir.readSync())?

addaleax · 2019-10-09T00:37:18Z

So this means that instead of 1 Dirent the APIs will now return an array of Dirents? If so, doesn't the documentation need to be updated?

No, the API is unaffected and will still only return one item at a time.

Also, wouldn't it be a good idea to have the number be configurable (via options passed to dir.read()/dir.readSync()`)?

Maybe? If you intentionally want a large number of directory entries at once you probably wouldn’t use the streaming API…

src/node_dir.cc

src/node_dir.h

nodejs-github-bot · 2019-10-09T15:34:19Z

CI: https://ci.nodejs.org/job/node-test-pull-request/25926/

BridgeAR

This is pretty great work!

benchmark/fs/bench-opendir.js

lib/internal/fs/dir.js

src/node_dir.cc

This is unlikely to be necessary in any case, and causes much unwarrented complexity when implementing further optimizations. Refs: nodejs#29893 (comment)

This is unlikely to be necessary in any case, and causes much unwarrented complexity when implementing further optimizations. Refs: #29893 (comment) PR-URL: #29908 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Anna Henningsen <[email protected]> Reviewed-By: Ruben Bridgewater <[email protected]>

Fishrock123 · 2019-10-09T22:08:29Z

Ok I landed #29908, lets see what this looks like without all that extra encoding nonsense. 😅

addaleax · 2019-10-09T22:40:26Z

@cjihrig @devnexen @Fishrock123 I’ve rebased this and it’s quite a bit simpler now, feel free to take another look

nodejs-github-bot · 2019-10-09T22:42:28Z

CI: https://ci.nodejs.org/job/node-test-pull-request/25934/

jasnell · 2019-10-09T23:34:44Z

Just thinking about it... given that opendir() is an async iteration over the directory entries, it may be good to document what happens when new entries are created while the iterator is still iterating...

e.g.

const fs = require('fs');

let n = 0;

async function print(path) {
  const dir = await fs.promises.opendir(path);
  for await (const dirent of dir) {
    fs.mkdirSync(`./foo${n++}`);
    console.log(dirent.name);
  }
}
print('./').catch(console.error);

and

const fs = require('fs');

const dir = fs.opendirSync(__dirname);

console.log(dir.readSync());
fs.mkdirSync('foo');
console.log(dir.readSync());

(the answer, of course, is that the newly created entries are not included in the iteration, but that should be documented :-) ...)

This is unlikely to be necessary in any case, and causes much unwarrented complexity when implementing further optimizations. Refs: #29893 (comment) PR-URL: #29908 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Anna Henningsen <[email protected]> Reviewed-By: Ruben Bridgewater <[email protected]>

Fishrock123

(Also if you could amend the commit to note that it it also includes misc cleanup that would be ideal)

lib/internal/fs/dir.js

Fishrock123 · 2019-10-10T19:22:56Z

@jasnell I believe that also applies to uv_fs_scandir aka fs.readdir(), since other threads could operate on the directory on disk between I/O calls.

Read up to 32 directory entries in one batch when `dir.readSync()` or `dir.read()` are called. This increases performance significantly, although it introduces quite a bit of edge case complexity. confidence improvement accuracy (*) (**) (***) fs/bench-opendir.js mode='async' dir='lib' n=100 *** 155.93 % ±30.05% ±40.34% ±53.21% fs/bench-opendir.js mode='async' dir='test/parallel' n=100 *** 479.65 % ±56.81% ±76.47% ±101.32% fs/bench-opendir.js mode='sync' dir='lib' n=100 10.38 % ±14.39% ±19.16% ±24.96% fs/bench-opendir.js mode='sync' dir='test/parallel' n=100 *** 63.13 % ±12.84% ±17.18% ±22.58%

addaleax · 2019-10-10T19:52:08Z

Just thinking about it... given that opendir() is an async iteration over the directory entries, it may be good to document what happens when new entries are created while the iterator is still iterating...

[...]

(the answer, of course, is that the newly created entries are not included in the iteration, but that should be documented :-) ...)

That is not the answer, sorry – POSIX says:

If a file is removed from or added to the directory after the most recent call to opendir() or rewinddir(), whether a subsequent call to readdir() returns an entry for that file is unspecified.

I think it’s fair for our docs to reflect that, though. I’ve pushed c66840d for that.

@jasnell I believe that also applies to uv_fs_scandir aka fs.readdir(), since other threads could operate on the directory on disk between I/O calls.

(I assume the same goes for that, too.)

(Also if you could amend the commit to note that it it also includes misc cleanup that would be ideal)

I’m not really seeing anything unrelated to the buffering, besides maybe removing the GetAsyncWrap() fn?

Fishrock123 · 2019-10-11T04:49:45Z

benchmark/fs/bench-opendir.js

+
+        read();
+      });
+    } {


@addaleax This patch seems wrong? There should be an else here... I think the benchmark you ran included readSync() along with the callback one...

Oops, yes … funny the linter wouldn’t complain about this? The correct benchmark results for changing to setImmediate are

confidence improvement accuracy (*) (**) (***) fs/bench-opendir.js mode='callback' dir='lib' n=100 3.40 % ±15.95% ±21.22% ±27.62% fs/bench-opendir.js mode='callback' dir='test/parallel' n=100 *** -43.16 % ±10.70% ±14.27% ±18.66%

This result seems strange to me, how is that worse than before?

Regardless though after further though I think that nextTick is the reasonable thing to do.

@Fishrock123 I think it makes sense – as you noted, the readSync() part was run unconditionally before, so its (unchanged) performance was included in the -30 %. If we only benchmark what has been slowed down, seeing more negative impact seems about right to me.

nodejs-github-bot · 2019-10-11T18:16:12Z

CI: https://ci.nodejs.org/job/node-test-pull-request/25966/

Read up to 32 directory entries in one batch when `dir.readSync()` or `dir.read()` are called. This increases performance significantly, although it introduces quite a bit of edge case complexity. confidence improvement accuracy (*) (**) (***) fs/bench-opendir.js mode='async' dir='lib' n=100 *** 155.93 % ±30.05% ±40.34% ±53.21% fs/bench-opendir.js mode='async' dir='test/parallel' n=100 *** 479.65 % ±56.81% ±76.47% ±101.32% fs/bench-opendir.js mode='sync' dir='lib' n=100 10.38 % ±14.39% ±19.16% ±24.96% fs/bench-opendir.js mode='sync' dir='test/parallel' n=100 *** 63.13 % ±12.84% ±17.18% ±22.58% PR-URL: #29893 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: David Carlier <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Jeremiah Senkpiel <[email protected]>

PR-URL: #29893 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: David Carlier <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Jeremiah Senkpiel <[email protected]>

addaleax · 2019-10-11T21:10:08Z

Landed in 5c93aab 7812a61

Read up to 32 directory entries in one batch when `dir.readSync()` or `dir.read()` are called. This increases performance significantly, although it introduces quite a bit of edge case complexity. confidence improvement accuracy (*) (**) (***) fs/bench-opendir.js mode='async' dir='lib' n=100 *** 155.93 % ±30.05% ±40.34% ±53.21% fs/bench-opendir.js mode='async' dir='test/parallel' n=100 *** 479.65 % ±56.81% ±76.47% ±101.32% fs/bench-opendir.js mode='sync' dir='lib' n=100 10.38 % ±14.39% ±19.16% ±24.96% fs/bench-opendir.js mode='sync' dir='test/parallel' n=100 *** 63.13 % ±12.84% ±17.18% ±22.58% PR-URL: #29893 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: David Carlier <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Jeremiah Senkpiel <[email protected]>

PR-URL: #29893 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: David Carlier <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Jeremiah Senkpiel <[email protected]>

Read up to 32 directory entries in one batch when `dir.readSync()` or `dir.read()` are called. This increases performance significantly, although it introduces quite a bit of edge case complexity. confidence improvement accuracy (*) (**) (***) fs/bench-opendir.js mode='async' dir='lib' n=100 *** 155.93 % ±30.05% ±40.34% ±53.21% fs/bench-opendir.js mode='async' dir='test/parallel' n=100 *** 479.65 % ±56.81% ±76.47% ±101.32% fs/bench-opendir.js mode='sync' dir='lib' n=100 10.38 % ±14.39% ±19.16% ±24.96% fs/bench-opendir.js mode='sync' dir='test/parallel' n=100 *** 63.13 % ±12.84% ±17.18% ±22.58% PR-URL: #29893 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: David Carlier <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Jeremiah Senkpiel <[email protected]>

PR-URL: #29893 Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: David Carlier <[email protected]> Reviewed-By: James M Snell <[email protected]> Reviewed-By: Jeremiah Senkpiel <[email protected]>

addaleax requested a review from Fishrock123 October 8, 2019 23:26

nodejs-github-bot added c++ Issues and PRs that require attention from people who are familiar with C++. fs Issues and PRs related to the fs subsystem / file system. labels Oct 8, 2019

addaleax added the performance Issues and PRs related to the performance of Node.js. label Oct 8, 2019

devnexen reviewed Oct 9, 2019

View reviewed changes

src/node_dir.cc Outdated Show resolved Hide resolved

src/node_dir.h Show resolved Hide resolved

cjihrig approved these changes Oct 9, 2019

View reviewed changes

devnexen approved these changes Oct 9, 2019

View reviewed changes

addaleax added the author ready PRs that have at least one approval, no pending requests for changes, and a CI started. label Oct 9, 2019

BridgeAR reviewed Oct 9, 2019

View reviewed changes

benchmark/fs/bench-opendir.js Outdated Show resolved Hide resolved

lib/internal/fs/dir.js Outdated Show resolved Hide resolved

lib/internal/fs/dir.js Outdated Show resolved Hide resolved

Fishrock123 reviewed Oct 9, 2019

View reviewed changes

src/node_dir.cc Outdated Show resolved Hide resolved

Fishrock123 mentioned this pull request Oct 9, 2019

fs: remove options.encoding from Dir.read*() #29908

Closed

3 tasks

addaleax added blocked PRs that are blocked by other issues or PRs. and removed author ready PRs that have at least one approval, no pending requests for changes, and a CI started. labels Oct 9, 2019

addaleax force-pushed the opendir-buffering branch from 0ccb46c to 4165094 Compare October 9, 2019 22:39

addaleax removed the blocked PRs that are blocked by other issues or PRs. label Oct 9, 2019

jasnell approved these changes Oct 9, 2019

View reviewed changes

Fishrock123 suggested changes Oct 10, 2019

View reviewed changes

lib/internal/fs/dir.js Outdated Show resolved Hide resolved

addaleax added 3 commits October 10, 2019 21:48

fixup! fs: buffer dir entries in opendir()

6d209fc

doc: fs dir modifications may not be reflected by dir.read

c66840d

addaleax force-pushed the opendir-buffering branch from 4165094 to c66840d Compare October 10, 2019 19:50

fixup! fixup! fs: buffer dir entries in opendir()

e2f1071

fixup! fs: buffer dir entries in opendir()

59cf84d

Trott requested a review from Fishrock123 October 11, 2019 01:44

Fishrock123 suggested changes Oct 11, 2019

View reviewed changes

fixup! fixup! fs: buffer dir entries in opendir()

751a8b6

addaleax force-pushed the opendir-buffering branch from c23f280 to 751a8b6 Compare October 11, 2019 09:12

Fishrock123 approved these changes Oct 11, 2019

View reviewed changes

addaleax added the author ready PRs that have at least one approval, no pending requests for changes, and a CI started. label Oct 11, 2019

addaleax closed this Oct 11, 2019

addaleax deleted the opendir-buffering branch October 11, 2019 21:10

addaleax mentioned this pull request Oct 12, 2019

fs.Dir.read() is very slow #29941

Closed

BethGriggs mentioned this pull request Oct 21, 2019

2019-10-22, Version 13.0.0 (Current) #29504

Merged

targos mentioned this pull request Nov 10, 2019

v12.13.1 release proposal #30352

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fs: buffer dir entries in opendir() #29893

fs: buffer dir entries in opendir() #29893

addaleax commented Oct 8, 2019

nodejs-github-bot commented Oct 8, 2019

mscdex commented Oct 9, 2019 •

edited

Loading

addaleax commented Oct 9, 2019

nodejs-github-bot commented Oct 9, 2019

BridgeAR left a comment

Fishrock123 commented Oct 9, 2019

addaleax commented Oct 9, 2019

nodejs-github-bot commented Oct 9, 2019

jasnell commented Oct 9, 2019 •

edited

Loading

Fishrock123 left a comment

Fishrock123 commented Oct 10, 2019

addaleax commented Oct 10, 2019

Fishrock123 Oct 11, 2019

addaleax Oct 11, 2019

Fishrock123 Oct 11, 2019

addaleax Oct 11, 2019

nodejs-github-bot commented Oct 11, 2019

addaleax commented Oct 11, 2019

fs: buffer dir entries in opendir() #29893

fs: buffer dir entries in opendir() #29893

Conversation

addaleax commented Oct 8, 2019

Checklist

nodejs-github-bot commented Oct 8, 2019

mscdex commented Oct 9, 2019 • edited Loading

addaleax commented Oct 9, 2019

nodejs-github-bot commented Oct 9, 2019

BridgeAR left a comment

Choose a reason for hiding this comment

Fishrock123 commented Oct 9, 2019

addaleax commented Oct 9, 2019

nodejs-github-bot commented Oct 9, 2019

jasnell commented Oct 9, 2019 • edited Loading

Fishrock123 left a comment

Choose a reason for hiding this comment

Fishrock123 commented Oct 10, 2019

addaleax commented Oct 10, 2019

Fishrock123 Oct 11, 2019

Choose a reason for hiding this comment

addaleax Oct 11, 2019

Choose a reason for hiding this comment

Fishrock123 Oct 11, 2019

Choose a reason for hiding this comment

addaleax Oct 11, 2019

Choose a reason for hiding this comment

nodejs-github-bot commented Oct 11, 2019

addaleax commented Oct 11, 2019

mscdex commented Oct 9, 2019 •

edited

Loading

jasnell commented Oct 9, 2019 •

edited

Loading