From 4a3c5c3177d82e6f3f3ac9c97db186b2931e2f04 Mon Sep 17 00:00:00 2001 From: Jeremiah Senkpiel Date: Tue, 27 Aug 2019 17:14:27 -0700 Subject: [PATCH] fs: introduce `opendir()` and `fs.Dir` This adds long-requested methods for asynchronously interacting and iterating through directory entries by using `uv_fs_opendir`, `uv_fs_readdir`, and `uv_fs_closedir`. `fs.opendir()` and friends return an `fs.Dir`, which contains methods for doing reads and cleanup. `fs.Dir` also has the async iterator symbol exposed. The `read()` method and friends only return `fs.Dirent`s for this API. Having a entry type or doing a `stat` call is deemed to be necessary in the majority of cases, so just returning dirents seems like the logical choice for a new api. Reading when there are no more entries returns `null` instead of a dirent. However the async iterator hides that (and does automatic cleanup). The code lives in separate files from the rest of fs, this is done partially to prevent over-pollution of those (already very large) files, but also in the case of js allows loading into `fsPromises`. Due to async_hooks, this introduces a new handle type of `DIRHANDLE`. This PR does not attempt to make complete optimization of this feature. Notable future improvements include: - Moving promise work into C++ land like FileHandle. - Possibly adding `readv()` to do multi-entry directory reads. - Aliasing `fs.readdir` to `fs.scandir` and doing a deprecation. Refs: https://github.com/nodejs/node-v0.x-archive/issues/388 Refs: https://github.com/nodejs/node/issues/583 Refs: https://github.com/libuv/libuv/pull/2057 PR-URL: https://github.com/nodejs/node/pull/29349 Reviewed-By: Anna Henningsen Reviewed-By: David Carlier --- doc/api/fs.md | 165 ++++++++++++++++++------------- lib/internal/fs/dir.js | 60 +++-------- lib/internal/fs/promises.js | 54 +++++----- src/node_dir.cc | 122 +++++++++-------------- src/node_dir.h | 16 ++- src/node_file.h | 59 +++++++++-- test/parallel/test-fs-opendir.js | 62 +----------- 7 files changed, 247 insertions(+), 291 deletions(-) diff --git a/doc/api/fs.md b/doc/api/fs.md index 643efaae03..b58393cb0d 100644 --- a/doc/api/fs.md +++ b/doc/api/fs.md @@ -286,13 +286,14 @@ performance implications for some applications. See the ## Class fs.Dir A class representing a directory stream. -Created by [`fs.opendir()`][], [`fs.opendirSync()`][], or -[`fsPromises.opendir()`][]. +Created by [`fs.opendir()`][], [`fs.opendirSync()`][], or [`fsPromises.opendir()`][]. + +Example using async interation: ```js const fs = require('fs'); @@ -306,9 +307,19 @@ async function print(path) { print('./').catch(console.error); ``` +### dir.path + + +* {string} + +The read-only path of this directory as was provided to [`fs.opendir()`][], +[`fs.opendirSync()`][], or [`fsPromises.opendir()`][]. + ### dir.close() * Returns: {Promise} @@ -321,7 +332,7 @@ closed. ### dir.close(callback) * `callback` {Function} @@ -334,104 +345,84 @@ The `callback` will be called after the resource handle has been closed. ### dir.closeSync() Synchronously close the directory's underlying resource handle. Subsequent reads will result in errors. -### dir.path - - -* {string} - -The read-only path of this directory as was provided to [`fs.opendir()`][], -[`fs.opendirSync()`][], or [`fsPromises.opendir()`][]. - -### dir.read() +### dir.read([options]) -* Returns: {Promise} containing {fs.Dirent|null} +* `options` {Object} + * `encoding` {string|null} **Default:** `'utf8'` +* Returns: {Promise} containing {fs.Dirent} Asynchronously read the next directory entry via readdir(3) as an [`fs.Dirent`][]. -After the read is completed, a `Promise` is returned that will be resolved with -an [`fs.Dirent`][], or `null` if there are no more directory entries to read. +A `Promise` is returned that will be resolved with a [Dirent][] after the read +is completed. -Directory entries returned by this function are in no particular order as -provided by the operating system's underlying directory mechanisms. -Entries added or removed while iterating over the directory may or may not be -included in the iteration results. +_Directory entries returned by this function are in no particular order as +provided by the operating system's underlying directory mechanisms._ -### dir.read(callback) +### dir.read([options, ]callback) +* `options` {Object} + * `encoding` {string|null} **Default:** `'utf8'` * `callback` {Function} * `err` {Error} - * `dirent` {fs.Dirent|null} + * `dirent` {fs.Dirent} Asynchronously read the next directory entry via readdir(3) as an [`fs.Dirent`][]. -After the read is completed, the `callback` will be called with an -[`fs.Dirent`][], or `null` if there are no more directory entries to read. +The `callback` will be called with a [Dirent][] after the read is completed. -Directory entries returned by this function are in no particular order as -provided by the operating system's underlying directory mechanisms. -Entries added or removed while iterating over the directory may or may not be -included in the iteration results. +The `encoding` option sets the encoding of the `name` in the `dirent`. -### dir.readSync() +_Directory entries returned by this function are in no particular order as +provided by the operating system's underlying directory mechanisms._ + +### dir.readSync([options]) -* Returns: {fs.Dirent|null} +* `options` {Object} + * `encoding` {string|null} **Default:** `'utf8'` +* Returns: {fs.Dirent} Synchronously read the next directory entry via readdir(3) as an [`fs.Dirent`][]. -If there are no more directory entries to read, `null` will be returned. +The `encoding` option sets the encoding of the `name` in the `dirent`. -Directory entries returned by this function are in no particular order as -provided by the operating system's underlying directory mechanisms. -Entries added or removed while iterating over the directory may or may not be -included in the iteration results. +_Directory entries returned by this function are in no particular order as +provided by the operating system's underlying directory mechanisms._ ### dir\[Symbol.asyncIterator\]() -* Returns: {AsyncIterator} of {fs.Dirent} +* Returns: {AsyncIterator} to fully iterate over all entries in the directory. -Asynchronously iterates over the directory via readdir(3) until all entries have -been read. - -Entries returned by the async iterator are always an [`fs.Dirent`][]. -The `null` case from `dir.read()` is handled internally. - -See [`fs.Dir`][] for an example. - -Directory entries returned by this iterator are in no particular order as -provided by the operating system's underlying directory mechanisms. -Entries added or removed while iterating over the directory may or may not be -included in the iteration results. +_Directory entries returned by this iterator are in no particular order as +provided by the operating system's underlying directory mechanisms._ ## Class: fs.Dirent -A representation of a directory entry, as returned by reading from an -[`fs.Dir`][]. +A representation of a directory entry, as returned by reading from an [`fs.Dir`][]. Additionally, when [`fs.readdir()`][] or [`fs.readdirSync()`][] is called with the `withFileTypes` option set to `true`, the resulting array is filled with @@ -2701,6 +2692,46 @@ Returns an integer representing the file descriptor. For detailed information, see the documentation of the asynchronous version of this API: [`fs.open()`][]. +## fs.opendir(path[, options], callback) + + +* `path` {string|Buffer|URL} +* `options` {Object} + * `encoding` {string|null} **Default:** `'utf8'` +* `callback` {Function} + * `err` {Error} + * `dir` {fs.Dir} + +Asynchronously open a directory. See opendir(3). + +Creates an [`fs.Dir`][], which contains all further functions for reading from +and cleaning up the directory. + +The `encoding` option sets the encoding for the `path` while opening the +directory and subsequent read operations (unless otherwise overriden during +reads from the directory). + +## fs.opendirSync(path[, options]) + + +* `path` {string|Buffer|URL} +* `options` {Object} + * `encoding` {string|null} **Default:** `'utf8'` +* Returns: {fs.Dir} + +Synchronously open a directory. See opendir(3). + +Creates an [`fs.Dir`][], which contains all further functions for reading from +and cleaning up the directory. + +The `encoding` option sets the encoding for the `path` while opening the +directory and subsequent read operations (unless otherwise overriden during +reads from the directory). + ## fs.read(fd, buffer, offset, length, position, callback) * `path` {string|Buffer|URL} * `options` {Object} * `encoding` {string|null} **Default:** `'utf8'` - * `bufferSize` {number} Number of directory entries that are buffered - internally when reading from the directory. Higher values lead to better - performance but higher memory usage. **Default:** `32` * Returns: {Promise} containing {fs.Dir} Asynchronously open a directory. See opendir(3). @@ -4885,9 +4909,10 @@ Creates an [`fs.Dir`][], which contains all further functions for reading from and cleaning up the directory. The `encoding` option sets the encoding for the `path` while opening the -directory and subsequent read operations. +directory and subsequent read operations (unless otherwise overriden during +reads from the directory). -Example using async iteration: +Example using async interation: ```js const fs = require('fs'); @@ -4901,7 +4926,7 @@ async function print(path) { print('./').catch(console.error); ``` -### fsPromises.readdir(path\[, options\]) +### fsPromises.readdir(path[, options])