From b76a2e502c6f227f7df5f35ac3bbb0dcb2a8372d Mon Sep 17 00:00:00 2001 From: Jeremiah Senkpiel Date: Tue, 27 Aug 2019 17:14:27 -0700 Subject: [PATCH] fs: introduce `opendir()` and `fs.Dir` This adds long-requested methods for asynchronously interacting and iterating through directory entries by using `uv_fs_opendir`, `uv_fs_readdir`, and `uv_fs_closedir`. `fs.opendir()` and friends return an `fs.Dir`, which contains methods for doing reads and cleanup. `fs.Dir` also has the async iterator symbol exposed. The `read()` method and friends only return `fs.Dirent`s for this API. Having a entry type or doing a `stat` call is deemed to be necessary in the majority of cases, so just returning dirents seems like the logical choice for a new api. Reading when there are no more entries returns `null` instead of a dirent. However the async iterator hides that (and does automatic cleanup). The code lives in separate files from the rest of fs, this is done partially to prevent over-pollution of those (already very large) files, but also in the case of js allows loading into `fsPromises`. Due to async_hooks, this introduces a new handle type of `DIRHANDLE`. This PR does not attempt to make complete optimization of this feature. Notable future improvements include: - Moving promise work into C++ land like FileHandle. - Possibly adding `readv()` to do multi-entry directory reads. - Aliasing `fs.readdir` to `fs.scandir` and doing a deprecation. Refs: https://github.com/nodejs/node-v0.x-archive/issues/388 Refs: https://github.com/nodejs/node/issues/583 Refs: https://github.com/libuv/libuv/pull/2057 PR-URL: https://github.com/nodejs/node/pull/29349 Reviewed-By: Anna Henningsen Reviewed-By: David Carlier --- doc/api/errors.md | 6 + doc/api/fs.md | 216 ++++++++++- lib/fs.js | 27 +- lib/internal/errors.js | 1 + lib/internal/fs/dir.js | 201 ++++++++++ lib/internal/fs/promises.js | 2 + lib/internal/fs/utils.js | 52 ++- node.gyp | 3 + src/async_wrap.h | 1 + src/env.h | 1 + src/node_binding.cc | 1 + src/node_dir.cc | 350 ++++++++++++++++++ src/node_dir.h | 60 +++ src/node_file.cc | 90 ----- src/node_file.h | 89 +++++ test/parallel/test-bootstrap-modules.js | 2 + test/parallel/test-fs-opendir.js | 174 +++++++++ test/sequential/test-async-wrap-getasyncid.js | 7 + tools/doc/type-parser.js | 1 + 19 files changed, 1165 insertions(+), 119 deletions(-) create mode 100644 lib/internal/fs/dir.js create mode 100644 src/node_dir.cc create mode 100644 src/node_dir.h create mode 100644 test/parallel/test-fs-opendir.js diff --git a/doc/api/errors.md b/doc/api/errors.md index 4b92452cd362a0..329c369d152a30 100644 --- a/doc/api/errors.md +++ b/doc/api/errors.md @@ -824,6 +824,11 @@ A signing `key` was not provided to the [`sign.sign()`][] method. [`crypto.timingSafeEqual()`][] was called with `Buffer`, `TypedArray`, or `DataView` arguments of different lengths. + +### ERR_DIR_CLOSED + +The [`fs.Dir`][] was previously closed. + ### ERR_DNS_SET_SERVERS_FAILED @@ -2380,6 +2385,7 @@ such as `process.stdout.on('data')`. [`dgram.disconnect()`]: dgram.html#dgram_socket_disconnect [`dgram.remoteAddress()`]: dgram.html#dgram_socket_remoteaddress [`errno`(3) man page]: http://man7.org/linux/man-pages/man3/errno.3.html +[`fs.Dir`]: fs.html#fs_class_fs_dir [`fs.readFileSync`]: fs.html#fs_fs_readfilesync_path_options [`fs.readdir`]: fs.html#fs_fs_readdir_path_options_callback [`fs.symlink()`]: fs.html#fs_fs_symlink_target_path_type_callback diff --git a/doc/api/fs.md b/doc/api/fs.md index c22e169f1c3937..a6736d127e9bbc 100644 --- a/doc/api/fs.md +++ b/doc/api/fs.md @@ -284,13 +284,148 @@ synchronous use libuv's threadpool, which can have surprising and negative performance implications for some applications. See the [`UV_THREADPOOL_SIZE`][] documentation for more information. +## Class fs.Dir + + +A class representing a directory stream. + +Created by [`fs.opendir()`][], [`fs.opendirSync()`][], or [`fsPromises.opendir()`][]. + +Example using async interation: + +```js +const fs = require('fs'); + +async function print(path) { + const dir = await fs.promises.opendir(path); + for await (const dirent of dir) { + console.log(dirent.name); + } +} +print('./').catch(console.error); +``` + +### dir.path + + +* {string} + +The read-only path of this directory as was provided to [`fs.opendir()`][], +[`fs.opendirSync()`][], or [`fsPromises.opendir()`][]. + +### dir.close() + + +* Returns: {Promise} + +Asynchronously close the directory's underlying resource handle. +Subsequent reads will result in errors. + +A `Promise` is returned that will be resolved after the resource has been +closed. + +### dir.close(callback) + + +* `callback` {Function} + * `err` {Error} + +Asynchronously close the directory's underlying resource handle. +Subsequent reads will result in errors. + +The `callback` will be called after the resource handle has been closed. + +### dir.closeSync() + + +Synchronously close the directory's underlying resource handle. +Subsequent reads will result in errors. + +### dir.read([options]) + + +* `options` {Object} + * `encoding` {string|null} **Default:** `'utf8'` +* Returns: {Promise} containing {fs.Dirent} + +Asynchronously read the next directory entry via readdir(3) as an +[`fs.Dirent`][]. + +A `Promise` is returned that will be resolved with a [Dirent][] after the read +is completed. + +_Directory entries returned by this function are in no particular order as +provided by the operating system's underlying directory mechanisms._ + +### dir.read([options, ]callback) + + +* `options` {Object} + * `encoding` {string|null} **Default:** `'utf8'` +* `callback` {Function} + * `err` {Error} + * `dirent` {fs.Dirent} + +Asynchronously read the next directory entry via readdir(3) as an +[`fs.Dirent`][]. + +The `callback` will be called with a [Dirent][] after the read is completed. + +The `encoding` option sets the encoding of the `name` in the `dirent`. + +_Directory entries returned by this function are in no particular order as +provided by the operating system's underlying directory mechanisms._ + +### dir.readSync([options]) + + +* `options` {Object} + * `encoding` {string|null} **Default:** `'utf8'` +* Returns: {fs.Dirent} + +Synchronously read the next directory entry via readdir(3) as an +[`fs.Dirent`][]. + +The `encoding` option sets the encoding of the `name` in the `dirent`. + +_Directory entries returned by this function are in no particular order as +provided by the operating system's underlying directory mechanisms._ + +### dir\[Symbol.asyncIterator\]() + + +* Returns: {AsyncIterator} to fully iterate over all entries in the directory. + +_Directory entries returned by this iterator are in no particular order as +provided by the operating system's underlying directory mechanisms._ + ## Class: fs.Dirent -When [`fs.readdir()`][] or [`fs.readdirSync()`][] is called with the -`withFileTypes` option set to `true`, the resulting array is filled with +A representation of a directory entry, as returned by reading from an [`fs.Dir`][]. + +Additionally, when [`fs.readdir()`][] or [`fs.readdirSync()`][] is called with +the `withFileTypes` option set to `true`, the resulting array is filled with `fs.Dirent` objects, rather than strings or `Buffers`. ### dirent.isBlockDevice() @@ -2505,6 +2640,46 @@ Returns an integer representing the file descriptor. For detailed information, see the documentation of the asynchronous version of this API: [`fs.open()`][]. +## fs.opendir(path[, options], callback) + + +* `path` {string|Buffer|URL} +* `options` {Object} + * `encoding` {string|null} **Default:** `'utf8'` +* `callback` {Function} + * `err` {Error} + * `dir` {fs.Dir} + +Asynchronously open a directory. See opendir(3). + +Creates an [`fs.Dir`][], which contains all further functions for reading from +and cleaning up the directory. + +The `encoding` option sets the encoding for the `path` while opening the +directory and subsequent read operations (unless otherwise overriden during +reads from the directory). + +## fs.opendirSync(path[, options]) + + +* `path` {string|Buffer|URL} +* `options` {Object} + * `encoding` {string|null} **Default:** `'utf8'` +* Returns: {fs.Dir} + +Synchronously open a directory. See opendir(3). + +Creates an [`fs.Dir`][], which contains all further functions for reading from +and cleaning up the directory. + +The `encoding` option sets the encoding for the `path` while opening the +directory and subsequent read operations (unless otherwise overriden during +reads from the directory). + ## fs.read(fd, buffer, offset, length, position, callback) + +* `path` {string|Buffer|URL} +* `options` {Object} + * `encoding` {string|null} **Default:** `'utf8'` +* Returns: {Promise} containing {fs.Dir} + +Asynchronously open a directory. See opendir(3). + +Creates an [`fs.Dir`][], which contains all further functions for reading from +and cleaning up the directory. + +The `encoding` option sets the encoding for the `path` while opening the +directory and subsequent read operations (unless otherwise overriden during +reads from the directory). + +Example using async interation: + +```js +const fs = require('fs'); + +async function print(path) { + const dir = await fs.promises.opendir(path); + for await (const dirent of dir) { + console.log(dirent.name); + } +} +print('./').catch(console.error); +``` + ### fsPromises.readdir(path[, options])